The Uluguru Mountains, which are part of the Eastern Arc Mountains, is a hotspot for diversity and an important area for the conservation of biodiversity in Tanzania, but faces increasing disturbance due to anthropogenic activities leading to a high rate of forest degradation and fragmentation. Changes in habitat structure have caused significant changes in the faunal assemblage of the region. However, information on these effects in the Uluguru Mountains is currently lacking. We conducted a survey of the small rodent assemblage on the Uluguru Mountains in three different habitats: forest, fallow, and cultivated land along two elevation gradients: 900-1,400 and 1,500-2,000 m a.s.l. Additionally, we recorded the habitat structure for each transect by looking at grass cover, tree density and shrub density. Generalised linear mixed models were used to examine the effects of habitat structure on both species richness as well as abundance. Our results revealed that, even while species richness remained similar over the whole study area, the species composition significantly changed depending on habitat structure. This finding arose particularly from changes in the numbers of Praomys delectorum and Mastomys natalensis, with the latter more abundant in cultivated and fallow land but not in forest, where P. delectorum is more abundant. This outcome may indicate that an increase in forest degradation and expansion of agriculture could have an impact on rodent assemblage and potentially on their population dynamics.