Purpose: This cross-sectional study investigated the association between hilliness and walking speed in community-dwelling older adults, and whether it varied according to their car-driving status. Methods: Data were collected from 590 participants aged 65 and older living in Okinoshima Town, Shimane prefecture, Japan, in 2018. Comfortable walking speed (m/s) was objectively assessed. Hilliness was measured by the mean land slope (degree) within a 500-m or 1000-m network buffer around each participant's home using a geographic information system. A multiple linear regression examined whether the land slope was associated with walking speed, adjusted for sex, age, body mass index, smoking habits, alcohol consumption habits, exercise habits, chronic disease, and living arrangements. A stratified analysis by car-driving status was also conducted. Results: After adjusting for all confounders, the land slope within the 500-m or 1000-m network buffer was negatively associated with walking speed (B = -0.007, 95% CI [-0.011, -0.002]; B = -0.007, 95% CI [-0.011, -0.003], respectively). The stratified analysis by car-driving status showed that living in a hilly area was negatively associated with walking speed among non-drivers in the 500-m or 1000-m network buffer (B = -0.011, 95% CI [-0.017, -0.004]; B = -0.012, 95% CI [-0.019, -0.006]), though there were no associations among drivers. Conclusions: A hilly environment is positively associated with slow walking speed in community-dwelling older adults in Japan. Moreover, car-driving status potentially modifies the relationship between living in a hilly environment and slow walking speed.