The characteristics of heavy metal pollution in 0–20 cm soil layers of the farmlands surrounding a nonferrous metal smelter in central southern China were assessed from three directions: northwest, southwest and southeast. The correlation analyses among heavy metal elements and between heavy metal concentrations and the distance to the smelter were conducted to identify metal spatial distribution and the pollution source. The results showed that the soils in this area were mainly acidic and seriously contaminated by Cd. Approximately 60.9% and 86.1% of the soils exceeded local soil environmental background values and the National Soil Environmental Quality Standards, respectively. The soil was polluted lightly by Cu, Zn, Cr and Ni, but not polluted by Pb. According to the “Nemerow comprehensive pollution index”, only 8.7% of the samples were within the level of “security”, which suggested that crop production in this area faces serious risk of heavy metal contamination. One factor analysis of variance demonstrated that the difference of heavy metal concentrations in dry land and paddy was not significantly different, indicating that farming methods are not the main cause of the different distribution of heavy metals in this area. Soil Cd, Pb and Zn were significantly correlated and showed remarkable negative relationship with the distance to the smelter. Exponential and power functions can be used to fit heavy metal distribution patterns with the distance, indicating that these pollutants might come from the high metal containing dust of the smelter.