Improving the soil fertility of abandoned farmland is important for the efficient utilization of land resources. A pot experiment was conducted using soil collected from the abandoned farmland to compare the effect of Vicia villosa L. and Astragalus sinicus L. on following rice growth. The results showed that planting Vicia villosa L. in abandoned farmland significantly increase the subsequent rice biomass, comparing with control, the dry weight of rice grown after Vicia villosa L. and Astragalus sinicus L. was increased by 53% and decreased by 38%, respectively. Compared with control, the soil organic matter content significantly increased by 17.14% after planting Vicia villosa L., but the soil available phosphorus content significantly decreased by 43.32%; the soil pH and total nitrogen content increased by 4.56% and 11.43% after incorporating, respectively. After following rice growth, the soil organic matter content significantly increased by 26.32%. In contrast, no significant differences in different soil fertility indicators at different stages after planting Astragalus sinicus L. compared with the control treatment. A significant positive correlation was observed between rice dry weight and soil organic matter content, while a significant negative correlation was observed between rice dry weight and soil available P content. Independent with sampling time, soil organic matter content and available P content contributed mostly to rice dry weight. Moreover, the activities of β-glucosidase, β-cellulase, and phosphatase in the soil after planting Vicia villosa L. increased significantly, with a random forest model analysis, we revealed that carbon-converting enzymes and phosphatases had the greatest impact on rice dry weight. In summary, the planting of Vicia villosa L. in abandoned farmland results in enhanced enzymes activities related to soil carbon and phosphorus transformation, which further increases soil organic matter and available phosphorus content, and ultimately facilitate nutrient absorption and biomass increase in subsequent rice crops.