Using the D-optimal mixture design method, taking the agronomic properties, fruit quality and yield of cucumber plants as evaluation indicators, the effects of different ratios of Volvariella volvacea residue, peat and perlite on cucumber cultivation were explored, the regression model between the ratio of each substrate and the response value was established, and the effects of the interaction of each substrate in the formula on the response value of cucumber were investigated. The test results show that Volvariella volvacea residue is rich in nutrients, but electrical conductivity (EC) and the potential of hydrogen (pH) are higher. The peat is acidic, has strong water retention and air permeability, can lower pH, and perlite has strong water absorption and air permeability. EC and pH of the matrix compounded with peat and perlite are lower, while EC and pH of the compound matrix containing Volvariella volvacea residue are increased with the increase of the added amount of Volvariella volvacea residue. The final optimized substrate ratio is Volvariella volvacea residue 32.02%, peat 60.00% and perlite 7.98%. It has been verified that, compared with peat, plant height, stem diameter, vitamin C and soluble sugar contents and yield per plant are 350.00 cm, 11.74 mm and 101.01 mg/kg 1.313%, 2 123.33 g for the optimized formula, increased by 20.69%, 1.73%, 12.79%, 14.79% and 41.56%, respectively, while nitrate content is 351.04 mg/kg, reduced by 14.66%, and production cost is reduced by about 33%. This study shows that optimizing Volvariella volvacea residue composite substrate through the mixture design can solve the problems of high EC and pH in a single substrate, and the cultivation effect is better, thus, applying Volvariella volvacea residue in soilless cultivation of cucumbers is completely feasible.