The accurate estimation of grain yield in rice breeding is crucial for breeders to screen and select qualified cultivars. In this study, a low-cost unmanned aerial vehicle (UAV) platform mounted with an RGB camera was carried out to capture high-spatial resolution images of rice canopy in rice breeding. The random forest (RF) regression techniques were used to establish yield models by using (1) only color vegetation indices (VIs), (2) only phenological data, and (3) fusion of VIs and phenological data as inputs, respectively. Then, the performances of RF models were compared with the manual observation and CERES-Rice model. The results indicated that the RF model using VIs only performed poorly for estimating yield; the optimized RF model that combined the use of phenological data and color VIs performed much better, which demonstrated that the phenological data significantly improved the model performance. Furthermore, the yield estimation accuracy of 21 rice cultivars that were continuously planted over three years in the optimal RF model had no significant difference (p > 0.05) with that of the CERES-Rice model. These findings demonstrate that the RF model, by combining phenological data and color Vis, is a potential and cost-effective way to estimate yield in rice breeding.