Gradient Boosting and bagging applied to regressors can reduce the error due to bias and variance respectively. Alternatively, Stochastic Gradient Boosting (SGB) and Iterated Bagging (IB) attempt to simultaneously reduce the contribution of both bias and variance to error. We provide an extensive empirical analysis of these methods, along with two alternate bias-variance reduction approaches --- bagging Gradient Boosting (BagGB) and bagging Stochastic Gradient Boosting (BagSGB). Experimental results demonstrate that SGB does not perform as well as IB or the alternate approaches. Furthermore, results show that, while BagGB and BagSGB perform competitively for low-bias learners, in general, Iterated Bagging is the most effective of these methods.

PDF, PS

In Proceedings of the 16th European Conference on Machine Learning, pp. 741-749, Porto, Portugal, October 2005.

@InProceedings{suen:ecml05, title={Combining Bias and Variance Reduction Techniques for Regression}, author={Y. L. Suen and P. Melville and Raymond J. Mooney}, booktitle={Proceedings of the 16th European Conference on Machine Learning}, month={October}, address={Porto, Portugal}, pages={741-749}, url="http://www.cs.utexas.edu/users/ai-lab?suen:ecml05", year={2005} }