Abstract

Hyperparameter optimization (HPO) methods are usually compared by the best score they reach after a fixed number of trials. That number says nothing about how quickly a method improves as time, money or energy is spent. This paper surveys HPO methods with cost in mind, points out that the score-versus-cost record of a search is rarely reported, and suggests reporting it as a Performance-per-Cost Curve (PpCC) with a few summary numbers. To see whether this matters, four methods (grid search, random search, Gaussian-process Bayesian optimization and the tree-structured Parzen estimator) were run on three classification tasks, ten times each. The two Bayesian methods got the highest cross-validation scores, and the tree-structured Parzen estimator beat grid and random search by a clear margin on two of the three datasets. On held-out test data, though, the differences were small and mostly within chance. On one dataset the ranking of the methods changed depending on the budget. Equal numbers of trials also did not mean equal cost: total search time differed by up to about 13% between methods. The extra computation of the Bayesian methods was under 1% of total cost here. The results argue for reporting the whole score-versus-cost record of a search and not just its final score.

References

• F. Hutter, L. Kotthoff, and J. Vanschoren, Eds., Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.
• R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Proc. IJCAI, 1995, pp. 1137–1143.
• M. Feurer and F. Hutter, “Hyperparameter optimization,” in Automated Machine Learning. Springer, 2019, pp. 3–33.
• J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res., vol. 13, pp. 281–305, 2012.
• B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas, “Taking the human out of the loop: A review of Bayesian optimization,” Proc. IEEE, vol. 104, no. 1, pp. 148–175, 2016.
• L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” J. Mach. Learn. Res., vol. 18, no. 185, pp. 1–52, 2018.
• S. Falkner, A. Klein, and F. Hutter, “BOHB: Robust and efficient hyperparameter optimization at scale,” in Proc. ICML, 2018, pp. 1437–1446.
• M. Feurer, J. T. Springenberg, and F. Hutter, “Initializing Bayesian hyperparameter optimization via meta-learning,” in Proc. AAAI, 2015, pp. 1128–1135.
• E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in NLP,” in Proc. ACL, 2019, pp. 3645–3650.
• C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. MIT Press, 2006.
• J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Proc. NIPS, 2012, pp. 2951–2959.
• J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” in Proc. NIPS, 2011, pp. 2546–2554.
• F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in Proc. LION, 2011, pp. 507–523.
• K. Swersky, J. Snoek, and R. P. Adams, “Freeze-thaw Bayesian optimization,” arXiv:1406.3896, 2014.
• K. Jamieson and A. Talwalkar, “Non-stochastic best arm identification and hyperparameter optimization,” in Proc. AISTATS, 2016, pp. 240–248.
• K. Eggensperger, F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Efficient benchmarking of hyperparameter optimizers via surrogates,” in Proc. AAAI, 2015, pp. 1114–1120.
• T. Domhan, J. T. Springenberg, and F. Hutter, “Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves,” in Proc. IJCAI, 2015, pp. 3460–3468.
• R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Commun. ACM, vol. 63, no. 12, pp. 54–63, 2020.
• C. Coleman et al., “DAWNBench: An end-to-end deep learning benchmark and competition,” in NIPS ML Systems Workshop, 2017.
• P. Mattson et al., “MLPerf training benchmark,” in Proc. MLSys, 2020.
• F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
• L. Breiman, “Random forests,” Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001.
• T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in Proc. KDD, 2019, pp. 2623–2631.
• F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945.
• G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation,” J. Mach. Learn. Res., vol. 11, pp. 2079–2107, 2010.
• K. Eggensperger et al., “HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO,” in Proc. NeurIPS Datasets and Benchmarks Track, 2021.
• C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “NAS-Bench-101: Towards reproducible neural architecture search,” in Proc. ICML, 2019, pp. 7105–7114.
• M. Mitchell et al., “Model cards for model reporting,” in Proc. ACM FAT*, 2019, pp. 220–229.