Survey on meta-learning-based hyperparameter optimization

Abstract:

Hyperparameter optimization (HPO) is a pivotal technology in Automated Machine Learning, aiming to automate the tuning process and alleviate the burden on practitioners. In robotic systems, HPO plays a critical role in enhancing neural network training for perception modules, controller parameter calibration, and performance optimization of multimodal data fusion algorithms. However, despite significant progress, efficiency remains the primary bottleneck limiting its widespread adoption. Recent advances in meta-learning have opened new avenues for improving HPO efficiency, particularly demonstrating unique advantages in robotic systems that require rapid adaptation to dynamic environments and novel task scenarios. This technique enables models to automatically assimilate and apply knowledge from prior tasks, thereby improving learning efficiency for unseen tasks. Currently, researchers are actively exploring meta-learning techniques to enhance HPO search capabilities. In this paper we aim to provide a systematic overview of relevant research. First, we provide a formal definition of the HPO problem and review state-of-the-art methods. Subsequently, we systematically summarize meta-learning-based HPO strategies and analyze prevailing meta-learning algorithms. Furthermore, we introduce benchmark datasets in HPO research and compare the performance of mainstream methods. Finally, we discuss future research directions in hyperparameter optimization technology.