基于元学习的超参数优化方法综述

Survey on meta-learning-based hyperparameter optimization

关键词:  自动机器学习  超参数优化  元学习  数据挖掘  机器学习

Key words:  automatic machine learning  hyperparameter optimization  meta-learning  data mining  machine learning

摘要:

超参数优化是自动机器学习领域中的关键技术之一,旨在通过实现超参数调优的自动化,减轻机器学习从业者的工作负担。在机器人系统中,超参数优化对感知模块的神经网络训练、控制器的参数整定以及多模态数据融合算法的性能提升具有关键作用。然而,尽管该技术已取得显著进展,但其效率问题仍是限制其广泛应用的主要瓶颈。近年来,元学习技术的迅猛发展为提升超参数优化的效率开辟了新路径,该技术在机器人系统需要快速适应动态环境及新任务场景时展现出独特优势。元学习的核心在于使模型能够从大量先验任务中自动吸收并应用相关知识,从而显著提升其对未知任务的学习效率。基于此,众多研究者正致力于探索如何利用元学习技术来增强超参数优化的搜索能力。本文旨在系统梳理相关研究进展:首先对超参数优化问题进行形式化定义,并综述当前主流方法;其次,总结基于元学习理论的超参数优化策略,并分析当前主流元学习算法的特点；再次,介绍超参数优化领域的基准数据集,并对比分析主流方法在其上的实验性能;最后,对超参数优化技术的未来发展趋势进行展望。

Abstract:

Hyperparameter optimization (HPO) is a pivotal technology in Automated Machine Learning, aiming to automate the tuning process and alleviate the burden on practitioners. In robotic systems, HPO plays a critical role in enhancing neural network training for perception modules, controller parameter calibration, and performance optimization of multimodal data fusion algorithms. However, despite significant progress, efficiency remains the primary bottleneck limiting its widespread adoption. Recent advances in meta-learning have opened new avenues for improving HPO efficiency, particularly demonstrating unique advantages in robotic systems that require rapid adaptation to dynamic environments and novel task scenarios. This technique enables models to automatically assimilate and apply knowledge from prior tasks, thereby improving learning efficiency for unseen tasks. Currently, researchers are actively exploring meta-learning techniques to enhance HPO search capabilities. In this paper we aim to provide a systematic overview of relevant research. First, we provide a formal definition of the HPO problem and review state-of-the-art methods. Subsequently, we systematically summarize meta-learning-based HPO strategies and analyze prevailing meta-learning algorithms. Furthermore, we introduce benchmark datasets in HPO research and compare the performance of mainstream methods. Finally, we discuss future research directions in hyperparameter optimization technology.