Second-Order Information and Applications

Kristof Schröder will talk about second-order information and its applications in machine learning.

Abstract

Second-order information, primarily represented through Hessians or curvature matrices, offers profound insights beyond mere gradient information. Incorporating this kind of information typically results in a substantially higher computational cost. To alleviate this, common strategies involve utilizing low-rank approximations or introducing elements of randomness to manage complexity. In this presentation, we will explore recent applications in the field that highlight the utility and scalability of second-order algorithms.

References

[Liu23S]

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, Hong Liu, Zhiyuan Li, David Hall, Percy Liang, Tengyu Ma.

May 2023

Given the massive cost of language model pre-training, a non-trivial improvement of the optimization algorithm would lead to a material reduction on the time and cost of training. Adam and its variants have been state-of-the-art for years, and more sophisticated second-order (Hessian-based) optimizers often incur too much per-step overhead. In this paper, we propose Sophia, Second-order Clipped …

[Zha24S]

Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer, Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, Ivor W. Tsang.

Feb 2024

Fine-tuning large language models (LLMs) with classic first-order optimizers entails prohibitive GPU memory due to the backpropagation process. Recent works have turned to zeroth-order optimizers for fine-tuning, which save substantial memory by using two forward passes. However, these optimizers are plagued by the heterogeneity of parameter curvatures across different dimensions. In this work, we …

[Rat24C]

Challenges in Training PINNs: A Loss Landscape Perspective, Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, Madeleine Udell.

Feb 2024

This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing …

Abstract

References

In this series →