SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning

JunHoo Lee¹,Jayeon Yoo¹,Nojun Kwak¹

NeurIPS 2023

¹Seoul National University

The teaser summarizes the core setup of SHOT. In gradient-based meta-learning, every inner loop begins with an unseen sampled task and must adapt in only a few large-step updates. SHOT explicitly targets this difficult regime by suppressing trajectory distortion caused by the Hessian rather than leaving that suppression implicit.

Abstract

Gradient-based meta-learning (GBML) adapts to a new task with only a few inner-loop updates, yet those updates use very large learning rates and therefore should be highly sensitive to high-order terms such as the Hessian.

SHOT starts from the hypothesis that successful GBML already suppresses the Hessian implicitly along the optimization trajectory. It then makes that prior explicit by minimizing the distance between a fast target model and a smoother reference model that starts from the same initialization and support set but uses more, smaller inner-loop steps.

The resulting objective is algorithm-agnostic, architecture-agnostic, and lightweight in practice. Across standard few-shot benchmarks, cross-domain evaluation, and Hessian-free or one-step GBML variants, SHOT consistently improves adaptation quality while keeping the same test-time inference cost as the baseline.

Method Overview

SHOT frames the GBML inner loop as a fast model: it has to reach a task-specific solution in one to a few gradient steps, so curvature terms can dominate the optimization dynamics much more strongly than in ordinary deep learning.

The method therefore introduces a reference trajectory that is less distorted by the Hessian. The target model takes the usual aggressive few-step path, while the reference model follows the same task with more granular updates. The outer loop minimizes the distance between their endpoints, turning trajectory distortion into a directly optimizable signal.

In the efficient SHOT setting used in the paper, this can be added with one extra forward pass and no extra backward pass in the common case. The method is only used during training, so the baseline and SHOT share the same inference-time cost.

\theta_\tau^\star = \underset{\theta}{\arg\min} \sum_{(x, y) \in \mathcal{X}_{s}^{\tau}} L(x, y \mid \theta; \theta_0), \qquad \theta^{k+1} = \theta^k - \alpha \nabla_{\theta} L(\theta^k)

\theta_0^\star = \underset{\theta_0}{\arg\min} \sum_{\tau} \sum_{(x, y) \in \mathcal{X}_{t}^{\tau}} L(x, y \mid \theta_\tau^\star; \theta_0)

L_{\text{SHOT}} = D(\theta_t^T, \theta_r^R), \qquad \alpha_r = \frac{T}{R}\alpha_t, \qquad T < R

Main Results

SHOT improves MAML on all four 4-Conv benchmarks reported in the paper: miniImageNet, tieredImageNet, Cars, and CUB. The strongest 5-shot gain is on tieredImageNet, where SHOT raises accuracy from 66.12 to 69.08, and Cars also improves from 62.24 to 64.84.

The benchmark summary below is derived directly from the SHOT paper tables and keeps the same single-image presentation style as the template page.

Transfer & Variants

SHOT is not tied to a single benchmark or a single GBML formulation. In 5-shot cross-domain evaluation, SHOT improves tieredImageNet to miniImageNet from 66.86 to 70.70 and Cars to miniImageNet from 37.23 to 40.79.

The same idea also carries over to Hessian-free and one-step variants such as FoMAML and BOIL, supporting the paper's claim that Hessian suppression is a broadly useful prior rather than a MAML-specific trick.

Transfer and variant summary table for SHOT.

Analysis

SHOT changes the geometry of inner-loop optimization in a way that is visible throughout training. The validation and cosine-similarity curves show that both SHOT variants reach a high-alignment regime much earlier than the MAML baseline, and that alignment closely tracks adaptation quality.

The training-epoch comparison further supports the paper's core hypothesis: successful gradient-based meta-learning benefits from keeping the inner-loop path close to a more stable direction, rather than letting the optimization trajectory bend sharply under Hessian effects.

The loss-surface visualization then makes that claim tangible. Compared with a random initialization, the SHOT pretrained model exhibits a much more linear landscape along the optimization trajectory, showing that Hessian-induced distortion has been explicitly suppressed.

\int_0^1 \nabla L(\theta(t)) \cdot \nabla L(\theta^k)\, dt \approx \int_0^1 \left( \lVert \nabla L(\theta^k) \rVert_2^2 - \alpha t \, \nabla L(\theta^k)^\top H(\theta^k)\nabla L(\theta^k) \right) dt > 0

Validation accuracy and cosine similarity analysis for SHOT.

BibTeX

@inproceedings{lee2023shot,
      title={SHOT: Suppressing the Hessian along the Optimization Trajectory for Gradient-Based Meta-Learning},
      author={JunHoo Lee and Jayeon Yoo and Nojun Kwak},
      booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
      year={2023},
      url={https://arxiv.org/abs/2310.02751},
}