Tuning Analysis

This page provides an analysis of the hyperparameter sweeps to get a handle on how different tuning methods are performing. We use these results from early data sets to select the most efficient strategies for other data sets.

All results are validation error.

Trial Performance

These charts display the maximum performance so far as the search progresses through its trials. The vertical lines are at 60 trials.

RBP

NDCG

Note

Intelligent search methods (HyperOpt and Optuna) are searching for RBP, so the NDCG performance may not be fully reflective.

Loss

Our goal is to determine where to stop searching, and which methods are more efficient at searching the space. To better assess this, let’s look at the loss relative to each method’s best performance if we stop at each trial point.

Loss drops pretty quick for intelligent searches, going under 5% by about 30 iterations on ML10M.