L2R Callbacks

General purpose callbacks needed for L2R learner

Essential

---

source

TrainEval

 TrainEval (after_create=None, before_fit=None, before_epoch=None,
            before_train=None, before_batch=None, after_pred=None,
            after_loss=None, before_backward=None,
            after_cancel_backward=None, after_backward=None,
            before_step=None, after_cancel_step=None, after_step=None,
            after_cancel_batch=None, after_batch=None,
            after_cancel_train=None, after_train=None,
            before_validate=None, after_cancel_validate=None,
            after_validate=None, after_cancel_epoch=None,
            after_epoch=None, after_cancel_fit=None, after_fit=None)

Callback that tracks the number of iterations done and properly sets training/eval mode

Tracking

---

source

AvgSmoothMetric

 AvgSmoothMetric (beta=0.98)

Smooth average of the losses (exponentially weighted with beta)

class _TstLearner(): pass
learn = _TstLearner()
_data = torch.randn(24)

tst = AvgSmoothMetric()
tst.reset()
val = tensor(0.)
for i,o in enumerate(_data): 
    learn.moi = o
    tst.accumulate(learn)
    val = val*0.98 + o*(1-0.98)
    test_close(val/(1-0.98**(i+1)), tst.value)

---

source

TrackResults

 TrackResults (train_metrics=False, beta=0.98)

Basic class handling tweaks of the training loop by changing a Learner in various events

---

source

Monitor

 Monitor (monitor='ndcg_at_6', comp=None, min_delta=0.0,
          reset_on_fit=False)

Basic class handling tweaks of the training loop by changing a Learner in various events

---

source

XParamScheduler

 XParamScheduler (scheds)

Schedule hyper-parameters according to scheds

scheds is a dictionary with one key for each hyper-parameter you want to schedule, with either a scheduler or a list of schedulers as values (in the second case, the list must have the same length as the the number of parameters groups of the optimizer).

---

source

XParamScheduler.before_fit

 XParamScheduler.before_fit ()

Initialize container for hyper-parameters

---

source

XParamScheduler.before_batch

 XParamScheduler.before_batch ()

Set the proper hyper-parameters in the optimizer

---

source

XParamScheduler.after_batch

 XParamScheduler.after_batch ()

Record hyper-parameters of this batch

---

source

XParamScheduler.after_fit

 XParamScheduler.after_fit ()

Save the hyper-parameters in the track_results if there is one

---

source

L2RLearner.fit_one_cycle

 L2RLearner.fit_one_cycle (n_epoch, lr_max=None, div=25.0,
                           div_final=100000.0, pct_start=0.25, moms=None,
                           cbs=None)

Fit self.model for n_epoch using the 1cycle policy.

The 1cycle policy was introduced by Leslie N. Smith et al. in Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates. It schedules the learning rate with a cosine annealing from lr_max/div to lr_max then lr_max/div_final (pass an array to lr_max if you want to use differential learning rates) and the momentum with cosine annealing according to the values in moms. The first phase takes pct_start of the training. You can optionally pass additional cbs and reset_opt.

---

source

TrackResults.plot_sched

 TrackResults.plot_sched (keys=None, figsize=None)

XLRFind

---

source

XLRFinder

 XLRFinder (start_lr=1e-07, end_lr=10, num_it=100, stop_div=True)

Training with exponentially growing learning rate

---

source

TrackResults.plot_xlr_find

 TrackResults.plot_xlr_find (skip_end=5, return_fig=True,
                             suggestions=None, nms=None, **kwargs)

Plot the result of an LR Finder test (won’t work if you didn’t do learn.xlr_find() before)

---

source

L2RLearner.xrl_find

 L2RLearner.xrl_find (start_lr=1e-05, end_lr=0.1, num_it=400,
                      stop_div=True, show_plot=True,
                      suggest_funcs=(<function valley at
                      0x7f2f4c36d670>,))

Launch a mock training to find a good learning rate and return suggestions based on suggest_funcs as a named tuple

Progress

---

source

ProgressBarCallback

 ProgressBarCallback (after_create=None, before_fit=None,
                      before_epoch=None, before_train=None,
                      before_batch=None, after_pred=None, after_loss=None,
                      before_backward=None, after_cancel_backward=None,
                      after_backward=None, before_step=None,
                      after_cancel_step=None, after_step=None,
                      after_cancel_batch=None, after_batch=None,
                      after_cancel_train=None, after_train=None,
                      before_validate=None, after_cancel_validate=None,
                      after_validate=None, after_cancel_epoch=None,
                      after_epoch=None, after_cancel_fit=None,
                      after_fit=None)

Basic class handling tweaks of the training loop by changing a Learner in various events

Saving

---

source

SaveCallBack

 SaveCallBack (fname, monitor='ndcg_at_6', comp=None, min_delta=0.0,
               reset_on_fit=False)

Basic class handling tweaks of the training loop by changing a Learner in various events