pytorch_lightning.trainer.evaluation_loop module

Validation loop

The lightning validation loop handles everything except the actual computations of your model. To decide what will happen in your validation loop, define the validation_step function. Below are all the things lightning automates for you in the validation loop.

Note

Lightning will run 5 steps of validation in the beginning of training as a sanity check so you don’t have to wait until a full epoch to catch possible validation issues.

Check validation every n epochs

If you have a small dataset you might want to check validation every n epochs

# DEFAULT
trainer = Trainer(check_val_every_n_epoch=1)

Set how much of the validation set to check

If you don’t want to check 100% of the validation set (for debugging or if it’s huge), set this flag.

limit_val_batches will be overwritten by overfit_batches if overfit_batches > 0

# DEFAULT
trainer = Trainer(limit_val_batches=1.0)

# check 10% only
trainer = Trainer(limit_val_batches=0.1)

Set how much of the test set to check

If you don’t want to check 100% of the test set (for debugging or if it’s huge), set this flag.

limit_test_batches will be overwritten by overfit_batches if overfit_batches > 0

# DEFAULT
trainer = Trainer(limit_test_batches=1.0)

# check 10% only
trainer = Trainer(limit_test_batches=0.1)

Set validation check frequency within 1 training epoch

For large datasets it’s often desirable to check validation multiple times within a training loop.

Pass in a float to check that often within 1 training epoch. Pass in an int k to check every k training batches. Must use an int if using an IterableDataset.

# DEFAULT
trainer = Trainer(val_check_interval=0.95)

# check every .25 of an epoch
trainer = Trainer(val_check_interval=0.25)

# check every 100 train batches (ie: for IterableDatasets or fixed frequency)
trainer = Trainer(val_check_interval=100)

Set the number of validation sanity steps

Lightning runs a few steps of validation in the beginning of training.

This avoids crashing in the validation loop sometime deep into a lengthy training loop.

# DEFAULT
trainer = Trainer(num_sanity_val_steps=5)

You can use Trainer(num_sanity_val_steps=0) to skip the sanity check.

# Testing loop

To ensure you don’t accidentally use test data to guide training decisions Lightning

makes running the test set deliberate.

test

You have two options to run the test set. First case is where you test right after a full training routine.

# run full training
trainer.fit(model)

# run test set
trainer.test()

Second case is where you load a model and run the test set

model = MyLightningModule.load_from_checkpoint(
    checkpoint_path='/path/to/pytorch_checkpoint.ckpt',
    hparams_file='/path/to/test_tube/experiment/version/hparams.yaml',
    map_location=None
)

# init trainer with whatever options
trainer = Trainer(...)

# test (pass in the model)
trainer.test(model)

In this second case, the options you pass to trainer will be used when running

the test set (ie: 16-bit, dp, ddp, etc…)

class pytorch_lightning.trainer.evaluation_loop.TrainerEvaluationLoopMixin

Bases: abc.ABC

_evaluate(model, dataloaders, max_batches, test_mode=False)

Run evaluation code.

Parameters

• model (LightningModule) – The model to evaluate.

• dataloaders (List[DataLoader]) – A list of PyTorch dataloaders.

• max_batches (Union[int, List[int]]) – An integer or list of integers with length of the number of dataloaders. Each entry is the number of batches to process in the corresponding dataloader.

• test_mode (bool) –

abstract add_progress_bar_metrics(*args)

Warning: this is just empty shell for code implemented in other class.

abstract copy_trainer_model_properties(*args)

Warning: this is just empty shell for code implemented in other class.

evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode=False)

abstract get_model()

Warning: this is just empty shell for code implemented in other class.

Return type

LightningModule

abstract is_overridden(*args)

Warning: this is just empty shell for code implemented in other class.

abstract log_metrics(*args)

Warning: this is just empty shell for code implemented in other class.

abstract reset_test_dataloader(*args)

Warning: this is just empty shell for code implemented in other class.

abstract reset_val_dataloader(*args)

Warning: this is just empty shell for code implemented in other class.

run_evaluation(test_mode=False)

abstract transfer_batch_to_gpu(*args)

Warning: this is just empty shell for code implemented in other class.

abstract transfer_batch_to_tpu(*args)

Warning: this is just empty shell for code implemented in other class.

callback_metrics: ... = None

current_epoch: int = None

data_parallel_device_ids: ... = None

fast_dev_run: ... = None

global_rank: int = None

model: LightningModule = None

num_test_batches: List[int] = None

num_val_batches: int = None

on_gpu: bool = None

on_test_batch_end: Callable = None

on_test_batch_start: Callable = None

on_test_end: Callable = None

on_test_start: Callable = None

on_validation_batch_end: Callable = None

on_validation_batch_start: Callable = None

on_validation_end: Callable = None

on_validation_start: Callable = None

process_output: ... = None

progress_bar_dict: ... = None

reload_dataloaders_every_epoch: ... = None

single_gpu: bool = None

test_dataloaders: DataLoader = None

tpu_id: int = None

use_ddp: bool = None

use_ddp2: bool = None

use_dp: bool = None

use_horovod: bool = None

use_tpu: bool = None

val_dataloaders: DataLoader = None

world_size: int = None