HorovodPlugin

class pytorch_lightning.plugins.training_type.HorovodPlugin(parallel_devices=None, checkpoint_io=None)

Bases: pytorch_lightning.plugins.training_type.parallel.ParallelPlugin

Plugin for Horovod distributed training integration.

all_gather(result, group=None, sync_grads=False)

Perform a all_gather on all processes.

Return type

Tensor

barrier(*args, **kwargs)

Synchronizes all processes which blocks processes until the whole group enters this function.

Parameters

name – an optional name to pass into barrier.

broadcast(obj, src=0)

Broadcasts an object to all processes.

Parameters

• obj (object) – the object to broadcast

• src (int) – source rank

Return type

object

model_to_device()

Moves the model to the correct device.

post_backward(closure_loss)

Run after precision plugin executes backward.

Return type

None

pre_dispatch()

Hook to do something before the training/evaluation/prediction starts.

reduce(tensor, group=None, reduce_op='mean')

Reduces a tensor from several distributed processes to one aggregated tensor.

Parameters

• tensor – the tensor to sync and reduce

• group (Optional[Any]) – the process group to gather results from. Defaults to all processes (world)

• reduce_op (Union[ReduceOp, str, None]) – the reduction operation. Defaults to ‘mean’/’avg’. Can also be a string ‘sum’ to calculate the sum during reduction.

Returns

reduced value, except when the input was not a tensor the output remains is unchanged

setup()

Called by the accelerator to finish setup.

Return type

None

teardown()

This method is called to teardown the training process.

It is the right place to release memory and free other resources.

Return type

None

property root_device

Return the root device.