Here are some best practices to increase your performance.
When building your Dataloader set num_workers > 0 and pin_memory=True (only for GPUs).
Dataloader(dataset, num_workers=8, pin_memory=True)
The question of how many num_workers is tricky. Here’s a summary of some references, , and our suggestions.
num_workers=0 means ONLY the main process will load batches (that can be a bottleneck).
num_workers=1 means ONLY one worker (just not the main process) will load data but it will still be slow.
The num_workers depends on the batch size and your machine.
A general place to start is to set num_workers equal to the number of CPUs on that machine.
Increasing num_workers will ALSO increase your CPU memory consumption.
The best thing to do is to increase the nun_workers slowly and stop once you see no more improvement in your training speed.
When using distributed_backend=ddp_spawn (the ddp default) or TPU training, the way multiple GPUs/TPU cores are used is by calling .spawn() under the hood. The problem is that PyTorch has issues with num_workers > 0 when using .spawn(). For this reason we recommend you use distributed_backend=ddp so you can increase the num_workers, however your script has to be callable like so:
python my_program.py --gpus X
.item(), .numpy(), .cpu()¶
Don’t call .item() anywhere on your code. Use .detach() instead to remove the connected graph calls. Lightning takes a great deal of care to be optimized for this.
Don’t call this unnecessarily! Every time you call this ALL your GPUs have to wait to sync.
construct tensors directly on device¶
LightningModules know what device they are on! construct tensors on the device directly to avoid CPU->Device transfer.
# bad t = tensor.rand(2, 2).cuda() # good (self is lightningModule) t = tensor.rand(2,2, device=self.device)
Use DDP not DP¶
DP performs three GPU transfers for EVERY batch:
Copy model to device.
Copy data to device.
Copy outputs of each device back to master.
Whereas DDP only performs 1 transfer to sync gradients. Because of this, DDP is MUCH faster than DP.
Use 16-bit to decrease the memory (and thus increase your batch size). On certain GPUs (V100s, 2080tis), 16-bit calculations are also faster. However, know that 16-bit and multi-processing (any DDP) can have issues. Here are some common problems.
- CUDA error: an illegal memory access was encountered.
The solution is likely setting a specific CUDA, CUDNN, PyTorch version combination.
CUDA error: device-side assert triggered. This is a general catch-all error. To see the actual error run your script like so:
# won't see what the error is python main.py # will see what the error is CUDA_LAUNCH_BLOCKING=1 python main.py
We also recommend using 16-bit native found in PyTorch 1.6. Just install this version and Lightning will automatically use it.