Shortcuts

Train on the cloud (intermediate)

Audience: Anyone looking to train a model on the cloud in the background


What is background training?

Background training lets you train models in the background without you needing to interact with the machine. As the model trains you can monitor its progress via Tensorboard or an experiment manager of your choice.


0: Install lightning-grid

First Navigate to https://platform.grid.ai to create a free account.

Next, install lightning-grid and login

pip install lightning-grid
grid login

# Login successful. Welcome to Grid.

1: Create a dataset

Create a datastore which optimizes your datasets for training at scale on the cloud. Datastores can be created from all sorts of sources such as .zip and .tar links, local files/folders and even s3 buckets.

Let’s create a datastore from this .zip file

grid datastore create https://pl-flash-data.s3.amazonaws.com/tinycifar5.zip --name cifar5

Now your dataset is ready to be used for training on the cloud!

Note

In some research workflows, your model script ALSO downloads the dataset. If the dataset is only a few GBs this is fine. Otherwise we recommend you create a Datastore.


2: Choose the model to run

You can run any python script in the background. For this example, we’ll use a simple classifier:

Clone the code to your machine:

git clone https://github.com/williamFalcon/cifar5-simple.git
cd cifar5-simple

Note

Code repositories can be as complicated as needed. This is just a simple demo.


3: Run on the cloud

To run this model on the cloud with the attached datastore, use the grid run command:

grid run --datastore_name cifar5 cifar5.py --data_dir /datastores/cifar5

The grid command has two parts the [run args] and the [file args]

grid run [run args] file.py [file args]

4: Monitor and manage

Now that your model is running in the background, monitor and manage it here.

You can also monitor its progress on the commandline:

grid status

Cost

Lightning (via lightning-grid) provides access to cloud machines to the community for free. However, you must buy credits on lightning-grid which are used to pay the cloud providers on your behalf.

If you want to run on your own AWS account and pay the cloud provider directly, please contact our onprem team: mailto:onprem@pytorchlightning.ai


Next Steps

Here are the recommended next steps depending on your workflow.


© Copyright Copyright (c) 2018-2022, Lightning AI et al... Revision f4fcad36.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: latest
Versions
latest
stable
1.8.3post1
1.8.3.post0
1.8.3
1.8.2
1.8.1
1.8.0.post1
1.8.0
1.7.7
1.7.6
1.7.5
1.7.4
1.7.3
1.7.2
1.7.1
1.7.0
1.6.5
1.6.4
1.6.3
1.6.2
1.6.1
1.6.0
1.5.10
1.5.9
1.5.8
1.5.7
1.5.6
1.5.5
1.5.4
1.5.3
1.5.2
1.5.1
1.5.0
1.4.9
1.4.8
1.4.7
1.4.6
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0
1.3.8
1.3.7
1.3.6
1.3.5
1.3.4
1.3.3
1.3.2
1.3.1
1.3.0
1.2.10
1.2.8
1.2.7
1.2.6
1.2.5
1.2.4
1.2.3
1.2.2
1.2.1
1.2.0
1.1.8
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.8
1.0.7
1.0.6
1.0.5
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
0.10.0
0.9.0
0.8.5
0.8.4
0.8.3
0.8.2
0.8.1
0.8.0
0.7.6
0.7.5
0.7.4
0.7.3
0.7.2
0.7.1
0.7.0
0.6.0
0.5.3
0.4.9
Downloads
html
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.