Alternatively, one could forget all activations during the forward pass and recompute them on demand during the backward pass. In order to compute the gradients during the backward pass all activations from the forward pass are normally saved. Gradient CheckpointingĮven when we set the batch size to 1 and use gradient accumulation we can still run out of memory when working with large models. Next we have a look at another trick to save a little bit more GPU memory called gradient checkpointing. If we wanted to train with a batch size of 64 we should not use per_device_train_batch_size=1 and gradient_accumulation_steps=64 but instead per_device_train_batch_size=4 and gradient_accumulation_steps=16 which has the same effective batch size while making better use of the available GPU resources. So in our case, the batch_size of 4 was already pretty close to the GPU’s limit. In general you would want to max out the GPU usage as much as possible. Of course, this would change as you increase the number of accumulation steps. We can see that the memory footprint was dramatically reduced at the cost of being only slightly slower than the vanilla run. Let’s see how it impacts the models memory footprint: We can use gradient accumulation in the Trainer by simply adding the gradient_accumulation_steps argument to TrainingArguments.

In turn, however, the added forward and backward passes can slow down the training a bit. This way we can easily increase the overall batch size to numbers that would never fit into the GPU’s memory. When enough gradients are accumulated we run the model’s optimization step. The way we do that is to calculate the gradients iteratively in smaller batches by doing a forward and backward pass through the model and accumulating the gradients in the process. The idea behind gradient accumulation is to instead of calculating the gradients for the whole batch at once to do it in smaller steps.

A simple trick to effectively train larger batch size is gradient accumulation. So ideally we want to tune the batch size to our model’s needs and not to the GPU limitations. However, a larger batch size can often result in faster model convergence or better end performance. We see that already a relatively small batch size almost fills up our GPU’s entire memory. First, we set up a few standard training arguments that we will use across all our experiments: So now we can start training the model and see how the GPU memory consumption changes. We get the same number as before and you can also see that we are using a V100 GPU with 16GB of memory. | GPU GI CI PID Type Process name GPU Memory | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr.

Infinity trainer state of decay 2