A killer application of Stable Diffusion is training your own model. Being an open-source software, the community has developed easy-to-use tools for that.

Training LoRA models is a smart alternative to checkpoint models. Although it is less powerful than whole-model training methods like Dreambooth or finetuning, LoRA models have the benefit of being small. You can store many of them without filling up your local storage.

Why train your own model? You may have an art style you want to put in Stable Diffusion. Or you want to generate a consistent face in multiple images. Or it’s just fun to learn something new!

In this post, you will learn how to train your own LoRA models using a Google Colab notebook. So, you don’t need to own a GPU to do it.

This tutorial is for training a LoRA for Stable Diffusion v1.5 models. See training instructions for SDXL LoRA models.

Train a Stable Diffuson v1.5 LoRA

Software

The training notebook has recently been updated to be easier to use. If you use the legacy notebook, the instructions are here.

You will use a Google Colab notebook to train the Stable Diffusion v1.5 LoRA model. No GPU hardware is required from you.

If you are a member of this site, access the training notebook and images below.

Alternatively, you can purchase the notebook using the link below.

You will need Stable Diffusion software to use the LoRA model. I recommend using AUTOMATIC1111 Stable Diffusion WebUI.

Get the Quick Start Guide to learn how to use Stable Diffusion.

Step 1: Collect training images

The first step is to collect training images.

Let’s pay tribute to Andy Lau, one of the four Heavenly Kings of Cantopop in Hong Kong, and immortalize him in a Lora…

Google Image Search is a good way to collect images.

Let’s collect at least 15 training images.

Pick images that are at least 512×512 pixels for v1 models.

Make sure the images are either PNG or JPEG formats.

I collected 16 images for training. You can download them to follow this tutorial on the training notebook’s product page OR below if you are a member (login required).

Step 2: Review the training settings

Open the Easy LoRA Trainer SD 1.5 notebook.

Here is some input to review before running the cell.

Project folder

A folder in Google Drive containing all training images and captions. Use a folder name that doesn’t exist yet.

Pretrained model name

The Hugging Face name of the checkpoint model. Here are a few options.

The Stable Diffusion v1.5 model is the official v1 model.

runwayml/stable-diffusion-v1-5

Realistic Vision v2 is good for training photo-style images.

SG161222/Realistic_Vision_V2.0

Anything v3 is good for training anime-style images.

admruul/anything-v3.0

You can find other models on Hugging Face using this link or this link. They have to be diffusers models. You can tell by a similar folder structure to the models above.

Image repeats

How many times the training images are repeated in each training epoch.

Number of epoches

The number of training rounds. Increase this number to increase training cycles.

Learning rate

How big a step is it to update the model.

Trigger keyword

The token associated with your subject.

Lora name

The name of your LoRA file. In AUTOMATIC1111, It looks like <lora:AndyLau001:1> when you use the LoRA.

Lora output path

A folder in Google Drive where the Lora file will be saved. The default path lets you use the LoRA with our AUTOMATIC1111 Google Colab notebook without moving.

Step 3: Run the notebook

Run the notebook by clicking the Play button on the left.

It will ask you to connect to your Google Drive.

You must accept because there’s no easy way to download the final LoRA model from Google Colab.

Click Choose Files and select your training images.  (The images, not the zip file.)

It may prompt for restarting the runtime. Click Cancel.

It will take a while to complete running. It will

• Set up the training software
• Generate captions for your images. You can find them in the project folder in your Google Drive. They are the .txt files with the same name as your images.
• Train the LoRA model.

The LoRA model is saved in your Google Drive folder: AI_PICS > Lora in the default setting.

Note: Google Colab may not work well with browsers other than Chrome. Try using Chrome if you experience an issue uploading the image.

When you are done, don’t forget to click the caret on the top right, and click disconnect and delete the runtime. Otherwise it will keep consuming your compute credit.

Using the LoRA

If you save the LoRA in the default output location (AI_PICS/Lora), you can easily use the Stable Diffusion Colab Notebook to load it.

Open AUTOMATIC1111 Stable Diffusion WebUI in Google Colab. Click the Select the Lora tab and click the LoRA you just created.

Here are the prompt and the negative prompt:

Andy Lau in a suit, full body <lora:AndyLau001:1>

ugly, deformed, nsfw, disfigured

Since we have used “Andy Lau” as the triggering keyword, you will need it in the prompt for it to take effect.

Although the LoRA is trained on the Stable Diffusion v1.5 model, it works equally well with the Realistic Vision v2 model.

Here are the results of the Andy Lau LoRA.

Revise captions

The captions are generated automatically during training. But they can be incorrect sometimes. You may want to revise them and retrain your model.

To do so, go to your Google Drive and navigate to the project folder. (AI_PICS > training > AndyLau with the default setting).

You should see a sub-folder that contains all your images and captions. They are the files ending with .txt and with the same names as the images.

Revise them to fit the image.

In the notebook, check Skip_image_upload and run the notebook to start a new training. You can optionally change the Lora_name to avoid overwriting the previous one.

You don’t need to disconnect and reconnect the notebook for new training.

General tips

Experimentation

Training a LoRA model requires patience and experimentation. You should treat the default parameters as a starting point.

Observe the result and change the settings one at a time. Observe the result with the same seeds. Generate multiple images to draw conclusion.

Just like a good old scientist would do.

A systematic approach may take longer. But in the end, the knowledge and intuition you gain will make you a better trainer.

Overcook and undercook

We say a LoRA is overcooked when your training subject shows color saturation like using a high CFG value.

On the other end, a LoRA is undercooked when your subject is not showing enough.

The images below used a LoRA that’s undercooked. This guy doesn’t look quite like Andy Lau. (50 repeats)

The LoRA is trained just right for the following images. (100 repeats)

The following images are overcooked (150 image repeats). See the backgrounds are all the same.

You can increase the image repeats or number of epochs to cook more. Likewise, reduce them to cook less.

There’s probably a setting between 100 and 150 that gives the optimal result.

If the training is too aggression that it is easy to overcook, you can lower the learning rate. This will cause a smaller update to be made to the model. You may need to increase the image repeats or the number of epochs to compensate.

Reference

LoRA training parameters – An authoritative reference of training parameters.

LEARN TO MAKE LoRA – A graphical guide to training LoRA.

kohya_ss Documentation – English translation of kohya_ss manual.

FAQ

I got ModuleNotFoundError: No module named ‘torch._custom_ops’

You likely have restarted the runtime when prompted. Click cancel when prompted to restart the runtime.