You can train your own SDXL LoRA model with the Google Colab Notebook created by this site. The following instruction trains a LoRA model for a person’s face.
Software
You must have a Google Colab Plus subscription to use this training notebook.
Download the Easy LoRA Trainer SDXL and sample training images below. You can use the sample images to go through the whole process before using your own.
Step 1: Prepare training images
For the best result, crop the images to 1024×1024 pixels. However, this is not strictly necessary because the training script supports bucketing.
See my training examples below.
Step 2: Come up with a good triggering keyword
For training SDXL LoRA models, it is better NOT to use a rare token for the triggering keyword but something that resembles your subject.
Since we are training a woman’s face, we need to find someone in the SDXL model who looks like her. That would increase your chance of success.
Let’s try prompting the SDXL Base 1.0 model.
photo of jane
Nah, Jane won’t be a good triggering keyword because it generates black-and-white images with a woman who doesn’t look like ours.
Let’s try another name.
photo of emma
The keyword “emma” generates color pictures of a young woman who resembles our subject. (Likely thanks to Emma Watson) Let’s use “emma” as our triggering keyword. Note you have to stick with a single word.
Step 3: Review the training settings
Open the Easy LoRA trainer SDXL notebook.
Project name
This is the folder that will be created in your Google Drive. Use a different project folder for each training.
Image Repeats
The Image repeats is how many times the training images are repeated in each training epoch. Keep this at 1 for the default training workflow.
Number of Epochs
The number of epochs is the number of training rounds. Increase this number to increase training cycles.
This is the main parameter to adjust how much want to train the model. Increase to train more. Decrease to train less.
Learning rate
How big a step for each model update.
A larger value trains faster and requires fewer training epochs.
But a learning rate too large may cause error or bad results.
Triggering keyword
The triggering keyword is the token associated with your subject. You need to use this keyword in the prompt.
Lora name
The Lora name is the name of your LoRA file. In AUTOMATIC1111, It looks like <lora:emma_XL:1> when you use the LoRA.
Lora output path
The LoRA file will be saved in this location in your Google Drive.
Skip Image upload
Select this option if you want to reuse the previously uploaded images. This is useful for retraining a LoRA model with a different setting or revising the captions.
Step 4: Start training
Start the training by clicking the play button on the left of the settings.
It will ask for your permission to access your Google Drive. You must accept the connection to save the final LoRA model in your Google Drive. (There’s no good way to download the model except by saving it in your Google Drive.)
A button will appear that allows you to upload the training images. Click Choose Files and select your training images. (The images, not the zip file.)
It will take a while to complete running. It will
- Set up the training software
- Generate captions for your images. You can find them in the project folder in your Google Drive. They are the
.txtfiles with the same name as your images. - Train the LoRA model.
It may prompt for restarting the runtime. Click Cancel.
Monitor the training progress with the printouts. It is done when it shows 100% steps.
You can rerun the training cell without disconnecting and reconnecting the notebook.
Disconnect the notebook when you are done. Otherwise, it will continue to consume your compute credit.
Using the SDXL LoRA model
Your model is saved in your Google Drive in the AI_PICS > Lora folder. It is ready to use with the Stable Diffusion Colab Notebook.
Alternatively, download and install the LoRA model locally on your machine.
For AUTOMATIC1111, put the LoRA model in stable-diffusoin-webui > models > Lora.
To use the LoRA model in AUTOMATIC1111, you first need to select an SDXL checkpoint model.
Use the prompt with the LoRA:
photo of emma <Lora:emma_XL:1>
Don’t forget to set the image size to 1024×1024.
Now, we get our subject!
Note: You must apply the LoRA AND use the triggering keyword emma to get the effect.
Testing the LoRA weight (e.g. <lora:emma_XL:weight>) when using the LoRA. Sometimes, 1 is not the optimal value.
Tips for successful training
The default setting is good for training a realistic face. You may need to tweak the settings for your training.
Below are some tips for tweaking.
- The quality of the training images is more important than the quantity. The training images should show clear faces if you are training a face. You can get good training with as few as five images.
- The images should have a diverse background.
- Try adjusting the number of epochs to increase or decrease the training. It is possible to overtrain a model.
Experimentation
Training a LoRA model requires patience and experimentation. You should treat the default parameters as a starting point.
Observe the result and change the settings one at a time. Observe the result with the same seeds. Generate multiple images to draw conclusion.
Just like a good old scientist would do.
A systematic approach may take longer. But in the end, the knowledge and intuition you gain will make you a better trainer.
Undercook and overcook
You should not undercook (training too little) or overcook (training too much) your model. You should aim at training the just right amount and stop.
A basic way to do that is to change the number of epochs.
Just right
Below are from a model trained with the default value of 50 epochs. Her face shows up nicely.
Undercooked
Training with 25 epochs is not enough. The face doesn’t show.
Overcooked
100 epochs overcook the model. The pose and the face lost diversity. Her face doesn’t look natural.
FAQ
I got ModuleNotFoundError: No module named ‘torch._custom_ops’
You likely have restarted the runtime when prompted. Click cancel when prompted to restart the runtime.