Do you want to create large images with Stable Diffusion with a lot of details? You will need to use an upscaler. In this post, you will learn 3 methods to upscale images.
- AI upscalers
- SD upscale
- ControlNet tile upscale
You will see a comparison and learn about the pros and cons of these methods.
You may want to learn the basics of upscalers if you are not familiar with the topic.
Software
We will use AUTOMATIC1111 Stable Diffusion GUI to perform upscaling. You can use this GUI on Windows, Mac, or Google Colab.
We will need the Ultimate SD Upscale and ControlNet extensions for the last method.
Colab users
If you use the Colab Notebook provided by the site, all you need to do is to select ControlNet and Ultimate SD Upscale extensions before starting the notebook.
Windows or Mac users
Installing Ultimate SD upscale extension
Ultimate SD upscale is an extension offering functions similar to the built-in SD Upscale script but with some advanced options.
To install Ultimate SD upscale in AUTOMATIC1111:
- Navigate to the Extensions page.
2. Select the Install from URL tab.
3. Put the following URL in the URL for extension’s repository field.
https://github.com/Coyote-A/ultimate-upscale-for-automatic11114. Click the Install button.
5. Wait for the confirmation message saying the extension is successfully installed.
6. Restart AUTOMATIC1111.
If the installation is successful, you should see Ultimate SD upscale available in the Script dropdown menu at the bottom of the img2img page.
Installing ControlNet Extension
ControlNet is an extension that gives you extra control over your image generations. See the ControlNet tutorial to learn more.
To install Ultimate SD upscale in AUTOMATIC1111:
- Navigate to the Extensions page.
2. Select the Install from URL tab.
3. Put the following URL in the URL for extension’s repository field.
https://github.com/Mikubill/sd-webui-controlnet4. Click the Install button.
5. Wait for the confirmation message saying the extension is installed.
6. Restart AUTOMATIC1111.
7. Visit the ControlNet models page.
8. Download all model files (filename ending with .pth). If you don’t want to download all of them, you can just download the tile model (The one ends with _tile) for this tutorial.
9. Put the model file(s) in the ControlNet extension’s model directory.
stable-diffusion-webuiextensionssd-webui-controlnetmodels10. Restart AUTOMATIC1111.
You should see the ControlNet section on the txt2img page. The tile model should be available for selection in the Model dropdown menu. Press the refresh button next to the menu if you don’t see it.
Input image
We will use the following image generated by Stable Diffusion.
Model: Lyriel v1.5
Prompt and negative prompt:
“The Last Dance”: In a grand ballroom, our star-crossed lovers share a dance. Time seems to slow as they waltz, lost in each other’s eyes. Unbeknownst to them, this will be their final moment together before fate intervenes.
(I got this prompt from GPT-4)
disfigured, deformed
The original size is pretty small (768×512 pixels). Let’s enlarge it 4x to 3,072×2,048 pixels. We will test with three upscaling methods.
- AI Upscaler
- SD Upscale
- ControlNet tile upscale
Method 1: AI Upscaler
An AI Upscaler is an AI model trained to enlarge photos and fill in details so that the image can stay sharp. This is the basic way to upscale images in AUTOMATIC1111.
- In AUTOMATIC1111, navigate to the Extras page.
2. Upload the image to the image canvas.
3. Set Resize to 4. (Enlarge the image by 4x.)
4. Select an upscaler in Upscaler 1. I used Universal Upscaler v2, but you can use any. (See instructions to download and install new upscalers here)
5. Click Generate.
After a short while, you will see the upscaled image on the output window on the right.
Let’s see how it goes. Let’s first look at the dress.
It’s doing a decent job of faithfully filling in the details of the dress.
But the woman’s face will need some enhancement beyond what this upscaler can do.
For sure, you can turn on face restoration, which uses another AI model to restore faces. But because the model is not trained with this particular style, it could introduce artifacts and make faces look unnatural.
Method 2: SD Upscale
Is there any way to fix the face without using face restoration?
First of all, the face is kind of like that in the original image. That’s why we got what we got when scaling it up.
But what if you are OK with more changes since the original image is computer-generated anyway? You can perform an additional round of image-to-image to regenerate details.
This is exactly what the SD upscale script does.
The benefits of this method are
- New zoom-in details can be generated with the same model and prompt.
- Remove artifacts introduced by the AI upscaler.
You can use the SD Upscale script on the img2img page in AUTOMATIC1111 to easily perform both AI upscaling and SD img2img in one go. The script performs Stable Diffusion img2img in small tiles, so it works with low VRAM GPU cards.
Step-by-step guide
Follow these steps to perform SD upscale.
- On the img2img page, upload the image to Image Canvas.
2. Put in the original prompt and the negative prompt. You can use “highly detailed” if you don’t have the original prompt.
Alternative 1: If you just generated the image on the txt2img page, click the Send to img2img button. The image and prompts will be populated automatically.
Alternative 2: If you have a generated image saved in your local storage, you can upload it to the PNG info page. You should see all the information for generating this image, including the prompts and models. Click Send to img2img. The image and prompts will be populated automatically.
3. In the Stable Diffusion checkpoint dropdown menu, Select the model you originally used when generating this image
3. Set both the image width and height to 512. This is the tile size to be used for SD upscale.
4. Set denoising strength to 0.3. A higher value will result in more details and recovery, but you should not set it higher than 0.4. Otherwise, you will see artifacts.
5. Find the Script dropdown menu near the button of the img2img page. Select SD upscale.
6. Set the Scale Factor to 4 for scaling up 4x.
7. Select an Upscaler. I selected the same upscaler used in the previous method. But you can select ESRGAN or R-ESRGAN if you don’t have it.
8. Click Generate.
Result
This is what I got.
The hardwood floor has a lot of details, thanks to Stable Diffusion img2img.
Now let’s compare the face.
Nice improvement! SD upscale fixes the faces without introducing artifacts.
Let’s inspect the dress, shall we?
They have slightly more details too.
You can experiment with the following parameters.
- Sampling steps: Setting to higher, e.g. 50 – 100, can result in more details.
- Sampling method. The details can be different, especially with ancestral samplers.
- Denosing strength. Controls how much detail to add. The workable range is between 0.1 and 0.4 for this method.
- Seed: Change the details.
The level of detail is controlled by the denoising strength parameter. As I said earlier, this method breaks down if you set it higher than 0.4 because Stable Diffusion will change the image too much. You will start to see artifacts.
Is there any way to increase the denoising strength without sacrificing the original composition? Mmmm… This sounds awfully similar to the function of an extension that copies the composition from a reference image…
That’s right. The solution is using ControlNet!
Method 3: ControlNet tile upscale
ControlNet tile is a ControlNet model for regenerating image details. If the local image details does not match the prompt, it will ignore the prompt and fill in the local details. This behavior makes it ideal for upscaling in tiles, so it works with a low VRAM setup.
You will need to install Ultimate SD Upscale and ControlNet extensions in order to use this method. Follow the instructions in the Software section if you haven’t done so already.
I have no idea why this method doesn’t work with the built-in SD Upscale script. It should, but it doesn’t.
Step-by-step guide
To perform ControlNet tile upscale:
- Navigate to the img2img page in AUTOMATIC1111.
2. Upload your image to the img2img canvas.
3. Put in the original prompt and the negative prompt. You can use “highly detailed” if you don’t have the original prompt.
Alternative 1: If you just generated the image on the txt2img page, you can click the Send to img2img button. The image and prompts will be populated automatically on the img2img page.
Alternative 2: If you have a generated image saved in your local image, upload it to the PNG info page. You should see all the information for generating this image, including the prompts and models. Click Send to img2img. The image and prompts will be populated automatically. on the img2img page.
4. Set Sampling steps to 50.
5. Set Denoising strength to 0.5.
The following steps are for the ControlNet section.
6. Upload the same original image to ControlNet’s image canvas.
7. Select Enable to enable the ControlNet extension.
8. Select tile_resample in the Preprocessor dropdown menu.
9. Select control_v…_sd15_tile in the Model dropdown menu.
The following steps are for the Ultimate SD Upscale extension.
10. In the Script dropdown menu near the bottom of the page, select Ultimate SD Upscale.
11. In the Target size type dropdown menu, select Scale from image size.
12. Set the Scale to 4.
13. Select an Upscaler. I used the Universal Upscaler v2. But you can select ESRGAN or R-ESRGAN for a generic upscaler.
14. Confirm the Tile width is 512 and height is 0. This will use 512×512 tiles during upscaling.
15. Click Generate.
Results
Here’s the upscaled image.
Let’s zoom in and compare.
Here are the faces.
Amazing! The quality of ControlNet tile upscaling is nothing short of life-changing. The faces and hairs are so clearly rendered with lots of detail.
Now let’s inspect the dress.
It blew my mind to see how much detail is added!
The downside of this method is perhaps too much detail is added. It could change the image in a way that it is no longer a faithful scale-up of the original. But it depends on your goal.
Parameter adjustments
You can experiment with the following parameters.
- Denoising strength: Lower it to reduce the amount of detail added. The highest reasonable value is 0.5, in my experience.
- Downsampling rate (ControlNet): Control how much downsampling is done to the control tile. Increasing this value makes the control image more blurry. The more burry the control image, the more liberty the model has in making up the details.
- Sampling steps: Increasing the number of steps tends to improve details and sharpness.
- Sampling method: Like SD Upscale, the sampler can play a role in the final details.
- Seed: Change the seed to change the details.
- Seam fix (Ultimate SD Upscale): Enable if you feel you can see the 512×512 tiles. This option runs an extra round of upscaling along the boundaries of the tiles.
Tips
If you see artifacts such as extra objects, try reducing the denoising strength or increasing the down-sampling rate.
Which one should you use?
Out of the three upscaling options — AI upscale, SD Upscale and ControlNet tile upscale, which one should you use?
Before getting to the recommendations, I would argue that the use cases of SD upscale and ControlNet tile upscale are similar. ControlNet tile upscale is superior. There would not be a case that you would want to use SD upscale instead of ControlNet tile upscale.
So it comes down to 2 choices: (1) AI upscaler, and (2) ControlNet tile upscale.
Since you would normally upscale the image with AI upscale before the ControlNet tile operation, essentially, it comes down to whether to perform an additional image-to-image with ControlNet tile conditioning.
If you are working with real photos or fidelity is important to you, you may want to forego ControlNet tile and use only an AI upscaler. There are a multitude of AI upscalers you can use for different types of images.
In fact, even these AI upscalers make up details, so use them wisely.
ControlNet tile upscale is the way to go if you are ok or even prefer to add details. This applies to images generated with Stable Diffusion or other AI image generators.