Pony Diffusion v6 XL is a Stable Diffusion model for generating stunning visuals of humans, horses, and anything in between. Don’t let the model’s name turn you away if you are not into ponies. It can generate humans and scenes equally well, if not better, than any other model.
In this post, I will explain the Pony Diffusion model, how it is trained, what sets it apart, and how to use it in AUTOMATIC1111.
What is Pony Diffusion v6 XL?
Pony Diffusion is an SDXL model trained with ~2.6 million with roughly equal ratios of anime/cartoon and furry images.
The training images contain safe, explicit, and questionable images. You can generate only safe images by including appropriate tags in the prompt. (More on tags later.)
The author of the model manually ranked the training images by aesthetic quality. Score 9 is assigned to the highest-quality images. Score 8 is assigned to slightly less good images, and so on. So, to generate the best quality images, you need to use score_9
in the prompt. But using score 9 alone doesn’t give a strong effect, you should also include some lower scores. But because of a mistake during training, score_8 is mislabeled as score_8_up
. So, in practice, you should start the prompt with something like:
score_9, score_8_up, score_7_up, score_6_up, score_5_up, score_4_up
And add to this prompt what you want to generate.
What is Pony Diffusion good at?
Pony Diffusion is not just another fine-tuned model.
Compared to other SDXL models, Pony Diffusion is good at
- Generating artistic and creative styles.
- Generating horses, humans, and anything in between.
- Following the prompt closely.
- Generating popular and obscure cartoon/anime characters. It has a vast knowledge of characters!
- Generating interaction between subjects.
In addition, you can benefit from hundreds of style modifier LoRAs specifically trained for the Pony Diffusion model.
Software
We will use AUTOMATIC1111 , a popular and free Stable Diffusion software. Check out the installation guides on Windows, Mac, or Google Colab.
If you are new to Stable Diffusion, check out the Quick Start Guide.
Take the Stable Diffusion course if you want to build solid skills and understanding.
Check out the AUTOMATIC1111 Guide if you are new to AUTOMATIC1111.
Download Pony Diffusion v6 XL model
You can download the Pony Diffusion model on CivitAI. Download the pruned fp16 model and put the file in stable-diffusion-webui > models > Stable-diffusion.
The model also comes with a VAE. Download the VAE and put it in stable-diffusion-webui > models > VAE. You may want to rename to file to sdxl_vae.ponny.safetensors
to note that it is for the Pony model.
Using Pony Diffusion model
For the best results, you need to use the following settings.
- Use the Pony XL VAE.
- Set the CLIP SKIP to 2.
You can set up your AUTOMATIC1111 so these two settings can be easily changed. Go to Settings. Search for Quicksettings. Add the following two items.
- sd_vae
- CLIP_stop_at_last_layers
Click Apply settings and restart AUTOMATIC11111.
You should see the SD VAE and Clip skip settings at the top. Set the two settings like below when using Pony Diffusion.
The rest of the settings are pretty standard for SDXL models
- Sampling method: DPM++ 2M
- Schedule type: Automatic
- Sampling steps: 25
- Image size:
- 1024 x 1024 (square)
- 832 x 1216 (landscape/ portrait)
- 1344 x 768 (16:9)
Prompt examples for Pony Diffusion
In this section, you will find prompts and ideas to get you started using Pony Diffusion. You will see creative and dynamic compositions unique to the Pony model.
Tips
- The pony model is trained with a unique tagging system. I will detail them in a new post, but as a rule, it always starts with score_9, score_8_up, score_7_up in the prompt. These tags improve the quality of the image.
- You don’t have to use any negative prompt. But I found using simple generic negative prompts like ugly, deformed helped.
- Use realistic in the prompts to generate more realistic anime styles.
- Use the rating_safe tag in the prompt to generate SFW images.
Prompt examples
You can take advantage of Pony Diffusion’s vast knowledge of characters.
Prompt:
score_9, score_8_up, score_7_up, 1girl, Tifa Lockhart, destruction of the world, floating, beautiful eyes, mysterious, looking at viewer
Negative prompt:
ugly, deformed
Use a simple prompt after the quality tags (score_9, score_8_up, score_7_up) to let the model generate something to surprise you.
score_9, score_8_up, score_7_up, a fearsome dragon, biomechanical, snow mountain, moon, dimly lit
ugly, deformed
Or you can be very specific in describing the characters and composition in the prompt to get exactly what you want.
(score_9, score_8_up, score_7_up), 1girl, 20 year old, long black hair, hime cut, nose ring, perky breasts, cleavage, pretty face, looking at viewer, purple eyes, light freckles, oversized hoodie, black skirt, striped thigh-high socks, gothic boots, leaning on wall, detailed background, inside, bar, hidden area, volumetric lighting, vivid colours, glowing, neon, portrait shot, face focus
ugly, deformed
Experimenting with lighting keywords is a great way to explore the model!
score_9, score_8_up, score_7_up, 1boy, standing on the top of a building, short hair, bangs, black hair, long sleeves, realistic, backlight
ugly, disfigured, deformed
LoRA
Because of its usefulness in artistic creations, there are LoRAs specifically trained to be used with Pony Diffusion. On CivitAI, you can find them by filtering with
- Model type: LoRA
- Base model: Pony
Tips: You can download LoRAs directly in A1111 using the CivitAI Helper extension! You will also get the thumbnail images of the LoRAs as a bonus.
Styles for Pony Diffusion V6 XL
Stype for Pony Diffusion V6 XL Model Page
Not satisfied with cartoonish styles in Pony Diffusion? This collection of LoRAs adds an array of styles to your toolbox.
Sinfully Stylish for Pony
Sinfully Stylish (dramatic lighting) – For Pony v0.2 model page
Lighting is the soul of the images. The Sinfully Stylish LoRA adds dynamic lighting to any Pony images! Be gentle on the weight. Try lowering it to 0.8.
Tags
I will discuss tags in a separate post since there is so much to talk about! Stay tuned!
Reference
Pony Diffusion V6 XL – V6 (start with this one) | Stable Diffusion Checkpoint | Civitai
What is score_9 and how to use it in Pony Diffusion | Civitai