Pony Diffusion v6 XL is a Stable Diffusion model for generating stunning visuals of humans, horses, and anything in between. Don’t let the model’s name turn you away if you are not into ponies. It can generate humans and scenes equally well, if not better, than any other model.

In this post, I will explain the Pony Diffusion model, how it is trained, what sets it apart, and how to use it in AUTOMATIC1111.

What is Pony Diffusion v6 XL?

Pony Diffusion is an SDXL model trained with ~2.6 million with roughly equal ratios of anime/cartoon and furry images.

The training images contain safe, explicit, and questionable images. You can generate only safe images by including appropriate tags in the prompt. (More on tags later.)

The author of the model manually ranked the training images by aesthetic quality. Score 9 is assigned to the highest-quality images. Score 8 is assigned to slightly less good images, and so on. So, to generate the best quality images, you need to use score_9 in the prompt. But using score 9 alone doesn’t give a strong effect, you should also include some lower scores. But because of a mistake during training, score_8 is mislabeled as score_8_up. So, in practice, you should start the prompt with something like:

score_9, score_8_up, score_7_up, score_6_up, score_5_up, score_4_up

And add to this prompt what you want to generate.

What is Pony Diffusion good at?

Pony Diffusion is not just another fine-tuned model.

Compared to other SDXL models, Pony Diffusion is good at

• Generating artistic and creative styles.
• Generating horses, humans, and anything in between.
• Following the prompt closely.
• Generating popular and obscure cartoon/anime characters. It has a vast knowledge of characters!
• Generating interaction between subjects.

In addition, you can benefit from hundreds of style modifier LoRAs specifically trained for the Pony Diffusion model.

Software

Download Pony Diffusion v6 XL model

You can download the Pony Diffusion model on CivitAI. Download the pruned fp16 model and put the file in stable-diffusion-webui > models > Stable-diffusion.

The model also comes with a VAE. Download the VAE and put it in stable-diffusion-webui > models > VAE. You may want to rename to file to sdxl_vae.ponny.safetensors to note that it is for the Pony model.

Using Pony Diffusion model

For the best results, you need to use the following settings.

• Use the Pony XL VAE.
• Set the CLIP SKIP to 2.

You can set up your AUTOMATIC1111 so these two settings can be easily changed. Go to Settings. Search for Quicksettings. Add the following two items.

• sd_vae
• CLIP_stop_at_last_layers

Click Apply settings and restart AUTOMATIC11111.

You should see the SD VAE and Clip skip settings at the top. Set the two settings like below when using Pony Diffusion.

The rest of the settings are pretty standard for SDXL models

• Sampling method: DPM++ 2M
• Schedule type: Automatic
• Sampling steps: 25
• Image size:
  • 1024 x 1024 (square)
  • 832 x 1216 (landscape/ portrait)
  • 1344 x 768 (16:9)

Prompt examples for Pony Diffusion

In this section, you will find prompts and ideas to get you started using Pony Diffusion. You will see creative and dynamic compositions unique to the Pony model.

Tips

1. The pony model is trained with a unique tagging system. I will detail them in a new post, but as a rule, it always starts with score_9, score_8_up, score_7_up in the prompt. These tags improve the quality of the image.
2. You don’t have to use any negative prompt. But I found using simple generic negative prompts like ugly, deformed helped.
3. Use realistic in the prompts to generate more realistic anime styles.
4. Use the rating_safe tag in the prompt to generate SFW images.

Prompt examples

You can take advantage of Pony Diffusion’s vast knowledge of characters.

Prompt:

score_9, score_8_up, score_7_up, 1girl, Tifa Lockhart, destruction of the world, floating, beautiful eyes, mysterious, looking at viewer

Negative prompt:

ugly, deformed

Use a simple prompt after the quality tags (score_9, score_8_up, score_7_up) to let the model generate something to surprise you.

score_9, score_8_up, score_7_up, a fearsome dragon, biomechanical, snow mountain, moon, dimly lit

ugly, deformed

Or you can be very specific in describing the characters and composition in the prompt to get exactly what you want.

(score_9, score_8_up, score_7_up), 1girl, 20 year old, long black hair, hime cut, nose ring, perky breasts, cleavage, pretty face, looking at viewer, purple eyes, light freckles, oversized hoodie, black skirt, striped thigh-high socks, gothic boots, leaning on wall, detailed background, inside, bar, hidden area, volumetric lighting, vivid colours, glowing, neon, portrait shot, face focus

ugly, deformed

Experimenting with lighting keywords is a great way to explore the model!

score_9, score_8_up, score_7_up, 1boy, standing on the top of a building, short hair, bangs, black hair, long sleeves, realistic, backlight

ugly, disfigured, deformed

LoRA

Because of its usefulness in artistic creations, there are LoRAs specifically trained to be used with Pony Diffusion. On CivitAI, you can find them by filtering with

• Model type: LoRA
• Base model: Pony

Tips: You can download LoRAs directly in A1111 using the CivitAI Helper extension! You will also get the thumbnail images of the LoRAs as a bonus.

Styles for Pony Diffusion V6 XL

Stype for Pony Diffusion V6 XL Model Page

Not satisfied with cartoonish styles in Pony Diffusion? This collection of LoRAs adds an array of styles to your toolbox.

Sinfully Stylish for Pony

Sinfully Stylish (dramatic lighting) – For Pony v0.2 model page

Lighting is the soul of the images. The Sinfully Stylish LoRA adds dynamic lighting to any Pony images! Be gentle on the weight. Try lowering it to 0.8.

Tags

I will discuss tags in a separate post since there is so much to talk about! Stay tuned!

Reference

Pony Diffusion V6 XL – V6 (start with this one) | Stable Diffusion Checkpoint | Civitai

What is score_9 and how to use it in Pony Diffusion | Civitai