Diffusion Models & Image Generation 🎨

The big idea: instead of generating an image directly (hard), train a model to remove noise from a noisy image (easier). Then start from pure noise and gradually denoise.

Forward process: add tiny amounts of Gaussian noise over T=1000 steps until the image is pure noise. This is just math — no neural network needed.

Reverse process: train a neural network (typically a U-Net) to predict the noise that was added at each step. At inference, start from noise and call the network 1000 (or with DDIM, just 50) times to denoise back to a clean image.

!pip install -q diffusers transformers accelerate

from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
).to("cuda")

prompt = "Goan Warli folk art style poster, women dancing in coconut grove, white figures on terracotta background, traditional patterns, festive"
negative_prompt = "blurry, low quality, modern style, 3d render, photograph"

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=30,
    guidance_scale=7.5,
    height=1024, width=1024,
).images[0]
image.save("warli_poster.png")

Guidance scale controls prompt adherence. Too low (<5) ignores the prompt. Too high (>15) creates over-saturated, deep-fried images. 7–9 is the sweet spot.

Pure prompt generation is unpredictable. ControlNet adds a second image (edge map, pose skeleton, depth map) that the model must follow:

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
from PIL import Image
import cv2, numpy as np

# Load Tara's pencil sketch and convert to edge map
sketch = np.array(Image.open("grandmother_sketch.jpg").convert("RGB"))
edges = cv2.Canny(sketch, 100, 200)
edge_image = Image.fromarray(np.stack([edges]*3, axis=-1))

controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet, torch_dtype=torch.float16,
).to("cuda")

image = pipe(
    prompt="Warli folk art style, traditional Goan motifs, terracotta and white",
    image=edge_image,                  # composition controlled by edges
    controlnet_conditioning_scale=0.7,
    num_inference_steps=30,
).images[0]

Image generation is powerful and dangerous. The same Stable Diffusion that paints folk art can generate harmful content. Tara built these safeguards:

1. Negative prompts: Always include "nsfw, nude, child, violence, gore" in negative prompts.
2. Safety checker: The Hugging Face StableDiffusionSafetyChecker blocks NSFW outputs. Don't disable it.
3. Watermark: Add a visible "AI-generated · Mitra Heritage Project" watermark to every output.
4. Invisible watermark: Use imwatermark or Stability AI's invisible watermark to mark AI provenance even after cropping.
5. No real faces: Refuse to generate identifiable people without consent. Use generic figures only.
6. Source credit: Always credit "Trained on grandmother's original paintings, with permission."