Diffusion model
Also known as: diffusion models, denoising diffusion model, latent diffusion model
A diffusion model is a type of generative AI that creates content by starting with random noise and gradually removing it, step by step, until a clear image, video, or sound appears.
It learns how to do this by being trained to reverse the process of adding noise to real examples. While diffusion models power popular image and video generators, they raise serious questions about copyright, consent, and deepfakes.
Diffusion model, explained at three levels
At home
Imagine a photo slowly covered in static until it's just fuzz. A diffusion model learns to run that process backward, turning fuzz into a brand new picture that matches what you typed.
At work
Diffusion models let teams create images, product mockups, and video drafts quickly from text descriptions.
At home
The model makes an image over many small steps, cleaning up noise each time. Your text prompt guides each step so the final picture matches your description.
At work
Most image tools use latent diffusion, working in a compressed representation for speed. Controls include guidance strength, number of steps, image-to-image editing, inpainting, and conditioning tools such as ControlNet for pose or layout.
At home
These models learned from millions of images online, often without the creators' permission. In my opinion, that's a fair concern, even if the technology itself is remarkable.
At work
Business considerations include licensing of training data, indemnity offered by vendors, brand safety, content credentials such as C2PA, and policies against generating real people's likenesses. Newer approaches such as flow matching and diffusion transformers improve speed and quality.
Real-world examples
At home
Lena used an AI image generator to create a birthday card showing her cat as a medieval knight. Typing a description and waiting a few seconds produced several versions to choose from.
At work
Jerome is a creative lead at a Toronto ad agency. His team uses diffusion models to sketch concept boards quickly, then has designers create final artwork to avoid copyright and quality issues.
How it compares
| feature | Diffusion model | Large Language Model |
|---|---|---|
| How it creates | Refines noise over many steps | Predicts the next token |
| Main outputs | Images, video, audio | Text and code |
| Speed | Several steps per output | One token at a time |
| Guided by | Text prompt and settings | Prompt and context |
Diffusion models paint by removing noise. Language models write word by word.
For analysts
For commercial use, I'd choose image tools that offer licensed training data or legal indemnity. Cheap generation can become expensive if a campaign has to be pulled over rights issues.
History & origin
Jascha Sohl-Dickstein and colleagues introduced diffusion models in a 2015 paper inspired by physics. They became practical when Jonathan Ho, Ajay Jain, and Pieter Abbeel at UC Berkeley published 'denoising diffusion probabilistic models' in 2020. OpenAI's DALL-E 2 and Google's Imagen in 2022 showed striking results, Midjourney launched publicly in 2022, and Stable Diffusion, released openly in August 2022 by Stability AI with researchers from LMU Munich and Runway, made the technology widely available. Video models followed, including OpenAI's Sora, announced in February 2024, and Google's Veo. Lawsuits from artists and Getty Images challenged how training data was collected. In my view, diffusion models made image creation effortless and made proving an image is real much harder.