Generative AI, Deep Dive

Most AI before this wave was discriminative: given an input, decide which category it belongs to — spam or not spam, cat or dog. A generative model does the opposite. It learns the shape of a distribution from examples, then produces new samples that plausibly belong to it: a paragraph that reads like the training text, an image that looks like a photograph, without copying any single example directly.

That distinction holds across modalities. The mechanism differs between text, image, audio, and video generation — but the underlying move is the same in every case: sample something new from a learned distribution, rather than sort something given into a bucket.

Text generation is autoregressive: predict the single most plausible next piece of text given everything so far, append it, and repeat. A paragraph isn't retrieved whole — it's assembled one prediction at a time, each one conditioned on everything that came before it.

A setting called temperature controls how much randomness enters that choice. Low temperature picks the single most likely next piece almost every time, producing safe, repetitive output. Higher temperature samples from a wider range of plausible options, producing more varied but less predictable text. Neither setting makes the model more or less accurate — only more or less willing to take a less obvious next step.

The dominant approach for image generation today is diffusion. Training runs the process backward: take a real image, add random noise to it in small steps until it's pure static, and train a model to predict and remove that noise at each step.

Generation runs the trained process forward from pure noise: start with random static, and repeatedly ask the model to predict and subtract a little more noise, guided at each step by a text prompt describing what the final image should resemble. After enough steps, what began as static resolves into a coherent image — not because any single step "drew" anything, but because each step nudged the noise a little further toward the distribution the model learned from.

Training generally happens in two stages. Pretraining exposes the model to an enormous volume of raw data — text scraped from the web, or large image datasets — and simply asks it to get better at predicting the next piece, or removing noise, across all of it. This stage is what gives the model its raw capability.

A second, much smaller stage adjusts that raw capability toward being useful and safe to interact with — fine-tuning on curated examples of the kind of output that's actually wanted, sometimes combined with human feedback that rewards outputs people rate as better. Pretraining teaches the model what's statistically plausible; this second stage teaches it what's actually desirable to produce.

A prompt is the main lever available for shaping generated output without retraining anything. Specificity, examples, and explicit constraints in the prompt all shift what the model treats as plausible for that particular generation — which is why two people can get very different results from the same underlying model.

This is also why prompting has limits. A prompt can steer the model toward a region of what it already learned; it can't teach the model something new or force output outside what its training makes plausible. Asking more clearly helps; asking for something the model never learned a pattern for doesn't.

A classifier has a right answer to check against. Generated text or images usually don't — quality is closer to a judgment call than a verifiable fact, which makes evaluation genuinely harder than it is for discriminative tasks.

In practice, teams combine automated metrics that approximate quality, structured human review on a sample of outputs, and increasingly another model acting as a judge. Each has known blind spots: automated metrics reward what's measurable over what's actually good, human review doesn't scale to everything, and a model judge inherits the same kinds of blind spots as the model it's judging.

The most consequential limitation is hallucination: a model can generate fluent, confident, entirely wrong content, because generation only optimizes for plausibility, not for truth. A convincing wrong answer and a convincing right one can look identical from the outside.

Bias is a related, distinct problem — a model trained on real-world data reproduces the patterns in that data, including skewed or unfair ones, without being told to. And generative models are weak at anything requiring exact verification — precise arithmetic, checking a fact against a source — because nothing in the generation process actually checks an output against ground truth before producing it.

Generative models are trained on large volumes of existing text, images, and other content, much of it collected from the public web. Whether that kind of training use requires permission or compensation from the original creators, and who owns the rights to what a model generates, are questions still being actively litigated and legislated in multiple jurisdictions.

This is worth stating plainly rather than glossing over: there isn't yet a single settled answer, the rules differ by jurisdiction and are still moving, and anyone using generated output commercially should treat this as an open legal question to get real advice on, not a solved problem.

Text generation shows up in drafting, summarization, and code completion. Image generation spans concept art, product mockups, and marketing visuals. Audio and voice generation cover synthetic narration and voice cloning; video generation is the newest and least mature of the group. Code generation assists with boilerplate and translation between languages, still under close review.

A common thread across all of them: the more consequential the use case, the more a human is still expected to review the output before it goes anywhere that matters — generation proposes, a person still generally disposes.

Responsible adoption mostly means taking the limitations from earlier modules seriously in practice: human review before anything generated ships unsupervised, especially anywhere a hallucination or a bias would actually cause harm.

Increasingly it also means disclosure — being clear with an audience when content was AI-generated, particularly where authenticity matters to them — and keeping the open provenance questions from module eight in mind rather than treating generated output as free of any rights complexity by default.