ComfyUI Visual Stories — Concepts

The ideas behind a short ComfyUI course for filmmakers and graphic designers. A clarification first: this is not about video generation, but about static images that preserve identity and consistency across a sequence — as in comics and storyboards — in a hyperrealistic style, meaning with the precision and detail of a high-resolution photograph. Without going into individual nodes, this post uses metaphors to explain how a Diffusion Model turns text into an image (CLIP, noise and denoising, VAE), how two adapters — IP-Adapter and ControlNet — teach it to respect a face and a framing, and how these three guides come together at every cycle of generation. It closes with a broader reflection on how neural networks learn through repeated cycles. The technical details, the node graph and the downloadable pipeline are in the companion post ComfyUI Visual Stories — The Node Graph. More advanced examples using this technique can be found in Comics and Storyboards.

Today, using generative AI in a controlled way — not through tools that do everything in one shot, that offer a guided but limited interface, or that behave as a black box — means working the way this course describes. One alternative, sometimes of higher quality, is to use the APIs of proprietary models, whose inner workings are hidden. But both for teaching and for cost, being able to open up the internal structure of these processes remains the strength of open-source models.

1. 01

   Diffusion Models: the base chain

   This course's pipeline generates images through a Diffusion Model (DM) — the generative model that marked a turning point for 'visual' AI. More precisely, we use a Diffusion Model called Flux. Its basic chain is: CLIP → Noise → Denoising → VAE decoder. Let's take it one step at a time.

   The first component is CLIP (you won't see it as a standalone ComfyUI node, since it is built into the main Flux module). CLIP learned, from billions of ⟨text, image⟩ pairs, how to associate an image with the text that describes it. The process doesn't happen 'in plain sight' but in geometric spaces, where both images and texts are tokenised — chopped into pieces — and the various fragments are associated with one another.

   In our culture, so tied to verbal language, this process is intuitive with texts: a text splits into words, words into syllables, down to letters. With images it isn't, because we have no mental habit trained to break an image into fragments in order to understand it better. If anything, the opposite process comes more naturally: jigsaw puzzles break an image apart precisely to test how well we can put it back together. If we had in front of us a fragment of an image and the text describing it, we would only risk making our lives harder when we want the full picture and its description: it isn't natural to find the right words to define a small piece of an image.

   In geometric spaces, however, this is possible, because the system doesn't reason, break apart and reassemble the way we do, but works with measurements in abstract spaces, where objects (image + text) transformed into their 'numerical' form (or 'latent', to use a fashionable term) can be compared. This is exactly what CLIP does: it assigns a text label to an image fragment, chosen by geometric proximity. A symbolic example: think of a tree associated with every image of trees the model has learned, and imagine that every piece of the tree has a name. When I put the tree back together, I also get the detailed description of the tree. Of course, I can't do this in a space where trees are aggregates of pixels, only by manipulating abstract functions and metrics.

   CLIP's job, in the end, is to take every word of your prompt and associate it with the correct images it has learned from the enormous volume of the training set. That is the first step. But how does it generate new images that aren't in the training set? For example, all those fantastical images combining snails and skyscrapers? For that we need the other great insight behind Diffusion Models: the Noising/Denoising process.

   It means adding noise to an image and then removing it. Here a metaphor makes things easier. You're in the countryside, the sky is clear, you see a house and recognise it perfectly. Gradually the fog rises: the house is harder to recognise, but still visible; the fog thickens until the house becomes invisible. At that point it has vanished, but you keep a memory of it, perhaps a slightly vague one. This is the Noising process: adding noise until an object can no longer be identified.

   Now let's move to the reverse process, Denoising. The fog clears little by little. Since you remember the house, because you had seen it, very little is needed — a fog that leaves only 10% visibility — to recognise it as the same one as before. Picture a cycle of clearing in successive steps, until the fog disappears entirely and you're back at the starting image. At that point you're sure it's the same house.

   Intuitively, this is what happens at the geometric level so that the system learns the different tokens (fragments) that make up the house: it manages because it has broken the house into pieces and then put it back together — naturally, in geometric spaces made of numbers. And since every fragment in the training set has a textual meaning — or rather, comes very close to one, because the ⟨image, text⟩ pairs number in the billions, and that is precisely the space CLIP comes from — the final image can be reassembled even when it's made of fragments that may be invented, but that exist in the training set as fragments: a snail's horn, a skyscraper window. In short, the system in the latent space has 'understood' which text fragments to associate, step by step, with the images it discovered in the clearing fog.

   This is what the technical modules call cross-attention: the ability to focus on the text fragments best suited to describing the image that is emerging (in Flux it takes a slightly different form from the early Diffusion Models, but the idea is the same). To go from the language of numbers to what the human eye sees, the work is entrusted to the VAE (Variational Auto Encoder) in its decoder function¹. The same module also works in the opposite direction at the start of the Noising process during learning: everything is first transformed into numerical patterns by a VAE encoder.

   At this point, though, comes the hardest step. We've seen how the Diffusion Model manages to describe any image with a text, even when it doesn't know it in full: it is enough that it knows the fragments independently. But now you'll ask: the prompt you write is a sentence that is unlikely to appear, as it stands, in the dataset, and so has no associated images. And indeed the model often doesn't know it yet — but it manages to construct it, because a noise prediction mechanism comes into play, that is, a prediction of what the fog is covering in the image as it forms. During generation the model stays anchored to the text just as it did during training, with the difference that here the text isn't mapped onto an already formed image, but guides the image that is being formed. This with/without-fog passage drives a process of understanding and generation that is, in a rather interesting way, similar to how the human brain works (a similarity, however, discovered only afterwards).

2. 02

   IP-Adapter: showing instead of describing

   So far, text was the only thing guiding the fog. But a text, however detailed, describes a face poorly. Try describing a person in words to a police sketch artist: 'strong jaw, deep-set eyes, fifty-ish, tired look'. You'll get a plausible face, not that face. For a comic or a storyboard, though, you need the same face, shot after shot. The IP-Adapter (Image Prompt Adapter) solves the problem in a way that is easy to picture: instead of describing the face, it shows it. A reference photograph is placed alongside the text prompt.

   How does it work, staying within the metaphor? The text says: 'behind the fog there is a house'. The photograph says: 'this very house', the one with the green door, the slightly crooked roof, the chimney on the left. At every step of the clearing, as the system decides which fragment to bring out, it has two prompters at its disposal: one speaks in words, the other shows pictures. To be able to listen to both together, the photograph too is translated into the language of numbers — into fragments, as with text — and added as a second channel the model can pay attention to, alongside the text one.

   It isn't copy-and-paste: the model doesn't cut the face out of the photograph and paste it into the image, but 'remembers' its features while rebuilding the scene with a completely different pose, light and expression. That's why there is an intensity dial (a weight): if it's too low, the face fades into the generic; if it's too high, the image becomes a bad photocopy that can no longer follow the prompt. With two characters in the same scene there are two photographs and their voices overlap: if one speaks louder than the other, it wins. That's why the main character is usually given a higher weight.

3. 03

   ControlNet: marks on the floor

   Even with text and a photograph, there's one thing neither controls well: where each element sits within the frame. If I write 'two people at a small table, shot from above', the model decides distances, proportions and body positions on its own. Anyone who has worked on a set knows that this is the job of marks: the signs on the floor that tell the actor where to stop, how far from the camera, facing which way. ControlNet does the same job for the model: it hands over the geometry of the scene before the fog clears.

   What does this geometry look like? In our case it's a depth map: a black-and-white image in which light means near and dark means far, with no faces, colours or details. Back to the countryside in the fog: it's as if, before any house appears, someone let you feel the hidden volumes with your hands — here something large and close, there something small and far, and empty space in between. When the fog clears, things appear in exactly those positions and with those proportions. The text and the photograph decide what things are; the depth map decides where they stand.

   Technically, ControlNet is a 'twin' neural network, trained to work alongside the main model and read a guide image (depth is only one option: there are versions for outlines, body pose and more). Here too there is a dial, the 'strictness' of the marks: if it's high, the model follows them to the letter, but the image risks becoming stiff, almost mechanical; if it's low, the position is just a suggestion. In our pipeline we keep it fairly low: enough to respect the framings, but low enough to leave the characters their naturalness.

4. 04

   Three guides, one cycle

   Let's put the pieces together. At every step of the fog's clearing, the model receives three kinds of guidance:

   • The text says what is in the scene: characters, place, action, type of shot.
   • The reference photograph (IP-Adapter) says who the characters are and what they look like.
   • The depth map (ControlNet) says where things stand and in what proportions.

   The text, in turn, is read by two different 'readers': a more concise one, suited to short labels (such as a character's features), and a much more capacious one, suited to the detailed description of the scene. There is also a negative prompt, which says what to avoid — like a director who, besides saying what to shoot, says what must not be seen; in our pipeline it plays a marginal role, and we discuss it in the technical post. All of this is repeated, with the same three guides, for every shot in the sequence: this is how characters stay recognisable and the setting stays consistent from one shot to the next.

5. 05

   A broader reflection

   A final, broader reflection. The least intuitive thing — and it wasn't (almost) for anyone ten years ago — is that, thanks to neural networks, all the modules we've seen can be connected together and processed in cycles (many cycles) all at once. And the cycles are, in fact, of two kinds, which are worth keeping apart.

   The first is the cycle of generation, the one we've already met: the fog clearing in successive steps (in our pipeline there are thirty), each of which removes a little noise, guided by text, photograph and depth. In this phase the network's weights don't change: the network applies what it has learned.

   The second is the cycle of learning, which took place long before, during training. At every cycle the weights and internal connections of the network's neurons change, and this is how the network gradually learns and transforms the information it processes — even when that information is highly heterogeneous (since it includes everything: positions, character features, colours, environments, and so on). It is a gesture repeated an enormous number of times: each time, the network gets it a little less wrong.

   Today's networks are a combination of many layers (30–40 or more), where each layer is a stratum of neurons (millions of them, or more), and where the connections are guided by what's called an attention mechanism, which in short steers the flow of information. The parameters are the adjustable values the network has at its disposal, and their number depends on how many layers it has, how wide its vectors are, and how many attention mechanisms it contains. When you read that a network 'has 12 billion parameters' (as Flux does), it means there are 12 billion values to be adjusted, cycle after cycle, during learning. This is why hundreds or thousands of GPUs are needed for pre-training, and why the costs at OpenAI, Anthropic and similar companies are so high — and why, for now, these companies operate at a loss.

   Whether it's clearing the fog or learning how to clear it, the principle is the same: not one stroke of genius, but a simple gesture repeated many times, a little better each time.

1. In the language of statistical mathematics for neural networks, Encoder refers to the process that transforms a pixel image into a mathematical object within a geometric space — made of points, or more precisely N-dimensional vectors. The concept of a Zip file is often used to describe it: it is, in effect, a compressed space that allows all these geometric operations to take up far less space. The Decoder is the opposite process: from the geometric space back to the plain image. ↩

Technical details, node graph and downloadable pipeline: ComfyUI Visual Stories — The Node Graph.