ABOUT

ComfyUI · Workflow Instructions

Instructions for writing and running this section's workflow

These are the concrete, hands-on steps for writing and running the Python/bash workflow introduced in ComfyUI Visual Stories — The Python Scripts: the five workspace files, the run sequence, and the post-processing pass.

  1. 01

    The steps

    The workspace contains five files in two layers. The static layer (created once, never changed again): the base graph JSON and the two automation scripts. The active layer (edited before each generation run): the configuration file and the character reference images in ComfyUI/input/.

    File Generated by Role
    bar_14shots.json Claude — once Base graph: all node wiring. Never edited.
    shots_config.py You — each run Content: prompt, weights, image refs, seed per shot.
    patcher.py Claude — once, reused Reads base JSON + config, injects content, writes PATCHED files.
    sweep.py Claude — once, reused Fires N API calls varying one parameter, saves N output images.
    jim_face.png, barbara_face.png … You — uploaded once Character and environment references in ComfyUI/input/.
    1. Generate the base graph. Describe the bar_14shots pipeline to Claude: 14 shots, Jim and Barbara, wine bar interior, one PuLID + IPAdapter + ControlNet chain per single-character shot, a doubled chain for shots with both characters. Claude returns the complete bar_14shots.json. Copy it to your workspace. This is the only moment you interact with the graph structure — it stays frozen from here on.

    2. Configure shots_config.py. Open the file and fill in the fields for the shot you are working on. Eight fields define each shot: wall (the depth reference image that also sets the environment), framing, focal, and angle (the cinematographic language, appended to the prompt text), objects (props in the scene), clip_l and t5xxl (the main scene description for Flux's two text encoders), and neg (negative prompt). A ninth field, prev_image, is reserved for visual continuity between shots and not yet wired in the current version.1

      # shots_config.py (excerpt) BASE_GRAPH = "bar_14shots" # patcher.py and sweep.py derive all file paths from this constant SHOTS = { "1A_Barbara_Enters": { "wall": "wall1", # depth / environment reference "framing": "medium shot", "focal": "50mm", "angle": "eye level", "objects": "wine glass, dim ambient light", "prev_image": None, # reserved — not yet wired "clip_l": "Barbara walks into the wine bar, elegant, cinematic...", "t5xxl": "Barbara walks into the wine bar, elegant, cinematic...", "neg": "blurry, deformed, low quality", "pulid_weight": 0.50, "ipadapter_weight": 0.60, "seed": 42, }, "1B_Jim_Reacts": { ... }, }
    3. Run patcher.py. The script reads bar_14shots.json and the active config, injects all content into the correct node fields, and writes two files: bar_14shots_PATCHED.json (GUI format — loadable in the ComfyUI browser to inspect the wired nodes before generating) and bar_14shots_PATCHED_API.json (API format — used by sweep.py and direct API calls). Loading the GUI file in the browser before generating is optional, but useful when debugging a shot for the first time.

      python3 /workspace/patcher.py --shot 1A_Barbara_Enters # optional check before generating: # ComfyUI browser → Load → bar_14shots_PATCHED.json → inspect nodes → Queue
    4. Run sweep.py for parameter exploration. Instead of generating one image and guessing which weight is best, sweep.py loops over a list of values for a single parameter, fires an API call per value, and saves the corresponding output image. Use it to calibrate PuLID (face identity strength), IPAdapter (body and clothing reference strength), or to test different seeds before committing to a final generation.

      # calibrate face identity: 4 values → 4 output images compared side by side python3 /workspace/sweep.py --shot 1A_Barbara_Enters \ --param pulid_weight --values 0.2,0.35,0.5,0.65 # calibrate body/clothing reference strength python3 /workspace/sweep.py --shot 1A_Barbara_Enters \ --param ipadapter_weight --values 0.3,0.5,0.7,0.9 # test random seeds before the final run python3 /workspace/sweep.py --shot 1A_Barbara_Enters \ --param seed --values 42,99,123,777

      Sweepable parameters: pulid_weight, ipadapter_weight, denoise, seed, cfg, steps. The script derives the API JSON path from the BASE_GRAPH constant defined in shots_config.py — no additional arguments needed.

  2. 02

    Post-processing

    Once a shot is generated, small defects can be corrected without re-running the full pipeline. A separate lightweight set of three files applies img2img refinement: it takes the already-generated frame as a structural anchor and re-runs diffusion on it at a controlled noise level, guided by a corrective prompt. The result sits between the original and a full regeneration — closer to one or the other depending on how much noise you inject.

    The control parameter is denoise. The table maps its range to the kind of change it produces:

    denoise Effect Typical use
    0.15 – 0.30 Micro-fix — barely visible Skin artifacts, blurred edges, eye sharpness
    0.30 – 0.50 Moderate — structure stays Wrong background element, uneven lighting
    0.50 – 0.70 Strong — significant change Face drift, mood or style correction
    0.70 – 0.90 Near-regeneration — composition only Full palette or atmosphere overhaul

    The three-file set follows the same pattern as the main pipeline: one config file for the creative direction, scripts and graph JSON for the technical execution. postprocess_config.py is the only file you edit: declare the input image, the corrective prompt, and the denoise value. postprocess_run.py is the engine (generated by Claude). postprocess_graph.json is an 11-node API graph, static and never edited.

    python3 /workspace/postprocess_run.py # postprocess_config.py: # INPUT_IMAGE = "outputs/1A_Barbara_Enters_00042.png" # PROMPT = "Barbara, sharp face, correct the left hand" # DENOISE = 0.35 # moderate — keeps face and composition stable

Complete sequence — all steps

  • The Node Graph: a full walkthrough of the ComfyUI node graph, node by node — DualCLIPLoader, VAELoader, IP-Adapter, ControlNet depth, KSampler. See the companion post ComfyUI Visual Stories — The Node Graph.
  • Setup: install custom node packs, download model weights, upload character references to ComfyUI/input/. See the setup page.
  1. Describe the pipeline to Claude → receive bar_14shots.json → copy to workspace.
  2. Edit shots_config.py: set prompt, wall, framing, weights, seed for the target shot.
  3. Run patcher.py → bar_14shots_PATCHED.json + _PATCHED_API.json. Optional: load _PATCHED.json in the ComfyUI browser to verify node values.
  4. Run sweep.py --shot SHOT_NAME --param pulid_weight --values 0.2,0.35,0.5,0.65 → 4 variants generated and saved automatically.
  5. Set image + corrective prompt + denoise in postprocess_config.py, then run postprocess_run.py. Repeat with higher denoise if the first pass is insufficient.

Steps b–d form an iteration loop. For a 14-shot storyboard, expect several cycles: start with sweeps to calibrate weights, generate all shots, then post-process the frames that need correction.

  1. The prev_image field is intended to feed the output of a previous shot back into the IPAdapter of the next one — a technique for visual continuity between consecutive frames (consistent clothing, lighting match, camera continuity). It requires adding a dedicated IPA node per shot in the graph. It is documented in shots_config.py so that a future revision can wire it without changing the config format. ↩
POSTS