Character Consistency in AI Video: Why Faces Drift and How to Keep a Cast Stable Across 60 Episodes

Faces drift because a video model has no memory between calls: every segment is a fresh sample, and text only narrows the range. Fix it in cost order: appearance as concrete static features plus one anchor; a white-background, neutral-light sheet and a face crop; inject those references into every generation; storyboards that only name who is in frame and carry over the previous end state; LoRA for a studio cast; first/last-frame chaining only where references are unavailable.

Why faces drift: three mechanisms

Conclusion first: drift is not a model-quality problem. Three things happen at once.

Every generation is an independent sample. A video model keeps no state between calls. Segment 3 and segment 4 are unrelated draws; with identical prompts the face is still re-guessed. Text narrows the range of guesses, it does not pick one. Text alone gets you "the same kind of person", never "the same person".

Text overrides the reference. Supply a reference image and also write "sharp jaw, thick brows, almond eyes", and the two signals compete; most models follow the more specific one, usually the text. The commonest pipeline bug we see is a parsed appearance paragraph pasted verbatim into every segment prompt, demoting the reference to a hint.

Whatever the reference bakes in is inherited. If the sheet has rim light, a grin and a forward lean, the model reads those as the character, not the photo. Every later shot opens with that light, that grin and that high angle, then fights the per-shot lighting and performance you wrote. A reference image shows identity; it is not a poster.

Layer 1: appearance text that can be read back (cheapest)

"A cold, handsome man around thirty" contains nothing executable. Write static features item by item: face shape, brow shape, eye shape, nose bridge, lips, skin tone, hair style and colour, build, apparent age, plus one static anchor: a scar, a mole, a piece of jewellery, a haircut. The anchor must be static. Write "his eyes crinkle when he smiles" and the sheet will smile, and every segment will open smiling.

No expressions, no posture, no gait. In a full-body shot "leaning slightly forward" means "head near, feet far", which renders as a high-angle shot, and that angle is inherited too.

This text is reused along the whole chain: sheet prompt, identity block in every segment prompt, fallback when references fail. It is the only layer that costs no credits to change.

Layer 2: white-background, neutral-light sheets plus a face crop

The standard for a character sheet is neutral, not cinematic. The working rule for reference images: consistent light, an unambiguous subject, no visual noise competing with it. In practice: a pure white background (its own hard clause, repeated in the negative list; buried mid-paragraph it gets overridden), front-facing full body, one directional key at roughly 75 degrees with fill, so bone structure is legible. The goal is recognisability, not mood.

Add a tight face crop. Reference mechanisms weight attention by pixel area; in a 9:16 full-body sheet the face is under 5% of the frame, and close-ups need only the face. Full body carries build and wardrobe, the crop carries identity. Send both.

Delete cinematic, dramatic lighting, masterpiece. On a reference image they add noise, not information.

Layer 3: inject references into every generation. Mechanism comparison

This is the physical basis of "the same person". A tool with no reference mechanism drifts by construction; one that has it but is not fed on every segment behaves as if it had none. Official docs, checked 2026-09-02:

Two kinds of product expose this layer. Model-native entry points are in the table. End-to-end pipelines automate it: SceneMixer, for instance, keeps a persistent cast library (white-background full-body sheet plus face crop per character) and injects both images for every character present on each segment, with Seedance 2.0/2.5, Wan 3.0 or MiniMax H3 as the selectable tier. Kling Elements, Vidu reference-to-video and Wan r2v reach the same result by hand, at the cost of picking references on every segment.

Two common misuses: a moody poster as the reference (mechanism three again), and more people in frame than references supplied; the extra faces are improvised and pull the referenced ones off-model.

Layer 4: storyboard continuity rules

References say what someone looks like; the storyboard says who is present right now and in what state. Two rules.

Only name who is in frame, positively. Never write "Zhang is not in frame" or "Li exits". Naming someone who should be absent puts their name into the conditioning and raises the odds they wander in. Anyone unlisted is absent by default.

Carry over the previous segment's end state. If she ended the last segment seated with her coat over the chair back, this segment opens from exactly that state: position, posture, what is in her hands, where the light comes from, written into the segment header. Without it the model resets to defaults (standing, coat on, facing camera), and the viewer sees a jump that reads like a face change.

Shots inside one location share one set-dressing and light-source description; only framing and action change. Rewriting the set per segment re-rolls the room per segment.

Layer 5: LoRA / fine-tuning for studios

A reference tells the model about a person at call time; a fine-tune writes the person into the weights. For a fixed cast across a long series the ceiling is higher, and it frees reference slots for locations and props. The costs: data (dozens of multi-angle images per character), training fees and time, and vendor support. Alibaba Model Studio publishes a fine-tuning guide for Wan image-to-video; open-weight models can be LoRA-trained directly.

Worth it when one character appears in more than roughly 30 minutes of finished footage or is reused across productions. Below that, the reference layers pay back faster.

Layer 6: first/last-frame chaining and its trade-off

Chaining feeds segment N's last frame in as segment N+1's first frame. Motion and position are continuous by construction; it is the mechanised version of carry-over. The price:

• Mutually exclusive with references on several models. MiniMax H3's API states that if any reference_image / reference_video / reference_audio role is present, first_frame / last_frame must not be, and vice versa; API docs for Seedance 2.0 treat first/last-frame and multimodal reference as separate scenarios. Use a first frame and identity rests on that frame alone.
• Error accumulates. Each last frame carries its segment's drift; by segment five or six you are seeding from a face that has already moved.
• Some first/last-frame models are silent. Wan's kf2v series outputs video without audio.

Sensible use: only when references are unavailable and two segments must be physically continuous (one action split in two); return to reference mode every two or three segments to re-anchor.

Debug order: find which layer broke

1. Check output dimensions and the task log to confirm which model produced the frames; a user-level preference often silently overrides default routing, and evaluating the wrong model is wasted work.
2. Compare reference to output. If the sheet itself has rim light, a smile or a high angle, redo the sheet (layer 2).
3. Inspect each request: references attached on every segment, face crop included?
4. Inspect the identity block: appearance text contradicting the reference, expression or posture words?
5. Inspect the storyboard: anyone named as absent, any header without the carried-over end state?
6. Still drifting: shorten segments, reduce people per frame, avoid big head turns and occlusion; then consider a fine-tune.

Key figures and sources

Sources

• https://platform.minimax.io/docs/api-reference/video-generation-v2-create
• https://help.aliyun.com/zh/model-studio/wan-video-to-video-api-reference
• https://help.aliyun.com/zh/model-studio/use-video-generation
• https://www.alibabacloud.com/help/zh/model-studio/wan-video-generation-finetune-guide
• https://platform.vidu.com/docs/reference-to-video
• https://kling.ai/quickstart/klingai-element-library-3-user-guide
• https://kling.ai/quickstart/klingai-video-3-omni-model-user-guide
• https://kling.ai/blog/kling-3-subject-binding-character-consistency
• https://docs.cloud.google.com/vertex-ai/generative-ai/docs/video/use-reference-images-to-guide-video-generation
• https://www.volcengine.com/docs/82379/2298881

Tools mentioned

Read next