Why the Same Prompt Makes Different People
When you type "a woman with red hair and green eyes" into any AI image generator, you get a different woman every time. This isn't a bug — it's how diffusion models work.
AI image generators start with random noise and gradually refine it into an image guided by your prompt. The random noise is different every generation (unless you lock the seed). Your prompt constrains the output to the space of "women with red hair and green eyes," but that space contains millions of possible faces, body types, ages, skin tones, and bone structures. Each generation samples a different point in that space.
This means text descriptions alone — no matter how detailed — cannot produce the same character twice. "A woman in her early 30s with shoulder-length auburn hair, green eyes, high cheekbones, and a small mole on her left cheek" is still a description that matches thousands of possible faces.
The only way to get the same character is to show the model what that character looks like. Text constrains. Images anchor.
The Three Types of Consistency Failure
When characters drift, it happens in one of three ways:
Type 1: Identity Drift (The Slow Morph)
The character looks approximately right but gradually changes over multiple generations. The jawline softens. The eyes shift from hazel to brown. The nose widens slightly. Each individual image looks plausible, but placed side by side, images 1 and 10 are clearly different people.
Why it happens: Each generation interprets your reference slightly differently. Small variations compound. After 5-10 generations, the cumulative drift is visible.
Prevention: Re-anchor frequently. Every 3-5 generations, go back to your original reference images rather than using recent outputs as references. Recent outputs carry the accumulated drift.
Type 2: Feature Swap (The Sudden Break)
The character suddenly gains or loses a distinctive feature. Their signature scar disappears. Their hair changes from curly to straight. Their skin tone shifts noticeably. One generation is clearly a different person.
Why it happens: The AI weighted a prompt element differently, or a reference image was ambiguous. If your reference shows the character in shadow, the model may not have enough information to preserve skin tone in a brightly lit scene. If your prompt mentions "blonde highlights" and the model over-indexes on "blonde," the character might become fully blonde.
Prevention: Use multiple reference images showing the feature from different angles and lighting conditions. Explicitly reinforce critical features in every prompt text. Include negative instructions for features that must NOT appear.
Type 3: Style Contamination (The Aesthetic Shift)
The character's physical features are correct, but the rendering style changes — they look photorealistic in one image and slightly illustrated in another, or their skin texture shifts from natural to airbrushed.
Why it happens: Different prompt elements or reference images trigger different aesthetic modes in the model. Adding "cinematic" might push toward a film-grade render while "natural" stays photographic. Different environments can also trigger different rendering approaches.
Prevention: Lock your style separately from your character. Use style references (--sref in Midjourney, style ingredients in Flow) consistently across all generations. Keep your camera and processing specifications identical in every prompt.
The "Anchor and Constrain" Principle
Every consistency technique in this course is a variation of one principle: anchor the character's identity with images and constrain the variation with text.
ANCHOR = Reference images that show EXACTLY what the character looks like
(headshots, full body, expressions, turnaround sheets)
CONSTRAIN = Text instructions that prevent the model from deviating
(Identity Headers, negative prompts, explicit feature descriptions)
Images anchor. Text constrains. Neither works alone. A reference image without supporting text allows the model to reinterpret features. Text without a reference image allows the model to imagine a different face that matches the description.
The strength of your consistency is determined by:
- Quality of your anchor images — clear, well-lit, unambiguous references
- Specificity of your constraints — detailed text describing exactly what must and must not change
- Frequency of re-anchoring — how often you return to original references versus relying on recent outputs
The Feature Preservation Hierarchy
Not all features are equally easy to maintain. In order from most to least stable:
EASIEST TO MAINTAIN:
├── Hair color (strong signal, high contrast)
├── Gender presentation
├── Approximate age range
├── Skin tone (broad category)
├── Hair length (long vs. short)
MODERATELY DIFFICULT:
├── Face shape (oval, round, angular)
├── Eye color (especially distinctive colors)
├── Hair texture (curly, straight, wavy)
├── Build / body proportions
├── Distinctive marks (scars, birthmarks, tattoos)
HARDEST TO MAINTAIN:
├── Exact facial proportions (nose width, lip fullness)
├── Precise eye shape and spacing
├── Skin texture and complexion details
├── Ear shape, hand proportions
└── Subtle asymmetries that define a real face
This hierarchy tells you where to invest your reference effort. Hair color takes care of itself with a single mention. Exact facial proportions require multiple high-quality reference images from different angles and explicit reinforcement in text.
The Three Levels of Consistency
Different projects require different levels of consistency. Knowing your target level prevents both under-investing and over-investing.
Level 1: Recognizable (5-10 minutes setup)
The character is clearly the "same person" at a glance — same hair, same general features, same vibe. Close inspection reveals minor differences in facial structure.
Good enough for: Social media series, mood boards, concept exploration, rough storyboards.
Method: Single reference image + text description. Midjourney --oref at default weight, or Nano Banana Pro multi-turn conversation.
Level 2: Consistent (30-60 minutes setup)
The character maintains identity across 10-20 images. Side-by-side comparison shows the same person with only very subtle variation.
Good enough for: Brand campaigns, product launches, short video projects, portfolio work.
Method: 3-4 reference images (headshot, full body, expressions) + Identity Header text + tool-specific consistency features. The standard for most professional work.
Level 3: Locked (2-4 hours setup)
The character is virtually identical across 50+ images and video. Even pixel-level comparison shows remarkable consistency.
Good enough for: Long-running character IP, animated series, character-driven brand identities, recurring ad campaigns.
Method: LoRA training on 15-20+ images, or Higgsfield SOUL ID training. Requires upfront investment but produces the highest reliability.
Practical Exercise
Exercise: Diagnose Your Consistency Failures
- Generate the same character 5 times using only a text prompt (no reference images). Use the exact same prompt each time.
- Place all 5 images side by side.
- Identify which features stayed consistent and which drifted.
- Classify each difference: Identity Drift, Feature Swap, or Style Contamination?
- Note which features from the hierarchy were easiest and hardest to maintain.
This exercise makes the consistency problem concrete and personal. You'll see exactly where YOUR prompting approach breaks down — and the remaining modules will give you the tools to fix each failure type.
Key Takeaways
- Text alone cannot produce consistent characters. The same prompt generates a different person every time because diffusion models sample from a space of possibilities.
- Three failure types exist: Identity Drift (gradual morph), Feature Swap (sudden break), and Style Contamination (aesthetic shift). Each has a different prevention strategy.
- The "Anchor and Constrain" principle governs all consistency techniques: images anchor identity, text constrains variation.
- Features vary in preservation difficulty. Hair color is easy; exact facial proportions are hard. Invest reference effort where it matters most.
- Three consistency levels (Recognizable, Consistent, Locked) match different project needs. Know your target before choosing your method.
References & Resources
- Midjourney Documentation: Omni Reference
- Towards Data Science: Generating Consistent Imagery with Gemini
- Pinterest board — Character Design Reference Sheets: https://pinterest.com/search/pins/?q=character%20design%20reference%20sheet%20ai