One Agent Can’t Make a Video
We tried. We gave a single AI prompt the job of writing a script, specifying visuals, choosing music direction, and generating narration settings. The result was mediocre at everything and good at nothing.
The problem isn’t capability — modern LLMs can do all of these things. The problem is scope. The fix was a multi-agent AI pipeline where each agent owns a single creative role. A prompt that tries to be a Writer, Director, and Music Director simultaneously makes compromises that a specialist never would. The Writer optimizes for the ear. The Director optimizes for the eye. When one agent does both, neither gets full attention.
The 8-Agent Pipeline
We split video production into 8 specialized roles, each with defined inputs, outputs, and constraints:
Producer → Creative Director → Writer → Director → Voice Director → Music Director → Automation Engineer → AssemblerEach agent receives specific context from the agents before it and produces a defined deliverable for the agents after it. The Producer creates the brief. The Creative Director sets the aesthetic. The Writer creates the script. The Director creates the shot list. And so on.
Why Constraints Matter More Than Capabilities
The key insight: what an agent can’t do is more important than what it can.
- The Writer can’t specify visual details — that’s the Director’s job. This means the Writer focuses entirely on narration that sounds good spoken aloud, at exactly 150 words per minute.
- The Director can’t change the script — they must work with what the Writer produced. This forces creative problem-solving in the visual domain rather than rewriting narration to be easier to film.
- The Voice Director can’t edit narration text — they flag issues for the Writer. This preserves the Writer’s carefully timed pacing.
- The Music Director can’t adjust narration timing — music adapts to narration, not the other way around. This is why the Music Director runs after the Voice Director.
Without these constraints, each agent would “help” by encroaching on others’ domains, and the result would be the same mediocre compromise we got from a single prompt.
The Audio Pipeline: Narration + Music
Voice Director: Tone Calibration
Every scene has a tone intent — empathetic for problem scenes, confident for the product intro, enthusiastic for features, warm for the CTA. The Voice Director maps these creative intents to engine-specific parameters.
For ElevenLabs, “empathetic” means stability 0.35, similarity_boost 0.75, style 0.35. For OpenAI, it means speed 0.95 with the instruction “Speak with empathy and understanding.” Same creative intent, different technical implementation.
This abstraction means switching from ElevenLabs to OpenAI doesn’t require rethinking the creative direction — just the engine parameter mapping.
Music Director: Prompt Engineering for AI Composition
The Music Director is the newest addition to the pipeline — and arguably the most interesting. They take the Creative Director’s mood description (“tense opening that resolves into confident energy”) and translate it into a detailed generation prompt:
“Instrumental corporate tech background music. Starts with a tense, slightly urgent electronic mood for the first 40 seconds — muted pads, minor key, subtle pulse. Then transitions into an optimistic, confident, driving energy with brighter synths and a steady beat. Modern, clean production. 110 BPM. No vocals.”
The difference between a vague prompt and this level of specificity — with BPM, instrumentation, dynamics arc, and temporal markers — is the difference between generic stock music and a score that serves the video.
The Music Director generates 2-3 variants per video, each approaching the brief from a different angle. A human picks the winner.
Deduplication: Search Before You Create
Before the Producer writes a single brief, they search the video database. “Do we already have a scene that covers the deployment feature?” If yes, the Director can reference the existing asset instead of spec’ing a new one.
This sounds obvious, but without it, teams end up with three slightly different versions of the same product demo scattered across different videos — each produced independently at full cost.
The Production Queue
Video ideas don’t always become videos immediately. The production queue tracks ideas with priorities (high/normal/low) and lets you promote them to full briefs when the time is right.
This separation between “good idea” and “active project” prevents the pipeline from getting clogged with half-started videos that block higher-priority work.
What We Learned
1. Agent Ordering Matters
The Music Director runs after the Voice Director because music must accommodate narration — ducking under spoken words, swelling during transitions, fading for the CTA. If music were generated first, narration would compete with it.
2. Constraints Produce Better Output
Every time we removed a constraint (“let the Writer also suggest visuals”), quality dropped. Specialization with clear boundaries beats generalization every time.
3. Logging Everything Enables Iteration
Every narration clip and music track is generated with full metadata: engine, voice ID, settings, timestamp. When a clip sounds slightly robotic, you can adjust stability from 0.40 to 0.35 and regenerate just that scene — without losing the settings that worked for the other seven.
Try It
The AI Video Producer is available now as part of the EchoForgeX AI Workforce. It runs as an MCP server in your Claude Code environment — bring your own ElevenLabs API key, and you have a full production pipeline.
Pricing starts at $79/month for the Starter tier (3 videos, 15 minutes of TTS narration).
At EchoForgeX, we build AI-powered tools and help businesses integrate AI into their workflows. Get in touch to learn how we can help your team work smarter with AI.