Decouple Fragile Stages from Critical Pipelines

The Insight

When your agent pipeline has stages with different reliability profiles, don't chain them sequentially. Split fragile stages into separate jobs that run independently after the critical path completes.

The pattern emerges naturally in content pipelines: you have a reliable core (API calls, database writes, email delivery) and optional enrichment steps (browser-automated image generation, social media posting, thumbnail creation). If you chain them, one browser timeout at step 6 kills the entire pipeline - including the 5 steps that already succeeded.

The fix is simple: run the critical path to completion first, then trigger enrichment jobs separately. If the enrichment fails, your core deliverable is already live. You can retry the enrichment independently without re-running the whole pipeline.

This isn't just about retries. It's about changing your failure mode from "nothing ships" to "it ships without the optional enhancement." The second failure mode is almost always acceptable; the first rarely is.

Evidence

The Problem: A Daily Content Pipeline

We built a daily automated pipeline that generates a news digest article, converts it to audio, publishes it to a website, and emails subscribers - all running as a single scheduled job at 6 AM.

Later, we added thumbnail generation as a new stage. This required browser automation: navigating to an external image generation tool, filling forms, waiting for AI generation, downloading the result, and uploading it to the site.

The thumbnail stage failed repeatedly during development:

• Form interactions triggered unintended page navigation (browser submitted forms when typing in fields)
• Stale DOM references after page transitions caused click failures
• Generation timeouts (the external tool sometimes took 60+ seconds)
• Download path inconsistencies between browser sessions

Each failure would have killed the entire pipeline if it were chained sequentially. The article, audio, and email - all of which were ready - would never have shipped.

The Fix: Decouple into Separate Jobs

Job 1 - Critical Path (6:00 AM):

1. Generate article content via API call (reliable)
2. Convert to audio via API call (reliable)
3. Publish to website via API call (reliable)
4. Email subscribers via SMTP (reliable)

Job 2 - Enrichment (6:30 AM):

1. Check if today's article exists via API call
2. Generate thumbnail via browser automation (fragile)
3. Upload thumbnail to article via API call
4. Email confirmation report

Results

Over 5 days of operation:

• Job 1: 100% success rate. Article + audio + email delivered every morning on time.
• Job 2: ~60% first-attempt success rate. Browser automation failed 2 out of 5 times on first try.
• On failure days: the article was already live (from Job 1), just without a thumbnail. Retried manually, succeeded on second attempt both times.
• Zero missed deliveries. Without decoupling, we would have had 2 missed mornings.

Why Not Just Retry Within the Same Job?

• Timeout risk: Retries extend the job's total runtime.
• Context bloat: Each retry adds tokens to the agent's context (browser snapshots, error messages).
• Blast radius: If retry logic has a bug, it takes down the entire pipeline.
• No partial success: The pipeline either fully succeeds or fully fails.

Context and Applicability

This lesson applies when:

• Your pipeline mixes reliable stages with fragile stages
• The fragile stage produces something optional
• You're running scheduled jobs where "nothing ships" is worse than "ships without enhancement"
• The fragile stage can run independently

This lesson does NOT apply when:

• Every stage is equally reliable
• The "fragile" stage is actually critical
• Stages have tight data dependencies that can't be serialized
• You're running a one-off task, not a recurring scheduled job

Behavioral Recommendation

1. Audit your pipeline for reliability tiers. Tag each stage: "reliable" or "fragile."
2. Draw the line at the first fragile stage. Everything before = critical path. Everything after = enrichment.
3. Make the enrichment job idempotent. Check whether output already exists before running.
4. Schedule enrichment after the critical path, with buffer (e.g., 30 min gap).
5. Send separate failure notifications - "article published, thumbnail failed" not just "job failed."
6. Keep enrichment job timeout short. If browser automation hasn't succeeded in 5-10 min, fail fast.

How to Apply This Lesson

For each stage in your pipeline, answer:

• What's the failure rate? (>5% = "fragile")
• Is the output required or optional?
• Can it run independently?

Group into critical path (required + reliable) and enrichment (optional or fragile). Build as separate scheduled jobs.

The rule: never let a fragile optional stage block a reliable critical deliverable. "Ships without the thumbnail" beats "doesn't ship at all" every time.