An AI pipeline that finds, scores and routes leads — and stays reliable
We built a production automation that watches external sources for in-niche opportunities, scores each with an LLM, and syncs qualified leads into the outreach CRM — then re-engineered it so it runs unattended without breaking.
The challenge
Automation that impresses in the demo and breaks in production
Finding the right opportunities was manual and slow, and a first automation pass looked great in testing — then failed quietly after launch. Leads were missed or duplicated, the data source itself became unstable, and low-quality contacts slipped through to the outreach tool. In a workflow that touches real people and real accounts, “works in the demo” is not the bar.
What we built
Signals in, scored leads out — end to end
- Real-time detection — monitors external hiring / opportunity signals continuously (every few minutes), not on a manual schedule.
- LLM fit-scoring — each opportunity is scored against a defined niche, and a personalized opener is drafted in-voice.
- Contact resolution — resolves the actual decision-maker behind an opportunity, not just the listing.
- CRM sync — qualified leads flow into the outreach CRM over REST/OAuth, with a personalized field per lead.
- Human-approve gate — nothing is sent without a review step; the system proposes, a person confirms.
- Dedup + config-driven rules — niche, titles and thresholds live in config, not code.
How we made it reliable
From “breaks quietly” to “fails loudly and safely”
- Account-safe sourcing — authenticated scraping got challenged and risked the account, so we moved ingestion onto managed infrastructure. Sourcing became stable and the account is never exposed.
- Confidence-scored resolution — title/company matching with a minimum score and a per-run cap; anything below threshold routes to human review instead of auto-sending.
- Idempotency & dedup — retries never double-write; no duplicate records reach the CRM.
- Rate-limit-aware throttling + QA gate — the pipeline respects API limits and validates every record before the CRM write, so malformed data never ships.
Some client-specific details are generalized; the engineering and the failure-and-fix are exactly as built.
What broke in production, and how we made it hold
Most CRM-plus-AI pipelines demo well and fall over in week two. Here is exactly where this one strained, and what we changed so it stopped.
Pulling signal through a real logged-in account got challenged fast: redirect loops, hidden query IDs, and a live risk of the account itself being restricted. The whole pipeline was one lockout away from going dark.
We moved all ingestion onto managed scraping infrastructure, so the client accounts are never in the loop. Signal keeps flowing even if any single source tightens up.
Matching a real person to a company by name and title is noisy. A wrong contact is worse than none: it burns a send and can reach the wrong person under the client name.
Every match gets a confidence score. Above the threshold it flows; below it, it drops into a human review queue instead of guessing (Confidence Score Check to Human Review in the flow).
Re-runs and overlapping sources produced the same lead twice, which means the same person could be contacted twice: the fastest way to look like a bot.
Idempotency keys plus a dedup check before anything is written. The pipeline holds a hard “0 duplicate records” gate on every pass.
API limits and third-party hiccups used to break a run quietly. You only found out when results stopped, days later.
A rate-limit throttle keeps every call inside provider limits, and every error branch routes to a fail-loudly Slack and email alert. It fails in the open, not in the dark.
An LLM writing outreach at scale will eventually draft something wrong or off-voice. Under the client name, that is a real reputation cost.
A human approval gate sits at the last mile. The model drafts the opener; nothing leaves unreviewed. “Nothing sent unreviewed” is a rule, not a setting.
How we would architect it from day one
If you are building CRM-connected AI automation, these are the decisions that decide whether it survives contact with production. We build them in from the first commit, not after the first outage.
Every step is safe to re-run. Re-processing the same record changes nothing, so a retry can never corrupt or double-send.
Never the client account or API key on the risky edge. If a source tightens up, you swap the edge, not the whole system.
Confidence thresholds with a human fallback beat a confident wrong answer every time. Uncertainty routes to a person, not to the send queue.
Every branch has an error path that reaches a human within minutes. The worst outage is the one you find out about days late.
Anything client-facing passes an approval gate before it sends. Automation does the volume; a person owns the name on it.
Two cheap guards that prevent the two most common blow-ups: duplicate records and rate-limit bans.
The result
An automation that runs unattended — safely
The pipeline moved from an impressive demo that broke on its own to a system that runs continuously, fails loudly instead of silently, and keeps a human in the loop — the exact posture a regulated, review-driven environment needs.
Have an automation that works until it doesn’t?
Enrichment, scoring, CRM sync, LLM pipelines — we build them to run unattended and fail safe, not just to demo.