AI agents in B2B trade: from hype to production

When we started building LLM agents for our sourcing operation in late 2024, the demos were extraordinary. Production was a different story. Eighteen months later we run agents that handle real RFQs, real supplier shortlisting and real contract drafting — but only after we threw away most of the demo code. Here is what we learned, and what we now think production-grade B2B trade agents actually look like.

The demo trap

Every LLM demo looks miraculous. Drop a 40-page RFQ into ChatGPT, ask for a structured spec sheet and a supplier shortlist, and you get something that looks plausible. The first week we ran this on real RFQs, our team thought we were going to be redundant by Christmas.

Then we audited 200 real outputs against what our senior sourcing managers would have produced. The agents missed lead-time penalties on non-CE suppliers. They suggested manufacturers that had been delisted from Bloomberg Tier-1 six months earlier. They confidently quoted MOQs that did not exist. The demos were not lying — they were hallucinating with confidence.

Three things we threw away

1. Single-shot prompting

The first version was a giant prompt that tried to do everything: parse the RFQ, identify suppliers, draft the comparison sheet, write the email. It worked 60% of the time. The other 40% was a compounding-error nightmare: bad parsing led to bad supplier suggestions led to a confident email recommending the wrong manufacturer.

Production now is six small agents in a chain, each with one job and a strict output schema. RFQ parser. Supplier-fit checker. Compliance verifier. MOQ resolver. Lead-time estimator. Drafting agent. Each output is validated against a schema before the next agent reads it. Latency went up. Reliability went up more.

2. “Just use vector search”

For the first three months we believed our supplier database was a vector search problem. Embed the supplier descriptions, embed the RFQ, find the top 10 matches. The retrieval looked clean in eval. In production it surfaced the wrong supplier almost half the time.

The fix was to admit that B2B sourcing is a structured-query problem first and a semantic problem second. Filter by category, region, capacity, certification, then run semantic ranking inside the filtered set. Sounds boring. Doubled accuracy.

3. Open-ended generation in customer-facing flows

We had an early version of an agent that wrote contract drafts directly to the customer. It generated convincing legal text. It also occasionally invented clauses that did not exist in our template library — clauses that, if signed, would have created problems.

Open-ended generation is now confined to internal tools where humans review every output. Customer-facing agents pick from approved templates and fill specific slots. We lost some flexibility. We gained a system we can actually deploy.

Demo agents impress strangers. Production agents survive contact with real users, real edge cases and real money on the line. Those are very different artifacts.

What production looks like

Three patterns we settled on after eighteen months of trying everything else.

Strict input/output schemas. Every agent in our pipeline has typed inputs and validated outputs. If the schema check fails, the agent retries with an explicit error in the prompt. After two failures, the request escalates to a human. We log every failure and run weekly evals on the failure set — that is how the prompts actually improve.

Tool calls before generation. Where the LLM needs facts — supplier capacity, certification status, current lead times — the agent calls a tool, retrieves the structured answer, and then writes the response. Generation without grounding is where hallucinations come from. Tool-grounded generation has been our biggest reliability win.

Model routing. We do not use one model for everything. Cheap fast models for parsing and routing. The strongest model only for drafting and reasoning steps where the output goes to a customer. The savings are real, but the bigger win is consistency — you do not want a smart-but-occasionally-flaky model on parsing tasks.

What we still cannot automate

The hardest call in B2B sourcing is reading between the lines of a buyer’s ask. A new RFQ from a hospital procurement officer might say “3,000 boxes of nitrile gloves” but the real ask is “I need 3,000 boxes that pass our infection control review without a fight.” Senior sourcing managers know which suppliers ace that review. Agents do not, yet. We feed reviewer notes back into the system, but it is still a human-in-the-loop call for high-value deals.

Where this is going

The interesting frontier is multi-agent collaboration on complex deals. A trade agent that negotiates with a supplier agent, mediated by a compliance agent, all under our human broker’s supervision. We have early prototypes; they are not in production. The infrastructure for evaluating these flows — offline replay, structured eval datasets, deterministic test harnesses — is what we are building this quarter.

If you are building B2B trade software with LLMs and have run into the same walls, we would love to compare notes. And if you want to see what production looks like, our sourcing team uses these agents on every RFQ that comes through — the output is good enough that you would not know it from a senior human draft.