Why two LLM agents beat one on financial classification

How we classify a year of a business's bank transactions in about ten seconds, in the middle of a live underwriting decision.

In our post on the LLM platform that powers underwriting at Parafin, we described building a transaction classification workflow for Pay Over Time in roughly a week. We promised more details, and now, we're ready to share them.

The initial build was a week. Everything after that is what this post covers: optimizing for speed, cost, and reliability. Making it fast enough to serve live underwriting, cheap enough at scale, and trustworthy enough for real credit decisions.

What's actually in a bank statement

Here are four transactions, lightly scrubbed, of the kind we see every day:

The first is Fifth Third Bank's card settlement system depositing a day of card sales (revenue). The second is a repayment on a merchant cash advance. That's existing debt service, which matters a lot if you're about to extend this business more capital. The third depends on who the business is. A restaurant's revenue arrives as card settlements, so a Zelle credit there is usually something else, like the owner moving money in. A cleaning company is different: plenty of its customers pay their invoices over Zelle, so the same credit probably is revenue. And the fourth is a check which will be classified as revenue for some industries.

Parafin provides capital to small businesses through the platforms they already sell on (DoorDash, Amazon, Walmart, and others). For some products, bank transactions are most of what we can underwrite from. When a business first links its bank account, before it has any sales history with us, its statement is the application, and "is this credit revenue or a loan deposit" matters a lot for the offer.

For years, the standard answer to this problem was regex. Ours ran to hundreds of hand-crafted patterns, and it had the problems you'd expect: every bank formats descriptions differently, payment processors appear and disappear, and a pattern that matched last quarter silently misses this quarter's format change. Worse, regex can't use context. Amazon Marketplace on a debit is inventory purchasing; on a credit it's a seller payout, and the string alone can't tell you which.

Our first LLM attempt, back in early 2025, was one long prompt asking a small model to categorize batches of 100 transactions into 16 categories, pipe-separated, no structured output. It mostly worked. But we had no evaluations and no audit trail, so "mostly" was a feeling, not a number. As more and more of our underwriting started depending on the output, that stopped being acceptable.

The constraints

The rebuilt system had to live inside a real-time underwriting flow with a hard budget: when a merchant links a bank account, the whole offer generation cycle should finish in under a minute. Fetching transactions from Plaid can eat 30 seconds of that on its own. A p95 account has about 3,500 transactions from the past year (the average is around 1,100; both were higher when we designed the system, and they drift as the partner mix changes). So: classify a few thousand messy strings, accurately, in well under 30 seconds, thousands of times a day, without the cost curve ruining the product's economics.

Accuracy has a specific meaning here too. Of the 34 categories we classify into, five do most of the work downstream: sales or revenue, loan repayments, overdraft and NSF fees, bank fees, and account transfers or sweeps. Getting revenue recall right matters more than nailing the difference between office expenses and software subscriptions.

The workflow

Here's the production system:

Every box exists because something broke without it. In rough order of the pain that produced them:

Don't classify the same string twice

A year of one business's transactions is extremely repetitive. MAIN STREET CAPITAL PMT 240112 and MAIN STREET CAPITAL PMT 240119 differ only by a date. So before anything reaches a model, we tokenize each description and mask the parts that carry no signal: reference numbers become #, dates become [DATE], ACH SEC codes become [SEC(CCD)] and so on. Then we build a trie per direction, collapse branches that differ only in masked noise, and send one representative per group to the LLM with a hint about how many transactions it stands for. The label fans back out to the whole group afterward.

In production this collapses around 60% of rows before the model sees anything, and cuts total tokens by about 30% with no measurable accuracy change on our eval set. It also produced a free heuristic: a representative that compresses to pure noise, literally just #, can skip the LLM entirely. There's nothing to classify (Checks get the same treatment; CHECK 3051 matches a one-line regex and never costs us a token).

Two agents

We run separate agents for debits and credits, each with its own prompt and its own category list (26 debit categories, 8 credit). This sounds like an implementation detail. It was worth four accuracy points.

The reason is that direction resolves ambiguity better than any instruction we could write. The Amazon Marketplace problem disappears when the debit agent has never heard of seller payouts. We also inject what little business context we have at classification time (legal name, DBA, which platform the application came through). With this context, the agents can distinguish cases like Tony's Pizza on DoorDash, where a deposit is almost certainly a payout rather than a refund.

We tested adding more Plaid fields to the prompt (amounts, dates, Plaid's own category guesses). None of it improved accuracy. It just added tokens.

Picking a batch size

How many transactions per LLM call? We swept it.

At 400 transactions per call, the model simply stopped returning the right number of answers. At 100, accuracy held but each call took 22 seconds. Going smaller than 25 bought nothing: accuracy was already flat and the request count ballooned. We landed on 50 per call because, at up to 90 concurrent requests, a p95 account fits in a single wave of parallel calls; at 25 it would take two. End-to-end classification time is roughly the latency of one call: about 13 seconds, against the old pipeline's 72.

One thing the sweep taught us: per-call latency grows sublinearly with batch size (7 seconds at 5 transactions, 13 at 50, 22 at 100), because output tokens dominate and the model streams them at a fixed rate regardless of how cleverly you pack the input.

When the model drops a transaction

LLMs lose things. Ask for 50 classifications and you'll occasionally get 49, or 51, or one for a transaction ID that doesn't exist. We treat this as the central reliability problem of the whole system, and it's the reason our interface is strict: structured output with an enforced JSON schema, and transaction IDs re-indexed to 0..n-1 within each call (small integers are harder to fumble than UUIDs; the mapping back is persisted for audit). Exactly two errors are retryable: a returned ID that doesn't exist, and a count mismatch. Anything else fails fast.

Retries are partial. If a call comes back with 47 of 50, we keep the 47 and re-send only the 3 misses, up to five attempts. We used to re-run whole batches, which compounds badly: if each transaction independently fails with probability p, the chance a full batch of 50 comes back clean is (1-p)^50, so you can retry forever and keep losing a different straggler each time.

One answer for a thousand transactions is scary

Compression concentrates risk. A busy account's most common description can repeat hundreds or thousands of times a year, and after grouping, every one of those rows inherits whatever the model says once. So representatives with a fan-out of 50 or more get classified five times, with the copies shuffled into random positions across different batches so their errors are uncorrelated, and the majority label wins. Cheap insurance: these high fan-out groups are a tiny share of calls but a huge share of transactions.

Evals are the whole game

Each change above shipped only after it moved a number on the eval set.

The golden set started embarrassingly simple: 24 samples of about 100 transactions each, pulled from real Plaid links and uploaded bank statements across 15+ platforms, hand-labeled by us. About 2,400 transactions. It now lives in our warehouse, syncs to Braintrust every two weeks, and gets fresh production samples labeled by our risk ops team, whose corrections (with reasoning) flow back in. Their disagreements with the model are the most valuable rows we have.

Seven scorers run on every experiment. The one we watch first is coverage: did every input transaction get exactly one classification? It sits at ~1.0 and any drift blocks release, because silent drops corrupt everything downstream. Then overall accuracy across the 34 categories, and macro precision and recall on the five load-bearing ones, so a regression in a rare-but-important category can't hide behind a good average.

The eval harness is also where model choices get made, and it's regularly humbled us. When a major new model family launched, we tested it the same day: calls took three minutes instead of 13 seconds. We ship the full-size model of the family we settled on; its mini variant turned out slower AND less accurate. The nano variant couldn't reliably return five transactions, let alone fifty. None of this is visible on a leaderboard.

The before and after, on our eval set: the old pipeline classified at 74% accuracy with a 72-second p95. The rebuilt one hit ~99% at a 13-second p95, at about six cents per average account per month (the trie compression came later and cut that further). And the system has kept improving underneath those launch numbers. As of this week, production shows about 57,000 classification runs and half a million agent calls over seven days, an end-to-end p95 right around 10 seconds, and a per-call p95 of 4 to 6 seconds, on a model two major versions newer than the one we launched on.

The boring parts are the product

Everything runs on the LLM platform we wrote about previously, which is why the original build took a week. Every agent interaction (prompt, output, token counts, timing, model) is persisted to S3 and lands in a warehouse table an hour later, where anyone at the company can query it. Datadog error logs carry a deep link to the exact Braintrust trace. A WORKFLOW_VERSION constant gets bumped on any behavior change, so every stored classification is traceable to the exact code and prompt that produced it.

That discipline pays for itself. We've swapped underlying models multiple times since launch and the workflow code didn't change: new config, run the evals, compare, ship. In a system that decides whether a small business gets capital, "we can explain any output from any point in time" is not optional.

Failure semantics are deliberate too. If classification fails during an underwriting decision, we retry with backoff and then block the decision rather than guess. The same failure during an onboarding review degrades to a human review queue instead. Fail closed for money, fail human for everything else.

The other hundred million transactions

The real-time flow answers one question: classify this account, now. A different question shows up once you have a portfolio: we improved the workflow, now reclassify everything we've ever seen, so downstream models can retrain on consistent labels.

That's a separate offline path. Transactions are deduplicated by direction, description, and business context, then sent through OpenAI's Batch API at 100 transactions per request, 8,000 requests per file, with a 24-hour turnaround at half the per-token price. A nightly job keeps active accounts fresh for servicing and disbursement reviews in between. The agents, prompts, and evals are identical; only the deadline and the economics change.

What the categories feed

Classified transactions become features for the transaction risk model, a stacked ensemble that estimates the probability a business ends in significant loss:

Two TF-IDF n-gram models read the raw descriptions directly (with the vocabulary filtered so no n-gram can span two transactions, a bug we only caught because a nonsense bigram showed up in the top features). The XGBoost model reads what the LLM understood: revenue, loan repayment, fee and sweep totals over 30-, 90-, and 365-day windows, balance percentiles, all normalized by forecast daily sales. A logistic regression stacks them. A separate model predicts NSF risk, and simple ratios like loan_servicing_90d, the share of outflows going to debt service, turn out to be signals underwriters and models both love.

The model also explains itself: for any score, we can surface the ten n-grams pushing it up or down, which is the difference between "the model said no" and a reviewable decision.

What's next

The offline and real-time paths can drift apart in subtle ways, and reconciling them is ongoing. The model under-detects debt: some loan repayments look like ordinary ACH pulls, and missing them understates a business's obligations, which is the expensive direction to be wrong in. The category aggregates are promising features for the next generation of distress models, and that work has barely started. And continuous bank-based underwriting currently covers a fraction of the businesses it could; the ceiling is very far away.

Most of what's described here was designed, built, and shipped by a very small number of people, which is the part we'd actually like you to take away. If turning 5/3 BANKCARD SYS COMB. DEP. into a defensible credit decision in 13 seconds sounds like your kind of problem, we're hiring across engineering and data science.