How We Test an AI Finance Tool Before You See It

AI finance accuracy testing starts before a client sees the tool. We test the math in code, gate every change, and check the live number after each deploy.
The rule the tests protect
Every system we build follows one rule. Code computes, and the AI explains. Revenue, expenses, balances and variances are calculated by deterministic code. The same file goes in, the same numbers come out, and headline totals are summed from the detail underneath them, never copied from text a model wrote.
That rule is what makes testing possible. You cannot write a reliable test for a number a language model produced, because the model may produce a different one next time. You can write a test for a function. So the tests sit on the math, and the model only receives numbers that have already passed.
If the model's explanation ever disagrees with the computed figures, the computed figures win, and the report says so.
The bug our sample data hid
While building our own finance reporting tool, we found a bug in how one kind of row was read. Cost rows entered as positive numbers were being counted as revenue. On the affected data, reported revenue was overstated by about 20 percent.
It slipped through because the sample data we tested with followed a sign convention that hid it. Real ledgers do not all follow the same convention. The fix was small once we saw it.
What caught it was not a model reading the output and deciding it looked odd. It was deterministic checks comparing totals that should have agreed and did not. That bug is now covered by tests that run on every change.
Two lessons came out of it. First, test data has to look like client data, including the awkward parts such as sign conventions. Second, the most useful checks are often reconciliations inside the tool itself. If the detail does not sum to the total, something upstream is wrong, and nobody should be reading a summary built on it.
Every change goes through the same gate
Tests only help if nothing ships around them. On October 16, 2026, we ran a change to our own finance tool with no human touch after the request was written.
The request was small. Show the largest payment in the last 30 days on the cash snapshot card. One of us opened the issue and assigned it to an automated flow. That was the only human action.
An AI planner posted a plan. An AI builder opened a merge request with the engine change, the card row, tests and documentation. A separate AI reviewer read the change and posted its review with no blocking findings. A scheduled merge job then checked two things before merging, that the review was clear and that the pipeline had passed. The change went to staging with smoke tests, then to production with smoke tests, and all 5 passed. Finally we checked the live card in production, and it showed the largest payment of the period, a payroll run.
The whole run took 59 minutes, and about 50 of those were the merge job waiting for its 30 minute schedule.
We are careful about what counts. An earlier run, issue 3, also reached production, but we fixed the merge job's review parser partway through. That makes it not a no-touch run, so we do not cite it as one.
The point is not the speed. The point is that an AI wrote the code, and the AI never got to decide whether that code was right. Tests, a separate review and the pipeline decided that.
What a CFO should ask for
Before any AI finance tool reaches your team, ask the vendor, or us, for four things.
1. Tests on the math: Every calculation covered by tests that run on every change, not only at launch.
2. A rerun check: Run the same file twice. The totals should match exactly. If the answer is usually, keep looking.
3. Honest gaps: When data is missing, the tool should say what it could not compute instead of filling the gap and moving on.
4. A live check after release: Someone, or something, confirms the production number after each deploy, not just the test results.
This is the standard behind our finance systems work. A demo shows that a tool can be right once. Testing shows that it stays right while your data and the code both keep changing.