Designing a review workflow with AI that does not guess
I designed the rules, review flow, training and coverage for a site review process using AI that handled 30+ reviews a day.
The workflow reached 99% review consistency and became the team's standard process for turning AI flags into consistent human review decisions.
The problem
AI could flag potentially problematic sites, but the flag itself was not the final decision.
A reviewer still had to look at the evidence, check what had already happened and decide whether further action was needed. The risk was letting the workflow inherit assumptions from the AI instead of grounding each decision in the actual company rules and case history.
The process also needed to support more than 30 reviews a day while keeping pending work within a 24 hour handling target.
Start with the workflow, not the AI
I did not begin by asking what the AI could automate.
I first defined what a reviewer needed to know and what a correct review looked like:
- what evidence to check
- in what order to check it
- which rules applied to each scenario
- what counted as already handled
- what to do when evidence was incomplete or conflicting
- when the case needed escalation
- what should never be guessed
Only after the decision path was clear did the AI output have a useful place in the workflow.
What I built
I created the review operation from zero, including the SOP, decision rules, training and rostering.
I worked through normal cases and edge cases so reviewers did not have to improvise when the evidence did not fit the usual pattern. The workflow separated what could be handled by the standard path from what needed a different check or escalation.
Testing against real cases
A workflow can look correct on paper and still fail on real evidence.
I used actual review cases to find ambiguity, missing rules and situations where the workflow could lead someone toward the wrong conclusion. When a case did not fit, I treated that as a signal to improve the rule rather than letting the reviewer or AI invent an answer.
That same principle applies to prompts and reusable AI workflows: generic instructions are not enough. They need to be grounded in the real process, tested repeatedly and updated when the underlying rules change.
Human judgement stays in the loop
I use AI to extend judgement, not replace it.
If the evidence is ambiguous, the right response is not to force the case into the closest standard category. The workflow should make uncertainty visible and route the case to the right person or review path.
That is especially important when the cost of a confident but incorrect answer is higher than the cost of asking for another check.
Delivery
The initial process was running within one week. I completed the full workflow, documentation, training and operating setup within two weeks.
The operation handled more than 30 reviews a day and used the 24 hour target as part of the coverage and queue management process.
Keeping AI workflows current
An AI workflow is not finished when the prompt works once.
When a policy, product rule or review process changes, the instructions and examples that guide the AI also need to change. Otherwise the tool can keep applying an old rule very consistently.
I therefore treat prompts, skills and AI instructions as operational documentation: versioned in practice, tested against real cases and maintained alongside the process they represent.
Outcome
The workflow launched on schedule, supported the required review volume, reached 99% review consistency, and became the team’s standard operating process.
I checked the result for every site and every reviewer: whether the site should pass review or not. In 99% of cases, the reviewer’s pass or not pass decision matched my check.
More importantly, the system made the role of AI explicit: surface and assist, while the real decision remained grounded in documented rules, evidence and a defined review path.