Building a support quality system from scratch
I built a support QA framework from zero, then used real ticket and chat reviews, calibration and coaching to turn it into a shared operating standard.
After the new calibration process was introduced in Aug 2025, average calibration variance was 15% lower over the following three months than in May–Jul 2025.
The problem
The team needed a QA standard that matched its own support work. Another support function already had a guide, but copying it would have created a framework around someone else’s workflow instead of ours.
Without a clear shared standard, the same case could be interpreted differently by different reviewers. Feedback also risked becoming a single comment rather than something that improved future work.
What I built
I built the QA framework from scratch around real support cases.
That included:
- scoring criteria and review rules
- clear examples of what met or missed the standard
- ticket and chat review guidance
- calibration when reviewers interpreted the same case differently
- a way to track recurring issues instead of treating every miss as isolated
- coaching, refresher training or documentation changes to follow up on issues
During development, I reviewed five cases a day and used real examples to refine the guide.
How I handled disagreement
A scoring guide is only useful if two reviewers can use it and reach a similar conclusion.
When interpretations differed, I did not simply choose one answer and move on. I looked at what caused the disagreement: Was the rule unclear? Was an example missing? Was the case an exception the guide did not cover?
The standard was then updated so the same ambiguity was less likely to repeat.
In Aug 2025, I introduced a new calibration meeting approach, started logging disagreements and maintained a shared update sheet so decisions from calibration became explicit rules the QA team could keep using.
From QA score to improvement
A low QA score is not the end of the process.
I look at what sits behind the score. An isolated execution error may need coaching. A repeated mistake across several people may point to an unclear SOP, missing knowledge, weak training or a rule that is difficult to apply.
That creates a more useful loop:
Review → calibrate → find the cause → coach or fix the system → review again.
What changed
The team had a clearer shared QA standard instead of relying on individual judgement. Reviews became easier to calibrate, and recurring issues could be connected to the right response: coaching, refresher training, process clarification or knowledge updates.
After I introduced the new calibration approach in Aug 2025, average calibration variance over the following three months was 15% lower than in May–Jul 2025.
What I would keep improving
A quality system should change when the work changes. I would keep reviewing disagreement patterns, repeated misses and new edge cases, then update the scoring rules, examples and training material together rather than letting them drift apart.