A watcher is the quality-control person for work done by AI. They spot-check what the AI did in their own field, comparing it with the actual records, and they sign off on anything that can't be undone. A bookkeeper checks an AI's invoices, a support lead checks an AI's refunds, a claims adjuster checks an AI's payouts. It's one of the jobs AI creates rather than removes, because someone still has to answer for the decisions that can't be taken back.
What jobs will AI replace, and what's left
The work most exposed to AI is routine, rules-based and high-volume: entering data, answering common questions, coding invoices, processing standard claims, writing boilerplate code. Those are exactly the tasks AI agents now perform end to end.
What's hardest to replace is the part of those jobs that carries judgment and accountability:
- deciding whether an exception is really an exception,
- noticing when something technically allowed is still wrong,
- signing for a payment, a denial or a deletion that can't be reversed,
- and answering for it afterward to a customer, an auditor or a regulator.
Regulators, insurers and courts want a named person behind irreversible decisions. That's why the watcher role grows as automation grows.
What a watcher does
- Reviews a random sample of the AI's actions, one card at a time: what actually happened according to the company's records first, the AI's own account second, and a short checklist ("Is the refund within policy?").
- Answers Yes, No or Not sure. "Not sure" is always safe: it escalates to a senior watcher.
- Signs held actions that can't be undone, approving or denying each one with their own signature.
- Catches planted tests. Deliberate mistakes are hidden in the queue, like a mystery shopper, so everyone can see how good a watcher is at spotting problems.
- Turns incidents into rules, so the same mistake is blocked automatically next time.
The watcher career ladder
- 6Independent auditorChecks other companies' AI oversight and signs off on it, like an outside auditor
- 5Oversight leadDecides how much gets checked and what "good enough" means; reports to customers and regulators
- 4Rule authorTurns mistakes into rules and checklists so they can't happen again
- 3SignerGives the final sign-off on actions that can't be undone
- 2Senior reviewerSettles the "not sure" cases and confirms real problems
- 1ReviewerSpot-checks the AI's work in their own field
Side tracks include red-teamer (tries to trick the system, to find the gaps first) and test designer (writes the hidden test cases).
Watcher roles by industry
| Industry | Reviewer | Signer | Rule author | Comes from |
|---|---|---|---|---|
| Customer support | Ticket action reviewer | Refund and credit approver | Support policy author | Support agents, team leads |
| Finance operations | Transaction reviewer | Payment releaser | Controls author | AP/AR clerks, bookkeepers |
| Insurance claims | Claim decision reviewer | Payout and denial signer | Claims rules author | Adjusters, claims processors |
| Coding agents | Change reviewer | Production change approver | Guardrail engineer | QA testers, junior developers |
| Legal operations | Document reviewer | Filing approver | Review protocol author | Paralegals, document reviewers |
| Lending and KYC | Decision reviewer | Adverse-action signer | Model-risk rules author | Loan processors, KYC analysts |
Where watchers fit among AI governance jobs
"AI governance" jobs are growing, but most of them are policy and compliance roles that sit far from the work itself. Watchers are the hands-on layer that makes governance real:
| Role | Focus | Needs |
|---|---|---|
| AI governance analyst / lead | Policies, frameworks, regulatory mapping (NIST AI RMF, EU AI Act) | Compliance and policy background |
| AI auditor | Independent assessment of AI systems and their controls | Audit or risk background, evidence handling |
| Model risk manager | Validating models in banking and insurance | Quantitative and regulatory background |
| Watcher | Checking the actual actions an AI took, and signing the irreversible ones | Knowledge of the work in your own field |
A watcher who rises to oversight lead or independent auditor ends up doing much of what an AI auditor does, with something most AI auditors lack: years of experience doing the work being checked.
Skills you need, and skills you don't
Domain judgment
Knowing what a correct refund, invoice, claim or deploy looks like, and spotting the one that's technically allowed but wrong.
Evidence habits
Reading the facts before the AI's explanation, writing a clear note when you say No, and saying "Not sure" instead of guessing.
Good checking habits
Two people for big decisions, the person who does the work isn't the one who approves it, and checking a fair sample rather than everything. Most experienced staff already do these without the formal names (four-eyes approval, segregation of duties, audit sampling).
Coding or machine learning
Watchers judge the work, not the AI. The rules and the math are handled by the software.
How watchers are measured
Exams test what someone knew on one day. Watchers are measured all the time: hidden test cases that look exactly like real work are mixed into every queue, the way a mystery shopper tests a store. A watcher's credential is their catch rate, how many of those they spot, and it's always current. That's how pay can follow accuracy instead of speed, and it's the guard against oversight becoming low-paid clickwork.
How to start
- Pick your field. The best watchers already know the work: support, finance, claims, code, legal.
- Learn the basic checking ideas by name: four-eyes approval (two people for big decisions), segregation of duties (the doer isn't the approver), audit sampling (checking a fair sample) and a confidence bound (how sure the sample lets you be).
- Practice on the live demo. Review a sample of an AI support agent's actions and see how many planted tests you catch.
- Volunteer for oversight where your team is deploying AI agents. Someone has to own the review queue.
- Keep your numbers. A record of catch rate and decisions is the strongest case for the next level.
Frequently asked questions
What jobs are AI-proof?
No job is entirely AI-proof, but work that carries judgment on exceptions and accountability for irreversible decisions is the hardest to automate. Watcher roles (reviewing AI work against evidence and signing what can't be undone) grow as automation grows, because a named person must stay accountable.
What does an AI watcher do?
A watcher reviews a random sample of an AI system's actions in their own field, answers Yes, No or Not sure against a short checklist, signs held actions that can't be undone, and turns mistakes into rules. Their accuracy is measured with planted test cases.
What is an AI auditor?
An AI auditor independently assesses AI systems and their controls: what the system is allowed to do, whether its decisions are logged and reviewed, and whether the evidence holds up. Senior watchers who check other companies' AI oversight do this hands-on.
Do I need to know how to code to oversee AI?
No. Watchers judge the work, not the model. What matters is knowing what correct work looks like in your field. Picking what to check, the rules and the math are handled by the software.
Is there a watcher certification?
Not yet. Cosignary plans a certification based on how many hidden test cases a watcher actually catches, launched once real review data exists. Until then, a record of your review decisions and catch rate is the best evidence of skill.
Won't AI replace watchers too?
AI can take over routine review, and may help by flagging items. But regulators, insurers and courts want a named, accountable person behind irreversible decisions, the way a licensed pilot or a signing auditor is required today. Accountability is the part that lasts.
Be a watcher for five minutes
Spot-check an AI support agent's refunds, see if you catch the hidden mistakes, and approve a held refund from your phone.