Before You Automate Expert Work, Find the Exceptions

Most process documents describe the path that can be repeated. They show the required fields, the sequence of approvals, the normal handoff, and the answer that is right often enough to become policy. That is exactly why they are useful—and why they can be misleading as a complete specification of expert work.

Experienced operators spend much of their value at the edges. They notice when two ordinary signals conflict, when a familiar customer is behaving out of character, when the input is technically valid but contextually wrong, or when following the procedure would satisfy the letter of the rule while creating a larger risk.

If an AI system is trained, prompted, or evaluated only on the documented path, the organization may learn that it can reproduce routine execution. It has not yet learned whether the system—or the people overseeing it—can recognize when routine execution should stop.

The International AI Safety Report 2026 describes current general-purpose AI capability as jagged across tasks and contexts. It reports continuing difficulty with long-term planning, many-step tasks, and unexpected obstacles, while also emphasizing rapid improvement and uncertainty about how capabilities will develop. Benchmarks can overstate practical utility when controlled tests do not represent dynamic real-world conditions.

That evidence does not establish that every AI system fails on exceptions. It does not show that undocumented expert knowledge is a leading cause of failed enterprise deployments. The defensible inference is narrower: where an expert workflow contains consequential unusual cases, leaders should test those cases directly rather than assume success on routine examples will generalize.

This is especially important when an AI agent can act, not merely recommend. Permissions, reversibility, monitoring, and recourse determine how far one missed exception can travel before a human sees it.

Some exceptions are statistical outliers. Others occur regularly but remain difficult because the relevant facts arrive late, goals conflict, or local history changes the meaning of a signal. A case can be ordinary in frequency and exceptional in judgment.

Useful categories include boundary cases where a small fact changes the call; missing-information cases where proceeding would be premature; conflicting-authority cases where two legitimate policies point in different directions; and escalation cases where consequence or irreversibility exceeds the operator’s mandate.

The point of a taxonomy is not to create a universal list. It helps leaders sample the terrain deliberately. Without it, an evaluation set tends to become a collection of clean examples that the organization already knows how to solve.

The Critical Decision Method was developed to elicit critical cues, discriminations, and judgments from experienced people working in demanding naturalistic environments. Applied Cognitive Task Analysis similarly helps practitioners identify difficult cognitive demands, cues, strategies, and potential errors, then represent them for training and design.

For automation planning, those traditions suggest a practical line of inquiry. Begin with nonroutine incidents. Reconstruct the changing situation. Ask what the expert noticed, what made the ordinary response inadequate, what evidence would have reversed the call, and when the matter had to move to someone with different authority.

This should not be treated as mind extraction. Experts can misremember, disagree, and rationalize. The cases become stronger when compared with records, near misses, failures, and the accounts of adjacent roles. Preserving disagreement may be more honest than forcing a single golden rule.

An exception register has little value if it sits beside the deployment plan as background reading. Each consequential case should become a test: Does the system detect that the ordinary path no longer applies? Does it ask for the missing information? Does it escalate to the right owner? Can it refuse an action when the consequence is high and the evidence weak?

Use contrasting pairs where two cases look nearly identical but demand different responses. Hold some cases out of development. Vary details that should matter and details that should not. Test the full system—including retrieval, tools, permissions, and human handoffs—because a capable model inside a poorly designed workflow can still produce a fragile operation.

Passing the set does not prove general reliability. It supplies bounded evidence and a clearer failure vocabulary. Production monitoring should then look for new exception classes, changes in context, and recurring escalations that indicate the map is drifting.

A reasonable objection is that modern models can infer patterns no expert explicitly articulated. That may be true for a given task, and explicit rules can even make performance worse when they freeze an outdated interpretation. The answer is not to presume that elicitation must outperform the model. It is to run the comparison.

Test the proposed system on representative and adversarial cases with and without the elicited material. Ask subject-matter experts to examine errors as well as agreements. If the model handles a class of exceptions reliably without additional representation, the organization has learned something valuable. If it fails, the team has found a boundary before deployment rather than after consequence.

Skagway’s practitioner hypothesis is that succession artifacts built around cases, cues, and thresholds may sometimes help create stronger automation evaluations. That proposition is technically plausible and unproven. It belongs in experiments, not in promises.

A usable escalation design names the trigger, the human owner, the information they receive, the response time available, and their authority to pause or reverse the system. “Send to human review” is not a control design if nobody can explain what the reviewer is expected to recognize.

Review intensity should follow consequence, reversibility, observability, and recourse. Requiring a senior expert to inspect every low-risk output may erase the economic value of automation and encourage rubber-stamping. Allowing autonomous action on high-consequence, hard-to-reverse cases because routine accuracy is strong creates a different kind of false economy.

Skagway does not currently position this work as broad AI-governance consulting. The Map is relevant only when the automation project exposes a genuine critical-person dependency—when the organization must understand a departing expert’s decision terrain before it can preserve, transfer, or responsibly test the capability. Otherwise, the right partners may be AI assurance, cybersecurity, legal, risk, or domain-specific technical specialists.