AI planning
Before the AI prototype: a project readiness checklist
A convincing demo can answer a handful of friendly questions. A useful pilot has to cope with incomplete records, different permissions, and people who need an answer now. Before choosing a model, write down the task and the evidence that would justify putting it in someone's workflow.
Use the checks below in a short planning session with the person doing the work, the owner of the information, and someone who will support the system. The support assistant is an illustrative example, not a reported client deployment.
1. Describe one task with an observable finish
"Help support" is too broad. "Draft an answer to a delivery-status question using an order record and the current delivery policy" gives the pilot a clear boundary. The user still reviews the draft before sending it. Refunds, address changes, and promises about arrival times stay outside the first release.
Observe the existing process before estimating savings. Record where people look, what they copy, and which exceptions take the longest. A useful comparison includes review and correction time. A draft produced in seconds is not a gain if someone spends several minutes checking it.
- Name the user and the event that starts the task.
- Define the output: a draft, a classification, a summary, or an action.
- List what the pilot must never do on its own.
2. Check whether the information can support the answer
Collect representative records with permission. Include an ordinary case, an incomplete case, an outdated document, and two sources that disagree. Ask the data owner which source wins and how a retired policy is removed. If people cannot resolve the conflict, adding retrieval will not resolve it for them.
Treat access as part of the answer. A support employee who can view one customer's order should not gain access to every customer through the assistant. Map who may see each source and apply those rules when retrieving it. Keep sensitive examples out of development tools unless those tools are approved for that data.
3. Make uncertainty an explicit outcome
Define what happens when the relevant order is missing or the policy cannot support a promise. The system can ask for the order number, show the conflicting sources, or hand the task to a person. Each outcome is more useful than a fluent answer built on a guess.
For the delivery example, the review screen should show the draft, the order status, and the policy passage used. The reviewer needs enough context to reject the draft. A button labeled Approve is not meaningful oversight if the evidence is hidden.
A practical stop rule: if the answer would require a new fact that is absent from the permitted sources, request clarification or review.
4. Write the tests before tuning the prompt
Build a small, maintained set of examples with expected behavior. Separate factual correctness from style. A friendly tone does not compensate for the wrong order or a policy that does not apply. Keep some cases out of prompt tuning so the evaluation includes questions the team did not optimize for.
Choose acceptance limits with the workflow owner. There is no universal percentage that makes an AI application safe. A typo in an internal summary and an unauthorized disclosure have different consequences and should not be averaged into the same score.
| Case | Expected behavior |
|---|---|
| Complete, current record | Draft an answer supported by the visible evidence. |
| Missing order number | Ask for the missing identifier. |
| Conflicting policies | Show the conflict and request review. |
| Unauthorized customer record | Do not retrieve or expose the record. |
| Provider unavailable | Offer the established manual workflow. |
5. Assign ownership before inviting users
Name the person who updates sources, the person who reviews failures, and the person who can pause the feature. Estimate cost using actual tasks, retries, and review effort, then set a budget for the pilot. Keep a path back to the existing process.
At the end of the pilot, compare the same types of work with and without the assistant. Look at completion, corrections, failure reports, and cost together. Expand only when those observations support the next task. If the main problem is inconsistent source data, improving that data may be the most useful result of the pilot.
Sources and further reading
- NIST AI Risk Management Framework
A voluntary framework for managing AI risks across design, use, and evaluation.