A model that writes fluent copy can still fail the marketing task. Evaluate the output against examples with known answers, realistic edge cases, and the language people will actually use.
Build a small test set before launch
For a lead-classification assistant, include at least one ordinary inquiry, one missing-location inquiry, one out-of-coverage request, one conflicting request, and one Arabic message with dialect or spelling variation. For each case, write the expected route and why. Keep personal data out of the test set unless it is necessary and safely handled.
Score each output separately for:
- correct task classification;
- required fields extracted without invention;
- unsupported claim or fabricated detail;
- correct escalation when uncertain;
- Arabic meaning and tone;
- valid structure if another system consumes the result.
Do not hide failures in an average score. A single fabricated price or missed safety escalation may outweigh many correct routine classifications.
Compare against a baseline
Run the same test cases through the current human or rules-based process. Record model, prompt, source set, date, and exact version. Review failures by severity, not only frequency. Use a held-out set after tuning so the system is not being graded on examples it was optimized to memorize.
Define release gates in advance: what must be correct every time, what can be reviewed by a human, and which cases should fall back to manual handling. Re-run after changing the prompt, model, sources, or workflow. A passing sample supports only that tested scope; it does not establish universal reliability.
Optional AI-assisted work brief — Small evaluation set
Create a test set with routine, edge, ambiguous, bilingual, and adversarial examples. Define an answer key and severity rubric before running the model. Score factual accuracy, instruction following, tone, privacy leakage, and reviewer correction effort separately. Keep a holdout set for later comparisons. Report failure examples and confidence limits; do not claim quality from a few hand-picked successful outputs.
Worked scenario — Score an evaluation run
For each test case, mark factual support, instruction compliance, language fit, privacy, and usefulness from a defined rubric. A fabricated discount claim should count as a severe failure even if the copy sounds polished. Compare model output with a human reference and preserve the exact input and version. Report the denominator: “17 of 20 passed this rubric,” not “the model is 85% accurate” across all marketing tasks.
AI disclosure: AI tools assisted with drafting. The test scenario is illustrative and not a validated benchmark.