01 / 04
A demo shows possibility; a prototype tests a claim
A polished demo usually uses a handful of clean examples. Real work has incomplete documents, conflicting records, outdated policies, ambiguous requests, and permissions that vary by person. A prototype should encounter those conditions deliberately.
Its job is to answer a specific question: Can this approach meet an agreed bar on the cases we actually see? That means choosing a narrow workflow, running it against representative material, and deciding in advance what success looks like.
02 / 04
Build the test set before tuning the system
Collect examples from the team that owns the workflow. Include routine cases and the uncomfortable ones: missing fields, unusual formatting, contradictory sources, and requests the system should decline. Mark the expected outcome and who agreed on it.
Keep that set separate from the examples used to adjust prompts or retrieval settings. Otherwise the system may look better simply because it has been tuned to the test. Re-run the same cases after every meaningful change so improvements do not quietly break another category.
03 / 04
Measure more than answer quality
Accuracy is only one dimension. A useful prototype also tracks how often it refuses or escalates appropriately, whether cited sources support the answer, how long it takes, what it costs per case, and whether it respects access permissions.
Do not bury uncertainty in a single success percentage. Break the results down by case type. If extraction works well for standard forms and poorly for handwritten notes, that is a design constraint the rollout should reflect.
04 / 04
Design the failure path now
What happens when the model cannot find an answer, returns invalid data, or disagrees with a business rule? A system that hands the case to a person with its sources and uncertainty visible is often far more useful than one that always produces something fluent.
The prototype is ready to inform a build decision when the team can show both where the approach works and where it does not — with enough evidence to estimate the engineering, review, and operating work ahead.