Reading handwritten forms with AI: a proof of concept

This is a proof of concept I built during my last months at metricHEALTH Solutions. It ran end to end as a demo. It was not deployed to production.

The problem

Patient support programs enroll people with paper forms. The forms arrive as scans and phone photos, with printed text, cursive handwriting and ticked boxes on the same page. Before a data platform can use a form, the form has to become a structured record.

What I built

A pipeline that takes a scanned form and returns a record in FHIR, the standard format for health data. It has three steps.

1. Clean the image. OpenCV straightens the page, removes noise and evens out the contrast.
2. Read the form. Gemini 2.5 Flash on Vertex AI extracts every field. The prompt carries a fixed schema, so the model has to return typed data and cannot answer in free text.
3. Check the result. The output is validated against the FHIR R4B questionnaire schema before anything is saved, so a malformed extraction stops there.

How it handles doubt

The model reports a confidence score for every field. A field at or above 0.85 passes. A field below that is read again by a stronger model, Gemini 2.5 Pro.

What was measured

In the proof-of-concept runs, 85% of fields cleared the confidence bar on the first pass and 15% went to the second model. That number is a pass rate. It is not an accuracy score, and I do not quote an accuracy figure here.

Privacy by design

Health data rules shaped the build. The model ran in Google's Montreal region, which kept the data in Canada under PHIPA and PIPEDA. Logs, file names and test fixtures never contained patient information. The demo used short-lived credentials, with no stored keys.

What it shows

A working demo of one approach to document AI: extraction against a schema, a confidence gate, and a second reader for the uncertain fields. The same pattern applies to invoices, intake forms and any other paperwork that has to end up as clean data.