KYC document verification

Onboarding a customer means answering two questions about two documents, and they are not the same question.

The first is whether the person submitting the application is the person on the identity document. The second is whether the utility bill or bank statement they uploaded actually shows the address they typed into the form. One is a face problem and one is a text problem, and almost all of the difficulty is in the second.

Matching an address against a blob of OCR

Reading a proof of residence gives you one long run of text with no structure. The document is a bank statement or a council bill, so it contains the address somewhere, surrounded by account numbers, dates, tariffs and marketing copy. You know what you are looking for. You do not know where it is.

Two obvious approaches both fail. Substring matching breaks on the first OCR error, and there is always an OCR error: a scanned street name comes back with a zero for an O or a missing space. Fuzzy matching the address against the whole extracted text also fails, in the opposite direction, because similarity against a thousand-character blob is dominated by the thousand characters you do not care about. A perfect match scores badly simply because the haystack is large.

So the matcher slides a window instead. It takes the target string, measures it in words, and walks a window of that many words across the OCR output, scoring each position and keeping the best. Comparing like with like: a three-word street name is only ever compared against three-word spans, so the score means what it appears to mean.

Around that sits the cheap-first ordering. The address on file is split into its lines and each is checked independently, first by plain substring containment, and only if that fails by the sliding fuzzy match against a similarity threshold of 90. Most lines on most documents match exactly and cost nothing.

The output is the part I would defend hardest. It does not return a boolean. It returns which elements of the address failed to verify, so a human reviewer or a support agent knows whether the applicant fat-fingered a house number or uploaded a bill for an entirely different property. A KYC rejection that says only “denied” turns into a support ticket.

Photographs taken by people holding phones

The face check compares the portrait on the ID against the submitted selfie using a pretrained face recognition model and cosine distance. That part is mostly a matter of picking sensible defaults.

What is not standard is that the pipeline retries the comparison at four orientations. People photograph their ID card on a table, sideways, in whatever rotation the phone happened to record, and face detection on a sideways face does not fail gracefully so much as fail confidently. Rotating through 0, 90, 180 and 270 degrees and accepting the first orientation that verifies costs nothing when the image is upright, because that is the first attempt, and rescues the submission when it is not.

It is a small piece of code that exists entirely because of how the input actually arrives rather than how it is supposed to arrive, which is usually where this kind of system spends its errors.

From pipeline to service

I built the above as a self-contained pipeline. A colleague later moved it into a Flask endpoint that enqueues to Redis with a worker draining the queue, which is the right shape for the workload: the models load real weights, so a synchronous request would spend its life in cold starts and timeouts, and the imports belong inside the worker rather than in a web process whose only job is to accept a job.

One thing did not survive the port, and it is worth noting because it is the ordinary way robustness leaks out of a system. The rotation retry did not make it across. The service version calls the face comparison once, at whatever orientation the image arrived in. Nothing about the refactor was wrong, and the behaviour it dropped was the kind that looks like an implementation detail right up until a user submits a sideways photograph.

The bug I would fix first

The verdict logic has a real defect, and since the flow is mine it is mine to own.

Each check writes an overall status. The face check writes a denial if it fails. The address check then writes its own verdict afterwards, unconditionally. So a submission where the face does not match but the address does ends up reported as verified, because the second write overwrites the first. The individual field recording the face result stays false, so a careful consumer could still catch it, but the field that reads as the verdict is wrong.

Two checks that must both pass should be combined once at the end rather than each writing the shared answer as it finishes. It never mattered in practice because this never carried real applicants, which is exactly the condition under which a bug like this survives review.

Where it went

Nowhere. The product it was built for never launched, so the pipeline was finished, deployed to a serverless endpoint for testing, wrapped into a service, and then shelved with the rest of it.

I still think the address matcher was the right answer to that problem, and it is the piece I would lift out and reuse. Matching a known short string against unstructured extracted text is not really a KYC problem, and every OCR pipeline eventually runs into it.