โ† All tutorials
Licence Recognizer Lab

Lab โ€” train a driver-licence recognizer from scratch

A hands-on, no-prior-ML-experience lab. By the end you'll have trained your own model that, given a photo of a driver licence, says which type it is (New Zealand, or which Australian state) and reads the fields off the card (name, date of birth, licence number, expiryโ€ฆ) โ€” returning clean JSON.

๐ŸŽฏ What you'll build

An object detector (finds the card type & each field) + OCR (reads the text) + a small API.

๐Ÿงฐ Main tools

Ultralytics YOLO (training), Label Studio (labeling), EasyOCR (reading), FastAPI (serving).

๐Ÿง‘โ€๐Ÿ’ป For whom

Developers new to machine learning. Every ML term is explained in a Why this works box.

The mental model. You won't write the "recognition" logic by hand. Instead you'll show the model examples (labeled images), it will learn patterns into a file of numbers called weights, and you'll use those weights to find and read fields on new cards. That loop โ€” label โ†’ train โ†’ use โ€” is all of supervised machine learning.

Read first  โš ๏ธ Responsible use & privacy

Driver licences are sensitive personal information. Building recognition for them is legitimate (identity verification, onboarding, accessibility) โ€” but you must handle the data lawfully and safely.

Non-negotiables.
  • Have a lawful basis / consent to collect and process each licence image.
  • Comply with NZ Privacy Act 2020 (the 13 Information Privacy Principles) and, for Australia, the Privacy Act 1988 / Australian Privacy Principles โ€” plus state ID laws.
  • Prefer synthetic or properly-consented sample data for training and testing (see Step 3). Never scrape real IDs.
  • Never commit real licence images to git. Encrypt at rest, restrict access, and set a retention/deletion policy.
  • Data-minimize: extract only the fields you actually need; redact the rest.
  • Don't build this to enable fraud, impersonation, or surveillance.

This lab is educational and not legal advice โ€” check your jurisdiction and consult a privacy specialist before handling real documents.

High-level architecture

Two phases. You train once (offline) to produce a weights file, then run inference per image using those weights:

Training and inference pipeline Training lane: collect images, label them, write a data config, train YOLO, produce a weights file. Inference lane: an input image goes to the detector (which loads the weights), fields are cropped, OCR reads them, results are parsed and validated, and structured JSON is returned. TRAIN ยท once INFER ยท per image collect images label (boxes) data.yaml train YOLO weights .pt load weights input image detect: type+ fields crop + OCR parse &validate JSON out

Why this design? A single object-detection model can locate both the card (labelled with its type) and each field region in one pass. OCR then reads only those small regions, which is far more accurate than running OCR on the whole photo.

Step 1  Set up your environment

You need Python 3.10+. A GPU makes training faster but is optional โ€” this works on a laptop CPU (just slower).

terminal
# 1) isolated environment
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate

# 2) the core libraries
pip install ultralytics        # YOLO training + inference (installs PyTorch)
pip install easyocr            # reads text from image regions
pip install opencv-python      # image loading / cropping
pip install fastapi uvicorn python-multipart   # the API in Step 11
pip install label-studio       # labeling tool (Step 4)

# 3) sanity check
python -c "import ultralytics, easyocr, cv2; print('ok')"
Why this works: Ultralytics YOLO is a batteries-included object-detection framework โ€” one command trains a model and produces weights. It's built on PyTorch (the deep-learning framework), which pip installs for you. You don't have to write the neural network yourself.

Step 2  Define what you'll detect (the schema)

Decide your classes โ€” the categories the model can output. You need two groups: card-type classes (one box around the whole card) and field classes (a box around each piece of information).

GroupExample classes
Card typecard_nz, card_au_nsw, card_au_vic, card_au_qld, โ€ฆ (one per state you support)
Fieldsf_surname, f_firstnames, f_dob, f_licence_no, f_expiry, f_address, f_class, f_photo
What's typically on the card (fields to consider)

NZ licence: surname, first names, date of birth, licence number, card/expiry dates, licence class (Learner/Restricted/Full), conditions, donor status, photo, signature, a card/version number. Australian licences vary by state (NSW, VIC, QLD, WA, SA, TAS, ACT, NT) โ€” different layouts, colours, number formats and card numbers โ€” which is why "Australia" really means "detect which state." Security features (holograms, UV) verify authenticity and are out of scope for text extraction. Always verify field formats against current official specs.

Start small. For your first run, support just card_nz + card_au_nsw and 4โ€“5 fields. You can add states and fields later by re-labeling and re-training.

Step 3  Collect (or generate) data

The model is only as good as its examples. Aim for ~150โ€“300 images per card type to start, in varied conditions: different lighting, angles, phone cameras, slight blur, and both real-world backgrounds and flat scans.

๐Ÿงช Synthetic (best for privacy)

Make blank licence templates and script random names/dates/numbers onto them (e.g. with Pillow). No real PII โ€” ideal for a first dataset.

๐Ÿค Consented samples

Real images collected with consent and a lawful basis, stored securely and never committed to git.

๐Ÿ” Augmentation

YOLO auto-applies flips, rotations, colour jitter and mosaics during training โ€” so each image teaches more. No extra work.

Put images in a folder; you'll split them into train/ (โ‰ˆ70%), val/ (โ‰ˆ20%) and test/ (โ‰ˆ10%) in the next steps.

Why a val/test split: you measure the model on images it didn't train on, to check it truly learned the patterns rather than memorizing the training set (that failure is called overfitting).

Step 4  Label the data (draw the boxes)

Labeling = telling the model the right answer for each training image by drawing a bounding box around each thing and tagging it with a class. Do this in a labeling tool:

terminal
# launch Label Studio (opens in your browser)
label-studio start
# โ†’ create a project โ†’ import your images โ†’ choose the
#   "Object Detection with Bounding Boxes" template โ†’
#   add your labels (card_nz, f_dob, โ€ฆ) โ†’ start drawing.

Labeling guidelines

  • Draw one box per item; keep boxes tight around the content.
  • Draw the card_* box around the whole card; draw f_* boxes around each field's text.
  • Be consistent โ€” same field, same class, every image. Label every instance you can see.
  • Skip fields that are missing or unreadable on a given card.

When done, export in YOLO format (Label Studio โ†’ Export โ†’ "YOLO"). Alternatives: Roboflow (web, auto train/val split + augmentation), CVAT, or LabelImg.

Consistency beats volume. 200 carefully, consistently labeled images beat 1,000 sloppy ones. Most "the model is bad" problems are actually labeling problems.

Step 5  Understand the label files

YOLO stores labels as one .txt per image (same filename). Each line is one box: class_id x_center y_center width height, with coordinates normalized 0โ€“1 (fractions of image size).

labels/train/licence_042.txt
# class_id  x_center  y_center  width   height   (all 0..1)
0          0.512     0.500     0.940   0.900    # card_nz (whole card)
4          0.300     0.360     0.360   0.070    # f_licence_no
2          0.300     0.470     0.300   0.060    # f_dob

And a dataset config ties folders to class names:

licences.yaml
path: ./dataset          # root folder
train: images/train      # images (labels/ mirror these paths)
val:   images/val
test:  images/test

names:                   # class_id โ†’ name (order matters!)
  0: card_nz
  1: card_au_nsw
  2: f_dob
  3: f_surname
  4: f_licence_no
  5: f_expiry
  6: f_address
Why normalized coordinates: storing boxes as fractions (not pixels) means the labels stay correct no matter what size the image is resized to during training.

Step 6  Train the model โ†’ generate the weights

This is the "learning" step. One command reads your labeled data and produces a weights file.

terminal
# start from a pretrained model (transfer learning) and fine-tune on your data
yolo detect train data=licences.yaml model=yolo11n.pt epochs=100 imgsz=640 batch=16

# โ†’ results + weights land in runs/detect/train/
#   runs/detect/train/weights/best.pt   โ† your trained weights

Prefer Python? Identical result:

train.py
from ultralytics import YOLO

model = YOLO("yolo11n.pt")          # pretrained "nano" model โ€” small & fast
model.train(
    data="licences.yaml",
    epochs=100,                     # how many passes over the data
    imgsz=640,                      # image size the model trains at
    batch=16,                       # images processed at once
)
# best weights: runs/detect/train/weights/best.pt
Key terms, plainly: Weights are the millions of numbers the model adjusts as it learns โ€” the trained model is the weights file (best.pt). An epoch is one full pass over your images. Transfer learning means starting from a model already trained on millions of general images (yolo11n.pt) and nudging it toward your task โ€” so you need far less data. yolo11n is the smallest ("nano"); use yolo11s/m for more accuracy if you have a GPU.

Step 7  Evaluate & iterate

Training auto-reports metrics on the val set (charts in runs/detect/train/). Check the numbers, then improve.

terminal
yolo detect val model=runs/detect/train/weights/best.pt data=licences.yaml
# prints per-class Precision, Recall, and mAP; saves a confusion matrix image
MetricMeansAim
PrecisionOf the boxes it predicted, how many were rightHigh (few false alarms)
RecallOf the boxes it should have found, how many it foundHigh (few misses)
mAP50Overall detection quality (0โ€“1)> 0.8 is a good start

To improve: add more images for weak classes, fix inconsistent labels, train more epochs, or step up to yolo11s. Re-train and re-check โ€” this loop is normal.

Step 8  Use the weights to detect type + fields

Load best.pt and run it on a new image. You get boxes, each with a class and a confidence score.

detect.py
from ultralytics import YOLO
import cv2

model = YOLO("runs/detect/train/weights/best.pt")   # your trained weights
img = cv2.imread("sample_licence.jpg")
result = model(img)[0]                                # run detection

card_type, regions = None, {}
for box, cls, conf in zip(result.boxes.xyxy, result.boxes.cls, result.boxes.conf):
    name = model.names[int(cls)]
    x1, y1, x2, y2 = map(int, box)
    if name.startswith("card_"):
        card_type = name                              # e.g. "card_nz"
    else:
        regions[name] = img[y1:y2, x1:x2]             # crop each field

print("Type:", card_type, "| fields found:", list(regions))
Why crop: the detector tells you where each field is. Cropping to that little rectangle means the next step (OCR) reads clean, isolated text instead of hunting across the whole busy card.

Step 9  OCR โ€” read the text in each field

OCR (Optical Character Recognition) turns the pixels of a cropped field into a string. EasyOCR is the simplest to start with.

ocr.py
import easyocr
reader = easyocr.Reader(["en"])          # load once; reuse for every crop

text_by_field = {}
for name, crop in regions.items():       # regions from Step 8
    lines = reader.readtext(crop, detail=0)   # detail=0 โ†’ just the strings
    text_by_field[name] = " ".join(lines).strip()

print(text_by_field)
# {'f_dob': '14/09/1994', 'f_licence_no': 'AB123456', ...}
Alternatives & accuracy: PaddleOCR and Tesseract are other options. Improve results by upscaling small crops, converting to grayscale, and de-skewing. For messy cards, a document-AI model (Step 12) can beat crop-then-OCR.

Step 10  Parse & validate the fields

Raw OCR text is noisy. Clean it and validate formats with plain code (regex) โ€” and reject anything that doesn't look right.

parse.py
import re

def parse(card_type, text):
    out = {"card_type": card_type, "fields": {}, "warnings": []}

    dob = re.search(r"\b(\d{2}/\d{2}/\d{4})\b", text.get("f_dob", ""))
    if dob: out["fields"]["date_of_birth"] = dob.group(1)
    else:   out["warnings"].append("DOB not recognised")

    # Licence-number formats DIFFER by issuer โ€” verify against official specs.
    # This is only an illustrative pattern (letters + digits).
    lic = re.search(r"\b([A-Z0-9]{6,10})\b", text.get("f_licence_no", "").upper())
    if lic: out["fields"]["licence_number"] = lic.group(1)

    out["fields"]["surname"] = text.get("f_surname", "").title()
    return out
Formats vary and change. NZ and each Australian state use different licence-number and card-number formats. Don't hard-code a format from memory โ€” confirm the current spec for each issuer you support, and treat validation as best-effort, with a human in the loop for anything important.

Step 11  Serve it as an API

Wrap the whole pipeline (detect โ†’ crop โ†’ OCR โ†’ parse) behind one HTTP endpoint so your app can POST an image and get JSON back.

app.py โ€” run: uvicorn app:app
from fastapi import FastAPI, UploadFile
from ultralytics import YOLO
import easyocr, cv2, numpy as np

app    = FastAPI()
model  = YOLO("runs/detect/train/weights/best.pt")   # load ONCE at startup
reader = easyocr.Reader(["en"])

@app.post("/extract")
async def extract(file: UploadFile):
    buf = np.frombuffer(await file.read(), np.uint8)
    img = cv2.imdecode(buf, cv2.IMREAD_COLOR)
    res = model(img)[0]

    card_type, fields = None, {}
    for box, cls in zip(res.boxes.xyxy, res.boxes.cls):
        name = model.names[int(cls)]
        x1, y1, x2, y2 = map(int, box)
        if name.startswith("card_"):
            card_type = name
        else:
            crop = img[y1:y2, x1:x2]
            fields[name] = " ".join(reader.readtext(crop, detail=0))
    return {"card_type": card_type, "fields": fields}
# test:  curl -F file=@sample_licence.jpg http://localhost:8000/extract
Why load once: loading the model and OCR reader is slow; do it at startup (module level), not inside the request handler, so each request is fast.

Step 12  Make it better (and productionize)

๐Ÿ” Active learning

Log low-confidence predictions, label those hard cases, and re-train. Your dataset (and accuracy) compounds.

๐Ÿ“ Perspective fix

Detect the card corners and warp to a flat rectangle before detection โ€” big accuracy win on phone photos.

๐Ÿ“„ Document AI

Swap crop-then-OCR for a layout model like Donut or LayoutLMv3 that reads structured fields end-to-end.

๐Ÿš€ Deploy

Export to ONNX (yolo export format=onnx) for faster serving; containerize with Docker; add auth & logging.

๐Ÿ›ก๏ธ Guardrails

Reject blurry/partial images, flag low-confidence fields for human review, and never auto-approve on a shaky read.

๐Ÿ”’ Compliance

Encrypt storage, restrict access, log access, honour deletion requests โ€” revisit the privacy section as you scale.

Knowledge check

Six questions on the lab. Pick an answer for each, then check your score.

1. In supervised ML, what is "labeling"?

2. What are "weights"?

3. Why start from a pretrained model (yolo11n.pt)?

4. Why crop each field before running OCR?

5. What does a val/test split protect against?

6. What's the responsible way to source training data here?

Next steps

๐ŸŽฏ How YOLO & Label Studio work

The tools, explained โ€” YOLO's detection architecture and Label Studio's labeling model.

๐Ÿ”ฅ PyTorch

The framework underneath โ€” YOLO is built on PyTorch; learn what's happening below.

๐Ÿ”ถ TensorFlow

The other giant โ€” an alternative training stack with strong deployment.

๐Ÿงฌ Embedding models

Turning data into vectors โ€” the idea behind document-AI models.

๐Ÿ““ All tutorials

The hub โ€” LLMs, agents, ML, cloud & more.