Lab โ train a driver-licence recognizer from scratch
A hands-on, no-prior-ML-experience lab. By the end you'll have trained your own model that, given a photo of a driver licence, says which type it is (New Zealand, or which Australian state) and reads the fields off the card (name, date of birth, licence number, expiryโฆ) โ returning clean JSON.
๐ฏ What you'll build
An object detector (finds the card type & each field) + OCR (reads the text) + a small API.
๐งฐ Main tools
Ultralytics YOLO (training), Label Studio (labeling), EasyOCR (reading), FastAPI (serving).
๐งโ๐ป For whom
Developers new to machine learning. Every ML term is explained in a Why this works box.
Read first โ ๏ธ Responsible use & privacy
Driver licences are sensitive personal information. Building recognition for them is legitimate (identity verification, onboarding, accessibility) โ but you must handle the data lawfully and safely.
- Have a lawful basis / consent to collect and process each licence image.
- Comply with NZ Privacy Act 2020 (the 13 Information Privacy Principles) and, for Australia, the Privacy Act 1988 / Australian Privacy Principles โ plus state ID laws.
- Prefer synthetic or properly-consented sample data for training and testing (see Step 3). Never scrape real IDs.
- Never commit real licence images to git. Encrypt at rest, restrict access, and set a retention/deletion policy.
- Data-minimize: extract only the fields you actually need; redact the rest.
- Don't build this to enable fraud, impersonation, or surveillance.
This lab is educational and not legal advice โ check your jurisdiction and consult a privacy specialist before handling real documents.
High-level architecture
Two phases. You train once (offline) to produce a weights file, then run inference per image using those weights:
Why this design? A single object-detection model can locate both the card (labelled with its type) and each field region in one pass. OCR then reads only those small regions, which is far more accurate than running OCR on the whole photo.
Step 1 Set up your environment
You need Python 3.10+. A GPU makes training faster but is optional โ this works on a laptop CPU (just slower).
# 1) isolated environment
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
# 2) the core libraries
pip install ultralytics # YOLO training + inference (installs PyTorch)
pip install easyocr # reads text from image regions
pip install opencv-python # image loading / cropping
pip install fastapi uvicorn python-multipart # the API in Step 11
pip install label-studio # labeling tool (Step 4)
# 3) sanity check
python -c "import ultralytics, easyocr, cv2; print('ok')"
Step 2 Define what you'll detect (the schema)
Decide your classes โ the categories the model can output. You need two groups: card-type classes (one box around the whole card) and field classes (a box around each piece of information).
| Group | Example classes |
|---|---|
| Card type | card_nz, card_au_nsw, card_au_vic, card_au_qld, โฆ (one per state you support) |
| Fields | f_surname, f_firstnames, f_dob, f_licence_no, f_expiry, f_address, f_class, f_photo |
What's typically on the card (fields to consider)
NZ licence: surname, first names, date of birth, licence number, card/expiry dates, licence class (Learner/Restricted/Full), conditions, donor status, photo, signature, a card/version number. Australian licences vary by state (NSW, VIC, QLD, WA, SA, TAS, ACT, NT) โ different layouts, colours, number formats and card numbers โ which is why "Australia" really means "detect which state." Security features (holograms, UV) verify authenticity and are out of scope for text extraction. Always verify field formats against current official specs.
card_nz + card_au_nsw and 4โ5 fields. You can add states and fields later by re-labeling and re-training.Step 3 Collect (or generate) data
The model is only as good as its examples. Aim for ~150โ300 images per card type to start, in varied conditions: different lighting, angles, phone cameras, slight blur, and both real-world backgrounds and flat scans.
๐งช Synthetic (best for privacy)
Make blank licence templates and script random names/dates/numbers onto them (e.g. with Pillow). No real PII โ ideal for a first dataset.
๐ค Consented samples
Real images collected with consent and a lawful basis, stored securely and never committed to git.
๐ Augmentation
YOLO auto-applies flips, rotations, colour jitter and mosaics during training โ so each image teaches more. No extra work.
Put images in a folder; you'll split them into train/ (โ70%), val/ (โ20%) and test/ (โ10%) in the next steps.
Step 4 Label the data (draw the boxes)
Labeling = telling the model the right answer for each training image by drawing a bounding box around each thing and tagging it with a class. Do this in a labeling tool:
# launch Label Studio (opens in your browser)
label-studio start
# โ create a project โ import your images โ choose the
# "Object Detection with Bounding Boxes" template โ
# add your labels (card_nz, f_dob, โฆ) โ start drawing.
Labeling guidelines
- Draw one box per item; keep boxes tight around the content.
- Draw the
card_*box around the whole card; drawf_*boxes around each field's text. - Be consistent โ same field, same class, every image. Label every instance you can see.
- Skip fields that are missing or unreadable on a given card.
When done, export in YOLO format (Label Studio โ Export โ "YOLO"). Alternatives: Roboflow (web, auto train/val split + augmentation), CVAT, or LabelImg.
Step 5 Understand the label files
YOLO stores labels as one .txt per image (same filename). Each line is one box: class_id x_center y_center width height, with coordinates normalized 0โ1 (fractions of image size).
# class_id x_center y_center width height (all 0..1)
0 0.512 0.500 0.940 0.900 # card_nz (whole card)
4 0.300 0.360 0.360 0.070 # f_licence_no
2 0.300 0.470 0.300 0.060 # f_dob
And a dataset config ties folders to class names:
path: ./dataset # root folder
train: images/train # images (labels/ mirror these paths)
val: images/val
test: images/test
names: # class_id โ name (order matters!)
0: card_nz
1: card_au_nsw
2: f_dob
3: f_surname
4: f_licence_no
5: f_expiry
6: f_address
Step 6 Train the model โ generate the weights
This is the "learning" step. One command reads your labeled data and produces a weights file.
# start from a pretrained model (transfer learning) and fine-tune on your data
yolo detect train data=licences.yaml model=yolo11n.pt epochs=100 imgsz=640 batch=16
# โ results + weights land in runs/detect/train/
# runs/detect/train/weights/best.pt โ your trained weights
Prefer Python? Identical result:
from ultralytics import YOLO
model = YOLO("yolo11n.pt") # pretrained "nano" model โ small & fast
model.train(
data="licences.yaml",
epochs=100, # how many passes over the data
imgsz=640, # image size the model trains at
batch=16, # images processed at once
)
# best weights: runs/detect/train/weights/best.pt
best.pt). An epoch is one full pass over your images. Transfer learning means starting from a model already trained on millions of general images (yolo11n.pt) and nudging it toward your task โ so you need far less data. yolo11n is the smallest ("nano"); use yolo11s/m for more accuracy if you have a GPU.Step 7 Evaluate & iterate
Training auto-reports metrics on the val set (charts in runs/detect/train/). Check the numbers, then improve.
yolo detect val model=runs/detect/train/weights/best.pt data=licences.yaml
# prints per-class Precision, Recall, and mAP; saves a confusion matrix image
| Metric | Means | Aim |
|---|---|---|
| Precision | Of the boxes it predicted, how many were right | High (few false alarms) |
| Recall | Of the boxes it should have found, how many it found | High (few misses) |
| mAP50 | Overall detection quality (0โ1) | > 0.8 is a good start |
To improve: add more images for weak classes, fix inconsistent labels, train more epochs, or step up to yolo11s. Re-train and re-check โ this loop is normal.
Step 8 Use the weights to detect type + fields
Load best.pt and run it on a new image. You get boxes, each with a class and a confidence score.
from ultralytics import YOLO
import cv2
model = YOLO("runs/detect/train/weights/best.pt") # your trained weights
img = cv2.imread("sample_licence.jpg")
result = model(img)[0] # run detection
card_type, regions = None, {}
for box, cls, conf in zip(result.boxes.xyxy, result.boxes.cls, result.boxes.conf):
name = model.names[int(cls)]
x1, y1, x2, y2 = map(int, box)
if name.startswith("card_"):
card_type = name # e.g. "card_nz"
else:
regions[name] = img[y1:y2, x1:x2] # crop each field
print("Type:", card_type, "| fields found:", list(regions))
Step 9 OCR โ read the text in each field
OCR (Optical Character Recognition) turns the pixels of a cropped field into a string. EasyOCR is the simplest to start with.
import easyocr
reader = easyocr.Reader(["en"]) # load once; reuse for every crop
text_by_field = {}
for name, crop in regions.items(): # regions from Step 8
lines = reader.readtext(crop, detail=0) # detail=0 โ just the strings
text_by_field[name] = " ".join(lines).strip()
print(text_by_field)
# {'f_dob': '14/09/1994', 'f_licence_no': 'AB123456', ...}
Step 10 Parse & validate the fields
Raw OCR text is noisy. Clean it and validate formats with plain code (regex) โ and reject anything that doesn't look right.
import re
def parse(card_type, text):
out = {"card_type": card_type, "fields": {}, "warnings": []}
dob = re.search(r"\b(\d{2}/\d{2}/\d{4})\b", text.get("f_dob", ""))
if dob: out["fields"]["date_of_birth"] = dob.group(1)
else: out["warnings"].append("DOB not recognised")
# Licence-number formats DIFFER by issuer โ verify against official specs.
# This is only an illustrative pattern (letters + digits).
lic = re.search(r"\b([A-Z0-9]{6,10})\b", text.get("f_licence_no", "").upper())
if lic: out["fields"]["licence_number"] = lic.group(1)
out["fields"]["surname"] = text.get("f_surname", "").title()
return out
Step 11 Serve it as an API
Wrap the whole pipeline (detect โ crop โ OCR โ parse) behind one HTTP endpoint so your app can POST an image and get JSON back.
from fastapi import FastAPI, UploadFile
from ultralytics import YOLO
import easyocr, cv2, numpy as np
app = FastAPI()
model = YOLO("runs/detect/train/weights/best.pt") # load ONCE at startup
reader = easyocr.Reader(["en"])
@app.post("/extract")
async def extract(file: UploadFile):
buf = np.frombuffer(await file.read(), np.uint8)
img = cv2.imdecode(buf, cv2.IMREAD_COLOR)
res = model(img)[0]
card_type, fields = None, {}
for box, cls in zip(res.boxes.xyxy, res.boxes.cls):
name = model.names[int(cls)]
x1, y1, x2, y2 = map(int, box)
if name.startswith("card_"):
card_type = name
else:
crop = img[y1:y2, x1:x2]
fields[name] = " ".join(reader.readtext(crop, detail=0))
return {"card_type": card_type, "fields": fields}
# test: curl -F file=@sample_licence.jpg http://localhost:8000/extract
Step 12 Make it better (and productionize)
๐ Active learning
Log low-confidence predictions, label those hard cases, and re-train. Your dataset (and accuracy) compounds.
๐ Perspective fix
Detect the card corners and warp to a flat rectangle before detection โ big accuracy win on phone photos.
๐ Document AI
Swap crop-then-OCR for a layout model like Donut or LayoutLMv3 that reads structured fields end-to-end.
๐ Deploy
Export to ONNX (yolo export format=onnx) for faster serving; containerize with Docker; add auth & logging.
๐ก๏ธ Guardrails
Reject blurry/partial images, flag low-confidence fields for human review, and never auto-approve on a shaky read.
๐ Compliance
Encrypt storage, restrict access, log access, honour deletion requests โ revisit the privacy section as you scale.
Knowledge check
Six questions on the lab. Pick an answer for each, then check your score.
Next steps
๐ฏ How YOLO & Label Studio work
The tools, explained โ YOLO's detection architecture and Label Studio's labeling model.
๐ฅ PyTorch
The framework underneath โ YOLO is built on PyTorch; learn what's happening below.
๐ถ TensorFlow
The other giant โ an alternative training stack with strong deployment.
๐งฌ Embedding models
Turning data into vectors โ the idea behind document-AI models.
๐ All tutorials
The hub โ LLMs, agents, ML, cloud & more.