#🪨 Rosetta — the handwritten notebook that answers back

Take notes by hand. End a calculation with = and the answer writes itself. Draw something and the notebook names it.

Online handwriting recognition from raw pen trajectory (ink, not images), in the spirit of the iPad's Math Notes — one task-agnostic ink encoder, multiple heads, all trained locally on a laptop GPU.

Real output: the blue ink is handwritten; every orange mark was drawn by the models.

#📓 What it does

The web app is a notebook: a long ruled sheet (with pages) where you write anything — class notes, text, math, doodles — with pen, eraser and undo. Two things make it answer back:

1. Math solves itself. End a calculation with = and pause the pen for a second. The page's strokes are segmented amid the notes (no lasso, no selection), the calculation is recognized (seq2seq, ink → LaTeX), solved with SymPy, and the answer draws itself in orange ink right after the =:

  you write:   2 + 3 =            the notebook writes:   2 + 3 = 𝟓  (orange, animated)
  you write:   2x + 4 = 10        the notebook writes:   x = 3

Re-running never duplicates — the orange ink itself marks a calculation as done. There's an auto toggle and a manual resolver = button; a side panel keeps the solved history rendered with KaTeX.

2. Drawings get named. Sketch something, press desenho?: the most recent drawing (spatially clustered strokes) goes to a classifier trained on Google QuickDraw — 21 categories, 92% validation accuracy — and the label is written in orange ink under the drawing ("gato", "casa", "bicicleta"…), confidence in the toolbar.

3. Equations get identified and plotted. Write x²+y²=4 and the notebook recognizes it, classifies it — "circunferencia · raio 2 · centro (0,0)" — and sketches the graph with axes in orange ink right below, R² and R³: conics (line, circle, ellipse, parabola, hyperbola), quadrics (sphere, ellipsoid, paraboloids, hyperboloids, cone, plane — drawn as isometric wireframes) and arbitrary y = f(x) / z = f(x,y) graphs. Classification is classic linear algebra (quadratic-form eigenvalues via SymPy + numpy), so rotated and translated forms work too. One-variable equations like 2x+4=10 get their solution (x = 3) written below.

Everything persists locally: pages, ink and history survive reloads.

#🧠 Architecture: one encoder, multiple heads

The core design decision (ADR 0006): the input and the encoder are task-agnostic. The same BiGRU ink encoder powers both heads today — swapping tasks is a config line.

flowchart LR
    A["🖊️ Online ink<br/>strokes = points {x, y}"] --> B["Per-point features<br/>x, y, dx, dy, pen, eos"]
    B --> C["<b>Ink encoder</b><br/>BiGRU (or Transformer)<br/><i>task-agnostic</i>"]
    C --> D["<b>LaTeX head</b><br/>Transformer decoder<br/>beam search · 6.4M params"]
    C --> E["<b>Sketch head</b><br/>classifier · 2.1M params<br/>21 QuickDraw classes"]
    D --> F["2 + 3 =  →  5 (SymPy)"]
    E --> G["🐱 'gato · 87%'"]

There is one single ink contract (schemas/ink.schema.json), mirrored in TypeScript (web), Pydantic (api) and dataclasses (ml) — training and inference use the exact same representation. Geometry is canonicalized server-side (bbox normalization + fixed-step arc-length resampling), so pen density and scale never matter.

#📊 Status & results

The honest caveat: CROHME alone is small, so the math recognizer still struggles with long expressions (5% exact on valid) — but the retrain with fixed-step resampling cut the character error rate from 1.41 to 0.81 and made recognition independent of pen density by construction: interpolating the same ink ×3 now yields bit-identical features and 15/15 identical predictions (it used to drop exact-match from 12/15 to 0/15). Scaling to MathWriting (~230k samples) is the next jump. Details in docs/roadmap.md · decisions in docs/adr/.

#🚀 Run the notebook

# terminal 1 — model API (loads both checkpoints at startup)
.\scripts\serve_api.ps1                      # defaults: contas + desenhos, CPU, :8000

# terminal 2 — the notebook
cd web; npm install; npm run dev             # → http://localhost:3000

Write 2 + 3 = and pause; draw a cat and press desenho?. Without checkpoints the API starts in stub mode (HTTP 501) — handy for frontend work against the contract.

If a port gets stuck (orphaned server on Windows): Get-NetTCPConnection -LocalPort 8000 -State Listen | % OwningProcess | % { Stop-Process -Id $_ -Force }

#🧪 Prove the pipeline in 2 minutes (no dataset download)

The repo ships a synthetic ink generator — enough to prove data → training → inference with zero downloads:

uv sync                                                   # or venv + PYTHONPATH=ml/src

python -m hmer_ml.data.synth --out data/synth --n 32      # 1. synthetic InkML
python -m hmer_ml.train --config ml/configs/overfit_synth.yaml   # 2. ~2 min on GPU
python -m hmer_ml.evaluate --config ml/configs/overfit_synth.yaml \
    --ckpt checkpoints/overfit_synth/last.ckpt            # 3. exact_match = 1.0

💡 RTX 50xx GPUs (Blackwell) require the cu128 PyTorch build: pip install torch --index-url https://download.pytorch.org/whl/cu128.

#🏋️ Training on real data

# math: CROHME (~9k expressions; see docs/datasets.md for download)
python -m hmer_ml.train --config ml/configs/crohme.yaml
python -m hmer_ml.evaluate --config ml/configs/crohme.yaml \
    --ckpt checkpoints/crohme_rs/last.ckpt --root data/crohme/valid --beam 4

# sketches: QuickDraw (Phase 4 — same encoder, classification head)
python scripts/download_quickdraw.py            # 21 categories, ~170 MB (byte-range)
python -m hmer_ml.train_sketch --config ml/configs/quickdraw.yaml   # ~50 min on RTX 5050

Built for a single laptop GPU (6–8 GB): mixed precision, gradient accumulation, length bucketing, and checkpointing with automatic resume — interrupt with Ctrl+C and pick up right where it left off with the same command.

#📁 Monorepo layout

├── ml/          # PyTorch: data (InkML/QuickDraw), tokenizer, encoder/heads, train loops,
│   ├── configs/ #   beam search, "=" segmentation. YAML configs with inheritance (_base_)
│   └── tests/   #   40 tests
├── api/         # FastAPI: /recognize (ink→LaTeX), /evaluate (SymPy), /page/process
│   └── tests/   #   (notebook: contas amid notes), /sketch/recognize. Hershey ink font.
│                #   63 tests
├── web/         # Next.js: the notebook — pages, pens/eraser/undo, auto-solve, KaTeX panel
├── xournalpp-plugin/  # paused: Lua plugin with the same backend (no stroke events in Lua API)
├── scripts/     # serve_api.ps1, dataset downloads, glyph generator, e2e checks
├── schemas/     # ink.schema.json — single ink contract (web = api = ml)
└── docs/        # vision, datasets, roadmap, and ADRs (architecture decisions)

Xournal++ (paused experiment): before the notebook, the same backend powered a Lua plugin for Xournal++ (scripts/install_plugin.ps1, Ctrl+M). It works, but the Lua API has no stroke events — no write-time trigger — so the web notebook became the primary interface.

#🔩 Architecture decisions (ADRs)

#🗺️ Roadmap

• Fix math generalization, step 1: CROHME retrained with fixed-step resampling (train = inference, like the sketch head) — CER 1.41 → 0.81
• Step 2: scale to MathWriting (~230k samples), the real generalization jump
• Phase 4 (next): more sketch categories; auto-triage math/text/sketch in the same pass (replace the = pair heuristic with a learned detector)
• Report CER/exact-match on the CROHME 2019 test set after the retrain
• Future: multimodal fusion (ink + rendered image) and LLM-based output refinement