Digiplot

Digiplot takes an image of a line chart (a figure from a paper, a datasheet curve, a screenshot) and gives back a CSV of the data behind it, with one column per curve, named from the legend. It’s live at digiplot.ai. I built all of the machine learning and backend: the synthetic training data, the models, the post-processing, and the serving and release infrastructure. Ivan Ng built the web app.

67 → 2 px

curve error after reframing as segmentation (synthetic validation)

0.995

chart-element detection mAP50 (synthetic validation)

3.8 GiB

peak memory for any upload, inside a 6 GB serverless limit

35

versioned releases in 17 days, each gated on evaluation

The problem

Researchers, engineers and students regularly need the numbers behind a published chart, for a meta-analysis or to compare against a datasheet. The standard tools make you click every point by hand and calibrate the axes yourself, which is slow and painful for dense or multi-series charts. The people doing this are precise and sceptical of “AI magic”, so the output has to be right, and it has to fail honestly when it can’t be.

No labelled data, so render it

There’s no dataset of chart images with exact per-curve masks, axis calibration and legend pairings, and labelling one by hand at pixel accuracy isn’t realistic. Therefore I built a synthetic data engine on top of matplotlib that reads the ground truth straight from the plotting library’s internal objects as each chart is rendered.

A few decisions made the biggest difference:

• Masks follow the curve’s geometry, not its ink. A dashed or marker-styled line still gets a continuous mask, so the model learns where the curve is rather than where pixels happen to be drawn.
• Coherent themes, not independent random knobs. Randomising every style setting separately produced charts that looked like no real tool. Sampling whole themes (Excel-like, Plotly-like, scientific, editorial, graph paper) made the data look like the charts people actually upload.
• Deliberately breaking shortcuts. The detector learned that “digits are ticks, words are titles”. Adding categorical axes and a share of gibberish text forced it to use position and layout instead.

Rendered charts then go through JPEG compression, blur and perspective warps, with the labels warped in lockstep.

Six months on the wrong formulation

My first model treated each curve as a set-prediction problem: a DETR-style transformer where every query regresses a fixed number of points along one curve. I iterated on it for months, through deformable attention, refinement stages, hybrid curve and point queries, a DINO-style polyline variant and finally a full rewrite with curve-aware attention.

It plateaued. The validation error sat around 67 pixels, and not a single chart came out fully correct. When I wrote up the failure analysis, the cause turned out to be structural. Wherever two curves cross, their queries attend to almost the same location, so the model is most confused exactly where the task is hardest. More tuning wasn’t going to fix that.

In late May I reframed the task as instance segmentation followed by classical geometry. A Mask2Former-based segmenter (LineFormer, from ICDAR 2023) finds each curve as a mask, and deterministic post-processing turns masks into calibrated points. Curve error dropped to about 2 pixels (1.88 px in the final model).

The two numbers aren’t a strict A/B, because the metrics differ slightly. However, the gap is large enough that the conclusion doesn’t depend on the details. This was the most useful lesson of the project: a published, simpler framing beat a novel architecture I’d invested months in, and the failure analysis is what made it clear when to stop.

The pipeline

The production pipeline combines three small models with a long chain of deterministic geometry:

• Curve segmenter (Mask2Former, Swin-T backbone) produces one mask per curve.
• Chart-element detector (YOLOv8, 14 classes) finds ticks, tick labels, axis titles, legend entries and scale factors.
• Style matcher (a small encoder) pairs each legend swatch with its curve, at 93% top-1 accuracy.
• OCR (PaddleOCR) reads tick values, including units, currency and exponents.
• Post-processing cleans and skeletonises masks, calibrates each axis with a robust fit that rejects outliers, handles log and categorical axes, and samples each curve onto a shared x-grid.

Every model metric above is on synthetic validation data. There isn’t yet a benchmark on real charts with known values.

Fitting it into a serverless CPU box

To keep costs near zero when nobody is using it, inference runs on one CPU container on SageMaker Serverless, which caps memory at 6 GB and scales to zero. Getting a research pipeline into that box took some work:

• All three models were exported to ONNX and checked against PyTorch with parity tests (mask IoU of 1.000). PyTorch was removed from the serving image entirely.
• Benchmarking showed peak memory grew by about 1.7 GiB for every megapixel of input, so large uploads ran out of memory. Resizing every input to the scale the training data was rendered at fixed that and also kept inference in-distribution. Peak memory now stays around 3.8 GiB regardless of upload size.
• A plain standard-library HTTP server replaced the default model server, and the processing Lambda in front of it shrank from a 2 GB, 15-minute worker to a 512 MB invoker.

Warm inference takes about 4 to 7 seconds per chart on CPU.

Knowing which model produced a result

Once there are several versions of code and models in flight, “which model made this CSV?” has to have an exact answer. Code and models are versioned independently. Each code release is a tagged wheel built in CI, and each model bundle lives in a versioned catalog. Every prediction is stamped with the code version and a content hash of the exact model files that were loaded, because a label like “v1” can’t be trusted if someone re-uploads different files under the same name.

Promotion to production is gated on evaluation, and a preflight check reconciles the catalog, the stored files, their hashes and the endpoint’s runtime before a deploy goes out. This made it possible to ship 35 releases in 17 days without losing track of what was running where.

Bugs only production finds

A few of the most instructive problems only appeared once real users did:

• Labels leaking between charts. The app sends several uploads at once, and they landed on the same warm container. The OCR engine wasn’t thread-safe, so one chart’s axis labels could come back on another user’s chart. A coarse lock around inference fixed it, and costs nothing here because SageMaker scales by adding containers rather than threads. A test now drives the real server concurrently.
• A silently wrong model file. A default export shipped the detector’s raw output head instead of the version with non-maximum suppression built in, and a parity test that should have caught it had quietly skipped. The export now checks output shapes, and the parity test can no longer skip.
• Development and production disagreeing. Tuned settings reached the command-line tool but never reached production. Both entry points now build config the same way, with a test that fails if any setting can’t reach both.
• Failing loudly. A chart whose axes can’t be calibrated now returns a clear error (and refunds the user’s credit) instead of pixel coordinates dressed up as data.

The test suite grew from 155 to 519 tests over this period.

Precision as a feature

For this audience, output has to look like it was read off the chart by a careful person. The CSV is rounded to an error budget derived from each axis’s calibration, so it never shows more precision than the chart supports. Sample points snap to the chart’s own gridlines when that doesn’t cost accuracy. For instance, a curve of y = x² that used to come back as 1.85, 2.85 … 9.75 now comes back as 2, 3 … 10. A robust axis fit handles charts with broken axes, cutting calibration error on one test chart from 17 pixels to under 1.

Stack

Models: Mask2Former (LineFormer) via Hugging Face Transformers, YOLOv8, a custom metric-learning encoder, PaddleOCR. Earlier research on DETR, Deformable DETR and DINO-style set prediction.

Training: PyTorch Lightning, Hydra, Weights & Biases, SageMaker training jobs, synthetic data with matplotlib.

Serving: ONNX Runtime on SageMaker Serverless, AWS Lambda, EventBridge, API Gateway (HTTP and WebSocket), S3, GitHub Actions, CodeArtifact, ECR.

The web app (Next.js) was built by Ivan Ng.