Training and evaluating image-to-LaTeX models with IM2LATEX-100K

1 / choose the data

Use one documented release.

The original IM2LATEX-100K dataset page provides raw and processed downloads, split files, and checksums. The original Zenodo archive was prepared and deposited by coauthor Anssi Kanervisto. The two Hugging Face exports maintained by Yuntian Deng provide smaller subsets in Parquet format.

The HF repository named im2latex-100k-raw contains 203,745 variant rows, representing the same 67,915 source rows as the normalized export. It does not contain 203,745 independent formula images or the complete original archive. Its preprocessing field describes the target string, not a transformation to apply to the image.

2 / inspect before training

Load a pinned example.

This Python example uses Hugging Face datasets and Pillow. It streams the normalized training split at a fixed repository revision and inspects one image–formula pair without materializing the full dataset. Streaming reads the required remote Parquet data; it is not a promise of one-image network traffic.

from contextlib import closing
from io import BytesIO
from PIL import Image
from datasets import Image as DatasetImage, load_dataset

repo = "yuntian-deng/im2latex-100k"
revision = "2cf5f2534bc6d9fb2a333cc99b46c0d331553200"
train = load_dataset(
    repo, revision=revision, split="train", streaming=True, token=False
)
train = train.cast_column("image", DatasetImage(decode=False))
with closing(iter(train)) as rows:
    row = next(rows)
image = Image.open(BytesIO(row["image"]["bytes"])).convert("RGB")
print(row["filename"], image.size)
print(row["formula"])

The available split names are train, val, and test. Each normalized row contains image, formula, and filename. Keep the repository, commit revision, split, and row position alongside predictions; a filename is not a unique row identifier. Inspect the image size and preserve the model’s required channel and pixel normalization conventions.

Select one target representation

For the companion export, filter on preprocessing. The filter below is lazy and scans rows until it finds the selected representation. Use "tokenized" or "normalized" in place of "none" for the other targets.

variants = load_dataset(
    "yuntian-deng/im2latex-100k-raw",
    revision="7bbe85a01359115dcea263bccb35c14d444198c4",
    split="train", streaming=True, token=False,
).cast_column("image", DatasetImage(decode=False))
raw_targets = variants.filter(
    lambda mode: mode == "none", input_columns=["preprocessing"]
)
with closing(iter(raw_targets)) as rows:
    raw_row = next(rows)
print(raw_row["filename"], raw_row["preprocessing"], raw_row["formula"])

Preserve memberships and check overlap

The existing normalized export contains 12 training filenames twice, four filenames shared by training and validation, and two shared by training and test. The variant export inherits these memberships. The dataset cards document these filename checks; they are not an exhaustive image or semantic deduplication audit.

For a comparison using the released splits, preserve and report them. For a new split, group all variants and repeated source examples before splitting, and check source, image, and formula overlap. Keep a manifest of every retained and excluded row. Never shuffle all three representations into independent train and test rows: that puts the same source example on both sides of the evaluation.

3 / preserve the supervision

Keep images, indices, and targets aligned.

Original file indices are zero-based

A raw split line has the form formula_index image_name render_type; the image name has no extension. A processed split line has the form image_filename formula_index, with the extension included. Both indices select a line in the corresponding formula list. Each hosted formula list has 103,559 indexed lines.

# Python 3: LF separates formula records. Preserve blank records and raw bytes.
with open("im2latex_formulas.lst", "rb") as source:
    formulas = [line[:-1] if line.endswith(b"\n") else line for line in source]

with open("im2latex_train.lst", encoding="ascii") as split_file:
    formula_index, image_name, render_type = next(split_file).split()

formula_bytes = formulas[int(formula_index)]
image_filename = image_name + ".png"
print(image_filename, formula_index, len(formula_bytes))

Do not remove blank normalized or tokenized lines, renumber surviving lines, or use a parser that treats embedded carriage returns as extra formula records. The byte-preserving reader above splits on LF. Decide and record a text-decoding policy separately: the HF none targets omit invalid UTF-8 bytes present in 13 training rows and one validation row of the original source. Preserve the original bytes when exact source recovery matters.

Choose the target before tokenizing it for the model

none retains the source-style formula; tokenized separates LaTeX tokens; normalized rewrites parsed expressions into the project’s normalized form. Different LaTeX strings render the same expression. Fix one representation for supervision and state the transformation used for evaluation. Fit a new vocabulary on training data, then freeze it for validation and test.

The historical formula preprocessor calls the repository’s Node/KaTeX parser and preserves one output line per input line. Parser failures leave targets that the filtering step rejects when blank. The filter excludes missing or oversized images, blank targets, and targets over its length limit from training and validation. The original test conversion uses --no-filter.

Record image preprocessing independently

The original image pipeline crops the formula, pads it with white space, groups it into a size bucket, and downsamples it. Its defaults add eight pixels of padding on each side before bucket padding and downsample by a factor of two. Reusing the hosted processed images avoids applying this sequence twice. For a new model, record cropping, resizing, aspect-ratio handling, channels, padding, and pixel scaling; use the same policy at inference.

4 / train a model

Define the image-to-sequence contract.

An image-to-LaTeX model encodes the formula image and decodes a sequence of markup tokens. The original model combines a convolutional image encoder, a row encoder, and an attention-based recurrent decoder. A new encoder–decoder implementation must preserve the same supervision contract:

1. Image input: feed the processed formula image to the visual encoder. When a model supports an image mask, mark padding according to that model’s documented convention.
2. Teacher forcing: for target tokens t1 … tn, the decoder input is BOS t1 … tn and its prediction targets are t1 … tn EOS. Use the chosen framework’s shift operation exactly once.
3. Loss and masks: compute token prediction loss on real target positions, including EOS. Exclude padded target positions. Apply causal and padding masks according to the decoder API; do not assume that a token’s numeric ID is the loss ignore value.
4. Validation: select checkpoints and decoding settings using val. Generate test predictions from the image and the start token, without supplying the gold target prefix.
5. Decoding: save the beam size or sampling policy, maximum output length, EOS behavior, and every truncated or failed prediction. A maximum-length cutoff changes the measured result.

The original data loader constructs shifted targets and batches images with identical dimensions. Its special IDs are padding 1, start 2, end 3, and unknown 4; the loss assigns zero weight to padding. Those IDs belong to the Torch implementation. Use your own tokenizer’s IDs in a different model.

This section specifies the training inputs and checks for a new implementation. The commands below document the original implementation and its sample data.

Original implementation

The Torch7 workflow.

The original repository includes a toy formula dataset and these preprocessing, training, and prediction paths. Run the commands from its repository root. They require the historical Torch7/CUDA/cuDNN environment and Python 2-era preprocessing dependencies listed in the README, together with Node/KaTeX for normalization. They are historical commands, not a newly validated installation or training run.

Prepare the included sample data

python scripts/preprocessing/preprocess_images.py \
  --input-dir data/sample/images --output-dir data/sample/images_processed
python scripts/preprocessing/preprocess_formulas.py --mode normalize \
  --input-file data/sample/formulas.lst --output-file data/sample/formulas.norm.lst

python scripts/preprocessing/preprocess_filter.py --filter \
  --image-dir data/sample/images_processed --label-path data/sample/formulas.norm.lst \
  --data-path data/sample/train.lst --output-path data/sample/train_filter.lst
python scripts/preprocessing/preprocess_filter.py --filter \
  --image-dir data/sample/images_processed --label-path data/sample/formulas.norm.lst \
  --data-path data/sample/validate.lst --output-path data/sample/validate_filter.lst
python scripts/preprocessing/preprocess_filter.py --no-filter \
  --image-dir data/sample/images_processed --label-path data/sample/formulas.norm.lst \
  --data-path data/sample/test.lst --output-path data/sample/test_filter.lst

python scripts/preprocessing/generate_latex_vocab.py \
  --data-path data/sample/train_filter.lst --label-path data/sample/formulas.norm.lst \
  --output-file data/sample/latex_vocab.txt

The default training/validation filter limits images to 500 × 160 pixels and targets to 150 tokens. Vocabulary generation keeps tokens whose training frequency exceeds one. For the full hosted processed release, supply its image directory, normalized formula list, and processed split lists instead; skip transformations already applied to those files.

Train and write predictions

th src/train.lua -phase train -gpu_id 1 \
  -model_dir model/sample -input_feed -prealloc \
  -data_base_dir data/sample/images_processed/ \
  -data_path data/sample/train_filter.lst \
  -val_data_path data/sample/validate_filter.lst \
  -label_path data/sample/formulas.norm.lst \
  -vocab_file data/sample/latex_vocab.txt \
  -max_num_tokens 150 -max_image_width 500 -max_image_height 160 \
  -batch_size 20 -beam_size 1

th src/train.lua -phase test -gpu_id 1 -load_model \
  -model_dir model/sample -visualize -output_dir results/sample \
  -data_base_dir data/sample/images_processed/ \
  -data_path data/sample/test_filter.lst \
  -label_path data/sample/formulas.norm.lst \
  -max_num_tokens 500 -max_image_width 800 -max_image_height 800 \
  -batch_size 5 -beam_size 5

-visualize enables prediction output at results/sample/results.txt. Each row has five tab-separated fields: image filename, gold formula, predicted formula, prediction score, and gold score. The model’s vocabulary is stored in its checkpoint. Check that model/sample/final-model exists before testing: -load_model loads that file when present; it does not by itself establish that a checkpoint was loaded.

The loader skips images it cannot load or that exceed the configured image limits, and truncates long targets. Audit the emitted predictions against the complete evaluation manifest. Use th src/train.lua -h and the actual options when changing the configuration.

5 / evaluate every test example

State the metric and its denominator.

Keep a fixed test manifest and emit one prediction status for every row. Count missing, invalid, and truncated predictions explicitly. For exact accuracy, divide the number of successful matches by the full declared evaluation count; a missing prediction is a failure. Report any separately defined subset alongside its membership rule and size.

Original evaluation commands

python scripts/evaluation/evaluate_bleu.py \
  --result-path results/sample/results.txt \
  --data-path data/sample/test_filter.lst --label-path data/sample/formulas.norm.lst

python scripts/evaluation/evaluate_text_edit_distance.py \
  --result-path results/sample/results.txt

python scripts/evaluation/render_latex.py \
  --result-path results/sample/results.txt \
  --data-path data/sample/test_filter.lst --label-path data/sample/formulas.lst \
  --output-dir results/sample/rendered --no-replace
python scripts/evaluation/evaluate_image.py \
  --images-dir results/sample/rendered/

Rendering uses the original, unnormalized formula list for gold references, so gold and predictions are rendered with the same template. The historical renderer uses pdflatex, ImageMagick at 200 DPI, and cropping. Use a fresh output directory for a new set of predictions: --no-replace reuses existing rendered files.

The historical image matcher binarizes pixels, compares image columns, and applies tolerance rules; it reports matching with and without blank columns. Its reported image accuracy is not strict byte-for-byte or pixel-for-pixel equality. Name the implementation and settings when reproducing that score, and label strict rendered equality separately if you compute it.

Include enough information to repeat the comparison

Record dataset repository and revision or file checksums; split membership and counts; target representation; image transforms; model and tokenizer versions; training configuration; checkpoint selection; decoding settings; metric implementation; and prediction/render failures. Keep original research citations and derivative contributions together. This makes a new model’s result interpretable without changing the benchmark’s history.