Original research · 2016 / ICML 2017
Image-to-Markup
(im2latex)
Learning to recover structured markup from images. Neural visual decompilation for LaTeX and HTML, and the IM2LATEX-100K mathematical-expression dataset.
Authors of Image-to-Markup Generation with Coarse-to-Fine Attention, ICML 2017.
The research
Recover the structure behind the image.
Image-to-markup generation
The 2016 work, What You Get Is What You See: A Visual Markup Decompiler, studies learning to reconstruct presentational markup from rendered images. It demonstrates a shared neural approach on mathematical expressions in LaTeX and synthetic web pages in HTML.
The project connects visual markup decompilation with model design, datasets, and evaluation. Read the original paper.
Coarse-to-fine attention
The ICML 2017 paper develops an attention-based encoder–decoder for image-to-LaTeX generation. A coarse-to-fine attention mechanism selects a support region before attending at a finer scale, reducing the cost of visual attention.
The study evaluates printed expressions and transfer to handwritten expressions. Read the published paper.
The dataset
IM2LATEX-100K
Mathematical expression images paired with their LaTeX source, introduced in the image-to-markup research of Yuntian Deng and collaborators.
The formulas originate in scientific articles from arXiv, distributed through the Cornell KDD Cup source collection. The dataset pairs rendered formulas with markup for training and evaluating mathematical expression recognition.
Our dataset page brings together the original archive, the hosted raw and processed files, and the Hugging Face exports, with the distinctions needed for reproducible comparisons.
Publication history
Original sources, dated.
Original dataset archive
IM2LATEX-100K on Zenodo, prepared and deposited by coauthor Anssi Kanervisto. This is the publication date displayed by the archive.
What You Get Is What You See: A Visual Markup Decompiler
Yuntian Deng, Anssi Kanervisto, and Alexander M. Rush. The original preprint presents image-to-markup generation for LaTeX and HTML and introduces IM2LATEX-100K. Dated preprint · PDF
Image-to-Markup Generation with Coarse-to-Fine Attention
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. Published in PMLR 70, pages 980–989. The revised arXiv version was submitted on 13 June 2017. Publication record · Dated revision
Explore the work
Code and demonstrations.
Original implementation
The Harvard NLP repository includes the Torch model, preprocessing scripts, image-based evaluation, pretrained-model download links, and the synthetic HTML example.
Attention and qualitative results
The original demonstration lets you browse model outputs and inspect attention over the formula image. It also includes the network diagram and qualitative results.
Citation
Cite the original work.
If you use IM2LATEX-100K, its processed versions, or the Image-to-Markup model, please cite the ICML 2017 paper and identify the exact data release and preprocessing you used. When using a derivative, credit its additional contribution alongside the original dataset.
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. Image-to-Markup Generation with Coarse-to-Fine Attention. ICML 2017, PMLR 70:980–989.
@inproceedings{pmlr-v70-deng17a,
title = {Image-to-Markup Generation with Coarse-to-Fine Attention},
author = {Deng, Yuntian and Kanervisto, Anssi and Ling, Jeffrey and Rush, Alexander M.},
booktitle = {Proceedings of the 34th International Conference on Machine Learning},
pages = {980--989},
year = {2017},
volume = {70},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR},
url = {https://proceedings.mlr.press/v70/deng17a.html}
}
For the original dataset archive, also identify Anssi Kanervisto’s 2016 Zenodo release, DOI 10.5281/zenodo.56198. The archive citation and the research-paper citation describe related contributions.