Original research · ICML 2017
Image-to-Markup
(im2latex)
Learning to recover structured markup from images. Neural visual decompilation for LaTeX and HTML, and the IM2LATEX-100K mathematical-expression dataset.
Authors of Image-to-Markup Generation with Coarse-to-Fine Attention, ICML 2017.
The research
Recover the structure behind the image.
Coarse-to-fine attention
The ICML 2017 paper develops an attention-based encoder–decoder for image-to-LaTeX generation. A coarse-to-fine attention mechanism selects a support region before attending at a finer scale, reducing the cost of visual attention.
The study evaluates printed expressions and transfer to handwritten expressions. Read the published paper.
Image-to-markup generation
The 2016 work, What You Get Is What You See: A Visual Markup Decompiler, studies learning to reconstruct presentational markup from rendered images. It demonstrates a shared neural approach on mathematical expressions in LaTeX and synthetic web pages in HTML.
The dataset
IM2LATEX-100K
Mathematical expression images paired with their LaTeX source, introduced in the image-to-markup research of Yuntian Deng and collaborators.
The formulas originate in scientific articles from arXiv, distributed through the Cornell KDD Cup source collection. The dataset pairs rendered formulas with markup for training and evaluating mathematical expression recognition.
Compare the splits and preprocessing of the original archive, hosted raw and processed files, and Hugging Face exports before choosing data for an experiment.
Research history
History. The September 2016 preprint, What You Get Is What You See: A Visual Markup Decompiler, introduced IM2LATEX-100K and studied image-to-LaTeX and image-to-HTML. 2016 BibTeX.
PDF · Original dataset archive, deposited by coauthor Anssi Kanervisto on 21 June 2016.
Explore the work
Code and demonstrations.
Original implementation
The Harvard NLP repository includes the Torch model, preprocessing scripts, image-based evaluation, pretrained-model download links, and the synthetic HTML example.
Attention and qualitative results
The original demonstration lets you browse model outputs and inspect attention over the formula image. It also includes the network diagram and qualitative results.
Further reading
Community resources and discussion
OpenAI im2latex challenge
Historical challenge and solution listing
OpenAI’s archived im2latex challenge links to the Image-to-Markup results, dataset and code, and describes its HTML-generation experiments.
Seq2Seq for LaTeX generation
Implementation tutorial · Guillaume Genthial ·
Guillaume Genthial’s tutorial explains a later TensorFlow implementation and links the original Image-to-Markup research.
Citation
Cite the original work.
If you use IM2LATEX-100K, its processed versions, or the Image-to-Markup model, please cite the ICML 2017 paper and identify the exact data release and preprocessing you used. When using a derivative, credit its additional contribution alongside the original dataset.
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. Image-to-Markup Generation with Coarse-to-Fine Attention. ICML 2017, PMLR 70:980–989.
@inproceedings{pmlr-v70-deng17a,
title = {Image-to-Markup Generation with Coarse-to-Fine Attention},
author = {Deng, Yuntian and Kanervisto, Anssi and Ling, Jeffrey and Rush, Alexander M.},
booktitle = {Proceedings of the 34th International Conference on Machine Learning},
pages = {980--989},
year = {2017},
volume = {70},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR},
url = {https://proceedings.mlr.press/v70/deng17a.html}
}
For the early visual decompiler and HTML experiments, see the 2016 history note and sources.
For the original dataset archive, also identify Anssi Kanervisto’s 2016 Zenodo release, DOI 10.5281/zenodo.56198. The archive citation and the research-paper citation describe related contributions.