research-article

ChemInk: a natural real-time recognition system for chemical drawings

Authors:

Randall DavisAuthors Info & Claims

IUI '11: Proceedings of the 16th international conference on Intelligent user interfaces

Pages 267 - 276

https://doi.org/10.1145/1943403.1943444

Published: 13 February 2011 Publication History

Abstract

We describe a new sketch recognition framework for chemical structure drawings that combines multiple levels of visual features using a jointly trained conditional random field. This joint model of appearance at different levels of detail makes our framework less sensitive to noise and drawing variations, improving accuracy and robustness. In addition, we present a novel learning-based approach to corner detection that achieves nearly perfect accuracy in our domain. The result is a recognizer that is better able to handle the wide range of drawing styles found in messy freehand sketches. Our system handles both graphics and text, producing a complete molecular structure as output. It works in real time, providing visual feedback about the recognition progress. On a dataset of chemical drawings our system achieved an accuracy rate of 97.4%, an improvement over the best reported results in literature. A preliminary user study also showed that participants were on average over twice as fast using our sketch-based system compared to ChemDraw, a popular CAD-based tool for authoring chemical diagrams. This was the case even though most of the users had years of experience using ChemDraw and little or no experience using Tablet PCs.

References

[1]

C. Alvarado and R. Davis. Sketchread: A multi-domain sketch recognition engine. In Proc. UIST, 2004.

Digital Library

[2]

R. Casey, S. Boyer, P. Healey, A. Miller, B. Oudot, and K. Zilles. Optical recognition of chemical graphics. Document Analysis and Recognition, pages 627--631, 1993.

[3]

L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1):59--70, 2007.

Digital Library

[4]

L. Gennari, L. Kara, T. Stahovich, and K. Shimada. Combining geometry and domain knowledge to interpret hand-drawn diagrams. Computers & Graphics, 29(4):547--562, 2005.

Digital Library

[5]

M. Gross. The electronic cocktail napkin - a computational environment for working with design diagrams. Design Studies, 17(1):53--69, 1996.

[6]

T. Hammond and R. Davis. Ladder: a language to describe drawing, display, and editing in sketch recognition. In International Conference on Computer Graphics and Interactive Techniques, 2006.

Digital Library

[7]

L. Kara and T. Stahovich. An image-based trainable symbol recognizer for sketch-based interfaces. AAAI Fall Symposium: Making Pen-Based Interaction Intelligent and Natural, 2004.

[8]

J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. ICML, pages 282--289, 2001.

Digital Library

[9]

J. LaViola Jr and R. Zeleznik. Mathpad 2: a system for the creation and exploration of mathematical sketches. ACM Transactions on Graphics, 23(3):432--440, 2004.

Digital Library

[10]

D. Lowe. Object recognition from local scale-invariant features. In International Conference on Computer Vision, volume 2, pages 1150--1157, 1999.

Digital Library

[11]

K. Murphy, Y. Weiss, and M. Jordan. Loopy belief propagation for approximate inference: An empirical study. In Proc. of UAI, pages 467--475, 1999.

Digital Library

[12]

J. Nocedal and S. Wright. Numerical optimization. Springer, 2000.

[13]

M. Oltmans. Envisioning Sketch Recognition: A Local Feature Based Approach to Recognizing Informal Sketches. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 2007.

Digital Library

[14]

T. Ouyang and R. Davis. Recognition of hand drawn chemical diagrams. In Proc. AAAI, 2007.

Digital Library

[15]

T. Ouyang and R. Davis. Learning from neighboring strokes: Combining appearance and context for multi-domain sketch recognition. In NIPS, pages 1401--1409, 2009.

[16]

T. Y. Ouyang and R. Davis. A visual approach to sketched symbol recognition. In Proc. IJCAI, 2009.

Digital Library

[17]

B. Paulson and T. Hammond. Paleosketch: accurate primitive sketch recognition and beautification. In Proc. IUI, pages 1--10, 2008.

Digital Library

[18]

T. Sezgin and R. Davis. Sketch based interfaces: Early processing for sketch understanding. In International Conference on Computer Graphics and Interactive Techniques, 2006.

Digital Library

[19]

T. Sezgin and R. Davis. Sketch recognition in interspersed drawings using time-based graphical models. Computers & Graphics, 32(5):500--510, 2008.

Digital Library

[20]

F. Sha and F. Pereira. Shallow parsing with conditional random fields. In Proc. ACL, pages 134--141, 2003.

Digital Library

[21]

M. Shilman, H. Pasula, S. Russell, and R. Newton. Statistical visual language models for ink parsing. AAAI Spring Symposium on Sketch Understanding, 2002.

[22]

M. Shilman, P. Viola, and K. Chellapilla. Recognition and grouping of handwritten text in diagrams and equations. In Frontiers in Handwriting Recognition, 2004.

Digital Library

[23]

M. Szummer. Learning diagram parts with hidden random fields. In International Conference on Document Analysis and Recognition, pages 1188--1193, 2005.

Digital Library

[24]

D. Tenneson and S. Becker. Interpretation of Molecule Conformations from Drawn Diagrams. PhD thesis, Brown University, 2008.

Digital Library

[25]

Y. Xiong and J. LaViola Jr. Revisiting shortstraw: improving corner finding in sketch-based interfaces. In Proc. Eurographics Symposium on Sketch-Based Interfaces and Modeling, pages 101--108, 2009.

Digital Library

Cited By

Mapari S(2024)A Neural Network-Based Framework to Recognize the Handwritten Chemical ExpressionProceedings of World Conference on Information Systems for Business Management10.1007/978-981-99-8346-9_34(405-413)Online publication date: 1-Mar-2024
https://doi.org/10.1007/978-981-99-8346-9_34
Morin LDanelljan MAgea MNassar AWeber VMeijer IStaar PYu F(2023)MolGrapher: Graph-based Visual Recognition of Chemical Structures2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01791(19495-19504)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01791
Cheddi F(2023)Implementing Machine Learning-Based Simulation in Physics Virtual LaboratoryInnovations in Smart Cities Applications Volume 610.1007/978-3-031-26852-6_27(280-290)Online publication date: 2-Mar-2023
https://doi.org/10.1007/978-3-031-26852-6_27
Show More Cited By

Index Terms

ChemInk: a natural real-time recognition system for chemical drawings
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

PaleoSketch: accurate primitive sketch recognition and beautification
IUI '08: Proceedings of the 13th international conference on Intelligent user interfaces

Sketching is a natural form of human communication and has become an increasingly popular tool for interacting with user interfaces. In order to facilitate the integration of sketching into traditional user interfaces, we must first develop accurate ...
Technical Section: Using data mining for digital ink recognition: Dividing text and shapes in sketched diagrams

The low accuracy rates of text-shape dividers for digital ink diagrams are hindering their use in real world applications. While recognition of handwriting is well advanced and there have been many recognition approaches proposed for hand drawn sketches,...
Sketch2Tag: automatic hand-drawn sketch recognition
MM '12: Proceedings of the 20th ACM international conference on Multimedia

In this work, we introduce the Sketch2Tag system for hand-drawn sketch recognition. Due to large variations presented in hand-drawn sketches, most of existing work was limited to a particular domain or limited predefined classes. Different from existing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '11: Proceedings of the 16th international conference on Intelligent user interfaces

February 2011

504 pages

ISBN:9781450304191

DOI:10.1145/1943403

General Chairs:
Pearl Pu
EPFL, Switzerland
,
Michael Pazzani
Rutgers University, USA
,
Program Chairs:
Elisabeth André
Augsburg University, Germany
,
Doug Riecken
CCLS Columbia University, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 February 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI '11

Sponsor:

IUI '11: 16th International Conference on Intelligent User Interfaces

February 13 - 16, 2011

CA, Palo Alto, USA

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

52
Total Citations
View Citations
454
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Mapari S(2024)A Neural Network-Based Framework to Recognize the Handwritten Chemical ExpressionProceedings of World Conference on Information Systems for Business Management10.1007/978-981-99-8346-9_34(405-413)Online publication date: 1-Mar-2024
https://doi.org/10.1007/978-981-99-8346-9_34
Morin LDanelljan MAgea MNassar AWeber VMeijer IStaar PYu F(2023)MolGrapher: Graph-based Visual Recognition of Chemical Structures2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01791(19495-19504)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01791
Cheddi F(2023)Implementing Machine Learning-Based Simulation in Physics Virtual LaboratoryInnovations in Smart Cities Applications Volume 610.1007/978-3-031-26852-6_27(280-290)Online publication date: 2-Mar-2023
https://doi.org/10.1007/978-3-031-26852-6_27
Seddati ODupont SMahmoudi SDutoit T(2022)Towards Human Performance on Sketch-Based Image RetrievalProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549582(77-83)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549582
Jain GChopra SChopra SParihar A(2022)Attention-Net: An Ensemble Sketch Recognition Approach Using Vector ImagesIEEE Transactions on Cognitive and Developmental Systems10.1109/TCDS.2020.302305514:1(136-145)Online publication date: Mar-2022
https://doi.org/10.1109/TCDS.2020.3023055
Tumen RSezgin M(2022)Segmentation and Recognition of Offline Sketch Scenes Using Dynamic ProgrammingIEEE Computer Graphics and Applications10.1109/MCG.2021.306986342:1(56-72)Online publication date: 1-Jan-2022
https://doi.org/10.1109/MCG.2021.3069863
Li LZou CZheng YSu QFu HTai C(2021) Sketch-R2CNN : An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition IEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.298762627:9(3745-3754)Online publication date: 1-Sep-2021
https://doi.org/10.1109/TVCG.2020.2987626
Wang YZhang TYu X(2021)A Component-detection-based Approach for Interpreting Off-line Handwritten Chemical Cyclic Compound Structures2021 IEEE International Conference on Engineering, Technology & Education (TALE)10.1109/TALE52509.2021.9678874(785-791)Online publication date: 5-Dec-2021
https://doi.org/10.1109/TALE52509.2021.9678874
Zhang TDu SYu XZhou J(2021)Rotation Invariance for Offline Handwritten Chemical Organic Ring Structure Symbols Recognition2021 IEEE International Conference on Engineering, Technology & Education (TALE)10.1109/TALE52509.2021.9678644(844-848)Online publication date: 5-Dec-2021
https://doi.org/10.1109/TALE52509.2021.9678644
Arnia FAisyah SSaddami KMunadi K(2021)The Performances of Pre-trained Convolutional Neural Networks in Clothing Sketch Classification2021 18th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)10.1109/ECTI-CON51831.2021.9454854(107-111)Online publication date: 19-May-2021
https://doi.org/10.1109/ECTI-CON51831.2021.9454854
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents