--- title: Visual Document Understanding with Multi-Modal Image & Text Mining in Spark OCR 3 --- [![John Snow Labs](https://events.johnsnowlabs.com/hs-fs/hubfs/John_Snow_Labs_February2020/Images/johnsnowlabs_logo.png?width=1364&height=468&name=johnsnowlabs_logo.png "John Snow Labs")](https://www.johnsnowlabs.com/) - [Home](https://www.johnsnowlabs.com/) - Products - [Spark NLP](https://www.johnsnowlabs.com/spark-nlp/) - [Healthcare NLP](https://www.johnsnowlabs.com/spark-nlp-health/) - [Spark OCR](https://www.johnsnowlabs.com/spark-ocr/) - [Annotation Lab](https://www.johnsnowlabs.com/annotation-lab/) - [Live Demos](https://www.johnsnowlabs.com/demos/) - [Install Software](https://www.johnsnowlabs.com/install/) - [Install on Databricks](https://www.johnsnowlabs.com/databricks/) - [Healthcare Data Library](https://www.johnsnowlabs.com/welcome-to-the-healthcare-data-library/) - [Healthcare AI Lab](https://www.johnsnowlabs.com/healthcare-ai-platform/) - Solutions - [De-Identification](https://www.johnsnowlabs.com/deidentification/) - [Trial Master File Migration](https://www.johnsnowlabs.com/trial-master-file-migration/) - [Clinical Data Abstraction](https://www.johnsnowlabs.com/clinical-data-abstraction/) - [Clinical Decision Support](https://www.johnsnowlabs.com/clinical-decision-support/) - [Real World Data Curation](https://www.johnsnowlabs.com/real-world-data-curation/) - [Biomedical Question Answering](https://www.johnsnowlabs.com/biomedical-question-answering/) - [Customers](https://www.johnsnowlabs.com/customers/) - [Company](https://www.johnsnowlabs.com/our-story) - [Our story](https://www.johnsnowlabs.com/our-story) - [Our Team](https://www.johnsnowlabs.com/our-team) - [Careers](https://www.johnsnowlabs.com/careers/) - [Awards & Press](https://www.johnsnowlabs.com/awards/) - [Social Impact](https://www.johnsnowlabs.com/social-impact/) - [Announcements](https://www.johnsnowlabs.com/announcements/) - [Learn](https://www.johnsnowlabs.com/resources/) - [NLP Summit](https://www.nlpsummit.org/) - [Webinars](https://events.johnsnowlabs.com/webinars) - [Training](https://www.johnsnowlabs.com/spark-nlp-training/) - [Peer-Reviewed Papers](https://www.johnsnowlabs.com/peer-reviewed-papers/) - [NLP for business Leaders](https://www.johnsnowlabs.com/nlp-for-business-leaders/) - [AI in Healthcare Blog](https://www.johnsnowlabs.com/ai-and-healthcare-blog/) - [Spark NLP Blog](https://www.johnsnowlabs.com/spark-nlp-blog/) - [Spark NLP for Academia](https://www.johnsnowlabs.com/spark-nlp-for-research-and-education/) - [Install Software for Free](https://www.johnsnowlabs.com/install/) - [Schedule a Call](https://www.johnsnowlabs.com/schedule-a-demo/) - - [Home](https://www.johnsnowlabs.com/) - [Spark NLP](https://www.johnsnowlabs.com/spark-nlp) - [State of the Art NLP](https://www.johnsnowlabs.com/spark-nlp) - [NLP For Healthcare](https://www.johnsnowlabs.com/spark-nlp-health/) - [Spark OCR](https://www.johnsnowlabs.com/spark-ocr/) - [Live Demos](https://www.johnsnowlabs.com/spark-nlp-in-action/) - [Free Trial](https://www.johnsnowlabs.com/spark-nlp-try-free/) - [Case Studies](https://www.johnsnowlabs.com/nlp-case-studies/) - [Healthcare AI](https://www.johnsnowlabs.com/ai-platform) - [De-Identification](https://www.johnsnowlabs.com/deidentification/) - [Annotation Lab](https://www.johnsnowlabs.com/annotation-lab/) - [Healthcare AI Lab](https://www.johnsnowlabs.com/ai-platform/) - [AI Lab Architecture](https://www.johnsnowlabs.com/healthcare-ai-platform-features/) - [Data Library](https://www.johnsnowlabs.com/welcome-to-the-healthcare-data-library/) - [AI Case Studies](https://www.johnsnowlabs.com/ai-case-studies/) - [Company](https://www.johnsnowlabs.com/our-story) - [Our story](https://www.johnsnowlabs.com/our-story) - [Our Team](https://www.johnsnowlabs.com/our-team) - [Careers](https://www.johnsnowlabs.com/careers/) - [Awards & Press](https://www.johnsnowlabs.com/awards/) - [Social Impact](https://www.johnsnowlabs.com/social-impact/) - [Announcements](https://www.johnsnowlabs.com/announcements/) - [Learn](https://www.johnsnowlabs.com/resources/) - [NLP Summit](https://www.nlpsummit.org/) - [Webinars](https://events.johnsnowlabs.com/webinars) - [Training](https://www.johnsnowlabs.com/spark-nlp-training/) - [Papers & Conferences](https://events.johnsnowlabs.com/conferences-videos?_ga=2.12548788.1674323299.1603701754-1154577709.1587485642) - [AI in Healthcare Blog](https://www.johnsnowlabs.com/ai-and-healthcare-blog/) - [Spark NLP Blog](https://www.johnsnowlabs.com/spark-nlp-blog/) - [Spark NLP for Academia](https://www.johnsnowlabs.com/spark-nlp-for-research-and-education/) - [Schedule a Demo](https://www.johnsnowlabs.com/schedule-a-demo/) - [Free Trial](https://www.johnsnowlabs.com/spark-nlp-try-free/) - [Account](https://www.johnsnowlabs.com/account/) #### Webinar sign up #### Visual Document Understanding with Multi-Modal Image & Text Mining in Spark OCR 3 ###### Watch live on Wednesday, May 12 @ 2:00 p.m. ET   The Transformer architecture in NLP has truly changed the way we analyze text. NLP models are great at processing digital text, but many real-word applications use documents with more complex formats. For example, healthcare systems often include visual lab results, sequencing reports, clinical trial forms, and other scanned documents. When we only use an NLP approach for document understanding, we lose layout and style information - which can be vital for document image understanding. New advances in multi-modal learning allow models to learn from both the text in documents (via NLP) and visual layout (via computer vision). We provide multi-modal visual document understanding, built on Spark OCR based on the LayoutLM architecture. It achieves new state-of-the-art accuracy in several downstream tasks, including form understanding (from 70.7 to 79.3), receipt understanding (from 94.0 to 95.2) and document image classification (from 93.1 to 94.4).   ###### ![Mykola Melnyk](https://events.johnsnowlabs.com/hs-fs/hubfs/Mykola%20Melnyk.jpeg?width=125&name=Mykola%20Melnyk.jpeg) ###### About the speaker Mykola Melnyk is a senior Scala, Python, and Spark software engineer with 15 years of industry experience. He has led teams and projects building machine learning and big data solutions in a variety of industries - and is currently the lead developer of the Spark OCR library at John Snow Labs. ### Recent blog posts ### Get to work Need NLP Software? [Try Free](https://www.johnsnowlabs.com/spark-nlp-try-free/) Need NLP in Healthcare? [Schedule a Demo](https://www.johnsnowlabs.com/schedule-a-demo/) Need to Learn More? [Register for NLP Summit](https://www.nlpsummit.org/) #### Let’s Talk! 16192 Coastal Highway Lewes, DE 19958, USA [info@johnsnowlabs.com](mailto:info@johnsnowlabs.com) [+1 (302) 786-5227](tel:+13027865227) ### Join our newsletter © 2020 John Snow Labs. John Snow Labs Inc. [Terms of Service](https://www.johnsnowlabs.com/terms-of-service/) | [Refund Policy](https://www.johnsnowlabs.com/refund-policy/) | [Privacy Policy](https://www.johnsnowlabs.com/privacy-policy)