The use of Mathpix OCR with EDICO scientific editor to help blind Students in STEM education

1. Introduction

In this tutorial we'll show how Mathpix OCR can be used to instantly transpose math and science materials in a way that can be read by Blind and Visually impaired students. In the next section we'll briefly describe technologies used by blind people to access to electronic information. Then we'll present the EDICO scientific editor and the Mathpix software. In the section 4, we'll discuss the process a teacher use to convert a math assignment from a PDF into a format that can be read by a blind student. Finally in section 5 we'll demonstrate how the student can read and solve the assignment.
What we want to point out here is that we're not just discussing a way to make math and science texts readable by students who are blind or visually impaired. We're discussing a viable method that, thanks to the powerful Mathpix and EDICO software working together, let blind people read, write and solve math and scientific problems as their sighted schoolmate are able to do.

2. Assistive technologies to access electronic text

A blind person who wants to access to electronic information interacts with the computer by using a software called screen reader.
Screen reader software are available for all major operating systems including Microsoft Windows, Mac OS X and Linux.
Screen reader are specifically designed to be used by blind people, that interact mainly by using the keyboard. By the use of different key combination (called keyboard shortcuts), a blind person can move around the screen, interacts with GUI control and access to the installed application including web browser, email clients, word processors etc.
For more information about how a screen reader works you can refer to The ability net webpage on An introduction to screen readers.
A screen reader let the users decide how to output information. It can use speech, by using a technology called TTS (Text To Speech). A speech synthesizer reads aloud the content of the screen and the user can decide to read one character at a time, by word, by sentence etc. This is not so convenient to explore scientific formula, because the user needs to remember the whole formula in its mind to get a complete overview of it.
A photo containing a blind person in front of a computer with a keyboard and a braille display
In addition to speech synthesizer users can access information by using a Refreshable Braille Display. They provide a tactile access to a text line by representing it in braille.
The braille system of raised dots, a way to represents information that can be read with the fingers by people who are blind or who have low vision. Braille offers a linea representation of information. As reported by AFB, Braille symbols are formed within units of space known as braille cells. A full braille cell consists of six raised dots arranged in two parallel rows each having three dots. The dot positions are identified by numbers from one through six. Sixty-four combinations are possible using one or more of these six dots. A single cell can be used to represent an alphabet letter, number, punctuation mark, or even a whole word. For more information about the Braille System please visit the AFB page related to Braille.
Electronic braille is formed by braille cells of 8 dots. In this way all possible characters in ASCII can be represented, since all possible combinations are 2⁸ = 256 patterns which can be use to represents all differents 256 character.
As reported by AFB, braille display can show up to 80 characters from the screen and is refreshable—that is, it changes continuously as the user moves the cursor around on the screen, using either the command keys, cursor routing keys, or Windows and screen reader commands. The braille display sits on the user’s desk, often underneath the computer keyboard.
Braille cells are dynamically refreshed by push up and down plastic pins, driven by a specifically designed software that interacts with a screen reader. For a more advanced description on how a braille display works Refreshable Braille Now and in the Years Ahead article is a good starting point. In Internet are available a lot of videos that show how a user can access information using Refreshable braille display.

3. Reading and writing STEM using MahtPix and EDICO

The most relevant issue in access of math and science text, using both speech and braille display, is that in many times it is not represented in a linear way (latex, ascii-math, markdown etc). Instead it is represented only in a graphical representation. Screen reader are unable to offer access to information that is not structured. In the case of Math, a screen reader can offer access to MathML, because it is structured, but it can't offer access to a PDF file.
There are a lots of tools that can be used by authors to produce accessible PDFs containing scientific formulas. Actually, they are not largely adopted, because only few people knows about this problem, and, by this way, blind and visually impaired are condemned to remain distant from a full access of scientific information.
Mathpix Snipping Tool is an AI-Empowered OCR that can be easily used by teacher to "take a snapshot" of a math formula and convert it in MathML. Using this tool, the Teacher don't have to manually transcribe examples, tests and homework to be specifically accessible to blind students. She simply take a snapshots of the formula, a process that requires just few seconds, and the blind student can instantly read it using his favorite screen reader rendered both in braille and in speech.
By using Mathpix you can export formula in MathML, so that it can be accessed using advanced technologies contained in recent screen readers, you can export it in LaTex, so that blind students who knows this linear language can easily read and understand it, or you can export it in ASCII math, an easy way in which math formula can be represented.
As we have discussed on the introduction of this tutorial, this is only a part of the problem. By this, the blind student can read the formula. But she can not interact with it. She can read the equation, but the objective is that she learn how to solve it. This is where EDICO comes into play.
EDICO is a Scientific Editor specifically designed for the blind. It is developed and maintained by the ONCE, the Spanish Organization for the Blind and the UCM University. You can find more information and download it for free from the EDICO page on the ONCE Website.
The interface of EDICO has been designed to let blind and sighted people working togeter on scientific texts. It is well integrated with the two major screen reader for the Microsoft Windows OS, so that it can provide information both in braille and in speech.
A screenshot of EDICO Scientific Editor
The screen is horizontally splitted in three parts. In the first part we find the linear representation of the document. It is in a format, specifically designed to be easily readable by blind people using a braille display. In the second area, we'll see a "real" representation of the formula, where a sighted person can read math and chemistry as she's used to. At the bottom there is a visual representation of dot patterns that the blind student read on his braille display. By this, a teacher can help him in understanding the braille and can teach him how to approach to the math using this special tool.
The user can move around the formula, by using arrow keys. When it moves, the speech syntesizer reads the character on the cursor position. The student can also write symbols and letters using keyboard. Keyboard shortcuts are provided for open/close construction like fraction, braces, parenthesis, sum etc. Specific tools have been designed to help the blind student in understanding how the formula is structured and to make the formula exploration easy to do.

4. How to convert a math assignment from a PDF to a text that a Visually impaired student can read

The process is as fast as simple. First of all be sure to have both the MathPix and the EDICO software installed on the computer where you have to perform the transposition.
Then we select in the MathPix editor to export the formula in LaTeX format:
A screenshot of MathPix Snipping Tool window
Then open the PDF containing the math exercise or scan it using a flatbed scanner.
A screenshot of the PDF opened in Adobe Reader
For instance, we have opened this PDF freely available online from the Italian Zanichelli Website:
We then press the shortcut key Control-Alt-1 and we select the area of the equation we want to transpose using the mouse.
A screenshot showing how to capture a formula using MathPix
Finally we open EDICO and select the "Paste from LaTeX" option.
A screenshot showing the Edit menu of the EDICO editor
Then we fix some small semantic errors (like power represented as superscript) from this:
A screenshot showing the replace dialog of EDICO editor
And the exercise is ready to be solved by the blind student!
A screenshot of EDICO with the formula written both in linear format, in real format and in braille format

5. How a visually impaired student can solve an exercise after it has been recognized by MathPix Snipping Tool

After this, a blind student can read the expression using her braille display. First of all she copies the whole exercise without the exercise title and explores it with the EDICO powerful Structure View tool.
A screenshot of EDICO where the user solves the assignment
She is able to identify the three bloks of the formula and to solve each single small block:
A screenshot of EDICO where the user solves the assignment
and then to solve it
A screenshot of EDICO where the user solves the assignment
And to put all together for the final result:
A screenshot of EDICO where the user solves the assignment
Finally the completed homework can be exported as a PDF file or printed to be checked by teacher.
A screenshot of the PDF export window of EDICO software

6. Conclusion

In conclusion we've show how an AI Empowered OCR like MathPix Snipping Tool can work together with a specifically designed Scientific Editor to let blind students access math and scientific materials and to solve complex math assignments.

Recommended for you

Jintang Li
Adversarial Learning on Graph
Adversarial Learning on Graph
This review gives an introduction to Adversarial Machine Learning on graph-structured data, including several recent papers and research ideas in this field. This review is based on our paper “A Survey of Adversarial Learning on Graph”.
7 points
0 issues