Best Paper Award at the 12th Web for all Conference
Best Paper Award at the 12th Web for all ConferenceConsider two images: In one, three kittens look towards the camera, and in the other there’s a diagram of the molecules in a single aspirin. The former shows three fluffy cuties with eyes and noses, the latter only black characters and lines on a white background.
Now imagine that you are blind and have to rely on a screen reader to tell you what’s displayed on your computer monitor. If no other information is available, you’ll need computer software to help you decipher the images.
Companies such as Google, Facebook and Microsoft have come up with ways to identify the cats in images, but they’ve not done as well with helping people who have low or no vision recognize diagram, such as the one of the aspirin molecule.
While at first glance the cats may seem to be more complex an image than that of the Aspirin diagram, once you’ve seen lots of cat pictures, your eye is trained to instantly recognize the shape of yet another cat, even if you’ve never seen the picture before. You can even identify the vague outline of a cat, because the details are not critically important. But with images of science content, details are everything.
When you want to describe diagrams to teach them to students, you need to be precise. It does not help a student to get a vague idea of a diagram. They need to know precisely what it depicts. If you have the picture of a square, it will not help only to know that it is a polygon. To understand a chemical molecule, you need to know all the atoms it contains, and how they are combined via bonds. So if you get a small part wrong, you might as well not depict the diagram at all.
Consequently, science subjects are the most difficult to teach to students with visual or print impairments. For sighted people, think back to your own math or chemistry classes in high school or college: Diagrams and images of geometric shapes or images of atoms and bonds were an integral part of your studies.
check this out
Ideally students who are blind are taught science diagrams with a mixture of tactile graphics, 3D models, and teacher assistance. Students with low vision use tools to magnify images both on paper and on the board. And students with other impairments, such as dyslexia, often need material presented in high contrast colors, such as yellow lines on a blue background, or white on a green one.
In reality, these tools are not always available. When visually impaired students are being taught in mainstream schools, there may be little understanding of their needs, or in environments where there’s not enough money to fund special support, this population can be clearly disadvantaged. In fact, in many countries, even in developed ones, visually impaired students may be actively discouraged from taking science classes, regardless of their ability level, simply because they can’t get the material or support they need for their studies. One would think that moving towards more computerbased teaching, replacing printed textbooks with their electronic version, would resolve this dilemma. But moving from printed material to electronic books only solves parts of the accessibility problem. Although text is immediately accessible and can, for instance, be voiced by an eBook reader, without the need for audio recording of a human reader, graphical content is generally only transformed into electronic images. While visually this achieves the same result as in print, it does the opposite in terms of a graphic’s accessibility. Once you have an image, the information it contains is effectively lost; all you are left with is a rectangle of meaningless pixels. It is totally inaccessible by screen readers, and any text contained in the graphic is also now just a collection of pixels that can no longer be voiced.
If you are relying on magnification, an enlarged, fixed-resolution image will become grainy and illegible. For dyslexic readers, images do not allow them to change foreground and background colors for their reading preferences, or enable highlighting of text within the image.
While there might be accessible captions, they are generally not sufficient to explain the graphic. So to make it fully accessible, one needs to explicitly add the explanations in an alternative format. However, in the mass transition of printed material into electronic books, this problem is often inadequately addressed, or not addressed at all. As a consequence, science textbooks in electronic form are often less accessible than their audio-recording counterparts.
One can solve this problem by creating specialist software for drawing diagrams in an accessible format that allows a reader to listen to descriptions of the diagram and interact with it. But this raises two new barriers:
On the one hand authors actually have to be aware of the existence of the software and use it to draw diagrams. On the other hand, students need to use the same software to read the diagrams, meaning that they not only need to purchase the program, but also learn how to work with it, which imposes a learning curve above and beyond the actual subject they’re studying.
The problem is further compounded by more and more material being produced on the fly. It’s become easy and cheap for everyone to create their own custom handouts, including graphics prepared with drawing programs that they can put online for students to work through independently. But many teachers are unaware of the technology available to create accessible content, or may not have the means to do so. Therefore the knowledge gap between students who can work with graphics and those who can’t continues to widen.
The main motivation for our work is to bridge this gap, and to lower the barrier for students with visual or print impairments to study the sciences, and also to support already working scientists who acquire such impairment now or in the future.
We are building a technology that does not erect new hurdles between student and content. Our software does not rely on authors to generate accessible diagrams, and can deal with images of from and an array of sources, regardless of how they were created.
Images are automatically analyzed and turned into a format that is open, standardized, and accessible. Students can use them on their desktops, laptops or tablets, along with their regular web browser and normal screen-reader or magnification tool, without investing additional time and money in a specialist-reading tool.
While this solution might sound obvious, it’s technically a highly challenging task. Since we have no control over the type and format of the images that come in to be read by our software, the most challenging part is the initial image analysis.
Diagrams can be well-drawn, high-resolution images, but they can also be cropped from scanned books with dirty pages and fingerprints. These images are cleaned so they can be correctly interpreted.
Interpretation naturally depends on the type of diagrams we need to recognize. The current technology concentrates on chemical diagrams of molecules, which are crucial for studying chemistry and the biosciences. As there are many ways to express the same molecule as a diagram, depending on author preference or author emphasis, our software has to deal with these style differences and interpret different depictions of molecules in a uniform way.
Once the basic structure of the diagram, and thus of the chemical molecule is determined, the program further analyzes the molecule, and computes as much information as is needed to generate a detailed and full explanation of the diagram.
But even if sufficient explanatory text is provided, just listening to it generally is not enough for a student to fully grasp the nature of a diagram. In an ideal teaching situation, students are presented with tactile graphics or 3D models that they can explore to get a feel for the structure and shape of an object, and to be able to ask the teacher what the different parts are.
check this out
If we want to simulate that on a computer, we have to provide a means for students to interact with a diagram, to explore its structure step by step, and to obtain meaningful information while doing so.
We consequently do not just provide a single explanation for the whole diagram, but also for its components, as well as summaries for sub-structures. Thus when a reader encounters a diagram in the text; they can first listen to its summary, and then dive into it, exploring it in depth using simple keyboard interaction.
During exploration, sub-structures are described, highlighted and magnified. Print impaired users are further supported by the possibility of changing contrast and color combinations. This interactive exploration provides students with an experience similar to tactile graphics or models. Moreover, the spoken explanations can substitute for teacher explanations providing the learner with additional independence.
We have evaluated our software in experiments with high school students, mainly at specialist education colleges. In particular the reaction of chemistry teachers was very positive, with one stating that it is “of benefit when students are accessing written documentation electronically, as the student would not need to divert to a tactile diagram (which may not be available), but can investigate the structure using the web browser.”
Our vision for the future is that most, if not all diagrams in electronic books and on web pages, can be automatically identified and replaced by accessible formats to provide a seamless reading experience for visually and print impaired students, enabling them to thrive in the sciences. Currently our product realizes this vision for diagrams in chemistry. But we are also working on other subject areas, such as geometry, physics, and biology.
check this out
While providing visually and print impaired students equal access to teaching resources is an important issue to ensure equal opportunity for all learners, regardless of their ability, there is little industry involvement and limited funding available for these tasks. Nevertheless, Volker Sorge is currently taking time off from his academic position to concentrate on these efforts in a dedicated start-up company, Progressive Accessibility Solutions— seeking to raise awareness of the potential diagram access that technology can offer, and to help the technology evolve for the greater good of the next generation of students.
by Dr. Volker Sorge
Dr. Volker Sorge is an Associate Professor at the University of Birmingham, UK. He is working with his research group on the recognition and analysis of scientific documents. He was a visiting scientist at Google, working on accessible mathematics and has been working on accessible chemical diagrams with a team of colleagues in Birmingham: Dr. Mark Lee and Dr. Sandy Wilkinson.
