Active from 2007 to 2010, this project was the first exploration of using a digital smartpen as a platform for creating and presenting audio/tactile graphics – a system of using touch-based audio to annotate tactile figures to improve access to graphical information for the blind. The project, a collaboration between Smith-Kettlewell and Vanderbilt University, used the Livescribe Pulse Pen – a commercial smart pen device that includes a video camera in its tip, a microprocessor that can run Java programs, memory, an audio speaker, a microphone, and a rechargeable battery. When the tip of the pen is touched to specially-printed paper, the camera “sees” a pattern of tiny dots (commercially known as the Anoto pattern) that encodes page and pixel coordinate data, and can deduce its position on a printed surface. In the audio/tactile applications explored here, the pen was programmed to announce labeling and other annotation information for raised-line graphics touched by the pen’s tip. We also explored complex interactivity with materials such as maps, games, and reference tools. Users were able to explore with their hands while holding the pen to tap the pen on features of interest, allowing them to hear labeling and other annotation information. By this means, blind and visually-impaired users could access complex spatial materials such as maps, charts, graphs, and images. Pen-enabled materials were also shown to make effective graphical user interfaces such as the pen-based scientific calculator.
Demonstration Applications
Under this project we developed four pen-based applications to demonstrate a variety of different educational use cases. These demos were further expanded upon and developed into a successful educational product – the Audio/Tactile STEM Binder – by Touch Graphics, our valued technology transfer partner. Demonstration applications included:
- Periodic Table - A complete reference tool on a single 11” x 17” page showing the periodic table of the elements. Tapping on the page with the pulse pen allows the user to elicit several levels of audio information. This application allowed a blind student to easily experience the form and structure of the Periodic Table, while allowing exploration of detailed information about each element using the smartpen.
- Scientific Calculator - This application turned a single page of “tactile” buttons” into a powerful scientific calculator when tapped with the pen. This application demonstrated ways for embedding context-sensitive help features to simplify and speed up the process of learning about functions and performing potentially complex operations that would otherwise require the user to press multiple keys in sequence.
- Sudoku Game - This mathematical logic game proved extremely popular with our testers! In sudoku the player fills in grids of digits through deductive reasoning, drawing on logic and spatial reasoning skills. This application demonstrated an inexpensive, elegant, and portable way for blind and visually-impaired players to enjoy this popular and educational game.
- Motor Neuron - This tactile diagram of a motor neuron allowed the user to use touch to explore the morphology and anatomy of this nerve cell, while using the smartpen to identify the different parts of the cell and their functions.
Technology Transfer
A number of smartpen-based tools for education and wayfinding have been developed as a result of this Smith-Kettlewell research. Developers of audio/tactile smartpen-based products and materials include:
- American Printing House for the Blind - APH produces a high-quality smartpen-based map of the United States that can be obtained on quota. They are also reported to be developing other audio/tactile materials using this technology.
- The LightHouse for the Blind and Visually Impaired - This San Francisco agency produces a variety of smartpen-based materials, including the Audio/Tactile BART Station Maps developed in collaboration with Smith-Kettlewell.
- Touch Graphics, Inc. - Touch Graphics worked closely with Smith-Kettlewell during and after this project. In addition to the STEM Binder discussed above, they have produced a wide variety of smartpen-based audio/tactile booklets for parks, museums, schools, and other clients.
