Multisensory Interactive documents for children with visual impairments: Examples of pedagogical applications and users experience
JULIE: Hello, everybody and thank you for your interest. My name is Julie Mulet. I'm a French postdoctoral fellow in psychology and learning sciences. I'm currently working in a multidisciplinary research laboratory in collaboration with researchers from several fields and professionals. I'm working on the evaluation and the deployments of digital touch design to help visually impaired people to learn, as well as their caretakers. The purpose of my presentation will be to introduce you to some of the tools developed by my laboratory, as well as some feedback on their use. But let's start with some introductory elements. Visual representations are very common in education because they are useful for accessing specific concepts in several learning fields, for example, mathematics, science, history, geography and many more. Specialist centres in charge of education for visually impaired students usually provide them and their ordinary teachers with adaptations, such as Braille transcriptions for written contents and raised-line documents for graphical representations, like maps and diagrams. However, these adaptations have several limitations for both learners and caretakers. For learners, even when Braille reading is mastered, which not always the case, access to written information is slow, sequential and limited to a small perceptual field. Moreover, young students don’t always employ the best exploratory procedure. Another cognitive limitation is that graphical representation with a Braille legend provide information from two distinct sources. For example, a tactile city map and a Braille legend. This configuration generates interruptions of the tractile exploration and a split attention effect, resulting in an overload of working memory. Thus, in a cognitive point of view, these learning supports are not optimal for the learners. For caretakers, such adaptations necessitate purchasing expensive materials. They're also time consuming and require expertise onsite educators often lack. Finally, they're also hard to modify and refresh. More generally, exploratory studies identify three major needs. First, the need for tools to help children access symbolic representations, including maps, in their own empowering way. Secondly, the need to enable caretakers to easily design and produce highly specific educational materials. Finally, the need for more collaboration between children, sighted or not, as well as between children and caretakers. In this context, the Cherchons Pour Voir research laboratory, which can be translated as Looking To See, developed two innovative digital devices, DERi, and the Tangible Box. Both emerged from a participatory design method, involving researchers in psychology, computer sciences, learning sciences but also caretakers and persons with visual impairment. I will now present these tools, some exciting applications and first results from pilots to this. Let's start with DERi whose name means in French interactive raised-line drawing. DERi is made of a raised-line document to cover, laid on a touch-sensitive surface that makes specific elements of the covering interactive by delivering an audio feedback. Thus, visually impaired students can explore a tactile picture and autonomously get access to sounds or verbal descriptions without the help of the teacher nor legend. A point of interaction can provide several geo feedback according to the gesture performed. For example, the points directed on Paris, capital of France, may trigger name of the city with one tap and some explanations with a double tap. DERi can be used on surfaces of different sizes depending on the intended uses. A large tactile table allows collective use during group learning. A tablet allows one board use, useful in the case of auto-teaching, for example, for orientation and mobility. Caretakers can easily design the interactions associated with tactile map using the editing software. They first have to load the drawing they want to make interactive. The editor allows the use of different image formats. Then they draw the areas they want to make interactive. Finally, they select the sound file, as well as the action that will trigger the sound. As mentioned earlier, a safe zone allows the triggering of several sound interactions depending on the gesture performed. The sound can be verbal, for example, recorded explanations, or nonverbal, for example, music or sounds from real environment. DERi has been used for several years by caretakers. For example, maps were proposed for geography, history, orientation and mobility. Shimmer were used in sciences and mathematics. Also applications involve the use of quiz or the consultation of mind map. All these documents were interested in relearning situations. Several qualitative studies assessed the perceptions and the effect of DERi. They found that the tool led to strong satisfaction from both caretakers and children. The tool is easy to use, even for children with motor impairments. Students can use it independently, which has good effects on their sense of competence. We also found better retention of information when they were delivered by DERi in comparison with a non-interactive map with Braille legend. Caretakers also confirmed that the editor is easy to use and make it simple to create or modify documents. Thus, DERi seems to have great potential for learning performances and is well accepted by both caretakers and visually impaired students. The Tangible Box is another interactive device at very low cost, similar on the principle that integrates in addition tangible interaction with objects. More specifically, the Tangible Box is a multipurpose interactive tabletop device with camera-tracking coloured objects moving onto the surface. Depending on the usage context, it is possible to put different types of covers to set it up. For example, one can use swirl paper, a thin wooden plate with engraved REF, or a LEGO small-scale model. Then the user can move objects, for example, a small character onto the surface in order to explore the drawing or the small-scale model. The objects can be made with an additive printer. There are two pieces of software associated with the Tangible Box. As for DERi, the Tangible Box editor allows the teacher to define different areas of the drawing or model with specific geo feedback. The Tangible Box player processes the real-time location of some specific pieces, for example, a pawn on the board. Depending on the sitting configuration, the player triggers the sound of the text file when the pawn enters an interactive zone. I will now present some exciting devices that use the Tangible Box. One of them was co-designed with an orientation and mobility teacher. The teacher's goal was to encourage children to acquire a multi-representation of their education centre with its many rooms and corridors. She used a regular REF drawing made of magnets stuck on a board. Some magnets represented the corridors and walls of the centre and then a magnet represented a character that can be moved onto the drawing. The teacher played pre-recorded sounds from her cell phone in order to provide children with sound feedback about the actual location of the character. The purpose of these sounds was to associate audio experiences that had occurred during the actual journey with the exploration of the symbolic map. However, for young students, the representation of a moving person by a magnet is very abstract and difficult to represent. Moreover, triggering sound information at the right time and location is demanding activity for the teacher that requires constant attention. In response to these limitations, we co-designed an interactive device with the teacher. It is made of a tangible, small-scale mock-up built with LEGO pieces that represents the simplified settings of the centre and a software that triggers sound feedback according to the pawn location in a model. The sounds were recorded during the actual exploration of the building. They can be verbal, like a person's voice when the pawn enter his or her office or nonverbal, like the sound of the elevator. (indistinct computer voice) We tested this device with five visually impaired students during individual learning sessions and their orientation and mobility teacher. These learning sessions were recorded. After this, we assessed the teacher's perceptions toward the tool through some instructor-led interview. During this interview, we identified an issue with the use of the Tangible Box. The magnet under the pawn was easily detachable. The supervision of this issue required additional attention from the teacher. It was also an additional instruction that the teacher provided to the student during the exploration task. Be careful, this is a magnet, you must not lift the pawn. Hence, it represented an additional task for the student as well. In addition, the pawn could sometimes be difficult to manipulate. The teacher mentioned that the student wanted to push it by holding the body instead of the head but by doing so, it was blocked in the corridor. This is an information I had to provide all the time. On the other hand, the teacher expressed a strong perceived usefulness of the Tangible Box. One of the positive aspects is that using the tool is pleasant and motivant. She pointed out that the students had positive reactions after a sound was triggered. Moreover, the interactive feedback provides immediate cues but it doesn't sound like corrective feedback when the child is mistaken or disoriented. Another reason for satisfaction concerns more specifically the case of impulsive students who had difficulties in planning their trips and orienting themself when needed. This is the case for one of the students as the teacher explained. He's very impulsive. When you provide him with a model made of LEGO bricks, he touches every part very quickly. The sounds really slow him down. In addition, according to the teacher, sound feedback make it easier to maintain attention when compared to human verbal feedback, especially for one of the students with concentration problems. The teacher tried to give students instructions but reported that she's already tired because she has to concentrate so hard to get an answer that she gets it wrong almost every time. She had so much trouble concentrating. With the Tangible Box, she hears and immediately understands where to go. Finally, the teacher emphasized the interest of the Tangible Box for students who do not use the surrounding sounds in the environment to find their way around. For instance, she explains about one of them. He doesn't choose sounds enough when I'm inside with him. It would well be a specific goal when working on the Tangible Box with him to start focusing on auditory cues. Finally, when the teacher was asked for an overall assessment of the system, her feedback was very positive. She expressed that compared to the benefits of the tool, having to reposition the magnet when it's lost after lifting the pawn is a minor issue. We designed another application for learning the notions of time and the use of a clock. The clock reacts according to the manipulations of the student by delivering a sound feedback. An exercise models the student to test their knowledge. COMPUTER: It's 2:40. It's 2:25. JULIE: Finally, to enable sighted and blind children to share collaborative activities, we designed the Little Red Riding Hood game. Using visual, auditory and tactile information, participants must work together to develop the best strategy to avoid the big bad wolf. The devices and their applications that we have just presented are still in progress. According to the needs of the professionals, their desires and their constraints, we are currently working on several other projects. Some of them also involve visually impaired students in a participatory design process. If you have any ideas to propose, a willingness to discuss these projects or to collaborate, please do not hesitate to contact us. Thank you for your attention.