Til hovedinnhold

Video transcription Boguslaw Bob Marek, Phia Damsma

Blending digital and tactile learning to develop skills for tactile reading

PHIA: Blending digital and tactile learning to develop skills for tactile reading. Sonokids, Australia, and Hungry Fingers, Poland. Hello, my name is Phia Damsma from Sonokids Australia.

BOB: And I am Boguslaw Bob Marek, Hungry Fingers, Poland.

PHIA: We have developed a blended teaching method that aims to build a bridge between traditional tactile graphics and grids on the one hand and digital grids and mental mapping skills on the other, moving from the concrete to the imagined. Digital learning is done with Ballyland. Ballyland is an imaginary world full of songs, stories and sound where the Ballylanders live. These are six characters who are ball-shaped, each have a signature sound and a specific way of moving around. This imaginary world of Ballyland forms the backdrop of the Ballyland suite of software and apps, that support young students who are blind or visually impaired to develop fundamental digital and technology STEM skills. Ballyland offers a fun, interactive and inclusive learning environment that is also suited for remote at-home learning. The apps provide spoken instructions and guidance, audio alerts and effects and storytelling to support children who are blind or visually impaired in finding their way around the touch screen. The slide shows photos of young children, who are learning and playing with the Ballyland apps and from their smiles, it is clear that they are having fun. The Sonokids's Ballyland accessible gamified e-learning pathway currently includes apps for iOS, for iPads and iPhones, for Android, one for Alexa smart speakers, software for Windows computers and supporting tactile learning tools. The apps are available in multiple languages. The first app we would like to discuss in this presentation is called Ballyland Sound Memory. The slide shows two screenshots from this app. The app doesn't use voiceover or talkback, but it's fully accessible for children who are blind because it is self-voicing. It is a matching game, this app. You need to pair the same sounds or cards. Players only need to use basic finger gestures such as finger drag, flick and double tap to open the cards. The game uses different size grids. You can also choose from a variety of sound sets such as musical instruments, animal sounds or the sounds of the Ballylanders. To give you an impression of the game and the required skills, here is a short video of a young boy who is blind playing and learning with Ballyland sound memory. He uses the flick gesture to move to different cards and double-tap to open the card. He can hear whether he has found a correct match or not through the fun audio feedback. (bell ringing) - [AI] Row two, co- - [Ballylander] Oh no! (application clicks) - [AI] Row two, column three. Row one, column three. (bell ringing) Row one, column two. (bell ringing) - [Ballylander] Oh, yes!

PHIA: Moving from the concrete to the imagined. To help children with the gameplay, Sonokids has developed tactile learning tools with the digital apps. This slide shows, on the left, an image of Wheelie, one of the Ballylanders. He is a little blue car on wheels and likes to race around. Next, a line image of Wheelie, which can be printed on swell paper to make a tactile image. For schools or organisations who have access to 3D Print Technology, Sonokids has developed a large size model of Wheelie with turning wheels, which a child can explore to get to know what Wheelie looks like. From concrete to imagine, navigating a grid. To understand how to move around a grid in the sound memory game, the child requires conceptual understanding of a grid with rows and columns, an understanding of directional concept, mapping and mental mapping skills, and spatial orientation skills.

BOB: For a sighted person, it is almost impossible to imagine how someone with congenital blindness can make sense of what is happening on an iPad screen, which is just a smooth piece of glass. How can a blind child imagine that the screen can be divided into sections be it four, six, eight, or 10, arranged into some rows and columns, which together make a grid? And as if building a mental picture of a grid wasn't a big enough challenge, how can a blind child, on tapping on one of the cards, go back to the card, which the child remembers as a good match? Or finally, how does the grid change when a matching pair is removed? To cope with all this, the child must first feel confident with the concept of a grid and with the audio instructions identifying individual sections of the grid. To explain this concept, one must use a language which a blind child can understand best, the language of touch. One way to do it is to start with this simple crossword puzzle, for example, a horizontal tactile row of squares filled with letters, making the word flower, and a vertical tactile column made of three squares, making the word dog. When dropped together, they make a simple crossword puzzle grid. Grids can have different numbers of sections, for example, the tactile grid made of six squares, arranged into two rows and three columns. Good understanding of grids is important in a wide range of contexts, word games, word diagrams, all kinds of graphs with coordinates and chess, to mention just a few. To help children understand what is happening on the screen and how the grid changes when a pair of matching cards is removed, a special manipulative was designed. Configurations of the cards on iPad screen can be checked on this manipulative, which is a plate with removable blocks, each representing one card. Configuration of cards can be additionally confirmed on a tactile graphics grid, a useful intermediate two-dimensional stage between the concrete, 3D manipulative, and an abstract imagined or virtual grid. Tapping twice on a card, in this game on card in row one column one, activates the voice of the barking dog. On the manipulative, the activated card can be identified by turning over the corresponding block and revealing a tactile symbol, the circle. On the tactile grid, the activated card is shown as a textured rectangle. In this slide, an incorrect guess is shown. The card in row one column one, and the one in row two column three, are not a matching pair. This is confirmed by different tactile symbols on the manipulative, the circle and the triangle, and on the tactile grid, where the two cards have different textures. In this slide, the child has moved to row two, column two, which hides a matching card for the one selected earlier, in row one, column one. Voice output, the word dog confirms the right choice, and so does the manipulative, where the corresponding blocks have identical tactile circles. On the tactile grid, the two matching cards have the same textures. As in any memory game, the matching cards are removed. On the iPad screen, they disappear, leaving two blank spaces in the grid. The child can check this by removing, from the manipulative, the two blocks with tactile circles, which leave indentations symbolising blank spaces. Also on the tactile grid, removed cards are replaced by blank spaces. The child can now plan the next move, and can, at any point, check the location of the remaining cards, either on the manipulative or on the tactile grid. As a result, rather than leaving it to chance and an uncontrolled chain of flicks, the child will learn to make conscious choices as to which way to move within the virtual grid and which cards to open. With time, the child will be able to make these choices without the aid of the concrete manipulative or a tactile diagram. Given the complexity of the process of building a mental image of a grid and various configurations of the elements which it contains, it is always good to make sure that the child really understands the spatial relations, which appear during various stages of the game. One way to do it is to make a simple two by three grid, cut out from a sheet of cardboard, start in your game and ask the child to place a small object here, round magnets, in compartments described by voice instructions. For example, in row one column one, as in picture A, or in row two, column three, as in picture B. The same grid can be used for practising and checking understanding directional concepts. The child can check, tactilely, how the location of active cards changes with each flick left or right and confirm that flick down and flick up can be used to switch between the rows. Understanding of spatial relations represented graphically can be practised and reinforced in a number of situations in real life or in play, as in this slide showing a boy whose task is to place the cup on a placement mat in a location shown in wonderful, tactile drawings and then to move the cup to a new location. To do that, the child will need good understanding of concepts describing location and direction, such as right, left, top, bottom, up and down.

PHIA: The Ballyland code apps also use grids. This is a series of three apps. Ballyland code one, Say Hello, Ballyland code two, Give Rotor, and Ballyland code three, Pick Up. They increase in difficulty and should be played in order. The code apps provide an audio-based introduction to coding for young students who are blind or visually impaired with no coding experience and limited voiceover skills. The slide shows three app icons of the code apps for iOS. If we do a coding workshop, we like to start children by moving around a life-sized grid. This could be at the school playground or with bright tape or tech tiles on the floor. Children can then experience moving around a grid one step at a time, which is relevant for coding, where you can only give one coding command at a time. They learn to move forward, turn right and avoid obstacles. The code apps uses three by three grid, and moving in this grid, Willie needs to reach a target, avoid obstacles and find the shortest route. This requires computational thinking. You need to break the process down in small steps, decomposition, and give Wheelie the correct commands in the correct order, sequencing, to move one step at a time. Children also need to understand, which way is Wheelie facing? Where will he move if he moves forward?

BOB: In decoding games, a three by three grid is used, within which Wheelie will be moving around, visiting friends or helping to solve unexpected problems. The grid can best be illustrated by noughts and crosses, which children may be familiar with. While directing Wheelie and planning his different routes, a wooden plate divided into nine compartments comes useful. Small objects can be placed as obstacles in selected compartments, allowing children to find ways around them, as in these two photos of children in India, getting ready for a coding challenge. The tactile plate makes it possible to illustrate with concrete objects, the situation on the virtual grid, spaces within which Wheelie can move and those which are blocked. In this slide, Wheelie has only one route available to reach his friend. While here, the child can plan several routes for Wheelie to reach Ballicopter waiting for a new rotor. Before engaging in coding, the child can first practise and try to choose the shortest route with the help of the wooden board and concrete objects. Such exercises involving concrete objects are important as they build children's confidence and give them a chance to practise various options, which they will have to decide on when they engage in proper coding and learn with Wheelie how to plan available routes.

PHIA: Having prepared and learn through tactile means, students can then use the Ballyland code apps on the touch screen. As mentioned before, this requires the use of orientation skills, decomposition skills and sequencing skills. Together, these are coding skills. On the left, a photo of a young boy studying the tactile grid with Wheelie, and on the right, an image showing a boy using both a tactile grid and an iPad with Ballyland code one. This is an image, on this next slide, of Ballyland code one, Say Hello, on an iPad screen, and also represented by 3D printed mini movable Ballylanders on the 3D printed tactile grid. In coding challenge one, Wheelie is in the grid of three rows by three columns. He starts in row one, column one, facing right. Tinkleball, another Ballylander, is in row one, column three. There are two obstacles that Wheelie needs to avoid, a pond with ducks in row two, column one, and a rubbish bin in row two, column two. The student needs to find the shortest route for Wheelie to move to Tinkleball, and when he is in the same cell as her, to say hello. The coding commands that can be used are, MoveForward, TurnRight, and SayHello. This slide shows the coding panel of Ballyland code one. The app is fully accessible and a double-tap opens this coding panel in which you enter the code. You can flick right and left through the lines of code and flick down and up to select the correct coding commands. In this case, in line one, what to do first, you need to enter, move forward, in line two, move forward, and in line three, when he is in the same cell as Tinkleball, say hello. We will now share a short video which I will describe before it plays so that you can hopefully enjoy the audio effects from the app in the video. After having acquired all the necessary skills and understanding, this student uses 3D-printed tactile models to plan Wheelie's route on the tactile grid, and then, step by step, enters the corresponding correct commands in the coding panel on the iPad, then you can hear how he double taps the screen to run his code. The output is both visual, with Wheelie moving to Tinkleball, but also as an audio story, to make it fully accessible so that the student who is blind can hear that he coded it correctly and have fun with his code. - [Instructor] So what do we need to do, to get to Tinkleball? - [Child] Move forward. - [Instructor] Okay. (child speaking indistinctly) - [Instructor] Okay, to- - [AI] Move forward. Line two, what to do next? - [Child] Move forward. - [AI] Empty line. - [Child] Oh. - [Instructor] Okay, squeeze them in (giggles). (child giggles) - [AI] Move forward, line three. - [Child] Say hello. - [AI] What to do next? Move forward, turn right, say hello. Your code is running. - [AI] Wheelie is in row one, column one facing right. (Wheelie's engine roaring) He speeds up and moves forward. (Wheelie's engine roaring) Wheelie is in row one, column two, facing right. (Wheelie's engine roaring) He speeds up and moves forward. (Wheelie's engine roaring) Wheelie is in row one, column three, facing right. Tinkleball is also here. (Tinkleball hoots) He says hello. - [Wheelie] Hello, Tinkleball. - [Tinkleball] Hi there, Wheelie. You brought Wheelie and Tinkleball together. (Ballylander hoots) (bell ringing) - [Instructor] Well done, baby. - [Ballylanders] Hurray!

PHIA: The blended teaching method is designed to build a bridge between traditional tactile graphics and grids on the one hand, and digital skills in a digital environment and mental mapping skills on the other. We hope to have demonstrated that tactile reading skills support the learning of digital skills, and that accessing digital learning also supports learning of skills that are important for tactile reading and exploration. The learning outcomes of the blended teaching method are, mental mapping and memory skills, spatial awareness on a touch screen understanding and navigating around a grid with rows and columns, which is important for orientation and mobility, for technology, mathematics, and also for chess as we just saw, computational thinking and coding skills, and for the tactile skills, improved spacial orientation and ability to move around tactile floor plans, maps, and large diagrams. And all of this together, this learning, empowers children. We kindly thank you for your attention.