Interactive Storytelling for Children with Autism

Interactive Storytelling for Children with Autism
Adhish Bhobe
College of Computing
Georgia Institute of Technology
Atlanta, GA 30332 USA
[email protected]
Andrew Harbor
College of Computing
Georgia Institute of Technology
Atlanta, GA 30332 USA
[email protected]
Sanika Mokashi
College of Science
Georgia Institute of Technology
Atlanta, GA 30332 USA
[email protected]
Amol Shintre
College of Computing
Georgia Institute of Technology
Atlanta, GA 30332 USA
[email protected]
“Kinect the Dots” is an application designed to help therapists, teachers and parents of autistic children engage the
child in an interactive storytelling experience. It ties the
power of traditional storytelling with the full body interactions enabled by the Kinect. Its novel interaction model
aids innumerable uses in therapy and promotes social and
peer interaction.
ACM Classification: H5.2 [Information interfaces and
presentation]: User Interfaces. - Graphical user interfaces.
General terms: Design, Human Factors
What is Autism?
Autism is a complex neuro-developmental disorder. It affects 1 in a 110 children in the US today. It is manifested as
speech and language delay, repetitive stereotypical behavior, challenges with motor planning and difficulty with social interactions. Different people with autism can have
very different symptoms. Autism is thought of as a “spectrum” disorder, a group of disorders with similar features.
The various interventions and strategies used to deal with
autism include Applied Behavioral Analysis (ABA),
Speech Therapy, Occupational Therapy and Physical Therapy.
Storytelling plays a big part in therapy for autistic children.
It is used in conjunction with songs and the children are
encouraged to enact the parts of different characters from
stories. Storytelling helps them master language skills, improves their listening skills, increases their attention span
and develops their curiosity and creativity.
Keeping a child engaged while telling them a story isn’t
easy though. Children with autism especially tend to get
distracted easily. The teachers from the Lionheart School (a
specialized school for children on the autism spectrum) use
a lot of props, songs and other activities to keep the children engaged. A teacher from the school described her experience with the story of Jack and the beanstalk. To explain the concept of climbing up they made the children
pull a rope down and asked them to imagine Jack climbing
up on the beanstalk.
One of the teacher commented that the students can’t visually see the character Jack climb the beanstalk and thus
couldn’t understand the action or connect very well with
the character where they were teaching up and down
movements to the children using the story.
Digital Storytelling
A lot of different approaches use digital interactive storytelling to engage children in stories and let them play a
more active part than what traditional storytelling would
allow. IPad apps such as the Fantastic Flying Books of Mr.
Morris let children (and adults!) be a part of the story by
flying through the magical world, tumbling through a storm
and even learning piano. The TellTable system from Microsoft Research lets children author stories collaboratively
using Microsoft Surface. “People in Book” from Nokia
Research is a system that supports children and longdistance family members to act as characters in children's
storybooks while they read stories together over a distance.
Approaches like these lack in utility for children with autism on two levels:
(1) Trying to keep them engaged in such technology can be
a challenge sometimes
(2) It’s not as useful for children with autism, since they
don’t use their full body, often not speech, and they don’t
interact with their peers or teachers much.
The use of Kinect
On further research,we realized that the Kinect has the potential to create a big impact in therapy for children with
autism. A lot of children with autism seem to naturally take
to the Kinect.
The Lakeside center for autism uses the Kinect for multiple
goals. There are some specific goals of therapy that the
Kinect can address – as the Lakeside center for autism has
been doing
These are very encouraging results but they had to adapt
their goals to fit the technology which was limiting the extent to which the technology could aid the therapy.
However, we decided to go a step forward. Our solution is
a customized interactive storytelling application using the
Kinect that parents, teachers and therapists can use.
Use Case
At a very high level, our solution lets a teacher or a parent
tell a story in a more engaging way to a child with autism.
Unlike traditional storytelling, which is to a large extent
one-way communication, our application lets the child be
an active part of the process in the following ways:
--through controlling the characters via his/her body
--coloring different scenes from the story and
--even select components for particular scenes.
The entire experience of illustrating the story through kinect is both engaging for the child as well as intuitive.
Architecture and Technologies
--We are using C# for our code base
--Microsoft’s Kinect SDK
Architecture Diagram:
The entire code base is based on C#. Traditional Digital
storytelling consists of sounds and pictures. Through Kinect we extend this engagement further.
Through the inherent skeletal movement tracking we set
trackers to track the movements concerning selection required for selection of colors and scenes/images.
Trackers are set for tracking user movement. Movement
pertaining to foot movement is tracked for particular
scenes. On initiating the tracker, the characters on the scene
are corresponding set to move based on the user’s move-
Trackers are set for coloring, motion detection (climbing
action) and drag/drop action.
Evaluation and Results
We evaluated our system at the Lionheart School for children with Autism. We targeted the use of our system
against the most common behavioural characteristics of
autistic children. These included
1) Motor skills, Full body movement, Gross motor control
2) Speech
3) Communication,Responsiveness and Situational Awareness
We found the following results (seen in our videos) regarding these characteristics:
The child was able to effectively show responsiveness to
the scenes and the happenings in the story as well as communicate with the teacher and respond quickly to the instructions and requests made by the teacher. In one instance, the child was required to color a black and white
image of the Castle using the colors on the screen. The
child was able to pick the appropriate color matching the
Castle and color the Castle with minimal assistance from
the teacher
Certain scenes required the child to use leg and hand
movements. With minimal or no instructions from the
teacher/therapist, the child was able to intuitively use his
leg movements to play with the characters on the screen. In
one of the scenes the child was required to move his legs to
initiate a corresponding movement of the character ‘Jack’
up the beanstalk on the screen. The child was able to do
this with minimal assistance or prodding.
There were instances where the child was required to make
decisions by picking one of the scenes based on the theme
of the story. He was required to drag and drop the appropriate image. This required him not only to be decisive but
also use motor skills to bring the most appropriate image to
the forefront. We saw that he was able to do this no intervention from the therapists/ teachers.
Other miscellaneous findings included the improved engagement of the child with the story as well as the teacher
while the story proceeded.
We will continue evaluating the system with more participants (every child with autism is different). The measures
that we will use include: how often the children respond
appropriately (responsive communication), how often the
child asks questions or engages with the teacher on his or
her own (initiation of communication), instances of use of
speech, instances of full body movement. Specific
measures for speech and gross and fine motor control can
be used for specific children who have problems in these
Business case
The healthcare industry is growing from $100 billion today
to $150 billion in the next 2 years. Families spend anywhere from $6000 to $50000 per year on autism. Also the
cases of autism have increased significantly over the
Autism Statistics:
Current scenario lacks any specific and efficient solution to
tackle autism and its effects. As scientists and medical researchers keep looking for cures, it is critical that we embrace new technologies to help support the therapists today.
Storytelling is one of the constituents of therapy aids in
tackling autism but faces several obstacles.Kinect and storytelling together can be an effective combination to tackle
autism. It provides a customisable and low cost method in
combating autism. Pre- designed stories customised to various levels and kinds of autism can be useful and easy to
use by therapists and parents alike and intuitive for autistic
children. We plan to package the software primarily for
educational, research and therapy institutions that are looking for direct and simple ways to collect results and evaluate them.
Kinect the dots will do this without significantly increasing
the burden on parents.
Growth Strategy
-- Currently our ecosystem consists of software and content
developers along with our consumers which includes parents, therapists as well as teachers.
Software developers are involved with developing and
maintaining the system. Content developers are involved
with developing the story content and its constituent parts.
The consumers include therapists, parents and teachers
using the software in educational and therapy sessions.
-- In the future we plan to transform this into a platform and
a distribution channel which can be used by a network of
developers. The content can be developed by content developers who can then contribute to and add to the existing
library of content.This content can then be customized and
used by our consumers based on their needs.
Interventions and therapies used in treatment of
Children and Autism :
What is Autism? Do you know the Signs? :
Lionheart School:
Telltable from Microsoft Research:
“People in Books”: Sean Follmer, Rafael (Tico)
Ballagas, Hayes Raffle, Mirjana Spasojevic, and
Hiroshi Ishii. 2012. People in books: using a
FlashCam to become part of an interactive book
for connected reading. In Proceedings of the ACM
2012 conference on Computer Supported Cooperative Work (CSCW '12)
Importance of storytelling for young
Lakeside Center for Autism (Video):
Lakeside center: