There are a vast number of children throughout the nation and world who suffer from the devastating effects of minimal or delayed language exposure. This can impact children in various home environments with reduced adult contact, particularly socially rich language interactions. This is especially challenging for over 90% of young deaf infants who initially have little (or no) exposure to a fully accessible natural signed language. Minimal or delayed early first-language exposure can have a lifelong negative impact on healthy learning, cognitive development, language acquisition, and reading success.

​

Professor Laura-Ann Petitto (Principal Investigator, grants from the W. M. Keck Foundation and NSF INSPIRE, IIS-1547178), the Petitto Brain and Language Lab for Neuroimaging (BL2) at Gallaudet University, additional scientists from three universities, and spanning four disciplines, aim to solve this problem with a revolutionary learning tool called the Robot AVatar thermal-Enhanced prototype, or RAVE.

​

What is RAVE?
Created and led by Petitto with funding from two (2) three-year research grants from the W.M. Keck Foundation and the National Science Foundation (NSF-INSPIRE), NSF RAVE is a robot-avatar learning tool that senses when babies’ emotional-attentional interest is most engaged and “ready to learn” (with Thermal Infrared imaging and eye-tracking). If so, NSF RAVE’s thermal system then triggers the robot and virtual human to start a variety of social interactions with the baby, and especially triggers the virtual human when to begin, and when to stop, language conversations and nursery rhymes with the baby in American Sign Language. (A speech option is planned for later.)

The work grows out of Petitto’s brain imaging discoveries using functional Near Infrared Spectroscopy (fNIRS) showing that all infants are sensitive to highly specific patterns at the heart of human language structure in early life: specifically, rhythmic temporal patterns with maximal contrast about the size of phonetic-syllabic units in natural language—all natural language, be it signed language or spoken language (see Petitto publications; Petitto et al., 2016). Like a lock and key, NSF RAVE is designed to provide babies with minimal language exposure with just those specific language patterns that they are most sensitive to in early life and just when their developing brains need to experience it most to ensure latter language and reading success.

​

Depending on the babies’ communicative interactions, NSF RAVE’s virtual human will produce rhythmic-temporal nursery rhymes specifically designed to help all babies discover the words, small meaningless parts of words (phonetic-syllabic units) and their patterning to promote healthy language growth.

​

One very special feature of NSF RAVE is that it can change the types of language interactions it has with the baby based on the baby’s own communications. Here, the virtual human is built to produce language communications that are meaningfully tied to what the baby is doing and communicating. For example, if a baby points to the virtual human, the virtual human may respond, “Are you pointing to me?” and may then begin to sign a happy nursery rhyme in ASL. Thus, beyond being socially interactive, NSF RAVE makes possible first-time human-machine interactions that approximate a meaningful socially contingent conversation (see Petitto Refereed Abstracts; Petitto, 2017).

These scientific achievements permit for the first time the advancement of a machine-human device that can (1) engage in socially contingent rudimentary conversations with young babies (ages 6-12 months), (2) provide natural language when babies are most “ready to learn” (even before they can even produce language to communicate this to us), and (3) provide just the right type of honed language stimulation (with specific rhythmic frequencies) that their brains require to “wedge” or “hold open” neural systems for language learning until fuller, richer language input becomes accessible.

​

The leading outcome goal of NSF RAVE is to produce an augmentative learning tool that makes available natural language patterns to young babies during the precise critical or sensitive periods in early brain and behavior development when they need it most.

To build NSF RAVE, basic science studies were first conducted with Dr. Petitto’s fNIRS brain imaging system that measures language and higher cognitive processing in the infant brain with Dr. Merla’s Thermal IR imaging system that measures baby’s levels of emotional-attentional interest. This work yielded evidence that (a) all babies’ are sensitive to specific phonetic-syllabic rhythmic patterning at the heart of human language structure (see Petitto references, e.g., Petitto et al., 2016) and that (b) increased interest to these specific aspects of human language patterns are coupled with peaked emotional arousal as determined by the Thermal IR imaging (Merla/Universita Gabriele D’Annunzio, Chieti-Pescara, Italy). Following this basic research, Dr. Merla and team built algorithms to signal a baby’s emotional state to the avatar, specifically, to stop and start its language productions (Avatar science: Traum team/USC), which was further interfaced with the robot’s solicitation actions to the baby (e.g., head nodding, eye blinking, etc.; Robotics science: Scassallati team/Yale).