JJ Moritz, president of Sapien LLC, demonstrates the Cthulhu Shield, a tongue-based sensory substitution device.
All photos by John Eisele.
Listening to a favorite song…with your tongue. Receiving turn-by-turn navigation…with your tongue. Tracking stock market returns…with your tongue.
That these are strange concepts for most consumers isn’t lost on the team at Sapien LLC, a startup company based on technology developed at Colorado State University.
That’s why Sapien’s first commercial product, a tongue-based sensory substitution and augmentation kit, is aimed at engineers, hobbyists, tinkerers and creators who salivate over exotic ideas. The Sapien team hopes to stimulate the creative juices of technology enthusiasts willing to experiment and bring sensory substitution via the tongue into the mainstream.
The open-source product was launched at the end of January and is called the Cthulhu Shield, named after the mythical, tentacled creature of early 20th-century pulp fiction/horror and subsequent pop art. The company has launched a Kickstarter campaign to raise funds and allow continued product development.
The Cthulhu Shield is a circuit board connected to a removable, flexible mouthpiece. It’s outfitted with small electrodes that send electrical impulses to the tongue in response to signals from mobile applications, computers or sensors. It also senses the position of the tongue, allowing users to, for example, control video games or mobility and assistive devices with the tongue alone. Sapien engineers have even opened and sent email with the device.
“All schematics, designs and code are online, and users can write whatever software they want for the product,” explained Moritz, who completed his master’s degree in mechanical engineering at CSU in 2016. His thesis covered an early prototype of the device.
Sapien’s Cthulhu Shield, which can be purchased with an Arduino Uno and flexible electrode array.
New option for those with hearing loss
The inventors hope enthusiasm for the Cthulhu Shield creates a market for a related product they have been developing for several years and was the original inspiration for the company. Born out of research Moritz began as a student, the product is a retainer-like device, using the same technology as the Cthulhu Shield, that people who are deaf or hard of hearing would place in their mouths and use to “hear.”
Since Sapien was founded in 2014 with assistance from CSU Ventures, the university’s technology transfer office, the company has received investment from a key partner: DynaFlex, a national leader in orthodontic products. With that support, Sapien is developing “Smart Mouth Ware” – an electrode-filled device that fits into the roof of the mouth. This device would use the lingual nerve as a pathway to help the brain translate electrical stimuli into sounds – hearing with the tongue, one might say. The inventors envision it as a non-invasive tool to help people who have lost their hearing but may not be candidates for cochlear implants or hearing aids.
The inspiration for Sapien started when Moritz was a student working with John Williams, associate professor in the CSU Department of Mechanical Engineering. An expert in electric-propulsion systems used in space travel, Williams, who was suffering from tinnitus and high-frequency hearing loss, became interested in a subset of neuroscience called sensory substitution – the idea that the brain can be trained and retrained to process information from more than one source. Braille, the tactile alphabet for people who are vision impaired, is an example of sensory substitution, in which users learn to read text with their fingertips.
The CSU researchers became intrigued by an extensive body of work by the late University of Wisconsin-Madison scientist Paul Bach-y-Rita, who first developed the idea of placing spatially encoded images on the human tongue with an electrical device, in essence using the tongue to “see.” The University of Wisconsin team, building on Bach-y-Rita’s legacy, has successfully commercialized medical products that use tongue stimulation to help people with vision impairment perceive their surroundings, and those with vestibular disorders to find relief from severe vertigo.
Studying the tongue
As luck would have it, CSU has a tongue expert – Leslie Stone-Roy, assistant professor in the Department of Biomedical Sciences, who has spent most of her scientific career studying the sense of taste. “When John and JJ took me over to their lab for the first time, they had me listen to a song while the electrotactile stimulation was playing on my tongue. It was like a laser light show on the tongue. I thought, ‘This is pretty interesting.’”
Stone-Roy joined the research collaboration, and has led studies mapping the somatosensory pathway of the human tongue. The team has been refining the concept ever since, and they’ve been issued two patents to date.
“Electrotactile stimulation devices have been around for a while, but no one had completely characterized them according to the neurobiology of the tongue,” Stone-Roy said.
With the assistance of undergraduates enrolled in a biomedical sciences research course, Stone-Roy’s research is helping Sapien engineers define the most sensitive regions of the tongue for device optimization. One study looked at two-point discrimination, or how far apart certain sensations can be distinguished on the tongue’s surface. The researchers have found that tongues vary quite a bit; one person might be able to discriminate 99 different electrodes, while another might only be able to sense 14. Such information will be crucial for product development, and eventual training and customization for users.
The team has presented the research in journal articles and at several conferences, including the Front Range Neuroscience Group meeting in December 2018.
The Sapien team hopes the Cthulhu Shield is only the beginning of tongues going mainstream – whether that means inspiration for new medical applications or other ways to expand a user’s sensory experience through gaming or entertainment.
“Hopefully, we can create some hype that will normalize this technology and build a development community,” Moritz said.
The team at Sapien.
