At first glance, what Timothy Gentner and colleagues from the University of California San Diego are doing seems a bit strange: The researchers recorded the neural activity in what is known as HCV of zebra finches – a brain region that is active when singing. With the help of the nerve signals, they then trained a neural network that was supposed to synthesize the singing associated with brain activity. The researchers describe the technical details of their experiment in a study, which was recently published in the journal Current Biology.
An intermediate goal
The zebra finches only serve the researchers as an intermediate target. You want to develop a voice prosthesis that can be controlled simply by reading out brain activities. “Going from a songbird model to a system that will eventually be used in humans is a pretty big evolutionary leap in the minds of many people,” says Vikash Gilja, one of the study’s authors. In fact, however, the brains of birds are surprisingly complex, and there are numerous parallels to humans.
“Although birdsong differs from human language in important ways, the two vowel systems share many similarities, including features of sequential organization and strategies for acquiring them,” the researchers write. There are “analogies in neural organization and function, genetic foundations and physical mechanisms of sound production”.
Lots of neural patterns
Technically speaking, however, translating patterns of neural activity into sound patterns has not been an easy task. “There are simply too many neural patterns and too many sound patterns to be able to find a single solution that can map one signal directly onto the other,” says Gentner. To solve the problem, the team used a biomechanical model that describes the pressure fluctuations and changes in muscle tone in the finch’s vocal organ – the membranes in the vocal organ behave like non-linear oscillators. The researchers then trained their algorithms to map the neural activity directly onto these representations.
The team’s next step is to show that their system can reconstruct the birdsong from the neural activity in real time. This is even more difficult because songbirds listen to their own singing and constantly adapt their singing to what they really want to sing. A successful voice prosthesis will ultimately have to work on a timescale that is similarly fast and complex enough to take into account the birds’ entire feedback loop, including the correction of errors, the researchers write.
(wst)
