There have been a number of studies that have successfully decoded speech articulation and other motor signals from brain signals to restore a subject’s lost ability to speak. While effective, these decoders require neurosurgical access to the brain and are not applicable in most scenarios.
A recent study published in Nature Neuroscience describes a novel non-invasive decoding approach that uses functional magnetic resonance imaging (fMRI) recordings to reconstruct continuous natural language from cortical representations of semantic meaning. This non-invasive brain-computer interface can be used to identify meaning in perceptions, imaginations, and silent videos, and to generate intelligible word sequences. The study demonstrates the feasibility of a non-invasive brain-computer interface for semantic reconstructions.
On May 31st, young scientists from Tianqiao and Chrissy Chen Institute voluntarily organized the second session of “AI for Brain Science Journal Club”. Dr. Yuan Ze, a young scientist from the Peking University Sixth Hospital and also a member of the Chinese Academy of Sciences, expounded on the research. Please find below the essence from his interpretation as compiled by the NextQuestion editorial team.
How to decode language without invasively decoding the brain?
This study describes a novel decoder that uses non-invasive fMRI brain recordings to reconstruct arbitrary stimuli that a subject is hearing or imagining in continuous natural language.
To compare word sequences to subjects’ brain responses, the researchers trained an encoding model that predicted how subjects’ brains would respond to phrases in natural language. The experiment recorded the fMRI BOLD responses of subjects’ brains over a 16-hour period of listening to a narrative story and constructed an encoding model for each subject, which was then trained to predict the brain’s response based on the semantic features of the stimulus words.
It turned out that the decoded word sequences not only captured the meaning of the stimuli, but even predicted the exact words and phrases.
Where does the cerebral cortex register linguistic messages?
The study divided the brain data into a classic language network, a combined parietal-temporal-occipital network, and a prefrontal network. After decoding from each network in each hemisphere individually, the researchers found that predictions of the decoders from each network in each hemisphere were significantly more similar to actual stimuli than random predictions.
The researchers also calculated the time course of decoding performance for each network and found that most of the time points that were significantly decoded from the whole brain could be decoded from both the combined network and the prefrontal network. They similarly compared decoder predictions across networks and across hemispheres and found that the similarity between each pair of predictions was significantly higher than random ones. This suggests that cortical networks carry a great deal of redundant information, and that future brain-computer interfaces may be able to selectively record from the most accessible brain regions to obtain sound decoding performance.
Applications: where can we apply the non-invasive language decoder?
·Imagined speech decoding: decoding based on brain activity during imagination
·Cross-modal decoding: linguistic reconstruction of non-verbal tasks
·Attention decoding: semantic representations are modulated by attention, and semantic decoders should thus selectively reconstruct the stimuli under attention
·Privacy implications: an important ethical consideration of semantic decoding is that it may compromise mental privacy.
Lessons learnt: where do all the data noises come from?
The study also assessed whether decoding errors reflected random noise in brain recordings, model-setting errors, or both. The results found that model setting errors were the main source of decoding errors, in addition to random noise in training and testing data. Also, collecting more data may not significantly improve decoding performance.
Newsletter
WeChat
RSS