there has been a growing interest in recent years in combining brain scanning technologies with natural language processing and machine learning techniques to study the neural basis of language processing and to develop brain-computer interfaces for communication. In this report, I will review some of the recent advances in this area and provide a bibliography of relevant research.
State of the Art:
One promising approach for combining brain scanning with AI language models is to use functional magnetic resonance imaging (fMRI) to measure brain activity while participants read or listen to language, and then use machine learning algorithms to analyze the data and build predictive models of brain activity based on the language input. These models can then be used to decode or generate language from brain activity data, or to gain insights into how the brain processes language.
For example, recent work by researchers at the University of California, San Francisco (UCSF) used fMRI and machine learning to decode brain activity related to spoken words, and then used a natural language processing model to generate predicted speech from the decoded brain activity. The researchers trained a neural network to predict the sound spectrogram of spoken words based on fMRI data, and found that the predicted speech matched the original speech input in terms of word identity and phonetic features.
Other studies have used EEG and machine learning to decode brain activity related to language processing, and to develop brain-computer interfaces for communication. One recent study by researchers at Carnegie Mellon University used EEG and machine learning to decode imagined speech from brain activity data, and demonstrated the potential for using such systems as a communication tool for people with speech impairments.
Bibliography:
Chang, E. F. (2019). Towards a neural decoder of speech. Current Opinion in Neurobiology, 55, 120-129.
Hermes, D., Miller, K. J., Noordmans, H. J., Vansteensel, M. J., & Ramsey, N. F. (2015). Automated electrocorticographic electrode localization on individually rendered brain surfaces. Journal of neuroscience methods, 242, 65-73.
Martin, S., Brunner, P., Holdgraf, C., Heinze, H. J., Crone, N. E., Rieger, J. W., & Knight, R. T. (2018). Decoding spectrotemporal features of overt and covert speech from the human cortex. Frontiers in neuroengineering, 11, 3.
Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J. M., ... & Slutzky, M. W. (2014). Direct classification of all American English phonemes using signals from functional speech motor cortex. Journal of neural engineering, 11(3), 035015.
Zhang, Q., Song, Y., Sun, H., & Chen, W. (2021). EEG-based classification of imagined speech: A review. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29, 271-281.
Conclusion:
The combination of brain scanning technologies and AI language models has the potential to revolutionize our understanding of language processing in the brain, and to develop new tools for communication and assistive technologies for people with speech impairments. While much work remains to be done, the recent advances in this area suggest that we are moving closer to achieving these goals.
