‘Mental handwriting’: brain–computer interface turns neural signals into text

18 May 2021 Natalie Rhodes

Two microelectrode arrays in the “hand area” of the brain measure neural activity. A recurrent neural network (RNN) then converts the signals into probabilities for each character. These probabilities are either thresholded to provide real-time output or combined with a language model, which acts like an autocorrect. (Courtesy: Nature 10.1038/s41586-021-03506-2)

Locked-in syndrome, a neurological disorder that causes complete paralysis of nearly all voluntary muscles, leaves an estimated one in 100,000 people unable to communicate naturally. However, by using computational brain function imaging tools and specially designed software, researchers have enabled paralysed participants to communicate once again.

One method of computer-mediated communication uses flashing “mind spellers”, in which the participant looks at a screen with a keyboard of letters flashing at different speeds. Focusing on a given letter generates a neural response in the visual cortex that corresponds to the flashing frequency, and by measuring this response using electroencephalography, the target letter is determined.

Alternatively, “point-and-click” brain–computer interfaces (BCIs) use implanted intracortical electrodes that measure neural activity to visualize and control the movements of a cursor over an on-screen keyboard.

To date, these two methods have reached communication speeds of 60 and 40 characters per minute, respectively. Now, researchers at Stanford University have designed a novel BCI, based upon the imagined act of handwriting, that has achieved an unmatched 90 characters per minute. They describe this new approach in a paper published in Nature.

Decoding the neural signals of handwriting

The researchers tested their BCI in a single participant selected from the BrainGate study. The participant, known as T5, was paralysed from the neck down following a spinal cord injury, leaving him with no functional hand movement. During five experimental sessions, the team recorded T5’s neural activity using two microelectrode arrays, which were surgically implanted before the study into the area of his brain corresponding to hand control. T5 was instructed to attempt to write as if his hand were not paralysed, imagining holding a pen over a piece of paper and writing characters on top of one another.

The team employed principal component analysis (PCA), a method of reducing the dimensionality of large data sets, to visualize the neural activity recorded while “writing” single characters, by displaying the top three neural dimensions with the greatest variance in the signal. The signals proved to be repeatable over trials but had variations in timing, thought to be due to different handwriting speeds, which had to be corrected for.

To reveal the handwritten characters, the researchers used averages across trials of the same character to decode the pen-tip velocity, displaying distinctive and legible letters. To decode whole sentences, they trained a recurrent neural network (RNN) using thousands of characters written over multiple days. In turn, this network turned the neural activity into probabilities describing the likelihood of the participant writing a particular character at each time.The interpreted handwritten characters from the decoded neural signals. (Courtesy: F Willett)

The researchers investigated three methods to analyse the measured signals. First, they used thresholding of the probabilities to expose single characters, allowing for real-time on-screen feedback and resulting in an error rate of 5.4%. Second, they used a language model as an autocorrect feature, which decreased the error rate to only 0.89%. Following these experiments, they trained a new RNN on all the available data. This led to extremely high accuracy, with a low error rate of 0.17%, demonstrating the efficacy of the method and the high-performance ceiling that is attainable.

Why is this technique more effective?

This novel BCI method has more than doubled the speed of communication, while maintaining similar accuracy to “point-and-click” BCIs, which employ similar brain imaging methods. The researchers theorized that this rate increase is due to the variety in the spatial and temporal patterns of the neural activity when attempting to handwrite characters, compared with the straight-line pointing movements. They tested this hypothesis with a model and confirmed that the variety of the temporal patterns (different movement speeds) increases the separability of each movement, enabling a faster BCI with sufficient accuracy.READ MORE

BrainGate: untangling the brain–computer interface

BCIs based on movements such as “mental handwriting” also have notable advantages over visual-based interfaces, giving the participant the freedom to look around and communicate at their own pace. This system, and the results from participant T5, provide a proof-of-concept that a higher pace of communication is possible. While notable challenges remain, such as how this system works across different participants and lengths of time, this study demonstrates an important step-change in the development of BCIs for real-time communication.

from physicsworld.com 21/5/2021