A team of Swedish neuroscientists recently spread waves of panic through the scientific community when they published findings in the journal PNAS of a software bug that may have nullified 15 years of neuroimaging research. The team, lead by Anders Eklund, a biomedical engineer, initially said the glitch might have affected the results of as many as 40,000 fMRI studies. After the paper presumably caused some researchers to cower in a corner, however, Eklund and colleagues revised their estimate to a few thousand studies and lamented the overblown reaction to their work.
fMRI, which stands for functional Magnetic Resonance Imaging, is a neuroimaging technique used to measure activity in and between brain regions. Actually, fMRI indirectly measures brain activity. Brain scanners are Hulk-esque magnets that detect subtle changes in blood oxygen levels. The idea is: Where blood flows, neurons are dancing.
The brain-imaging technology has only been around for about two decades and comes with a hefty pricetag for researchers looking to give their work a neuro upgrade. Studies on the brain basis of mindfulness, jazz and delayed gratification, among other hobbies and mental phenomena, have proven to be media catnip, but scientists have also bristled at trumped-up claims over neon blobs that the software uses to indicate neuronal activity. (See: the Salmon Study.) After all, hype on facebook doesn't necessarily translate to hype inside the academy. But, while Eklund's investigation reminds us that machines are made by humans and therefore not perfect, it doesn't render 15 years of imaging findings bunk. In sleep science, one of many disciplines that make use of fMRI, researchers profess continued, careful use of a tool that's ultimately a boon to neurobiological exploration — in other words, brain-imaging business as usual.
Through fMRI, scientists can compare what's happening inside the brain-at-rest to what's happening during a task (e.g. a facial recognition test) or a heightened state (e.g. being mother-f-ing exhausted). They can also see how brain activity differs between "controls" and people who share some unifying quality — a disease, personality trait or income bracket. But fMRI results aren't dispositive. Researchers can discover a burst of brain activity in a frontal-lobe section or temporparietal structure without knowing what that burst means. And it's not as though the only barriers to neuro-truth lie in understanding how to apply fMRI findings outside the lab. Because, in imaging studies, computer software does a lot of the interpretive lifting.
Brain scans are noisy. Noise refers, literally, to sounds from the machine and the person inside the machine, as well as to neural noise that interferes with the magnetic signal picked up by fMRI. To separate the signal from the noise, researchers rely on statistical software programs that generate 3-D models of gray-matter activity, comprised of units called voxels. One voxel equals roughly a million brain cells. The brightly colored blobs in brain scans are clusters of voxels. Programs detect patterns by assuming that activity coursing through one voxel implies activity in adjacent voxels. So, scientists don't directly analyze brain activity.
In their study, as Wired reported, Eklund and colleagues made use of brain-scan databases, collecting fMRI data on 499 healthy people. They used the three most popular software programs to analyze the data. They should have seen similar results across different software programs because the participants underwent scans during rest. As a result, any observed brain-activity bursts would be a product of noise, explained Bryce Mander, a researcher ath the UC Berkeley Sleep and Neuroimaging Lab. Instead, researchers found inconsistent data and false positives, indicating statistical detection of brain activity where it didn't exist.
It's no surprise the study made a splash, given the the use of fMRI across scientific disciplines, including sleep. We reached out to rest researchers, who differ both in their views on the study and the use of fMRI in sleep studies.
"fMRI is remarkably helpful in sleep research," said Eti Ben-Simon, a researcher in the same lab as Mander, who pointed out the issue at hand hand isn't unique to functional MRI, and could emerge in other imaging techniques. "Unlike EEG, it allows access to deep brain structures like the amygdala and the hippocampus in a non-invasive manner, regions that are critical to the study of emotion\memory as a function of sleep."
Pierre Maquet, a neurologist at the University of Liege in Belgium, didn't see the study as revelatory. "I'm afraid there is nothing new under the sun. It is known for ages that cluster level inference requires a conservative threshold."
Similarly, Michael Chee, a cognitive neuroscientist at the Duke-NUS Medical School Singapore, urged against applying the study too widely, or reading too much into it. "My exhortation," Chee said, "is [to] leave this particular sleeping dog to lie. There are many different analyses and questions that one can pose with fMRI and the kinds of problems alluded to in the Eklund study affect a number of studies but not all."
In sleep research, Chee has found that fMRI data deviation is a reality — and can be easily monitored. "Some effects, like the reduction of fronto-parietal activation in attention demanding tasks performed by sleep deprived young adults are highly reproducible. My group has replicated this basic finding in different test samples using variations of tasks requiring attention. Others like those on economic decision making are less robust but in my experience, that is true of both imaging and behavioral studies."
While Ben-Simon sees the paper as important in its potential for strengthening fMRI analysis, he thinks the identified problem might be more controlled in sleep studies, in which researchers "compare subjects' activity before and after sleep manipulation, so only regions that are consistently manipulated by sleep are discussed and random false activations tend to be minimized."
Mander, however, doesn't think the bug will affect many sleep studies because, he noted, sleep researchers rely on EEG and behavior more than fMRI. "However," he said, "all the functional neuroimaging studies of sleep and sleep deprivation will have to be closely scrutinized and replicated in future studies in order to determine which effects are real, and which are driven by overly liberal false positive correction."
Vinod Venkatraman, a researcher at Temple University, similarly described sleep studies, at least those on sleep deprivation, as somewhat insulated from consequences of the glitch. "Most fMRI studies on sleep deprivation use paradigms that are well validated in normal adults," said Venkatraman. "[They] also use within-subject controls for rested wakefulness. These control sessions provide additional tests of reproducibility for the main paradigm, reducing concerns of the nature raised in the recent PNAS paper."
So while future fMRI-based sleep studies will likely face increased scrutiny, there's likely no need to disregard the body of work that already exists. And, going forward, it's probably not a good idea to get neuroscientists unnecessarily riled up about their life's work.