Defending fMRI, Brain Mapping, and Discovery Science

This blog post was initiated by Dr. Vince Calhoun, director of the Tri-institutional Center for Translational Research in Neuroimaging and Data Science and of Georgia State University, Georgia Institute of Technology, and Emory University. Vince shot me an email asking if I saw this editorial in Brain by Dimitri Kullman (Brain, Volume 143, Issue 4, April 2020, Page 1045) https://academic.oup.com/brain/article/143/4/1045/5823483. He also made the suggestion that we write something together as a counterpoint. I heartily agreed. While there are many valid criticisms of fMRI and brain mapping in general, this particular editorial struck me as uninformed, myopic and cynical – thus requiring a response. I usually err on the side of giving the benefit of the doubt when reading or hearing of a different opinion, but my first visceral reaction to reading this article was simply: “Wow…” Vince and I quickly got to work and within a week submitted the below counterpoint to Brain.

---

Rebuttal to Editorial (Brain, Volume 143, Issue 4, April 2020, Page 1045)

Vince Calhoun¹ and Peter Bandettini²

¹Tri-institutional Center for Translational Research in Neuroimaging and Data Science: Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, Georgia, USA.

²National Institute of Mental Health

In his editorial in Brain (Volume 143, Issue 4, April 2020, Page 1045), Dr. Dimitri Kullmann takes several cheap shots at fMRI as a field and at most of the research findings that it produces. He argues that fMRI-based findings describing functional differences in activation or connectivity have no place in Brain and that fMRI functional contrast is fundamentally flawed. He rants that fMRI is drawing away talented young researchers whose time and energy would be better spent using other modalities. This salvo misses the mark however, as it is woefully uninformed and incorrect.

Dr. Kullmann seems to equate brain mapping itself with flawed and non-hypothesis driven research: “Showing that activation patterns or functional connectivity motifs differ significantly is, on its own, insufficient justification to occupy space in Brain.” There is no need to argue the utility of brain mapping, as the thousands of outstanding papers in the literature speak for themselves. One just has to attend the Organization for Human Brain Mapping or Society for Neuroscience meetings to appreciate the traction that has been made by fMRI in generating insight into brain organization of healthy and clinical subjects.

Dimitri Kullmann’s central premise is that somehow the science performed with fMRI, to a greater degree than other modalities, is ineffective in penetrating meaningful neuroscience questions or leading to clinical applications – something akin to doing astronomy with a microscope. He states two reasons. The first: “… the fundamental relationship between the blood oxygenation level-dependent (BOLD) signal and neuronal computations remains a complete mystery. As a direct consequence, it is extremely difficult to conclude that functional connectivity as measured by functional MRI genuinely measures information exchange between brain regions.” This is partially true, as the relationship between ANY measure of neuronal firing or related physiology and neuronal computations IS a complete mystery. We really do not know what a neuronal computation would even look like no matter what is measured. However, the relationship between neuronal activity and fMRI signal changes is far from a complete mystery, rather it has been extensively studied. While this relationship is imperfectly understood, literally hundreds of papers have established the relationship between localized hemodynamic changes and neuronal activity, measured using a multitude of other modalities. Nearly all cross-modal verification has provided strong confirmation that where and when neuronal activity changes, hemodynamic changes occur – in proportion to the degree of neuronal activity. Certainly, issues related to spatial and temporally confounding effects of larger vascular and other factors are still being addressed, yet, sound experimental design, analysis, and interpretations can take these limits into account, allowing useful information to be derived. Additionally, multiple functional contrast manipulations and normalization approaches have reduced these vascular confounds. In contrast to what is claimed in the editorial, high field in fact does allow mitigation of large blood vessels thanks to higher sensitivity that enables scientists to use contrast manipulations less sensitive to large vein effects. Hundreds of ultra-high resolution fMRI studies are revealing cortical depth dependent activation that shows promise in informing feedback vs. feedforward connections.

The second of his reasons: “…effect sizes are quasi-impossible to infer, leading to an anomaly in science where statistical significance remains the only metric reported.” Effect sizes in fMRI are in fact quite straight-forward to compute using standard approaches and are very often reported. What is challenging is that there are many different fMRI-related variables that could be utilized. One might compare voxels, regions, patterns of activation, connectivity measures, or dynamics using an array of functional contrasts including blood flow, oxygenation, or blood volume. Thus, there are many different types of effects, depending on what is of interest. Rather than a weakness, this is a powerful strength of fMRI in that it is so rich and multi-dimensional.

The challenge of properly characterizing and modeling the meaningful signal as well as the noise is an ongoing point of research that is, in fact, shared by virtually every other brain assessment technique. In fMRI, the challenge is particularly acute because of the wealth and complexity of potential neuronal and physiological information provided. Singling out these issues as if they were specific to fMRI is indicative of a very narrow and perhaps biased perspective. Dr. Kullmann is effectively stating that indeed fMRI is different from all the rest – a particularly efficient generator of a disproportionately high fraction of poor and useless studies. This perspective is cynical and wrong and ignores that ALL modalities have their limits and associated bad science, ALL modalities have their range of questions that they can appropriately ask.

Dr. Kullmann’s editorial oddly backpedals near the end. He does admit that: “This is not to dismiss the potential importance of the method when used with care and with a priori hypotheses, and in rare cases functional MRI has found a clinical role. One such application is in diagnosing consciousness in patients with cognitive-motor dissociation.” He then goes on to praise one researcher, Dr. Adrian Owen, who has pioneered fMRI use in clinical settings with “locked in” patients. The work he refers to in this article and the work of Dr. Owen are both outstanding, however, the perspective verbalized by Dr. Kullmann here is breathtaking as there are literally thousands of similar quality papers and hundreds of similarly accomplished and pioneering researchers in fMRI.

An additional point to emphasize in this age of big neuroscience data is that the editorial also expresses a cynicism against science that generates results that it cannot fully seal into a tight-fitting story. Describing a unique activation or connectivity pattern with a specific paradigm or demonstrating differences between populations or even individuals, while not always groundbreaking, usually advances our understanding of the brain, and can lead to clinical insights or even advances in clinical practice. Dr. Kullmann implies that the only legitimate use of fMRI in a study is in an hypothesis driven study. This view dismisses out of hand the value of discovery science, which casts a wide and effective net in gathering and making sense of large amounts of data. Both hypothesis driven and discovery science have importance and significance.

In summary, Dr. Kullmann argues that studies that compare activity or connectivity maps, as many fMRI studies do have no place in Brain. He claims that fMRI attracts too many talented researchers at the expense of better science performed with other tools. He describes two aspects of fMRI: the vascular origin of the signal and reporting on statistical measures, as being fatal flaws of the technique. However, he states that there are very rare exceptions – certain rare people are doing fMRI well.

We argue that location and timing of brain activity on the scales that fMRI allows is informative and useful information for both understanding the brain and clinical practice. One just has to take a more in depth view of the literature and growth fMRI over the past 30 years to appreciate the impact it has had. His cynicism that most fMRI users are misguided appears to dismiss the flawed yet powerful process of peer review. His specific criticisms of fMRI are incorrect as they bring up legitimate challenges but completely fail to appreciate how the field has dealt – and continues to effectively deal with them. These two criticisms also fail to acknowledge that limits in interpreting the measurements are inherent to all other brain assessment techniques – imaging or otherwise. Lastly, his highlighting of a single researcher and study in this issue of Brain is myopic as he appears to imply that these are the extreme exceptions – inferred from his earlier statements – rather than simply examples of a high fraction of outstanding fMRI papers. He mentions the value of hypothesis driven studies without appreciating the vast literature of hypothesis driven fMRI studies nor acknowledging the power of discovery science. Functional MRI is a tool and not a catalyst for categorically mediocre science. How it is used is determined by the skill of the researcher. The literature is filled with examples of how fMRI has been used with inspiring skill and insight to penetrate fundamental questions of brain organization and reveal subtle, meaningful, and actionable differences between clinical populations and individuals. Functional MRI is advancing in sophistication at a very rapid rate, allowing us to better ask fundamental questions about the brain, more deeply interpret its data, as well as to advance its clinical utility. Any argument that an entire modality should be categorically dismissed in any manner is troubling and should in principle be strongly rebuffed.