We submitted our rebuttal to Brain and received a prompt reply from the Editor-In-Chief, Dr. Kullman himself, offering us an opportunity to revise – with the main criticism that our letter contained unfounded insinuations and allegations. We tried to interpret his message as best we could and respond accordingly. To most readers it was pretty clear what he wrote and the message he intended to convey. Nevertheless, in our revision, we stayed much closer to the words of editorial itself. We also tried to bolster our response with tighter arguments and a few salient references.
Essentially our message was:
- The editorial is striking in two ways: The tone is cynical and dismissive of fMRI as a method and the arguments against Brain Mapping, Discovery Science, and fMRI are outdated and weak.
- Dr. Kullmann does have valid points: Many fMRI studies are completely descriptive and certainly don’t really reveal underlying mechanisms. The impact of these studies are somewhat limited but certainly not of no value. Functional MRI is challenged by spatial, temporal, and sensitivity limits as well. We try to address these points in our response
- The limits that fMRI has are not fatal nor are they completely immovable. We have made breathtaking progress in the past 30 years. The limits inherent to fMRI are shared by all the brain assessment methods that we can think of. They are part of science. We make the best measurements we can using the most penetrating experimental designs and analysis methods that we can.
- All techniques attempt to understand the brain at different spatial and temporal scales. The brain is indeed organized across a wide range of spatial and temporal scales, and it’s likely we need to have an understanding of all of them to truly “understand” the brain.
- Discovery (i.e. non-hypothesis driven) science is growing in scope and insight as our databases grow in number and in complementary data.
- Lastly, what the heck? Why would an Editor-In-Chief of a journal choose to publicly rant about an entire field?! What does it gain? Let’s have a respectful discussion about how we can make the science better.
Defending Brain Mapping, fMRI, and Discovery Science: A Rebuttal to Editorial (Brain, Volume 143, Issue 4, April 2020, Page 1045) Revision 1
Vince Calhoun1 and Peter Bandettini2
1Tri-institutional Center for Translational Research in Neuroimaging and Data Science: Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, Georgia, USA.
2National Institute of Mental Health
In his editorial in Brain (Volume 143, Issue 4, April 2020, Page 1045), Dr. Dimitri Kullmann presents an emotive and uninformed set of criticisms about research where “…the route to clinical application or to improved understanding of disease mechanisms is very difficult to infer…” The editorial starts with a criticism about a small number of submissions, then it quickly pivots to broadly criticize discovery science, brain mapping, and the entire fMRI field: “Such manuscripts disproportionately report on functional MRI in groups of patients without a discernible hypothesis. Showing that activation patterns or functional connectivity motifs differ significantly is, on its own, insufficient justification to occupy space in Brain.”
The description of activity patterns and their differences between populations and even individuals is fundamental in characterizing and understanding how the healthy brain is organized, how it changes, and how it varies with disease – often leading directly to advances in clinical diagnosis and treatment (Matthews et al., 2006). The first such demonstrations were over 20 years ago with presurgical mapping of individual patients (Silva et al., 2018). Functional MRI is perfectly capable of obtaining results in individual subjects(Dubois and Adolphs, 2016). These maps are windows into the systems level organization of the brain that inform hypotheses that are generated within this specific spatial and temporal scale. The brain is clearly organized across a wide range of temporal and spatial scales – with no one scale emerging yet as the “most” informative(Lewis et al., 2015).
Dr. Kullmann implies in the above statement that the only hypotheses-driven studies are legitimate. This view dismisses out of hand the value of discovery science, which casts a wide and effective net in gathering and making sense of large amounts of data that are being collected and pooled(Poldrack et al., 2013). In this age of large neuroscience data repositories, discovery science research can be deeply informative (Miller et al., 2016). Both hypotheses-driven and discovery science have importance and significance.
Finally, in his opening salvo, he sets up his attack on fMRI: “Given that functional MRI is ∼30 years old and continues to divert many talented young researchers from careers in other fields of translational neuroscience it is worth reiterating two of the most troubling limitations of the method..” The author, who is also the editor-in-chief of Brain, sees fMRI research as problematic not only because a disproportionally large number of studies from it are reporting group differences and are not hypothesis-driven, but also because it has been diverting all the good young talent from more promising approaches. The petty lament about diverted young talent reveals a degree of cynicism of the natural and fair process by which the best science reveals itself and attracts good people. It implies that young scientists are somehow being misled to waste their brain power on fMRI rather than naturally gravitating towards the best science.
His “most troubling limitations of the method” are two hackneyed criticisms of fMRI that suggest for the past 30 years, he has not been following the fMRI literature published worldwide and in his own journal. Kullman’s two primary criticisms about fMRI are: “First, the fundamental relationship between the blood oxygenation level-dependent (BOLD) signal and neuronal computations remains a complete mystery.” and “Second, effect sizes are quasi-impossible to infer, leading to an anomaly in science where statistical significance remains the only metric reported.”
Both of these criticisms, to the degree that they are valid, can apply to all neuroscience methods to various degrees. The first criticism is partially true, as the relationship between ANY measure of neuronal firing or related physiology and neuronal computations IS still pretty much a complete mystery. While theoretical neuroscience is making rapid progress, we still do not know what a neuronal computation would look like no matter what measurement we observe. However, the relationship between neuronal activity and fMRI signal changes is far from a complete mystery, rather it has been extensively studied (Logothetis, 2003; Ma et al., 2016). While this relationship is imperfectly understood, literally hundreds of papers have established the relationship between localized hemodynamic changes and neuronal activity, measured using a multitude of other modalities. Nearly all cross-modal verification has provided strong confirmation that where and when neuronal activity changes, hemodynamic changes occur – in proportion to the degree of neuronal activity.
While inferences about brain connectivity from measures of temporal correlation have been supported by electrophysiologic measures, they have inherent assumptions about the degree to which synchronized neuronal activity is driving the fMRI-based connectivity as well as a degree of uncertainty about what is meant by “connectivity.” It has never been implied that functional connectivity gives an unbiased estimation of information transfer across regions. Furthermore, this issue has little to do with fMRI. Functional connectivity – as implied by temporal co-variance – is a commonly used metric in all neurophysiology studies. Functional MRI – based measures of “connectivity” have been demonstrated to clearly and consistently show correspondence with differences in behavior and traits of populations and individuals(Finn et al., 2015; Finn et al., 2018; Finn et al., 2020). These data, while not fully understood, and thus not yet perfectly interpretable, are beginning to inform systems-level network models with increasing levels of sophistication(Bertolero and Bassett, 2020).
Certainly, issues related to spatially and temporally confounding effects of larger vascular and other factors continue to be addressed. Sound experimental design, analysis, and interpretation can take these factors into account, allowing useful and meaningful information on functional organization, connectivity, and dynamics to be derived. Acquisition and processing strategies involving functional contrast manipulations and normalization approaches have effectively mitigated these vascular confounds (Menon, 2012). Most of these approaches have been known for over 20 years, yet until recently we didn’t have hardware that would enable us to use these methods broadly and robustly.
In contrast to what is claimed in the editorial, high field allows substantial reduction of large blood vessel and “draining vein” effects thanks to higher sensitivity at high field enabling scientists to use contrast manipulations more exclusively sensitive to small vessel and capillary effects(Polimeni and Uludag, 2018). Hundreds of ultra-high resolution fMRI studies are revealing cortical depth dependent activation that shows promise in informing feedback vs. feedforward connections(Huber et al., 2017; Huber et al., 2018; Finn et al., 2019; Huber et al., 2020).
Regarding the second criticism involving effect sizes. In stark contrast to the criticism in Dr. Kullmann’s editorial, effect sizes in fMRI are quite straight-forward to compute using standard approaches and are very often reported. In fact, you can estimate prediction accuracy relative to the noise ceiling. What is challenging is that there are many different fMRI-related variables that could be utilized. One might compare voxels, regions, patterns of activation, connectivity measures, or dynamics using an array of functional contrasts including blood flow, oxygenation, or blood volume. In fact, you can fit models under one set of conditions and test them under another set of conditions if you want to look at generalization. Thus, there are many different types of effects, depending on what is of interest. Rather than a weakness, this is a powerful strength of fMRI in that it is so rich and multi-dimensional.
The challenge of properly characterizing and modeling the meaningful signal as well as the noise is an ongoing area of research that is, shared by virtually every other brain assessment technique. In fMRI, the challenge is particularly acute because of the wealth and complexity of potential neuronal and physiological information provided. Clinical research in neuroscience generally suffers most from limitations of statistical analysis and predictive modeling because of the limited size of the available clinical data sets and the enormous individual variability in patients and healthy subjects. Again, this is a limitation for all measures, including fMRI. Singling out these issues as if they were specific to fMRI is indicative of a narrow and biased perspective. Dr. Kullmann is effectively stating that indeed fMRI is different from all the rest – a particularly efficient generator of a disproportionately high fraction of poor and useless studies. This perspective is cynical and wrong and ignores that ALL modalities have their limits and associated bad science, ALL modalities have their range of questions that they can appropriately ask.
Dr. Kullmann’s editorial oddly backpedals near the end. He does admit that: “This is not to dismiss the potential importance of the method when used with care and with a priori hypotheses, and in rare cases functional MRI has found a clinical role. One such application is in diagnosing consciousness in patients with cognitive-motor dissociation.” He then goes on to praise one researcher, Dr. Adrian Owen, who has pioneered fMRI use in clinical settings with “locked in” patients. The work he refers to in this article and the work of Dr. Owen are both outstanding, however, the perspective verbalized by Dr. Kullmann here is breathtaking as there are literally thousands of similar quality papers and hundreds of similarly accomplished and pioneering researchers in fMRI.
In summary, we argue that location and timing of brain activity on the scales that fMRI allows is useful for both understanding the brain and aiding clinical practice. One just has to take a more in-depth view of the literature and growth of fMRI over the past 30 years to appreciate the impact it has had. His implication that most fMRI users are misguided appears to dismiss the flawed yet powerful process of peer review in deciding in the long run what the most fruitful research methods are. His specific criticisms of fMRI are incorrect as they bring up legitimate challenges but completely fail to appreciate how the field has dealt – and continues to effectively deal with them. These two criticisms also fail to acknowledge that limits in interpreting any measurements are common to all other brain assessment techniques – imaging or otherwise. Lastly, his highlighting of a single researcher and study in this issue of Brain is myopic as he appears to imply that these are the extreme exceptions – inferred from his earlier statements – rather than simply examples of a high fraction of outstanding fMRI papers. He mentions the value of hypothesis driven studies without appreciating the growing literature of discovery science studies.
Functional MRI is a tool and not a catalyst for categorically mediocre science. How it is used is determined by the skill of the researcher. The literature is filled with examples of how fMRI has been used with inspiring skill and insight to penetrate fundamental questions of brain organization and reveal subtle, meaningful, and actionable differences between clinical populations and individuals. Functional MRI is advancing in sophistication at a very rapid rate, allowing us to better ask fundamental questions about the brain, more deeply interpret its data, as well as to advance its clinical utility. Any argument that an entire modality should be categorically dismissed in any manner is troubling and should in principle be strongly rebuffed.
Bertolero MA, Bassett DS. On the Nature of Explanations Offered by Network Science: A Perspective From and for Practicing Neuroscientists. Top Cogn Sci 2020.
Dubois J, Adolphs R. Building a Science of Individual Differences from fMRI. Trends Cogn Sci 2016; 20(6): 425-43.
Finn ES, Corlett PR, Chen G, Bandettini PA, Constable RT. Trait paranoia shapes inter-subject synchrony in brain activity during an ambiguous social narrative. Nat Commun 2018; 9(1): 2043.
Finn ES, Glerean E, Khojandi AY, Nielson D, Molfese PJ, Handwerker DA, et al. Idiosynchrony: From shared responses to individual differences during naturalistic neuroimaging. NeuroImage 2020; 215: 116828.
Finn ES, Huber L, Jangraw DC, Molfese PJ, Bandettini PA. Layer-dependent activity in human prefrontal cortex during working memory. Nat Neurosci 2019; 22(10): 1687-95.
Finn ES, Shen X, Scheinost D, Rosenberg MD, Huang J, Chun MM, et al. Functional connectome fingerprinting: identifying individuals using patterns of brain connectivity. Nat Neurosci 2015; 18(11): 1664-71.
Huber L, Finn ES, Chai Y, Goebel R, Stirnberg R, Stocker T, et al. Layer-dependent functional connectivity methods. Prog Neurobiol 2020: 101835.
Huber L, Handwerker DA, Jangraw DC, Chen G, Hall A, Stüber C, et al. High-Resolution CBV-fMRI Allows Mapping of Laminar Activity and Connectivity of Cortical Input and Output in Human M1. Neuron 2017; 96(6): 1253-63.e7.
Huber L, Ivanov D, Handwerker DA, Marrett S, Guidi M, Uludağ K, et al. Techniques for blood volume fMRI with VASO: From low-resolution mapping towards sub-millimeter layer-dependent applications. NeuroImage 2018; 164: 131-43.
Lewis CM, Bosman CA, Fries P. Recording of brain activity across spatial scales. Curr Opin Neurobiol 2015; 32: 68-77.
Logothetis NK. The underpinnings of the BOLD functional magnetic resonance imaging signal. J Neurosci 2003; 23(10): 3963-71.
Ma Y, Shaik MA, Kozberg MG, Kim SH, Portes JP, Timerman D, et al. Resting-state hemodynamics are spatiotemporally coupled to synchronized and symmetric neural activity in excitatory neurons. Proc Natl Acad Sci U S A 2016; 113(52): E8463-E71.
Matthews PM, Honey GD, Bullmore ET. Applications of fMRI in translational medicine and clinical practice. Nat Rev Neurosci 2006; 7(9): 732-44.
Menon RS. The great brain versus vein debate. NeuroImage 2012; 62(2): 970-4.
Miller KL, Alfaro-Almagro F, Bangerter NK, Thomas DL, Yacoub E, Xu J, et al. Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat Neurosci 2016; 19(11): 1523-36.
Poldrack RA, Barch DM, Mitchell JP, Wager TD, Wagner AD, Devlin JT, et al. Toward open sharing of task-based fMRI data: the OpenfMRI project. Front Neuroinform 2013; 7: 12.
Polimeni JR, Uludag K. Neuroimaging with ultra-high field MRI: Present and future. NeuroImage 2018; 168: 1-6.
Silva MA, See AP, Essayed WI, Golby AJ, Tie Y. Challenges and techniques for presurgical brain mapping with functional MRI. Neuroimage Clin 2018; 17: 794-803.