Quality and denoising in real‐time functional magnetic resonance imaging neurofeedback: A methods review

Abstract Neurofeedback training using real‐time functional magnetic resonance imaging (rtfMRI‐NF) allows subjects voluntary control of localised and distributed brain activity. It has sparked increased interest as a promising non‐invasive treatment option in neuropsychiatric and neurocognitive disorders, although its efficacy and clinical significance are yet to be determined. In this work, we present the first extensive review of acquisition, processing and quality control methods available to improve the quality of the neurofeedback signal. Furthermore, we investigate the state of denoising and quality control practices in 128 recently published rtfMRI‐NF studies. We found: (a) that less than a third of the studies reported implementing standard real‐time fMRI denoising steps, (b) significant room for improvement with regards to methods reporting and (c) the need for methodological studies quantifying and comparing the contribution of denoising steps to the neurofeedback signal quality. Advances in rtfMRI‐NF research depend on reproducibility of methods and results. Notably, a systematic effort is needed to build up evidence that disentangles the various mechanisms influencing neurofeedback effects. To this end, we recommend that future rtfMRI‐NF studies: (a) report implementation of a set of standard real‐time fMRI denoising steps according to a proposed COBIDAS‐style checklist (https://osf.io/kjwhf/), (b) ensure the quality of the neurofeedback signal by calculating and reporting community‐informed quality metrics and applying offline control checks and (c) strive to adopt transparent principles in the form of methods and data sharing and support of open‐source rtfMRI‐NF software. Code and data for reproducibility, as well as an interactive environment to explore the study data, can be accessed at https://github.com/jsheunis/quality‐and‐denoising‐in‐rtfmri‐nf.

The full text of each article in the list of studies, including supplementary material, were searched for the following key terms: averag*, band, cutoff, difference, differential, drift, filter, frequency, heart, high, linear, low, motion, movement, nuisance, outlier, parameter, pass, physiol*, respir*, retroicor, scale, scrub, slice, smooth, spike, trend. We then coded whether these 128 studies reported the use of the following real-time preprocessing steps and other information: • Slice timing correction (stc) We classified studies as Did Not Report (DNR) if no mention of the particular method was made in the article or supplementary material, and if we could not confidently infer its use from studying the particular article's content. Some studies reported a processing step but did not provide further detail (e.g. "data were spatially smoothed...", with no smoothing kernel size provided). In such cases we coded the study and particular processing step as "Y".
Some entry types were simplified to allow for easier interpretation. Specifically, for Fig.7G where real-time fMRI neurofeedback software use was indicated, some studies (for example Koush et al., 2013 [2] ) were classified as OpenNFT even though they were conducted and published before the published release of OpenNFT (Koush et al., 2017 [3] ). Because these earlier studies used software developed by the same authors and containing essentially the same infrastructure and processing steps, they were classified for the purposes of our manuscript as using OpenNFT rather than "Custom Matlab + SPM".
An important point that was further examined during the review process is that there could be discrepancies between the default steps implemented in the particular software tool, the steps implemented based on the researchers' choices, and the steps that were eventually reported. Real-time fMRI software defaults could potentially present an accurate reflection of the unreported literature, if it is true that such default steps and parameters were indeed implemented and not reported. On the other hand, in the absence of accurate reporting of methods, we can also not be certain that default values were indeed used. Researchers might have had valid reasons for not implementing a specific step, but might still have failed to report this. Thus, whether DNR or the default value is used, an assumption is made in either direction and these have to be balanced.
To balance these unknowns for the data under consideration, we first distinguish between different software implementations in the set of 128 studies. Many (about 24%, see Figure 7G in the main manuscript) were not done with mature software packages but rather with custom pipelines and scripts. We deem these (often one-off) implementations to be more likely to have included the reported steps than to have unreported defaults. On the other hand, more wellknown software packages have been used for multiple studies, including Turbo-BrainVoyager (~56% of studies), AFNI real-time plugin (~11% of studies), OpenNFT (~5.5% of studies), BioImage Suite (~3% of studies), and FRIEND (~3% of studies). In these cases we contacted the software developers and asked for their input on standard preprocessing steps and default values. The general feedback without exception was that, while some default steps and parameters are made available to the users, it is up to the users (and they are indeed encouraged) to select their own pipeline steps and set their own parameters (in some cases with additional plugin functionality, e.g. slice-timing correction in Turbo-Brain Voyager). The developers cannot take responsibility for the accuracy of the information reported in publications. Even so, some possible default steps and parameters were provided by the developers.
With these distinctions, we recoded the dataset such that studied that used mature and widely used software packages reflected default options where particular steps were not reported, while for custom and one-off scripts/software we took the reported information to be accurate as reported. The recoded dataset is available as an additional Tab Delimited Text file as part of the supplementary material. This dataset was re-analysed to generate a new set of figures given below.