Seeking reproducibility: Assessing a multimodal study of the testing effect
Abstract
Low‐cost devices have widened the use of multimodal data in experiments providing a more complete picture of behavioural effects. However, the accurate collection and combination of multimodal and behavioural data in a manner that enables reproducibility is challenging and often requires researchers to refine their approaches. This paper presents a direct replication of a multimodal wordlist experiment. Specifically, we use a low‐cost Emotiv EPOC® to acquire electrophysiological measures of brain activity to investigate whether retrieval during learning facilitates the encoding of subsequent learning as measured by performance on recall tests and reflected by changes in alpha wave oscillations. Behavioural results of the wordlist experiment were replicated, but physiological results were not. We conclude the paper by highlighting the challenges faced in terms of replicating the previous work and in attempting to facilitate the reproducibility of our own experiment.
Lay Description
What is already known about this subject matter?
- Multimodal data provides a more complete picture of behavioural effects.
- Low‐cost devices are enabling the greater use of multimodal data.
- Multimodal data requires advanced methods of collecting and combining data.
- Reproducibility is a key tenet of science, yet many studies are not easily reproducible.
What this paper adds?
- An exploration of the complexity multimodal data adds to reproducibility.
- A replication study of the behavioural and physiological effects of testing versus restudying.
- A critical analysis of the challenges faced in reproducing a multimodal study.
Implications for practitioners:
- Reproducible studies need to be effectively planned from the start.
- Reproducibility is likely to become a requirement for publication.
- Multimodal data makes reproducibility more challenging.
