Inclusive multisensory science and immunology books for blind, low‐vision and diverse‐needs audiences

This paper presents interdisciplinary research exploring the development of inclusive multisensory science books, communicating immunology data for blind, low‐vision and diverse‐needs audiences. The research adopted an inductive theory‐building approach, practice‐based art methods and music and design methods, leveraging the lived experience of a legally blind artist. The research also involved designers and scientists in a cocreation process, producing books that incorporate tactile artworks, Braille‐inspired protein models, image sonification and interaction. Two multisensory book titles, “The Heroes Within You: A Multisensory Exploration of Infection and Immunity” and “My Goodness: A Multisensory Exploration of Nutrition and Immunity”, were developed for the Monash Sensory Science 2023 Exhibition Day. The books offer an innovative way to make science and art more accessible and engaging, addressing the limitations of traditional museum methods. Feedback from audiences has been positive, emphasizing the fascination, sensory engagement and ease of understanding. This paper highlights the potential for an interdisciplinary and inclusive approach to science and art, demonstrating the value of multisensory books as tools for science communication. The findings highlight the positive reception of this novel approach and suggest its potential for broader applications, promoting inclusivity and accessibility.


MULTIMODAL AND MULTISENSORY EXHIBITION
Museums and galleries have traditionally attempted to engage blind, low-vision and diverse-needs (BLVDN) audiences through the provision of multimodal descriptions of painted canvases, experiential walk-throughs and touch tours, 1 the provision of audio descriptions, 3D-printed textured images and raised relief or high-quality tactile paintings. 2lthough novel, these approaches struggle to convey the true content of a painted work, its color, brushstrokes and composition.In addition, multimodal descriptions are not appealing to sighted audiences and therefore fail to promote inclusion within an exhibition.STEM (science, technology, engineering, and mathematics) exhibitions for BLVDN audiences exploring astronomy have also utilized multimodal approaches.In the United Kingdom, the Wonder Dome 3 educational initiative has offered a variety of tactile exhibitions tailored for BLVDN audiences.These exhibitions of "Accessible Astronomy" included three-dimensional renditions of NASA's (National Aeronautics and Space Administration) Hubble Telescope images 3 ; Braille star maps depicting astronomical constellations 4 ; tactile reproductions of 2D images, including the Whirlpool Galaxy (M51 in the Messier catalog) and sonifications of radio astronomy objects, such as pulsars and planets within the solar system. 3everaging the lived experience of a blind artist, Monash Sensory Science has embraced an inclusive multisensory approach to communicating immunology-based art to BLVDN audiences.This approach acknowledges that a sense is any faculty that accesses stimuli from outside or inside the body and that perception is not just a process of awareness but fundamentally one of recognition, understanding and interpretation of a given stimuli. 5ather than adopting the traditional multimodal methods of 3D-printed tactile images and audio descriptors, we chose a multisensory perceptual approach to support understanding.For example, we advocate model making using found objects such as food, paper, seeds, grains, material, yarn and toys, to evoke the perceptual sense memories, providing deeper understandings of the information being conveyed.Perception also relies on a fusion of the senses and therefore we also advocate music to aid multisensory understanding, providing temporal descriptions of immunology data, biological interactions or event streams.

INTERDISCIPLINARY RESEARCH
During the Australian summer of 2019-2020, the authors were collaborating 6 through digital data visualization and sonification of RNA proteins of HIV and severe acute respiratory syndrome coronavirus 1 (SARS-CoV-1) unaware of the approaching coronavirus disease 2019 (COVID-19) pandemic and the profound impacts it would exert on exhibition plans going forward.Melbourne's six lockdowns continued for a total of 262 days from March 2020 to October 2021.Lockdowns shut researchers out of their workspaces and transformed home garages, spare rooms (and a loft) into makeshift laboratories and studios.
During the lockdowns, Dr Erica Tandori conceived a novel portable interactive exhibition format she called "an exhibition in a book."Conceived for an audience of one, it was considered that the format would satisfy social distancing, cleaning and other COVID-19 stringent requirements.The idea also built upon prior multisensory exhibition work including tactile posters, Braille-inspired proteins and protein sonification (Figure 1).It was envisioned that a book could contain these elements and be exhibited continuously, throughout the day at schools, libraries or BLVDN community hubs and resource centers.
As a result, an interdisciplinary research project was designed with the aim of exploring multisensory bookmaking to communicate immunology to BLVDN audiences.The research adopted inductive theory building, practice-based art and design methods.Monash scientists were also engaged through cocreation activities which generated narrative vignettes for the books.Creative practice-based artmaking and musical composition were also brought to bear.To drive the exhibition design, a number of research questions were developed, including the following: 1 How can an interdisciplinary approach enhance the creation of First, a prototype washable book was made comprising 15 double-sided A3 foam-core leaves.The leaves had waterproof A3-laminated pouch covers and afforded a range of elements including tactile artworks,

large-scale text,
Braille-inspired proteins and optical interaction markers to be placed therein.Various textures and materials were employed in crafting multisensory artifacts, imparting intricacy and fascination for BLVDN readers.The artworks were intentionally designed with a slight elevation to facilitate the exploration of textural details, while maintaining a flatness that allows the pages to close.
Interaction design was employed to develop a software solution to cue the audio playback with the corresponding pages within the books.A number of ideas were explored before settling on an optical scanning approach using reacTIVision, 7 an open-source software development kit available for Mac, PC and Raspberry Pi. reacTIVision uses optical markers called fiducials, each providing its own identity number which is identified by the software in real time (Figure 2).
A standalone app was then created in Max 8 and installed onto Mac mini-computers, configured to run without the internet, keyboard or mouse.The computers connected to just a single USB (universal serial bus) camera which was positioned above the open book via a bespoke acrylic stand designed to also house the computer.AKG professional headphones (Harman International, Stamford, CT) completed the equipment build.
After testing initial prototypes, it was decided that the books should be raised to an incline to better facilitate hand and arm posture during tactile exploration.In response, a second bookstand iteration was designed.A range of materials and finishes were also trialed in its development including 4.5-mm wood, poly-finish, stained and oiled plywood and additionally 6-mm clear and black acrylic (Figure 3).The Braille-inspired protein models were adapted from lessons  learned from earlier exhibitions.A combination of found objects (e.g.black peppercorns for carbon) and clay-modeled balls with painterly textures were assembled into 2D molecular structures.To further extend the multisensory experience of the books, audio sonifications of the images were then added.
Sonification is the use of nonspeech sounds to communicate data and has had a wide application in scientific data display. 9In our multisensory book application, we wished to explore the potential for sonification to sound the image details of the tactile Braille-inspired models.An oscillator bank was created to sound a vertical array of image pixels via fast Fourier transform.
Another scrolling algorithm was then created to read each vertical band continually across the video frame image from left to right, sounding out the structure of the image in real time.Image-processing control was then added to isolate artworks from the page, stretch ads or transpose them vertically or horizontally across the video sampling window.A controller console was added for real-time multiparameter control, which greatly sped up the process of finding suitable sounding mappings.We then added a waveform editor and sound effects including reverb and spectral filtering, to complete the sonification instrument.The instrument was then used to record a set of image sonifications which were included in the book soundtracks, interspersed among the narration and musical illustrations.

EXHIBITION AND FEEDBACK
Two multisensory book titles have been created so far: (i) "The Heroes Within You: A Multisensory Exploration of Infection and Immunity" and (ii) "My Goodness: A Multisensory Exploration of Nutrition and Immunity".The Heroes book included 10 double-page spreads exploring a diverse range of immunity themes including vaccines, immunity to viruses, anticancer immunotherapy, autoimmunity, major histocompatibility complex and T-cell receptor, B cells and antibodies, and innate immunity."My Goodness" included 12 two-page spreads covering vitamin B metabolites, the microbiome, celiac disease, intestinal immunology, obesity and amino acids, immune memory cells, dendritic cells, vitamin A and gut health, short-chain fatty acids, allergies and high-fiber diets, fatty liver disease, diet and inflammatory bowel disease and colon and bowel cancers.Each book title was completed in a set of 10 duplicate volumes together with cover sleeves, all painstakingly produced by Dr Erica Tandori and her biomedical and disability interns.For the readers of Braille, supplemental Braille booklets were printed to accompany each book.
The books were presented for the 2023 Monash Sensory Science Annual Exhibition (Figure 4) plus exhibitions for blind primary and high school students at the Statewide Vision Resources Center and the Australian Hearing Hub at Macquarie University, Sydney.The books have also been presented for the Sensory Science Cambridge Exhibition in March 2024, as part of the Cambridge Festival.
Both general and BLVDN audience responses have been complimentary and positive.Following up with focus groups and short interviews, several themes emerged.
First, many in the audience found the multisensory book exhibition to be fascinating, sensory and engaging."My most favourite exhibit was the multisensory books!" said one "very satisfied with the exhibition", said another.
"There are a lot of things to see and very interactive . . .a sense thing", commented one mother who attended with her young son.
"The science book room was fascinating to me . . . the books really helped me to engage with the scientific content", commented another participant.
Some participants loved the simplicity of the page-turning interaction."I wanted to keep turning the pages and wanted to know how the interaction was made", said one participant.
One elderly woman found the page turning and voice-over not to her liking, preferring to read the book without headphones, which her husband wore instead while they shared the tactile experience together.
There were many opinions among the audience concerning 3D-printed accurate models versus the multisensory book's flatpack tactile multisensory models.
A younger participant commented: "I have been to a number of science exhibitions, and I really like accurate 3D printed models, especially ones that are kinetic and move".
A scientist added, "I find accurate 3D printed models only somewhat interesting.They're good for first year students as they can learn more than stick models."When asked about the multisensory books, the scientist continued ". . .these are very good not just for visually impaired and general public but also for scientists like me.They help me visualise and I feel them which always adds more dimension." A BLVDN participant commented, "The (multisensory book) tactile models engage so much better, and the use of colour is really good as there are a lot of low vision people who love colour".The same participant found the science understandable too."It is also quite easy to understand the tactile models." The image sonification was commented on during the interviews."Definitely like the sonification!I have come here with my young son, and he can definitely experience much more.The voice recording is very good too.Easy to access the data." A BLVDN participant commented, "Again, to people who can understand a screen reader with one axis e.g. the horizontal one being time and the other being sonification datagenerally yes you can explain it".This participant also talked at length about their experience using data sonification via phone apps and so was clearly experienced.
One audience participant had reactions against the audio."I am neurodivergent (Autism) and I really struggle with auditory processing.I find music, audio very distracting."The participant remained in the exhibition room.However, they wore their own noise-canceling earbuds under their headphones.

DISCUSSION
The research presented in this paper explores the innovative concept of multisensory books as a means of engaging and educating BLVDN audiences, specifically in the context of immunology-based art and science communication.We have described an interdisciplinary approach combining practice-based artmaking and music composition alongside interaction design, industrial design and audio design, to develop an interdisciplinary artifact in the form of a multisensory science book.
The multisensory books highlight the limitations of traditional approaches employed by museums and galleries to engage BLVDN audiences, such as the use of multimodal descriptions and 3D printed models.These methods fall short in conveying meaning as they do not take advantage of touch memory or other sensory gestalts.Other examples, such as Braille star maps or gray PLA 3D print relief versions of oil and canvas works, do not provide color for people with low vision while also not appealing to sighted audiences.The books also advocate a multisensory approach that leverages perception through associations prompted by everyday objects, materials and even music to evoke sense memories and provide a deeper understanding of the content.
The research project involved collaboration between artists, designers and scientists, highlighting the importance of interdisciplinary cooperation.The integration of artistic and musical elements alongside scientific data not only enriches the multisensory experience of the books but also generates original methods of image sonification utilizing a range of audio and video processing techniques with real-time control.
The themes that emerged from interviews and focus groups were fascination, sensory engagement and ease of understanding.Some audience participants also appreciated the simplicity of the interaction, while others had varying opinions on the use of audio.A noteworthy theme for discussion is the comparison between the multisensory books tactile art and 3D-printed accurate models.While some participants expressed a preference for the latter, especially if they were kinetic, others found the multisensory books more engaging and informative.
The books' tactile nature and use of color were particularly praised.This can be directly attributed to Dr Erica Tandori who, despite being legally blind and having peripheral vision, implying fewer cones and therefore reduced color vision, has continually advocated for the inclusion of vibrant color.The inclusion of audio elements and image sonification received positive feedback from many participants, allowing for a more comprehensive experience.However, sensitive audience participants may find audio distracting.

CONCLUSION
The findings suggest that multisensory books have the potential to be a valuable tool for science communication and education, benefiting both BLVDN and general audiences.They offer a unique way to make complex scientific concepts accessible, engaging and inclusive.
In summary, the research presented in this paper showcases the development and positive reception of multisensory books as a novel and inclusive approach to science and art communication.The interdisciplinary collaboration, innovative use of everyday materials and music all contribute to a more immersive and educational experience for inclusive BLVDN audiences.
Further exploration and refinement of this approach could have broader implications for inclusive education and accessibility in museums and science communication.
portable multisensory science books for BLVDN audiences and inclusion? 2 How can the lived experience of blindness guide and inform the development of multisensory representations of immunology data for BLVDN audiences?3 What key insights may be gained from a cocreation process between artists, designers and scientists, in the development suitable book chapter themes.Next, the scientists were engaged to write short vignettes describing these themes in the context of their own research.The themes were further explored through artmaking and the composition of several hours of immunology-inspired electronic music.The scientists' written vignettes were then voice recorded and produced with supporting audio soundtracks of music.

Figure 1 .
Figure 1.Examples of multisensory work: (from left to right) co-designed tactile posters, Braille-inspired amino acid models and sonification of RNA proteins.All images by the authors.

Figure 3 .
Figure 3. Second design iteration example; 6-mm clear acrylic bookstand with camera.Images by the authors.

Figure 2 .
Figure 2. Left: reacTIVision software and Max 8 standalone app.Right: a multisensory book with optical fiducial (top left page) and Braille-inspired protein models and MAIT cell tactile model (right).Images by the authors.

Figure 4 .
Figure 4. Multisensory Science show at the Monash Sensory Science 2023 Exhibition Day.Exhibition images by Gerard Hynes (photographer for Monash Sensory Science).