A Survey on Visualizations for Musical Data

Digital methods are increasingly applied to store, structure and analyse vast amounts of musical data. In this context, visualization plays a crucial role, as it assists musicologists and non‐expert users in data analysis and in gaining new knowledge. This survey focuses on this unique link between musicology and visualization. We classify 129 related works according to the visualized data types, and we analyse which visualization techniques were applied for certain research inquiries and to fulfill specific tasks. Next to scientific references, we take commercial music software and public websites into account, that contribute novel concepts of visualizing musicological data. We encounter different aspects of uncertainty as major problems when dealing with musicological data and show how occurring inconsistencies are processed and visually communicated. Drawing from our overview in the field, we identify open challenges for research on the interface of musicology and visualization to be tackled in the future.


Introduction
Probably anyone has personal experiences with music, a medium that has the unique feature to unite people.This sociocultural aspect of music is one of the main driving forces for music research [Lam12].Merriam-Webster defines musicology as 'the study of music as a branch of knowledge or field of research as distinct from composition or performance' [MW18].This encloses all information related to music, for example, sound patterns, scores, biographical information about artists, music genres and their dependencies.Similar to other subdomains of the humanities, in recent years, digital methods became increasingly important in musicology to store, structure and analyse vast amounts of digitally available musicological data [Urb17].To achieve these tasks, visualization is a key element in this context, as it enables easier access to the data and has the capability to highlight relationships between structural elements of music [Lam12].As the data to be observed are manifold, visualization designs offered to analyse data occur in many different forms.Our survey focuses on this unique interface between musicology and visualization research.State-of-the-art reports in related fields have already been conducted.Most are domain specific, such as the survey by Chan et al. [CQ07] reviewing visualizations of structural features of music.It focuses on scores without discussing other entities related to musicology.Casey et al. [CVG*08] focus on the retrieval of content-based information and give an overview of existing projects and future challenges in this context.While they primarily discuss analysis, classification and retrieval methods, visualization solely plays a secondary role.However, the listed-related applications will also be covered by our survey.Related surveys situated in our field concern visual text analysis methods in digital humanities [JFCS17], focusing on textual data rather than musicological data.Cultural heritage collections [WFS*18], that explicitly exclude musical collections and persons from the analysis, were found as well.Here, the clear need for a survey considering visualizations for musicology arises.
In our survey, we shed light on applied visualization techniques depending on the underlying data, that is, musical entities-like musical instruments or musicians-and relationships among those entities, and typical tasks (exploring, presenting, comparing, etc.) musicologists perform with the given visualizations.On the one hand, it is complementary to related survey papers as it focuses on a different subdomain of the digital humanities.According to McNabb et al.'s three-dimensional hierarchical classification of visualization surveys [ML17], we will offer a new category in real world and applications, thus, extending the current spectrum.On the other hand, our State-of-the-Art-Report (STAR) aims to introduce well-established visualization scenarios for typical tasks in musicology to the visualization community.Furthermore, in contrast to related publications from our community, our STAR will include visualization techniques published in musicology and digital humanities-related realms.In addition, we will include online available visual analysis tools, primarily developed for users interested in music and commercial software dealing with music.
Hence, our STAR provides a useful resource for future developments in visualization on the basis of musicological data.First, we provide an overview of already existing techniques alongside supported typical user tasks.Second, we discuss arising challenges due to the nature of humanities data.These challenges are (1) vastness in the size of the data emerging through the long history of musicology, (2) inhomogeneity through fragmented data and an imbalanced state of research in parts of musicology, (3) imprecision because of undocumented but necessary information on historical entities and (4) incompleteness being a typical issue of cultural heritage data.
Third, we list future challenges and summarize unsolved problems as well as topics that have not yet been sufficiently addressed.

Scope
Means of visualization to communicate musical information can be found in diverse realms.Our main priority when surveying related works was reflecting this diversity.Therefore, we decided to take innovative, but for the visualization community rather atypical visualization design approaches into consideration.The result is a collection of 129 works that distribute over different areas, as shown in Table 1.
Consequently, our survey is situated on the intersection of visualization and musicology.While visualization has rather seldomly been applied in the realm of Musicology (4) like the Violin Society of America, the most common source of included works fall under the visualization of musicological data, published in diverse Visualization (22) realms.In addition, the area of Human Computer Interaction (18) provides a suitable platform for presenting visual interfaces to analyse digital audio and computer music.This includes publications at global conferences like the Conference on Human Factors in Computing Systems (CHI) and conferences specifically directed towards applications in musicology like The International Conference on New Interfaces for Musical Expression (NIME).In Digital Humanities, an interdisciplinary community that brings together people with humanities and computer science backgrounds, musicology is still considered a niche.Nevertheless, eight related works have been collected and included in this survey.Visualization plays an important role in Music Information Retrieval (MIR) applications (18) to support the analysis of retrieved datasets.One of the major journals that yield a large number of related works are The Proceedings of the International Society for Music Information Retrieval (ISMIR), which mainly focuses on the similarity of music.Multimedia experts and practitioners apply visualizations for a diversity of analysis tasks on scores, performances and emotions of This first category lists a series of visualizations inviting website visitors to browse and to interact with musicological data.The second category specifically includes game software providing atypical, keyboardless means of interaction and visual design approaches that support the playful acquisition of musical knowledge-both aspects are valuable to be highlighted as they offer future prospects for visualization research and can be found using search interfaces and typical keywords like 'music' or 'visualization'.

Considered research papers
To limit the large body of related works on the intersection of visualization and musicology, a reference needed to fulfill two requirements in order to be considered for our survey.
First, the visualization needs to support a domain-specific task or helps to investigate a research question concerning data related to musicology.This includes visualizations for individual musical works, entire musical collections, musicians and instruments.We likewise considered visualizations on the basis of metadata as well as musical contents.In contrast, we excluded works using visual input to generate music [LL05, PIE*11, MKSM16, Cho18, CW18] from our survey.In interdisciplinary settings, the word visualization is frequently used to refer to traditional charts.Although meaningful information can be extracted from these representations, we excluded such works from our survey, for example, Cano et al.'s work [CKGB02] offering a scatterplot to analyse the similarity among audios of musical works or Plewa [PK15] positioning the numbers of songs on a regular grid.
Our second criterion is based on the information visualization definitions given by Card [CMS99] and the University of Illinois' Digital Libraries Initiative (UIUC DLI) Glossary [oI98].Here, we only considered papers that provided computer-supported, nontraditional visual representations of abstract data.As musicologists also gain valuable insights using non-interactive visualizations, interactivity was not a necessity.Thus, we also included sophisticated static visual representations of musicological data, such as Heller's heat map [Hel17] plots illustrating wood thickness of instruments.At last, a variety of works including visual representations of music can be found in proceedings of other conferences like The International Conference on Technologies for Music Notation and Representation (TENOR).However, those works are often focused primarily on the notations part, therefore we did not consider them in this survey.

Methodology
Most of the surveyed visualizations have been designed to communicate features of musicological data to certain user groups.On the one hand, tools are developed for domain experts with a musicology background [MFH04, Hel17, KSKE17, KJKF19], on the other hand, easy-to-understand visualizations are designed for the broad public [Har07a,Dan14,Spo18]-including a category of papers addressing hearing-impaired persons [YLL*07, FF09].In all cases, the application domain specifies the tasks to be supported, and, consequently, the complexity of the visualization design.Therefore, we structure the related works on the basis of Munzner's nested model for the visualization design [Mun09], but focus on the level of data-rather than task-abstraction that builds the bridge between domain situation and visual encoding.The classification of the works, depending on the type of data for which the visualization has been designed for, includes four main data categories: musical works, musical collections, musicians and musical instruments.
We decided to split our first category on musical works into the subcategories musical scores and musical sound to better structure the large variety of proposed methods for this data type.Whereas score is the composed blueprint to reproduce a musical piece, sound is the actual interpretation, for example, a recorded song or performed operatic aria.These human interpretations are marked by variance from the blueprint like not monotone and unvarying tempo or improvising and replacing noted features on the fly.As a special case, examples of note sheets exists that contain so-called paranotations, added notes on the blueprint to describe planned changes to the notation in a non-standardized form.Such deviations lead to special visualization challenges and approaches.While the first category offers detailed views on music, diverse distant overviews for musical collections-including large numbers of musical pieceshave been designed.Depending on the actual user task, the musical piece can be compared in regard to specific features or be transformed to allow for easier browsing.The last two categories are closely related to musicology that does not only focus on the result of the musical process (notes and performances), but also on musicians (composers, performers, instrument makers and so forth)related visualization techniques are surveyed in the musicians (Section 4.3)-or instruments that have been used (see Section 4.4).Due to the difference of features related to these categories, the means of visualization offered to observe and to interact with the data vary.For better guidance through the collection, we grouped the related works of each data category dependent on the general use of the visualization.The classification according to data types is discussed in Section 4.
Next to structuring the related works according to data characteristics, we skimmed through the papers of the collection and analysed what abstract [BM13] and domain-specific visualization tasks are supported.Considering the large amounts of works in some categories, we divided them into related subtasks.We included information about typical tasks for each data type within the data classification.The survey is complemented with an overview of how and if visualizations cater for communicating occurring uncertainties (see Section 5).This includes issues arising due to the vastness of musicological datasets, the imprecision of data features, the incompleteness of data and the inhomogeneity throughout and beyond the collected data.Finally, we marked open challenges on the intersection of visualization and musicology reported in Section 6.

Domain-related terminology
As our survey puts the spotlight on musicology, a humanities research domain having its own terminology, we briefly explain a few terms that will repeatedly occur in the following sections.• The pitch is quantified by a frequency, describing the physical phenomenon of oscillation of sound waves.It gains a musical dimension through the relation to other frequencies in a complete range of sounds.Thus, it is the feature that defines the height position in a musical score notation.Examination of it is interesting, especially for comparison between today's and music of the past, as the concert pitches of instruments have changed throughout the centuries.• A musical note is a symbolization of a musical sound encoding pitch and duration of it.In order to define the position of a musical note in a score, we refer to scientific pitch notation which is a method for the explicit description of a pitch using its note name and octave number.• The musical score is the notated version of a musical piece.It consists of multiple notes, varying according to its system and media, like written sheet music or digital MIDI.It encodes the musical features for storage, exchange or replay.One example of a classical sheet notation is visible in Figure 2 (top), where the notes are complemented by additional information such as tempo or repetition for its performance.• The key of a musical piece is the root (tonic note) in which it is composed.This note and its corresponding chords form the tonality of the piece.• The dynamic describes the variability of loudness within a musical piece.• Timbre is the perceived sound quality of a sound arising from the mixture of different frequencies by overlapping of the fundamental tone and partial tones.It describes the tone colour of individual instruments or voices even if they are equally tuned.• Music is described by its features that can be divided into low level features, physical attributes like pitch, tempo or loudness, perceived high level features, for example, timbre and structural features such as dynamics and repeating motifs.

Classification
Musicology as an application area for visualization is a domain that requires user-centred design approaches [AMKP04] to lower barriers and to enable intuitive interpretation and to foster engagement with the visualized data for musicologists and non-expert users.
During our extended research, we encountered a large variety of data types for which visualizations have been designed to support various user tasks in a musical or musicological context.These data types cluster the surveyed visualization techniques the best.Table 2 gives an overview of the classification having musical works, musical collections, musicians and instruments as main categories.The references for each category are further subdivided depending on domain-specific tasks.Figure 1 provides an overview of the visualization techniques used for communicating features of the main data categories visually.Different visualization approaches are grouped together under abstract names.For example, charts include a multitude of rather basic visualizations like scatterplots, bar charts, pie charts or boxplots.The group of piano roll views is defined by the digital representation of a piano roll and plain score sheets.Both holding all information a traditional and analog score sheet offers, and were defined from us as special forms of timelines.Also, the three-dimensional rendering group is an aggregation of different principles.Next to the expected renderings like volume and sur- face rendering, also renderings of avatars are included.The miscellaneous group consists mainly of glyph visualizations that are too abstract to be categorized into other groups or would lead to groups of three elements or less.While charts are the only visualization used for all four data types, they are rather seldomly used in general.
Especially noticeable is the high amount of map visualizations used for musical collections.As collections dealing with a multitude of different songs that are to be put into an easily accessible way, this is not surprising.Often, the maps are used with a self-organizing approach (SOM) or clustering for positioning, resulting in the song's proximity representing similarity.Musical works, which reserve the major space of our survey, are visualized with very different and special visualization means, including 16 miscellaneous visualizations that appear too rarely to be grouped explicitly.A further anomaly is the piano roll view only being used for musical pieces.As this being a digital adaptation of score sheets, the high amount of musical pieces is less striking and the missing other types are caused by this visualization approach being very specialized for notes in a temporal context.Instruments are mainly visualized by three-dimensional renderings.Mostly, available data for this type are given as computed tomography (CT) data, pre-defining it for these renderings.Metadata visualizations of instruments (charts and heatmaps) are the exception.The musicians are represented in different visualization strategies.A lot of works deal with network visualization, calling for typical graphs, while temporal data like living or In the following, we give a detailed overview of each data category and the typical user tasks when dealing with such data.We report on the necessity of designing visualizations for musicology, and we likewise outline the relevance of dealing with such data and applications for the visualization community.

Visualization of musical works
The majority of visualizations have been designed for musical works that represent different products of music.Those appear either in written form as Musical Scores that are composed manuals for musicians to reproduce the music, or in the form of produced Musical Sound, wearing the fingerprint of the performer and showing (minor) discrepancies to the blueprint of musical scores [DR06].As the data of those products are different, we divided the references on visualizations for musical works into those two categories.Further, this section focuses on works that visualize aspects of a single musical work, thus, providing a detailed view of a single musical work.Visualizations made for entire collections of musical works, which focus on other aspects and support other user tasks, are discussed in Section 4.2.
The data formats for digital musical works are manifold.While musicXML [G*01] is a standard for sharing musical scores, musical sounds are stored in various audio file formats [DR06] like MP3 or WAV.Both, musical sounds and musical scores can also be stored as MIDI files.MIDI is a standardized format developed for exchanging information of events and music data like pitch, velocity (volume), vibrato, panning to the right or left of stereo and tempo [MID19], and encodes them into control signals for electronically instruments.MIDI files can be generated automatically by an electrical instrument playing (where all information are encoded and saved) [Swi97] or composed digitally with the help of (commercial) composing software [OTG19].Thus, MIDI files differ from other files for storing musical information as they do not contain records of sounds that are simply saved digitally and later be played again.Instead, they save information as instructions that are redirected to an electrical instrument and then used to 'reinterpret' [Swi97], recreating an original sound.Hence, MIDI can store both kinds of musical works, depending on how the file is generated (recorded input and transformed back into a midi notation format, or composed input).In some cases, the authors do not explicitly report on the data format being used, but we can assume that the MIDI file format was the most often used one.

Musical scores
Throughout time, scores have been the main way of transferring, documenting and teaching musical pieces [Ben19].Figure 2 (top) shows a part of the classical score representation of Beethoven's 'Für Elise'.The sheet includes notes for two parts that are played simultaneously.Each part offers different information in a temporal context [HB82,Ben19] like key (violin on top and bass on bottom), beat (three-eighth time), dynamics (starting with pp-pianissimofor 'very soft'), tempo (Poco moto meaning 'little motion') and a list of notes and breaks.With the information contained in the music sheet and the knowledge of how to read it and how to play an instrument, musicians are capable of re-interpreting a musical piece.Although such a traditional score of a musical piece itself is already a type of visualization [SW97], a variety of alternative score representations, discussed in this section, exist.
Relevance for VIS.Musical scores are given in a data format that is unique to be used as a basis for visualization.Musicology has found its own principles to present scores effectively in a visual form.It is used by a large community including musicologists and non-expert users.Visualization research can further enhance those representations by applying generic visual design and interaction principles.Most of the visualizations are used to teach scores and only a few case studies exist, which examines how visualization can be used for teaching [YV15, RRJH18, FIB*19].Thus, visualization researchers can learn strategies from the presented visualizations on how to design in a way that the data are easily understood by the observer.
Relevance for musicology.The prior advantage of visualizing scores is the ability to turn a complex traditional score notation into an easily understandable visual form, thus, enabling less-skilled users quick access to the data.In addition, musicologists profit from visualizing scores because a reinterpretation of a musical piece always comprises a unique fingerprint of the corresponding musician.Further engagement of visualization scholars could help to contrast different interpretations of the same score or even adaptations of such fingerprints left on the score notations through paranotations (handwritten notes and instructions added to a note sheet).
We group the surveyed works according to three main user tasks.First, visualizations are tailored to give an overview of the score.Second, such representations can be enhanced exposing structural score features.Third, scores are visualized to analyse instrument performances.
Task: Score overview.Giving an overview of the whole score is useful for both, experts trying to analyse a musical piece and less-skilled users that aim to comprehend music scores.Miller et al. [MHK*18] offer a pipeline for designing and visualizing music notation overviews to assist in performing musicology tasks utilizing information visualization principles.To differentiate from the classical score notation, typical scores are visualized in their temporal context and are augmented through a combination of colour, shape and placement or even complex glyphs.Some works enhance traditional score notations with other visuals, for example, a coloured similarity matrix showing recurring passages and similarities between multiple tracks [WBK09], contextual information and annotations shown on demand in a fish-eye view [WHF03] or further visualizations like box plots and heat maps [CNP16].Typically, traditional score notations are transformed into a so-called 'piano roll notation' [CKS10, MT11, CNP16].Each note's pitch, temporal placement and length is mapped to y-axis position, x-axis position and length, respectively [CNP16].Additionally, information may be encoded using colour.Figure 3 (left) shows a typical composition software [OTG19] using a plain implementation of the piano roll, whereas Figure 3 (right) demonstrates the work of Ciuha et al. [CKS10] that visualizes the musicological aspects of consonance and dissonance in harmonies by colour and saturation.Instead of visualizing single notes, other approaches concentrate on showing the distribution of pitches throughout different time steps [MC07,Hal13].While Mardirossian and Chew [MC07] depict only the currently played keys and tones without further context information, Hall et al. [Hal13] provide an illustration of the whole musical piece.Figure 4 shows the visualization for Johann Sebastian Bach's Prelude in C Major.The y-coordinate's zero position is a C Major pitch and higher pitches are placed on top in a colour map ranging from red to yellow and lower pitches on the bottom using colours ranging from blue to yellow.Longer played notes have a greater width and simultaneous tones overlap.Score overviews can further assist users in learning or teaching musical compositions [SW97,Gud11].Using score rather than performance data facilitates improving the understanding of music and not directly improving the performances.Therefore, different score views-for a casual user normally not achievable through score notation-are offered.This includes three-dimensional coloured spheres indicating low-level features of the scores [SW97].Biophillia [Gud11] uses physical processes in nature as a visual metaphor to teach musictheoretical concepts like rhythm or dynamics with its artistic illustrations.
Task: Structure analysis.Besides the visualization of the pure score, researchers provide deeper insight into the structure of musical pieces and highlight the underlying characteristic patterns, repetitions, dynamics, keys and harmonies.Therefore, a harmonic analysis yields the harmonic structure of musical compositions, and relationships between key regions can be extracted.While a couple of works show static representations of a song [Sap01, Wat01, Wat02, MFH04, CQM09, MPZZ15], others offer animations progressing throughout the song [BKH07,GC16].Malandrino et al. [MPZZ15] highlight structural features of musical compositions by mapping similar tonalities to similar colours.Chan et al. [CQM09] communicate the structure of classical music works.First, they illustrate the interaction among instruments, for example, if they are played dominantly or if they are played in an ensemble, in a timeline.Second, played themes and their variations are illustrated as glyphs and the connections between them show repetitions.Sapp [Sap01] did not focus on the musical composition itself but on the evaluation of key-finding algorithms applied to the composition.This enables the user to inspect the (key) structure of a piece, as the different window sizes and algorithmic outputs show the development of keys within a piece.Goss and Carson [GC16] visualize the leitmotivs, harmonies, phrases and orchestration of Richard Wagner's 'Götterdämmerung' (Act II Scene I) using an animated, foursegment polar area chart.The size of a segment stands for the 'energy and direction of the music using an expanding and contracting motion'.Other approaches determine the 'shape' of a song [Wat01,Wat02,BKH07].Wattenberg [Wat01] used this term to question how music looks like, trying to map relevant musical to visual features while focusing on the repetition of structural elements [SS69].Wattenberg offers arc diagrams [Wat02] that group notes into sequences, recurring sequences are linked using widespread arcs.gaming industry also brought forth games where the player performs musical pieces.Most of those games let the player immerse into the song through simplified piano roll visualizations [Son04, Har07a, Har07b, Ubi11, Rea13] or map score features to game objects like space ships [Fit08,Col09].Miller [Mil13] gave insight into pedagogic aspects of different video games using visualization to link the user's desire to enjoy music and enhance the experience through gameplay elements.The data used by these games are either pre-included music files [Son04,Har07a,Har07b,Rea13] or local music files from the user [Fit08,Col09].The visualizations offered by GuitarHero [Har07a], Rock Band [Har07b] (Figure 7, right) and Band Fuse [Rea13] are similar to those of Rocksmith [Ubi11].In these games, users are presented with a rotated piano roll notation, moving towards the screen.Single notes are moving towards the player and they have to be played at the time they reach the viewport.SingStar [Son04] follows a similar paradigm but uses a horizontal piano roll to show the notes that have to be sung.In comparison to the above-mentioned works, other games do not require users to play or sing a note as gameplay, but to interact differently on the notes.Audiosurf [Fit08] (Figure 7, left) and Beat Hazard [Col09] both offer to playback private audio files, from which musical score features are analysed and used to generate a 'race track'.

Musical sound
In contrast to musical scores, this section focuses on visualizing sound data.Audio features extracted from music performances like pitch, loudness, tempo and timing are the basis for related visualizations.Some works use a combination of audio features and score data [HWF02,Isa03].For the works considered in this section, score data play a secondary role.Typically, the data are received from audio files, recorded sound information via microphone or MIDI interfaces-either real-time performed or pre-recorded.We further consider music videos as a visual enhanced form of audio art.Relevance for VIS.Sound and visualization both aid at communicating information to humans.While visualization uses the human eye for transmitting expression, emotion or meaning, the ear is the organ to experience auditory impressions.Both sensory organs detect signals, and the transmitted information carries meaning or emotion [Coo59].For hearing-impaired persons, related approaches try to compensate for the loss of the hearing sensation (the loss of the audio signal) and still allow the transfer of emotions and information of music through visual means [YLL*07, FF09].In this context, visualization scholars can learn how to design barrier-free representations of data-a research direction that is hitherto untouched in visualization.
Relevance for musicology.The visualization of sound is relevant for diverse purposes.While visualizing performances gives valuable feedback in learning sessions [MW03, YLL*07], characteristic patterns in interpretation strategies of musicians get easily explorable [PDW04,SH05].In addition, visualizations enhance the listening experience when offering interaction mechanism with the sound [Dix01, LL04, YGK*07].A further benefit could be drawn from adapting sequence alignment algorithms and visualizations to support comparing different interpretations (sounds) of the same score.
We group visualizations based on three main tasks for sound data.We first give an overview of visualizations that support analysing actual performances.While the traditional sheet notation of scores exists for centuries, sound features that prepare the ground for the visualizations discussed in this section are more experimental and less standardized.This is due to the rather subjective nature of sound perception.Many visualizations support analysing the mood of a song, which we summarize in the second paragraph.The last paragraph is dedicated to differences and similarities among different reproductions of music due to the subjective nature of music perception but also due to the human-typical deviation in performing music.
Task: Performance analysis.Performance data can be streamed with a microphone or with cartridges, transformed and shown on a display, while performing, to enable quick feedback loops.The corresponding scores can be loaded as MIDI files.Applications supporting performance analysis serve different purposes.Subtask: Structural analysis.Many abstract performance views enable structural analysis and comparison tasks.The former is achieved by including a temporal dimension to the visualizations, for example, scatterplots map time progression to saturation [DGW02,LG03], thereby illustrating the dynamic-tempo relation throughout the performance's progression.For visualizations that already contain a temporal dimension, like the piano roll notation, specific structural features are highlighted to allow quick navigation and overview of the distribution of such features [Got06].Visualizations can also aid to compare playing characteristics of musicians.Either concentrating on Jazz musicians [SH05] or classical music pianists [PGW03], tendencies in improvisations during performances are visualized using contour maps [SH05] or smoothed data histograms [PGW03].Subtask: Feedback.Giving feedback during live or recorded performances helps performers in improving their skills.Next to sound data, two works [HWF02,Isa03] make use of score information that is used (secondarily) to highlight discrepancies to the actually played tones.This is especially important for evaluating the correctness of performances [HWF02].Using Chernoff Faces [Che73], differences in pitch, flow, harmony, volume and keys can be shown.Alternatively, tools may help in deepening the understanding of the user's own performance, by visualizing different structural features and similarities in the performance [Isa03] 9).For drum manipulation, the tool offers three methods.The first method allows for timbre manipulation by selecting different kinds of timbre.The second method comes with a slider to change the volume for each drum separately.Lastly, a graphical editor (Figure 9) allows the user to rearrange the drum notes with click and drag interactions.
Task: Emotion analysis of sound.Fundamental for the perceived sound of music is the resonating mood and the communicated emotions [Coo59].This is an important task, as music is perceived by humans and can trigger different moods or emotions and researchers are not only interested in why but also in how music can achieve this.For instance, emotion can be transmitted with images that appropriately reflect the mood of sound [HLZ04,CWJC08].Sound snippets as well as images are tagged with related emotion adjectives, and, during playback, images reflect the current music mood

Visualization of musical collections
So far, we focused on the analysis of a single or very few music pieces.Collections, in comparison, can be a whole music album, a playlist or the music archive.General issues when working with large musical collections are classification, recognition, annotation and the retrieval of music due to the increased required technology capacity, large amounts of available music data and acoustic information of sound that needs to be processed [CVG*08, FLTZ11].Visualizations of large music collections are based on diverse features of musical data, and they are of interest for users who desire new perspectives on their musical archive that are different from plain file lists.Relevance for VIS.This section includes a large variety of visualization approaches designed for non-expert users to support tasks like navigating, exploring and editing music archives.The visualizations themselves are grounded on diverse features of musical pieces, ranging from low-level sound features to descriptive metadata and crowdsourced information.Thus, visualization scholars can learn how such a large palette of information can be processed, and how the results can be visually represented in an intuitive way.
Relevance for musicology.While the visualizations listed in this section are typically designed to help non-expert users in navigating through their music archives, the relevance for musicology is limited.However, most systems are applicable also to music collections of musicological interest.Further, there is a growing interest in music alignment visualizations that help musicologists in analysing recurring sound patterns [Mül15].
Most of the surveyed methods support casual exploration tasks for music collections.This includes that users get an overview of the collection and can interact with it to perform typical tasks like generating playlists or playing back music of interest.Further tasks that are supported by visualizations are the analysis of listening histories as well as exploring music alignments.
Task: Explorative analysis.Though music pieces that compile a collection are typically arranged hierarchically in a file system, visualizations aid to give a more comprehensive overview by allowing to see what is contained in a music collection containing thousands of songs.Further, interactive visual interfaces support a variety of tasks, ranging from user-driven playlist creation [vGVvdW04, VGV05, GG05, CRV*06, CB09, PEP*11] to automatized music recommendation [BCD04, HHKB06, Don07, SWT08, ZL17].The musical pieces of a collection can be also arranged circularly based on their features.This can be done for a song of interest where the other songs are mapped to a circular layout based on audio features and their similarity to the target song [HHKB06], or as visible tempo or genre distribution of the collection in a music player [SPKW06], or as a mapping of artists to the colours of a circular rainbow based on the audio features of their songs [PG06].Three-dimensional spaces are also used to arrange glyphs representing musical pieces [TC00, NM02, LT07].For example, Notess and Minimayeva [NM02] use a three-dimensional coordinate system, Subtask: Focusing on genre.Genre is one of the most important metadata of music that we use to select the music we like to listen to.Two works focus on communicating genres, sub-genres and the dependencies among them with interactive overviews [CC18,McD18].Both platforms encourage casual visual exploration by allowing to playback audio samples for selected genres.While Everynoise [McD18] provides a tag cloud to display genresclicking a genre generates a tag cloud of related musicians and music bands-Musicmap [CC18] offers a zoomable treemap (see Figure 12) for that purpose.Torrens et al. [THA04] organize personal music libraries also by genre in treemaps, rectangle maps or circularly on a disc.Each song is represented with a tiny diamond glyph and changes-through playlist editing-are highlighted.Spatial arrangements of music items by genre are also produced by SOMs [RPM03, MUNS05, KSPW06].A representative example-the 'Island of Music' [PRM02, PDW04]-uses a geographical metaphor and organizes genres as islands like shown in Figure 13.Knees et al. [KSPW06] extend the visual metaphor with an auditive one.When moving through the genre map, audio files closely positioned to the mouse cursor are played.
Subtask: Focusing on mood.In contrast to the genre, other applications place songs having a similar mood closely [HHKB06,HG13].Andjelkovic et al. [APO16] formulate the need to include mood similarity into content-based similarity classification.For that, they offer an interactive user-interface to enter the names of favourite performers that are used to project musicians in a latent mood-space spanned by the mood categories sublimity, vitality and unease.Closely positioned musicians aid as recommendations for the user.Van et al. [vGVvdW04,VGV05] also offer a mood map that is laid out using a force-directed placement approach.'Magnets' representing mood categories are placed in the two-dimensional space, and they attract related musicians.Musicream [GG05] colours discs that symbolize musical pieces according to mood.The user can interactively assemble playlists via drag and drop.When doing so, songs with similar moods can be easier stitched together.Subtask: Focusing on popularity.Crowdsourced information can also be used to feed an algorithm that arranges musical items in a two-dimensional area.Donaldson [Don07] analyses existing playlists for that purpose.The more often two songs are listed together, the more similar they are.After applying an MDS algorithm, the music collection is explorable in 2D.Sprague et al. [SWT08] present a democratic approach to select the next songs to be played during a social event.The offered collection of songs is mapped in a two-dimensional plane, and users' votes are visually highlighted.Figure 14 is an example of a music collection (on the right) and voting area on the left side.The preferences form a convex hull that includes all song candidates that can be played next.A user's vote influences the 'weight' of the chosen song and all nearby songs, thus, increasing the probability of a whole region of similar songs of being selected.
Task: Analysing listening statistics.Exploring the popularity of songs is of interest for individuals who are curious about what they have listened to in the past.Due to the time-related reference of listening histories, timelines are the means of choice for visualization.Byron and Wattenberg [BW08] visualize last.fm[Lim02] listening histories with a stream graph.Each stream stands for a musician, and the colour of a stream indicates the personal popularity of the musician as well as the initial onset time.A more detailed view is provided by Baur and Butz [BB09], who represent each song with a thumbnail icon and connect subsequently heard songs in the form of a graph overlaying a timeline.Applicable for listening histories spanning multiple years, Baur et al. [BSSB10] generate heat maps juxtaposing daytime and years, thus, enabling to discover patterns of how and when people listen to music-as seen in Figure 15

Visualization of musicians
Besides musical pieces, musicology also focuses on the people associated with music.These are not only composers, instrumentalists and singers but also instrument makers, musical teachers or music publishers, and further.A variety of biographical information pre- Musicology provides real-world data with diverse types of uncertainty that can be used to develop comprehensive models to visualizing uncertainties prevalent in different data types.Some case studies report on how uncertainty can be dealt with and how uncertain features can be visually communicated [MWP12,Jän18,KJKF19].We focus on this subject more detailed in Section 5.
Relevance for musicology.Digital tools have only been rarely applied or designed to support prosopographical research in musicology.Traditional prosopographical research typically focuses on one of the famous composers and their main works in a philological manner [JFS16].A few works illustrate that visualizations, which communicate contents of large biographical databases [Foc19], serve musicologists with intuitive views on large person groups [KJ16], thus, focusing on an entire community rather than individuals.The value of visualizations to engage with new research questions in musicology is reported by Jänicke et al. [JFS16,JF17].Further, visualizations aid to make relations among musicians that have not been considered visible for the first time [KJKF19].
Visualizations can be grouped according to three major user tasks discussed in this chapter.First, they enable multifaceted explorative visual analysis of biographical data in order to give an overview that allows detecting trends, noticeable features and patterns.Second, visualizations aid at analysing social networks of musicians.Third, tools have been developed to enable visual analyses of similarities among musicians.
Task: Explorative analysis of musicians.For hypothesis verification and generation, many visualizations follow Shneiderman's Information Seeking Mantra [Shn96] to provide an overview of the biographical database at hand, and means of zooming and filtering are to be used to analyse patterns of interest, to explore clusters of musicians having similar characteristics and to detect outliers.Typically, biographical information of various types given in databases is used to generate an overview of the data [AWR*07, KJ16, Doi17   with seven widgets to exploratively analyse the data-two sunburst plots for musical and non-musical professions of musicians, two tag clouds for their denominations and divisions, a pie chart to contrast male and female musicians, a timeline for activity times and a map to inspect places where musicians worked.Other works take fewer biographical data into account.Doi [Doi17] focuses on music bands and the locations of their performances.Those are plotted on an interactive map that supports analysing the popularity of places.Continuum [AWR*07] illustrates the careers of musicians, that is, their creative works, on a timeline.Means of filtering enable the detection of dependencies among musicians of the database.Instead of using biographical information, Daniels [Dan14] gives an overview of a rap musicians dataset according to vocabulary size extracted from their lyrics on a horizontal axis.To avoid occlusions, nodes are vertical.
Task: Social network analysis.Graph visualizations make relations among persons of biographical databases visible, and they intuitively reveal societal structures of communities.Biographical databases like the BMLO [eJF15], the Linked Jazz database [Pat18] or the Red Hot Jazz Archive [Ale02] provide relationship information among the musicians they include.Relations are often categorized-typical relations are familial, academic or work related.Two projects focus on visualizing social networks of Jazz musicians [GD03,MWP12].While the Community Structure Jazz project [GD03] provides a force-directed drawing of the network to communicate the problems of racial segregation, the Linked Jazz project [MWP12] places 20 Jazz musicians having the most registered relationships on an ellipsis and a barycentre drawing is applied to layout the entire network like shown in Figure 17.By additionally scaling node sizes according to importance, influential Jazz musicians are quickly detectable.Laying out the social network graph on a timeline allows detecting by what musicians musicological knowledge has been transferred over time.Jänicke et [KJKF19] place musicians depending on their life dates on a timeline, and related musicians are connected with each other.As musicians are grouped either by institutions they belonged to or by musical professions they exercised in, potential relationships among musicians can be hypothesized.Different zoom levels-focusing on one, a few dozens or all musicians in the filtered dataset-are provided to support this task.
Task: Musicians similarity analysis.As opposed to exploring general characteristics or social networks of musicians, many visualizations have been designed to support the analysis of similarities among musicians.Most visual interfaces are tailored for casual users, typically, to encourage intuitive exploration of music recommendations [Gib11,Vav17,Spo18].The similarity of musicians is determined using crowdsourced information, for example, Amazon sale statistics [Vav17] or Spotify listing histories [Spo18].All approaches follow the same concept.The user searches for a musician that is laid out alongside a fixed number (max.20) of most similar musicians in the form of a graph.While the Artist Explorer [Spo18] uses a tree for that purpose, LivePlasma [Vav17] and Music-Map [Gib11] use a force-directed graph where edge lengths, or distances, respectively, reflect similarity strengths between musicians.A seamless exploration of the similarity data is enabled by refocusing on any shown musician that can be selected via mouse click, and the playback of music samples can support this task [Vav17].The works by Cano and Koppenberger [CK04] and Gleich [GZL05] (see Figure 18) differ slightly from the previous ones as the graphs, which likewise reflect similarity with proximity, give overviews of entire databases.Two works focus on exposing similarity in the form of influences between musicians visually.Schedl et al. [SKW05] extract prototypical musicians per genre and vaguely define influence among musicians based on the number of co-occurrences in online texts.The result is shown using a circular graph layout, that places similar musicians closely.The project 'History of Rock' [LA18] uses a manually curated dataset of 100 music songs assessed as the most important milestones of rock history [The11].Multiple coordinated views are offered for exploring

's network graph 'Linked Jazz' [MWP12] uses the musician images for quick recognition. The linkages between the nodes encoding relationships between the jazz musicians, making it possible to derive further information about the social network, the interaction between the artists and influences. The size of images encodes the impact on the community through the number of relationships.
the corresponding musicians; Figure 19 shows visualization when focusing on The Beatles.A force-directed graph illustrates Who influenced whom? relations between musicians, their songs are listed on a timeline to analyse trends and temporal dependencies, and bar charts convey energy (from low to high) and emotion (from sad to happy) levels.While the above-mentioned approaches are merely designed for casual users and similarity is based on the works of musicians, Jänicke et al. [JFS16] designed a visual analytics system for musicologists to support the profiling of musicians based on biographical characteristics (provided by the musiXplora [Foc19]) like places of work, musical professions or denominations of musicians.After choosing a musician of interest, the similarity to all other musicians in the database is determined based on eight similarity measures.The biographical details of the chosen musician and the most similar ones can be comparatively analysed in a stream graph illustrating textual features, a social network graph and a map showing related places of work.

Visualization of instruments
Besides musical works and associated musicians, the played instruments are in the focus of interest for musicologists, too.Musical instruments are part of previously mentioned visualizations like Haro et al. [HXF*10] who links avatars stereotypically to genres and mood, or games such as Guitar Hero [Har07a] in which parts of instruments are visualized.In this section, we focus on the visualizations made for physical instruments and their digital representations, for example, in the form of three-dimensional object scans.
Relevance for VIS.While the work with instruments has a century-old tradition in musicology, we observed that physical instrument data are rarely used as a basis for visualization research.In recent years, many works have been published in different domains having different challenges depending on the 'objects' to be scanned, for example, the industrial domain [DCCK*14], like metal joins [ZVMK17], the medical domain dealing with organic subjects such as small animals [LZM*17], human infants' heads [HPDCSC*16] or muscle tissue [PM16] as well as inorganic objects like nanoparticles [JAV12].The specific characteristic of musical instruments is that they are made of very different materials provoking research questions that are hitherto untouched by visualization research.A few related works, mostly published in musicology, offer basic approaches in dealing with instrument data [KBF*16, Hel17, KSKE17, Kir19], thus, pioneering work interesting for the visualization community.In addition, we list numerous future challenges related to instrument data in Section 6.
Relevance for musicology.From a musicologist's point of view, working with visual representations of instruments offers diverse opportunities.The work with digital representatives of original instruments is non-destructive, and many users can have access to a digitized instrument that can be documented and annotated with metadata.Further, visualizations can help to explain the functionality of instruments.Our collaboration partners from a musicology department mentioned that visualizations illustrating concepts of instruments, their function (like sound generation), their cultural contexts or provenance research (which deals with inspecting the origin of instruments) are missing.Engagement of visualization scholars in musicology is desired and provides new research challenges as outlined in Section 6.
We divide related works into two groups of tasks.Firstly, visualizations aid at supporting to analyse structural features and special material properties of instruments.Secondly, visualizations have been designed to communicate how musical instruments function and how sound is generated.
Task: Instrument structure analysis.Three-dimensional visual representations of musical instruments are frequently used to communicate structural features.Especially, if instruments are very old or if they are unique, a digital copy enables an explorative analysis of the instrument's construction or manufacturing without harming the original.Different measurement and imaging techniques are applied to digitize musical instruments.While laser or X-ray scanners [Hel17] and computed models based on photos of instruments [KSKE17] are rarely used, most frequently,   [Hop18], is shown in Figure 20.In this example, the transfer function is defined to highlight glued parts of the instrument in white to highlight previously repaired parts that may affect the value and sound of the instrument.A typical task for documentation and conservation purposes is analysing the thickness of musical instruments, for which heat maps, encoding thickness values with colours, are the means of choice [BS09, dBLD*17, Hel17].Figure 21 shows a wood thickness heat map of a violin tracked by laser scans.Such representations are used in musicology to compare and to categorize different instruments of the same type, and they can even help to determine the instrument maker [Hel17].Heat maps can also be applied to visualize long-term influences on instruments due to storage and aging processes [KSKE17].Using finite element simulations, the observer can inspect the estimated deformation throughout the years.Figure 22 shows expected longterm structural deformations due to humidity changes in red and blue colours.Such views help conservators in handling and storing musical instruments accordingly.While the former visualization approaches serve musicologists with three-dimensional digital copies of instruments to support research tasks in musicology, Kusnick [Kus18] and Schott [Sch18] created a system that allows museum visitors to playfully observe musical instruments in augmented and virtual reality environments.Figure 23 shows how a tablet device enables museum visitors in getting a non-destructive, new perspective on a historical instrument-not only from the outside as when the instrument is exhibited in a case but also from the inside.The advantages of such immersive analytics tools are manifold, they can support pedagogical, conservational as well as documentation tasks, and, perspectively, they can also support organologists and instrument makers in understanding the sound generation of the instrument.
Task: Instrument functioning analysis.Communicating how musical instruments operate is a mandatory task in musicology.Pioneering work in this regard has been done by Askenfeld and Jansson [AJ91] who illustrated the physical effects of the human touch on strings and hammers of pianos during sound generation.Though presenting such results in non-interactive charts is sufficient for documentation purposes, sophisticated visualization and interaction techniques ease understanding how instruments function and ease the access to such information with appealing representations.Two works offer to view and interact with digital instruments in Augmented and Virtual Reality environments [MPLKT05, SEK*16].With gesture capturing, virtually represented, three-dimensional instruments can be played, and the user can interact with colourcoded rectangles, each encoding a different, playable sound spectrum.Further, two approaches communicate the dynamics of instruments [BGW08,BMSH13].Bouënard et al. [BGW08] study the real membrane of a drum in detail to generate dynamic and kinematic motion data through motion capturing.They display the trajectories of the drum sticks, which allows to analyse special hitting techniques of a player.Berthaut et al. [BMSH13] record actions when playing an instrument through sensors and generate virtual animations of three-dimensional shapes.Figure 24 communicates the functionality of a synthesizer.Depending on the pressed button or regulated knobs, the animation on the display below the controllers shows movements of represented mechanical parts and how they interact to generate a sound.

Dealing With Musicological Data
The long history of musicology yields large datasets that comprise a diversity of inconsistencies.We skimmed through the paper collection and investigated how researchers deal with those inconsistencies and how and if the proposed interfaces reflect them visually.Additionally, we reviewed if and to what degree vastness of data yields challenges to accurately design a data visualization.As a result, we observed different forms of uncertainty [Mch19].

Vastness
The vastness of a dataset refers to the size that leads to problems in visual representation, for example, visual clutter.Although musical collections can be rather small compared to other areas of visualization, vastness still becomes a problem due to musicological metadata that includes different types of entities.So, transforming the metadata of musical collections and prosopographic datasets into intuitive visualizations becomes a problem.Typically, multiple views [WBWK00] are used for filtering purposes to reduce the number of data items to be visualized, for example, when browsing biographical characteristics of around 30,000 musicians [KJ16].In order to visually highlight a subset of recommendations out of millions of musical pieces, Donaldson et al. [Don07] offer basic means of filtering and apply a liquid browsing technique [WB04] to get a hand on occluding nodes representing recommended audios.Other visualization approaches are not reflecting vastness as they consider its effect as negligible.Everynoise [McD18] arranges all available 2635 music genre tags on the screen and offer scrolling functionality for browsing purposes.As the tags are arranged according to the similarity of subgenres, 'Serendipitous Browsing' [Mas11] supports the task of finding related genres.Commercial software for emulating songs, for example, Audiosurf, Rocksmith or Gui-tarHero [Har07a, Fit08, Ubi11], solely show a short sequence of notes in a time-dependent frame.Hiraga et al. [HWF02] offer a one-dimensional fisheye focus + context view to inspect the entire score sheet of a composition on a single line having a fixed width.The vastness of data items is crucial when graphs are the means of choice to support related user tasks.Jänicke and Focht [JF17] propose a semantic graph clustering strategy [dRdSP*10] to condense the number of nodes to be displayed within a social network graph of musicians-laid out by traditional force-directed placementwithout harming the user's capabilities to investigate the intended research questions.In contrast, Ono et al. [OCF*15] opt for a visual clustering by applying edge bundling and a focus + context metaphor to investigate the relations between audios in a similarity graph.However, many works do not cater for occlusions in graphs at all, taking occlusions among labels [CRV*06, YSB09] or edges [GD03,CK04,MWP12] harming the readability of the graph into account.

Imprecision
Imprecise data features are manifold in musicological applications.Typically, audio signals are perturbed with noise, and it is unclear whether the measured noise relates to 'deliberate expressive strategies, motor noise or imprecision of the performer, or even measurement errors' [LG03].Though different strategies to handle noise exist [TEC01, HHKB06, SY09], none of the surveyed works communicate the influence of noise on a similarity analysis visually.Pampalk et al. [PDW04] leave the noise handling to the user who tweaks a neighbourhood radius that smooths the spatial mapping of the musical collection.Next to auditive imprecision, musicological metadata like datings or geographical information is inaccurate.In historical datasets, this typically relates to the limited information available for a subject of interest.In MusikerProfiling [JFS16],  Concerning the imprecision of time, genres are also ambiguous entities of music history [SPKW06].Crauwels [CC18] highlights that a music genre does not suddenly emerge 'as a shocking revolution without any trace or evolution in the past'.Consequently, a precise mapping on a timeline is not possible.In his Musicmap he relies on the year, two or more different artists published work in the same genre as a genre's starting point, neglecting the mentioned trace in the past.As genres are hierarchically arranged, the mean year is used when joining sub-genres with different years of origin.Hilliges et al. [HHKB06] replace a traditional genre clustering by emotional genres after comparing the emotional qualities of the audios, to encounter the ambiguity of the genre names.
As opposed to communicating the imprecision of time, the geospatial inaccuracy that is manifested in different levels of granularity when assigning places to musical entities has been rarely addressed in related works.The current procedure is representing each geographical entity (e.g.city, region, country) as a location on the map [Foc19].When generating CT scans of instruments, noise artefacts occur due to different characteristics of materials an instrument is composed of.Though such inconsistencies are mentioned [Kus18], appropriate solutions to overcome those do not ex-   ist.For streaming purposes, CT scans need to be downsampledthus, an intended imprecision due to the loss of information-to enable fluent interaction with the instruments' three-dimensional models [EKK*17, Kus18].

Incompleteness
Referring to a dataset with entities having different features, incompleteness refers to the percentage of missing feature entries.For these entries, it is known that the values actually exist, but it is not given in the dataset [Mch19].For a note sheet, this might be a corrupted, thus, unreadable note line or a missing page-typical issues when dealing with historical material.Consequently, Optical Musical Recognition (OMR) software induces errors [Bul08].Existing visual representations of note sheets [Wat02, Hal13] do not account for this issue.When developing visualizations [JFS16,KJKF19] for the musiXplora [Foc19], the activity time of a musician was defined by the beginning of a musician's career and the musician's date of death.For around 30 000 musicians in the database, approximately 10% of the former date and 30% of the latter date are missing.Different strategies were applied to approximate the time range in such cases, typically, by taking other temporal metadata such as the date of birth or the last mentioning of a musician into account.By not displaying the marker for the beginning of the musician's career [JFS16], this approximation is indirectly communicated to the observer.If no temporal information was given, the musician is disregarded from the visualization.Timages [Jän18] arranges the portraits of musicians in the musiXplora on a timeline to communicate their historical influence.Missing portraits are visually expressed by a dummy image instead.The Linked Jazz project [MWP12] involves another kind of incompleteness.Personal relationships between jazz musicians are extracted from interview transcripts that are not completely processed by now.Music learning software, for example, Rocksmith [Ubi11] offer different levels of difficulty.Training with low difficulty, a certain number of notes are excluded, thus, benefits are drawn from drafted incomplete data.Due to the different physical processes during a CT scan of a musical instrument consisting of manifold materials, aberrations in the resulting datasets and images are unavoidable [EKK*17, Kus18].Especially, radiation artefacts lead to irretrievably lost holes in the three-dimensional models.

Inhomogeneity
As opposed to the incompleteness of a dataset that can be measured as the actual number of missing information of data items in a given dataset is known, the inhomogeneity of a dataset cannot be quantified, as it depends on the state of research undertaken on the topic addressed in the dataset [Mch19].Datasets in musicology are partly built on qualitative research [Foc19], thus, the knowledge on single cultural heritage objects or individuals is uneven.Next to relevant data items unknown to the creator of a dataset, undocumented or overseen attributes lead to a distorted representation of reality.MusikerProfiling [JFS16], which operates on a biographical database of musicians, deals with inconsistencies due to inhomogeneity.As of the late 19th century, musicology focuses primarily on 50 musicians and their main works in a traditional philological manner.Next to adjusting the popularity weight in the visual analytics profiling process, the domain expert can reduce the weight of relationship similarity, a value strongly affected by the inhomogeneous state of research in musicology.This way, the musicologist is enabled to throw the spotlight on less popular musicians.Furthermore, crowdsourcing projects suffer from inhomogeneity.Music-Map [Gib11] gives an overview of the proximity of artists in dependency of casual users who list their favourites that will be placed closer to each other in the global overview taking all crowdsourced information into account.As crowdsourcing projects are usually biased [Kos09]-as they are fed by a specific community-the spatial distribution of artists is consequently inhomogeneous.A special form of inhomogeneity is the non-standardized terminology in musicology due to the century-long history of music.For example, a comprehensive and scalable classification of instruments and instrument classes does not exist.To resolve occurring ambiguities, a mapping between different sources is needed.In the best case, this is offered with high confidence through a domain expert.Otherwise, this inhomogeneity has to be communicated by visual means.

Future Challenges
During our extensive literature review, we encountered open problems in musicology that could be supported by suitable visualization approaches.Combined with the needs that musicologists reported us throughout our perennial collaboration and our own experiences gained during interdisciplinary projects, we introduce a list of future challenges for visualization in musicology.Despite the manifold application fields of the researched visualization, the backflow into a mainstream practice is rather low and not documented in surveyed publications.Probably the group of interfaces for music production is the most affected, as seen through the commonly used piano roll visualization.

Strengthen interdisciplinary collaboration.
The papers included in this survey are typically written by either musicologists or computer scientists, but seldomly in a cooperative setting.While we do not want to devalue these works, we want to highlight advantages for both fields in cooperating, as well as stress that works dealing with musicology and visualization should meet requirements from both fields.Furthermore, collaborations with graphic designers or visual artists can lead to more artistic visualizations like the work of Lupi and King [LK18].For the visualization and musicological domain, we included a short paragraph for each data section, highlighting advantages, challenges and in general relevance for each separately.These data-type specific points will not be repeated here.Instead, we want to highlight that the collaboration is crucial, as the musicological part in music visualizations brings the expertise needed to analyse features and comprehend reasons and causes.For the visualization part, there exists a big effort to learn how visualization can be used to improve the communication of features to users and how to reduce distortion.Further, visualizations or more general computer science techniques are crucial to providing infrastructure and knowledge about data handling, digitization and processing.Next, musicology offers a vast amount of a wide range of data types, often not fully tapped.Scientists are interested in literally seeing data, which is the high value of visualization, as it does not only offer the data itself but users (casual users, researchers, experts, ...), research questions and in general a demand of visualization for making use of the potential in data, also.
Distant-reading analysis of score collections.While we do have a whole section dealing with musical collections (see Section 4.2), all found works deal exclusively with sound data.As seen in Section 4.1, not only the sound itself is of interest to musicology but also the scores of songs.When taking this into account, it is quite astonishing that there is a clear lack of collection-wide analysis tools of musical scores.This could be caused by the collection section mainly holding tools used by casual users, who are more interested in the sound itself than the written score.Nevertheless, these questions exist and can be answered with distant-reading approaches, making it easier to find similarities and significant outliers in score texts.Plausible questions can be included to find strict similarities and plagiarism Metadata visualization of instruments.As described in the instrument visualization section (see Section 4.4), researchers started to include digital representatives of instruments in their consideration.While the found and included works show usage of mainly CT data and three-dimensional rendering, we also encountered textual metadata usage.From a visualization point-of-view, these works were too weak to be included in the survey.We found this lack of textual metadata visualization quite surprising, as this is quite common for the other listed data types (especially the musicians and their biographies).One example is the timeline visualization of instruments from multiple musical instrument museums by Kusnick et al. [KKFJ20b].The single instrument's lifetime events are enhanced with hypothesized relations to matching musical pieces, through the analysis of metadata about the two entity classes.Further, a metadata visualization approach is rather simple, compared to a three-dimensional one, as knowledge of other sources can be used and tools adapted.Like in the previous paragraph, we encountered the potential for especially distant-reading tools, which allow inspection of, for example, trends of instrument usage throughout the time.An indirect example of a usage scenario can be seen by Khulusi et al. [KFJ18,KJKF19], who visualized the temporal development of the lute instrument, represented through metadata of lute players.A more direct approach to visualizing instruments developments and trends has been marked as interesting by our collaborating musicologist, as well.
Visualizations of an instrument's history.The value to visually analyse musicians embedded in a temporal context is documented in various works [AWR*07, YSB09, KJKF19], as it enables under-standing trends and seizing historical events.While the above paragraph dealt with general trends for a type of instrument, our collaborating musicologists desired similar approaches to investigate the career of a single instrument, for which far fewer metadata exists.The musicological term for inspecting the career of an instrument is provenance research and is built on the knowledge that an instrument has different properties throughout the time and is to be inspected with regard to these changes and not as a static object.As an example, an instrument may have undergone a restoration process, changing physical components or, been modernized, resulting in a change of sound range.Other examples of events shaping the career of an instrument may be production, owner change, presentation or performance.Each of these events can be associated with geospatial and temporal data, associated persons and further metadata (like prize, value, collection, etc.).Currently, such information is collected and digitized in musicology projects such as the musiXplora [Foc19] or Music Instruments Museums Online [Mus11], and first visualization approaches are available as browser applications (see a prototype-like screenshot in Figure 26).But, sophisticated approaches are necessary to align an instrument's career to historical musical pieces, such as the Répertoire International des Sources Musicales (RISM) [LBPP19].Supported by interactive visual means, this would enable to investigate which instruments may have been used in performances like operas.Still, this needs further basic research focussing on musicological terms, that is (especially throughout the centuries) non-standardized and hence hard to match (see Section 5.4).

Workflow for musicology.
Musicology is a heterogeneous field, dedicated to different types of users, and including different types of data and different tasks.In order to ease the process of developing novel visualization strategies for musicology, a generic workflow, guidelines and best practices are highly requested.Miller et al.
[MHK*18] propose a pipeline for designing visualizations for musical scores, which serves a clear need but only includes one of the data types.A holistic workflow or further pipelines for musical collections, musicians and instruments are highly appreciated.
Working with historical data.The lack of works dealing with historical data was quite surprising.Musicology, a field with a long history, offers data aged for centuries and interesting questions on the development of cultural aspects.Next to works dealing with biographical data [AWR*07, JFS16, KJKF19], only a few visualizations have been designed to communicate sound patterns [Wat02,Hal13].As stated in Section 5, dealing with historical data is not trivial and requires novel workflows.Our collaborating musicologist highlighted the need to regard musicological data under historical aspects.An example is a pitch of instruments (the difference in distance of the halftones of instruments), which changed throughout the time, resulting in a distorted perception of music, for example, emotion and harmony when addressing historic instruments.
Communicating inconsistencies in musicological data.Our review on inconsistencies in musicological data and how these are processed (see Section 5) revealed a lack of solutions to visually inform users about these inconsistencies.Additionally, there seems to be an awareness that inconsistencies like imprecision, incompleteness or inhomogeneity exist.Only a few works introduce techniques to visually encode temporal imprecision [JFS16,KJKF19].Future works should not only disclose such information to avoid misinterpreting visualizations but also produce novel visualization means to communicate uncertainties.
Multi-modal data visualization.Many research inquiries in musicology address only one specific data type, and only a few works tackle multi-modalities [YGK*07, KJ16, AGC*17, KJKF19].As the musiXplora [Foc19] shows, the combination of multiple types of entities provides a high potential for musicology research that enables gaining new insights.It offers a database that can be queried through diverse web-based visual interfaces [KKFJ20a].The database includes the present state of musicological biography data for more than 30,000 persons relevant for music history from around 2000 years.As an ongoing research project that started in 2004, the musiXplora deals with historical and present sources of data of different facets of musicology.Although it currently focuses primarily on person's biographies, the responsible researchers hinted on further, unpublished data about other musicological-relevant data types like places, objects, institutions, media, events and terms [Lei17, Lei18].The offered data are already used by multiple research papers included in this STAR [JFS16, KJ16, JF17, Jän18, KJKF19].Besides this, the work with multi-modal data, shown through many associated works gives insight into potential new approaches to provide data access through visualization.Data-wise, all available data are stored in a relational database and is accessible as textual data, linking features to IDs of the persons.This leads to data like a person's working time, information about professions like soprano singer and violin player as well as institutions he or she worked at like the 'Bayerische Staatsoper' ('Bavarian state opera') and 'Würzburger Hofkapelle' ('Würzburger's court orchestra').Due to the historical nature and dependency on documentation, the data are missing a linkage between the different subtypes.Hence, for example, information about a person's profession that was held in the Bayerische Staatsoper or the working time of the musician in the court orchestra is not extractable, only a list of all professions and the general working time.Thus, the database does not only bring its own challenges and issues but also shows musicological approaches to tackle many of the previously listed challenges, arising in working with musicological data in general (see Section 5 and previous paragraphs of Section 6).Examples include temporal uncertainties, communicating inconsistencies in terms, non-standardized, incomplete and uncertain sources and further.
Annotating three-dimensional models of musical instruments.Recently, the visualization of three-dimensional CT scans became a valuable tool to analyse and interact with musical instruments, supported by a general trend in science and a rocket-like explosion of new advances in the computed tomography sector in the last 30 years [H*09].Especially for restorers, this method could support the non-destructive inspection of valuable objects and the discovery of weak or damaged parts, not visible from the outside.A useful tool would not solely require the functionality and pipeline of an instrument being scanned but also to enable annotations.This would help to directly mark and document conspicuous parts, even through changes of scale and the colour transfer function.We identified work going into this direction (see Section 4.4 and the work of Konopka et al. [KSKE17]), but they rather use simulations to get insights into expected changes on an object in the future, instead of actual ones as of today.The rendering of virtual instruments alongside with annotated parts is also desired for teaching and presentation purposes.In addition, animations could aid to communicate how instruments work.Visualization of the internal movements of music instruments, the flow of air and mechanical interaction could further enhance the understanding of learners and museum visitors.For such a diversity of visualization applications addressing multiple user groups, the major challenge is not the visualization in the first place, but rather the design of intuitive, interactive user interfaces.

Simulation, visualization and exploration of instrument data.
Many historic instruments cannot be played without the risk of damaging the object.Still, researchers are interested in hearing the sound of these instruments and the understanding mechanisms involved in the sound generation.Kirsch and Plath [SP18] showed that the visual animation of instruments' sound generation based on scanned instruments is possible.For enjoyment and pedagogical purposes, this information may be presented with visualizations.Besides the reconstruction of sound generation itself, visual reconstruction of the surfaces of CT scanned instruments may allow simulation of the original sounds.

Reconstruction and reproduction of instruments.
A high-level goal is reconstructing (rare) musical instruments based on CT scans.First approaches use CT data and 3D scanners to print plastic variants [SP18].While giving a hint on the feasibility of this task, even allowing for sound generation, these prototype-like prints lack both, quality and material components, to be considered as a kind of replication.A possible pipeline for such a project may be the reconstruction, followed by the reproduction either through the automatic generation of blueprints or direct linkage to 3D printers or woodworking machinery.

Conclusion
The intersection of musicology and visualization has brought forth a diversity of innovative applications designed for a variety of purposes.On the one hand, musicologists are served with interactive tools to analyse musicological data of a different kind, on the other hand, applications are tailored for the broad public to communicate and to teach aspects of music in a more intuitive, playful manner.Our survey includes 129 related works from different disciplines, all of them include sophisticated visualization techniques to illustrate features and relationships among musicological data entities.With a strong focus on the targeted musicology domain that provides data, we classified the works first by the type of data-musical works, musical collections, musicians and instruments-with sub-categories indicating the intended use of the offered visualization, followed by a deeper analysis of supported domain-specific user tasks.This allows to easily detect visualizations designed for specific purposes, for example, for comparative analysis, for enjoyment or for annotation support.The diversity of application scenarios yields datasets of various scales comprising different inconsistencies.We define aspects of uncertainty in the scope of musicology, and we investigated in what form vastness, imprecision, incompleteness and inhomogeneity occur in musicological datasets and how upcoming issues are tackled.Though such inconsistencies are prevalent in many datasets of the collection, a few works communicate them visually.The visualization of uncertainty in musicological data marks a major future challenge.In addition, our survey summarizes diverse future prospects including the intense engagement with historical data to serve a broad palette of research questions and solutions for in-depth analysis of three-dimensional models of instruments aiming to enable the reconstruction of unique items.

Figure 1 :
Figure 1: Overview of all used visualization techniques for each class of data.

Figure 2 :
Figure 2: The first notes of Ludwig van Beethoven's "Für Elise" in classical sheet notation (top) and Wattenberg's Arc Diagram Visualization of structural repetition for the whole piece (bottom) [Wat02].
Figure 3: Commercial software like Liquid Notes [OTG19] (left) can be used to help the user in composing music.The piano roll view (A) enables an overview of the compositions.Instruments or sound pitches were encoded along the y-axis, position and length of notes were drawn along the x-axis.Additional controllers adjust a single sound or the whole song.Ciuha et al. [CKS10] (right) visualize note sheets and piano rolls.The colour represents the different keys or the harmonic relations between tones using excerpts (from top to bottom) from Pachelbel's 'Canon in D major' and Debussy's 'Clair de Lune'.

Figure 2 (Figure 4 :Figure 5 :
Figure 2 (bottom) shows the visualization of Beethoven's 'Für Elise', and repeated sections are seizable.Arc diagrams can be additionally applied for providing an overview of different works and compare them as shown in Figure 5. Bergstrom et al. [BKH07] introduce Isochords to shape a musical piece.The structure is laid out on a two-dimensional, triangular isometric coordinate grid.Distance in the grid indicates the consonance and dissonance of tones, providing a quick look into the structural features.The sight of structures in musical pieces can support composing new or editing existing music.Using different MIDI channels of an input file, which can be seen in the upper part of Figure 6, Miyazaki et al. [MFH04] enhance a score structure visualization by a three-dimensional circular representation using cone-trees, representing subsequences of the musical piece and encoding sound features in a circular piano roll.Task: Instrument performance analysis.One of the main functions of scores is enabling instrumentalists to (re-)interpret musi-

Figure 6 :
Figure 6: An extract of Comp-I's timeline (top) and structure view (bottom), where each note is represented as a cylinder on the timeline and the different subsequences of the song are drawn in a circular layout [MFH04].
First, (real-time) visualizations of (live) performances can help in classifying and describing performance styles of musicianseither generally [TED85, DGW02, LG03, SH05, Got06, SML*16, Huy17, WZBKB17, LK18] or explicitly through structural features [PGW03, SH05].Second, a teaching perspective gives feedback to users helping to improve their performances [HWF02, Isa03, MW03, YLL*07].Third, performance sound can be edited or manipulated [Dix01, YGK*07].Subtask: Single performance analysis.An in-depth analysis of a single performance is subject to many related works.Single performances can be analysed in real time using web services [TED85, Huy17].They offer means to record performances and the corresponding sound is represented in piano roll notation.Other works focus on different dynamic roles of instruments in ensembles [SML*16, WZBKB17].Schedl et al. [SML*16] show the current position and involved instruments in an orchestra score with additional structural elements of the performance.Wu et al. [WZBKB17] visualize similar aspects, but for the distinct

Figure 7 :
Figure 7: Music games utilizing visualization as enhancing enjoyment.Audiosurf [Fit08] (left), which visualize notes as obstacles in different colours and Rock Band (right) [Har07b] where scores are represented as coloured tiles.
[CWJC08].Zhang et al.  [ZHJ*10] created a system to derive the mood of music from its music video content through image analysis.They align music videos with extracted moods in a two-dimensional 'affective space'.Next to images and videos, features of sound can be communicated turning the 'shape of music' into visual shapes transmitting the emotion of music [Got96, LL04, SY09, HXF*10].While Sauer et al.[SY09] use avatars, performing a Celtic dance (no arm movement), Goto et al.[Got96] showed more abstract human-like dancers.Both map tempo, beat position or dynamic to movements, and users can select dance performance parts to create a whole choreography.A screenshot of such a performance is shown in Figure10.In contrast, Haro et al. [HXF*10] do not show the shape of a song, but the shape of a user's musical taste.Therefore, around 60 audio features are used to generate a (static) avatar, depicting tastes stereotypical by mapping to background, head (eyes, mouth, hair, hat), suit or instrument.Levin et al.[LL04] mainly focus on the question 'If we could see our speech, how might it look like?'They take input from live performances (speech, sound and song) and generate real-time visualizations, inspecting features including pitch, spectral content and autocorrelation data.The resulting visualizations, which are described as 'consensual hallucination', are presented in an augmented reality environment, accessible for multiple users at once.To help hearing-impaired persons to develop a feeling for the music, Fourney et al.[FF09] offer a Music Animation Machine that includes a piano roll view (see Figure11 (left)) and a mood view (see Figure11 (right)).The latter one displays notes as circles and encodes note length by circle size.When playing back sound, the core of a circle moves to the next notes and disappears after some time to communicate a feeling for the music for example, speed or fading of a tone.Task: Sound similarity analysis.Repetitions are key elements in musical compositions that are often found within the structure of sound.The development of methods to measure similarities plays a central role in the field of MIR [AP*02].Foote [Foo99] used a two-dimensional matrix visualization to show acoustic similarities in the same piece of music, allowing to investigate structural and rhythmic characteristics.This leads to repeated or modified themes being recognizable.The resultant visual fingerprints of sound structure can be used to derive knowledge on how similar other musical pieces are.

Figure 8 :
Figure 8: The stacked timelines represent different instrument players' performances.On the left, different audience votes for predefined modes are shown.These modes define if and how the instrumentalist shall perform and play notes [WZBKB17].

Figure 9 :
Figure 9: An excerpt of the Drumix tool, which shows the distribution of the drum patterns through a geographical metaphor based on a SOM [YGK*07].The drums (snare vs. bass drum) are listed in their temporal progression and can be rearranged.

Figure 10 :
Figure 10: Goto's virtual dancers whose motions and positions change to musical beats in real time [Got96].
Subtask: Focusing on sound features.Similarity of songs can also be determined by low-level sound features like spectrum, amplitude, metronome or beat points [TC01, TEC01, BFTC02, LE07, PEP*11].Kolhoff et al. [KPL06] compute bloom-like shapes whose form and colour represent such features.In contrast, Muelder et al. [MPM10] derive similarities by analysing audio frequencies, and the musical collection is displayed in a graph layout that accentuates those similarities.Likewise, Leitich et al. [LT07] use frequency characteristics of audio signals to project icons representing songs of a music library on a globe.A multifaceted view for this is provided by the 'Sonic Browser' that displays sound as coloured shapes in scatterplots, treemaps and graphs [BF03].It maps file size to the size of visual symbols, file types to symbol shape and sampling rates to colour.
. Zhang et al.[ZL17] arrange the music history of a user on a circular timeline.When selecting individual songs, connections to related songs are shown for music recommendation purposes.

Figure 11 :
Figure 11: The Music Animation Machine [FF09, MT11].The piano roll visualization (left), where coloured bars represent the notes and the Part Motion (right), which uses coloured circles showing the same information.The size of the circle encodes the note's length.

Figure 12 :Figure 13 :
Figure 12: An excerpt of the super-genres Musicmap created through a top-down approach [CC18].
Figure 14: PartyVote's Interface, allowing attendances of social events to vote democratically on the next selected songs [SWT08].

Figure 15 :
Figure 15: The heatmap visualization of Baur et al. [BSSB10] show patterns in listing history of users.Listening to the same song and song sequences is highlighted through red lines.

Figure 16 :
Figure 16: Similarity Graph for Ono et al.'s [OCF*15] musical collection exploration tool.Songs are presented as rectangles and similar music segments in different songs are connected.

Figure 17 :
Figure 17: Miller et al.'s network graph 'Linked Jazz'[MWP12] uses the musician images for quick recognition.The linkages between the nodes encoding relationships between the jazz musicians, making it possible to derive further information about the social network, the interaction between the artists and influences.The size of images encodes the impact on the community through the number of relationships.

Figure 18 :
Figure 18: The Yahoo! metadata graph of Gleich et al. [GZL05].Showing the similarity of artists (nodes) based on positive user ratings (likes).The edges are alpha-blended to highlight local density and connect each artist with their 20 nearest neighbours.The colours represent different clusters based on the artist and the user rating data.

Figure 19 :
Figure 19: History of Rock showing connections between Rock bands, their temporal developments and classification of their music [LA18].Selected are The Beatles.

Figure 20 :
Figure 20: A coloured three-dimensional visualization of a violin scanned by computed tomography.The colour transfer function highlights glued areas (bottom) [Hop18].

Figure 21 :
Figure 21: Wood thicknesses distribution of a top plate from an asymmetric violin (top) in a heat map-like two-dimensional visualization [Hel17].

Figure 22 :
Figure 22: Konopka et al. examine long-term deformations of wooden instruments by experimenting with real instruments and a computed simulation [KSKE17].The image shows the simulation results of a replicate of a clavichord in three states, where the heat map-like colour scale indicates the level of deformation.

Figure 23 :
Figure23: Extract of Kusnick's Masterthesis[Kus18] showing a volume rendering of a hurdy-gurdy in an augmented reality context.The printed marker on the table is detected by the tablet's camera as a reference point for the placement of the virtual object.

Figure 24 :
Figure 24: Berthaut el al.'s 'Rouages' visualize effects and their interplay of different buttons and controller from digital music instruments and virtual extensions[BMSH13].The controller settings are symbolized with different three-dimensional shapes.

Figure 25 :
Figure 25: Visual encodings of imprecise datings [JFS16]: Small horizontal lines − are used for precise datings, circles • for around datings, triangles for before datings and upside down triangles for after datings.
, similar as shown by Ono et al. and De Prisco et al. [OCF*15, DPLM*16].The usage of score information requires a stronger text analysis as shown in [JGBS14].

Figure 26 :
Figure 26: Comparison of temporal and geospatial information on production events of different clavichords[KKFJ20a].Each colour represents a type of clavichord.

Table 1 :
Publication realms of considered related works.Notable realms are the ACM International Conference on Multimedia and IEEE Transactions on Multimedia.Computer music research is cumulated under the tag Digital Audio (13)- with The International Computer Music Conference (ICMC) as a representative realm-providing related works using interactive visual exploration tools for sound analysis.The group of Miscellaneous related works (9) includes further origins, four related master and doctoral theses that we took into consideration.In addition to scientific publications, we considered interactive visualization approaches offered on Websites (15) or as Commercial Software (8).

Table 2 :
Classification of visualization references by firstly their data type and secondly their main use.
. McLeod et al. [MW03] and Yang et al. [YLL*07] communicate low-level musical features like frequencies during live performances using rather simple visualizations.While the former target beginners, the latter try to enable the group of hearing-impaired people to learn to perform a musical instrument.Subtask: Manipulation.Lastly, visualizations can support manipulating performance data.Such signal editing tasks are either applied to live performance data [YGK*07] or to audio files [Dix01, YGK*07].For this purpose, an interactive graphical representation of sound signals is offered, allowing a user to interactively add or delete specific sound patterns.Both works use drum and rhythm patterns for manipulation and represent the result visually and acoustically.For example, Yoshii et al. [YGK*07] visualize a self-organizing map (SOM) that shows the distribution of the different drum patterns throughout the song (see Figure al. [JF17] and Yim and Bartram [YSB09] both use a horizontal time axis and apply a one-dimensional force-directed placement strategy to place nodes vertically.Next to individually listed relationships between musicians, social networks can further be derived when institutions that employed musicians are taken into account [AGC*17, KJKF19].Like the latter two approaches, Ables et al. [AGC*17] combine timeline and graph in order to visualize the careers of Venetian musicians in the late 17th century.Khulusi et al.