David Kut'ák

Pere-Pau Vázquez

Tobias Isenberg

Michael Kron

Marc Baaden

Jan Byška

Barbora Kozlíková

Haichao Miao

State of the Art of Molecular Visualization in Immersive Virtual Environments

Keywords: MB21, RFK*21, OBD CCS Concepts, Computing methodologies → Virtual reality, • Human-centered computing → Scientific visualization, • Applied computing → Molecular structural biology

: For this state-of-the-art report, we surveyed the literature for papers focusing on molecular visualization in immersive environments. Many of them explore educational use cases ( ) or collaborative environments ( ). We also report on various enabling technologies, such as head-mounted displays ( ) or augmented/mixed reality ( ). Furthermore, we also report on papers tackling interaction techniques ( ) or providing solutions over the web ( ).

Introduction

The significant benefits of virtual environments for the visualization of scientific data have been established decades ago [START_REF] Bryson | Virtual reality in scientific visualization[END_REF] Haa96; vDFL*00; LSSB12; MGK *13]. Yet only relatively recent technological advancements-partially driven by the video game industry [LTD*13]-have led to an increase in the widespread availability of affordable immersive hardware, especially in the form of head-mounted displays (HMDs) [START_REF] Morrison | Will chemists tilt their heads for virtual reality?[END_REF]. This development, in turn, has led to the establishment of the subfield of Immersive Analytics [DMI*18; FP21] within visualization research. As part of this work, research has been carried out and tools have been developed that rely on virtual or augmented reality-usually summarized under the umbrella term mixed reality, occasionally also called extended reality or the "metaverse" [LBZ*21]-that are now available outside of visualization research labs and have become a feasible and promising way of data exploration and/or analysis [START_REF] Millais | Exploring data in virtual reality: Comparisons with 2D data visualizations[END_REF] for many labs in the domain sciences. Furthermore, the omnipresence of small mobile devices, such as smartphones, with both a display and a camera now allows virtually everyone to experience some form of mixed reality-even if only with a monoscopic display-, for instance for educational purposes. Due to the importance and complexity of the three-dimensional structure of molecules, molecular visualization started to adopt immersive technologies already decades ago [START_REF] Akkiraju | Viewing geometric protein structures from inside a CAVE[END_REF]. Besides providing access to the intricacies of molecular shapes, a mixed reality approach also has the considerable benefit that it can show relevant non-surface data that affects many molecular interactions (e. g., electrical fields or charges), and it inherently facilitates collaborative data exploration and analysis scenarios [SWB *22].

Researchers have proposed many techniques, setups, and application cases to use immersive technology for the visualization of molecular data. Several surveys in the past have covered research on genome [START_REF] Nusrat | Tasks, techniques, and tools for genomic data visualization[END_REF] or molecular visualization [KKL*15; KKF *17] in general, on immersive environments for (bio-)medical visualization [VBB*04; VMR*21; FFV22], or for visualization in general [vDFL*00; vDLS02; OOKO15; DMI*18; FP21]. In our overview, however, we focus on the analysis of the state of the art of molecular visualization in mixed reality. Our specific focus is the use of modern head-mounted platforms, but we also discuss relevant work in the related areas, such as "general" VR in form of CAVEs [CSD*92] and stereo rendering, as well as hand-held augmented reality. The research overviews most closely related to our own are those by Goddard et al. [GBS*18] and Calvelo et al. [START_REF] Calvelo | An immersive journey to the molecular structure of SARS-CoV-2: Virtual reality in COVID-19[END_REF]. In contrast to Goddard et al.'s survey, we examine the research landscape more broadly and without a particular own set of approaches as a comparison. Compared to Calvelo et al., we do not focus only on one specific application case or easily accessible software-we cover the field of immersive molecular visualization as a whole, considering wider range of challenges and future opportunities. Furthermore, we take a visualization-centric perspective on the existing work, while Calvelo et al. primarily provide an application-centric perspective, i.e., an overview of the available VR tools that can be used to view molecules in relation to COVID-19. Recent work also suggests advantages of immersive hardware specifically for molecular visualization, such as for perceiving the volume of a pocket [CŠR *20], for seeing data aspects previously unnoticed [GBS *18], or for engaging students in education [START_REF] Brown | Visualizing molecular structures and shapes: A comparison of virtual reality, computer simulation, and traditional modeling[END_REF]. In our survey, we thus focus also particularly on these benefits and analyze specific application scenarios of immersive visualization approaches. Finally, we classify all approaches that we review in this report based on their application domain and supported tasks, noting that much past work focused on applications in education, often created by researchers within the bio-molecular research fields, instead of originating from visualization researchers (see Figure 1).

Background

We start our discussion by introducing the necessary knowledge, background, and terminology for understanding this report.

Definition of Immersive Environments

As we later need to distinguish between different aspects of virtual environments, we first clarify the respective terms. In the literature, there is no clear consensus about how to differentiate between extended reality (XR), augmented reality (AR), mixed reality (MR), and virtual reality (VR) and the terms are, consequently, not always used consistently. To avoid misunderstandings, we stick to the terminology established in the scientific literature, which follows the classical definition of Milgram and Kishino [START_REF] Milgram | A taxonomy of mixed reality visual displays[END_REF], who proposed to use the term MR for the whole "virtuality continuum" between AR and VR:

Virtual Reality (VR) puts the user into a purely virtual, fully immersive, simulated environment, which shuts them off from the real environment.

Augmented Reality (AR) combines digital information with the real world (either by embedding it into a live video stream or using see-through displays). The digital information can be only overlaid on top of the real environment or the virtual content can interact with real-world objects. The latter requires AR applications to maintain a detailed, three-dimensional model of the real world. AR is sometimes also called hybrid reality.

Mixed Reality (MR) is an umbrella term that encompasses the whole spectrum ranging from AR to VR. It stands for a digital, simulated world that can be merged with the real world to create an immersive, interactive experience.

Recently, the term Extended Reality (XR), which was not used by Milgram and Kishino, has emerged and is nowadays often used as an alternative umbrella term encompassing the whole spectrum from AR to VR. That is, XR can be considered to be equivalent to Milgram's and Kishino's definition of MR. This is partially to avoid confusion after Microsoft's introduction of the Windows Mixed Reality ecosystem, which adopts the term MR for this specific context. However, there is also no clear consensus on the term XR, as it is sometimes used synonymously with the abovementioned definition of AR. Therefore, we decided to follow Milgram and Kinshinos's terminology and use the terms MR and immersive environments [DMI*18; FP21; LBZ *21] throughout this work to describe the settings on which we focus. We use these two terms interchangeably and mean them to encompass both AR and VR environments, i.e., the whole XR spectrum, as defined above.

Molecular Visualization

Molecular Visualization or Molecular Graphics-more specifically, the graphical depiction of three-dimensional molecular structures-has been a very active area of research for more than five decades [START_REF] Levinthal | Molecular model-building by computer[END_REF]. Therefore, a plethora of molecular representations or models has been proposed in that time [KKF *17]. On the one hand, this is since molecular structures are available on different scales, ranging from all-atom or even quantum mechanics data to coarse-grained data, where whole molecular complexes are represented as single data points. On the other hand, especially for atomistic data, there are many representations that visualize different properties of the molecular structure. For example, the ball-and-stick model shows the bonds between atoms, molecular surface models highlight the interface between the molecule and its environment, and there are specific abstractions like the common twisted-ladder representation commonly used for DNA. Due to advances in data acquisition and simulation methods, data set sizes are continuously growing, which poses a challenge for interactive molecular visualization and drives the development of new, accelerated rendering methods. Time-dependent data that capture a molecular process and ensembles of data sets further add to the complexity. That is, suitable molecular representations used in interactive molecular visualization have to be chosen based on the data, the desired task, and the available processing power. For more details on these various aspects we refer the reader to Kozlíková Especially the widespread availability of capable graphics hardware on almost all modern commodity computing devices-ranging from smartphones and tablets to desktop PCs-has led to considerable advances in interactive molecular visualization, not only concerning the visual quality but also regarding the size of the structures that can be displayed [ Generally speaking, molecular visualization uses complex 3D models to represent rather abstract data that is inaccessible to a human observer in the physical world. Making these depicted structures easy to understand is thus an important task. Much research has focused on advanced rendering and illumination to improve perception [START_REF] Skånberg | Real-time molecular visualization supporting diffuse interreflections and ambient occlusion[END_REF] or on making tangible models. The latter, however, is not feasible for dynamic data and the use of this physicalization is mostly limited to educational scenarios [START_REF] Gardner | 3D printing of molecular models[END_REF]. To make such complex data and the spatial relations between entities easier to understand, the use of stereoscopic displays has a long tradition in molecular visualization, going back to the very early VR HMDs [START_REF] Ihlenfeldt | Virtual reality in chemistry[END_REF] or the original CAVE [CSD*92]. It is thus not surprising that molecular visualization nowadays often relies on the increasingly available, affordable, and user-friendly consumer-grade stereoscopic HMDs [START_REF] Morrison | Will chemists tilt their heads for virtual reality?[END_REF]. As these devices have to process the scene separately for each eye, to achieve the stereoscopic effect and keep high refresh rates to avoid motion sickness, the computer graphics research focusing on improving the performance when rendering molecular data remains a vital endeavor along the visualization advancements [HL00; SLM*02; RKN*13; SSS16; MKH*18].

Hardware

Since different MR technologies require specific hardware, we discuss in the following sections the available display technologies for MR, associated interaction devices, and their advantages and shortcomings. Our goal is to provide the reader with the basic overview and terminology, whereas, for more in-depth tour, we would like to refer to some of the other existing reviews, such as those by

Consumer-grade Immersive Hardware

From the proposed technologies that have dominated the market for the past 50+ years, HMDs are the only form factor that seems suitable for the public. Consequently, although in the past some molecular visualization systems were developed for CAVE-like systems [HDS96; YPHS04], most of the research and development in visualization and interaction in MR environments is nowadays commonly developed using HMDs [HP17; GBS*18; MKH*18; RBDR18; GK19; MBE19; PTA *20].

Though the first HMDs date from the 1960s [START_REF] Sutherland | A head-mounted three dimensional display[END_REF], it was not until 2012 that the first successful consumer-oriented device, Oculus DK1, was put on the market. There have been many previous attempts (almost 150 devices developed until 2007 are listed by Bungert [Bun]), but Oculus' products really caught the attention of the general public: probably thanks to the improvements in some key aspects of the technology and the right content. The year 2012 was also important for AR headsets, particularly thanks to the Google Glass presented at Google's I/O event that year. It was released to the public in 2013, but discontinued shortly after and re-released again in 2017 as an enterprise-only product. Microsoft took over the hype with their HoloLens, an AR HMD released for developers in 2016. Together with its successor, the HoloLens 2, it is currently the most common AR device in research laboratories.

In general, there are three types of dedicated immersive MR devices, though the first entry in the list cannot be considered consumer-grade due to the investment required:

CAVE-like systems [CSD*92] that consist of an enclosed room,
where the user is surrounded by screens showing the virtual environment and can move freely within the room.

3D screens of different sizes and form factors [CPS*97; MRE13]

with users in front of the screen; e. g., PowerWalls or 3D TVs.

Head-mounted displays (HMD) [START_REF] Sutherland | A head-mounted three dimensional display[END_REF], where a head-worn display is placed right in front (and around) the user's eyes.

Besides these three types of specialized MR hardware, handheld devices with front-facing cameras, like smartphones or tablets, can be used for simple AR applications.

HMD Features

There are two main families of HMDs: the ones oriented towards VR experience that block out other visual stimuli from the environment, and AR headsets, intended to produce synthetic images on top of the real information. The former usually consist of a flat panel worn close to the eye, with a pair of lenses, one for each eye, to increase the field of view and achieve the proper focal length. AR HMDs usually feature a see-through display and are commonly built using a light engine and an optical combiner. The light engine is responsible for generating the synthetic information, while the optical combiner delivers the images to the eye, while also transmitting the environment light [XHH*21]. To track users' position in space, HMD devices use integrated cameras or external sensors, assisted by inertial measurement units, such as accelerometers and gyroscopes, integrated into the HMD.

As for the visual quality, two important aspects of HMDs are resolution and field of view (FOV). Early VR HMD devices had a low resolution, and even the Oculus DK1's resolution was only 1200×1080 pixels (resulting in 640×800 pixels per eye) with a FOV of 110 • . Wider FOVs improve the sense of immersion and have positive effects on distance judgment and motion sickness [LDP*02]. However, popular VR HMDs still are limited in this aspect (e.g., HTC Vive Pro 2's approximately 116 • ×96 • ). Though the resolution of commonly available devices also has not seen a major leap, there are products that are promising, such as the Pimax 8K, that already offers an exceptional resolution (3840×2160 per eye), and human-like FOV (≈ 160

Hardware-related Specifics of MR Devices

One of the often-mentioned key advantages of MR is the ability of the users to observe the complexity and internal relationships of the data. This is quite true for 3D, but in visualization, we commonly use a set of 3D and 2D views in so-called Multiple Coordinated View layouts. These configurations have only been studied in very few cases-and using older technology-for MR environments [MCH*18] and best practices for embodied interactions and embodied user interfaces are still not properly developed [START_REF] Gra Čanin | Immersion versus embodiment: Embodied cognition for immersive analytics in mixed reality environments[END_REF]. A second advantage of MR is that some aspects of the interaction, such as changing the viewpoint, are achieved with natural movements. The increased field of view is also important so that the application can fit more data, although, as mentioned, this can be challenged by the limited resolution of the displays. Furthermore, modern MR hardware often also enables quite intuitive two-handed manipulation, which increases the potential interaction abilities. Finally, HMDs equipped with eye tracking sensors have become recently available (e. g., HoloLens 2, HTC Vive Pro Eye, or Varjo XR-3) and other devices can be retrofitted with eye tracking kits. This setup facilitates novel interaction concepts and input mechanisms (see, e. g., Groß et al. [GBR*19]).

However, MR environments are not short of disadvantages either. In the rendering side, current untethered stand-alone hardware devices like HoloLens or Oculus/Meta Quest 2 are significantly less powerful than a desktop [MKH*18; MBE19]. This limits the complexity of the 3D scenes that can be dealt with (and the framerates that can be achieved). Wired devices, tethered to a PC exhibit better behavior (the PC is the one rendering the scenes), but the connections can be annoying for the users as it restricts free movement. Foremost, some tasks that are easy to achieve using mouse and keyboard, such as data introduction, are difficult to achieve through 3D menus and virtual keyboards or voice commands. Goddard et al. [GBS*18] state that "a virtual keyboard floating in the VR scene is feasible but tedious." Many modern HMDs like the HoloLens 2, the Meta Quest 2, or the HTC Vive Focus 3 offer hand tracking using multiple cameras integrated into the HMD. While this technology is convenient and user-friendly as it does not require additional hardware, the accuracy is still limited and the hands of the user have to be within the field of view of the cameras. Furthermore, no haptic feedback-that is, letting the user feel the interaction with virtual objects-is available, which lessens the user experience. To obtain a fully immersive experience (that is, reacting to users' senses), haptic feedback is desired. Operations such as palpably touching or grasping greatly improve the sense of embodiment. A possible solution is haptic gloves, which have been used already in early VR systems [AF98; KPL*04]. However, in addition to the inconvenience of wearing gloves while using the MR system, modern haptic gloves like the Noitom Hi5 or the Manus VR haptic that are commercially available are mostly not in a typical consumer price range. Therefore, especially in VR, the most prevalent interaction devices are tracked controllers tailored to gaming, which are usually included with consumer-grade VR HMDs. Although the only haptic signal provided by most of these common controllers is just a small vibration (vibrotactile feedback), it already helps to improve the sense of embodiment. MR, on the other hand, mixes physical objects with virtual ones and therefore allows for some degree of physicality [NPD *21]. Another important aspect is collaboration. AR devices allow users to communicate naturally because they can see each other. However, this is not true for VR HMDs, where only virtual representations-"avatars"-of the other participants are typically available. Though high-bandwidth connections may allow for remote collaboration setups, there are still numerous open challenges in this area. Furthermore, movements in the virtual world that are not aligned with the real world have to be designed with great care as otherwise motion sickness can occur [START_REF] Kaufeld | Chewing gum reduces visually induced motion sickness[END_REF].

Software-related Specifics of MR Devices

Software development for MR needs to take the sensor data and input signals described in Section 2.3 into account to deliver a seamless and sophisticated user experience. Usually, it needs to generate highly interactive 3D graphics for a stereoscopic display. The Software Development Kits (SDK) for modern HMDs usually include functionality for automatic tracking and localization, as well as stereo rendering. As an alternative, SDKs like Vuforia [PTC] offer image recognition to create AR applications (mainly for hand-held AR using smartphones or tablets). Developing an MR visualization application from scratch that handles all necessary input and output, however, is still a challenging task. Most visualization researchers and developers thus use the existing frameworks. Especially game engines often provide support for various HMDs. Unity [Tec] and Unreal [Gam] Engine are commonly used, because they provide instant functionality for developing applications with advanced interactions and optimized rendering. An often-cited advantage is also the large, powerful community, customer support, and asset stores. The game engines do not come without limitations. While they are easy to use and their scripting features encourage rapid prototyping, they offer less control over rendering and the general tool functions. Greater flexibility is possible at the cost of more technical effort if the prototype is built from less-restrictive frameworks, or even from scratch. In case of using existing engines, costs may also play a role. While Unreal Engine is free if no revenue is generated, Unity requires paid license for institutions whose annual revenue is more than $100,000. However, academic institutions can apply for a free education grant. Apart from these, free open-source game engine Godot also offers MR support. Finally, many web-based frameworks have MR support through the WebXR standard (e. g., three.js, A-Frame, or Babylon.js).

In summary, the technological stack for the development of molecular visualization MR applications can take various forms, and many approaches have been indeed used. As a by-product of our paper search procedure, we collected list of existing ready-to-use MR applications offering molecular visualization features. As this content is not the main focus of this report, we list these application in Supplementary Material S2.

Typology

After describing the necessary background and providing the readers with basics about the current status of MR hardware and software, we dive into the approach we used when compiling this publication.

Paper Collection Methodology

For our paper discovery process, we opted for a systematic procedure, combining database queries and manual searches, that went as follows:

1. We agreed on a list of keywords that describe the intended area of focus. These keywords were chosen from the three relevant areas: mixed reality, (molecular) biology, and visualization (see point 4 for the list of keywords colored by area). 2. Then, we collected a list of databases and search engines potentially containing relevant papers (listed also in point 4). 3. Based on the keywords, we assembled various search queries and tested their compatibility and outcomes on the selected databases. 4. Next, we performed several iterations of the first three steps. During these iterations, we filtered the keywords and databases, and optimized the query to provide more relevant results. In the end, we searched five databases (IEEE Xplore, Web of Science, ACM Digital Library, EG Digital Library, PubMed) and used the query ("virtual reality" OR "virtual-reality" OR "virtual environment" OR CVE OR HMD OR head-mounted OR "virtual collaborative environment" OR "mixed reality" OR "augmented reality" OR "visual analytics" OR "visual representation"). 5. We created a shared Zotero (www.zotero.org) library for the storage of all discovered publications. 6. We performed the first iteration over the query results to collect an initial set of papers. 7. Furthermore, after this step, we defined several categories which were used for the classification of papers based on their application area, used technology, publication type or other characteristics (see Supplementary Material S1). Assignment of categories to the individual papers was realized via corresponding Zotero tags.

8. After the first iteration, we decided to narrow down the search results to papers published in the year 2010 or later, as we focus primarily on recent technologies. We then went back through the selected databases and reviewed all returned results to ensure that we had included all relevant publications. 9. In the end, our Zotero library contained about 250 selected articles after this step. This also included publications that were not discovered by the query, but by manual search in the references of discovered papers or using other means. 10. We re-processed all the papers to assign the categories more precisely. During this step, we also filtered out irrelevant publications, as well as duplicates. 11. After narrowing down the paper count to 199 articles, we moved the Zotero library to a shared Google Docs table. This table allowed for more easy-to-use addition of textual comments. We divided the publications included in this table based on their core categories, which were used as a basis for the final classification of publications presented in this report. When processing the assigned papers, we further excluded the less relevant ones. In the end, we ended up with 127 publications, forming the core of the paper (see Table 1).

Review Structure

Our report consists of two main parts that arose from the categorization performed during the paper collection. In the first part, Application Domains (Section 4), we focus on areas where the presented publications are used. This part is further split into two distinct categories-Education (Section 4.1) and Research (Section 4.2)-dividing the approaches between learning-oriented and experts-focused ones. In the second part, Tasks (Section 5), we discuss papers from the perspective of the different goals that can be achieved using the presented approaches. We further divide this part into two self-explanatory sub-parts-Visualization (Section 5.1) and Collaboration (Section 5.2).

Application Domains

We begin our discussion of contributions by looking at domains to which the papers contribute most.

Education Use Cases

Among all application domains, education is a major application area for molecular visualization in immersive virtual environments, receiving a lot of attention from the researchers over the years.

Target Audience

The goal of educative applications is usually straightforward-to deepen the target audience's knowledge of the presented subject. This is the case in most approaches focused on students, being a prevalent target audience. Nevertheless, even the student-focused solutions can be further split into several categories based on the expected level of education. Molecular Zoo 18] in their paper on the mixed reality exploration of a cell, the attractiveness of the immersive virtual environments can also make it easier to engage the public. Thus, MR molecular visualization can target not only experts but also members of the public by providing a highly immersive introduction to the core concepts of molecular and structural biology. In consequence, this may open doors to an upcoming generation of biologists. , who provided strong evidence that the students considered VR as a helpful tool when trying to better understand specific structural aspects of the presented molecules. Furthermore, the two VR labs conducted over a duration of a single semester by Coan et al. also exhibited additional positive results. In the first place, the learning outcomes of the labs were met. Therefore, the positive aspects of employing VR were not at the expense of educational goals. Then, Coan et al. discovered that the students' responses to the VR were even more positive when they were introduced to the second VR lab. This observation suggests that the initial enthusiasm was not purely due to the novelty of the technology but due to its actual benefits and strengths.

An additional important-and often mentioned-outcome of user studies performed in the area of MR molecular visualization is increased user engagement: the participants generally seem to enjoy the ability to see and interact with the structures in the immersive environment [SJPG18; FCM*19; WMT*19; QCC21; SF21]. This can make lectures potentially more enjoyable. In addition, increasing the students' engagement may have positive effects on the overall learning outcomes and interest in the study subject [CKK06; developed an immersive virtual environment for the teaching of valence shell electron pair repulsion theory, a model for prediction of molecular geometry. In their user study, participants were split into two groups-one used the virtual environment while the other completed a traditional learning procedure-with the former group achieving better learning outcomes. Finally, an interesting case for employment of MR devices in the education scenario was presented by Lu et al. [START_REF] Lu | Designing and implementing VR 2 E 2 C, a virtual reality remote education for experimental chemistry system[END_REF], proposing a virtual environment-focused on students with limited mobility-allowing them to perform chemical laboratory experiments by remotely controlling a specialized robot.

Open Questions

Despite the achievements mentioned above, there remain challenges and open questions with respect to the use of MR molecular visualization in education. Brown et al. [START_REF] Brown | Visualizing molecular structures and shapes: A comparison of virtual reality, computer simulation, and traditional modeling[END_REF], e. g., compared virtual reality to traditional modeling and computer simulation, with students performing tasks such as the building of small molecules. Despite the researchers' expectations, the study outcomes did not show any significant differences between the performance of students using the three methods. Brown et al. conclude that VR might be more suitable for complex processes than those presented in the study-therefore, this technology could not fully utilize its strengths in this case. Contrary to their results, in the studies performed by Brůža et al. [START_REF] Br Ůža | VRdeo: Creating engaging educational material for asynchronous studentteacher exchange using virtual reality[END_REF] it was observed that the students did not sufficiently consider the 3D structural aspects of the presented enzymes, despite the fact they are naturally available thanks to the immersive visualization, and instead focused mainly on color-coded electron density map when performing the given tasks. Patterson et al. [START_REF] Patterson | Molecular genomics education through gamified cell exploration in virtual reality[END_REF] faced additional MR challenges in their gamified cell exploration. They discovered during the design process that the immersive environment could lead to an overly high cognitive load since it places the user in an unknown environment, simultaneously offering multiple points of interest. Therefore, the depth of scientific information and structural detail should be adequately tailored.

Educational Goals

Educational approaches can be further divided based on the educational goal. In some cases, the focus is on broadening the understanding of structural aspects of molecules-focusing on their overall shape, symmetries, or other spatial characteristics [NSM*12; BJ14; CŠR*20; LT20; PTA*20; SWL *20]. Therefore, the user mostly perceives and explores the presented structure as is. On the contrary, in other applications the student is expected to be more involved with the visualized content by designing new parts or interacting with the data to better understand their interactions and dynamic be- *20]. In this application, students try to predict the geometry of the given molecule, while being guided by the real-time simulation of atomic forces. Simulation-backed approach is also chosen by the authors of MOF-VR [START_REF] Von | MOF-VR: A Virtual Reality Program for Performing and Visualizing Immersive Molecular Dynamics Simulations of Guest Molecules in Metal-Organic Frameworks[END_REF], an application focused on construction, simulation, and visualization of metal-organic frameworks. In VR application Peppy [DDG*20] (see Figure 2), the students modify various properties of amino acid residues, such as torsion angles and strength of hydrogen bonds, while also being able to mutate residues, to better understand the effects of these changes on the final polypeptide structure. The effects themselves are computed by underlying molecular simulation approach to provide as real behavior as possible. While Peppy is developed primarily for VR, it also works on a desktop without any headset or stereo vision. This is an interesting addition, allowing one to experience the application, albeit less immersive, also in cases when the desired hardware is not readily available. This may be especially useful for students who want to examine the content individually outside of school.

Gamification

A special category can be defined by approaches that build on the concept of gamification. Despite the proven educational benefits of this technique [START_REF] Bai | Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts[END_REF], however, the number of applications using this approach in combination with MR molecular visualization is rather low. One of the most recent examples of gamification seems to be Pepblock Builder VR by Yallapragada et al. [YXW*21]. This application teaches the concepts of protein design, while employing a post-apocalyptic narrative and LEGO-style concepts to make the educative aspects more approachable. Another recent gamification approach has been presented by Patterson et al. [START_REF] Patterson | Molecular genomics education through gamified cell exploration in virtual reality[END_REF] focusing on cell exploration. In this case, the gamification aspects are influencing both the visual side of the application and the interactions. For example, the students board a virtual platform onto which they descend into the cell nucleus. In the nucleus, they can perform various interactions, such as grabbing an RNA polymerase molecule and placing it onto the DNA strand to initiate transcription. A focus on cell exploration is present also in the work of Johnston et al. [JRA*18]. The colorful art style chosen for the visual [LLZC], who proposed an application providing a steering wheel support, enabling to use it to control a virtual racing car driving along an HIV secondary structure, for example.

Technologies

Used technology is an additional factor distinguishing the educational approaches. In this area, current research covers both virtual reality [dSNGF17; SJPG18; FCM*19; BBMK21; MB21] and augmented reality [NSM*12; SW19; AD20; PTA*20; FCS21]. The chosen technology and hardware play an important role for educational institutions, being an environment that often works with more limited resources and less flexibility than individuals or small laboratories. The choice of technology thus does not only influence the students' experience but also additional factors-ease-of-use, portability of the application (Section 2.3.2), the maximum number of students using it simultaneously (Section 2.4), as well as overall cost. In this regard, web-based solutions appear as a promising approach, to reduce some burden [NSM*12; Abr20; CŠR*20; RFK*21; FCS21]. As demonstrated by MoleculARweb platform [RFK*21] (see Figure 3), web applications can offer a wide range of readily available study content, being a huge advantage for education. In any case, the accessibility of the chosen technology also plays an important role in the content creation process. While most applications provide content prepared by the developers [SJPG18; FCM*19; SWL*20], some researchers also focus on means for customized content creation, aimed at teachers [CGY20; BBMK21; RFK*21; CDA22; BF17] or students [START_REF] Argüello | Fast, simple, student generated augmented reality approach for protein visualization in the classroom and home study[END_REF]. Some researchers thus published guides describing a procedure for creation of MR molecular content [GJK19; Abr20; ENP20]. Eriksen et al. [START_REF] Eriksen | Visualizing 3D molecular structures using an augmented reality app[END_REF], e. g., present a step-by-step guide on the development of a molecular AR visualization application and its deployment to the Google Play store, all without requiring any programming skills.

Rendering Techniques

Since molecular visualization is inherently coupled with computer graphics, educational approaches can be further distinguished based on their approach to the rendering of structures. In some applications, the visualized structures are pre-generated in some existing molecular visualization tool and then imported as a common three-

Research Use Cases

Apart from education, another important domain is biology research, offering a variety of focus topics. For example, the study of the constituents and behavior of molecules is of great importance in fields such as drug design, analysis of molecular dynamics simulations, and structural analysis.

Common Activities in Use Cases

Based on our analysis of the most common activities performed on molecular structures, we found four recurring basic activities that a user performs in the virtual environment. The first activity, molecular exploration, typically focuses on the perception of a larger-scale molecular data [XLX*21; CŠR *20]. For the analysis of molecular data, being a second activity, we note that it often involves experts who, with specific domain goals in mind, take advantage of immersion and the natural interactions of MR to analyze structure and dynamics [GBS*18; MB21; RBDR18]. The In the following subsections we largely omit the respective use cases in education that relate to these four activities, since we have already discussed them in detail in Section 4.1. Many ideas covered by these educational applications could give rise to interesting use cases. Surprisingly, this does not seem to have happened yet, as the academic and industrial use cases described in the literature so far only reflect a limited subset of features compared to educational applications.

Intrinsic and Non 3D-intrinsic Data

Given the intuitive spatial perception in an immersive environment and the inherent 3D structure of molecules, traditional use cases in the molecular domain can be directly transferred from desktop-based approaches. Therefore the three-dimensional spatial awareness that the immersive devices intuitively add to any molecular structure dataset naturally leads to use cases that explore these spatial features and are very often cited as a major advantage. Natural candidates that rely on intuitive perception of spatial structures include interactive drug design [AW99; NGEB15], molecular dynamics simulation analysis [AF98; NMB*16; GK19; DWH*20], and protein structure analysis [RBDR18; XLX*21]. Interestingly, several works use the added dimension in the immersive environment and the natural way of interacting to visualize molecular data to add elements that are not intrinsically 3D as depicted in Figure 4. The added dimension in virtual space is used to lay out the data, which can be directly related to the object under study or to generally useful information. Todd and Emsley [START_REF] Todd | Development and assessment of CootVR, a virtual reality computer program for model building[END_REF], e. g., allow users to place a widget into the scene to show the current time. Similarly, research data that is not naturally mapped to three dimensions can also be enhanced. Probst and Raymond [START_REF] Probst | Exploring DrugBank in virtual reality chemical space[END_REF], for instance, represent the interrelationships of a molecular data set in a chemical space as a cloud of points, in which each molecule is represented as a point when viewed from a distance. The molecular structure is only displayed when the user gets closer. As another example, for genome-wide association data, this approach could consist in representing allele frequency, p-value, and chromosomal position in cylindrical coordinates to arrange the data around the user. Relationships between biological elements, often expressed as networks [SAKW02; LKF*17; PMI*21], are visualized in 3D to allow the user to explore and analyze big data without visual occlusion. These are examples of data without spatial attributes, where the immersive environment is used only for layout purposes. However, there are also hybrids that lie between intrinsic and non-3D intrinsic data, such as Hi-C data that indirectly provide spatial information through 2D interaction frequency views [ZSW *15]. In this case, the 3D spatial positions must first be reconstructed and then visualized in an immersive environment to enhance the representation of chromosomal interactions and relationships, highlighting another advantage of using a 3D environment: the direct representation of positions in 3D instead of indirect 2D views.

Structural Analysis

For structural analysis tasks, the stereoscopic view provided by immersive devices is often cited as an advantage, allowing for identification of features in the dynamic 3D environment that would otherwise remain hidden in a 2D display. Moritz and Meyer [START_REF] Moritz | Interactive 3D protein structure visualization using virtual reality[END_REF] present a case study of structural exploration of pyruvate kinase where they also note that certain perspectives and representations in VR may hinder the recognition of some structural motives. On the other hand, VR provides clear advantages, one of which is described in [CŠR*20]: clipping of surfaces is more easily avoided due to fine camera position control and a wider field of view. The authors illustrate this observation by comparing a 2D ligand-protein view with a VR view in their ProteinVR tool [CŠR *20]. ProteinVR specifically addresses hypothesis generation for research purposes based on immersive visual analysis (in addition to classroom use) of modeling results from docking and molecular dynamics calculations. Such structural analysis may also include the calculation of specific properties of the molecular structure under study, such as the electrostatic potential, which can be visually analyzed in VR once the calculation is complete as described by Laureanti et al. [LBO*20]. Their VR version also interfaces with the Adaptive Poisson-Boltzmann Solver that can be configured in the VR environment without a command line interface by allowing users to prepare input files for the solver by selecting and grouping atoms. Similar concepts are used in ChimeraX in VR [GHM*18] that connects the VR environment to several analysis and modeling components. With ChimeraX, Goddard et al. provide a generic VR-enabled toolbox for structural work as well as a way to share such experiences in collaborative VR sessions for analysis that can be implemented in either ChimeraX or AltPDB [GBS *18]. The presentation and sharing of such molecular visualization experiences, particularly structural analysis, connects to the sharing process described by Martinez et al. [START_REF] Martinez | UnityMol prototype for FAIR sharing of molecular-visualization experiences: From pictures in the cloud to collaborative virtual reality exploration in immersive 3D environments[END_REF] following the principles of FAIR. Sharing also finds echo in approaches that allow users to create guided tours, as was proposed by Alharbi et al. [ASL*23]. Similarly, VRdeo provides the means to create interactive experiences by recording user actions that can be replayed [START_REF] Br Ůža | VRdeo: Creating engaging educational material for asynchronous studentteacher exchange using virtual reality[END_REF]. Instead of painting the molecular scene, their approach incorporates visibility management and a camera to create trips through two mesoscale biomolecular models. An important aspect for all such use cases is the consideration of hardware limitations. In the context of AR, such limitations for developing use cases are discussed by Müller et al. [MKH*18] for the HoloLens device. In [START_REF] Chan | Augmented reality in scientific visualization and communications: A new dawn of looking at antibody interactions[END_REF] it is argued that AR has been somewhat overlooked so far, but provides enticing opportunities for research as is illustrated with the use case of antibody interaction analysis. Details on visual tasks related to structural analysis use cases can be found in Section 5.1.5.

Interactive Drug Design

We have found that interactive drug design is one of the most prominent use cases described in the literature, going back decades [START_REF] Anderson | VRDD: Applying virtual reality visualization to protein docking and design[END_REF]. Unsurprisingly, the most common reasons for using VR are the added spatial awareness [START_REF] Norrby | Molecular Rift: Virtual reality for drug designers[END_REF] and the natural interactions that support particularly the modeling tasks [TLL*11]. In drug design, visually exploring the spatial conformation of molecules such as ligands and proteins [TLL*11; DWH*20] is essential, and the ability to select a molecule and view it from different angles while changing its representation is key. Drug development teams often bring together scientists from diverse backgrounds, for whom the intuitive VR approach opens a natural window for sharing insights about 3D molecular data, even for researchers who do not routinely work with the 3D shapes of molecules. An example is the ability to go inside molecules and look around, just like buildings, which is a completely unique way of looking at these objects [START_REF] Norrby | Molecular Rift: Virtual reality for drug designers[END_REF]. For drug development, and in particular docking, it is beneficial to manually adjust the molecules in VR [START_REF] Liu | Using virtual reality for drug discovery: A promising new outlet for novel leads[END_REF]. This approach, guided by the user's intuition, demonstrates that, in VR, manual adjustments are no longer limited the same way they are on the desktop. Automated approaches can thus be supported in VR by user interventions using 3D interactions that are otherwise difficult to achieve on the desktop [XLX*21; KSB *22]. A key challenge is that the manipulation combines two purposes: either customizing the representation or actually modifying the molecular structure through modeling. For example, one can change the position of parts of a molecule merely to better view the interior, but the same interaction could also be interpreted as a change in the actual spatial conformation.

Molecular Dynamics Simulation

Immersive manipulation allows the user to go beyond simply examining a fixed model [DWH*20; LFB22]. The idea, which is common in the literature, it to provide the user with a tool to quickly understand the response of a molecular system to a given user input. Such manipulations may be done by a single user or when multiple users coexist in the virtual environment. Narupa [OBD*19] allows such multi-user manipulation of MD simulations in real-time. The way in which the user interacts, e.g. using haptic and or visual feedback or not, is an important element. Koutek et al. [START_REF] Koutek | Virtual spring manipulators for particle steering in molecular dynamics on the ResponsiveWorkbench[END_REF] describe an immersive control environment for positioning particles for MD simulations in real-time. They use a spring manipulator that allows the user to navigate the particles through a molecular system and provide visual force feedback of physical plausibility. Because the interactive manipulation may cause changes to the molecular system, dynamic visual cues are also important. An example concerns encoding bond saturation when changing molecular conformation in VR and could be implemented in a ludic way using emojis [SWF *21]. Visual cues may also be adapted to specific types of data handled in a given MD simulation, as in Nakano et al. [NMB*16]'s use case of visualizing and analyzing electron transfer data from MD simulations in virtual reality. They use an HMD and describe how the immersive environment allows the user to walk through the data, and provide the user with means to identify trends in dynamic systems that might otherwise be missed. In a similar spirit of modeling data exploration, Salvadori et al. [SFP*16] present a toolbox and illustrate use cases for docking, molecular dynamics analysis, and exploration of modeling results at the QM and MM levels (see Figure 5). The authors explore both CAVE and HMD implementations. Ratamero et al. [START_REF] Ratamero | Touching proteins with virtual bare hands: Visualizing protein-drug complexes and their dynamics in self-made virtual reality using gaming hardware[END_REF] focus more specifically on the utility of HMDs for consumers and propose a system that allows users to interact with static displays but also to consider protein dynamics. They conclude that immersion helps inexperienced users analyze proteinligand interactions and understand the conformational changes in protein dynamics. NOMADVR [START_REF] García-Hernández | NOMAD VR: Multiplatform virtual reality viewer for chemistry simulations[END_REF] targets VR environments at different price points, including HMDs but also cardboard-type devices and is focused on materials systems. In another focus on materials' science, Hagita et al. [START_REF] Hagita | Study of commodity VR for computational material sciences[END_REF] describe molecular dynamics simulations of phase separation of ABA block copolymers and experimental observations of filler morphologies in rubber. They present a methodology and benchmark measurements to evaluate such experiments, which is a welcome step forward in assessing such experiences. Molecular dynamics often is perfomed subsequent to model building activities. MOF-VR [START_REF] Von | MOF-VR: A Virtual Reality Program for Performing and Visualizing Immersive Molecular Dynamics Simulations of Guest Molecules in Metal-Organic Frameworks[END_REF] specifically targets an integrated package that goes from model building to molecular dynamics of guest molecules in Metal-Organic Frameworks (MOFs). Model building use cases are described in more detail in the next section.

Modeling and Model Building Approaches

The key challenge in molecular modeling is to generate 3D models based on experimental data. The examples discussed in the literature include building atomic models from X-ray crystallography or electron cryo-microscopy (CootVR [START_REF] Todd | Development and assessment of CootVR, a virtual reality computer program for model building[END_REF]), modeling DNA shape from Hi-C data (CSynth [TTM*21]) or even directly manipulating atoms via hand-controlled scanning probe microscopy manipulation of single molecules [LGE *15]. The authors of CootVR [START_REF] Todd | Development and assessment of CootVR, a virtual reality computer program for model building[END_REF] report an order of magnitude speedup when modeling is performed in VR instead of using the desktop. Their work offers insight on the challenges of VR molecular modeling with 6 DOF controllers. First, the arms must rest on a surface, as the user can quickly become fatigued from the elevated arm position (i. e., gorilla arm such experiences[tb] ard in assessing syndrome). Second, modeling sessions should be less than 25 minutes to limit eye and neck strain. The biggest challenge of working in a virtual environment using HMDs is isolation: actions like drinking coffee, using other software, or even taking notes become very tedious as the headset must be removed each time. While authors of CSynth [TTM*21] focus primarily on the computational aspects of the modeling problem, they also remark that VR helps with the exploration of the 3D models during the modeling process and can be useful for education. VR visual control can also be used to enhance direct experimental manipulation of single molecules as described in [LGE *15]. The authors of this paper add 3D visual feedback relevant for scanning probe microscopy (SPM) by displaying the currently executed trajectory and the position of the SPM tip during manipulation in real-time. This information is combined with additional control data to assist the experimentalist.

The construction of novel biomolecular structures in VR, such as DNA, was demonstrated by Schkolne et al. [START_REF] Schkolne | Immersive design of DNA molecules with a tangible interface[END_REF] and Kut'ák et al. [KSB*22]. Modeling of DNA molecules is different from other biomolecular structures because it consists of a chain of repeating units that behave predictably. For DNA nanotechnology, an interesting observation in [KSB *22] is that the exact alignment of structures is difficult as the controller has to be held in 3D space. However, the authors deal with this accuracy problem by adding constraints to the modeling process that are dictated by biology, e. g., the user can only place the DNA at a specific spatial position with a given rotation, so it remains biologically relevant. Schkolne et al. [START_REF] Schkolne | Immersive design of DNA molecules with a tangible interface[END_REF] go beyond the software and propose peripherals (tongs, ray gun, handle) specifically designed to build DNA molecules. ations can be entered without peripherals but via gestures, such as selecting and placing molecular fragments [START_REF] Alsayegh | A novel virtual environment for molecular system design[END_REF]. Finally, while the main contribution of Abriata et al. [START_REF] Abriata | Building blocks for commodity augmented reality-based molecular visualization and modeling in web browsers[END_REF] is based on education, it also gives a perspective on interesting AR-based applications in modeling biological macromolecules. These examples indicate that some fine-tuning is needed to make 3D modeling tasks in VR really efficient, be it related to the specific nature of the molecules, to the interaction modalities or to the hardware used.

Illustration

Illustrative visualization of molecular structures for the desktop has recently gained popularity (e. g., [KAK*18; HMK*20; KIK*21]). Although generic 3D painting has been explored as a powerful method for creating art in VR, we have found only few approaches that focus on molecular illustration in VR, despite the positive benefits of this technique, for example for research dissemination. Cellpaint-VR [Cen12] is the 3D version of Gardner et al.'s Cell-PAINT [GAB*18], which allows non-artistic users to create molecular scenes. Their approach uses stereoscopic rendering and includes isomorphic interactions that mimic painting, erasing, spraying, and coloring. The approach by Johnston et al. [JRA*18] also starts from the cellular level to create a VR model based on experimental data that serves as a template, with the future idea of further populating this vivid model up to the molecular level. LifeBrush [START_REF] Davison | LifeBrush: Painting, and visualizing dense biomolecular environments[END_REF] allows users to paint molecular agents that can be arranged to form a molecular simulation for illustrative purposes. The authors use the controllers to enable various interactions, such as sculpting, path line and event trace visualizations, to create complex molecular scenes for educational purposes. While MR illustrative visualization has been explored by the visualization community to some degree, there are still many open questions for the generation of molecular illustrations, given their importance for education and science communication. We have found few examples that incorporate many biological models to facilitate the creation of static and dynamic illustrations. An exception to this rule is the work of Alharbi et al. [ASL*23], focusing on VR guided-tours in dense molecular environments. A fundamental question in this context is how much biological context is required to make complex molecular illustrations, i. e., to what extent does biological information need to be included, compared to an approach that uses general 3D modeling and painting? We believe that incorporating the repetitive nature of biomolecular structures, as done in the work described above, should be included in illustration-based approaches, even though they may also limit the flexibility of illustrators.

Genomics

We found several papers that focus on visualizing genomics data in VR. Genomics data is often large and has complex relationships that take advantage of the generous space in a virtual environment for visualization. Pirch [START_REF] Zhang | BioVR: A platform for virtual reality assisted biological data integration and visualization[END_REF], who connect sequence data and structural spaces using proof-of-concept software to integrate protein and nucleic acid data, using Gria2 and its gene product as an example. Each amino acid or nucleotide of interest in the sequence is linked to the corresponding site in the protein structure. Although genomics is not intrinsically 3D, Stolk et al. [START_REF] Stolk | Mining the human genome using virtual reality[END_REF] demonstrate a creative use of 3D space in VR and report that bioinformaticians can identify new relationships between genes that may have otherwise remained hidden. Although most work focused on 3D data, the work described above demonstrated the potential of VR for non-3D data. We believe there is even more potential here. The key, however, is to find a good method for layouting or mapping to 3D in VR.

Potential For Visualization Research

Much of the work covered in the use cases we described addressed the possibility of visualizing and interacting with molecular data in immersive environments. These were mostly research prototypes, except for a few established molecular visualization systems [SSS16; GBS *18]. Many of the papers were published in the domain itself rather than in journals, such as IEEE TVCG. While many papers report the benefits of spatial perception and natural interactions, some also point to limitations, indicating the need for research to reach a pivot point where these specific use cases are primarily facilitated in an immersive environment. We believe that additional research should focus less on translating a desktop approach to VR and more on evaluating the effectiveness of specific visual encoding and interaction relating to the use case. A hybrid setup that combines desktop and immersive environments as it is being investigated for other application domains [WBR*20] may also be possible. Evaluation between modalities (e. g., VR vs. desktop) may not always be feasible due to the novelty of immersive devices or the lack of a comparable desktop tool, but some generic comparisons of specific aspects such as input and output devices [START_REF] Wang | Understanding differences between combinations of 2D and 3D input and output devices for 3D data visualization[END_REF] are possible. Evaluation within a modality through a quantifiable measurement of task completion time, however, could advance the field of molecular visualization in immersive environments.

Tasks

After having discussed education and research use cases, we now turn our attention to specific tasks supported by immersive tools.

Visualization Tasks

In particular, we have analyzed the papers regarding the basic visualization tasks needed for understanding any kind of data as described by Shneiderman [START_REF] Shneiderman | The eyes have it: A task by data type taxonomy for information visualizations[END_REF]: Overview, Zoom, Filter, Detailson-demand, Relate, History, and Extract. Because most of the papers we surveyed do not explicitly refer to these tasks in any way, we provide a summary based on our observations. In addition, we analyze how these traditional visualization tasks align with the immersive tools that often rely on isomorphic interactions [START_REF] Frutos-Pascual | Head mounted display interaction evaluation: manipulating virtual objects in augmented reality[END_REF].

Task 1: Overview

The vast majority of techniques described in the surveyed papers provide the readers with an overview of the data by showing the whole molecule in 3D using common spatial representations, such as molecular surface or ball-and-stick. The reason is that, in the immersive environment, zooming out is a rapid way to get an overview of the entire data by reducing the overall size of all molecules. Nevertheless, one could still argue that some representations are better suited for overview purposes (e. g., ribbon representation), while others give more details (e. g., ball-and-stick or van der Waals representations). Only a few surveyed papers provide representations tailored explicitly for overview purposes. Probst and Raymond [START_REF] Probst | Exploring DrugBank in virtual reality chemical space[END_REF] represent every molecule in the dataset as a point in a point cloud, and only if the users get closer, the actual spatial representation of a molecule is used. An overlayed minimap, such as in Eukaryo [START_REF] Yuen | Eukaryo: Virtual reality simulation of a cell[END_REF] (see Figure 6), can also aid users to understand the location and spatial relationships of molecular structures and often acts as a navigational device. In addition, derived quantitative measures often provide a good overview of specific structural properties. Numerous quantitative measures depicted as 2D plots (e. g., [ZPCC19; SFP*16; DDG*20]), however, are challenging to display in immersive environments and are only rarely used. Using semantics to link them to the underlying 3D objects seems to have great potential, but has been explored only little so far [TFF *18]. Such representations, therefore, remain a good opportunity for future research.

Task 2: Zoom

Depending on the type of MR devices, the users can use a pair of 6 DOF controllers, special gloves [KPL*04], or even fingers to perform gestures (e. g., moving the controllers closer to each other) that will zoom in or out on items of interest [LLZC; KSB*22; DDG*20; TE21]. The MR zoom in and out operations are equivalent to the desktop-based zoom operations, where users increase or decrease the size of the structure to see more or fewer details, respectively. The users are also often allowed to move the molecules closer to them (which makes them naturally larger due to the perspective camera). This operation is achieved by grabbing the molecules with controllers or using remote interactions via ray casting to select and pull distant objects. We can achieve a similar effect by letting users walk or move the device depicting the scene (e. g., cellphone or tablet) closer to the molecule of interest. Solutions such as teleporting within the VR scene are often used to cross larger distances and reach a more advantageous viewpoint. A unique solution was suggested by Patterson et al. [START_REF] Patterson | Molecular genomics education through gamified cell exploration in virtual reality[END_REF] who use a slowly moving platform (similar to an elevator) that carries the user through the scene. Zoom does not only provide a better view of details, but also accounts for the lower pointer accuracy of VR controllers that cannot be stabilized in the air by making the selection easier through more prominent visual elements [KSB *22]. Zooming into molecules can also be achieved using voice commands, as described by Goddard et al. [GBS*18].

Task 3: Filter

Tasks such as selection and filtering are inherently coupled with visualization due to the various structural properties that can be calculated from the atoms and bonds models. Nevertheless, in the papers we surveyed, especially in early approaches, we often only saw uses of VR/AR technology to display molecular data, without more advanced interaction techniques such as filtering or selecting data. Additional detail is often provided by labels that appear when users touch, hover over, or point towards a part of a molecule with a controller. A useful functionality is measuring distances and angles between selected atoms [ZW17; KBL*19; RB20]. Selecting the correct atoms of interest, however, may prove challenging due to the visual clutter caused by the complex molecules. Kazatzis [START_REF] Kazatzis | Molecular Visualization and 2D Interaction in Virtual Reality[END_REF] thus suggested an alternative context-aware selection technique based on selecting the relevant atoms using 2D directional input.

A related challenge is controlling the amount of actual detail shown to users to avoid cluttered views. We saw two main approaches to this challenge-either the control is left to the user or an appropriate representation is selected automatically. For manual approaches, users may be able to adjust the level of detail globally by moving a slider [START_REF] Yuen | Eukaryo: Virtual reality simulation of a cell[END_REF] or, as shown by Kut'ák et al. [KSB*22], use more complex lenses, allowing them to set a local level of detail. Some systems also handle the visual clutter by providing parts of the user interface (e. g., axes, view panels, interactive buttons, etc.) only when requested by the user [START_REF] Todd | Development and assessment of CootVR, a virtual reality computer program for model building[END_REF]. In automatic approaches, the distance from the camera is often used as the main criterion [START_REF] Probst | Exploring DrugBank in virtual reality chemical space[END_REF].

In desktop applications, details are often retrieved via brushing and linked views. But we have not yet seen a fully-fledged solution that would use a similar approach in immersive environments (at least not for molecular data). The most common examples include static 2D charts showing results of various computations [START_REF] Theart | Virtual reality assisted microscopy data visualization and colocalization analysis[END_REF]. More interactive solutions, however, are starting to appear as well [SFP *16]. A middle-ground option is to combine the mixed reality with a desktop-like interface in multi-window applications that often utilize web technologies [TFBB16; MMD*18; JTO*20]. While the spatial aspects are investigated using, e. g., head-mounted displays, the standard 2D desktop interface provides non-spatial views. By using semantic approaches, the details can be accessed with very little disruption [TFF *18].

An interesting take on a details-on-demand task was provided by Martinez et al. [START_REF] Martinez | UnityMol prototype for FAIR sharing of molecular-visualization experiences: From pictures in the cloud to collaborative virtual reality exploration in immersive 3D environments[END_REF] who consider immersive environments only as one of the modalities that can be used to explore data. In their FAIR sharing model, the overview may be provided with one modality, while details can be provided with another.

Task 5: Relate

The relate task pertains to the understanding the relationship between the visual elements in the virtual scene. These visual elements can either encode the same data, different data, or the relationship itself. Within the papers, we found that a quite commonly represented task was to relate various parts of datasets to each other. We found that there are four distinct ways how this visual relationship can be formed, which we use to structure our following discussion: colocation, spatial, dynamic, and explicit visual elements.

Colocation is used when visual elements appear at the same location and the user can switch between them. Because they appear at the same location the user will know that they refer to the same data. Tools such as UnityMol [LTD*13], ChimeraX [GHM*18], or custom visual representation designs [FKH*20; KSB*22; MB21] thus enable users to switch between different representations of a molecule or its constituents, depending on the derived molecular property of interest (e. g., all-atom representation vs. secondary structures). As the representations occupy the same space, it indicates to the user that the same data or its derivatives are depicted. However, in some cases, the elements do not have to align exactly on top of each other. This is useful when comparing multiple similar structures where the users are interested primarily in the differences. For example, Todd et al. [TTM*21] proposed a method where multiple results of DNA structure modeling can be shown and the corresponding areas are connected by semitransparent polygons. The technique immediately highlights regions with low and high variation (see Figure 7). Another special form of collocation is the possibility to share viewpoints in collaborative environments, as proposed by Chastine et al. [CYB*05]. We discuss this topic further in Section 5.2.

The spatial relation is used when, due to the proximity and geometric features of visual elements, users can derive interaction between these two visual elements. Particularly in molecular visualization the key challenge is understanding the spatial relationships between or within molecules, such as the distance of fragments or their geometric fit. This concerns two spatial properties: position and geometry of objects. In VR, users can relate these two properties by "grabbing" two objects and positioning them side by side. Two structural conformations after MD simulation, e. g., could be compared in The dynamic relation refers to the case when the same visual element appears and can be recognized at different time steps. It appears quite often since many of the papers we analyzed were concerned with dynamic data, such as the MD simulations we described in Section 4.2.5. Here, the task is often to set up the computation parameters [GJB*20; JTO*20] or understand actionreaction relationships when exploring the results of the MD simulation (e. g., [FZG11; HKM*19; DWB*20]). Among the physically inaccurate solutions we can list Molecular Zoo [GBS *18], which simulates interactions between molecules that are freely moving in the space around the user. The notion of random movement within the crowded molecular environments can be also achieved using particle effects [START_REF] Yuen | Eukaryo: Virtual reality simulation of a cell[END_REF]. Finally, users are sometimes encouraged to interact with the environment directly to influence the simulations of dynamic processes [OBD*19; PLAM19; vWGK22].

The most common method to encode relationships and properties is via explicit visual elements such as lines, highlights, and glyphs. While multiple coordinated views are employed on desktop applications, this task is slightly more challenging in an immersive environment as these views coexist in the same space, hence becoming spatial-visual objects themselves. The typical task using this approach is to explore the properties of individual atoms, e. g., their charges [LBO*20], by placing additional visual cues, such as lines depicting the electrostatic field next to them. Another variant of the same task is the exploration of bonds. For instance, Seritan et al. [SWF*21] use emoji placed on top of atoms to show the bond saturation (see Figure 8). Slightly more challenging are cases where there are one-to-many or many-to-many relationships between the spatial elements (e.g., While some solutions abstract from the spatial representation of molecules and use node-link diagrams to depict relationships [SAKW02; LKF*17; AGM*18; PMI*21], others only "flatten" the spatial representations and, while technically still being 3D, the representations appear two-dimensional [START_REF] Sharma | LearnDNA: An interactive VR application for learning DNA structure[END_REF]. For direct fully-fledged spatial representations we observed that, especially when using AR technology, multiple approaches depict possible interactions between molecules when a user places the molecules in proximity (e. g., by moving AR markers representing the molecules close to each other) [RFK*21; Abr20; ABT*20].

Task 6: History

The vast majority of papers we surveyed describe research prototypes that only rarely keep track of user actions. The few exceptions include the ability to reset the application to one of the few preset states, such as in the case of Rodríguez-Sotres et al. [RRG*09], or tracking the user actions for undo functionality that can be found in more complex tools. As for matured desktop applications such as ChimeraX [GBS*18] that got extended with VR viewing capabilities, undo/redo functions are already built-in. In our opinion, there is a clear opportunity to explore provenance visualization when it comes to immersive environments. The tracking of the user's head and controllers provides valuable information on their behavior, which can also be analyzed in experiments and used for further navigation as shown by Leinen et al. [LGE*15].

Task 7: Extract

Extraction of subsets to another file for sharing, further analysis, and presentation is one of the last tasks in visualization. None of the papers and techniques we surveyed explicitly discusses possibilities for extracting or exporting subsets of the data for further use and analysis. Although crucial for an effective workflow, these tasks are often not considered as a key paper contribution, hence are rarely implemented or described. Furthermore, many publications focus on demonstrating the system using prototypes rather than such saving/export features. Some systems that focus on the integration into the workflow of domain scientists provide export functionality, such as DNA sequence export into FASTA format after modeling in VR [KSB *22]. The purpose of exporting functions can be grouped into two categories. The first is to export data for further analysis in different tools. Defining these interfaces through data formats is essential, as VR often does not provide a full feature set but only deals with certain subsets, such as visualization and modeling. In this category, Martinez et al. Alternatively, the immersive environments can be used to enable the user to create biomolecular scenes for educational purposes (see Figure 9) and save and export the results [GAB*18; DSJ19; BBMK21]. While the extraction task is often neglected in research prototypes, we believe that more systems will need this built-in functionality as molecular visualization in MR environments matures, providing new research opportunities. While seemingly easy on the desktop, this task involves navigation through a file structure and text input, which is inherently challenging in VR.

Collaboration Tasks

Visualization plays an important role in the success of researchers' efforts. Similarly, the means of collaboration may significantly determine what can be accomplished. In this regard, MR interfaces are providing users with extensive options for collaborative work, namely when comparing them with desktop solutions. Especially in biology and biochemistry, where we face very complex phenomena, collaborative efforts and joining forces and experience of researchers worldwide are necessary for future progress in these fields. For that, MR represents an ideal environment and opens completely new possibilities.

In general, a meaningful collaboration in virtual environments can foster research, dissemination, and educational activities. Billinghurst et al. [START_REF] Billinghurst | Collaborative immersive analytics[END_REF] introduced and discussed the general concept of collaborative immersive analytics. They viewed the topic from different perspectives: spanning from the different types of possible collaboration to diverse users, their roles, and related interaction possibilities. Although they outline many insightful general aspects, in this section we address the topic of collaborative VR from the perspective of molecular visualization, where several very interesting approaches were presented throughout the years.

Asynchronous Collaboration

In 2009, Lee et al. [LQK*09] developed VR system for remote collaboration, with particular application in molecular docking and crystallography. They focused primarily on asynchronous collaboration, solving the problem of different time zones of users. One of their solutions is inspired by the version control concepts for collaborative work. The system records the session of the user as a set of files that are stored in the collaboration server and can be later downloaded, reviewed, and further enhanced. Asynchronous communication is supported also by the 3D-Lab web-based platform introduced by Grebner et al. [GNE*16], which aimed to provide the users with a collaborative and user-friendly interface for 3D modeling of molecules, with particular applications in conformer generation, ligand alignments, or molecular docking. The platform has a modular architecture and its ultimate goal was to promote interactions between drug designers. Another example of asynchronous communication was recently presented by Brůža et al. [START_REF] Br Ůža | VRdeo: Creating engaging educational material for asynchronous studentteacher exchange using virtual reality[END_REF].

Here, the tutor prepares a VR scene with educational content, which can be later entered and interactively explored by students.

Synchronous Collaboration

One of the earlier synchronous collaborative multi-view virtual environments for molecular visualization and modeling was proposed in 2005 by Chastine et al. [CYB*05]. They focused on one of the most challenging tasks in collaborative VR: how to handle situations when different users manipulate the same part of a molecular structure. In their case, each user determines an "area of interest" with a bounding box. Objects outside this area are de-emphasized, which also improves the clarity of the user's view. This area of interest is visible to the other users as well for location and focus awareness. They represented the user in the environment by 3D hand models and they experimented with the size of the models when the user is working. This concept supports the three main issues that must be addressed in such collaborative environments: awareness of the presence, attention awareness, and action awareness.

Synchronous communication between users was also addressed in research efforts by Corrêa et al. [START_REF] Francischetti-Corrêa | Immersive environment for molecular visualization to interaction between research groups geographically dispersed[END_REF], who targeted multiprojection and collaboration capabilities between geographically dispersed research groups. The interaction mode they implemented is based on handing the token of the active user, as only a single user at a given time can interact with the system. Other users are queuing requests for the control, and negotiations between users are realized through text and voice communication. Similarly, Corrêa et al. [START_REF] Francischetti-Corrêa | Molecular visualization with supports of interaction, immersion and collaboration among geographically-separated research groups[END_REF] focused on the integration of two or more geographically separated research groups working on the same molecular visualization. They decided to utilize one of the standard applications for molecular visualization, JViewer, operating in VR. *18] introduced MinOmics, a visualization framework dedicated to multi-omics interactive visual analysis. By utilizing UnityMol WebGL, it supports stereoscopic representations, and WebVR is used for integrating the framework into VR. They proposed four different scenarios for possible interaction between users. The first one utilizes the wall-sized display, showing a single monoscopic instance of the scene to all users in the room. Only one user can interact with such a setup. The second scenario combines the wall-sized display with VR, using a VR headset. In their arrangement, only a single user can use the VR setup, other users only see a restricted 2D view of the 3D scene the VR user is controlling. In a third scenario, they experimented with stereoscopic projection on wall-sized displays, while a fourth scenario finally offers a fully immersive experience using only VR. At the time of publication, however, this last setup was still restricted to only a single person.

Maes et al. [MMD

In 2018, Goddard et al. [GBS*18] released three VR applications: ChimeraX for analyzing molecular structures and electron and light microscopy data, AltPDB for collaborative discussions about atomic models, and Molecular Zoo for teaching young students characteristics of biomolecules. In ChimeraX, two or more users can join the same VR session. Each participant is represented by simple cones for hands and a postage stamp, where the user can upload a photo. Any user can then manipulate the molecular structure and point at features of interest, as a part of collaborative discussions. In AltPDB, the social VR site AltspaceVR handles the technical complexity of initiating the multi-person VR session, maintaining the synchronized views, audio connection, and customizable avatars. This environment as well as the users' avatars are more elaborated than in ChimeraX, leading to a richer overall user experience.

Probably one of the most elaborate VR applications to date for collaborative viewing, manipulating, and modifying chemical and macromolecular structures is Nanome [KBL*19]. Kingsley et al. present the architecture of the tool, three proposed collaborative modes, and a demonstration of its usage in designing a small molecular structure. The goal of Nanome is to minimize the need for the technical know-how, and thus reduce the communication barrier between structural biologists and other disciplines to enhance the idea flow and collaboration within drug-discovery teams. Nanome can host more than 10 users at the same time. A user is represented by an avatar with a head and hands (see Figure 10). Furthermore, the user can act in one of three possible user roles: virtual participant, 2D viewer, and ghost mode. The first role offers the full VR experience and the user is represented by an avatar in the scene. At a given time, only a single user (presenter) can manipulate the loaded structures, to avoid clashes with the other virtual participants. These can still move autonomously in space and freely explore the structures. In addition, the presenter can teleport all other avatars to his or her location, giving all participants the same view onto the scene. The 2D viewer is visible in the virtual scene as a video camera that is visible to other participants. Such users can freely move in the space and speak to other participants, but cannot become a presenter and are thus prevented from manipulating the scene. The ghost mode provides the users with a passive viewing experience from a fixed position and is intended for streaming the scene to a large audience. same VR space and interact with real-time molecular simulations. It is based on the iMD-VR software framework and extends its functionality by enabling a multi-person VR experience, setting up and customizing interactive user manipulation with MD simulations, and running the application on a local network. Via controlled studies in the laboratory, they demonstrated that Narupa and iMD-VR tools enable researchers to complete molecular modeling tasks more quickly than using conventional interfaces, such as mouse or touchscreen.

Gauthier et al. [START_REF] Gauthier | Dynamic virtual proteins: Visualization, interaction and collaboration in virtual reality[END_REF] developed Dynamic Virtual Proteins, a methodology of interaction design that combines human interactions and virtual agents that are assisting humans with the manipulation of a virtual protein. The methodology operates with three levels of user interface: (a) manual, (b) collaboration with a virtual agent, where some actions are taken by the agent, and (c) automatic, where all actions are performed by the agent. They are targeting mixed audiences-experts and non-experts-, operating in the shared space. They focus specifically on collaborative tasks from engineering and decision-making about new drugs and treatments.

Collaborative VR was recently thoroughly studied by Martinez and Baaden [START_REF] Martinez | UnityMol prototype for FAIR sharing of molecular-visualization experiences: From pictures in the cloud to collaborative virtual reality exploration in immersive 3D environments[END_REF]. They examined options for sharing curated visualizations of structural biology, modeling, and bioinformatics datasets for interactive and collaborative exploration. Their goal is to enable easy sharing of the visual experience with others. In their setup, each participant is represented by an avatar and has their point of view, separated from the others. However, participants may choose to group to adopt a similar viewing angle.

One of the latest approaches that was motivated by the communication issues caused due to the COVID-19 pandemic, is ProMVR by Xu et al. [START_REF] Xu | ProMVR -Protein multiplayer virtual reality tool[END_REF]. This tool aims to support the collaborative tasks of protein designers and provide them with a virtual environment containing the protein 3D model and intuitive interactions with peers. Although the work on this tool is still in progress, it already shows interesting results.

Collaboration in Augmented Reality

Augmented reality opens another perspective on collaboration inside complex molecular scenes. An indisputable benefit is that users can walk around the molecular model and choose their views without influencing others. Moreover, AR allows users to point out interesting locations in the molecule to each other naturally. Moreover, the communication between collaborators is natural, as there is no need for avatars. However, a downside is that such a solution can only support collaboration between co-located users.

Very recently, Noizet et al. [NPD*21] published their application framework for augmented 3D printing for molecular modeling. The molecule of interest is first 3D-printed, and this physical model is subsequently used as a support for the visual augmentation by the superimposition of additional visual representations. This is realized using a HoloLens. The results of their experimental testing showed that such a mode facilitates collaboration and dissemination.

Concluding Remarks on Collaborative Environments

Although the importance of communication and collaboration is evident and indisputable, only the recent pandemic showed us that the usage of collaborative virtual environments provides an excellent option in cases when the physical presence is not possible. The above-mentioned examples of already existing approaches and tools that support the collaborative aspects are first pioneers in these efforts. As a consequence of the recent dramatic events that influenced lives of all of us, we believe that this domain of building collaborative virtual environments will become one of the leading research fields in the near future.

Evaluating MR Research

As the domain of MR molecular visualization is still rather new, there are no long-term established procedures for the evaluation of the presented results. Nevertheless, as presented in Table 1, many of the authors use some kind of validation of their concepts. Five basic levels of evaluation seem to be currently employed in general, similarly to common non-MR visualization research [LBI*12; IIC*13]:

• No evaluation. Some authors completely omit evaluation or, at least, do not mention it in their publication. • Informal evaluation. In some cases, the evaluation is informal, i. e., it is mentioned that some people tried the presented work and shared their impressions but no specific procedure is presented. • Software-based evaluation. In some publications, the evaluation is performed by comparing the properties of achieved results (e. g., novel algorithms) with other existing solutions. • Qualitative evaluation. This type of evaluation usually involves smaller number of users, often experts in particular domain, providing often subjective feedback on the given work. • Quantitative evaluation. This type involves more participants to evaluate a given application, while the specific procedure is often designed to allow authors to statistically analyze results.

In the case of our collection of papers, the quantitative approach seems to be the most commonly used one. However, the choice is also affected by the domain and target audience of the application. For example, for applications aimed at education it is often natural choice to use quantitative evaluation with a group of students (as outlined in Section 4.1.1). On the other hand, researcher-focused approaches may be more suited for qualitative evaluation, as the Table 1: The list of surveyed papers with the dots indicating the topics covered by the given paper: education ( ), collaboration ( ), web applications ( ), HMDs ( ), AR/MR ( ), and interaction ( ). Furthermore, the table also shows if the given paper was evaluated and if so, in what way: no evaluation ( ), informal evaluation ( ), quantitative evaluation ( ), qualitative evaluation ( ), and software-based evaluation ( ).

Reference

Evaluation Reference Evaluation Reference Evaluation [Abr20] • • • [GHM*18] • • • • • [AF98] [GNE*16] • • [PMI*21] • [APM13] • [GBR*19] • [PR18] • [AW99] [HMO19] • [PB20] [AGM*18] • • [HP17] • • [QCC21] • • [AD20] • • [HL00] [RTH16] • [ABT*20] • • [JOB*20] • • [RB20] • • [BWE17] • • [JRA*18] • • [RBDR18] • [BRD*19] • • [JJS05] • • [RFK*15] • [BJ14] • • [JTO*20] • • [RKN*13] • [BF17] • • [Kaz20] • • [RRG*09] • [BM20] • [KPL*04] • • [RFK*21] • • • [BWH20] • [KBL*19] • • [RSH18] • • • [BBMK21] • • • [SD18] • • [SW19] • • [TW13] • [KHPB02] • • [SFP*16] • [CŠR*20] • • • [KSB*22] • • [SIS04] • [CPWG20] • [LNQ*19] • • [SWF*21] • • • [CYB*05] • • [LBO*20] • • [SLM*02] [CGY20] • [LT20] • • [SJPG18] • • [FSTdP10] • [LKKK11] • • • [CDA22] • • • • [LQK*09] • • [SBKW18] • [dCN17] • • [LGE*15] • • [SKE*20] • [DSJ19] [LLNW14] • • [SAKW02] [DWH*20] • • [LMKT05] • [SSS16] • [DWB*20] • [LKF*17] [SWL*20] • • [DDG*20] • [LXZ21] • [SCM18] • • [dSNGF17] • [LLZC] • • • [TLN17] • [DGC*15] • • • • [MMD*18] • • • • • • [DPSR98] [MTK*10] • • [TE21] • • [EDC*19] [MB21] • • • • [TFBB16] [ENP20] • • [MFVB15] • • [vWGK22] • • [FWA*21] • [MGK*13] • [WNM20] • • [FCS21] • • • [MM04] • [WMW*18] • [FCM*19] • • [Mor16] • • [WMT*19] • • • [FTG11] • [MKH*18] • • • • • • • [FKH*20] • [NMB*16] • [XLX*21] • • [GJB*20] • • • [NSM*12] • • • [YXW*21] • • • [GJK19] • • [NPD*21] • • • • [YPHS04] • [GK19] [NGEB15] • • • • • [GAB*18] [OBD*19] • • • [ZPCC19] • • [GMS19] • • • [ODD*18] • • [ZW17] • • [GSS*04] • • [PLAM19] • • [ZSW*15] [GBS*18]
• • • potential target audience is rather small and qualitative feedback may turn out to be more valuable.

Examples of well-thought quantitative evaluations can be found in works of O'Connor et al.[OBD*19; ODD*18] describing a series of studies measuring the performance of users throughout various tasks. Similarly, an interesting qualitative study was performed by the authors of StereoChem tool [START_REF] Swamy | Investigating the nature of students' reasoning in connecting molecular structures of stereoisomers with their physical properties using an AR app[END_REF], evaluated with a diverse set of users, having a different level of chemistry knowledge. This allowed for better examination of the reasoning process and, in consequence, also more precise identification of drawbacks of the chosen approach.

Conclusion and Future Challenges

For this state-of-the-art report, we systematically surveyed, analyzed and classified papers that report on molecular visualization in MR environments. We found, for example, that many developments in the field somehow relate to education. In addition, a particular feature of the papers we surveyed for this STAR is the fact that many research projects come from the domain experts themselves, rather than from the field of visualization. This observation shows that there is a dedicated need within the molecular science community for MR solutions. However, it also means that we need to communicate the research that has been carried out within the application domain to our visualization community. Furthermore, to keep this work up to date with developments in the field, we also created an online curated list of Molecular Visualization MR software, available through our GitHub repository † .

Currently, much research is carried out in domains that may pass † github.com/davous267/molecular-visualization-in-virtual -environments undetected by the visualization community, and both fields can crossfertilize each other. Moreover, molecular structures exhibit a high degree of complexity that can be better understood in specialized MR setups (e. g., the volume of pockets [CŠR*20]), requiring a better integration of molecular visualization with MR research. Also, as already said, the adoption of MR for molecular visualization is driven by the widespread adoption of MR devices at a consumer level, with the affordable hardware making it increasingly easy to be used in domain labs in practice.

While a large amount of research has been already done, we found that much of it targets the creation of prototypes for using MR for viewing and interacting with molecules for a specific use case, most commonly within drug design, structural analysis, and MD simulations. While this is a valid first step, future research should also focus on experiments to understand which interactions and representations are better suited for given tasks in MR. We envision that the established paradigms, rooted in mouse and keyboard-based interactions, are simply not valid anymore. Moreover, there is a set of tools, such as 2D views for charts, that are not easily ported to MR, and the number of studies regarding the perception of such views is small. This does not mean that we need to mimic every desktop task or tool in an MR application. Researchers rather have to start thinking about MR applications from scratch, take advantage of MR, and design with MR (instead of only the desktop) in mind.

Therefore, some areas that still present important challenges are:

• Enhanced interactions with molecular models. 3D interaction is a particular challenge in immersive environments. Molecular visualization research cannot be done in isolation, but instead interaction design needs to be considered at the same time, taking an inspiration in the field of human-computer interaction. Often, the natural ego-centric perspective through HMDs is seen as a key advantage of MR. One of the key challenges with 6DOF controllers, however, is that are held in unsteady hands, which inevitably decreases precision when interacting with the virtual objects. As discussed in Section 4, one of the key use cases is molecular modeling and drug design, which would benefit from dedicated attention on interaction research with stable input or control configurations (e. g., [START_REF] López | Towards an understanding of mobile touch navigation in a stereoscopic viewing environment for 3D data exploration[END_REF]). • Improved rendering times for complex molecules. Molecular landscapes are often large-scale containing millions or even billions of atoms. Rendering imposes a particular challenge due to basic requirement of VR (90Hz refresh rate + stereoscopic rendering). Therefore, the opportunities lie in accelerating and adopting the existing rendering techniques for VR applications. • Incorporation of 2D charts into molecular visualization immersive applications. The majority of the papers we surveyed focused on understanding the spatial structure and relationships. We found a few papers that have been using VR to layout and view abstract data [WNM20; PMI*21]. The sense of depth and immersive nature were mentioned as the major benefits of VR. Hence, we believe that other 2D-centric visualization could similarly benefit. • Evaluation of perceptual issues in Immersive Analytics. Based on the surveyed papers, we conclude that the process of validation of outcomes is still in a rather early stage and some standards for evaluation of MR molecular approaches might be necessary. Ultimately, we need to have a better understanding of perceptional issues that may arise when large-scale molecular landscapes are viewed. While a large body of research in VR perception exists in general, the question is how applicable it is to molecular visualization.

The papers we found are also distributed along the MR continuum, but with a larger portion of them lying on the side of the VR end. This may be attributed to the fact that molecular structures already exist on a different scale, hence having no connection to the world the human eye perceives. The AR work we found mostly focused on augmenting molecular visualizations for educational purposes that use markers such as QR codes.

Despite its advantages, MR devices are not free from issues. For example, HMDs can isolate the users, especially those that are entirely blocking out the view of the world. This creates several challenges that make simple tasks, such as taking notes, inputting text, or even drinking coffee, quite cumbersome. Fully virtual settings, however, also have advantages since remote users can connect to such environments, as we commented in Section 5.2. Consequently, this is an area that needs to be explored further.

We also want to point out possible further extensions that have not yet received much attention. One example is making deeper use of touch. By using haptic devices in VR or physicalization in AR, user experiences can be improved and become more engaging by increasing immersion. Other researchers also suggested the use of sonification, to extend the communication features [OBD *19].

Figures and Tables License Copyright

With the exception of those images from external authors whose licenses/copyrights we have noted in the respective figure captions (labeled as Permission for re-use obtained), we as authors state that all of our own figures and data tables in this article (i.e., Table 1, Figure 2, and Figure 3) are and remain under our own personal copyright, with the permission to be used here. We also make them available under the Creative Commons Attribution 4.0 International (c b CC BY 4.0) license and share them at osf.io/5yjku.

  , Krone et al.'s [KKL*15; KKF*17] surveys of common methods and representations in molecular visualization.

  [GBS*18] and ProteinSca-nAR [NSM*12], for instance, focus on high-school students, while BiochemAR [SWL*20] and Peppy [DDG*20] aim for undergraduates. A distinction can also be drawn based on the target audience of user studies-most researchers verify their approaches directly with students [SD18; BWH20; CGY20; PTA*20], while some perform an evaluation with teachers instead [NSM*12; SCM18]. As noted by Johnston et al. [JRA*

4. 1

 1 .2. Learning Challenges and Outcomes Generally speaking, students are tackling several challenges when learning about the structure and function of molecules. These challenges are not related only to the subject itself, but also to the techniques and approaches that are used to facilitate the study processsuch as molecular visualization. The report by Jones et al. [JJS05] divides the students' difficulties when dealing with molecular visualization into four areas, namely visual subtlety, complexity, abstractness, and conceptual depth. The first mentioned area-visual subtlety-is particularly interesting in the context of immersive environments, since it describes difficulties in interpreting spatial relationships in molecular visualization. Since MR is supposed to improve the spatial understanding of the visualized content [She03; DSKG06; KPB22], it has the potential to tackle the issue of visual subtlety naturally. Indeed, in the comparison of an AR application and a traditional desktop molecular visualization tool performed by Sung et al. [SWL*20], the students showed increased spatial awareness when using the AR solution. Moreover, it was generally accepted as being easier to use. Similarly, in the work of Peterson et al. [PTA*20] and Safadel et al. [SW19], students exhibited increased satisfaction with the spatial aspects of the AR molecular visualization. In the user study performed by Qin et al. [QCC21] based on the Nanome VR application [KBL*19], the spatial characteristics of the VR solution-influencing both the visualization of structures, and manipulation with them-were also strongly appreciated by the participants. This result aligns with the discoveries of Coan et al. [CGY20]

Figure 2 :

 2 Figure 2: Screenshot captured in the VR application Peppy [DDG*20] focused on teaching of principles of polypeptide structure. The students can interact both directly with the structure, as well as with the two-dimensional interaction dashboard.

  havior [RSH18; FCM*19; ABT*20; DDG*20; FKH*20; SWF*21; vWGK22]. For example, in NuPov [ABT*20], students are simulating a nucleophilic attack on a molecule by firing the nucleophile using their fingers. In InteraChem [SWF*21], students are performing various interactions, such as distorting molecules or trying to move hydrogen atoms through a benzene ring. All actions are supported by interactive molecular dynamics simulation, providing immediate feedback. InteraChem is also related to the topic of valence shell electron pair repulsion theory, which is the main focus of the VR tool presented by Fujiwara et al. [FKH

Figure 3 :

 3 Figure 3: DNA-protein and protein-protein interactions being examined in AR application MoleculARweb [RFK*21].

  dimensional model [BJ14; SCM18; SW19; ENP20; SWL*20; SF21; FCS21]. In other approaches, the structures are rendered in real-time based on underlying data, as common in regular molecular visualization tools [MKH*18; KBL*19; CŠR*20; LT20]. Each of these approaches comes with its strong and weak points. For example, with the former approach, the creator can create precisely crafted structural visualization decoupled from a specific application, making it thus possible to open the molecule in a variety of programs. On the other hand, real-time rendered molecules give users a possibility to render any of the many available molecules, achieving higher flexibility in the pipeline together with an option to dynamically change the properties of visualizations based on the current needs. The choice thus depends on a variety of factors, ranging from the needs of the course curriculum, through capabilities of the expected hardware, up to the skills and time constraints of the development team.

  Immersive applications have emerged naturally, from traditional molecular modeling software packages [SSS16; GBS*18] to newly developed use cases and applications [KBL*19; OBD*19]. While much of the relevant work in the literature focuses on direct translation to the virtual environment [GBS*18; DWH*20; KSB*22], a small category exploits the additional degrees of freedom and modalities that MR offers to represent non-3D intrinsic molecular data, such as genome-wide association data [WNM20]. In the end, MR-related research use cases are broad with many application areas, as introduced in the following subsections.

Figure 4 :

 4 Figure 4: The example on the top shows a common intrinsic 3D use case where users take advantage of an immersive environment for drug design [DWH*20]. The example on the bottom shows a typical non-3D intrinsic use case where the three dimensions in an immersive environment are used to lay out non-spatial data [PMI*21]. Permission for re-use obtained.

Figure 5 :

 5 Figure 5: The combination of 2D chart being embedded directly into the 3D scene in Caffeine [SFP*16]. The two linked views allow for a better understanding of the data. The image was adapted. Permission for re-use obtained.

  For collaborative modeling, Grebner et al. [GNE*16] present a user-friendly web-based platform for 3D modeling of molecules, with applications such as ligand alignments and molecular docking. Similarly, the work of O'Connor et al. [OBD*19] also provides multiple users with the ability to edit MD simulations in real-time. Modeling oper-

Figure 6 :

 6 Figure 6: The interface of Eukaryo [YSCJ16] with a minimap (topright corner) that indicates the relative location of the user inside the cell. Permission for re-use obtained.

  Alternatively, focus and context techniques [CYB*05; KSB*22] can be used to define a region of interest. Doutreligne et al. [DGC*15] also suggested a method of constraining the camera trajectory to minimize the occlusion.

  Yet, several researchers recently recognized the challenges posed by cluttered molecular environments and allow users to hide molecules or their parts on demand [ZSW*15; dCN17; GBS*18; CŠR*20; DDG*20] or use clipping [RBDR18; TE21]. While most of these techniques completely remove the filtered data, some solutions only suppress unimportant data by modulating their opacity [CYB*05]. 5.1.4. Task 4: Details-on-demand The more recent molecular viewers (e. g., [CGY20; MB21]) allow users to change between multiple molecular representations, such as ball-and-stick, ribbon, or surface, each providing a different level of detail. While means of smoothly transitioning between such different visual abstraction [VI18; VCI20] stages exist in the nonimmersive literature (e. g., [vdZLBI11; MDS*18; HMK*20]), we did not find any that would offer similar means in MR environments.

Figure 7 :

 7 Figure 7: The history trace view in CSynth [TTM*21] showing the differences between several predicted DNA models. Permission for re-use obtained.

Figure 8 :

 8 Figure 8: The emojis are used in InteraChem [SWF*21] to label happy/angry atoms based on a bond saturation. The image was adapted. Permission for re-use obtained.

Figure 9 :

 9 Figure 9: Screenshot from the VRdeo tool, showing a user setting up a camera trajectory before exporting the video recording [BBMK21]. Permission for re-use obtained.

  's [MB21] extract task relies on the FAIR model of sharing the data among users, while Goddard et al. [GBS*18] describe ChimeraX' capabilities for exporting data to AltPDB. The second export use case is to create images and videos for presentation and publication, as described by Spark et al. [SKE*20].

  Narupa [ODD*18; OBD*19; DWH*20; JOB*20] is an opensource software package where multiple users can cohabit in the

Figure 10 :

 10 Figure 10: Nanome environment with two collaborators sharing the view of the protein and its binding pocket. Image taken from [KBL*19]. Permission for re-use obtained.

  CLK*11; MKA*19]. Modern molecular visualization research, however, does not only focus on scaling up existing visualization methods: to understand and explore huge, more complex data sets, traditional molecular graphics is not sufficient anymore and the development of advanced visual analysis methods is necessary. Examples include the interactive extraction and visualization of cavities in proteins [KKL*16], novel representations that support summarizing and comparing data sets [KFS*17], and visual analytics approaches for molecular dynamics simulations [BTM*19; SFS*21] or large-scale chemical compound screening [SUS*21].

  Hu et al. [HLC*21], Cárdenas-Robledo et al. [AHRA22], and Dincelli et al. [DY22].

  • ×115 • ) [HR*95; Str20]. As for the world of AR HMDs, the field of view of the Microsoft HoloLens 2 is significantly lower ≈ 30-43 • ×29 • than of its VR counterparts. Latency is another important aspect in MR devices, since it may cause cybersickness, especially in VR. Related characteristic is the refresh rate [CGGG21], nowadays expected to be 90 Hz at minimum for a comfortable experience. For more information about the MR display technology, we would like to refer the reader to the review by Zhan et al. [ZYX*20].

  OR "extended reality") AND (molecular OR molecule OR biological OR biomolecular OR biochemical OR chemical OR bioinformatics OR cell OR cellular OR microbiology OR pharmacology OR protein OR DNA OR lipids OR membranes OR ligands OR QM OR RNA OR docking OR biomaterials OR drug design OR atom OR atomic OR crystal OR membrane) AND (visualization OR visualisation

  and Bennie et al. [BRD*19], VR produced better educational outcomes than corresponding desktop approaches. In the virtual reality study performed by Won et al. [WMT*19],

  et al. [PMI*21] allow users to explore large genomic networks in VR. The authors argue that VR offers new approaches to combine human cognition with advanced data science methods. Zhu et al. [ZSW*15] argue that the 3D view facilitates the discovery of patterns that are difficult to see on the desktop. They demonstrate visualization for spatial reconstruction of the human genome in VR. A different type of spatial reconstruction is used by Zhang et al.

© 2023 The Author(s). This is an open access article under the terms of the Creative Commons Attribution 4.0 International (c b CC BY 4.0) license. Computer Graphics Forum © 2023 The Eurographics Association and John Wiley & Sons Ltd.

Acknowledgments

This work was supported by the US DOE LLNL-LDRD 22-FS-026 and was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 (LLNL-CONF-831245) and by projects TIN2017-88515-C2-1-R(GEN3DLIVE), from the Spanish Ministerio de Economía y Competitividad, by 839 FEDER (EU) funds, and by Ministerio de Ciencia e Innovación/AEI (PID2021-122136OB-C21 by 10.13039/501100011033/FEDER, UE). The work received further support by the French National Research agency (Grant "PIRATE", ANR-21-CE45-0014) and by the "Initiative d'Excellence" program from the French State (Grant "DY-NAMO", ANR-11-LABX-0011 and grant "CACSICE", ANR-11-EQPX-0008), and from Carl Zeiss Stiftung.