Jiayi Hong

Alain Trubuil

Tobias Isenberg

LineageD: An Interactive Visual System for Plant Cell Lineage Assignments based on Correctable Machine Learning

Keywords:

We describe LineageD-a hybrid web-based system to predict, visualize, and interactively adjust plant embryo cell lineages. Currently, plant biologists explore the development of an embryo and its hierarchical cell lineage manually, based on a 3D dataset that represents the embryo status at one point in time. This human decision-making process, however, is time-consuming, tedious, and error-prone due to the lack of integrated graphical support for specifying the cell lineage. To fill this gap, we developed a new system to support the biologists in their tasks using an interactive combination of 3D visualization, abstract data visualization, and correctable machine learning to modify the proposed cell lineage. We use existing manually established cell lineages to obtain a neural network model. We then allow biologists to use this model to repeatedly predict assignments of a single cell division stage. After each hierarchy level prediction, we allow them to interactively adjust the machine learning based assignment, which we then integrate into the pool of verified assignments for further predictions. In addition to building the hierarchy this way in a bottom-up fashion, we also offer users to divide the whole embryo and create the hierarchy tree in a top-down fashion for a few steps, improving the ML-based assignments by reducing the potential for wrong predictions. We visualize the continuously updated embryo and its hierarchical development using both 3D spatial and abstract tree representations, together with information about the model's confidence and spatial properties. We conducted case study validations with five expert biologists to explore the utility of our approach and to assess the potential for LineageD to be used in their daily workflow. We found that the visualizations of both 3D representations and abstract representations help with decision making and the hierarchy tree top-down building approach can reduce assignments errors in real practice.

Introduction

How individuals of species develop is an essential question in specific subfields of biology. By looking at the historical development of a plant embryo, botanists investigate whether the plant develops normally or not. As an embryo develops, single cells divide into two new cells and these parent-child relationships can be described using temporal hierarchies. The process of creating these hierarchies is called cell lineage assignment and requires biologists to (1) take images of a multi-cell plant embryo using a confocal microscope, (2) segment the resulting 3D volumetric data to identify the embryo cells, and (3) indicate pairs to reconstruct the hierarchy backward toward the root as the first cell (i. e., the ovum) of the embryo.

In our collaboration with botanists, we identified several parts of this workflow that can be improved with a dedicated visual analytics tool. Existing tools (a) only use two-dimensional slice-based interaction to mark sister cells, (b) do not take advantage of either 3D interaction or interaction with a visual representation of the hierarchy, (c) do not allow the scientists to interactively divide the 3D geometry into parts that can be identified as being part of early hierarchy levels in the cell division, and (d) do not provide any means to automate the cell assignment process based on known cell division patterns. Given these limitations of existing tools, the current, largely manual workflow of biologists does not scale to embryos with hundreds of cells (i. e., 7 or more generations of cell divisions). A purely automatic approach would also not be ideal as they want to retain manual control over the process, to be able to deal with incorrect assignments and the special behavior of mutants.

We thus developed LineageD to address these limitations. Lin-eageD is based on specific domain requirements that we elicited with our collaborator and described in detail later in the paper. Specifically, it was important for our collaborator that the tool would be interactive to retain control over the hierarchy construction process, contain the familiar 3D representations of the embryo state, display a temporal hierarchy tree that encodes abstract temporal information about cell divisions, closely link the 3D and abstract representations, and include machine learning approaches to propose cell lineages and reduce repetitive work. We built the tool based on these require-ments, while ensuring that all views are interactive and proposed cell lineages could be efficiently reviewed and interactively corrected if needed. Our goal was for biologists to arrive at a final cell lineage more easily and with higher confidence. To better understand whether our design actually meets these goals, we also conducted an evaluation with five experts from the domain and described their feedback and our observations of their tool use.

Overall, in this work, we contribute (1) an interactive, web-based approach to establishing cell lineage, in which we combine 3D spatial and abstract data representations with correctable machine learning; (2) design considerations and decisions that led to our approach, including about the visual and interaction mappings that we used; (3) the approach of top-down and bottom-up hierarchy building and the interaction concepts that link the 3D and 2D representations at different scale levels; and (4) the evaluation with five experts to learn about the suitability of the employed approach, which also generated insights for applied visualization at large.

Related Work

Past work from three major fields is important for our own: the visual encoding of hierarchical structures, the use of visualization techniques in biological applications, and the application of machine learning in visualization. We review these fields below.

Visualization of and with Hierarchies

Hierarchical structures are essential for organizing many types of data and exploring inner relationships [START_REF] Schulz | The design space of implicit hierarchy visualization: A survey[END_REF]. Such inner correlations can be, e. g., about classification [BHGK14, BPT * 11], family development [BDF * 10], and evolution [START_REF] Horn | Visualizing biodiversity with Voronoi treemaps[END_REF][START_REF] Lee | How users interact with biodiversity information using TaxonTree[END_REF]. Over the past 40 years, much research has been conducted to design and compare different visual representations (e. g., treemaps, sundown charts, sunburst charts, and icicle plots) of both 2D and 3D tree structures (e. g., [RMC91, MGT * 03, Sch11]), to analyze hierarchical data (e. g., [START_REF] Shi | An evaluation of content browsing techniques for hierarchical space-filling visualizations[END_REF][START_REF] Tu | Visualizing changes of hierarchical data using treemaps[END_REF]), and to help with visual decision making (e. g., [START_REF] Asahi | Visual decisionmaking: Using treemaps for the analytic hierarchy process[END_REF]). Work has also been done to evaluate the different representations [START_REF] Woodburn | Interactive visualisation of hierarchical quantitative data: An evaluation[END_REF] and to propose guidelines for future design in hierarchical trees [START_REF] Kong | Perceptual guidelines for creating rectangular treemaps[END_REF]. However, most of these 3D trees were designed to add one dimension to store more information [SKW * 07, TON03, LSTD17]. Our collaborating biologists are most familiar with 2D node-linked diagrams for representing the hierarchy, while the actual cell shapes have to be interpreted in 3D views to assist them with making lineage decisions. We thus rely on a combination of both a 3D view and a 2D tree. One of the most closely related approaches is HyperLabels [KIK * 21], which uses a model's hierarchy to navigate the 3D structures. While Kouřil et al. also combine the abstract hierarchy data with 3D shapes that represent the actual physical entities, their abstract data representations do not represent additional information such as, in our case, the plant embryos' temporal development.

The size of datasets is constantly increasing, so researchers have also explored methods to flexibly adjust the space a tree representation takes to increase its efficiency [LTX * 20, BRZW17, Ahu86]. For treemaps, for example, Tu et al. [START_REF] Tu | Visualizing changes of hierarchical data using treemaps[END_REF] used a spiral layout to visualize the updates of the hierarchical data. Similarly, van de Wetering et al. [START_REF] Van De Wetering | Spacereclaiming icicle plots[END_REF] redesigned the icicle plot to reclaim empty space, to avoid the situations where nodes deeper in the hierarchy have less space. These techniques are not applicable in our scenario because we need to show all nodes equally such that we do not have excess empty space. Moreover, we do not have the full hierarchy at the start, and biologists are not always certain about the cell lineage hierarchy root in the beginning.

Another important aspect of our hierarchies is that they represent temporal development, and researchers have visualized temporal data with trees before. For example, Carvalho et al. [START_REF] Carvalho | Temporal data visualization technique based on treemap[END_REF] combined aspects of treemaps with a calendar, so that users could target data within a certain time slot. In our case, the division time is different for each cell and the former and latter stages should always be presented for reference, so that we cannot use filters to target at a single level. Shen [START_REF] Shen H.-W | Isosurface extraction in time-varying fields using a temporal hierarchical index tree[END_REF] also used index to mark the cell division within a time interval, but it is hard to read especially the tree is large. We thus encode the time periods with the layers.

In addition to innovative layout design, some work also focused on improving the interaction with trees to adapt the substructures and compare the results with the original data, which we also need in our work. AdaptiviTree [START_REF] Tan | AdaptiviTree: Adaptive tree visualization for tournament-style brackets[END_REF] encoded real-time game status information and indicated changes by color. Other work [BMW16, BvH * 11] used visual cues to allow viewers to compare different trees that were shown horizontally, with color-coded similarity metrics based on juxtaposed icicle plots. Because we use the trees proposed by ML only as an initial guess and there is no ground truth, we do not show multiple trees and simply rely on color coding to encode differences. Related to this approach is the encoding of the uncertainty of the input data, for which researchers have used, e. g., glyphs [LRCP07, YWL * 20]. While these methods can handle diverse forms of uncertainty including locations [START_REF] Lee | CandidTree: Visualizing structural uncertainty in similar hierarchies[END_REF] and subtree structures [START_REF] Lee | CandidTree: Visualizing structural uncertainty in similar hierarchies[END_REF], we can also use color-coding to indicate the ML model's certainty with respect to its results.

Biology-related Visualization

Biologists often study complex patterns or processes. While various stakeholders have different levels of interest in the respective datasets [START_REF] Lee | How users interact with biodiversity information using TaxonTree[END_REF], we focus on interactive tools for expertsspecifically botanists interested in plant embryo development.

Within the biology domain, several visualization approaches and tools are currently being used. For example, Briggs et al. [BPT * 11] combined dendrograms and heatmaps for the visualization of taxonomic diversity. Such representations, however, are usually not space-efficient [START_REF] Mcguffin | Quantifying the spaceefficiency of 2D graphical representations of trees[END_REF]. Moreover, we need to show both abstract cell lineage data as well as the actual 3D shape of the entities. Numerous platforms (e. g., OsiriX [START_REF] Rosset | OsiriX: An open-source software for navigating in multidimensional DICOM images[END_REF], Fiji ImageJ [SACF * 12], and Avizo) are designed to display such multi-scale 3D data, and these are currently being used by our collaborating experts to establish the cell lineage (e. g., the TreeJ plugin for Fiji ImageJ). form of cell lineage. Of course, also general tools exists that could support the representation of 3D shape such as VTK [START_REF] Schroeder | The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics[END_REF] which was used to create visual representations of biological data (e. g., [RSR04, SACF * 12]) and ParaView [START_REF] Ahrens | Paraview: An end-user tool for large data visualization[END_REF]. We also use the VTK library for creating the visual representations in our tool, but our emphasis is on the combination of 3D spatial and abstract data representations and their interactive use for the establishment of cell lineage for plant embryos.

Machine Learning in Visualization

In the visualization field, machine learning is used to ,[START_REF] Horvitz | Principles of mixed-initiative user interfaces[END_REF]. Inspired by these, we predict cell inheritance with a neural network model and display partial assignments to engage users [START_REF] Early | Test time feature ordering with FOCUS: Interactive predictions with minimal user burden[END_REF] and reduce manual labor.

Background and Design Considerations

As we just discussed, trees can be used to visualize cell hierarchies in embryos. Our collaborator also confirmed that biologists use and appreciate 3D views to assist them with understanding the spatial structure and to make decisions on the lineage. Here they rely on actual 3D meshes that they derive from the confocal microscopy slice data by segmenting and then processing them into the shapes of the embryo cells at the time of data capture. Yet, no suitable tools exist to determine cell lineage for several reasons:

A cell lineage tree only represents approximate division stages that apply to most cells, while some cells may remain unchanged from a given level to the next and the lineage hierarchy needs to account for such cases. Consequently, there exists no single generic (binary) tree that would represent the development of all embryos.

In addition, the tree does not yet exist at the start of the process and cannot be computed from the data, instead, it has to be established step-by-step by the experts based on what they observe in the 3D spatial and abstract data. Figure 1 illustrates this process, in which the experts reason based on the embryo's configuration at a given time to establish how it developed into this stage from a previous set of cells, and then they repeat the process. Existing tools are usually created to focus either on 3D spatial or on abstract data representations, while in our case both types of representations are tightly coupled and biologists need both to make decisions (the spatial cell data as well as abstract data such as the hierarchy as established so far and information such as shared area-either type alone is not sufficient).

Based on all these constraints, we set out to design a new interactive tool for establishing and visualizing the development of plant embryos. To reduce the tediousness of having to make many lineage decisions for larger embryos, we targeted a process that relies at least partially on machine learning, yet also leaves the biologists in control and allows them to override decisions as needed. We also based our design on inspirations from existing tree representations such as icicle plots [START_REF] Woodburn | Interactive visualisation of hierarchical quantitative data: An evaluation[END_REF]. Moreover, we followed the principles and advice on designing hierarchy structures [START_REF] Kong | Perceptual guidelines for creating rectangular treemaps[END_REF], cell visualization systems [DVV * 00], and mixed-initiative user interfaces [START_REF] Horvitz | Principles of mixed-initiative user interfaces[END_REF]. Next, we summarize our design considerations. D1: Provide an overview of the hierarchical structure to indicate the cell division process. We aim to create complete and accurate cell division hierarchies so that plant biologists can study the division history. Yet every cell divides in a different time and it is impossible to measure the precise division time from a single confocal microscopy dataset. So we rely on the fact that an embryo's cells roughly divide within similar time ranges, and use these division times to build the hierarchy. The tree should also clearly emphasize cells that divide outside of the regular time range. The tree thus not only needs to contain the parent and children relationships but also represents the time-dependent division process. As such, any node in the tree represents a given cell and thus its connected 3D representation at a certain stage, so navigation in the hierarchy needs to lead to adjustments of the 3D visuals. D2: Support the history building in two directions. Biologists traditionally build the tree in a bottom-up approach. From the overall shape of the embryo, however, they can also deduct how it divided in the very beginning. Introducing this knowledge early could improve the ML predictions and make it easier for the experts to specify assignments. It should thus be possible for biologists to build the tree from two directions: bottom-up and top-down (Figure 2). D3: Show related information to assist with decision making. According to our collaborator, biologists take the cells' shape, the shared surface between two adjacent cells, their approximate volumes, and the 3D positions into consideration when picking the sister for a target cell from its neighbors, and we need to show all this information. While the spatial properties are most effectively shown in a 3D view, the experts need to be able to correlate them to the other data. While 2D diagrams would be possible, each cell only has a few neighbors and a color coding thus promises to be better than separate 2D diagrams. In addition, we need to account for cell occlusion in the 3D view, so the system needs to support visibility techniques such as explosion views and layer peeling.

D4: Provide assistance for building the hierarchy with machine learning (ML). Our previous work [START_REF] Hong | Design and evaluation of three selection techniques for tightly packed 3D objects in cell lineage specification in botany[END_REF] indicated that, even with an interactive system, traversing neighbors to find sisters for more than 200 cells is too time-consuming, and ML-based assignment suggestions could greatly improve the workflow. These predictions then need to be checked by the experts with the help of an integrated hierarchy and spatial 3D interaction. Any mistake, however, would invalidate any prediction for earlier cell divisions. The system should thus use an ML model that predicts the most likely cell pairs for a given hierarchy level, and bases future levels on verified or corrected assignments. The full power of our approach relies on a fluid change between the bottom-up ML-based prediction, expert correction, and the top-down hierarchy specification so that the expert is always in full control over the process. At any time, the hierarchy view should show the status of the lineage, indicate decisions confirmed from the ML or corrected by the human, and navigate the temporal development of the embryo.

D5: Provide easy access to biologists who are not computers professionals. Many experts may be hesitant or not allowed to install extra software on their workstations. Moreover, such local installations would require cross-platform system development. We thus aim for a web-based implementation that facilitates easy access, interaction, and data sharing among researchers [LLC * 19].

System Design

Based on these considerations, we designed our web-based tool LineageD with two major components: a front-end for experts to interactively establish the lineages and a back-end that comprises a module to process the input data for the analysis, a set of default embryos for testing, a pre-trained classifier, and the partial lineage data currently being worked on (D4, D5).

Front-end Interface

Our web-based front end needs to allow the biologists to explore the 3D information in its hierarchy-dependent configuration, traverse the cells at the different hierarchy levels and understand their spatial and derived properties, make decisions about the cell lineage for all cells, and then record these decisions to specify further hierarchy levels. We thus offer the following elements to assist the experts.

Our LineageD system (Figure 3) centers around two main views: the main 3D view of the embryo and the hierarchy view that shows the cell lineage tree the biologists are building. The hierarchy on the right of the interface initially consists just of small blocks, where each represents a single cell at the time when the confocal microscopy dataset was captured; i. e., the leaf nodes of the tree (D1). We chose a tree layout that grows from top to bottom (i. e., root at the top) to match the traditional format used by the biologists (we illustrate the design process in Figure 4). As the experts work on establishing the hierarchy and declaring sister cells, this view actually becomes a hierarchical representation that we based on (completely filled) icicle plots. Unlike node-linked diagrams that take more space and are harder to navigate in a wide tree, this mapping has the benefit of establishing clear layers, which can then be navigated with the help of the slider on the left of the hierarchy view. We compute the widths of the leaf nodes based on their normalized volumes such that the biologists can directly see the relative sizes of the cells' volumes in the tree (D3). The icicle mapping then also nicely reflects that the volume of a mother cell is roughly the sum of the volumes of its daughters in consideration that the potential volume loss is not essential to biologists, and that all cells of a given level together always form the whole embryo. We chose the vertical direction for the hierarchy layering because the biologists are used to this layout.

The hierarchy view is tightly linked to the main 3D view on the left. In the latter, we show the whole embryo at the chosen division time, i. e., the level selected in the (partial) hierarchy view on the left. A regular 3D representation has the problem of the dense packing of the embryo cells, which results in inner cells being fully occluded by the outermost cell layer. To address this visibility issue we offer two visibility management techniques: explosion views (e. g., [START_REF] Agrawala | Automated generation of interactive 3D exploded view diagrams[END_REF][START_REF] Tatzgern | Multiperspective compact explosion diagrams[END_REF]; see Figure 3-main 3D view) and layer peeling (e. g., [START_REF] Sultanum | Point it, split it, peel it, view it" Techniques for interactive reservoir visualization on tabletops[END_REF]; see Figure 5-3), both controllable through sliders. The 3D view also provides typical 3D navigation interaction, including zoom, translation, and rotation to allow biologists to explore the spatial character of the embryo.

We closely coupled both the 3D view and the hierarchy, not only through cross-linked selections. When we select a given cell in either view, we not only select it in the other as well but also show this cell and its currently assigned sister (if any) in a secondary 3D view on the top, the target and sister view that shares the same camera orientation as the main view. Here we highlight the selected (target) cell in red and its assigned sister in its original color, while we show the rest of the embryo's cells transparently to illustrate the target and sister cell' relative positions in context. Unlike in the main view, in the target and sister view we always show all cells in a tightly packed configuration and do not apply any explosion. This way we allow the biologists to understand the combined shape of the proposed cell match and make it easy for them to identify In a pilot study, however, we found that these views were confusing to our collaborator, and we settled on only showing the target and sister view as a secondary 3D view and moved some of the other functionality to be accessible on demand (see below).

We use the same color mapping in all views, both the 3D views and the hierarchy. We initially color-code both views by spatial location [OBCS * 12], which we calculate based on the relative 3D positions of each cell's center of mass within the whole embryo, mapped to RGB. This mapping allows the users to clearly see if two cells are located close to each other, even in the abstract hierarchy view. During the later interaction we also offer color-coding by trust of the ML model, by the shared area between two matched cells, and a random color mapping that biologists are used to from their existing tools and that allows them to easily see the decomposition of the embryo into the cells of a given hierarchy level (D3).

Because the biologists can also use a top-down approach to establish the lineage (in addition to bottom-up), we need to represent both the top and the bottom section of the hierarchy at the same time-even if the tree has not been fully specified. We thus split the hierarchy into a top tree and a bottom tree (D2). For the top tree, we start with the root, i. e., the single ovum cell. As new layers are established top-down, we add them below the root. We cannot be sure about the layer number for the top-down layers at the beginning, so we named them N, N-1, etc., until we complete the hierarchy. We know all cell elements at the lowest level, so we can ensure that both representations correctly reflect this containment relationship. To ensure smooth layer navigation, as long as the hierarchy is not completed we introduce an in-between section between the top and the bottom tree marked by dots. We also add a corresponding layer entry to the hierarchy slider on the left, with newly established bottom-up layers being added directly below it and newly established top-down layers being added directly above it.

Due to the exponential nature of cell division, the corresponding cell lineage tree can become very large horizontally very quickly, already for embryos with only a few generations of development. We thus provided two means to cope with the resulting view limitations. First, we provide a slider on the top of the hierarchy to control the horizontal zoom, allowing the users to see the whole tree in one view or to zoom into details. For the latter case we can ensure that the respective labels of the cells (numbers in our case) are visible, yet a viewer may lose the context of what part of the hierarchy they are examining. We thus also added a horizontal World-in-Miniature thumbnail view to make it easier to navigate the wide icicle plot hierarchy tree and which is often used in virtual reality (e. g., [START_REF] Stoakley R | Virtual reality on a WIM: Interactive worlds in miniature[END_REF][START_REF] Danyluk | A design space exploration of worlds in miniature[END_REF]) and, more relevant to our case, in 2D video games (then often called "mini-map"; e. g., [START_REF] Gekker | Mini) mapping the game-space: A taxonomy of control[END_REF]). This view, which we include below the hierarchy view, always shows the whole hierarchy and highlights the subsection that is currently visible in the detailed hierarchy view and facilitates navigation.

Finally, most of the controls for the interaction (which we discuss in more detail below) are then assembled in the operations panel. It also contains a menu to load new datasets or export results as well as instructions for users about our 3D interaction mappings.

Server-side Back-end

Our tool's server-side takes care of data processing and analysis. Our datasets comprise the cells' names (i. e., numeric labels) and mesh data for their surfaces. The experts create these datasets with dedicated tools (e. g., Avizo, Fiji [SACF * 12]). In the back-end we use this input to build the necessary data structures to later be able to construct the hierarchy. We also derive the needed information such as the 3D cell locations (centers), neighbor counts and respective neighbor look-up tables, normalized cell volumes, normalized cell surfaces, and normalized shared areas between neighboring cells. After the completion of the lineage, the back-end also allows us to export the results in the tree data format needed by the biologists.

An essential part of the back-end is the creation of the ML model (D4). For this purpose we also maintain a collection of 97 past (manual) cell lineage assignments provided to us by the biologists. For these we also computed the data-derived parameters mentioned above and used this data to train a neural network model to allow us to predict the lineage for future datasets. The classifier computes a probability for all possible direct neighbors to be sister cells, and we then pick the one that has the highest chance. This reduces the number of possible pairs, and we continue picking the pair with the highest probability until this probability falls below 50%. This prediction functionality can then be accessed from the front-end.

Interaction Design

Our interaction design centers around the processes of proposing new matches of likely sister cells (manually and with the help of our neural network model) and the verification of whether these matches are correct. Both processes rely on abstract as well as 3D information, so our expert users have to constantly switch back and forth between the abstract hierarchy view and the spatial 3D views. Next, we describe how, in LineageD, we support the biologists both in the top-down and the bottom-up lineage specification.

For a new dataset we show all cells as blocks on the lowest hierarchy level on the right, and in the 3D view we show the embryo with the cells colored based on their 3D location. Usually the first step of the biologists is to remove those cells from the consideration that are part of the supporter (i. e., those that connect the plant embryo with the rest of the seed) by selecting them in the 3D or hierarchy view and assigning them a respective label (Figure 5-1).

With the top-down specification we offer a new functionality not supported by the traditional tools. We take advantage of the fact that, for most embryos, the first few cell division generations happen in a predictable way-one that can still be reconstructed from the geometric shape of the embryo and its cell decomposition, even at a later time. For this purpose biologists start to explore the shape of the embryo in the 3D view, usually via rotations. To better see and understand the inner structure, they can also use the explosion functionality. Then they decide on the most likely initial division-often a rather straight subdivision between the existing cells. We then allow them to mark one half using a spatial lasso selection [START_REF] Lucas | Design and Evaluation of 3D Multiple Object Selection Techniques[END_REF] (Figure 5-2) to register the selection as the first subdivision, which can also be further refined.

As they continue this top-down process, it is important to note that the interaction naturally relies on the selected level in the hierarchy. For example, to specify the third generation N -2 (i. e., the second split), the experts first select one of the daughters of the ovum in the hierarchy. This highlights all parts of the embryo that belong to this cell at the N -1 level in red, as if this was a single cell. Any potential explosion setting, however, applies to the level currently specified in the hierarchy just below; at the beginning this would be level 1. † The following lasso operation on a subset of these cells then selects them for the next split of only the selected cell, in a context-aware fashion [START_REF] Efstathiou | CAST: Effective and efficient user interaction for context-aware selection in 3D particle clouds[END_REF]. This step would then have to be repeated for splitting the second cell of the N -1 level. To mark these fully human-decided divisions, we display blue bars between the divided parent cells and their children in the hierarchy tree Figure 3. Naturally, this top-down process only works for a few generations, but it provides valuable constraints both for a manual and for the ML-supported process that we describe next.

For the bottom-up process we follow the biologists' traditional work process, as illustrated in Figure 1. They would start to traverse all cells at the lowest level and specify likely matches by examining each selected cell's direct neighbors to find its most likely sister. To reduce this workload, we can now use our neural network model and make ML-based predictions for the lineage. Initially we had used this process to predict the entire lineage, but in pilot studies it quickly turned out that this approach was not ideal. The experts disagreed with certain mappings, rendering the entire remainder of the path toward the root node obsolete when they corrected the mistakes. We thus now restrict the classifier to only predict the lowest non-specified hierarchy level (Figure 5-4).

After such a ML prediction step, the biologists can traverse the newly found pairings and either confirm or reject them (D4). To do that they can select a daughter cell in the hierarchy or 3D view, which shows it in a red highlight color, with the rest of the cells retaining their colors according to the chosen color scale (Figure 5-5-left). In the target and sister view, we also show its currently assigned sister in color, while the rest is transparent. A single click on a mother cell ‡ is similar, only then we highlight the whole mother in red and show no sister. With these techniques the experts can assess the mother cell's shape and decide if it is plausible. To better understand the alternatives, the experts can also double-click in either the hierarchy or the 3D view on a daughter cell, which leads to the 3D view showing a focused view with only the direct neighborhood of the † If the ovum cell was selected instead, however, after a first top-down division specification, the explosion would only separate the two daughter cells of the ovum to emphasize this position in the lineage tree.

‡ For performing such a selection in the 3D view one has to first select the corresponding hierarchy level in the hierarchy view, e. g., via the slider.

selected cell (Figure 5-5-center), while the target and sister view then only shows the cell and its currently specified sister. Upon a further click on one of the neighbors in the main 3D view we show this neighbor also in the target and sister view, along with a potentially assigned sister of the neighbor (Figure 5-5-right).

With these different mechanisms to see detailed views, the biologists then make a decision about whether the pairing is plausible or not. If the biologist then confirms a proposed mapping when a given mother cell is selected, we mark that pairing as confirmed by showing a green bar between the mother cell and its daughters in the hierarchy (Figure 3-right). If they disagree with the pairing, we break it and, for the time being, treat the cells as non-dividing cells for the chosen hierarchy transition. They can be left as such or can be re-assigned. In the latter case, biologists select one cell, show its neighborhood with a double-click, and then pick one of the neighbors to mark it as the new sister. To distinguish such human specifications from confirmed ML pairs, we mark them with an orange bar in the hierarchy between the mother and its daughters.

After examining all pairs of a given level and confirming or correcting them, the experts can then go on to predict the next level. We treat any pairing that has not been explicitly confirmed as implicitly confirmed in that case, and also mark it with a green bar. The process then continues until the whole hierarchy is specified, the top and bottom trees merge, and we can export the result.

Implementation Details

We implemented LineageD in JavaScript and Express.js (v. 4.17.1). For the front-end, we rely on D3 [START_REF] Bostock | D 3 : Data-driven documents[END_REF] to realize the hierarchy tree rendering and render the embryo in the 3D views with vtk.js (v. 19.0.4) [START_REF] Martin | The design and implementation of an object-oriented toolkit for 3D graphics and visualization[END_REF]. We implemented the back-end in Node.js and used MongoDB to create our database. We used TensorFlow's (v. 2.8.0) Keras library to train the multi-layer sequential neural network model offline with Python. We created two hidden layers with ReLU activation and adopted softmax for the final output layer to get the probability score, and the model's single-pair accuracy reaches approximately 94%. The raw data is produced by Avizo and contains the cells' information including names, vertices, and triangles. We then parsed the raw shape data and set each neighboring pair as one record. For each pair, we extracted features for the training data based on the recommendations by the biologists: the normalized distance between the cell pair, their normalized shared surface area, their neighbor counts, their normalized volumes, normalized surface area, and the directions of neighbors with target cells. With Tensor-Flow.js (v. 3.8.0), we used the trained model to predict assignments in the website back-end. For an embryo with 256 cells, such as one of our default datasets that we show in Figure 3, there are more than one million vertices with more than five million triangles. Depending on the network speed when connecting to the Web tool, typical datasets can thus be loaded in less than a minute and can be rendered at interactive speeds (approx. 20 fps for the 256-cell example on a macOS with AMD Radeon Pro and Intel UHD Graphics 630).

Empirical Case Study Evaluation

While we developed our overall approach and our LineageD tool based on conversations with our expert collaborator, we were also interested in feedback from independent plant biologists, by whom we envisioned our tool to be used. For this purpose it does not make much sense to run a quantitative experiment with many participants because, first, we do not have a reasonable alternative technique to which to compare our approach and, second, we would not find enough participants skilled in the traditional lineage process to draw meaningful conclusions. Instead we conducted case study evaluations [START_REF] Kitchenham | Case studies for method and tool evaluation[END_REF] with expert participants, as it has been done in the past in the visualization field (e. g., [LRF * 11, KGP * 12, HRD * 19, WBR * 20]). The benefit of this approach is that experts are trained in the traditional cell lineage process and can thus assess our new tool based on their experience.

In this section we present the overall experimental setup, our observations, and the user feedback from the biologists interacting with the system. The specific embryo we used in the study was a real Arabidopsis thaliana dataset provided by our collaborators, which comprises 64 cells (not counting the cells of the supporter). Datasets that are needed to be analyzed in practice typically range from several tens of cells to several hundreds of cells, so this example dataset is realistic (and still manageable in our study) as well as already challenging to analyze for biologists, and they could compare this experience to their past work. The whole investigation was approved by our institution's (Inria) ethical review board (AVIS n°2021-46) and we pre-registered the study setup and design (osf.io/rhyg4).

Participants

We recruited 5 expert biologists (2 females, 3 males, ages 31-54 years, mean 43.4 years) via social networking and mailing lists, denoted as P1-P5. We offered prospective biologists free access to our tool as compensation. Before the study, we asked them to fill in a consent form, a research media records release form, and a background and demographic data collection form. All of the participants had a PhD degree and have been doing post-PhD research on plant biology or related fields for 2.5-25 years (mean: 14.1 years, sd: 9.4 years, median: 12 years). Two of them were experts in plant biology, while the other ones focus on computational biology, biomathematics, or bio-images processing. We conducted our study via videoconferencing due to the COVID19 limitations.

Procedure

Our case study experiment consisted of two parts: first we conducted an observational study of the experts using our tool with the example data, followed by a survey and a semi-structured interview.

Observational study. We conducted the experiment with each participant individually. We began by presenting a brief introduction about the features of LineageD. Next, we asked the participant to open the website on Chrome using the credentials we provided and to share their screen. We then briefly explained to them the components of LineageD and introduced the main interaction mechanisms, as explained in Section 4. Then the participant could freely explore and establish the lineage for the 64-cell dataset. We used the thinkaloud protocol and with the participant's permission, we recorded the screen and audio during the process for our later analysis.

Survey and semi-structured interview. To better understand the usability of the tool and avenues for future improvement, we distributed a questionnaire which was designed based on the System Usability Scale (SUS) [START_REF] Harper | Improving user satisfaction: The questionnaire for user interaction satisfaction version 5.5[END_REF]. We also conducted a semi-structured post-study interview to understand our expert participants' usage experience and insights they may have gained.

Study results

A whole participant session took 60 to 100 minutes. All biologists reported that the 64-cell embryo was challenging for them to assign, and none of them finished the complete hierarchy history reconstruction within the study session. This was also not to be expected since they had to learn and get used to the new tool and its interaction concept first as well as answer our questions in the experiment. All of the participants, however, said that they plan to work with the tool more after the study and try it in practice.

Learning the Functionality. We found that three of our participating experts traditionally used 2D slice-based interaction (i. e., in TreeJ) to navigate the embryo dataset and to mark sister cells, as opposed to using 3D interaction with a surface-based dataset that we offer. This means that participants first had to get used to the fundamental 3D interaction techniques (rotation, translation, scaling) in projected 3D views. They also traditionally used text-based node-link representations of the hierarchy (see Appendix A), so they had to get used to the icicle plot representation we used. Webbased implementations as we used for LineageD were also new to one participant: P1 thought that the tool is quite different from the traditional software he is using. He stated that they generally use desktop software, but he believes that with sufficient time to play with it, most people could learn the new tool quickly. A final aspect was that P3 said that it was difficult for her to learn the actions to expect from the system based on specific input, such as what part of the tree to which the prediction applied. Overall, these observations meant that the training phase in our experiment was relatively long for four out of the five participants, but we expect that with more training, they can get used to the new interaction concepts. Interaction Design. After the training we observed how the experts freely specified the lineage of the cells in the embryo. While they all used different strategies, they all heavily relied on both the 2D abstract tree and the 3D views. Four of them primarily used the hierarchy view to select the cells for checking the predicted assignments. They used the 3D view only as a source to understand the cells' shapes and their locations within the embryo, rather than to select the cells in this view. P2, in contrast, used the hierarchy and the 3D views evenly to select the cells to check. P1 and P4 explicitly stated that they appreciated the linking between the tree and 3D views. Moreover, P2, P4, and P5 specifically expressed that the approach to creating the hierarchy tree in a top-down fashion was very interesting to them-as they do not have this ability with their traditional tools. For example, P4 suggested that, for large embryos, biologists could specify the lineage separately based on the top-down divisions or look only at a subset of the top tree as a form of grouping to reduce the complexity. In addition, P2 and P4 mentioned that the peeling and explosion functions have a great potential to help them with their decision making. Overall, P1, P2, and P4 stated that using the tool has the potential to reduce assigning time and possible errors, despite requiring about double the clicks as the traditional approach according to P1. This applies especially to embryos with more than 20 cells because it is very difficult to do it manually. P2 also believed that LineageD could improve his confidence in assignments.

Interface and Visuals. All participants stated that they appreciated the visual representations, both 3D and 2D. They also liked, for example, the mechanisms for encoding the cell volumes in block widths and the coloring by shared surfaces or location. Overall, P1 considered LineageD much more powerful than any tool that they currently have, especially in selecting cells from both 3D environments and the tree. P2 specifically appreciated the target and sister view that shows the proposed pairing as well as the transparent context. He liked the representations and interaction design of the tree, but also thought it may take time for biologists because they are used to node-link trees. He also added that some other functionality may need to be adjusted further to the mental concepts of the biologists, but thinks this is a normal process that he would expect for most tools. Overall, all comments about the design were on the positive side. For instance, P3 thought that all individual components are needed and easy to use and she liked the hover effects in the hierarchy tree. P4 was happy to see that the tool is versatile, because he could use peeling and explosion independently. P5, finally, liked the hierarchy tree and thought all the options are good.

Understanding the Embryo's Development. We also asked biologists whether the tool helps them to better understand the embryo's development. All participants agreed that this is the case. P1 envisioned that, with the hierarchy tree, he could perform quantification, data analysis, and create models because we show a lot of additional information about or related to the lineage. P2 was excited about the possible insights and wanted to try other datasets with the tool, such as animal embryos. P3 thought that some visualizations, like the target and sister view, make it easy for her to observe the development of the embryo's organization over time. P4 said he better perceived certain features of the embryo with the information we presented in the tool, such as the cells' volumes and shared surfaces. P5, finally, said that 3D views are essential for getting a good understanding of the embryo.

Machine Learning Experience. Next, we inquired whether our participants think that the automatic assignment based on machine learning helps them with the lineage process. P1 and P4 thought they cannot fully evaluate it because the experience time was not enough for assigning big embryos. Nonetheless, P1 and P5 reported that the ML model was essential and that it worked well for small embryos. P2 thought the machine learning can generally help a lot, but he would prefer the model to predict the lineage from both directions (top-down and bottom-up), rather than one direction here (bottomup). P2 and P4 also were curious about how the model actually made the specific assignments. P3, interestingly, said that she expects the model to help even more for larger embryos with more cells. She was, however, confused about whether the prediction function is applied to the top-down or bottom-up approach, and like P2 also wanted a model to predict the top tree. She also recommended to pre-visualize the model predictions for biologists to choose from before integrating them into the lineage tree. P3 and P4 were not sure whether the time needed to check and correct the ML predictions would ultimately be less than manually specifying the lineage.

Improvements. We asked the experts what interactions or elements they missed in the interface. P1 and P4 would like to have functions to label or categorize the cells so that they would have more features to examine and analyze the situation. P1 and P3 also missed a function to temporarily hide selected cells to get access to the inner cells and to distinguish checked pairs from unchecked ones. Also, P3 wanted the ability to explicitly highlight the shared surface between two neighboring cells to help her to decide between two possible cell pairings. The System Usability Scale rating we received for LineageD was 68 on average, which is exactly the average SUS score [START_REF] Mclellan | The effect of experience on System Usability Scale ratings[END_REF]. This means that we need to improve the overall usability of our tool, but this is to be expected for a research prototype. We will use the specific feedback that we received during the study to make these improvements.

Discussion

Based on the responses of our participants we can now summarize and reflect on the main take-aways from our work.

Influence of past interaction habits of experts. Our specific application domain required us to provide both abstract 2D and spatial 3D representations: the cell lineage hierarchy that the experts constructed along with data such as cell volumes or area of shared surfaces on the one side and the actual shapes of the 3D cells that make up the embryo on the other side. Interestingly, it turned out that most experts were not yet experienced with working with (projected) 3D representations, instead they used 2D slices from the segmented microscopy data to interactively mark cells. In addition, traditionally they do not use the hierarchy tree as an interaction proxy, and it only serves as the result of the lineage specification (also see Appendix A). This fact may be the reason that many of our experts were reluctant to work directly in the 3D view and, instead, primarily interacted with the abstract hierarchy as this somewhat resembled a version of their previous slice-based view (yet stripped from its spatial properties). More longitudinal studies will be needed to see if experts can learn the benefits of interacting in 3D space, and to take full advantage of the linked 2D and 3D views. Our observations, however, make this a likely development as some of our participants already understood the advantages of our 3D visibility techniques such as explosion and peeling.

Reliance on both 2D and 3D representations. Our specific application exemplifies a case where both 3D spatial and abstract information and, in particular, the tight connection between both is essential to solve a task. Nonetheless, both types of representations show different aspects. Only the tree view can show inheritance and, thus, only here can the biologists specify which cell division time stage they want to see. At the same time, only the 3D view can show aspects such as proposed sisters in a spatial context or the specific shape of a shared surface between two cells. Yet only together they provide enough information for the biologists to be able to solve their task. Moreover, unlike in many existing 3D interfaces, the navigation of the level in the abstract hierarchy actually has an impact on the shape composition itself (i. e., on the decomposition of the embryo into cells at the different hierarchy levels). In our case the actual compositional information is non-existent at the beginning and incomplete throughout the process, so the interaction effects change as further parts of the hierarchy are established. It is also interesting to note that, in both views, it is difficult to see all information for realistic datasets. In the 3D view we thus need to rely on techniques such as explosion and peeling, while in the 2D view we need the world-in-miniature thumbnail view.

Complexity of cell lineage. According to one of our participants, it is already very difficult for biologists to establish the lineage for embryos starting at a few tens of cells, especially when they want to reconstruct the hierarchy for several generations (i. e., from the observed generation to the very beginning of the division process). Our novel support of also top-down specification made this process easier for the experts, and all our participants took advantage of both ways of establishing the lineage. From their responses it seems that the top-down process makes it easier for them to control their interactions and have more confidence in the results, in particular since they deal with incomplete information most of the time. Nonetheless, further interactive assistance such as being able to focus and restrict operations (including the ML suggestions) to only subsets of the whole embryo, as suggested by some participants, may make the process even easier for them.

Generalization. Even though we focused on plant embryos, there is no reason that our findings could not also apply in a broader scope. In particular, biologists that study other kingdoms of life such as animals or fungi may similarly profit from our work directly. The most closely related past work on the depiction of parts may be research on the creation of assembly instructions [APH * 03,LACS08, TKS11, GYL * 13], even though the actual interactive control of such depiction was typically not discussed in much detail in such work. Other past work [KIK * 21] specifically discusses the navigation of 3D data through hierarchy interaction, yet in our work we actually change the composition of the 3D scene through this input. We thus think that our interactive combination of abstract hierarchy and spatial 3D views can inform future visualization tools that need to support such inter-dependencies.

Limitations and Future Work

Both our specific realization of LineageD and our empirical evaluation have some limitations that we will discuss next. We only focus on major issues, while minor usability issues as highlighted by our study naturally also need to be addressed. First, the actual shape of the cells depicted in the 3D views is a result of an approximative surface reconstruction from the segmentation of the original volumetric data. We only relied on the shapes on the lowest level and merged them for higher stages in the hierarchy. It would be better to actually apply any new hierarchy data to the segmentation masks and then re-extract the cell shapes at the coarser levels, which we also plan to do in the future. Second, for our machine learning support we adopted a neural network model to provide predictions for biologists to interact with. Other machine learning models such as reinforcement learning agents, however, could also work in our scenario and we are interested in comparing different models in the future. Also, we are interested in improving the interactions with the model, like sorting the pairs with model confidence and enabling users to pick to decide. As suggested by the biologists, we could also look into showing the reasons for the suggestions of the ML model yet we are skeptical if this would lead to a helpful representation-at least for our current neural network model. Instead, a more promising way could be to take the biologists' manual corrections into account (which we currently do not do) and to propose and visualize alternative pairings, for the experts to pick from.

Our relatively short case study evaluation is also limited. First, we largely did not get past the initial learning curve for the experts. While this also showed some interesting aspects about their interaction habits, it would be very interesting to conduct a longerterm evaluation with our participants and see how their interaction changes over time. Such a longitudinal evaluation would also allow us to better understand the challenges and benefits of the 2D-plus-3D interface on a deeper level, leading to better design guidelines for future tools. Second, we also only used data of a single species in the experiment, and a longer-term evaluation would allow us to use a variety of stimuli and even non-standard cases such as the scans of mutant embryos-a current research topic in the domain.

Conclusion

Having developed LineageD together with a domain collaborator, we were surprised to see that most domain experts did not have even basic visual representations such as tree representations [START_REF] Schulz | Treevis.net: A tree visualization reference[END_REF] as interactive elements in their traditional tools. A similar situation happens in the top-down and bottom-up approach where biologists spend reasonable amount of time to get familiar with the interaction. Moreover, when we provided them with this tool along with a standard 3D representation of their data, they primarily interacted in the 2D abstracted hierarchy as opposed to the 3D spatial view that showed the relevant data (cells within the context of the embryo)-almost like an extension of their previous 2D interaction on segmentation slices. So one of the main results of our work seems to be that, as a community, we may need to spend more effort on studying how established representations (from our perspective) can actually be transferred to tools in our application domains. We plan to continue our work with the biologists to do just that, and to also see how LineageD can evolve over a longer time to better suit the needs of the biologists.

Another interesting takeaway is that we may need to look further into ways that machine learning can be integrated into our tools. It is not necessarily essential to get the best or most correct prediction from a model, but instead how we can provide means to verify or correct algorithmic suggestions, to speed up or make less tedious otherwise manual processes. For this approach it is not needed to completely understand why an ML technique made a specific suggestion, but instead to accept that ML is not always perfect and to provide means to quickly analyze results and adjust them.

With the exception of those images from external authors whose licenses/copyrights we have specified in the respective figure captions, we as authors state that all of our own figures, graphs, and plots in this article (i. e., those not marked) are and remain under our own personal copyright, with the permission to be used here. We also make them available under the Creative Commons Attribution 4.0 International (c b CC BY 4.0) license and share them at osf.io/rhyg4. Users primarily interact in a 2D slice of the microscopy data, colored by the cell's assigned ID, to select a pair of cells to merge. The resulting tree is then represented in a textual form with numbers as cell labels, and it is no interactive. The 3D view (marked 1 and 4) only serves as an additional view and is also not used for interaction. Image © Elise Laruelle, Philippe Andrey, Jean-Christophe Palauqui, and Alain Trubuil, used with permission. They can adjust these divisions by selecting other cells to add to or subtract from the current selection. As they finalize the selection, the tree automatically is extended and a new daughter cell is added below the ovum (Figure 10-2). The biologists then continue this process until no further top-down decisions can be made, with the result that two or more layers at the top have been established (Figure 10-3). After the high-level divisions, biologists can use the ML model to predict a higher level assignment on the current tree. Next, they check the newly formed parent cells, and if they agree with the prediction results, they confirm the assignment. They can also disagree with the assignments and reassign a correct sister for the wrongly assigned cells. For a correction, they double-click the cell that has a wrongly proposed sister. Then, they traverse and single-click all the neighboring cells to check their properties by examining the respective color maps and the shared surfaces. Based on all this information, they then choose the most likely sister based on their experience and knowledge and mark it as such (Figure 10-4). After checking all the pairs in the current level, the biologists then continue to work on the next stage. This way the tree is being completed bottom-up. Thus, when the top part of the tree and the bottom part meet, we automatically merge both to form the final lineage tree (Figure 10-5). Finally, biologists can use the export function to record the assignments in their preferred format.

Appendix E: Data Structure

For each embryo, we start with with a dataset that comprises the 3D geometry (i. e., mesh information and a list of cell IDs) extracted by Avizo from the confocal microscopy data. We clean this data to ensure that every mentioned cell name actually has a geometry assigned to it, ignoring those IDs without associated meshes. We then parse the remaining data and extract all information related to the individual cells such as their centers, volumes, direct neighbors, and surface area. In addition, we prepare another data structure that will later hold the hierarchy information, and that needs to support specific cell orders in each layer based on the ML or manual assignments of sister cells. Moreover, this data structure needs to support the incomplete tree that has information on both ends and grows toward the middle. It gets updated with each cell assignment operation and also needs to handle non-hierarchy cells (supporters).

Finally, we use a third data structure for ML model training and prediction. Because the training records are information about pairs, we calculate, for every potential set of neighbors, the data such as distance and shared surface area, and normalize this information using all possible pairs within the same generation.

Figure 1 :

 1 Figure 1: A 2D illustration of biologists' cell lineage workflow.

Figure 2 :

 2 Figure 2: An ideal 32-cell embryo cell lineage hierarchy, which can be established by reasoning bottom-up and top-down.

Figure 3 :

 3 Figure 3: Screenshot of LineageD, for the example of assigning a 256-cell embryo (in progress). The core elements are the Main 3D View of the embryo, here exploded and with the target cell highlighted, and the Hierarchy Tree of the lineage, which is interactively established by the biologists. The Target and Sister View shows the relative position of the target cell within the whole embryo and its proposed sister. The Thumbnail View of the tree provides the context of the full hierarchy. Finally, the Operation Panel supports further control of the tool.

Figure 4 :

 4 Figure 4: An illustration of our reasoning for how we designed the lineage hierarchy tree representation, based on the traditional hand-written or (ASCII) text-based records used by the biologists.

Figure 5 :

 5 Figure 5: Several interaction techniques in LineageD.

Figure 6 :

 6 Figure 6: Screenshots of the interface in TreeJ (a plugin to Fiji ImageJ) that the experts traditionally use to specify the cell lineage.Users primarily interact in a 2D slice of the microscopy data, colored by the cell's assigned ID, to select a pair of cells to merge. The resulting tree is then represented in a textual form with numbers as cell labels, and it is no interactive. The 3D view (marked 1 and 4) only serves as an additional view and is also not used for interaction. Image © Elise Laruelle, Philippe Andrey, Jean-Christophe Palauqui, and Alain Trubuil, used with permission.

Figure 7 :Figure 8 :

 78 Figure 7: Example cell lineage result produced with TreeJ that shows several development stages (A-D) of the cell in the 3D view and the corresponding text-based lineage hierarchy. Image © Elise Laruelle, Philippe Andrey, Jean-Christophe Palauqui, and Alain Trubuil, used with permission.

Figure 9 :

 9 Figure9: Illustration of the interaction that links selections in the 2D hierarchy tree to the representations in the 3D view, including the highlighting of the selected cells and potential neighbors.

Figure 10 :

 10 Figure 10: Illustration of the typical interaction workflow.

© 2022 The Author(s) Computer Graphics Forum © 2022 The Eurographics Association and John Wiley & Sons Ltd.

Acknowledgments

We thank all participants in our evaluation for their input, the AVIZ team for their general feedback, and Hao Wu for technical advice. The work was partially supported by Inria's NaviScope project.

Images/graphs/plots license/copyright

APPENDIX

Appendix A: Traditional Lineage Specification Tool Figure 6 shows an example for a traditional tool (TreeJ plugin to Fiji ImageJ) to specify the cell lineage. Figure 7 shows the result of specifying the cell lineage for an Arabidospsi thaliana embryo of 122 cells. Figure 8 illustrates the overall traditional process further: we start from slices from confocal microscopy, segment this data, and then the biologists interact in the segmented slices rendered with ID colors to establish levels in the hierarchy.

Appendix B: Further Design Considerations for LineageD

Figure 4 illustrates how we arrived at our final design of the representation of the cell lineage hierarchy. We started from the hand-written (or text-based) records as used by the biologists in the traditional approach, encoded these as a traditional tree representation with the width of each node encoding the respective cell's volume, then elongated cells vertically to indicate the time period in which they existed in this configuration, and then removed the links between the nodes by arranging them in an icicle plot manner. The final step was then the addition of a color scheme and the interaction means as presented in the paper.

Figure 9 illustrates the linked interaction between hierarchy and 3D views in more detail. The first row shows that, upon clicking on a cell without a specified parent, the main 3D view highlights this cell and the target and sister view shows it in context. In contrast, if this cell already has a sister assigned (proposed by the neural network model or manually specified), then this sister is also highlighted in the target and sister view.

Appendix C: Availability of LineageD and Video Figure

The LineageD project website is available at aviz.fr/Research/ CellLineage. Nonetheless, we realize that LineageD runs on our own servers and so we understand that resviewers may be concerned about maintaining their anonymity (even though we do not log any access to the tool; any logs that we used in our experiment were collected on the local machines of the participants). We thus also make a video figure available for reviewers that shows the main functionality and interaction that we discuss in the paper, and plan to make LineageD openly available once our paper has been accepted for publication. We also plan to make the sources available on a public repository such as GitHub to make it possible for others to also run a local copy of LineageD.

Appendix D: Typical Interaction Process

With LineageD, biologists first need to select their dataset of interest from the menu. After the dataset is loaded, we show the whole embryo overview in the main 3D view, one default ovum cell at the top of the tree, and all the leave cells displayed at the bottom of the tree (Figure 10-1). Then biologists can observe the whole embryo, hide the supporters, and try to divide it into parts to make further assignments easier. To do that, they need to target the tree's


Lawrence Fyfe
email: lawrence.fyfe@ircam.fr

Daniel Bedoya
email: daniel.bedoya@ircam.fr

Elaine Chew
email: elaine.chew@ircam.fr

A Nutshell Guide to Annotating Recorded Piano Performances on the Web with CosmoNote

CosmoNote is a web-based citizen science tool for annotating expressive musical structures created by performers during piano performances. For this demonstration, we describe the annotation process in CosmoNote that allows citizen scientists to find and mark these expressive musical structures.

INTRODUCTION

In the course of performing scripted music, a pianist will use expressive variations in tempo and loudness that go beyond the score to communicate emphasis or structure through a performance. Understanding how performers create these structures is the key to comprehending how performers think about the piece and how they convey this thinking to listeners. However, it can be difficult to discern these structures with automated analysis and so, to find these structures in performed piano music, we have created CosmoNote [START_REF] Fyfe | Cosmonote: A web-based citizen science tool for annotating music performances[END_REF], a citizen science annotation tool that asks participants to lend their ears to finding and marking these structures. For this demonstration, we provide an overview of the CosmoNote annotation process for our citizen scientist participants.

THE ANNOTATION PROCESS

The following subsections describe the process for creating annotations in CosmoNote, from creating an account to navigating collections of recorded performances to listening to, visualizing, and annotating these performances.

Creating an Account

To begin the annotation process, participants must first create a CosmoNote account which requires an e-mail address and a password. Account creation is done from the CosmoNote start page 1 . In addition, participants must be at least 18 years old and agree to the user agreement which can be read in full by clicking the link provided. Note that the account and the data associated with that account sit on a secure, remote server.

Selecting a Collection

When participants create an account or log in with an existing account, they will see a list of collections like that shown in Figure 1. After reading the descriptions and deciding which collection they wish to explore, they then click on that collection to start navigating its set of recorded performances.

Navigating the Recorded Performances

Once a participant has selected a collection, they will be shown the recorded performances for that collection, one performance per page, starting with the first one.

Listening to the Recorded Performances

When a participant wants to listen to the audio for a performance they can start with the controls shown in Figure 3. The audio playback can be paused or stopped at any time during playback and, while audio is playing, participants can click on any time point in the corresponding visuals (as described in the next section) and playback will stop and start again from the time point selected.

Visualizing the Recorded Performances

CosmoNote shows musical data visually via superimposed information layers showing waveforms, note and pedal data, and via curves based on extracted feature data. The features include loudness, tempo, and harmonic tension with each type of visual data shown in a distinct layer. Each visual layer can be turned on and off individually with the controls shown in Figure 4a to allow for focus or comparison among the different layers. Figure 4b shows, superimposed, the waveform and note layers along with loudness and tempo curve layers.

Annotating the Recorded Performances

For annotating performances, CosmoNote offers four types of annotations: boundaries, regions, comments, and note groups.

Boundaries

Boundaries represent time points that separate a musical performances into segments of musical ideas. Boundaries have four different strength levels that are indicated visually via opacity and the thickness of lines with both increasing as the level increases.

Regions

Regions delineate entire segments or areas of interest in a performance. Their function overlaps with that of boundaries but regions encompass all of the notes between two boundaries rather than just the boundaries themselves.

Comments

Comments enable annotators to mark elements of interest and to write some text about them. They are meant to provide a means to point at something of interest that is not captured by the other annotation types.

Note Groups

Note groups enable annotators to select a subset of the notes in a recorded performance to highlight those notes as belonging to some expressive structure. To select notes, an annotator drags a rectangle around the notes that they wish to select. Once a note group is created, notes can be individually added to or removed from the group.

TECHNICAL REQUIREMENTS

CosmoNote can be demonstrated either in person or online since it runs in any modern browser. However, with detailed visuals, CosmoNote works best on desktop browsers.

ACKNOWLEDGMENTS

1

  https://cosmonote.ircam.fr/ Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: owner/author(s). Web Audio Conference WAC-2022, December 6-8, 2022, Cannes, France.© 2022 Copyright held by the owner/author(s).

Figure 1 :

 1 Figure 1: A list of collections: training, free exploration, and Glenn Gould's recording of Bach's Goldberg Variations

Figure 2 :

 2 Figure 2: The CosmoNote performance page including navigation, information, controls, and visuals

  Figure 2 shows the second performance in the Free Exploration collection. Participants can navigate through the collection with the backward and forward buttons at the top of the page while information about each recorded performance is displayed below the navigation as shown at the top of Figure 2.

Figure 3 :

 3 Figure 3: CosmoNote's audio controls

  (a) Visual controls for music data and extracted features: audio, notes, pedals, loudness, and tempo (b) Tempo and loudness overlaid on waveform and notes

Figure 4 :

 4 Figure 4: CosmoNote visual layers and their controls

  (a) CosmoNote annotation controls: boundaries, regions, comments, note groups, show/hide annotations, boundary strength, delete selected, delete all, and save (b) CosmoNote annotations: examples of (left-to-right) a boundary, a region, a comment, and a note group

Figure 5 :

 5 Figure 5: CosmoNote annotations and their controls

  This result is part of the project COSMOS that has received funding from the European Research Council under the European Union's Horizon 2020 research and innovation program (Grant agreement No. 788960).