Alexandre Ponomarenko

Olivier Vincent

Amoury Pietriga

Hervé Cochard

Eric Badel

Philippe Marmottant

É Badel

Ultrasonic emissions reveal individual cavitation bubbles in water-stressed wood

Keywords: bubble, nucleation, xylem, acoustic emissions

come

Introduction

Sap ascends in trees through the xylem tissues (wood), a porous structure containing a network of parallel and in-terconnected conduits that are a few dozen micrometers in diameter, using a seemingly hazardous mechanism relying on water under tension [START_REF] Tyree | Xylem Structure and the Ascent of Sap[END_REF][START_REF] Stroock | The Physicochemical Hydrodynamics of Vascular Plants[END_REF]. This mechanism has for long generated controversy because tensed water is metastable: water columns may be seen as "wires of water" [START_REF] Woodhouse | Sap hydraulics[END_REF] as opposed to the "ropes of sand" proposed by early opponents of the cohesion tension theory [START_REF] Darwin | Report of a discussion on the ascent of water in trees[END_REF][START_REF] Brown | The theory of the rise of sap in trees : some historical and conceptual remarks[END_REF]. Physicists have since established that pure liquid water does sustain extremely negative pressures, negative by tens of MPa [START_REF] Herbert | Cavitation pressure in water[END_REF]. Beyond this pressure, water vaporizes by cavitation. Because of the large contact area between the wood cell walls and the water column that greatly favors the bubble nucleation, the risk of heterogeneous cavitation in trees is very high. Cavitation occurs when the sap pressure is abnormally negative, in case of drought for instance, generating an embolism of the sap circulation which can prove to be fatal for the trees [START_REF] Tyree | Vulnerability of xylem to cavitation and embolism[END_REF]. Most trees have a narrow safety margin away from this hydraulic rupture [START_REF] Choat | Global convergence in the vulnerability of forests to drought[END_REF]. If cavitation is beneficial in other context such as the discharge of spores in ferns [START_REF] Noblin | The Fern Sporangium: A Unique Catapult[END_REF] (or harmless in human cracking joints [START_REF] Deweber | Knuckle cracking and hand osteoarthritis[END_REF]), it is now seen as a major player in the determination of tree mortality by drought [START_REF] Brodribb | Hydraulic failure defines the recovery and point of death in water-stressed conifers[END_REF].

However, the physical mechanism of cavitation development in trees remains unknown, the two main hypotheses being "nucleation" and "air seeding". Experimental investigations are still largely constrained by methodological issues. Most methods are invasive and subject to artifacts [START_REF] Cochard | Methods for measuring plant vulnerability to cavitation: a critical review[END_REF], while non-invasive observation tools such as magnetic resonance imaging [START_REF] Holbrook | In vivo observation of cavitation and embolism repair using magnetic resonance imaging[END_REF] or X-ray microtomography [START_REF] Brodersen | The dynamics of embolism repair in xylem: in vivo visualizations using high-resolution computed tomography[END_REF] do not provide fast high resolution images.

An indirect but very attractive approach is to record acoustic emissions from intact wood, as for non-destructive testing of engineering structures [START_REF] Grosse | Acoustic emission testing[END_REF]. Milburn and Johnson were the first to record "clicks" [START_REF] Milburn | The conduction of sap II. Detection of vibrations produced by sap cavitation in Ricinus Xylem[END_REF] in the audible acoustic range, while Tyree and Dixon [START_REF] Tyree | Cavitation Events in Thuja occidentalis[END_REF] evidenced ultrasonic emissions. It is established that ultrasonic emissions are more frequent under water stress conditions [START_REF] Tyree | Vulnerability of xylem to cavitation and embolism[END_REF] and are correlated with embolism patterns [START_REF] Johnson | Leaf xylem embolism, detected acoustically and by cryo-SEM, corresponds to decreases in leaf hydraulic conductance in four evergreen species[END_REF][START_REF] Rosner | Extraction of features from ultrasound acoustic emissions: a tool to assess the hydraulic vulnerability of Norway spruce trunkwood?[END_REF][START_REF] Vergeynst | New type of vulnerability curve gives insight in the hydraulic capacitance and conductivity of the xylem[END_REF] suggesting that ultrasounds may be linked to cavitation. But there is still 1 no direct evidence of the exact origin of ultrasounds, which represents a major limitation for its use in cavitation studies.

Here we investigated the question of the origin of ultrasounds emissions. We developed a new method to record simultaneously both acoustical emission and optical observations of thin wood slices molded in a transparent hydrogel reproducing the live conditions in a tree. This method could also be useful to study any porous media under hydric stress [START_REF] Brochard | Phase transitions of binary mixtures in random media[END_REF].

Experiment

To monitor functional xylem conduits, we cut 50 µm thick slices from fresh xylem samples (Scots pine, Pinus sylvestris L.) cut along the longitudinal axis of the conduits, called tracheids in gymnosperms. This thickness ensured that the sample became transparent to visible light while it still contained several intact conduits. We chose Scots pine species whose wood is made of short tracheids. The intact tracheids lengths we measured in our samples ranged from 20 µm to 1200 µm. The samples were around 5 mm long. This ensured that the samples contained intact tracheids.

This technique using slice samples was used in several studies [START_REF] Lewis | Collapse of Water-Stress Emboli in the Tracheids of Thuja occidentalis L[END_REF][START_REF] Kuroda | Detection of embolism and acoustic emissions in tracheids under a microscope: incidence in diseased trees infected with pine wilt[END_REF] but has the drawback of exposing the sectioned slice to air entry. In order to generate the hydric stress in wood and to avoid air infiltration, we molded the wood slices in polyhydroxyethylmethacrylate (pHEMA), a wet and stiff hydrogel formulated according to Wheeler & Stroock [START_REF] Wheeler | The transpiration of water at negative pressures in a synthetic tree[END_REF]. The hydrogel is porous to water, allowing mass transfer of water from the wood to the hydrogel surface, while preventing air invasion, since the mesh size of the hydrogel is nanometric [START_REF] Wheeler | The transpiration of water at negative pressures in a synthetic tree[END_REF]. The molding in between two half-cured hydrogel slabs is described in Fig. 1 and detailed in the Appendix A. Materials and Methods. Note that the final UV curing did not result in a shrinking of the gel, therefore the gel closely matched the shape of wood. It resulted in a close bounding of the hydrogel with the wood slice, without visible void space in between them where bubbles could propagate (see Fig. 1b).

We placed the sample in a controlled subsaturated atmosphere: evaporation at the surface of the porous hydrogel induced a hydric stress within the sample (Fig. 2a). The level of hydric stress is quantified by the water potential Ψ. At equilibrium, Ψ is directly set by the humidity level: it is 0 for an atmosphere saturated in humidity, and negative for a non-saturated atmosphere. The building of a hydric stress was similarly to what occurs naturally within the leaf of a tree. The atmospheric potential of water vapor equi-librated through the leaf/hydrogel (acting as a membrane) with the liquid contained in the conduits: xylem conduits or artificial conduits as in [START_REF] Wheeler | The transpiration of water at negative pressures in a synthetic tree[END_REF]. This hydric stress was thus transmitted to the liquid contained in the wood inclusion, which resulted in the lowering of the liquid pressure p compared to the atmospheric pressure p 0 (since Ψ = p -p 0 in a pure liquid), until cavitation occurred in the wood conduits, at a water pressure potential expected to be around -3.6 MPa for Scots pine [START_REF] Martínez-Vilalta | Hydraulic adjustment of Scots pine across Europe[END_REF]. Note that the water potential slowly built within the hydrogel sample, because it took time for water to diffuse out of the gel [START_REF] Vincent | Birth and Growth of Cavitation Bubbles within Water under Tension Confined in a Simple Synthetic Tree[END_REF]. The hydrogel was sufficiently stiff to sustain this very large pressure and did not cavitate (rupture) itself. Under evaporation both wood and hydrogel shrunk, and we did not observe the apparition of voids due to detachment of hydrogel from wood in our experiments reported.

During this drought experiment we monitored the inclusion both (i) optically with a camera mounted on a microscope (Fig. 2b) and (ii) acoustically with high sensitivity microphones (Fig. 2c).

Results

During the water stress, air bubbles suddenly appeared in conduits (defining an "optical event", that was detected by image analysis). Then, one air bubble usually took a characteristic time of 4 ± 3 s before it filled the entire conduit. We distinguished two types of optical events: on the one hand, the "nucleation" events, starting in a fully wet area and, on the other hand, the "air seeding" events, here defined as the appearance of bubbles near an already gas filled conduit. The first type of events showed that pre-existing bubbles were not necessary for the start of the cavitation process (Fig. 3ab). The second phenomenon resulted in extending patches of gas (Fig. 3c). This proved that the embolism development in our system was not the gradual propagation of a gas front, but the abrupt appearance of bubbles, and their intermittent development from several nuclei.

A second major finding was obtained from the detection of all "acoustic events" (signals whose amplitude exceeded a defined threshold value) in the ultrasonic range. We investigated what was the precise moment of these acoustic events, zooming on a few conduits and recording with a high-speed camera. Recordings showed that a sound was always synchronized with a bubble appearance at the millisecond time scale, see Fig. 3a. This result clearly indicated that the ultrasound emission was correlated in time with bubble nucleation.

Monitoring entire samples (at 3.75 frames per second)

we found that nearly all acoustic events (118 over a total of 118 and 84 over a total of 87 for samples 1 and 2 respectively) were correlated in time with one specific bubble nucleation with a delay that was less than the time interval between images. This therefore discards other possible origins of sounds, such as wood failures or conduit collapses [START_REF] Tyree | Cavitation Events in Thuja occidentalis[END_REF] at least for our specific case of a conifer species whose conduits are only tracheids.

From the correlation of acoustic events with optical events we obtained the spatiotemporal distribution of emissions, i.e. we could plot the position of the conduits where single ultrasound emissions were detected, obtaining the analog of a seismic map, see Fig. 4A, providing the point of origin of each sound. Plotting the time of filling for each conduit, we clearly observed on Fig. 4B the extending patches of gas propagating by "air seeding".

A striking result was that sound energies were widely scattered (size of circles in Fig. 4A), and that some bubbles do not even produce any recordable ultrasound. For a typical experiment with a millimetric sample the counts were 84 acoustic events and 463 optical events (sample 2). The acoustic recordings synchronized with optical events clearly suggested that the energy of each acoustic event increased with the volume of the corresponding conduit, for two samples (Fig. 5a for sample 1 and Fig. 5c for sample 2).

Discussion

We now interpret the acoustic results, considering the elastic energy that is stored in the conduits. Under the large hydric pressures developed by evaporation, the volumic elasticity of water comes into play. As water is tensed, it is elastically stretched with respect to its initial volume at rest: the larger the water conduit, the higher the storage of total elastic energy. Assuming that always the same fraction of elastic energy could be converted in acoustic energy, the variations during the experiment were thus explained by the variability of the conduit sizes and the degree of water tension at cavitation.

The stored elastic energy can be predicted quantitatively. It depends on the hydric negative pressure p of the liquid. The nucleation of a bubble suddenly relaxes this pressure to a near vanishing value p relax 0. Indeed, after nucleation the liquid pressure is in equilibrium with the gas pressure (neglecting capillary effects), the latter being close to vacuum in a freshly cavitated bubble, before water vapor and then air dissolved in surrounding liquid fills the bubble [START_REF] Vincent | Birth and Growth of Cavitation Bubbles within Water under Tension Confined in a Simple Synthetic Tree[END_REF].

The initial negative pressure p is stretching both (i) the volume of water that is contained in a conduit of volume V c and (ii) the conduit walls. The volume ∆V of the bubble just after nucleation (< 1µs) is therefore the sum of the volumes released by the relaxation of these two stretched volumes. It is such as ∆V /V c = p/K, where K is an effective elastic bulk modulus. Because water and wood volumes are stretched in series by the same pressure, the effective modulus writes K = 1/(1/K water + 1/K c ) [START_REF] Vincent | Birth and Growth of Cavitation Bubbles within Water under Tension Confined in a Simple Synthetic Tree[END_REF][START_REF] Vincent | The fast dynamics of cavitation bubbles within water confined in elastic solids[END_REF], with the compression modulus of water K water (2.2 GPa), and a volumetric elastic modulus K c ∼ G linked to the deformation of the material surrounding the cavity (G being the elastic shear modulus of the solid material surrounding the conduit). Here we followed [START_REF] Hölttä | A model of bubble growth leading to xylem conduit embolism[END_REF] and we used the value of K c = 0.75 GPa, typical of conifers. This results in an effective modulus of K = 0.5 GPa. With this new parameter we can now evaluate the density of elastic energy prior cavitation to be 1/2 × K(∆V /V c ) 2 = 1/2 × p 2 /K. The total elastic energy stored in the cavity volume therefore writes:

E elastic = 1 2 p 2 K V c (1)
The energy stored is a function of the stretching tension p, and the cavity volume V c . Upon bubble nucleation, this elastic energy was partly released into acoustic radiation [START_REF] Vincent | Birth and Growth of Cavitation Bubbles within Water under Tension Confined in a Simple Synthetic Tree[END_REF]. The microphone captured a part of this energy; the remaining energy being emitted in air or absorbed in the different materials (wood, hydrogel and structure). The same experiments were performed on artificial wood with a simpler geometry in order to validate this prediction. This mimetic material consisted of a hydrogel containing spheroidal cavities that were initially filled with water, and then set under hydric stress according to the Wheeler et al. method [START_REF] Wheeler | The transpiration of water at negative pressures in a synthetic tree[END_REF], providing nucleation at -20 MPa. Thanks to this calibration experiment, we showed that over a wide range of cavities volumes the acoustic energy of events was proportional to the water volume, see Fig. 6. Our results thus provide a first mechanistic demonstration of previous observations of Mayr et al. [START_REF] Mayr | Cavitation in dehydrating xylem of Picea abies: energy properties of ultrasonic emissions reflect tracheid dimensions[END_REF], who suggested that the energy of acoustic emissions may correlate with the average tracheid dimensions.

The ultrasound energy therefore mainly reflected the conduit volume, so that smaller volumes produced a sound level that was below the detection threshold value of our microphones. The threshold value was fixed in order to remove the noise. Variations in tension at cavitation (because of different conduit anatomies) or variations in distances to the microphone accounted for the distribution of acoustic energies at a given volume.

In addition, we noted a difference between the cavitation bubbles occurring near intact conduits ("nucleation"), and cavitation bubbles occurring near already cavitated conduits ("air seeding", extending the gaseous patches). A minority of bubbles originate from nucleations (15 %) at earlier times, but on average they were much more likely to produce a recordable emission (41 % against 21%). Our interperation is that the level of liquid pressure to be specified in equation ( 1) is less negative for "air seeding" bubbles. We expect such bubbles to be triggered when the pressure difference between the filled conduit and the cavitated conduit, ∆p = p -p cavitated p, overcomes a threshold value. This threshold value may be less negative on average than the "nucleation" pressure (else we would observe isolated nucleations only). This results in a lower released elastic energy according to equation ( 1), decreasing the likelihood of detecting such acoustic events. Note that two hypothesis remain for the origin of "air seeding" bubbles: (a) air propagates through a pit pore (a few dozens of nanometers [START_REF] Jansen | Plasmodesmatal pores in the torus of bordered pit membranes affect cavitation resistance of conifer xylem[END_REF]) or (b) the pit ruptures and breaks open irreversibly [START_REF] Cochard | Cavitation in trees[END_REF]. Note also that we cannot here detect optically the propagation of air, and a nucleation occurring near a cavitated conduit would be classified as an air seeding event. Last, we must specify that the acoustic signals were similar for both types of bubbles, but nucleations occurred only at the beginning of the experiments.

In the course of time, the average acoustic energy of each bubble clearly increased and then decreased after a maximum value (Fig. 5b for sample 1 and Fig. 5d for sample 2). We can also interpret this climax with the prediction. The stretching tension is expected to initially increase because of the progression of dehydration, which accounts for the energy increase, but the volume of cavitating conduits also decreased drastically (insert Fig. 5bd ), which accounts for the final decrease.

Conclusion

In summary, we proposed a technique to monitor the development of embolism in a slice of xylem wood under hydric stress. The propagation of embolism is found to be the consequence of two abrupt phenomena: nucleation and discontinuous propagation from conduit to conduit (air-seeding). The original combination of acoustic and fast optical microscopy brought insights in the induction of cavitation. Remarkably, we demonstrated that every ultrasound emission in xylem was triggered by a bubble nucleation. This discards other possible origins of sounds, such as wood failure or conduit collapse at least for our specific case of a conifer wood whose conduits are tracheids. Moreover, we showed that the energy we recorded with the microphones was linked to the mechanical elastic energy that was released during the cavitation event. In a living tree, water stressed bark also emits ultrasounds but they are not linked to xylem cavitation [START_REF] Kikuta | Vulnerability curves from conifer sapwood sections exposed over solutions with known water potentials[END_REF]: further experiments are needed to filter out those emissions. The origin of sound in xylem is the consequence of the sudden relaxation of tension by a bubble. The vibration causing the acoustic emission remains to be resolved. One relevant hypothesis is a bubble vibration: indeed within artificial wood in hydrogel it was found that nucleated bubbles oscillate in volume for a few periods, at a MHz frequency linked to the compression modulus of water and the surrounding matrix [START_REF] Vincent | The fast dynamics of cavitation bubbles within water confined in elastic solids[END_REF].

These findings strengthen the ultrasonic approach to monitor cavitation. The results enlightened that the energy of the acoustic signals was linked to the volumes of the cavitating conduits. A promising application opened by this study is that the energy of acoustic events goes through a maximum in the middle of the cavitation process, which proves useful to interpret the cavitation state of a living tree. With a simple and non-invasive ultrasonic sensor, there is now a great potential to follow in situ the effect of severe droughts associated with climate changes on the survival of forests.

Appendix. Materials and Methods

To prepare wood inclusions, 50 µm thick xylem slices were cut with a microtome, tangentially to the surface of the tree (radial plane) allowing most pits connecting conduits to be observed from the side. The fresh and still humid xylem was molded in pHEMA hydrogel, see Fig. 1. The pHEMA hydrogel was built by polymerization of a monomer solution of HEMA. The solution of HEMA was directly inspired from Wheeler et al. formulation [START_REF] Wheeler | The transpiration of water at negative pressures in a synthetic tree[END_REF]: 2-hydroxyethyl methacrylate (65 vol%), ethyleneglycol (6 vol%), methacrylic acid (1 vol%), de-ionized water (28 vol%) with an additional 1 vol% of photoinitiator solution (600 mg 2,2-dimethoxy-2-phenylacetophenone dissolved in 1 ml of n-vinyl pyrrolidone). All chemicals were from Sigma Chemicals. To include the xylem slice in hydrogel, we prepared two halfcured (1 minute in UV light, within a UV chamber, CL 508 from Uvitec Cambridge) hydrogel slabs 400 µm thick, prepared in between two glass slides spaced by a rectangular jig. Then we bound the three layers of gel/xylem slice/gel as shown on Fig. 1 and terminated the polymerization of the hydrogel with UV light in the UV chamber during 3 minutes. The assembly was done with watered slabs, in or-der to avoid trapping bubbles and keep the system always fully wet. Applying a slight finger pressure before curing was required in order to ensure a good binding and to avoid detachment during drying (as observed with unsuccessfully bound samples). Artificial wood was prepared as described in [START_REF] Wheeler | The transpiration of water at negative pressures in a synthetic tree[END_REF], by bubbling Argon gas in lightly cured monomer solution, resulting in spheroidal water-filled cavities with a wide range of sizes. Both artificial wood and wood inclusion were soaked for at least 24h in deionized and degassed water in order to dissolve potential bubbles.

The samples were held directly in between two highfrequency microphones (PICO-HF 1.2, bandwidth 0.5-1.85 MHz, from Physical Acoustic Corporation) facing each other, using a thin layer of coupling elastomer (Tensive gel, from Parker laboratories) to insure a good acoustic coupling between the gels and the microphones. The clamping configuration of the microphones avoided any contact with other surfaces and thus avoided frictional noises. The sample and microphones were placed in a closed chamber made of two Petri dishes and a home-made centimetric spacer, all being wrapped with parafilm. The hydric stress was imposed by a saturated KCl solution in the bottom dish, ensuring a relative humidity of 85 %, corresponding to a water potential of around -22 MPa at equilibrium [START_REF] Greenspan | Humidity fixed points of binary saturated aqueous solutions[END_REF]. This imposed hydric stress was well below the one causing 50 % reduction in hydraulic conductance of Scots pine (-3.6 MPa [START_REF] Martínez-Vilalta | Hydraulic adjustment of Scots pine across Europe[END_REF]), and was thus never reached in the conduits because of cavitation.

The samples were optically monitored under a zoom stereo-microscope with a camera (AVT Marlin F-421B, imaging at 3.75 Hz). Image analysis helped to detect appearance of bubble, by looking at the differences between images separated by around 1 second. The volume of conduits was estimated from the conduit length as measured on images, multiplied by the cross-section area. To follow the fast nucleation of bubbles, we used a fast camera (Miro 4, Vision Research, operating at 1000 frames per second).

An acquisition card (USB AE Node, Physical Acoustics) recorded the microphone signals during 700 µs when the acoustic pressure exceeded a threshold value of 4 mPa. The acoustic energy received by a microphone is given by E mic = S mic p(t) 2 ρpc dt, with S mic the microphone area in contact with the hydrogel, p(t) the acoustic pressure received by the microphone (using the calibration chart to convert voltages into Pascals), ρ p = 1274 kg/m 3 and c = 2000 m/s the density and the speed of sound in the hydrogel. The speed of sound within the hydrogel was measured with pure hydrogel samples of thicknesses e ranging from 0.3 to 1.8 mm, and placed inside a water tank and probed in echo mode with ultrasound pulses generated by a piezotransducer connected to a pulser-receiver (model 5073PR from Panametrics). To calculate cavities volumes, we considered them to be spherical. The radius is deduced from the pictures where we fit an ellipse to the observed cross section of the cavities and take the mean of the two ellipse axes as the radius.

Fig. 1 :Fig. 2 :

 12 Fig. 1: Inclusion of wood samples in a stiff hydrogel matrix: the slices of xylem are inserted underwater in between two half-cured hydrogel slabs. (a) schematic views of the inclusion after assembly and total curing of hydrogel slabs (dashes indicated the assembly). (b) Optical observations. Above, a cross-section cut performed with a scalpel clearly shows that the hydrogel is intimately bound to the wood. Intact channels are still in contact side by side. Below: top view of the wood inclusions.

Fig. 3 :Fig. 4 : 3 energy

 343 Fig. 3: (a) Bubble length as a function of time (red symbols, measured on some selected of images). The timing of the detected sound is indicated with the blue arrow. (b) Nucleation of a bubble in a conduit, recorded with a highspeed camera (1000 frames per second). The red line indicates the bubble length. (c) Air-seeding event, when a bubble nucleates near an already cavitated conduit, recorded with a 4 Hz camera. The evolution is sketched below each images, to increase the readability.

2 Fig. 5 :

 25 Fig. 5: (a) Variation of the acoustic energy correlated with bubble appearance as a function of the conduit volume, in xylem. The blue squares indicate the logarithmic average in a bin whose size is specified by the horizontal blue line. The vertical blue line represents the standard deviation of the points in the bin. The grey symbols show individual events. "Nucleations" are highlighted by dark symbols. The dashed line is a guide to the eye showing the microphone threshold. (b) Energy in the course of time. Insert: volume of each cavitating conduit in the course of time. For the figure and the insert, t = 0 s corresponds to the first acoustic event and the error bars are determined in the same way as in (a). The sample for (a) and (b) is the one presented in Fig.2 and Fig.4. (c) and (d): same graphs for a second sample.

Fig. 6 :

 6 Fig.6: Synthetic wood consisting in spheroidal cavities in hydrogel: acoustic energy as a function of the volume of artificial cavities. A picture of the experiment is shown in insert. To calculate cavities volumes, we considered them to be spherical. The radius is deduced from the pictures where we fit an ellipse to the observed cross section of the cavities and take the mean of the two ellipse axes as the radius.

Fig. 7 :

 7 Fig. 7: Supplementary material, Movie 1. Animated version of figure 4 (a) played a hundred times faster, with cavitation sounds. The original acoustic emissions (in the ultrasound range) are slowed down (1000 x) to become audible.

Fig. 8 :

 8 Fig. 8: Supplementary material, Movie 2. Animated version of figure 4 (b), with cavitation sounds, same parameters as Movie 1.

Acknowledgments

We would like to thank A. Stroock for fruitful discussions on hydrogels, S. Rosner, H. J. Schenk and K. Charra-Vaskou for pointing useful references, Camille Lucas for drawing illustrations, K. Ferhat for the optimization of the molding, J.-C. Vial for discussions on the optical visualization of wood, and J. Etienne for critical reading. The authors acknowledge financial support from Agence Nationale de la Recherche (grant Microacoustics ANR-08-JCJC-014-01) and Institut Rhônalpin des Systèmes Complexes (IXXI).


Hugo Romat

Nathalie Henry Riche

Ken Hinckley

Bongshin Lee

Caroline Appert

Emmanuel Pietriga

Christopher Collins

ActiveInk: (Th)Inking with Data

Keywords: CCS CONCEPTS, Human-centered computing → Touch screens, Visualization systems and tools, • Applied computing → Annotation, Digital pen, ink, visualization, active reading

During sensemaking, people annotate insights: underlining sentences in a document or circling regions on a map. They jot down their hypotheses: drawing correlation lines on scatterplots or creating personal legends to track patterns. We present ActiveInk, a system enabling people to seamlessly transition between exploring data and externalizing their thoughts using pen and touch. ActiveInk enables the natural use of pen for active reading behaviors, while supporting analytic actions by activating any of these ink strokes. Through a qualitative study with eight participants, we contribute observations of active reading behaviors during data exploration and design principles to support sensemaking.

INTRODUCTION

Pen and touch technology with high-resolution displays, such as the Microsoft Surface Studio or Apple iPad Pro, are becoming mainstream. These devices offer much potential for knowledge workers, affording large workspaces to handle multiple charts and documents, and natural interactions to manipulate and annotate them. Visualization researchers have started to explore this technology in systems tailored for pen and touch interactions (e.g., SketchInsight [START_REF] Lee | SketchInsight: Natural data exploration on interactive whiteboards leveraging pen and touch interaction[END_REF], Viz-Dom [START_REF] Crotty | Vizdom: Interactive analytics through pen and touch[END_REF], DataInk [START_REF] Xia | DataInk: Direct and Creative Data-Oriented Drawing[END_REF]). But these previous efforts have focused much less on how to enable people to externalize their thoughts as they engage in sensemaking activities.

Previous studies showed that a key affordance of analog pen and paper is to externalize one's thoughts and capture fleeting ideas via annotating content and taking notes [START_REF] Marshall | Annotation: From Paper Books to the Digital Library[END_REF][START_REF] Mehta | Metatation: Annotation as Implicit Interaction to Bridge Close and Distant Reading[END_REF][START_REF] Riche | As We May Ink?: Learning from Everyday Analog Pen Use to Improve Digital Ink Experiences[END_REF][START_REF] Sellen | The Myth of the Paperless Office[END_REF]. Externalizing thoughts has many cognitive and social benefits [START_REF] Kirsh | Thinking with external representations[END_REF] such as lowering working memory load, supporting idea reformulation, and providing common ground to share insights with others. These affordances are particularly important for sensemaking and data exploration tasks, as people need to keep track of numerous insights found while browsing through the data, which typically involves many visualizations and documents [START_REF] Pirolli | The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis[END_REF].

We contribute ActiveInk, a system that enables people to actively engage and think with data. It allows for a seamless transition between interacting with data and documents, and externalizing thoughts using a digital pen (Figure 1). We designed and evaluated two methods of working with ActiveInk. First, prefix provides pen modes and bimanual interaction to allow for switching between ink pens and action pens. Second, postfix provides ink for all pen strokes, and enables people to activate any ink stroke to perform analytic actions. For example, an annotation made at one point during the exploration process, such as underlining sentences in a document or circling a region in a map, may be activated at any time to filter, highlight, or color the underlying data. All actions propagate to all visualizations and documents in the workspace.

A study comparing these two approaches to a baseline of ink + touch (no actions) revealed benefits and drawbacks to each approach. We also observed that annotating data and taking ink notes were a common part of the analysis process across all conditions, but that ink actions, when available, supplanted some forms of annotation in active reading. Natural disasters can and do happen: worst places to be in the U.S.

===== States Affected By Tornados

Tornados have been documented in every single state in the U.S. but "Tornado Alley" is considered the area with the highest risk for these devastating natural disasters.

The states that comprise Tornado Alley include Kansas, Nebraska, South-Dakota and Oklahoma as well as the northern parts of Texas,western portions of Iowa and the very southern region of Minnesota. Southern California has a higher risk factor as well.

===== Areas Hit by Hurricanes

Areas most at risk for hurricanes include the southern Gulf Coast tip of Texas, most of Louisiana and all of the state of Florida. The state of Hawaii is also at great risk. ===== Regions at Risk for Earthquakes . . . Besides California Earthquakes aren't exclusive to California. While the Natural Disaster Risk report reveals that high risk regions include large portions of the Golden State, the entire western seaboard and western portions of Nevada are also at a high risk for earthquakes. Informed by the study results, we expanded ActiveInk with a hybrid interface, providing pen modes as well as a magic ink pen, which provides ink by default, but ink strokes can be activated. We enhanced the analytic capabilities of ActiveInk to provide new functions such as activating a sketched correlation line to trigger a regression or hatching an interval on an axis or legend to filter corresponding values. The resulting version of ActiveInk recognizes handwritten content as people take notes, allowing for words and shapes to be searched, filtered, and retrieved to facilitate revisitation.

In summary, our work contributes the following: (1) A novel set of pen-enabled actions for interacting with visualizations, documents, and images in a coordinated workspace;

(2) A comparison of inking, prefix actions and postfix actions for exploratory analysis tasks; and (3) A hybrid design for pen + touch analysis which provides features specifically targeting the types of operation needed to support thinking during exploratory data analysis. These contributions bring people closer to their data, enabling them to interweave reading and acting on data. ActiveInk provides for seamless annotation, note-taking, and invocation of analytical operations, without even putting the pen down.

RELATED WORK

This research sits at the intersection of information visualization and human-computer interaction, building on previous designs of pen and touch interfaces for visualization, general digital pen approaches for annotation and note-taking, and findings about the role of externalizing knowledge in solving data analysis tasks.

Pen and Touch Visualization Systems

Visualization researchers have recognized the potential of leveraging input modalities beyond mouse and keyboard in the past decade [START_REF] Lee | Beyond mouse and keyboard: Expanding design considerations for information visualization interactions[END_REF]. Advances in display technology and, in particular, the ability to offer simultaneous pen and touch on high-resolution displays have made possible new approaches to interact with visualizations.

There have been several approaches to design a coherent set of touch gestures for interacting with visualizations on tablets [START_REF] Drucker | TouchViz: A Case Study Comparing Two Interfaces for Data Analytics on Tablets[END_REF][START_REF] Jo | Touch-Pivot: Blending WIMP & Post-WIMP Interfaces for Data Exploration on Tablet Devices[END_REF][START_REF] Rzeszotarski | Kinetica: Naturalistic Multi-touch Data Visualization[END_REF][START_REF] Sadana | Designing and implementing an interactive scatterplot visualization for a tablet computer[END_REF][START_REF] Sadana | Designing Multiple Coordinated Visualizations for Tablets[END_REF], but these do not fully take advantage of the different inking capabilities. Visualization systems such as SketchInsight [START_REF] Lee | SketchInsight: Natural data exploration on interactive whiteboards leveraging pen and touch interaction[END_REF], PanoramicData [START_REF] Emanuel Zgraggen | PanoramicData: Data analysis through pen & touch[END_REF], and Viz-Dom [START_REF] Crotty | Vizdom: Interactive analytics through pen and touch[END_REF] offer novel experiences leveraging pen and touch interactions for data analysis tasks. The research focus of these systems is on the design of interactions to create and interact with visualizations. Pen interactions often support two modes: free-form note-taking and predefined gestures to invoke commands. For example, sketching a particular shape (e.g., a circle) creates a specific type of visualization (e.g., a pie chart) in SketchInsight; drawing a line between two visualizations activates coordination through brushing and linking in PanoramicData. These interactions require learning and remembering a potentially large set of gestures.

Although free-form inking supported by these systems enables analysts to externalize their thoughts as they interact with the data, none of these ink marks are leveraged for further interaction.

A recent system, DataInk [START_REF] Xia | DataInk: Direct and Creative Data-Oriented Drawing[END_REF], enables people to sketch arbitrary shapes on a canvas and bind their visual properties (color, size, thickness) to data. While this research focused on creating expressive visualizations for storytelling and art purposes, it demonstrates one way to leverage the unique strengths of using a digital pen for visualizing data. We aim to provide more access to these affordances in ActiveInk.

Unique Affordances of Digital Pens

A recent study by Riche et al. [START_REF] Riche | As We May Ink?: Learning from Everyday Analog Pen Use to Improve Digital Ink Experiences[END_REF] compiled and compared activities of people with an analog pen and using mainstream digital pen devices. They extracted a set of pen affordances, several of which are unique to digital pens. A key affordance of digital pens for working with visualizations is the accuracy for direct pointing and dragging, providing the ability to compose precise selections by sketching complex shapes on top of visualizations, while reducing the occlusion caused by touch. Digital pen systems have other capabilities beyond analog pens, most notably the ability for the same pen to perform different functions in quick succession.

Multiple affordances of an analog pen also appear to transfer to a digital pen [START_REF] Riche | As We May Ink?: Learning from Everyday Analog Pen Use to Improve Digital Ink Experiences[END_REF] such as externalizing thoughts and capturing fleeting ideas. Research in human-computer interaction studied and leveraged these phenomena, in particular, for making annotations during active reading of documents [START_REF] Matulic | Supporting active reading on pen and touch-operated tabletops[END_REF][START_REF] Mehta | Metatation: Annotation as Implicit Interaction to Bridge Close and Distant Reading[END_REF][START_REF] Bill N Schilit | Beyond paper: Supporting active reading with free form digital ink annotations[END_REF], taking notes [START_REF] Hinckley | InkSeine: In Situ Search for Active Note Taking[END_REF], gathering information [START_REF] Hinckley | Informal information gathering techniques for active reading[END_REF][START_REF] Tashman | LiquidText: A Flexible, Multitouch Environment to Support Active Reading[END_REF], generating ideas [START_REF] Li | Sketch-Comm: a tool to support rich and flexible asynchronous communication of early design ideas[END_REF], and sharing ideas with others [START_REF] Zhao | skWiki: A Multimedia Sketching System for Collaborative Creativity[END_REF]. However, we are not aware of works which specifically focus on studying and supporting externalization using a pen for data analysis and sensemaking scenarios.

Externalization in Visualization

Externalizing thoughts through annotations on content, taking notes, and drawing diagrams has multiple benefits when engaging in long and complex tasks [START_REF] Kirsh | Thinking with external representations[END_REF]. Externalizing thoughts enables people to limit their working memory load [START_REF] Scaife | External Cognition: How do Graphical Representations Work?[END_REF], articulate and reformulate thoughts which can lead to substantial improvements in understanding and retention [START_REF] Mueller | The pen is mightier than the keyboard: Advantages of longhand over laptop note taking[END_REF], or share their thinking with peers to generate a shared object of thoughts to support communication and decision-making [START_REF] Cherubini | Let's Go to the Whiteboard: How and Why Software Developers Use Drawings[END_REF]. Indeed, Kidd hints that the act of annotation itself creates knowledge in the knowledge worker, which may be more important to the process than the marks that remain on the page [START_REF] Kidd | The Marks Are on the Knowledge Worker[END_REF].

These benefits are particularly relevant for visual analytics and sensemaking activities. The sensemaking process is a loop [START_REF] Pirolli | The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis[END_REF] involving phases of analysis and interpretation of visualizations and documents to gain insights, and phases of revisitation to compare and contrast these insights to generate and investigate hypotheses. Externalizing thoughts is critical to facilitate revisitation and keep track of insights and hypotheses, especially over long periods of time.

Recent research started to study active reading of visualizations. Walny et al. [START_REF] Walny | Active Reading of Visualizations[END_REF] conducted a series of studies on active reading with both printed and digital visualizations. They demonstrated that active reading behaviors previously observed on textual documents (e.g., underlining, writing notes in margin) occurred when people interpreted visualizations, and that such behaviors transferred in the digital world. While they did not consider complex sensemaking activities involving multiple visualizations and documents, their results suggest that inking with a pen on or around visualizations is an activity that people naturally do when working with visualizations (and given the ability to do so). This work builds upon their findings and reveals externalization strategies people employ during sensemaking activities using a digital pen on interactive data visualizations.

ACTIVEINK

Our goal with ActiveInk is to empower people to think with data. We aim at leveraging the digital pen to enable people to interweave high-precision interactions performed on visualizations and documents with externalizing their thoughts during the exploration process, as they would do with a regular pen on documents and blank paper. In the following, we define note as ink separated from data views, annotations as ink on data views and actions as ink that changes the visual appearance of data.

ActiveInk does not target expert data analysts as they would likely require extensive computations, analytic functions, and the ability to handle large amounts of data (better supported by systems such as Vizdom [START_REF] Crotty | Vizdom: Interactive analytics through pen and touch[END_REF]). Rather, we seek to exemplify a more fluid sensemaking experience than what exists today for knowledge workers and data enthusiasts. We designed ActiveInk to primarily support thinking as people make sense of their data. We describe below the general principles that guided our design, and ActiveInk's key interface components.

Design Principles

We followed three driving principles to maximize opportunities for people to externalize thoughts while retaining the ability to interact with visualizations, minimizing the physical and cognitive costs of switching between the two.

D1. Support sensemaking through interaction with a set of heterogeneous information.

While many visualization systems focus on interacting with data visualizations, our goal is to support the broader activity of sensemaking, integrating documents and images as well. This activity involves extracting and cross-referencing information from multiple sources. Thus, ActiveInk should provide people with space to think [START_REF] Andrews | Space to Think: Large High-resolution Displays for Sensemaking[END_REF], i.e., support viewing multiple of these assets at once and enable individual spatial organizations, where space can be given meaning.

D2. Use the digital pen for interacting with data and for externalizing thoughts.

A pen is more precise than a finger and offers a more natural interaction than a mouse for tasks such as drawing. Composing a selection in a visualization often relates to drawing as it involves the creation of complex arbitrary shape enclosing a set of points while avoiding others. Having the pen already in hand also maximizes the opportunities for externalizing thoughts (annotation, note-taking). Thus, we designed a set of pen actions to enable analysis. To reduce the cost of switching between analysis and externalization, we propose two interface strategies (Figure 2), which operate on an infinite canvas:

(1) prefix: leveraging bi-manual interaction, in which people use their non-dominant hand to select an action for the pen they hold in their dominant hand; actions are directly performed as the pen is used; (2) postfix: the pen always lays ink on the screen; tapping ink strokes a posteriori with a finger reveals a menu of actions to perform on the data associated with that stroke. Activated strokes remain on canvas, allowing to recall and undo local actions at any time.

D3. Avoid requiring memorization of gestures.

Previous research on active reading and annotation has shown that the form of annotations varies between individuals, and even across tasks within individuals. A circle in one place may not serve the same annotative function as a circle in another, though similar annotations in spatio-temporal proximity often have the same intent [START_REF] Mehta | Metatation: Annotation as Implicit Interaction to Bridge Close and Distant Reading[END_REF]. Learning a collection of gestures may be difficult and recalling them may disrupt the flow of annotation. Instead, we aim to provide complete freedom of pen inking, and to support actions on the ink strokes that people naturally make.

Unified Infinite Canvas

ActiveInk is built around a zoomable infinite canvas, on which people can drop visualizations, documents, and images, which we collectively refer to as views, and rearrange them. The ability to lay out the workspace, reorganize and manipulate views was previously shown to be an important component of active reading of visualizations [START_REF] Walny | Active Reading of Visualizations[END_REF].

Freeform inking is provided anywhere on the canvas and on views alike. Ink on a view stays with the view when moved. Interaction with the canvas and views uses twofinger pinch to zoom. Single finger slide moves views or pans the canvas.

To provide powerful analysis across visualizations, the views imported into the ActiveInk canvas are tightly coupled. Each view responds to operations such as filter or highlight performed on the others. To maximize usability and minimize the cognitive effort of tracking coordination, we chose to coordinate all views of the data on the canvas. The data is loaded from tabular data file formats such as CSV, and is stored in-memory as a JSON data structure. In this first prototype we do not use a visualization toolkit, but rather rely on a set of basic, common chart types implemented from scratch in our environment for the sake of easy customization. Ac-tiveInk currently supports binned histograms, scatterplots, maps, and text documents. We selected this subset for the diversity of data, interactions, and insights they can offer. However, adding additional types of visualizations such as line charts or density plots is straightforward.

Pen Actions

ActiveInk supports the set of operations on views illustrated in Figure 3. The following scenario, illustrated in Figure 4, describes the pen actions of ActiveInk and how one can use these operations for sensemaking using the prefix strategy.

Emma is a data journalist writing a piece about the best states in the U.S. to start a family. She has collected data about each state, such as school quality, air pollution, and median home price, as well as documents about this topic. She organizes the most interesting views in her canvas, moving them with a finger and pinching them to resize. She annotates salient insights with ink as she encounters them. For example, she notes that the school quality index mostly correlates with math scores in Grade 8. She uses cut to slice the binned histogram of house prices, as the distribution is skewed. Providing a healthy environment being critical, she decides to remove all states that have higher air pollution by switching to the remove pen and using it on dark regions of the map. Data is removed from the entire canvas, and the scales in scatterplots and histograms are updated in response. Since school quality is an important factor to raise a child, Emma opts to hide the lower quadrant of the scatterplot, deemphasizing those states in all views. She then chooses a blue color to highlight the higher portion of happiness score. She notices that one of them, Vermont, pops up in the document and thus reads the corresponding paragraph. To identify all data about Vermont, she switches to a purple color to paint and label it from the text. Since Emma wants to have more details on house prices in Vermont, she uses the copy instrument and lassos the interval containing Vermont. She sees Vermont is in the lower range, concluding that it is the place to be! Prefix Interface: Switching between Types of Pen The prefix interface offers the pen actions described above as different types of instruments people can acquire with the pen (Figure 3). This paradigm of different modes of interaction is usually found in mainstream interfaces today, although modes generally appear in a menu or ribbon positioned at the top. In contrast, ActiveInk presents it on the non-dominant hand side of the workspace to afford bimanual Thumb+Pen interaction [START_REF] Pfeuffer | Thumb + Pen Interaction on Tablets[END_REF], in which a person taps on modes with the thumb of one hand while using the pen in the other hand. This design may lower the cost of interaction for mode switching [START_REF] Appert | Context matters: Evaluating Interaction Techniques with the CIS Model[END_REF].

The ActiveInk prefix interface uses a paradigm familiar to most people and provides a comprehensive view of all actions at all times, which is important for non-expertss [START_REF] Scarr | Improving Command Selection with CommandMaps[END_REF]. This paradigm is also efficient for accomplishing actions in batches. However, people need to decide on mode before performing actions, and interweaving different types of activities may induce numerous mode switches. Forgetting to switch modes may result in mistakes to undo later (e.g., people want to rapidly jot down thoughts next to highlighted points in a scatterplot but the pen highlights more points instead). Action pens in prefix lay down dashed ink to mitigate mode errors, and the interface includes buttons for global undo and redo to correct mistakes. The clear pen is used to clear all visual properties of data elements resulting from prior interactions (highlight, paint, etc.), affording some level of local undo.

Postfix Interface: Activating Ink Strokes

The postfix interface offers the pen actions described above through an in-place menu invoked by tapping an ink stroke (Figure 3, right). Activated ink strokes remain on canvas and can be deactivated to undo the action. We hypothesized that this design would stimulate externalizations as the pen always lays down ink first without requiring to switch back to this mode after interacting with the data.

We organized the actions in the menu along three semantic axes, inspired by the design of the ZoomCatcher menu [START_REF] Xia | WritLarge: Ink Unleashed by Unified Scope, Action, & Zoom[END_REF]. Once the menu is invoked by tapping a stroke, one can preview the effects of any action before executing it by sliding their finger from the central ink action to any other menu item. Lifting the finger triggers the action and fades out the ink stroke to limit visual clutter. Note that actions that are unavailable (e.g., due to no underlying data to highlight) are greyed out. Erasing an activated (faded-out) stroke undoes the action and reverts the stroke to a regular ink (that can potentially be activated again, or fully erased). This mechanism allows for local undo of actions during the exploration.

The postfix interface uses a paradigm that is less familiar to people, and certainly has a steeper learning curve, as all actions are not visible at all times. Given the additional Below are a few of our favorites:

==== Vermont: Hey now Vermont! Sounds like you guys got it all figured out up there in your little nook in the woods, huh?

Vermont is a great place for families. In fact, it's not even close. Vermont is the best place for education in the country, and it's also the 2nd safest.

The only weird thing is, only 24% of households are actually families.

==== Maine: Well, it's starting to look like the northeast is the best place in America for families to live. Is that a surprise? It isn't to us.

Maine is the safest state in America. While there are some misfits that cause issues in Maine, you're more likely to have a moose gun stolen out of your Subaru than you are to be attacked while ice fishing.

==== Kansas: Such a great state that a band named itself after it. cost of tapping after each ink stroke, this paradigm may slow down actions performed in batches. However, it enables the smooth interweaving of interactions with data and ink externalizations without mode switch (e.g., paint a set of points and hand write the reason for this action next to them). Additionally, actions can be carried out at any time: an underline, arrow, or circle made early during an initial read of the data can later be activated by tapping, thus turning externalizations into potential interaction locations. Since activated strokes remain on the canvas, we also hypothesized that they would provide additional benefits for analytic provenance -assist people in recalling their process to gain insights on the data. Finally, the postfix interface has fewer tool icons, and thus is less cluttered.

Implementation

ActiveInk is built as a web-based application in JavaScript and runs on a NodeJS server. The vector graphics canvas is implemented using Paper.js [START_REF] Lehni | [END_REF] and the visualizations are created with d3.js [START_REF] Bostock | D3 Data-Driven Documents[END_REF] and BubbleSets.js [START_REF] Collins | Bubble Sets: Revealing Set Relations with Isocontours over Existing Visualizations[END_REF]. To best associate ink strokes with data, ActiveInk tests for data item enclosure or intersection. This allows for marks such as circles and arrows to appropriately associate with the intended data.

QUALITATIVE STUDY

To gain insights on the externalization process during sensemaking using a digital pen, we conducted a qualitative observation study. We designed the procedure to investigate the following research questions:

Q1. How do externalizations manifest during sensemaking with digital inking?

Our first question is whether results from Walny et al.

[37] will transfer to more complex tasks involving extracting and comparing insights from multiple visualizations and documents to make sense of the data.

Q2. How do externalizations manifest during sensemaking with ActiveInk?

We seek to observe if and how annotation and note-taking behaviors change when offered the possibility to interact with the data to visually encode it (e.g., highlight). In particular, a key question is whether the cost of switching mode between interaction with data (e.g., highlight, filter) and inking in prefix may discourage spontaneous externalization of thinking. In contrast, while postfix may encourage such externalization, it uses a less familiar interaction paradigm where actions are not visible at all time, which may prove more difficult to use.

Study material and screenshots of canvases created by participants are available as supplemental material.

Study Design

We used a within-participants design as prior work on activereading [START_REF] Marshall | Annotation: From Paper Books to the Digital Library[END_REF][START_REF] Mehta | Metatation: Annotation as Implicit Interaction to Bridge Close and Distant Reading[END_REF][START_REF] Walny | Active Reading of Visualizations[END_REF] indicated that individual variability would be high given the diverse range of strategies for sensemaking and behaviors for externalization. Thus, participants experienced the following three interfaces in a single session.

Ink. The condition explored sensemaking using only digital ink and touch. Participants could browse through a set of views (visualizations and documents), adding and organizing them in the workspace with touch. To inspect underlying data, participants could tap and hold with a finger on a visual element to display its label and data values in all views. Inking was available anywhere on the canvas, including on views. This condition served as our baseline, enabling us to build knowledge on active reading and externalization behaviors without any analytic functions linked to pen interactions.

ActiveInk Prefix and Postfix. In the second and third conditions, participants used ActiveInk with prefix and postfix interfaces, as described above. Inking and touch capabilities were also provided as in the ink condition. As we hypothesized an order effect (due to fatigue or learning), we counterbalanced the presentation order of these two conditions. Participants also had to complete different scenarios in each of these two conditions. Assignment between a scenario and a condition was counterbalanced across participants in order to avoid observing an effect of the dataset instead of observing an effect of the ink condition.

Participants

We recruited 8 participants (2 females, 6 males; 7 in 30s and 1 in 20s) from a large software company (over 60000 employees), screening them to cover different age ranges, genders, and roles (team managers, administrative assistants, developers, data scientists, researchers). We screened participants for at least one year of experience with a pen-enabled device and recent experience reading and creating simple charts from data. Only P5 reported data analysis is not part of his/her job. We piloted the study with 2 people to streamline the training and fix minor usability issues.

Method

We used a Microsoft Surface Studio, a 28" screen with a resolution of 4500×3000 pixels, supporting multitouch and pen input. Participants were encouraged to adjust the orientation and incline of the device for comfortable pen and touch interactions. The study was held in a quiet room; experimenters observed live, from behind a space divider, using a camera. Data and Task. As our goal was to observe externalization during sensemaking, we designed high-level tasks involving browsing through a set of views, making hypotheses, and investigating queries to understand the data. We prepared 3 similar analytic scenarios based on identifying a subset of states in the U.S. In the ink condition, participants searched for states where tobacco and alcohol use was correlated to accidents and overdoses. In the prefix and postfix conditions we counterbalanced across two scenarios: identifying the best and worst states to retire, or start a new job and raise a family. A different multivariate dataset was provided for each scenario, containing factors such as nursing home quality (retirement) and home prices (new job). Examples appear in Figures 1 and4. Participants were instructed to imagine they were conducting research to write an article about their findings, and should create a canvas that would enable them to recall these findings in several weeks. To limit the duration of the session, we selected a subset of visualizations and documents (9 histograms, 9 scatterplots, 3 documents, and 3 maps) for each exploration task.

Procedure. An experimenter first instructed participants to sign a consent form and fill out a demographics questionnaire. The study was structured in three phases, one per condition. For each condition, the experimenter briefly described the interface and dataset at the start of the training. To cover the interaction mechanics, the experimenter instructed participants to perform each action available in the interface (pen and touch), explaining the interaction and resulting effects, and informing participants of efficient interaction techniques if needed (e.g., bi-manual interactions). This training lasted about 10 minutes. Since ActiveInk features a large number of actions, these conditions included an additional 5 minute practice task, in which participants completed a brief analysis task while asking questions and receiving experimenter guidance as needed.

After training, participants completed the main data exploration task for a maximum of 15 minutes. After each task, participants completed a questionnaire and had a brief interview with the experimenter about their likes, dislikes, and wishes. Participants were instructed to take a 5 minute break after the second condition. After experiencing all conditions, participants answered a preference questionnaire and verbally explained the advantages and drawbacks of each technique. The entire experiment lasted two hours. Participants received a $150 gift card as compensation.

Data Collection. We logged pen interactions, and used a screen video capture and video-audio recordings. At least two experimenters were present for each session: one person interacted with the participant while the other took notes.

Ink

Results

Figure 5 depicts the interactions of each participant over the course of each exploration task. All canvases created in the study are in supplemental material.

Ink. In the first condition, six participants used inking extensively for making sense of data (Figure 5). The largest portion of ink across participants are notes separated from data (78.0% of strokes), as opposed to direct annotations on views (22.0%). Participants exhibited different behaviors: P1 and P5 laid down most ink, however, P1 made both notes and annotations while P5 only took notes. While P4 and P8 did not ink as much as others, their ink mostly consisted of annotations. Figure 6 depicts externalizations in the ink-only condition. It is interesting to note that two participants (P1,P4 in Figure 6) used inking to coordinate multiple subsets of data permanently on the canvas. These behaviors correspond to the paint, highlight, and label actions of ActiveInk. We also observed the depiction of hypotheses about the data (e.g., drawing a correlation line) and externalized queries (e.g., drawing a threshold). Both of these annotations are illustrated in Figure 6-P1.

Most of the notes on the canvas were phrases summarizing findings, hypotheses, or describing the sensemaking process itself. However, we also saw instances of pictorial representations such as arrows (6 participants), or lines on the canvas to divide the workspace into regions (P3, P5). We also observed participants visually link a pictorial annotation in a visualization to a separate handwritten note. For example, P1 drew a box in red around the word "threshold" referencing the line sketched with the same pen in both scatterplots (Figure 6 Externalizations with ActiveInk. In contrast to the ink-only condition, the amount of regular ink in both ActiveInk conditions decreased, with fewer annotations. The prevalence of notes did not decrease as much: 8% lower in prefix and slightly higher in postfix. The total number of annotation + action strokes was consistently between 19-26% across all conditions. This may indicate that actions serve the role of some annotations in ActiveInk. While there were fewer annotations than in the ink condition, we noted several interesting types of annotations created after operating on the data: (1) depicting how views relate after applying cut or copy (e.g., copying a subset of a view), (2) depicting insights (e.g., correlation between two data dimensions), (3) identifying data of interest (e.g., circling), and (4) characterizing data of interest (e.g., good vs. bad candidates). In the ink condition, we saw annotations to connect specific data points across multiple views. We did not observe this type of annotation with ActiveInk since actions such as highlight and paint fulfilled this role.

The ink stroke log analysis (Table 1) indicates that postfix encourages more annotation than prefix. In observing the use of prefix, we saw that participants chose a pen and repeatedly made short strokes on items one by one, whereas with postfix they more commonly used enclosures to select groups of items, then activated the ink.

The most frequent action performed across both ActiveInk conditions was highlight (187), followed by label (55), paint (50), filter [START_REF] Scarr | Improving Command Selection with CommandMaps[END_REF], cut (6), copy (3), hide (3), and remove (2). Video analysis shows that cut and remove were primarily used on skewed distributions and outliers in scatterplots and histograms. In the prefix condition, the clear function was used to remove actions 47 times, and undo was used 39 times. Undo and clear were not provided in postfix as ink actions were locally reversible at all times. All participants commented that ActiveInk analytic functions on data were useful in contrast to the ink condition.

While a higher fraction of strokes were used for notetaking in postfix, the note-taking behavior was consistent in both ActiveInk conditions. For example, P7 wrote hypotheses on top of the canvas, interacted with data visualizations to investigate them, and wrote answers next to original notes in both conditions. Similar to the ink condition, participants created colored legends (e.g., Figure 7-P2) in both ActiveInk conditions. Overall, we observed meaning assigned to color in 75% of all exploration tasks across participants and conditions. We observed that participants generally followed one or more of these strategies: 1) write the data dimension name or value, 2) use the same color to mark the data subset, and 3) use a pictorial representation (e.g., underline, check mark) to connect notes in one place to data in different visualizations.

Sensemaking strategies with ActiveInk. We observed two different strategies that reflect advantages of each interface (Figure 7). P2 (Figure 7-P2) seamlessly interweaved paint action on data item, and in-place annotation with the same color to create a pictorial legend about this data and value. Child-Teacher Ratio Vermont is a great place for families. In fact, it's not even close. Vermont is the best place for education in the country, and it's also the 2nd safest.

The only weird thing is, only 24% of households are actually families. Child-Teacher Ratio Child-Teacher Ratio He sketched glyphs such as check marks and thumbs up to indicate positive aspects, and sad faces and thumbs down to indicate negative aspects. This strategy worked well with postfix as the pen "automatically" switched to ink after performing an action on the data. In contrast, P6's strategy was to do actions in batches (Figure 7-P6). Using the ink pen, P6 first handwrote a legend to associate certain data values to positives and negatives, then switched to the highlight pen to perform highlights in batches, only switching colors. This strategy worked well with prefix, as it did not require tapping strokes, but rather pick a single type of pen to apply successive strokes on the views. We observed differences between participants but each followed a personal sensemaking strategy, generally remaining consistent across conditions (Figure 5). For example, P1 used inking extensively in the first condition and continued to take more notes than others in prefix and postfix while most of the annotations were replaced by pen actions. In contrast, P4 mostly relied on annotations in ink condition, which were replaced by pen actions (mostly highlight) in ActiveInk.

State Population Air pollution Happiness

Most participants (6/8) attempted to lay out visualizations and documents on the canvas so they were all in view without requiring zooming or panning. P7 made use of the larger surface, storing handwritten notes on hypotheses and results out of screen. Two participants used a lower corner to place temporary visualizations. P8 used the histograms as one would use dynamic queries in conventional coordinated view interfaces, as a way to filter the data according to different attributes. P8 is also the only one who leveraged multiple tabs to segment the workspace by different questions investigated (e.g., one canvas for identifying the best states where to retire, another for the worst ones).

Ratings and Preferences. Participants praised the quality of both interfaces. For example, P1 said "I really like it, it's the optimal for this type of task really. " (referring to the general principle of ActiveInk, independently of prefix or postfix).

Note that while our questionnaire did not have an option to rate as equal, three participants verbally indicated that they did not have a clear favorite. Overall, six participants preferred prefix and two preferred postfix. In prefix, six participants used bimanual interaction, five of whom preferred prefix over postfix overall.

Results of the subjective experience questionnaire administered after each condition revealed similar ratings for all conditions, although postfix rated slightly higher than prefix on most measures. Most (6/8 in prefix and 8/8 in postfix) felt efficient with ActiveInk. Participants made more notes and annotations with postfix, while with prefix, they carried out more actions. In prefix, at least 3.3% were mode errors (actions on the canvas) (Table 1). Although, note that we could not detect mode errors resulting from accidental inking. Four participants specifically mentioned mode errors as frustrating, e.g., P3: "I keep just forgetting which one I'm in, " P7: "When I was highlighting and wanted to go back to my notes I forgot to click here. "

In postfix, the menu was activated without previewing or applying any action 6 times, which we also counted as errors. We also observed three participants experiencing usability issues with the postfix menu. In particular, P1 said: "I would like something faster, like tap and choose rather than tap, wait for animation, and drag to choose. " While dragging on the menu was intended to preview action results before executing them, these previews were not frequently used (only six instances from three participants). Several requested to see all available actions at once. This may be especially important during the learning period as participants are not familiar with the semantics of the actions available and thus do not fully comprehend their organization in the menu.

Wishes. Three participants mentioned the potential of a hybrid interface, integrating the best of both approaches. P3 said, "Can you make something where some people can choose the tool first and other people can select things and change them?" while P9 suggested using physical buttons on the pen to activate the most common actions. P5 suggested the ability to see the underlying data in a table, and to be able to change data values directly in the interface. P8 commented on the need for transient pen highlights "I would like to drag my ink across the thing and 'boom' things light up." In addition to local undo provided by erasing an activated ink in postfix, P2 and P6 suggested a global undo button would also be helpful. P2 and P6 also suggested functions to align or snap views together for neater canvases. Finally, both P1 and P5 suggested sketching dividing lines on the canvas to create different sets of coordinated views (in addition to different tabs).

Summary and Discussion

Our results suggest that the results of Walny et al. on active reading of one visualization [START_REF] Walny | Active Reading of Visualizations[END_REF] transfer to sensemaking tasks involving multiple visualizations and documents. When ink actions are available, people favor visually marking the data using actions rather than annotating with regular ink, though note-taking was present in all conditions. We observed that several annotations were used to add information not built into the action tools (e.g., sketched glyphs, semantic use of color). Annotations were reused by all participants in describing their findings and strategies to the experimenter, though we did not specifically test for the benefit of inking for recall. While we believe there are sensemaking benefits resulting from the low cost of externalizing thoughts with digital inking, more investigation is needed to check whether this behavior also happens in conventional systems using mouse and keyboards.

Most participants used notes, physically separated from the data, to record their questions and hypotheses, and evidence to confirm or reject them. Participants linked their handwritten notes to data by either rewriting the data dimension name or value, using the same ink color as used for data highlights, or using a pictorial representation (e.g., a check mark, an arrow). These techniques reflect the creation of ad-hoc visual indices to connect and cross-reference related information from different sources. This manual and iterative process may explain why most participants attempted to keep everything in one view, and suggests the need to better support management of large canvases or multiple pages of information.

Implications for Design

As evidenced by the ink stroke analysis, the prefix and postfix interfaces lead to different behaviors: prefix supported rapid sequential actions, while postfix better integrated with annotation and note-taking. This implies a better solution would be to support both paradigms. Thus, we created a hybrid version which offers the same set of pens available in the prefix interface, as well as the ability to activate strokes laid down with the ink pen and marker using the postfix activation menu. As before, activated ink strokes remain on the canvas, constituting a visual trace of the action performed, and a means to undo the action later on.

The observations we made about note-taking during sensemaking with ActiveInk prompted us to add a 4 th design principle:

D4. Make externalizations useful. We observed that, as people explore data, they took notes separated from the data visualizations or documents to capture their hypotheses or summarize their findings. These ink strokes contained explicit references to data dimensions or values through textual content, color, or associated shape to enable cross-referencing insights in multiple sources (notes, data visualizations, and documents). ActiveInk should leverage these externalizations for further analysis.

Advanced Actions to Support Sensemaking

The result of our qualitative study informed a set of 6 new functions depicted in Figure 8, inspired by our observations and comments from our participants.

Compute statistics: Sketching hypotheses about statistical properties of the data directly on the visualization is common (e.g., sketching a correlation line in a scatterplot). While laying ink is a mean to quickly mark potentially important insights discovered serendipitously while exploring the data, getting back to these and assessing their correctness is important when drawing conclusions about the data. For this reason, we added the ability to compute such statistics by tapping on a stroke and selecting the statistics action (i.e., adding a new options in the postfix menu). Note that Ac-tiveInk currently supports computing a correlation line in scatterplots and averaging the values of a set of data items in histograms as we mostly observed these behaviors in the study.

Interactive legends: We also observed participants attempting to draw on the map legend (and sometimes tap it with a finger). We incorporated the ability to sketch on the legend to perform actions (highlight, paint, label, hide, etc.) on the corresponding data elements.

Ink lenses: Study participants surfaced the need for transient and interactive pen interactions akin to brushing in conventional data visualization systems. We added the ability to create and activate any set of strokes on empty canvas with functions such as highlight or paint. Moving these activated strokes with a finger over views immediately performs the corresponding action on the underlying data, effective acting as lenses on the data [START_REF] Tominski | A Survey on Interactive Lenses in Visualization[END_REF] and acting as dynamic queries [START_REF] Shneiderman | Dynamic Queries for Visual Information Seeking[END_REF] when activated with filtering.

Recognizing ink content: Since we saw many references to data dimensions and values in the notes participants took, we added handwriting recognition to ActiveInk. This capability turns handwritten notes into content coordinated with the views. In particular, ActiveInk now offers the ability to tap a written word and perform an action on related data. This capability is inspired from the behavior we saw participants exhibit when referring to data dimensions or values in their notes to connect them to the data. For example, one would observe states with low unemployement, identifyng Texas and California. Highlighting Texas to further investigate its values along other dimensions, they would make a note to get back to California later. ActiveInk facilitates this process as one would simply tap the written word California, select paint from the postfix menu and see the corresponding data items in visualizations and text using the same ink color. Note that handwriting recognition is performed with the MS Ink Analysis API [START_REF]Ink Analysis API[END_REF].

Search similar ink shapes: Since we observed people draw the same glyph on different views, we added the ability to search for similar sets of strokes making up a glyph. When a search is triggered on a glyph, ActiveInk retrieves similar sets of strokes and brings them, and their underlying content, close to the location of the search action. This facilitates cross-referencing and may prove useful for a large canvas. Strokes are associated with glyphs using a spatiotemporal distance function and threshold, while glyphs are associated with each other using the $N-recognizer [START_REF] Anthony | N-protractor: A Fast and Accurate Multistroke Recognizer[END_REF].

Show and edit underlying data: Participants requested the ability to see the underlying data table for a subset of data, and one participant suggested she would like to edit the data directly. We added the show data action to reveal the underlying data table for items associated with a pen stroke. When writing numbers in the table, the written values are recognized and the corresponding data items are updated across all views.

CONCLUSION AND FUTURE WORK

ActiveInk represents a foray into active reading applications for heterogeneous data including visualizations, maps, and documents. It blends the operations of externalizing thoughts with analytic actions in a seamless combination of functions to be carried out with digital pen+touch. The hybrid interface of ActiveInk integrates two ways of working with pen actions, which our study revealed to be complementary for sensemaking activities. Prefix inking is useful for batch operations, while postfix ink activation is useful for interleaving multiple actions with externalizing thoughts.

This work offers several opportunities for extension. The ink recognition system we used does not distinguish between classes of marks, such as enclosures, underlines, and arrows. Our data selection model associates marks with any data items enclosed or intersected by an ink stroke. Distinguishing types of mark could be interesting to implement a finer inkdata association strategy.

ActiveInk was made to demonstrate the possibilities of thinking with ink, but does not scale to a large number of views or with very large volumes of data. Addressing scalability issues requires further development, and implementing the ability to save, recall, and share analysis canvases. It also implies adding support for a wider diversity of charts, which could be achieved by moving to, e.g., Vega-lite [START_REF] Satyanarayan | Vega-lite: A grammar of interactive graphics[END_REF].

To better understand the impact of ActiveInk on sensemaking and externalization, including any potential effects on the depth and quality of insights, we would like to run a longitudinal study in which participants analyze their own data using the hybrid interface. We suspect performance may improve with long-term use, and sensemaking strategies may change as people feel more engaged with their data. Furthermore, it would be interesting to study the use of ActiveInk in collaborative synchronous and asynchronous sensemaking scenarios.

Figure 1 :

 1 Figure 1: ActiveInk affords smooth transition between using a digital pen for high-precision selections of data and for externalizing thinking via notes and annotations. Ink strokes are leveraged to perform operations on underlying data.

Figure 2 :

 2 Figure 2: Different sequences of interactions to perform an action such as paint on the data underlying the ink stroke.

Figure 3 :

 3 Figure 3: Visual effects produced by our 8 different types of pens in prefix (a). In postfix (b), tapping an ink stroke reveals the activation menu. Sliding a finger from the center to an action previews its effect, while lifting the finger triggers it.

1

  Visual saliency of data elements that one can increase with Highlight and Paint , and decrease with Hide and ultimately Remove . 2 Display data about the underlying elements with Label displaying the name of data entities and their values in histograms. 3 View manipulations that do neither affect the data nor propagate to other views, including cut and copy .

Figure 4 :

 4 Figure 4: Depiction of the canvas of a data journalist organizing their workspace and annotating findings (1-ink , 2-cut , 3-remove ) and interacting on the data to identify salient subsets (4-hide , 5-highlight , 6-paint and label , and 7-copy a subset of a histogram).

Figure 5 :

 5 Figure 5: Inking and action sequences per participant across conditions. Each block reflects a 15 minute analysis task. Participants carried out the most annotations in the ink condition, followed by postfix. The most common operations were highlight, paint, and label, which provide details on the data. Split bars in postfix indicate ink strokes used for multiple actions. Note that P8-postfix logs were not saved due to a technical error.

Figure 7 :

 7 Figure 7: Examples of participants' externalizations during sensemaking with ActiveInk, including batch highlighting with prefix on the left, and interwoven actions and note-taking with colors using postfix on the right.

  , stable family sometimes requires moving to a new state. Below are a few of our favorites: ==== Vermont: Hey now Vermont! Sounds like you guys got it all figured out up there in your little nook in the woods, huh?

Figure 8 :

 8 Figure 8: Advanced actions added to ActiveInk, informed by results of our qualitative study.

Table 1 :

 1 Ink stroke analysis. Errors are data action on empty canvas.