Mihail Terenti
email: mihail.terenti@usm.ro

MANSiD Mintviz Lab

Radu-Daniel Vatavu
email: radu.vatavu@usm.ro.

Measuring the User Experience of Vibrotactile Feedback on the Finger, Wrist, and Forearm for Touch Input on Large Displays

We examine vibrotactile feedback delivered on the finger, wrist, and forearm with the goal of enriching the experience of touch input interactions with public displays. We focus on understanding the user experience of such interactions, which we characterize with a wide spectrum of UX measures, including subjective perceptions of the enjoyment, efficiency, input confidence, integration between touch input and on-body vibrations, distraction, confusion, and complexity of vibrotactile feedback for touch input with public displays. Our empirical findings, from a controlled experiment with fourteen participants, show positive and favorable perceptions of vibrotactile feedback as well as a significant preference for feedback on the finger compared to the wrist and forearm.

INTRODUCTION

Vibrotactile feedback accompanying touch input on mobile devices is known to reduce error rates during touch target acquisition [START_REF] Brewster | Tactile Feedback for Mobile Interactions[END_REF][START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF], improve user performance for text entry [START_REF] Mark | Tactile Feedback for Predictive Text Entry[END_REF][START_REF] Hoggan | Investigating the Effectiveness of Tactile Feedback for Mobile Touchscreens[END_REF] and dwell mode switching [START_REF] Liao | Dwell+: Multi-Level Mode Selection Using Vibrotactile Cues[END_REF] tasks, lower the cognitive demand of touch input interactions [START_REF] Brewster | Tactile Feedback for Mobile Interactions[END_REF][START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF], and even deliver new sensing capabilities to the device, such as identification of the touching fingers [START_REF] Bianchi | Disambiguating Touch with a Smart-Ring[END_REF][START_REF] Masson | WhichFingers: Identifying Fingers on Touch Surfaces and Keyboards Using Vibration Sensors[END_REF]. Moreover, vibrotactile feedback has been employed as an effective technique to increase the accessibility of mobile devices for users with visual impairments [START_REF] Grussenmeyer | Accessible Touchscreen Technology for People with Visual Impairments: A Survey[END_REF][START_REF] Tekli | Evaluating Touch-Screen Vibration Modality for Blind Users to Access Simple Shapes and Graphics[END_REF][START_REF] Tennison | Establishing Vibration-Based Tactile Line Profiles for Use in Multimodal Graphics[END_REF]. Unfortunately, public interactive displays do not offer native support for vibrotactile feedback and, consequently, cannot deliver the usability benefits of haptic sensations accompanying active touch on mobile devices. Although prototypes of large haptic surfaces [START_REF] Ezgi Emgin | HapTable: An Interactive Tabletop Providing Online Haptic Feedback for Touch Gestures[END_REF][START_REF] Ledo | The HapticTouch Toolkit: Enabling Exploration of Haptic Interactions[END_REF][START_REF] Marquardt | The Haptic Tabletop Puck: Tactile Feedback for Interactive Tabletops[END_REF][START_REF] Seifert | Adding Vibrotactile Feedback to Large Interactive Surfaces[END_REF] have been demonstrated in research laboratories, the technology has not yet been incorporated in public touchscreen displays.

However, mobile and wearable devices, such as smartphones, smartwatches, and fitness trackers, are prevalent and their users already familiarized with the language of vibrotactile feedback employed by these devices to deliver "click" and "springy" feelings [START_REF] Poupyrev | Tactile Interfaces for Small Touch Screens[END_REF] during touch input. Thus, delivering vibrotactile feedback on the user's body, e.g., on the finger wearing a smart ring or the wrist wearing a smartwatch, may be a feasible design option for augmenting public touchscreen displays with vibrotactile feedback, especially in the context where a niche market is forecast for wearables for authentication and payment at public terminals, e.g., smart rings for cashless payments [START_REF] Takezawa | Smart Rings Seen as New Frontier for Cashless Payments[END_REF]. In this work, we are primarily interested in the user experience (UX) of how distal vibrotactile feedback, physically decoupled from the point of touch on the display feels to users. To this end, we report results from a controlled experiment with N=14 participants and a diversified set of UX measures designed to collect subjective perceptions regarding vibrotactile feedback delivered on the finger, wrist, and forearm during touch input with two, horizontal and vertical, large displays. Our findings show a positive experience in terms of perceived enjoyment, efficiency, confidence, and integration between touch input and vibrations at various locations on the interactive hand as well as low perceived distraction, difficulty, confusion, and complexity for the vibrotactile feedback accompanying touch input. These results open the way towards designing cross-device, wearable-surface user experiences for interactions with public displays.

RELATED WORK

Wearable devices come in many form factors to address a variety of goals [START_REF] Shrestha | An Offensive and Defensive Exposition of Wearable Computing[END_REF] and ways to manage information [START_REF] Billinghurst | Wearable Devices: New Ways to Manage Information[END_REF], from applications in health monitoring and fitness tracking to communication and entertainment to tasks traditionally performed on other types of computers, e.g., writing documents from the smartwatch [START_REF] Nebeling | WearWrite: Crowd-Assisted Writing from Smartwatches[END_REF]. In addressing such diverse needs, designers can leverage a specific output modality for meaningful interactions with wearables: by being close to or in contact with the body, wearables can effectively deliver feedback via vibrations and other forms of haptics [START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF][START_REF] Liu | ThermoCaress: A Wearable Haptic Device with Illusory Moving Thermal Stimulation[END_REF][START_REF] Zhang | PneuMod: A Modular Haptic Device with Localized Pressure and Thermal Feedback[END_REF].

A few works have considered vibrotactile feedback for wearables to accompany interactions with other devices, such as the smartphone. For instance, Schönauer et al. [START_REF] Schönauer | Touch, Movement & Vibration: User Perception of Vibrotactile Feedback for Touch and Mid-Air Gestures[END_REF] examined vibrotactile feedback on the upper arm for gestures performed in mid-air to interact with the smartphone, and focused on aspects of user perception when vibrotactile feedback was physically decoupled from the smartphone. Vatavu et al. [START_REF] Vatavu | Digital Vibrons: Understanding Users' Perceptions of Interacting with Invisible, Zero-Weight Matter[END_REF] used vibrations on the index finger to create the illusion of "holding" digital content from the smartphone, e.g., a photograph, after being pinched on the screen, is taken out of the smartphone and brought into the physical world, where it manifests through vibrations of various duration and intensity. Le et al. [START_REF] Le | Ubitile: A Finger-Worn I/O Device for Tabletop Vibrotactile Pattern Authoring[END_REF] introduced Ubitile, a ring device for vibrotactile feedback during interactions with tabletops. Henderson et al. [START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF] showed that distal vibrotactile feedback, delivered to the user's nondominant wrist by a smartwatch, achieved similar performance characteristics in terms of time and error rate for touch target acquisition on a smartphone as under-the-finger vibrotactile feedback provided by the smartphone. Other work has examined substitution of audio with vibrotactile feedback for various applications, e.g., in healthcare, with devices designed for the upper arm [START_REF] Cobus | Vibrotactile Alarm Display for Critical Care[END_REF] and the wrist [START_REF] Rossa | Multiactuator Haptic Feedback on the Wrist for Needle Steering Guidance in Brachytherapy[END_REF].

Researchers have also explored designs of large displays that can natively deliver haptic feedback. For instance, HapTable [START_REF] Ezgi Emgin | HapTable: An Interactive Tabletop Providing Online Haptic Feedback for Touch Gestures[END_REF] is a multimodal interactive tabletop that integrates electromechanical and electrostatic actuation delivered by four piezo patches mounted at the edges of the tabletop. Poupyrev et al. [START_REF] Poupyrev | Haptic Feedback for Pen Computing: Directions and Strategies[END_REF] presented a haptic display for pen-based interaction, where the pen was augmented with vibrotactile feedback. Jansen et al. [START_REF] Jansen | Mudpad: Fluid Haptics for Multitouch Surfaces[END_REF] introduced a multitouch input device enhanced with active haptic feedback generated by varying the viscosity of the fluid located underneath the display using a magnetic field. Bau et al. [START_REF] Bau | TeslaTouch: Electrovibration for Touch Surfaces[END_REF] employed a customized 3M Microtouch panel to study users' perceptions of electro-vibration and reported frequency and amplitude thresholds for which feedback could be discriminated on the touch surface. Yamamoto et al. [START_REF] Yamamoto | Electrostatic Tactile Display with Thin Film Slider and Its Application to Tactile Telepresentation Systems[END_REF] explored small electrostatic tactile displays for VR systems.

Although new display prototypes have integrated creative solutions towards large display haptics, their technology is not available at scale and the displays from public places do not offer the possibility for haptic feedback. In this context, turning to wearables as a form of computing that is becoming mainstream, e.g., about one-in-five Americans use a smartwatch or fitness tracker as of 2020 [START_REF] Vogels | About One-in-Five Americans Use a Smart Watch or Fitness Tracker[END_REF], may represent a feasible design solution to augment public interactive Manuscript submitted to ACM displays with vibrotactile feedback on the fly. In this work, we are interested in the user experience of such a technical solution to complement results from prior work [START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF][START_REF] Schönauer | Touch, Movement & Vibration: User Perception of Vibrotactile Feedback for Touch and Mid-Air Gestures[END_REF] that focused on usability and user performance aspects for vibrotactile feedback delivered by wearables during touch and gesture input. To this end, we connect to UX reports that evaluated touch input and vibrotactile feedback for interactions with mobile devices with a diversity of UX measures, including preference rankings [START_REF] Brewster | Tactile Feedback for Mobile Interactions[END_REF][START_REF] Koskinen | Feel-Good Touch: Finding the Most Pleasant Tactile Feedback for a Mobile Touch Screen Button[END_REF][START_REF] Park | Tactile Effect Design and Evaluation for Virtual Buttons on a Mobile Device Touchscreen[END_REF], adjective ratings [START_REF] Park | Tactile Effect Design and Evaluation for Virtual Buttons on a Mobile Device Touchscreen[END_REF], and measures of perceived enjoyment, comfortability, and feeling of pressing physical buttons [START_REF] Koskinen | Feel-Good Touch: Finding the Most Pleasant Tactile Feedback for a Mobile Touch Screen Button[END_REF], on which we capitalize in the design of our experiment.

EXPERIMENT

We conducted a controlled experiment to measure aspects of the user experience of how vibrotactile feedback feels when delivered by a wearable device at various locations on the interactive arm during touch input with a large display.

Participants

A number of 14 participants (10 male, 4 female), representing young adults aged between 19 and 34 years old (M=25.0, SD=4.1), were recruited via convenience sampling. One participant was left-handed. All of the participants were smartphone users, five (35.7%) were also using tablets on a regular basis, eight (57.1%) were using smartwatches or fitness trackers, and four participants (28.6%) reported using smart earbuds. A number of eight participants (57.1%) had the keyboard vibration feature turned on on their smartphones.

Apparatus

We implemented an interactive map application using Leaflet, 1 a popular open-source JavaScript library for interactive maps. At startup, the application zooms into a city map with pinpoints shown for selected targets, e.g., the Museum of Contemporary Art of Barcelona; see Figure 1, left and middle. When the user touches a target, a popup window presents a brief description about the target and vibrotactile feedback is delivered to the user's interactive arm by a wearable device that we prototyped for our experiment; see Figure 1, right. The wearable incorporated three 10mm DC coin vibration motors2 into a velcro band with adjustable length so that the motors could be affixed comfortably to the index finger, the wrist, and the forearm, as shown in Figure 1, left and middle. (An alternative design approach would have been to implement three devices, one for the finger, one for the wrist, and one for the forearm, but the participants would have had to don and doff them repeatedly when moving from one experimental condition to the next; see the next subsections for details about the task.) The vibration motors were commanded by a CH340G NodeMcu V3 board based on the ESP8266 Wi-Fi module, 3 a self contained System-on-a-Chip with integrated TCP/IP protocol stack, which we programmed to communicate with the map application via the WebSocket4 protocol. Vibrations were delivered by each motor independently for a fixed duration of 150ms, a value determined empirically during our technical prototyping so that vibrations could be felt unambiguously on various parts of the arm. We ran the map application in the Google Chrome web browser (full screen mode) of two touchscreen displays: an horizontal 46-inch Ideum Platform5 (1920×1080 pixel resolution, 12ms touch response time, integrated CPU Intel Core i7-4790S 3.2GHz, RAM 16GB DDR3) and a vertical 55-inch Samsung UE55D display 6 with a CY-TD55LDAH touchscreen overlay 7 (1920×1080 pixel resolution, 13ms touch response time, connected to a Dell laptop with an Intel Core i5-4300U 2GHz processor, RAM 8GB DDR3); see Figure 1, left and middle.

Design

Our experiment was a within-subjects design with the following two independent variables:

(1) OnBody-Location, nominal variable with four conditions: finger, wrist, and forearm, specifying locations on the interactive arm where vibrotactile feedback was provided during touch input with the large display, and none, representing the condition where vibrotactile feedback was absent.

(2) Display-Orientation, nominal variable with two conditions: horizontal and vertical.

While OnBody-Location examines the effect of wearing a device, Display-Orientation covers potential effects of the presentation of the large display. The dependent variables are represented by several UX measures described in detail in Subsection 3.5.

Task

Participants were presented the two displays and asked to engage in a neutral task (they made a drawing in Windows Paint) to become familiarized with how touch input felt on each display. Afterwards, they were presented the wearable prototype and briefed about the interactive map application and the specifics of the task. For each combination of OnBody-Location × Display-Orientation, a different city map was presented and participants were asked a question about the targets indicated with pinpoints on the map, e.g., "What is the year of the oldest building?" To answer the question, participants had to touch all of the targets, which revealed their descriptions. Vibrotactile feedback was delivered during touch input by the wearable device (Figure 1, right) affixed to the dominant hand. The order of Display-Orientation was randomized per participant as was the order of the finger, wrist, and forearm vibrotactile feedback conditions of OnBody-Location for each display. The first condition for each display was always touch input without vibrotactile feedback, which was our control condition. On average, tasks were completed in 13.1 minutes (SD=7.7) with participants performing on average 187.9 touches (SD=51.5). At the end of the experiment, participants filled in a questionnaire with UX measures designed to collect their experience; see next.

Measures

We collected participants' experience of vibrotactile feedback delivered at various locations on the interactive arm with several UX measures representing dependent variables in our experiment. Most of the measures were collected with 5-point Likert-scale ratings encoding participants' level of agreement, from 1 "Strongly disagree" to 2 "Disagree," 3 "Neither agree nor disagree, " 4 "Agree, " to 5 "Strongly agree, " with various statements involving words with positive and negative connotations, respectively:

(1) Perceived-Enjoyment, in response to the statement "Interacting with the display felt more enjoyable with vibrotactile feedback than without. "

(2) Perceived-Distracteness, in response to the statement "Interacting with the display felt more distracting with vibrotactile feedback than without. "

(3) Perceived-Efficiency, "Interacting with the display felt more efficient with vibrotactile feedback than without. "

(4) Perceived-Difficulty, "Interacting with the display felt more difficult with vibrotactile feedback than without. "

(5) Perceived-Confidence, in response to the statement "Vibrotactile feedback made me feel more confident when interacting with the display compared to when vibrations were absent. " (6) Perceived-Confusion, "Vibrotactile feedback created confusion for me when interacting with the display. " (7) Perceived-Integration, "Touching the display and feeling the vibrations integrated well into one experience. " (8) Perceived-Complexity, "Vibrotactile feedback was unnecessarily complex when interacting with the display. "

Besides these measures focused on the perception of vibrations, we also elicited preferences, expectations, and desirability for vibrotactile feedback when interacting with a large display: [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF] Overall-Preference, a score from 1 (denoting the most preferred condition) to 9 (the least preferred) for each combination of OnBody-Location × Display-Orientation, including no vibrotactile feedback.

(10) Overall-Perception. We asked participants to provide three words that best described their experience with each combination of OnBody-Location × Display-Orientation, e.g., "Please use three words to describe your experience of vibrotactile feedback on the [finger] while interacting with the [horizontal] display. " [START_REF] Grussenmeyer | Accessible Touchscreen Technology for People with Visual Impairments: A Survey[END_REF] Expected-Meaningfullness collects participants' expectations for meaningful vibration patterns, i.e., patterns that are more complex than simple on/off vibrations. We measured Expected-Meaningfullness using a 5-point Likert-scale by eliciting agreement or disagreement with the statement "I would like for the vibrotactile feedback to be more complex than a mere vibration to have more information about the target that I am touching on the display. " [START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF] Expected-Implementation collects participants' preferences for the display to implement vibrotactile feedback by using their personal wearable devices, represented by a ring, watch, and armband, e.g., "I would see myself using vibrotactile feedback delivered to [my smartwatch] when interacting with a public display. "

Overall, we collected twelve UX measures spanning a wide spectrum of perceptions and expectations for vibrotactile feedback delivered by wearables during touch input with a large display.

Statistical Analysis

We report medians as preferred measures of central tendency for our ordinal variables. To analyze the ordinal data from our two-factor experiment, we employ ANOVA with the Aligned-Rank Transform procedure implemented with ARTool [START_REF] Wobbrock | The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures[END_REF]. 8 For post-hoc pairwise comparisons, we employ the ART-C algorithm [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF] of ARTool.

RESULTS

We report findings about our participants' experience with vibrotactile feedback. Overall, our results show that vibrotactile feedback was preferred to no feedback, and that different locations where vibrations were delivered on the interactive arm led to differences in the experience felt during touch input with the large displays.

Vibrotactile vs. No Vibrotactile Feedback

We asked participants to rate, on a scale from 1 (most preferred) to 9 (least preferred), the combined conditions of Display-Orientation × OnBody-Location, including the case of no vibrotactile feedback. We did not find a statistically significant effect of Display-Orientation on Overall-Preference (𝐹 (1,91) =0.004, 𝑝=.951>.05, 𝑛.𝑠.), but we detected a significant effect of OnBody-Location (𝐹 (3,91) =12.362, 𝑝<.001) of large size (𝜂 2 𝑝 =.29). Both the finger (Mdn=1) and wrist (Mdn=4) conditions of vibrotactile feedback were preferred to no feedback (Mdn=6) with large (𝜂 2 𝑝 =.26) and medium (𝜂 2 𝑝 =.11) effect sizes, respectively, but not forearm (Mdn=5, 𝑛.𝑠. at 𝛼=.05). These results indicate vibrations on the finger as a strong candidate for augmenting the user experience of touch input on a large display. Next, we provide a detailed analysis of the three conditions involving vibrotactile feedback.

Vibrotactile Feedback on the Finger, Wrist, and Forearm

Figure 2 shows participants' overall perception of vibrotactile feedback delivered on the finger, wrist, and forearm, and Table 1 highlights statistically significant effects detected for a significance level of 𝛼=.05. Next, we discuss each effect in detail.

Display orientation.

Except for the Perceived-Enjoyment measure (𝑝=.041), the orientation of the display did not influence participants' perceptions of the experience they felt with vibrotactile feedback. Participants reported enjoying vibrations more when interacting with the horizontal display (Mdn=4) than the vertical one (Mdn=3) with a medium to small effect size (𝜂 2 𝑝 =.06).

Location of vibrotactile feedback on the body.

The location where vibrations were delivered to the interactive arm influenced significantly the experience of touch input as detected by six of our eight UX measures of perception: Perceived-Enjoyment (𝜂 2 𝑝 =.28), Perceived-Efficiency (𝜂 2 𝑝 =.42), Perceived-Confidence (𝜂 2 𝑝 =.28), Perceived-Integration (𝜂 2 𝑝 =.33), Perceived-Confusion (𝜂 2 𝑝 =.13), and Perceived-Distracteness (𝜂 2 𝑝 =.21); see Table 1. Post-hoc contrast tests (Bonferroni corrected) revealed statistically significant differences between vibrations on the finger compared to the wrist and forearm, but no difference between wrist and forearm. Overall, vibrotactile feedback on the finger was preferred in terms of Perceived-Enjoyment (Mdn=5 and 4.5 for the horizontal and vertical display), Perceived-Efficiency (Mdn=5 and 5), Perceived-Confidence (Mdn=5 and 5), and Perceived-Integration with touch input (Mdn=5 and 4.5), where 5, the maximum rating from our scale, denotes "Strongly agree;" see Figure 2. Also, vibrations on the finger were preferable to the wrist and forearm when the experience was described in terms of

4.2.3

Free-form description of the experience of vibrotactile feedback. Figure 3 shows word clouds generated from a total number of 209 words proposed by our participants (M=14.9 on average), which they felt best described their experience with vibrotactile feedback for each combination of OnBody-Location × Display-Orientation. The finger condition received positive descriptions, such as "pleasant, " "interactive, " "useful, " and "responsive. " Positive words were also used to describe feedback on the wrist, although some participants considered it "unnecessary" or "little noticeable" compared to other conditions. Finally, the experience of vibrations delivered to the forearm received more words with negative connotations, such as "disturbing, " "uncomfortable, " "little noticeable, " and even "useless. " These descriptions corroborate the findings obtained with the UX measures from Figure 2 that revealed a preference for finger and wrist over forearm. An interesting finding regards the word "confirmation, " which stands out in all of the clouds, revealing the practicality of vibrotactile feedback to confirm successful detection of finger touches on the surface of the display.

Expectations for a Real-World System Implementation

In our experiment, we employed on/off vibrations of a constant intensity and duration to keep our experiment design simple. However, in order to learn about participants' expectations for more complex vibrotactile patterns for touch input, we measured Expected-Meaningfullness from 1 (participants strongly disagreeing that more complex patterns were needed) to 5 (strong agreement). The median rating of Expected-Meaningfullness was 3.5, located in-between "Neither agree nor disagree" and "Agree, " with a mean of 3.4 (SD=1.3). This result shows a slight preference for more complex vibrations, but not statistically significant according to a one-sample Wilcoxon test (𝑉 =46.5, 𝑝=.235>.05, 𝑛.𝑠.).

Nevertheless, a few participants provided examples of vibration patterns or applications where vibrotactile feedback more complex than an on/off vibration would be useful, which are interesting to report. For example, P 1 said that vibrations could be longer and stronger when the user finds the correct answer in a video game; P 3 considered that errors should be accompanied by longer vibrations; P 11 exemplified the case of a payment application, where the user could be alerted with a stronger vibration that they are about to confirm payment; and participant P 10 suggested complementing vibrations on the arm with a visual effect on the display, e.g., ripples, at the touch location.

Manuscript submitted to ACM

We also found that our participants were willing to receive vibrotactile feedback delivered from the display to the finger via their smart rings (Mdn=4.5), if they had such a wearable device, as well as on the wrist via their smartwatches and fitness trackers (Mdn=4), but were overall undecided regarding wearing a smart armband (Mdn=3) to receive vibrations on the forearm. This result corroborates our previous findings obtained with the UX measures of perception, where the forearm condition was the least preferred.

DISCUSSION AND LIMITATIONS

We reported positive reactions from our participants for vibrotactile feedback accompanying touch input on large displays, which recommend future work on designing cross-device, wearable-surface user experiences for public displays. One result that emerged with several of our UX measures was that vibrotactile feedback delivered on the finger was preferred to feedback on the wrist and forearm, respectively. One possible explanation for this result may lie with associations that users may form during their interactions (with the finger) with smartphones that provide vibrotactile feedback, while vibrations delivered by smartwatches and fitness trackers largely represent notifications, not confirmations of touch input. Also, our participants were not familiarized with devices for the forearm, such as smart armbands, which may explain the reported feelings of discomfort or unnecessary feedback at that location on the body. Moreover, the preference for the finger might have been influenced by the intensity and the duration of the vibrotactile feedback, for which we used 150ms-long vibrations; see Subsection 3.2. Longer or more intense vibrations, including using multiple actuators, may lead to different results, but also different form factors for wearables. We leave such explorations for future work.

There are two limitations to our experiment design. First, we did not randomize the order of the "no feedback" control condition as we did with vibrotactile feedback on the finger, wrist, and forearm. The control condition was always presented first (see Subsection 3.4) as we used it for our participants to become familiarized with the interactive map application and touch input with the two displays. Further experiment designs could look for significant effects between the control and vibrotactile feedback conditions. Second, we did not attempt to correct the time delay [START_REF] Henderson | Leveraging Distal Vibrotactile Feedback for Target Acquisition[END_REF] caused by the signal to propagate in the Wi-Fi network between the moment of touching the screen and the actual moment when feedback was provided at various locations on the interactive arm, although none of the participants mentioned any latency issues.

CONCLUSION

We focused in this work on understanding the novel user experience of distal, physically decoupled vibrotactile feedback on the interactive arm during touch input on a large display. Interesting future work lies ahead, such as exploring other locations on the body where to deliver distal vibrotactile feedback, technical implementation of such feedback using off-the-shelf wearables, and evaluating the user experience of other user groups. Also, exploration of a diversified set of vibrotactile feedback patterns for various categories of wearables [START_REF] Aiordăchioae | Addressing Inattentional Blindness with Smart Eyewear and Vibrotactile Feedback on the Finger, Wrist, and Forearm[END_REF][START_REF] Ariza | Ring-Shaped Haptic Device with Vibrotactile Feedback Patterns to Support Natural Spatial Interaction[END_REF][START_REF] Seifi | Exploring the Design Space of Touch-Based Vibrotactile Interactions for Smartwatches[END_REF] represents another future work direction for enriching interactions with large public displays.

Fig. 1 .

 1 Fig. 1. Participants interacting with the horizontal (left) and vertical (middle) large displays. On the right, a close-up of the wearable device with vibration motors that we prototyped for our experiment.

Fig. 2 .

 2 Fig. 2. User experience of vibrotactile feedback on the finger, wrist, and forearm during touch input with the horizontal and vertical displays. Notes: median values are shown in this figure; higher values denote a better experience for the UX measures from the top row; lower values denote a better user experience for the bottom row measures; seeTable 1 for statistical tests.

Fig. 3 .

 3 Fig. 3. Word clouds (https://www.wordclouds.com) generated from our participants' free-form descriptions of their experience with vibrotactile feedback accompanying touch input for each combination of Display-Orientation × OnBody-Location.

Table 1 .

 1 Table 1 for statistical tests. Statistical significance tests for the UX measures of the perception of vibrotactile feedback during touch input with large displays; see Figure 2 for the median ratings.

		Effect of	Effect of OnBody-Location
	UX measure	Display-Orientation	main effect	finger vs. wrist †	finger vs. forearm †	wrist vs. forearm †	Interac-tion
	■ Perceived-Enjoyment	𝐹 (1,65) =4.343, 𝑝<.05	𝐹 (2,65) =12.346, 𝑝<.001	𝑝<.005	𝑝<.001	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Efficiency	𝑛.𝑠.	𝐹 (2,65) =23.521, 𝑝<.001	𝑝<.001	𝑝<.001	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Confidence	𝑛.𝑠.	𝐹 (2,65) =12.418, 𝑝<.001	𝑝<.001	𝑝<.05	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Integration	𝑛.𝑠.	𝐹 (2,65) =15.896, 𝑝<.001	𝑝<.001	𝑝<.005	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Distracteness	𝑛.𝑠.	𝐹 (2,65) =8.473, 𝑝<.001	𝑝<.001	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Difficulty	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Confusion	𝑛.𝑠.	𝐹 (2,65) =4.912, 𝑝<.01	𝑝<.05	𝑝<.05	𝑛.𝑠.	𝑛.𝑠.
	■ Perceived-Complexity	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.	𝑛.𝑠.

† Bonferroni corrections applied.

https://leafletjs.com

https://nfpshop.com/product/10mm-coin-vibration-motor-3mm-type-model-nfp-c1030 Manuscript submitted to ACM

https://esp8266-shop.com/product/nodemcu-esp8266-esp-12e

https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API

https://www.ideum.com/products/touch-tables

https://displaysolutions.samsung.com/digital-signage/detail/86/UE55D

https://www.samsung.com/ph/business/smart-signage/touch-overlay-td55ldah/cy-td55ldahen Manuscript submitted to ACM

Manuscript submitted to ACM

https://cran.r-project.org/web/packages/ARTool/index.html Manuscript submitted to ACM

ACKNOWLEDGMENTS

This work is part of a project that has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 860114.


Mihail Terenti
email: mihail.terenti@usm.ro

MANSiD Mintviz Lab

Radu-Daniel Vatavu
email: radu.vatavu@usm.ro

How Do HCI Researchers Describe Their Software Tools? Insights From a Synopsis Survey of Tools for Multimodal Interaction

Keywords: CCS Concepts:, Human-centered computing → Systems and tools for interaction design; Human computer interaction (HCI) Multimodal interaction, tools, platforms, literature survey

Providing tools to support design and engineering of interactive computing systems has been encouraged in the HCI community.

However, little is known about the practices adopted by HCI researchers to describe their software tools in academic publications.

To address this aspect, we implemented a simplified literature survey procedure combining principles of population sampling and systematic literature reviews to enable rapid access to insights from a vast body of published academic work. Our findings show that screenshots and diagrams are among the most widely used descriptive elements by HCI researchers to present their software tools, a finding that we capitalize on to reflect about the dissemination of tools for the design and engineering of multimodal interaction at the intersection of software engineering and HCI.

INTRODUCTION

Tools are important in the HCI research and practice to support design and implementation of user interface software [START_REF] Myers | User Interface Software Tools[END_REF], rapid prototyping of interactive systems [START_REF] Beaudouin-Lafon | Prototyping Tools and Techniques[END_REF], encourage creativity [START_REF] Remy | Evaluating Creativity Support Tools in HCI Research[END_REF], inform design options [START_REF] Vanderdonckt | The Foldinterface Editor: A Visual Tool for Designing User Interfaces for Foldable Displays[END_REF][START_REF] Vatavu | GestuRING: A Web-based Tool for Designing Gesture Input with Rings, Ring-Like, and Ring-Ready Devices[END_REF], analyze user input [START_REF] Vatavu | Relative Accuracy Measures for Stroke Gestures[END_REF], manage user-elicited data [START_REF] Magrofuoco | GestMan: A Cloud-Based Tool for Stroke-Gesture Datasets[END_REF], and support research to understand users [START_REF] Ali | Crowdlicit: A System for Conducting Distributed End-User Elicitation and Identification Studies[END_REF][START_REF] Vatavu | Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit[END_REF] with evaluations, studies, and experiments [START_REF] Vanderdonckt | AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives[END_REF]. Consequently, "tool contributions, " as a specific type of an artifact contribution in HCI research [START_REF] Wobbrock | Research Contributions in Human-Computer Interaction[END_REF],

are particularly welcomed at HCI venues. For example, tools are explicitly encouraged by several subcommittees of CHI, e.g., the User Experience and Usability subcommittee "is suitable for papers that extend the knowledge, practices, methods, components, and tools that make technology more useful, usable, and desirable, " while the Design subcommittee of CHI specifies that "papers submitted here include [...] evaluation of new design tools, processes, methods, or principles, a b c d Fig. 1. Documenting software tools with screenshots, diagrams, code, and photographs: (a) screenshot of SAPIENS [START_REF] Schipor | SAPIENS: Towards Software Architecture to Support Peripheral Interaction in Smart Environments[END_REF], a software architecture for peripheral interaction, from EICS '19; (b) UML class diagram of Flecto [START_REF] Khaddam | Towards Rapid Prototyping of Foldable Graphical User Interfaces with Flecto[END_REF], a rapid prototyping tool for foldable UIs, from ISS '20; (c) API response of KeyTime [START_REF] Leiva | KeyTime: Super-Accurate Prediction of Stroke Gesture Production Times[END_REF], a web-based tool for estimating gesture production time, from CHI '18; and (d) photographs from using a C++ software library to test attacks on smartglasses [START_REF] Opaschi | Uncovering Practical Security and Privacy Threats for Connected Glasses with Embedded Video Cameras[END_REF], from IMWUT 2020.

including those that explore alternatives to scientistic ways of knowing" 1 (emphasis ours). Similarly, other HCI venues encourage tools, including "engineering design and evaluation tools" at EICS, 2 "multimodal platforms" at ICMI, 3 "tools for evaluating interactive systems" at INTERACT, 4 and "new methodologies and tools" at IMWUT 5 (emphasis ours); see Figure 1 for examples extracted from papers published in these subcommunities of HCI.

In this context, where tools are both needed and encouraged, it is useful to understand how HCI researchers describe their tool contributions when disseminating their results in order to document and understand this practice in our community. To this end, we perform a characterization of the ways in which HCI researchers describe their software tools in academic publications, and report results of a case study in multimodal interaction, an active area with a long tradition in HCI [START_REF] Dumas | Multimodal Interfaces: A Survey of Principles, Models and Frameworks[END_REF] and, hence, with established practices for providing tools. We use our findings to reflect on the dissemination of tools at the intersection of software engineering and HCI [START_REF] Manuel | A Newcomer's Guide to EICS, the Engineering Interactive Computing Systems Community[END_REF].

RELATED WORK

We discuss practices for documenting software applications in the software engineering community, and present an overview of prior surveys conducted on the topic of multimodal interaction related to the case study implemented in this work.

Documenting Software Applications

Presenting a software application or tool to others implies employing effective methods to document features and modes of operation to demonstrate the usefulness of the software for the target audience. Thus, software documentation for classes, libraries, frameworks, and platforms takes a variety of forms [START_REF] Mcconnell | Code Complete: A Practical Handbook of Software Construction[END_REF]. In formal projects, documentation is outside the source code, e.g., it takes the form of unit development folders containing the developers' notes or detailed-design documents describing class-level and routine-level design decisions. Internal documentation lies within the source code, and takes the form of comments inserted by programmers, but especially is the result of a good programming style [START_REF] Mcconnell | Code Complete: A Practical Handbook of Software Construction[END_REF].

A widespread practice among software developers is the use of the "readme" file, usually written in markdown, that provides a short description of the software, how to compile and install it, and how to use it. More detailed technical documentations employ wiki systems. 6 At community level, "get started" and "docs" sections and FAQ sections [START_REF] Ellmann | A Comparative Study of FAQs for Software Development[END_REF] of web pages and blogs [START_REF] Blood | How Blogging Software Reshapes the Online Community[END_REF][START_REF] Pagano | How Do Open Source Communities Blog[END_REF] are common. In the context of HCI, Mehlenbacher [START_REF] Mehlenbacher | Documentation: Not yet Implemented, but Coming Soon![END_REF] Several standards exist to provide valuable information to providers of software applications, tools, and platforms.

For instance, the ISO/IEC/IEEE 26515:2018 standard on developing user documentation in an agile environment 7acknowledges that "the documentation may be the first tangible item that the user sees, and so influences the first impressions the users have of the product" and "well designed documentation not only assists the users and helps to reduce the cost of training and support, but also enhances the reputation of the product, its producer, and its suppliers. "

The standard focuses on specific concepts, such as the burndown chart (a document for recording project status), system feature (a distinguishing characteristic of a system), persona (the archetypal user of a system), scrum report (documenting the daily activities of a scrum team), use case (behavioral requirements of a system and interaction with the user), user story (narrative illustrating user goals), and technical writer (person in charge of the user documentation).

The IEEE Standard for Software User Documentation8 provides "requirements for the structure, information content, and format of user documentation, to include both paper and electronic documentation used in the work environment by users of systems containing software." The standard focuses on concepts such as the user, action, illustration, tutorial, usage mode, and critical information, and identifies fourteen components of software user documentation, from identification data and table of contents to navigational features, indexes, and search capability.

Surveys of Multimodal Interaction

In this work, we choose to implement a case study in multimodal interaction. Our motivation is driven by the fact that multimodal interaction has a long tradition in HCI [START_REF] Dumas | Multimodal Interfaces: A Survey of Principles, Models and Frameworks[END_REF] and, consequently, more established practices are likely to exist for disseminating tools.

There is a wide literature on the topic of multimodal interaction, including many surveys that have adopted various approaches to describe design, implementation, evaluation, and application of multimodal interactions [START_REF] Zeeshan | Multimodal Systems: Taxonomy, Methods, and Challenges[END_REF][START_REF] Dumas | Multimodal Interfaces: A Survey of Principles, Models and Frameworks[END_REF][START_REF] Gürkök | Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles[END_REF][START_REF] Jaimes | Multimodal Human-Computer Interaction: A Survey[END_REF][START_REF] Martin | Multimodality in VR: A survey[END_REF][START_REF] Turk | Multimodal Interaction: A Review[END_REF].

For example, the goal of Jaimes and Sebe's [START_REF] Jaimes | Multimodal Human-Computer Interaction: A Survey[END_REF] survey was to present an overview of open issues in multimodal HCI by relating to computer vision approaches, the affective part of human-computer interaction (e.g., emotion processing to make multimodal systems more natural and empathic), and system architecture to support multimodal interactions. Dumas et al. [START_REF] Dumas | Multimodal Interfaces: A Survey of Principles, Models and Frameworks[END_REF] focused on principles, models, and programming frameworks for multimodal user interfaces, highlighting the potential of machine learning for this area. A survey of the state-of-the-art of the role played by and the benefits of multimodality in Virtual Reality was provided by Martin et al. [START_REF] Martin | Multimodality in VR: A survey[END_REF] that focused on the potential of multimodal interactions to contribute to compelling experiences in virtual environments. Gürkök and Nijholt [START_REF] Gürkök | Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles[END_REF] surveyed Brain-Computer Interfaces for multimodal interactions, stressing how they can significantly improve error handling, task performance, and user experience. Baig and Kavakli [START_REF] Zeeshan | Multimodal Systems: Taxonomy, Methods, and Challenges[END_REF] focused on evaluation, input and output modalities, and data collection and fusion as challenges for multimodal systems, and concluded that multimodal application developers focus mostly on input, while the most widely employed input modalities are speech and gesture.

Similar conclusions were drawn by Turk's [START_REF] Turk | Multimodal Interaction: A Review[END_REF] survey on designing multimodal interfaces, including input and output, biological sensory integration, and multimodal integration.

The research question addressed in our work-how do HCI researchers document their software tools to support design and engineering of multimodal interaction systems?-can be seen as a complementary survey to the existing literature [START_REF] Zeeshan | Multimodal Systems: Taxonomy, Methods, and Challenges[END_REF][START_REF] Dumas | Multimodal Interfaces: A Survey of Principles, Models and Frameworks[END_REF][START_REF] Gürkök | Brain-Computer Interfaces for Multimodal Interaction: A Survey and Principles[END_REF][START_REF] Manuel | A Newcomer's Guide to EICS, the Engineering Interactive Computing Systems Community[END_REF][START_REF] Martin | Multimodality in VR: A survey[END_REF][START_REF] Turk | Multimodal Interaction: A Review[END_REF], yet limited in depth (see the next section introducing synopsis surveys) reporting on how researchers have chosen to describe their prototypes, tools, libraries, frameworks, and applications designed to support multimodal interaction systems.

METHOD

To understand existing practices in describing tools for multimodal interaction, we conducted a guided, targeted search of the scientific literature by adopting the procedure of Systematic Literature Reviews (SLR) [START_REF] Brereton | Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain[END_REF], but without running a comprehensive and extensive search of the literature as SLRs do. This approach is justified for our specific case study in multimodal interaction, as would be in other cases where a huge amount of work exists, that prevents a complete analysis of all the available work in reasonable time. 9 Our simplified approach employs the method of SLRs, while adopting the rationale of conducting scientific experiments, where a sample is used for the analysis instead of the entire population. From this perspective, we follow the sound procedure of SLRs and keep their desirable characteristics [START_REF] Siddaway | How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses[END_REF], while arriving at rapid insights from a vast literature by exploring a sample of that literature, similar to how experimental research methods [START_REF] Lavrakas | Experimental Methods in Survey Research: Techniques that Combine Random Sampling with Random Assignment[END_REF] lead to findings by extrapolating the results obtained on a sample to the large population. We call this procedure a synopsis survey,10 with several convenient characteristics: synopsis surveys are methodical, transparent, replicable, empirical, and rapid. The first three characteristics emerge from adopting the procedure of SLRs [START_REF] Siddaway | How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses[END_REF] (described next), while the last two from the fact that we work with a sample of the literature. To implement our survey, we ran the following query:

"query": Title:(multimodal AND (tool* OR platform*) AND (interaction* OR interface* OR input))

"filter": NOT VirtualContent: true, ACM Content: DL on the titles of papers from the ACM Full-Text Collection and found 49 results. We searched our combination of keywords just in the titles to limit the large number of results corresponding to papers that used "'multimodal," "interaction," or "interface" as part of their abstracts or full text and that were not relevant to our research question about tools for multimodal interaction. We also considered the following eligibility criteria (ECs) to screen these results: EC 1 . Content availability. The paper is written in English, the full text is available, and the paper was peer reviewed.

Journal articles, conference papers, and other contributions are eligible according to this criterion, while invited keynotes, book abstracts, and white papers are not.

EC 2 . Eligible area. The paper is about multimodal interaction. EC 3 . Eligible topic. The paper presents, evaluates, or surveys tools or platforms to support multimodal interaction.

A number of six results were discarded by not meeting eligibility criterion EC 1 , three were excluded by EC 2 , and ten did not present tools (EC 3 ). After the eligibility stage, we arrived at a number of 30 papers for subsequent analysis.

During this analysis, we extracted the following characteristics and coded 11 the papers accordingly:

(1) Input modalities. We used the following categories that emerged from our analysis: voice, gesture, WIMP, eye gaze, stylus input, and head & face input.

(2) Output modalities: visual, aural, and haptic. Just like for input modalities, these categories emerged from our analysis.

(3) Target users: researchers, developers, and other users.

(4) Target platform addressed by the tool, with the following categories: desktop, mobile, and embodied (robots).

(5) Contribution. We extracted the contributions made by the papers describing tools for multimodal interaction:

software application (i.e., the tool is a software that developers can use to design and create new systems), hardware prototype (the tool relies on a hardware component, e.g., a specific sensor, required for multimodal interaction), source code (the authors release source code accompanying the previous two types of contributions),

user study (a study with users to validate the tool), and technical study (results are reported about the technical performance of the tool).

(6) Descriptive elements employed to present and document tools, besides the textual description from the paper, with the following categories emerging from our analysis: diagrams, code snippets, use cases, screenshots, photographs, and performance charts, respectively.

RESULTS

The papers included in our final set were published between 1998 (Cheyer and Julia's [START_REF] Cheyer | MVIEWS: Multimodal Tools for the Video Analyst[END_REF] multimodal tools for the video analyst) and 2020 (Sarmah et al.'s [START_REF] Ritam | Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications[END_REF] Geno tool for authoring multimodal interaction on existing web applications), and the majority (19/30=63.3%) were published at ICMI, the ACM International Conference on Multimodal Interaction (formerly on Multimodal Interfaces).

We found that the most common input modalities were voice (33.3%), mouse and keyboard for the WIMP interaction paradigm (27.5%), and gesture commands (20.3%), followed by eye gaze, stylus input, and head movements and facial expressions; see Figure 2a. These results match a conclusion from Baig and Kavakli's [START_REF] Zeeshan | Multimodal Systems: Taxonomy, Methods, and Challenges[END_REF] survey of multimodal systems based on 136 references, according to which the most widely employed input modalities in the literature were found to be speech and gesture-a result that builds confidence in our synopsis approach. Visual and aural modalities were approximately balanced at 51.1% and 46.7%, respectively, while haptics was addressed by just one tool [START_REF] Rose Mcgee-Lennon | User Evaluation of OIDE: A Rapid Prototyping Platform for Multimodal Interaction[END_REF]; see Figure 2b.

We identified a total of 48 contributions, of which software applications were the most common (50%), followed by user studies (16%) to validate the tools, and hardware contributions (15%) to enable new input modalities, e.g., the chemistry pod from Anderson et al.'s [START_REF] Khalil | Chemistry Pods: A Mutlimodal Real Time and Retrospective Tool for the Classroom[END_REF] multimodal tool for the classroom was designed to record audio from various locations in the classroom. The most widely addressed platform was desktop (73%), while mobile platforms and robots were less represented (20% and 17%, respectively). Finally, tools were specifically addressed at developers (61.1%) and researchers (30.6%) and, in a few cases, at end users, such as video analysts [START_REF] Cheyer | MVIEWS: Multimodal Tools for the Video Analyst[END_REF] and instructors [START_REF] Khalil | Chemistry Pods: A Mutlimodal Real Time and Retrospective Tool for the Classroom[END_REF]; see Figures 2c to 2e.

Figure 2f shows the distribution of descriptive elements used by the authors of these papers to document and present their tools. We found that screenshots and diagrams were the most used (34.9% and 26.2%), followed by photographs of the tools (12.3%) and descriptions of use cases (11.9%). A Pearson correlation analysis showed a significant positive correlation between the number of diagrams and code snippets (𝑟 (𝑁 =30) =.436, 𝑝=.05) employed to describe the tools, and several negative correlations (not reaching statistical significance at 𝛼=.05, however) between screenshots and photographs and screenshots and use cases, respectively. Screenshots and diagrams were also used in conjunction in 70%

of the papers identified in our synopsis survey, while diagrams and photographs were used together to present the tools in 43.3% of the papers. A percent of 93% of the tools were described using at least two descriptive elements, and 37% with at least four descriptive elements, e.g., Skantze and Al Moubayed [START_REF] Skantze | IrisTK: A Statechart-Based Toolkit for Multi-Party Face-to-Face Interaction[END_REF] employed diagrams, photographs, code snippets, and a description of an use case for IrisTK, their toolkit for multi-party face-to-face interaction; see Figure 2f, right.

Our results show that screenshots and diagrams have been preferred by HCI researchers and authors to describe their tools designed to support multimodal interaction. These findings make sense since, by definition, a tool implies engineering details that are communicated via platform-independent and programming language-independent diagrams

Manuscript submitted to ACM

showing the operation of the tool, e.g., dataflows or UML classes, while screenshots are useful to exemplify directly the end result to the audience. Using more descriptive elements, e.g., diagrams and photographs, enables authors to describe their tools better as well as corresponding use cases for the tools. However, our results also show a strong technical perspective for presenting tools, with authors insisting more on technical aspects compared to use cases. By connecting this finding to the IEEE/ISO/IEC standards discussed in Section 2, descriptions of personas as archetypal users of a tool, narratives illustrating user goals, tutorials, illustrations of actions and steps to achieve user goals are missing from tool descriptions. This finding may seem surprising on a first look, but can be interpreted from the perspective that tools for multimodal interaction are situated at the intersection of software engineering and HCI. In this context, the audience is primarily represented by researchers and practitioners that will use the tools to engineer new interactive computing systems (EICS), and that speak the same language of EICS [START_REF] Manuel | A Newcomer's Guide to EICS, the Engineering Interactive Computing Systems Community[END_REF], a strongly technically-oriented subcommunity of HCI.

CONCLUSION AND FUTURE WORK

We reported results from a synopsis survey to form a preliminary understanding of how HCI researchers and authors describe software tools for multimodal interaction, and found that screenshots and diagrams are representative of the technically-oriented EICS subcommunity of HCI. Future work will examine the description of tools in other areas of HCI, such as tools for fabrication (where more hardware contributions are expected) or tools for conducting user experiments (where use case descriptions and tutorials are likely to be more representative). These new case studies will complete our understanding of how the HCI community presents its tools, and will enable us to reapply synopsis surveys for other areas.

We also acknowledge the limitations of our synopsis survey approach as well as our focus on results from the ACM DL only and running our query on the paper titles alone. Nevertheless, a synopsis survey has useful features, among which being replicable and providing rapid access to insights from areas where a vast amount of literature exists.

Comparing synopsis surveys to other types of literature reviews [START_REF] Grant | A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies[END_REF] is recommended, along with formalizing synopsis surveys with population sampling methods and the use of inferential statistics to extrapolate the results obtained on a sample of papers to the wider scientific literature.

Fig. 2 .

 2 Fig. 2. Input and output modalities (a,b), target audiences (c), contributions (d), target platforms (e), and descriptive elements for documenting tools (f) extracted from the papers that we analyzed in our synopsis survey.

Compare them all | WikiMatrix, https://www.wikimatrix.org

https://www.iso.org/obp/ui/#iso:std:iso-iec-ieee:26515:ed-2:v1:en

https://ieeexplore.ieee.org/document/974401 Manuscript submitted to ACM

Many variants exist for conducting surveys, such as rapid reviews that employ the method of SLRs, but for which the completeness of searching is determined by time constraints, or systematized reviews, that attempt to include elements of the SLR process, while they stop short of a systematic review and are typically narrative; see Grant and Booth[START_REF] Grant | A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies[END_REF] for an analysis of fourteen types of reviews and associated methodologies.

According to Merriam-Webster, a synopsis is "a condensed statement or outline (as of a narrative or treatise)," https://www.merriam-webster.com/ dictionary/synopsis.Manuscript submitted to ACM

The coding was done by the first author of this paper, while the second author cross-checked 20% of the codings.Manuscript submitted to ACM

ACKNOWLEDGMENTS

This work is part of a project that has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 860114.

://www.interact2021.org/tracks/full_papers.php 5 https://dl.acm.org


Prasad Sudhakar

Rémi Gribonval

Sparse filter models for solving permutation indeterminacy in convolutive blind source separation

Keywords: convolutive BSS, permutation ambiguity, sparsity, ℓ1-minimization

Frequency-domain methods for estimating mixing filters in convolutive blind source separation (BSS) suffer from permutation and scaling indeterminacies in sub-bands. Solving these indeterminacies are critical to such BSS systems. In this paper, we propose to use sparse filter models to tackle the permutation problem. It will be shown that the ℓ1-norm of the filter matrix increases with permutations and with this motivation, an algorithm is then presented which aims to solve the permutations in the absence of any scaling. Experimental evidence to show the behaviour of ℓ1-norm of the filter matrix to sub-band permutations is presented. Then, the performance of our proposed algorithm is presented, both in noiseless and noisy cases.

I. INTRODUCTION

The problem of source separation arises in various contexts such as speech enhancement/recognition, biomedical signal processing, wireless telecommunication, etc. Mathematical models of BSS with different levels of complexity have been proposed by the signal processing community. The most difficult but close to reality model is of convolutive mixtures. The underlying model of having M mixtures x m (t), m = 1 . . . M from N source signals s n (t), n = 1 . . . N , given a discrete time index t, is given by

x m (t) = N n=1 K-1 k=0 a mnk s n (t -k) + v m (t) (1)
with v m (t) the noise term. In the matrix notation it can be written as

x(t) = K-1 k=0 A k s(t -k) + v(t) (2)
where x(t), v(t) are m × 1 vectors, A k is an M × N matrix which contains the filter coefficients at k th index. The notation A mn (t) = a mnt will also be used for each mixing filter, which is of length K. The ultimate objective of a BSS system is to recover back the original source signals s n , n = 1 . . . N given only the mixtures x m (t), m = 1 . . . M . A standard approach to separate sources is to first estimate the mixing matrix A(t) and then recover the sources s n (t) [START_REF] Pedersen | A survey of convolutive blind source separation[END_REF]. Fig. 1 shows the block diagram of such a system. This paper focusses on the second block of the system: estimation of the mixing matrix.

x m (t) ŝn (t) Estimate mixing matrix Â(t)

Estimate the sources Several methods have been proposed by the signal processing community to estimate the mixing matrices in convolutive BSS. Pedersen et. al. present an excellent survey of the existing methods [START_REF] Pedersen | A survey of convolutive blind source separation[END_REF]. Broadly, the techniques can be classified into time-domain and frequency-domain techniques. Both approaches have their own advantages and disadvantages. They are summarized in table 3 of [START_REF] Pedersen | A survey of convolutive blind source separation[END_REF].

The context of our problem arises in the frequency-domain convolutive BSS approach. A survey of these techniques is provided in [START_REF] Makino | Blind source separation of convolutive mixtures of speech in frequency domain[END_REF]. The principle behind the frequency-domain techniques is that the convolutive mixture case is transformed (under the narrowband assumption) into complex-valued instantaneous mixture case for each frequency bin:

x(f, t) = A(f )s(f, t) + v(f, t) (3)
where f = 1 . . . F are the sub-band frequencies. Several algorithms to estimate mixing matrix in case of instantaneous mixtures have been developed by the source separation community [START_REF] Bell | An information-maximization approach to blind separation and blind deconvolution[END_REF], [START_REF] Smaragdis | Blind separation of convolved mixtures in the frequency domain[END_REF]. Frequency domain convolutive BSS systems use one of these standard algorithms on each frequency bin to provide an estimate Â(f ) of A(f ). However, as the estimation is done on each frequency bin independently, this approach suffers from permutation and scaling indeterminacies in each sub-band f . Specifically, the estimated Â(f ) is related to the true filter matrix A(f ), for each f in the following form

Â(f ) = A(f )Λ(f )P(f ) (4)
where P(f ) is the frequency-dependent permutation matrix, Λ(f ) is a diagonal matrix containing the arbitrary scaling factors.

The frequency-domain methods have to invariably solve the permutation and scaling indeterminacy to eventually estimate A(t) up to a unique global permutation and scaling Â(t) = A(t)ΛP.

A. Existing approaches to solve the described problem

There are two main kinds of methods to solve the permutation indeterminacy in the sub-bands of the estimated mixing filters [START_REF] Pedersen | A survey of convolutive blind source separation[END_REF].

The first set of techniques use consistency measures across the frequency sub-bands of the filters to recover the correct permutations, such as inter-frequency smoothness, etc. This category also includes the beamforming approach to identify the direction of arrival of sources and then adjust the permutations [START_REF] Ikram | A beamforming approach to permutation alignment for multichannel frequency-domain blind source separation[END_REF].

The second set of techniques use the consistency of the spectrum of the recovered signals to achieve the same. The consistency across the spectrum of the recovered signals is applicable for only those signals which have strong correlation across sub-bands, such as speech [START_REF] Murata | An approach to blind source separation based on temporal structure of speech signals[END_REF].

There has also been some effort to combine the above mentioned approaches for better performance [START_REF] Sawada | A robust ans precise method for solving the permutation problem of frequency-domain blind source separation[END_REF]. Based on the different definitions of consistency, methods to correct permutations have been proposed in the literature. A categorization of these methods based on the definition of consistency has been presented in [START_REF] Pedersen | A survey of convolutive blind source separation[END_REF] (Table 4).

III. PROPOSED APPROACH: SPARSE FILTER MODELS

In our work, we propose to use a special type of consistency that can be assumed on the mixing filters: sparsity. That is, the number S of non-negligible coefficients in each filter A mn (t) is significantly less than its length K. The motivation behind this approach is that the acoustic room impulse responses (RIR) tend to have a few reflection paths relative to its duration. Hence, one can approximate the acoustic RIRs by sparse filters.

The idea behind our approach is that when the mixing filters are sparse, the permutations in the sub-bands disturb the sparse structure of the filters. That is, each filter in the reconstructed filter matrix Â(t) after sub-band permutations will no longer be sparse. So, one can hope to correct the permutations by bringing back the sparse structure in each of the filters.

Ideally, the sparsity of the filter matrix is measured by its ℓ 0 -norm, defined as

||A(t)|| 0 = M m=1 N n=1 K t=1 |A mn (t)| 0 . ( 5
)
Lesser the norm, sparser is the filter matrix. However, the ℓ 1norm of the filter matrix defined by

||A(t)|| 1 = M m=1 N n=1 K t=1 |A mn (t)| (6)
is also a sparsity promoting norm. It has been shown by Donoho that working with ℓ 1 -norm is as effective as working with ℓ 0 -norm while looking for sparse solutions to linear systems [START_REF] Donoho | For most large underdetermined systems of linear equations, the minimal ell-1 norm near-solution approximates the sparsest nearsolution[END_REF]. Furthermore ℓ 1 -norm being convex, has certain computational advantages over the former and is robust to noise.

In this paper, we concentrate primarily only on the permutation problem and will assume that the estimated filter sub-bands are free from any scaling indeterminacy. That is

Â(f ) = A(f )P(f ), f = 1 . . . F. (7)
Hence, when Â(t) is reconstructed using Â(f ), given by Eqn. [START_REF] Sawada | A robust ans precise method for solving the permutation problem of frequency-domain blind source separation[END_REF] one can expect an increase in || Â(t)|| 1 . In Sec. IV we experimentally show that this is indeed the case. Then in Sec. V we show that if there is no scaling indeterminacy, a simple algorithm which minimizes the ℓ 1 -norm of the filter matrix solves the permutation indeterminacy, even under noisy conditions.

IV. SOURCE PERMUTATIONS AND ℓ 1 -NORM This section presents some preliminary experimental study on how the ℓ 1 -norm of the filter matrix A(t) is affected by permutation in the sub-bands.

For experimental purposes, 50 different filter matrices were synthetically created with N = 5 sources and M = 3 channels, and each filter having a length of K = 1024 with S = 10 non-zero coefficients. The non-zero coefficients were i.i.d. Gaussian with mean zero and variance one. The locations of non-zero coefficients were selected uniformly at random. Then for each such instance of filter matrix A(t) the discrete Fourier transform (DFT) Âmn (f ) was computed for each filter A mn (t), to obtain the frequency-domain representation Â(f ) of A(t). These filters were used in the following two kinds of experiments.

A. Random source permutations

In a practical scenario, each sub-band can have a random permutation. So, for each filter matrix in the frequency domain, the sources were permuted randomly in an increasing of number of sub-bands (chosen at random), and their ℓ 1 -norms were computed. The positive and negative frequency subbands were permuted identically.

For one such experimental instance, Fig. 2 shows the variation of ℓ 1 -norm against increasing number of randomly permuted sub-bands. The circle shows the norm of the true filter matrix. Each star represents the norm after randomly permuting the sub-bands at random locations. Note the gradual increase in the norm as the number of sub-bands being permuted increases. Similar experiments were conducted with combinations of M = 3, N = 4 and M = 2, N = 3 and S = 10, 15, 20, leading to similar observations.

B. Sensitivity of ℓ 1 -norm to permutations

In order to show that even a single permutation in only one sub-band can increase the norm, only two sources, chosen at random were permuted in increasing number of sub-bands.

For one such instance, Fig. 3 shows a plot of the variation in ℓ 1 -norm with the number of sub-bands permuted.The circle in the plot shows the ℓ 1 -norm of the true filter matrix. Each star shows the norm after permuting the sources 2 and 3.

V. PROPOSED ALGORITHM

With the inspiration from the previous section, we present an algorithm in this section to solve the permutation indeterminacy.

Assumption:

The estimate Â(f ) of the sub-band coefficients A(f ) are provided by some other independent technique and

Â(f ) = A(f )P(f ), f = 1 . . . F .
Figure 4 shows the block diagram of our approach. The output from the first block is assumed to be available, and the algorithm presented in this section is what makes up the second block.

The absence of scaling indeterminacy while estimating Â(f ) is not totally a realistic assumption. But we feel that this assumption aids the understanding of connections between the correction Permutation Estimate Â(f ) Ã(f ) x m (t) Fig. 4. Block diagram of our approach to solve permutation indeterminacy time-domain sparsity of the filters and controlled disturbances in the frequency-domain. This understanding will be later useful in handling the arbitrary scaling of sub-band coefficients.

A. Description

We denote the set of all the possible source permutations by P (|P| = N !) and the inverse discrete Fourier transform by IDF T . At each sub-band (starting from the first subband), every possible permutation P ∈ P is explored, keeping the rest of the sub-bands fixed, and only that permutation is retained which minimizes the ℓ 1 -norm. This ensures that ℓ 1norm of the filter matrix is lowered to the minimum possible extent by aligning that particular sub-band. At the end of one such iteration through all the sub-bands, the norm of the filter matrix would have significantly reduced. However, as the subbands were locally examined, the resulting norm may not be the minimum. Hence, the entire process of locally aligning the sub-bands is iterated several times until the difference in the norms between iterations falls below a preset threshold θ. Algorithm 1 contains the pseudo-code of the description.

Input: Â, θ: The estimated sub-band coefficients of A(t)

and a threshold Output: Ã: The sub-band coefficient matrix after solving for the permutations (1) Initialize à ← Â;

(2) Update all sub-bands;

foreach f = 1 : F do old à ← Ã; foreach P ∈ P do Ã(f ) ← Â(f )P; val(P) ← ||IDF T ( Ã(f ′ ))|| 1 , f ′ = 1 . . . F ; end P(f ) ← arg min P∈P val(P); Ã(f ) ← Â(f )P(f ); end (3) Test if the algorithm should stop; if || Â(t)|| 1 ≥ ||old Â(t)|| 1 -θ then Output à else
Go to step (2) Algorithm 1: Algorithm to solve the permutation indeterminacy by minimizing the ℓ 1 -norm of the time domain filter matrix

B. Objective of the algorithm

The aim of the algorithm is to obtain the sub-band matrix Ã(f ) which would have the minimum ℓ 1 -norm in its corresponding time domain representation Ã(t). However, currently we do not have analytical proof about the convergence of the algorithm to the global minimum but only ample experimental evidence that minimizing the ℓ 1 -norm corrects the permutations. It should also be noted that once the algorithm stops, the sources will be globally permuted.

C. Complexity

A brute force approach to solve the ideal ℓ 1 -minimization problem would need N ! K K log K operations. In our approach, each outer iteration needs to inspect N ! × K permutations and at each step, an inverse FFT having a complexity of K log K has to be performed. Hence, the complexity in each outer iteration is N !K 2 log K. This is still expensive because it grows in factorial with the number of sources, but it is tractable for small problem sizes.

VI. RESULTS

In this section we present an illustration of the algorithm presented above.

A. The no noise, no scaling case

Firstly, we consider the case where the sub-band coefficients are assumed to be estimated without noise and scaling ambiguity. 20 filter matrices with the specifications N = 3, M = 2, K = 1024 and S = 10 were created, transformed using DFT and the sub-bands permuted in a similar way as explained in Sec. IV-A. These were the input to the algorithm. The value of θ was set to 0.0001 in all the experiments.

The output was transformed back to time domain to compute the reconstruction error. In all the experiments, the output filters were identical to the true filters up to a global permutation and within a numerical precision in Matlab.

B. Effect of noise

The estimation of Â(f ) by an actual BSS algorithm invariably involves some level of noise (as well as scaling, which we do not deal with here). Hence, the permutation solving algorithm needs to be robust to certain level of estimation noise. Experiments were conducted by permuting the subbands and adding noise to the coefficients:

Â(f ) = A(f )P(f ) + N(f ) (8)
where N(f ) is i.i.d. complex Gaussian with mean zero and variance σ 2 . For illustration, Fig. 5 shows an instance of the reconstructed filter matrix using Algorithm 1 with the input corrupted by additive complex Gaussian noise with σ 2 = 0.2. Each filter had 10 significant coefficients which have been faithfully recovered, along with some amount of noise.

For quantifying the effect of noise, an input-output SNR analysis was performed. The input SNR defined in Eqn.

SN R

in = 20 log 10 ||A(t)|| 2 ||N(t)|| 2 (9) SN R out = 20 log 10 ||A(t)|| 2 ||A(t) -Ã(t)|| 2 (10)
The experiments were repeated for S = 10 and S = 25.

VII. CONCLUSION AND FUTURE WORK DIRECTION

Frequency-domain estimation of mixing matrices in convolutive BSS suffers from the indeterminacies of source permutations and scaling of sub-bands. Hence, solving the permutation indeterminacy is an important aspect of such BSS systems. In this paper, it has been shown that in the absence of scaling, the ℓ 1 -norm of the filter matrix is very sensitive to permutations in sub-bands. An algorithm has been presented based on the minimization of ℓ 1 -norm of the filter matrix to solve for the permutations. Experimental results show that in the absence of scaling, the ℓ 1 -minimization principle to solve the permutations performs well even in the presence of noise.

Though the absence of scaling is not a realistic assumption, it can be a first step towards sparsity motivated permutation solving methods. Also, the complexity of the algorithm grows with the N ! and K 2 , which is expensive even for moderate values for the number of sources N and filter length K.

Our future work focusses on the theoretical analysis of the variation of ℓ 1 -norm with permutations. This may help to replace the combinatorial optimization step by an efficient convex optimization formulation. Further, working with real world mixing filters, and devising ℓ 1 -norm based methods to solve the sub-band scaling ambiguity are some of the interesting extensions to the presented work.

Fig. 1 .

 1 Fig. 1. Block diagram of a typical BSS system

Fig. 2 .Fig. 3 .

 23 Fig.2. The variation in the ℓ 1 -norm of the filter matrix against the number of sub-bands permuted when all the sources are permuted randomly

Fig. 5 .

 5 Fig. 5. Reconstructed filter matrix with additive noise in the sub-bands

Fig. 6 .

 6 Fig. 6. Plot of output SNR versus input SNR in dB. The definitions of the SNR are given in equations (9) and (10)


Muhammad Usman
email: muhammad.3.usman@kcl.ac.uk

Philipp G Batchelor
email: philip.batchelor@kcl.ac.uk

A QUANTITATIVE CRITERION FOR SELECTING THE OPTIMAL SPARSE REPRESENTATION OF DYNAMIC CARDIAC DATA IN COMPRESSED MRI

Keywords: Compressed Sensing (CS), x-f space, Sparsity, Temporal difference

One of the important performance factors in compressed sensing (CS) reconstructions is the level of sparsity in sparse representation of the signal. The goal in CS is to find the sparsest representation of the underlying signal or image. However, for compressible or nearly sparse signals such as dynamic cardiac MR data, the quantification of sparsity is quite subjective due to issues such as dropped SNR or low contrast to noise ratio (CNR) in sparse domains such as x-f space or temporal difference domains. Hence, we need a criterion to compare different sparse representations of compressible signals. In this paper, we define a model that can fit the decay of practical compressible signals and as an application; we verify that this model can be used as a basis for selecting the optimal sparse representation of dynamic cardiac MR data.

INTRODUCTION

Most of the energy in a compressible signal lies in few significant coefficients in its sparse representation and the amplitudes of the remaining coefficients are closer to 0. If we sort the coefficients of a compressible signal in decreasing amplitudes, the degree of 'sparseness' of the signal can be quantified by the decay rate at which the amplitude of coefficients is decreasing. For some signals, this decay can be modeled by an exponential law(|c i | = ae -bi ; b ≥ 0, a > 0). Alternately, we can use a power law as proposed in [START_REF] Jost | EUSIPCO 2008[END_REF]. For a signal of length N with sorted coefficient amplitudes |c 0 | ≥ |c 1 | ≥ .... ≥ |c N -1 |, the power law approximates the amplitude of the i th coefficient as |c i | = C.i -r r ≥ 0, where r is the decay rate and C is a positive constant. For different sparse representations of a compressible signal, we do curve fitting governed by power law and decaying exponential law to the coefficients sorted in decreasing amplitudes. The sparse representation corresponding to the power or exponential curve having the highest decay factor r is chosen to be the optimal representation.

METHOD

Our procedure for finding the optimal representation of dynamic cardiac data is as follows: Two sets of dynamic cardiac data of size (n f × n p × n t , n f : number of frequency encoding indices, n p : number of phase encoding indices, n t : number of time frames) (224x155x50) and (336x178x48) are acquired with Philips MRI scanner 1.5 T , SSFP sequence, FOV 430x320 mm 2 and TE/TR: 1.46/3 ms. The sparse representations to be tested are two difference frame representations [START_REF] Ji | [END_REF] and x-f space [3]. The difference frame representation is obtained by subtracting the test frame (frame to be reconstructed) from the reference frame; where the reference frame can be any time frame (example: first time frame) or the composite frame obtained by taking the temporal average of all time frames. Suppose I(t i ) is the i th time frame in a sequence of n t dynamic cardiac time frames and I ref is the reference frame, then the sparse representation obtained by using the difference operator is given as

I dif f (t i ) = I(t i ) - I ref , 1 ≤ i ≤ n t where I ref = 1 nt nt k=1 I(t k ), if the
reference frame is composite frame and I ref = I(t 1 ), if the reference frame is first time frame. The x-f space representation is obtained by taking the Fourier transform of dynamic cardiac data along the temporal dimension. For each sparse representation, we fit the exponential and power curves to transform coefficients sorted in decreasing amplitudes. For difference frame representation, the mean decay factor is obtained by taking the average of decay factors estimated for all n t time frames.For x-f space representation, the decay factors are estimated for x-f space corresponding to all n f frequency encoding indices and averaged to get the mean decay factor. To test whether the optimal representation selected on the basis of decay factor values also gives better results practically, we perform the CS reconstruction on randomly under-sampled data with different under-sampling factors. Our CS reconstructions are based on minimizing the l 1 norm as proposed in [4].

RESULTS

The three sparse representations of fully-sampled dynamic cardiac data i.e. difference frame with first frame as refer-ence, difference frame with composite frame as reference and x-f space centered around f = 0 (horizontal axis: temporal frequency f , vertical axis: spatial location x) are shown in Fig. 1

CONCLUSION AND FUTURE WORKS

The power law is a quite good fit for coefficients in sparse representations of dynamic cardiac data and can be used to select the optimal sparse representation of dynamic cardiac

Fig. 1 :Fig. 2 :Fig. 3 :

 123 Fig. 1: Three sparse representations of actual (fully-sampled) dynamic cardiac data

Fig. 4 :Fig. 5 :

 45 Fig. 4: Goodness of fit parameters for power law and exponential law curve fit


Anne-Lise Fourez

Mohamed Akkari

Gregory Gascou

Pierre-Henri Lefevre

Claire Duflos

Akil Kaderbay

Michel Mondain

Frederic Venail

Optimal Duration of MRI Follow-up to Safely Identify Middle Ear Residual Cholesteatoma

come   L'archive ouverte pluridisciplinaire