Jens Emil Grønbaek

Marcel Borowski
email: marcel.borowski@cs.au.dk

Wendy E Mackay
email: mackay@lisn.fr

Michel Beaudouin-Lafon

Mirrorverse: Live Tailoring of Video Conferencing Interfaces

Keywords: Human-centered computing → Collaborative and social computing systems and tools, Collaborative interaction, Webbased interaction, Interactive systems and tools Video conferencing, distributed meetings, end-user tailorability

Scripting Audio Routing

B Reprogramming Tools C Figure 1: Mirrorverse supports live tailoring of video conferencing interfaces. The figure illustrates three novel uses: (A) The recombination of tools enables live recording of camera streams during a yoga instruction session. (B) The audio routing scripting tool enables audio streams of participants to be routed in multi-room workshops. (C) The ability to edit tools enables live reprogramming of their functionality during meetings.

INTRODUCTION

Distributed meetings take many forms depending upon the configuration of people, activities, and shared content. Yet, video conferencing software is traditionally built as one-size-fits-all generic meeting with little support for tailorability of either functionality or the user interface. By contrast, physical meetings let people spontaneously rearrange furniture, equipment, and themselves to match the format and atmosphere of the meeting.

Our goal is to make end-user tailoring of virtual meetings as easy as rearranging physical furniture in a meeting room. Inspired by Mørch [START_REF] Mørch | Three levels of end-user tailoring: Customization, integration, and extension[END_REF], we introduce a five-level ladder of tailorabilitycustomization, recombination, extension, scripting, and reprogramming -to help us explore what it means to tailor video conferencing software live while in use.

We investigate how to reduce friction while moving through the levels of this ladder, especially in light of the increasing demand for remote work and the introduction of new, more tailorable apps and services such as Gather's map customization [14] and Zoom's and Teams' app marketplaces [START_REF]Microsoft Teams Marketplace[END_REF][START_REF]Zoom Marketplace[END_REF]. Current systems currently constrain live tailoring capabilities for end-users to include customization, e.g., enabling Zoom's moderation features; and recombination, e.g., combining elements in a room in the Gather map. Although some platforms such as the Zoom Apps Marketplace also support extensions, each requires using a separate development tool chain outside the application and none support automated scripting or new application behavior. Thus our primary research question is: How can we provide live support for all five tailoring levels in a video conferencing application?

The paper first analyzes what constitutes video conferencing and explores how applying principles from computational media can enhance tailorability. Next, we present Mirrorverse, a proof-ofconcept system that shows how live tailoring can be technically realized in a video conferencing interface. We evaluate the approach through demonstration [START_REF] Ledo | Evaluation Strategies for HCI Toolkit Research[END_REF] and heuristic analysis [START_REF] Olsen | Evaluating User Interface Systems Research[END_REF] using three diverse use scenarios: a virtual workshop activity, an online yoga class, and a stand-up team meeting. These scenarios illustrate how Mirrorverse's live tailoring capabilities enable more dynamic remote meetings with fluid transitions across activities. Tailoring in Mirrorverse is admittedly not yet as easy as rearranging furniture. However, it lets users with different levels of technical proficiency live tailor their interface together. This points to a future of more flexible collaborative software that can be adapted to local needs and practices, even while in use.

RELATED WORK

We review related research on video-mediated communication and commercial video conferencing tools, and tailorable software.

Video-Mediated Communication

The earliest video-conferencing systems were developed to permit remote collaboration within a corporation. In the early 1990s, Bly and colleagues introduced the notion of a Media Space [START_REF] Bly | Media Spaces: Bringing People Together in a Video, Audio, and Computing Environment[END_REF][START_REF] Mackay | Media spaces: Environments for informal multimedia interaction[END_REF], which led to a series of novel user interfaces for coping with the fundamental challenges of video-mediated communication. Buxton [START_REF] Buxton | Mediaspace -Meaningspace -Meetingspace[END_REF] introduced a taxonomy that divides a video communication space into person, task, and reference space. A consistent reference space is essential for effectively using deixis, i.e. when expressions such as "here" or "there" are accompanied by gestures that point at the shared space. A number of systems have been designed to integrate person and task space which establishes a more seamless reference space [START_REF] Ishii | Integration of Interpersonal Space and Shared Workspace: ClearBoard Design and Experiments[END_REF][START_REF] Hara | Blended Interaction Spaces for Distributed Team Collaboration[END_REF][START_REF] Sellen | Using Spatial Cues to Improve Videoconferencing[END_REF].

However, few commercial video conferencing tools have taken advantage of this research and have instead relied upon the same small set of standard interfaces since the 1990s [START_REF] Sellen | Remote Conversations: the Effects of Mediating Talk With Technology[END_REF], namely the "gallery" and "speaker view" layouts. The recent global switch to remote and hybrid work during the COVID-19 pandemic has sparked new interest in video-mediated communication. Recent research has identified key challenges for a new wave of video conferencing, including: navigating multiple audio communication channels [START_REF] Bleakley | Bridging social distance during social distancing: exploring social talk and remote collegiality in video conferencing[END_REF][START_REF] Hu | FluidMeet: Enabling Frictionless Transitions Between In-Group, Between-Group, and Private Conversations During Virtual Breakout Meetings[END_REF][START_REF] Hu | OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings[END_REF] and backchannel communication [START_REF] Kuzminykh | Classification of functional attention in video meetings[END_REF][START_REF] Rae | A Framework for Understanding and Designing Telepresence[END_REF][START_REF] Sarkar | The Promise and Peril of Parallel Chat in Video Meetings for Work[END_REF]; supporting everyday devices not just room hardware [START_REF]Steelcase: Hybrid Collaboration is Hard[END_REF]; managing turn-taking and the conversational floor [START_REF] Baym | Collaboration and Meetings -Chapter 1 of the 2021 New Future of Work report[END_REF][START_REF] Gonzalez Diaz | Making Space for Social Time: Supporting Conversational Transitions Before, During, and After Video Meetings[END_REF][START_REF] Hu | OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings[END_REF]; and supporting meeting configuration [START_REF] Grønbaek | MirrorBlender: Supporting Hybrid Meetings with a Malleable Video-Conferencing System[END_REF][START_REF]Remember Zoom-bombing? This is how Zoom tamed meeting intrusions[END_REF][START_REF] Saatçi | Re)Configuring Hybrid Meetings: Moving from User-Centered Design to Meeting-Centered Design[END_REF].

This has led to multiple new commercial platforms that use stronger and more flexible spatial metaphors, including Gather [14], SpatialChat [START_REF] Spatialchat | SpatialChat[END_REF], Sprout [START_REF] Sprout | Sprout[END_REF], Remo [START_REF] Remo | Remo[END_REF], Teamflow [START_REF] Teamflow | Teamflow[END_REF], Wonder [START_REF] Wonder | Wonder[END_REF] or Ohyay [START_REF]ohyay[END_REF]. In parallel, research prototypes have improved our understanding of spatial metaphors [START_REF] Gonzalez Diaz | Making Space for Social Time: Supporting Conversational Transitions Before, During, and After Video Meetings[END_REF][START_REF] Hu | FluidMeet: Enabling Frictionless Transitions Between In-Group, Between-Group, and Private Conversations During Virtual Breakout Meetings[END_REF][START_REF] Hu | OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings[END_REF][START_REF] Unhangout | Unhangout[END_REF] and introduced new ways of interacting with live and recorded video streams [START_REF] Grønbaek | MirrorBlender: Supporting Hybrid Meetings with a Malleable Video-Conferencing System[END_REF][START_REF] Hu | ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings[END_REF][START_REF] Balasaravanan | Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence[END_REF]. Finally, some popular solutions preserve the legacy gallery and speaker views but also integrate task-space apps and add-ons into their workspaces, including Zoom Apps [67, 68], Teams Apps [START_REF]Microsoft Teams Marketplace[END_REF], Miro's add-on video-conferencing features [START_REF] Miro | Miro[END_REF], and Google's integration of Meet and Docs [START_REF]Google Workspace[END_REF].

Despite some improvements in tailorability, these systems: enforce low ceilings -scope is limited to that provided explicitly by the platform; lack liveliness -solutions rarely support changes during meetings; and introduce friction -transitions to more complex levels of tailoring are difficult. Mirrorverse offers a novel video conferencing approach that addresses each of these problems.

Tailorable Software

Tailorable software, also called customizable software [START_REF] Mackay | Patterns of Sharing Customizable Software[END_REF], adaptable software [START_REF] Trigg | Adaptability and Tailorability in NoteCards[END_REF], or personalizable software [START_REF] Haraty | Designing for Advanced Personalization in Personal Task Management[END_REF][START_REF] Haraty | Online Customization Sharing Ecosystems: Components, Roles, and Motivations[END_REF], refers to software that can be modified and adapted by users to address idiosyncratic needs. Tailoring can occur at different levels [START_REF] Maclean | User-Tailorable Systems: Pressing the Issues with Buttons[END_REF][START_REF] Mørch | Three levels of end-user tailoring: Customization, integration, and extension[END_REF]. For example, MacLean et al. [START_REF] Maclean | User-Tailorable Systems: Pressing the Issues with Buttons[END_REF] describe two tailoring dimensions: tailoring power -which changes are possible at a particular level and tailoring skill requirements -which (technical) skills are needed to tailor the software. For example, changing parameters via a pre-defined menu requires a lower skill level than reprogramming software. Mørch [START_REF] Mørch | Three levels of end-user tailoring: Customization, integration, and extension[END_REF] defines three tailoring levels: customization -modifying the appearance or presentation of software through predefined configuration options; integration -creating a sequence of executions that results in new functionality and extension -writing new code to improve or add functionality. Later, we will expand Mørch's [START_REF] Mørch | Three levels of end-user tailoring: Customization, integration, and extension[END_REF] ladder of tailorability to include five levels.

MacLean et al. [START_REF] Maclean | User-Tailorable Systems: Pressing the Issues with Buttons[END_REF] further describe how different tailoring techniques should be employed at different levels of tailoring power and skill requirement. Their Buttons system lets users tailor functionality on multiple levels, enabling a more "gentle slope" of tailorability compared to the previous example of changing parameters vs. reprogramming. Mirrorverse follows a similar approach based on our five-level ladder of tailorability.

For tailoring to occur, users not only need access to technological solutions, but also a culture in which tailoring is the norm [START_REF] Maclean | User-Tailorable Systems: Pressing the Issues with Buttons[END_REF]. Mackay [START_REF] Mackay | Patterns of Sharing Customizable Software[END_REF] shows that "customization is not a purely individual activity. " Users with different levels of skills can work together, for instance when a peer called a translator can help a regular enduser to tailor their software or act as a link between end-users and programmers. These translators are also called handyman [START_REF] Maclean | User-Tailorable Systems: Pressing the Issues with Buttons[END_REF], gardeners [START_REF] Gantt | Gardeners and Gurus: Patterns of Cooperation among CAD Users[END_REF], or tailors [START_REF] Trigg | From Implementation to Design: Tailoring and the Emergence of Systematization in CSCW[END_REF]. More recently, Haraty et al. [START_REF] Haraty | Online Customization Sharing Ecosystems: Components, Roles, and Motivations[END_REF] have found similar roles in online customization sharing. Mirrorverse supports such practices by enabling translators to tailor the meeting space via programming in the same environment where regular end-users perform customization. This is inspired by a Computational Media approach to software, inspired by Kay's [START_REF] Kay | Personal Dynamic Media[END_REF] early vision of software as computing clay and diSessa's seminal work [START_REF] Disessa | Changing Minds: Computers, Learning, and Literacy[END_REF][START_REF] Disessa | Boxer: A Reconstructible Computational Medium[END_REF] on Boxer. These systems blur the line between using and developing software, making tailoring the modus operandi.

Today, the most prominent examples of computational media are spreadsheets and computational notebooks. Although Webstrates [START_REF] Borowski | Between Principle and Pragmatism: Reflections on Prototyping Computational Media with Webstrates[END_REF][START_REF] Klokmose | Webstrates: Shareable Dynamic Media[END_REF] demonstrates how to apply the principles of computational media to web-based systems, no current system has applied these principles to video conferencing.

DESIGN SPACE FOR TAILORABLE VIDEO CONFERENCING

Based on recent developments in video conferencing platforms and research on end-user tailoring, we articulate a design space for a new wave of tailorable video conferencing.

Design Space of Video Conferencing

Video conferencing systems let people in different locations communicate in real time through video and audio. Users can see and hear each other, as well as share documents and presentations. Although all these systems support standard gallery and speaker layouts, recent platforms have begun challenging these norms. Table 1 defines a design space for describing key characteristics of modern video conferencing systems, with examples of each. We identify three main categories: user presence, space, and interaction, and then specify a set of primitives (highlighted in bold) that describe common features and differences across these systems.

3.1.1 User Presence. All video conferencing software represents each user's presence and lets them control how they are represented to others. Minimal controls including switching video on or off and muting / unmuting the audio from their microphone. Unfortunately as meetings become larger, it becomes harder and harder for users to maintain an overview of each others' presence due to limited screen real-estate. To address this issue, most platforms include a meeting avatar in the form of an icon, image, figure, or name label in a participant list that continuously represents the user's presence in the meeting.

Space.

Most video conferencing systems rely on the room as a central metaphor. Newer systems have significantly expanded users' access to the space, e.g., by allowing users to move around a 2D map to support smooth conversational transitions [START_REF] Gonzalez Diaz | Making Space for Social Time: Supporting Conversational Transitions Before, During, and After Video Meetings[END_REF]. Systems such as in Gather [14], Sprout [START_REF] Sprout | Sprout[END_REF], MirrorBlender [START_REF] Grønbaek | MirrorBlender: Supporting Hybrid Meetings with a Malleable Video-Conferencing System[END_REF], and OpenMic [START_REF] Hu | OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings[END_REF] take advantage of the spatial metaphor's notion of proximity to provide access to virtual furniture, breakout rooms or ad-hoc group formations. When we compare the spatial metaphors of Zoom and Gather, we see that their conversational transitions can be either discrete, e.g., moving instantly between Zoom breakout rooms or continuous, e.g., moving within the Gather map.

The spatial metaphor can also divide users from a large virtual meeting into breakout rooms. These rooms provide a scope for a (sub)group and its content. Content may include video streams from cameras, screens, or recorded video, images, text notes or chat as well as more complex data such as sketches, or tools, e.g., adding a pedestal in the Gather map to broadcast audio from the speaker.

Interaction.

While all platforms support sharing of audio and video streams, they differ in how the interaction is designed, especially how views display streams back to the user and which tools are available to users. Today's most prominent platforms, including Zoom, Google Meet and Microsoft Teams, offer standard gallery and speaker layouts and standard tools such as screen sharing, virtual backgrounds, chat, recording and presence controls: mute / unmute audio, hide / show video. Some offer special user roles, where, for example, the host may have special permission to control other participants' access and presence.

The new wave of platforms has brought new tools and interaction techniques that take advantage of spatial metaphors, e.g., proximitybased group formations [START_REF] Gonzalez Diaz | Making Space for Social Time: Supporting Conversational Transitions Before, During, and After Video Meetings[END_REF] and live shared configuration of video and content [START_REF] Grønbaek | MirrorBlender: Supporting Hybrid Meetings with a Malleable Video-Conferencing System[END_REF][START_REF] Hu | OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video Meetings[END_REF][START_REF] Sprout | Sprout[END_REF]. These systems let users adapt the spatial layout to accommodate different collaboration needs and styles. The trend towards integrating person-space and task-space tools has also increased the need for tailoring, as demonstrated by the way tools have been customized to mitigate "Zoom-bombing" [START_REF]Remember Zoom-bombing? This is how Zoom tamed meeting intrusions[END_REF] or the main platforms' support for integrating apps [START_REF]Google Workspace[END_REF][START_REF]Microsoft Teams Marketplace[END_REF][START_REF]Zoom Marketplace[END_REF].

Tailorability for Video Conferencing

We address three key challenges for integrating tailorability into video conferencing: raising the ceiling, supporting liveliness, and reducing friction.

3.2.1

Raising the Ceiling. We build on Mørch [START_REF] Mørch | Three levels of end-user tailoring: Customization, integration, and extension[END_REF] to define five levels of software tailorability -customization, recombination, extension, scripting, and reprogramming -which we map onto the ladder of tailorability (see Figure 2). 1 Table 2 maps these levels onto current systems.

Customization adapts software functionality or aesthetics using only predefined settings. For example, users can change Zoom's configuration settings to enable or disable the waiting room or give participants permission to unmute themselves.

Recombination assembles ready-made blocks or the primitives in subsection 3.1 to produce new functionality. For example, Gather lets hosts build rooms and define zones with different functionality.

Extension adds new functionality from external sources. For example, Zoom users can add a Miro board or whiteboard.

Scripting adds new behavior using an exposed API or a built-in scripting language. For example, Zoom offers an exposed API for creating extensions.

Reprogramming changes an application's functionality by editing its source code or by interfacing with it directly. For example, technically, Jitsi [START_REF] Jitsi | Jitsi Meet[END_REF] supports changing the source code, although this requires rebuilding the entire application before the change takes effect. Smalltalk [START_REF] Goldberg | Smalltalk-80: the language and its implementation[END_REF] offers a canonical example of a system that supports reprogramming, 3.2.2 Supporting Liveliness. Video conferencing systems with multiple connected users should be able to support live tailoring at each level of the ladder of tailorability during the course of the meeting.

We distinguish among three levels of liveliness when tailoring video conferencing: offline, semi-live, and live: Offline requires restarting the meeting to apply a change. Semi-live means that changes can be applied while the meeting is running, but the tailoring itself cannot be done collaboratively. For example, an organizer can change the layout of a room in Gather while the meeting is running, but the changes are not applied until they save the new layout. Live means that changes can be applied while the meeting is running, and tailoring can be done collaboratively. For example, several users could change the configuration of the breakout rooms together, or edit scripts in a shared code editor.

Reducing Friction.

In theory, any user should be able to move up the ladder of tailorability from customization to reprogramming, if supported by the system. However, use of video conferencing is socio-technical in nature, and friction arises when a user's lack of programming skills, fear of breaking something, or the inherent complexity of the configuration prevents these transitions. Configuration work is a type of articulation work [START_REF] Schmidt | Taking CSCW seriously: Supporting articulation work[END_REF]: It is the work to make work work, i.e., in our case, to make video conference meetings work. Configuration causes technical friction, for example, when additional software such as an IDE is required to create an app for Zoom. Users' skills can also cause friction: scripting requires users to write code which requires programming knowledge. However, even when a user has the requisite skills and tools at hand, the potentially large amount of time required for scripting or reprogramming can render the transition insurmountable. We argue that friction can be greatly reduced by simplifying configuration work.

MIRRORVERSE

Mirrorverse is a system design and proof-of-concept implementation of video conferencing as computational media (see Figure 3). It was inspired by the metaphor of malleable mirrors in MirrorBlender [START_REF] Grønbaek | MirrorBlender: Supporting Hybrid Meetings with a Malleable Video-Conferencing System[END_REF], i.e. live streams that mirror people and screen content and that can be freely arranged in the same virtual workspace. Mirrorverse supports a broad range of meeting experiences by implementing the key primitives of modern video conferencing (described in subsection 3.1) and by supporting live tailorability. For example, Mirrorverse can recreate MirrorBlender as well as meeting experiences similar to Zoom, Teams, or Gather. The core design goal is to support meetings that change over time by allowing idiosyncratic adaptation to the needs that arise as a meeting progresses. Mirrorverse can be tailored at all five levels of the ladder, and changes are applied live, without the need to restart the system.

Video Conferencing as Computational Media

Mirrorverse applies principles from computational media [START_REF] Borowski | Between Principle and Pragmatism: Reflections on Prototyping Computational Media with Webstrates[END_REF][START_REF] Klokmose | Webstrates: Shareable Dynamic Media[END_REF] to realize live tailorability: shareability, malleability, computability, and distributability. Shareability means that all aspects of interaction with the software should be possible collaboratively, including changing the software itself. Malleability means that it should be possible to change the software to suit individual needs. Computability means that it should be possible to treat what is seen on the screen as data for computation or -as diSessa puts it -that the computational structure should be in the accessible parts of the medium [START_REF] Disessa | Changing Minds: Computers, Learning, and Literacy[END_REF]. Distributability means that software should be able to flexibly span across available devices. For video conferencing software, these principles mean that we should treat software as a malleable medium within which to conduct meetings, a medium that can be shaped to fit the specific meeting type and meeting culture. This should be possible before, during, and after the meeting and it should be possible to do so collaboratively. It implies that the fundamental primitives and mechanisms of video conferencing must be reified [START_REF] Beaudouin-Lafon | Reification, Polymorphism and Reuse: Three Principles for Designing Visual Interfaces[END_REF] into the user-accessible parts of the medium. For example, so that where a participant is located in a meeting space can be used computationally to control their audio (muffled, muted, volume, etc.) in a way that is computationally accessible to the user. Tools and computations should be polymorphic so they can be applied and reused wherever it makes sense to the user [START_REF] Beaudouin-Lafon | Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces[END_REF]. E.g., if a tool can manipulate the pixels of a video feed, it should be applicable to both live and recorded video.

Mirrorverse builds on the Webstrates family of software [6, 29] and the Varv [START_REF] Borowski | Varv: Reprogrammable Interactive Software as a Declarative Data Structure[END_REF] programming model to realize the principles of computational media (details in section 6). Mirrorverse is accessed through a Web browser.

Mirrorverse Overview

The design of Mirrorverse is based on a 2D nested canvas, 2 which we call the workspace, that combines person space and task space in the same environment -similar to, e.g., Sprout [START_REF] Sprout | Sprout[END_REF]. The space is structured into rooms that can contain elements that are either content, tools or nested rooms. Rooms can be navigated and elements can be moved from room to room depending on the use case. Tools make it possible to add functionality to a room -either before or during a meeting. Unlike systems such as Zoom, the layout of a meeting is persisted after a meeting ends so that a meeting can be continued at later points in time or a layout can be prepared beforehand, e.g., by setting up breakout rooms. This layout can be collaboratively manipulated live. Other key concepts include an audio routing mechanism that defines which clients are audible and how, and using recordings during a meeting. 4.2.1 Elements: Rooms, Content, and Tools. Elements are either rooms, content, or tools. They are displayed in the workspace as rectangles, which can be selected by clicking them and have options that can be customized with the inspector (see below). New elements can be added using the menu bar by selecting a content type or a tool to be added (see Figure 3A). Existing elements can be removed using the inspector. A room acts as a container for other elements, which can be nested. Our prototype features eight types of content (Camera, Screen, Video, Image, Note, Sketch, Chat, Avatar) and twelve tools (Grid, Speaker View, Layout Template, Doorway, Broadcast, Whisper, Proximity-based Audio, Pedestal, Recording, Room Recording, Template Store, Highlight, Camera Shaker). 3 Both content and tools are located in the same workspace.

Tools are a type of element that adds new functionality to a room, such as a tool for highlighting elements when clicking on them or a tool that arranges elements in a grid layout. A tool's functionality is tied to the room it is located in -tools from other rooms do not affect the current room. Elements and their location and size in the room and workspace follow the What-You-See-Is-What-I-See (WYSIWIS) principle [START_REF] Stefik | WYSIWIS Revised: Early Experiences with Multiuser Interfaces[END_REF]. Content in elements follows a relaxed WYSIWIS model: while the state is the same, the exact visual representation might differ, e.g., the content of a note is shared while the scroll position is private to each client. Similarly, tools might differ in their visual representation and whether they are active, e.g., when a tool is active for one user role but deactivated for another.

Workspace and Tool Panel.

The workspace is a central fixedsized 2D canvas (see Figure 3B). It always displays the elements of the current room a client is located in. As mentioned above, the canvas is WYSIWIS, so that its elements always have the same location, size and transparency for all clients. Users can move elements around in the canvas, resize them, change their opacity and layering order, and move them in and out of nested rooms.

To the left of the workspace is the tool panel (see Figure 3C): When adding a new tool to a room, the tool is by default added to the tool panel. This reduces the clutter in the workspace. However, as an option, a toggle can be switched so that the tool will move to the workspace, allowing users to treat it as a moveable element.

Inspectors.

To the right of the workspace are two inspectors (see Figure 3D): one for the current room and the other for the currently selected element. They are used to configure their options, e.g., the name of the room or the URL of an image, and to delete elements. For some tools, e.g., the inspector can set the roles for which the tool should be active.

4.2.4

Users, Clients, and the Client Panel. Each device, browser, or browser tab visiting a meeting is a named client. Each client, in turn, is connected to a user, who has a name and a role. The role can, e.g., be used to restrict a tool's functionality to groups of users. The client panel (see Figure 3E) is a sidebar that displays information The distinction between clients in the room and clients that are audible is needed since clients from other rooms may be audible (see below in the audio routing section).

Techniques for Tailorability

Mirrorverse supports tailorability through the following techniques, corresponding to the five levels of the ladder of tailorability (see Figure 2 and Table 2): 4.3.1 Level 1: Customizing Element Options. Elements in Mirrorverse have various options that can be customized using inspectors. These may range from preferences, e.g., the color of a note, to more functional options, e.g., which element is considered the "speaker" in a room when using the Speaker View tool. This level of tailorability is typically immediately accessible to users at all levels of proficiency. An example tool is the Recording tool. The ability to record the live media streams of a meeting is common in video conferencing but the recordings can rarely be used within the meeting (with a few exceptions, e.g., [START_REF] Hu | ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video Meetings[END_REF][START_REF] Balasaravanan | Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence[END_REF]). Mirrorverse supports the recombination of live and recorded videos, so that users can for example demonstrate something to the camera and then play it back and refer to the recording in the meeting (subsection 5.1 demonstrates this).

This and other types of recombination require configuration work, such as setting up rooms with different layouts and different types of tools. The Template Store tool makes it possible to store combinations of tools, content, and layout as templates and restore them later. It effectively reifies such combinations into new objects that can be reused. For example, we created templates that replicate Zoom-like and Gather-like interfaces 4 (see Figure 3) by combining different content elements and tools.

Level 3: Extending Content and Tool Types with Packages.

This level of tailorability allows the technically proficient user to seek out new functionality to support a given meeting situation, and for the user who is proficient in programming to share new functionality with other users. Content types in Mirrorverse, such as images, notes or sketches, are bundled as Webstrates packages [START_REF] Borowski | Codestrates v2: A Development Platform for Webstrates[END_REF]. Packages can be added and removed from a meeting at run-time, supporting tailoring by extension. If a user implements a new type of content, e.g., a chess game, they can add it to another meeting simply by dragging and dropping it into the Cauldron IDE of the other meeting (see section 6). Adding a tool package works in the same way and extends the list of available tools in the drop-down menu in the toolbar. The audio routing in Mirrorverse is a scripting technique that enables control of the above three properties based on the state of the system. It serves as one example technique at this level of tailorability, which lets users with a high degree of technical proficiency add or modify behavior of the interface without using conventional programming.

The audio routing can be authored either directly in Varv code or by using the audio routing GUI. Each property is calculated using a decision tree that can refer to a variety of states for determining its value. The GUI lets users create one root node for each property, decision nodes that depend on properties, and value nodes that set the value of the property at the leaves of a tree (see Figure 4; subsection 5.2 demonstrates the GUI). Nodes can be created and connected by dragging from the green connection boxes to other nodes. For example, the numeric property logicalDistance describes how many rooms a client is away from another client. This property can be used to determine the volume of an audio stream, e.g., if the logical distance of the client is zero, the volume is set to 100 %, if the distance is one it is set to 50 %, and if it is larger than one it is set to 0 % (see Figure 4).

Level 5:

Reprogramming Using the Built-in Editor. Everything in Mirrorverse, from content types and tools to the core of the system, can be reprogrammed live. This level of tailorability requires programming expertise, but as we will demonstrate later, live collaborative tailoring allows a user without the necessary expertise to realize a change to the interface through the help of a more technically proficient user.

To start reprogramming, users click the "Edit" button in the top right corner (see Figure 5A). This opens the built-in Cauldron IDE, giving access to the code (see Figure 5B). Reprogramming the system happens from within the web browser and no additional tools are required (subsection 5.3 demonstrates this). Edits to Varv concept definitions are immediately applied to the running system live, however, a concept definition can also be disabled while editing.

DEMONSTRATING SCOPE AND DEPTH

In the following three usage scenarios, we demonstrate how Mirrorverse addresses key challenges of video conferencing: managing audio routing, communicating through video, and adapting tools to impromptu activities. Together, these scenarios demonstrate the breadth and depth of tailoring with Mirrorverse through the power of the reification principle. 5

Dynamically Recombining Live and Recorded Video in a Yoga Class

Recombining video streams is traditionally done prior to a meeting, e.g., using OBS [START_REF] Obs | OBS: Open Broadcaster Software[END_REF], or after a meeting, e.g., recording and sharing meeting videos. We demonstrate how reified video in Mirrorverse enables users to dynamically recombine live and recorded video for remote instruction in an online yoga class (see Figure 6). two instances of the same camera stream with different crops. One crop shows the mat for portrait-mode shots of poses, the other for landscape mode (see Figure 6B).

Impromptu

Recording for Live Demonstration. The yoga session starts. John spots that several participants are doing one of the poses incorrectly with the risk of causing a knee injury. He grabs everyone's attention by dragging his video to the recording tool to record a video of himself performing the pose from the prepared sequence (see Figure 6C), but this time from a different viewing angle that better conveys the potential problem. He then navigates the video clip he just recorded and stops at a specific point to highlight an important detail in the body posture that helps avoid injuries (see Figure 6D).

Scripting Audio Navigation in a Multi-Room Workshop

Virtual workshop organizers often struggle to navigate audio in multi-room breakout sessions. We demonstrate how Mirrorverse reifies audio as the Audio Routing GUI to enable new forms of audio navigation among multiple breakout rooms (see Figure 7).

the Workshop Interface.

To prepare for the upcoming workshop, Ann, the organizer, sets up the meeting with a main room and a set of breakout rooms nested within it (see Figure 7B-C). She anticipates that hearing muffled sounds from the breakout rooms will help her monitor the activity. She opens the Audio Routing GUI (see Figure 7A) and tailors the audio routing so that people in rooms at a logical distance of 1 (i.e. the distance from the main room to the breakout rooms) are still audible but with a low-pass filter applied to convey the notion of distance.

Adding a New Tool for Listening

In. With the sound being muffled, Ann cannot hear the contents of the individual conversations, and she finds it cumbersome and disruptive to have to explicitly jump in and out of a breakout room to hear the sound clearly. Instead, she wants an interaction style similar to standing in the doorway of a physical breakout room. She finds and adds the Doorway tool, which allows her to listen in on the audio from a breakout room by hovering over it with the mouse cursor. The tool uses audio routing to lower the logical distance to 0 for the room at the cursor position (see Figure 7D). Finally, Ann wraps up the breakout activity by using the Broadcast tool, which sends her audio to all rooms, to ask participants to go back to the main room.

Impromptu Reprogramming of Deictic Tools in a Stand-Up Meeting

Distributed teams often need to tailor their workspace to the specific collaboration needs arising during an impromptu discussion. We demonstrate how the reification of tools that can be reprogrammed live can be used to support transitions from planned to ad hoc activities in a design team's stand-up meeting (see Figure 8).

Adding a New

Tool for Highlighting Elements. To prepare for their daily stand-up meeting, Roman, the project manager, restores the "Stand-Up Meeting" template using the Template Store tool (see Figure 8A). It includes the Grid and Highlight tools. The latter can be used to manage turn-taking in their status round. With the tool enabled, Roman can click on a participant's video to highlight it for everyone (see Figure 8B). This can accompany deictic expressions when referring to others as "you, " rather than using their names.

Tools are Polymorphic Across

Person and Task Space. Daniel, a designer, and Melissa, a programmer, get into an impromptu discussion about an urgent task. Roman decides to pause the status round and facilitate the discussion. They reorganize the workspace to focus on the task (see Figure 8C). As they switch their activity to sharing, producing, and arranging content together in the digital task space, they repurpose the Highlight tool to support deictic expressions related to the shared notes and images. This is possible because tools in Mirrorverse are polymorphic, meaning that they can be used on different types of content elements.

5.3.3

Reprogramming the Highlight Tool. The current Highlight tool can only select one element at a time. During the coffee break, Melissa and Roman reprogram the tool to enable the selection of multiple elements (see Figure 8D). Participants can now use deictic expressions such as "these two" or "you three" while referring to groups of people or content.

5.3.4

Storing the New Workspace as a Template. As they wrap up their meeting, they collectively agree that their new workspace layout and tool combination worked well and they want to keep it. Roman stores the final workspace as a template for future reuse.

Other Examples

To further demonstrate the power of reification and live tailoring in Mirrorverse, we implemented a few additional examples. communication. It works like the Doorway tool, but instead of hovering over a room to listen in, users can hover over a participant's video with their cursor to have a short one-to-one conversation with that participant, which others cannot hear.

Audio as Input to Modify Video Output.

To demonstrate the flexibility of audio reification, we implemented a Camera Shaker tool that uses audio as input to modify the video's appearance. It can be used to amplify and enhance non-verbal cues, such as making the video window shake when a person is too loud, i.e., when the volume exceeds a user-defined threshold.

Distributing Content Elements Across Devices.

Using the principle of transclusion [START_REF] Klokmose | Webstrates: Shareable Dynamic Media[END_REF] from the underlying Webstrates platform, it is possible to distribute content elements across multiple screens and devices. We implemented a self-contained chat mechanism that can be both embedded in a meeting as content and accessed directly on a companion device (e.g., a mobile phone; see Figure 9).

IMPLEMENTATION

Mirrorverse6 is built on top of the Webstrates software stack consisting of Webstrates [START_REF] Klokmose | Webstrates: Shareable Dynamic Media[END_REF], Codestrates v2 [START_REF] Borowski | Codestrates v2: A Development Platform for Webstrates[END_REF] and Varv [START_REF] Borowski | Varv: Reprogrammable Interactive Software as a Declarative Data Structure[END_REF]. It is implemented in Varv, HTML, and CSS. Extensions to Varv are written in JavaScript. It runs client-side in a Web browser and can be modified using Codestrates v2's built-in IDE, Cauldron.

Web-based Software Stack

Webstrates serves web pages, called webstrates, so that the document object model (DOM), including embedded JavaScript and CSS, is synchronized in real-time between clients and persisted on the server. A webstrate is self-contained and identified by its URL. Webstrates, further, allows for WebRTC streaming of audio and video channels. Codestrates v2 [START_REF] Borowski | Codestrates v2: A Development Platform for Webstrates[END_REF] is a development platform for Webstrates consisting of an execution engine and a package manager. Users author code via the Cauldron IDE from within a webstrate, without additional tools. Mirrorverse uses the state synchronization and WebRTC streaming capabilities of Webstrates, and the package manager and collaborative code editor of Codestrates.

Varv [START_REF] Borowski | Varv: Reprogrammable Interactive Software as a Declarative Data Structure[END_REF] is built on top of this stack as a new programming model for computational media. It supports authoring interactive behavior of software live and collaboratively. It uses a declarative data structure written in JSON to define interaction. 7 The declarative model enables accretive extensibility: applications in Varv are inherently extensible and their interactive behavior can be changed by adding code instead of having to modify existing code -similar to how CSS rules can be overwritten. A view layer supports creating GUIs using HTML templates and CSS. Varv persists the program's state in the DOM, which is synchronized among clients by Webstrates.

Using the Webstrates stack and Varv was a pragmatic choice as they support collaboration and live reprogramming within the same medium. Webstrates could be substituted with other clientserver architectures such as a Node.js server, and Codestrates by other web-based code editors such as CodeMirror [START_REF] Codemirror | CodeMirror[END_REF] and a package manager like NPM. Varv, however, was introduced as a novel programming model and we are not aware of other models with similar capabilities.

In terms of the ladder of tailorability, techniques at the levels of customization, recombination, and extension are mostly independent of the stack. It is only the techniques at the levels of scripting and reprogramming that rely on Varv. Yet, these techniques would also be applicable to other future live programming models.

Architecture

The code of Mirrorverse is structured in two types of packages. The first type are low-level extensions to Varv implemented in JavaScript. Among other things, they provide interfaces to the We-bRTC [START_REF]WebRTC API[END_REF] and Web Audio [START_REF]Web Audio API[END_REF] APIs, and enable recording video. The second type are packages that implement the Mirrorverse platform. Each of these packages contains a combination of Varv code, HTML templates, CSS styling, and audio routings. They are structured into managers for handling users, audio and video streams, tools, and the chat, and elements that implement rooms, content, and tools. The managers and the room elements are the core features of Mirrorverse, all other content and tool element types (e.g., Camera, Image, Grid tool) are implemented as independent packages. This enables adding and removing packages at runtime because dependencies between elements are rare. 8 Currently, Mirrorverse supports eight content types and twelve tools (see Appendix A). However, as Mirrorverse builds on the power of web technologies, new content types can, e.g., be created by reusing existing JavaScript libraries or by embedding existing web apps.

The Audio Routing Mechanism

Audio routing is implemented using both Varv and JavaScript. It is part of the Mirrorverse infrastructure and can be used by itself in Cauldron or when creating tools. The routing decision tree is defined in Varv -this is necessary to allow for the routing to be changed live during a meeting. Once the properties of audio streams are set in the Varv part of the routing, they are saved to the DOM, where they are picked up by the JavaScript part of the audio routing. The JavaScript part reads the properties from the DOM and changes the corresponding properties on the WebRTC stream using the Web Audio API, e.g., using a GainNode to change the volume.

The audio routing GUI (see Figure 4) was created on top of the audio routing mechanism as an authoring tool. It is implemented in JavaScript and integrated into the Cauldron editor of the Webstrates stack. Decision trees created with the audio routing GUI are exported as Varv code used by the audio routing mechanism. This also enables an escape hatch by allowing to author an audio routing using the GUI for the overall tree and implementing advanced features directly in the exported Varv code.

DISCUSSION

We discuss the key implications of our work and evaluate Mirrorverse using Olsen's heuristics for user interface systems [START_REF] Olsen | Evaluating User Interface Systems Research[END_REF]. (Relevant qualities are highlighted in bold).

Systems-Oriented Evaluation

Enabling tailoring of video conferencing empowers participants to become designers. Lowering the friction in climbing the ladder of tailorability facilitates the transition from user to designer, and encourages participants to design or change a feature in collaboration with a more experienced colleague.

A Mirrorverse meeting is a Varv program consisting of a set of primitives specific to video conferencing (see subsection 3.1). Because a Varv program is a data structure, it is also easy to generate Varv code procedurally. This means that a domain-specific scripting tool such as the audio routing GUI can be built without having to explicitly create an API for it (as required by others, see Table 2). Thus, programming with Mirrorverse improves expressive match as it encourages creating and using domain-specific abstractions [START_REF] Borowski | Varv: Reprogrammable Interactive Software as a Declarative Data Structure[END_REF] i.e. design thinking in terms of video conferencing primitives.

Mirrorverse supports inductive combination at multiple levels. For example, tool elements are primitives that can be recombined in countless ways. This lets users add new tools via extension thus enabling multiple new use situations not covered in this paper. Fundamentally, tailorability offers flexibility and liveness improves flexibility. Users can rapidly experiment with new features even during an ongoing meeting with multiple participants.

Tailorability, Friction, and Usability

We have demonstrated how Mirrorverse reduces friction from a technical perspective by making tailorability possible at each level and by enabling live transitions between them. However, future work is necessary to evaluate if and how users will develop, use and combine tools with Mirrorverse, and whether or not this significantly reduces friction under real-world use.

Mirrorverse poses several usability challenges. While Mirrorverse's ability to shift from configuration to implementation work while running a meeting raises the ceiling of tailorability, it also introduces a usability trade-off, since the increased flexibility also increases the risk of breaking the system as it runs. Borowski et al. [START_REF] Borowski | Between Principle and Pragmatism: Reflections on Prototyping Computational Media with Webstrates[END_REF] argue that computational media needs safeguards to prevent accidental breakage and should always support easy reversion to a previous working version. Mirrorverse let's users revert changes both to the source code and/or to the application state (e.g., the precise configuration of a meeting at a given point in time) using the version history of Webstrates [START_REF] Klokmose | Webstrates: Shareable Dynamic Media[END_REF].

Even without errors, live reconfiguration can be disruptive and distracting. Organizers of physical meetings handle this by preparing for changes in advance. Digital meeting organizers should also be able to prepare changes "on the side, " before making them live. For example, an organizer could rearrange breakout rooms during a plenary session, before sharing them with everyone. Moreover, building and recombining tools also require work. We provide Template and Extension Stores to facilitate easy reuse and make this work a community effort, which should reduce friction.

Another usability challenge is awareness: when live changes made by a user affect other users, awareness mechanisms should help the other users understand the change. This is a well-known challenge in groupware research, and this literature could inform good mechanisms for enhancing awareness of the who, what, and where [START_REF] Gutwin | A descriptive framework of workspace awareness for real-time groupware[END_REF] of live collaborative tailoring.

Performance and Scalability

Video and audio communication use only peer-to-peer communication using WebRTC. This does not scale well beyond a handful of users. Handling large meetings would require a media server for multiplexing video and audio streams in order to reduce resource demands on clients. Mirrorverse performs well in relatively complex room setups. However, there are numerous optimizations to be done in the Varv runtime engine that would increase performance.

Another scalability issue relates to the approach of recombining tools. As the collection of extensions and templates grows, recombination may lead to conflicts that cause unexpected behavior for the users. For example, two different tools added to the same meeting may lead to unexpected results or even errors if they operate on the same data, such as conflicting audio routing logic.

Hybrid Meetings

To reduce the complexity in our usage scenarios, we have focused on conventional video conferencing situations where participants connect from each their own device. But in hybrid meetings, physically co-located participants may share a device, such as a large meeting display, to connect to one or more remote participants. These types of meetings have become commonplace, but are also known to be fraught with challenges [START_REF] Saatçi | Hybrid meetings in the modern workplace: stories of success and failure[END_REF].

The core interactions in Mirrorverse replicate the experience of MirrorBlender [START_REF] Grønbaek | MirrorBlender: Supporting Hybrid Meetings with a Malleable Video-Conferencing System[END_REF], i.e., by letting users reposition, resize, and adjust translucency of their video windows. These interactions were originally designed to address challenges with deictic referencing in hybrid meetings. Although out of scope for this paper, the malleability of Mirrorverse provides a great foundation for further experimentation with new ways to address these challenges in hybrid meetings.

Directions for Future Work

Our design space covers the current trend of commercial video conferencing platforms and recent research prototypes, or what is sometimes referred to as the 2D Metaverse [START_REF] Larsen | Virtual communication without sniper rifles: How the 2D metaverse is changing virtual spaces[END_REF]. However, future work should consider support for configuring 3D environments, such as mixed and virtual realities [START_REF] Grønbaek | Partially Blended Realities: Aligning Dissimilar Spaces for Distributed Mixed Reality Meetings[END_REF][START_REF] Xia | Spacetime: Enabling Fluid Individual and Collaborative Editing in Virtual Reality[END_REF] and cross-device video conferencing [START_REF] Marquardt | AirConstellations: In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures[END_REF]. We demonstrated how to move chat windows onto a companion device. This type of cross-device interaction could potentially be extended to new interactive devices such as head-mounted displays.

Future work should further explore the possibilities of reifying video conferencing primitives into interactive objects. Our examples of applying filters and rules to video and audio only show the beginning. An obvious next direction is to reify text as a key data format. Video conferencing includes a variety of text content, such as chat, embedded notes, and live transcriptions. With the flexibility and support for quickly experimenting with such different parts of the interface, Mirrorverse is an ideal platform for conducting studies on how to improve different aspects of online meetings.

Finally, our demonstration of applying principles of computational media hints at a future where video conferencing is not trapped inside an application, but part of the computational fabric itself, allowing for deeper integration of person and task space.

CONCLUSION

The ongoing global switch to hybrid work calls for a new breed of video conferencing systems that can be more easily configured by end users to fit their needs. We have introduced a design space for live tailoring of video conferencing interfaces and a system design based on principles of computational media that spans all five levels of the ladder of tailorability. We have described Mirrorverse, a proofof-concept implementation of this design, and have demonstrated through a series of examples the range of video conferencing experiences and reconfiguration capabilities that it enables. Mirrorverse extends tailoring in two important ways: (1) it increases the ceiling of tailorability with reprogramming, and (2) it reduces tailoring friction by supporting live collaborative changes and providing constructs, such as templates and a domain-specific language, to simplify configuration work. Future work includes extending this approach to 3D environments and multi-device settings, as well as exploring the reification of more video conferencing primitives.

B VARV CODE EXAMPLE

Figure 2 :

 2 Figure 2: The Ladder of Tailorability: We raise the ceiling of tailorability and reduce the friction of climbing the ladder.

Figure 3 :

 3 Figure 3: The Mirrorverse prototype. New elements can be added using the menu bar (A). Content and room elements are added to the Workspace (B). The top left corner of the Workspace shows the name of the room and breadcrumbs if it is located in other rooms. The top right corner of the Workspace shows all clients currently in the room. Tools can exist as elements in the Workspace but are initially added to the Tool Panel (C). Selecting any element shows its options in the element inspector, e.g., the URL of an image (D). Navigation to move between rooms and options of the current room (room inspector) are placed above that inspector (D). The Client Panel (E) shows the current user, client, and other audible clients in the meeting. about one's client and user, provides an overview of the clients that are currently audible. Clients that are currently in the same room as oneself are displayed in the top right corner of the workspace.The distinction between clients in the room and clients that are audible is needed since clients from other rooms may be audible (see below in the audio routing section).

4. 3 . 2

 32 Level 2: Recombining Tools and Templates per Room. This level of tailorability allows more technically proficient users, such as meeting hosts, to customize the experience. Tools in Mirrorverse are located in rooms and their functionality is limited to the room they are located in. This enables the recombination of different tools (or content) in a room to create different meeting experiences, from existing setups similar to those of Zoom or Gather to novel examples as we demonstrate later.

4. 3 . 4

 34 Level 4: Scripting the Audio Routing. In Mirrorverse, every client has an audio stream with several properties: muted, which defines whether the stream is audible; volume, which defines the volume of the stream; and filter, which defines what audio filter (e.g., a low-pass or high-pass filter) is used on the stream.

Figure 4 :AFigure 5 :

 45 Figure 4: The audio routing GUI. In this example, the volume of audio streams is dependent on the client's logical distance.

5. 1 . 1 Figure 6 :

 116 Figure 6: Scenario 1: Dynamically Recombining Live and Recorded Video in a Yoga Class.

5. 4 . 1 Figure 7 :Figure 8 :

 4178 Figure 7: Scenario 2: Scripting Audio Navigation in a Multi-Room Workshop. Notice the cursor hovering over a room in (D), causing the audible clients in the client panel to change to the clients in the hovered room.

Figure 9 :

 9 Figure 9: A dedicated chat application (left) can run on a different device but connect to the same chat messages as Mirrorverse (right) using Varv's data store mechanism.

26 " 27 "

 2627 when ": { " click ": { " concept ": " element " }} ,

Table 1 :

 1 Dimensions and primitives of video conferencing, used to categorize a selection of commercial and research systems.

Table 2 :

 2 Levels of tailorability with examples of their realization in video conferencing systems ( * offline; † semi-live; ‡ live).

	Levels	Customization	Recombination	Extension	Scripting	Reprogramming
	Examples
	Zoom, Teams, Discord,	Pre-customize UI †		App integrations †	Zoom API *
	Slack, Google Meet
	Gather	Pre-customize the map ‡	Combining existing map		Gather API *
			layouts ‡
	MirrorBlender	Ad hoc customizing meeting space ‡				Reprogramming in the Cauldron editor *
	Mirrorverse	Customize tools with the inspector ‡	Adding tools to a room ‡ ; Templates ‡	Content and tools are bundled as packages ‡	Audio Routing ‡	Reprogramming in the Cauldron editor in Varv ‡

Given changes in software and vocabulary since the 1990s, we have nuanced and expanded on Mørch's original three levels: customization, integration, and extension.

While we present one design of such a system, the dimensions in our design space could also be used to create other system designs. E.g., we based our design on a 2D user interface, leaving

3D interfaces like virtual and augmented reality to future work.

A full list of element types and their descriptions is available in Appendix A.

These replicated examples are demonstrated in the accompanying video.

Mirrorverse on GitHub: https://github.com/Webstrates/Mirrorverse

A code example of Varv is provided in Appendix B.

A few packages have dependencies, e.g., the recording tool requires camera content.

ACKNOWLEDGMENTS

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 740548). Thanks to Rolf Bagge and Janus Bager Kristensen from CAVI at Aarhus University for their support in the development of Mirrorverse.

A MIRRORVERSE ELEMENT TYPES A.1 Content Types

Name Description

Camera Displays a live camera stream.

Screen

Displays a live screen capture stream.

Video

Plays a video file.

Image

Displays an image file.

Note

Displays a plain text note similar to a post-it.

Sketch

Displays a 2D sketch drawing board using an transcluded tldraw [START_REF] Tldraw | tldraw[END_REF] project.

Chat Displays an interactive chat.

Avatar Displays an avatar that is used in the proximity-based audio tool.

A.2 Tools

Name Description

Grid

Changes the layout of elements in the workspace to a grid.

Speaker View

Changes the layout of elements in the workspace to a view where one element-the speaker-is displayed large and other elements are displayed in a filmstrip above the speaker.

Layout Template

Stores the layout of elements in a room and restore the layout later (this only restores the position of existing elements but does not recreate the elements).

Doorway

Allows to listen in on nested rooms by hovering over it with the mouse cursor.

Broadcast

Routes the audio to all rooms.

Whisper

Allows to whisper to other clients in the same room by hovering over their camera feed.

Proximitybased Audio

Changes the audio routing to be based on the location of avatars in a room.

Pedestal

Makes avatars that are close to the pedestal audible to all other clients in a room when using the proximitybased audio.

Recording

Creates recordings of single camera streams.

Room Recording

Creates a recording of the whole workspace of the current room.

Template Store

Enables to store element layouts and restore them later (including recreating the elements).

Highlight

Enables to highlight content WYSIWIS in the workspace by double clicking on it.

Camera Shaker

Shakes the camera stream of clients that are talking loudly in a room.

{ " where ": { 46 " property ": " selected " , Varv code is written in the JSON format. It is structured in concepts, e.g., a specific tool is its own concept (similar to classes in regular object-oriented programming (OOP)). Each concept consists of a schema, actions, and extensions. The schema defines the properties of a concept. For example, we add the Boolean property globalHighlighted to the schema of an element to be able to set its highlight value to true or false. Actions define the interactive behavior of a concept and are trigger-action (when-then) rules. The when parts define when an action should be triggered, e.g., when the user clicks on a certain type of concept. This part is optional and actions can also only consist of a then-part. The then part defines which action chain should be run, passing an event along from one action to the next. Actions are applied to selections of concept instances (concept instances are similar to objects in OOP). Finally, extensions allow for polymorphism and can be used to inject one concept into another. For example, the element concept has properties for its position and size; by injecting the concept into the highlightTool it inherits these properties.

Actions in Varv are named and can be used in other action chains. This makes it possible to create a domain-specific language for a certain type of software. In Mirrorverse, we created the action checkInFullscreenRoom to check whether an element is in the current room that is active in the workspace (the fullscreenRoom).

Actions such as this one can be used when implementing elementsboth in content and tools. Similarly, in the highlight tool, we added the action checkHighlightActive which checks whether the highlight tool is active and only continues the action chain if it is.

C ADDING TOOLS TO MIRRORVERSE

Adding new concept types or tools to Mirrorverse works similar to reprogramming a tool as described in Section 4.3.5: The user also has to open the Cauldron IDE using the "Edit" button but instead of editing existing code needs to create a package for the new content type or tool and add a Varv concept definition and a Varv template with some boilerplate code (see below for an example). The tool can be used afterwards.