Sebastian Gottschalk
email: sebastian.gottschalk@uni-paderborn.de

Enes Yigitbas
email: enes.yigitbas@uni-paderborn.de

Eugen Schmidt
email: eschmidt@mail.uni-paderborn.de

Gregor Engels
email: gregor.engels@uni-paderborn.de

ProConAR: A Tool Support for Model-based AR Product Configuration

Keywords: Product Configuration, Augmented Reality, Model-based, Tool Support

Mobile shopping apps have been using Augmented Reality (AR) in the last years to place their products in the environment of the customer. While this is possible with atomic 3D objects, there is is still a lack in the runtime configuration of 3D object compositions based on user needs and environmental constraints. For this, we previously developed an approach for model-based AR-assisted product configuration based on the concept of Dynamic Software Product Lines. In this demonstration paper, we present the corresponding tool support ProConAR in the form of a Product Modeler and a Product Configurator. While the Product Modeler is an Angular web app that splits products (e.g. table) up into atomic parts (e.g. tabletop, table legs, funnier) and saves it within a configuration model, the Product Configurator is an Android client that uses the configuration model to place different product configurations within the environment of the customer. We show technical details of our ready to use tool-chain ProConAR by describing its implementation and usage as well as pointing out future research directions.

Introduction

In the last years, Apple and Google have pushed the topic of Augmented Reality (AR) in mobile apps within their mobile ecosystems. One area of focus is mobile shopping [START_REF] Chatzopoulos | Mobile Augmented Reality Survey: From Where We Are to Where We Go[END_REF], where AR can support the decision process of the customer with additional product information, direct placement of products in the environment, and a greater product choice [START_REF] Dacko | Enabling smart retail settings via mobile augmented reality shopping apps[END_REF]. Examples for these mobile apps are the placing of an atomic product in the environment as in IKEA Place app 3 , the configuration of products like windows in the VEKA Configurator AR app 4 , or the placement of multiple products in an environment as recently announced by Amazon 5 . While these approaches focus on simple placements, they neglect the runtime configuration of 3D object compositions based on user needs and environmental constraints.

To improve this configuration process, we previously presented a model-based AR-assisted product configuration [START_REF] Gottschalk | Model-based Product Configuration in Augmented Reality Applications[END_REF] that uses the concept of Dynamic Software Product Lines (DSPLs) [START_REF] Capilla | An overview of Dynamic Software Product Line architectures and techniques: Observations from research and industry[END_REF]. The approach consists of a Design Stage and a Runtime Stage. At the Design Stage, we use the concepts of feature models and reuseable assets from Software Product Lines (SPLs) [START_REF] Apel | Feature-Oriented Software Product Lines[END_REF] to split the product representation and its possible configurations from the corresponding 3D objects. Out of both, we export a configuration model that is used at runtime. At the Runtime Stage, we import the configuration model and use the MAPE-K architecture [START_REF] Kephart | The vision of autonomic computing[END_REF] to monitor the user needs in the form of user inputs, the actual product configuration, and the environment. Based on the measured inputs, we configure possible product configurations within the environment. In contrast to our research paper [START_REF] Gottschalk | Model-based Product Configuration in Augmented Reality Applications[END_REF], in this demonstration paper we focus on the technical implementation of the corresponding tool ProConAR and point our future research directions. ProConAR, as shown in Fig. 1, consists of a Product Modeler and a Product Configurator. The Product Modeler, which is used at the Design Stage, is an Angular web app based on an existing feature model editor in [START_REF] Gottschalk | Hypothesis-driven Adaptation of Business Models based on Product Line Engineering[END_REF]. It can be used to create feature models of possible product configurations (see Fig. 1 (a) for a developed kitchen model). Moreover, for each feature, a 3D object together with additional attributes can be saved. The created configuration model can be exported for the Product Configurator. The Product Configurator, which is used at the Runtime Stage, is an Android client based on the Unity Engine6 . It imports the configuration model and uses the information to configure products within the environment (see Fig. 1 (b) for kitchen configuration in the environment). Moreover, it adapts the product configuration to the user needs, the product requirements, and the environmental constraints (see Fig. 1 (c) for validation of the product in the environment).

The rest of the paper is structured as follows: Sect. 2 provides the solution architecture of ProConAR in the form of a component diagram. Sect. [START_REF] Chatzopoulos | Mobile Augmented Reality Survey: From Where We Are to Where We Go[END_REF] shows technical details in the modeling of the product, the transfer of the assets from the Product Modeler to the Product Configurator, and the configuration of the product. Finally, in Sect. 4 we conclude our paper and point out future research directions.

Solution Architecture

In this section, we provide a solution architecture for ProConAR based on a component-based architecture as shown in Fig. 2. The solution is divided into the Design Stage, which consists of the Product Modeler and the Asset Modeler, and the Runtime Stage, which consists of the Product Configurator.

In the Design Stage, the configuration parts of the products are modeled through a Feature Model Editor together with the attributes for additional product information as presented in [START_REF] Gottschalk | Model-based Product Configuration in Augmented Reality Applications[END_REF]. Moreover, the assets for the configuration parts are made available with the Asset Model Exporter which is an external graphic modeling tool. Both, the Feature Model and Assets, are connected so that each feature can consist of an asset with a specific type (e.g. product, part of product, texture) and positioning (e.g. left slot, right slot, upper slot) to other features. Next, both are serialized through the Asset Binder to create a single Asset Feature Bundle which can be transferred to the Product Configurator. By choosing the Asset Feature Bundle as a loose coupling between both stages, the tools of each stage can be exchanged independently.

In the Runtime Stage, the Asset Manager is used to deserialize the Asset Feature Bundle into components. Here, we use the Feature Model to store all possible configurations, the Configuration Model to store the actual product configuration, the User Model to store the user needs and the Environment Model to store the environmental constraints. While the Assets can be directly used in the Configurator Manager, the Features need to be analyzed by the Feature Modeler Interpreter to derive the Feature Model. Moreover, the Configuration Manager, whose activities are based on the MAPE-K architecture and explained in [START_REF] Gottschalk | Model-based Product Configuration in Augmented Reality Applications[END_REF], receives the requirements of the user, the configuration, and the environment during the runtime. While the User Input of the UI Interface can be directly restricted to use the Configuration Model and User Model, the mechanism of Unity by using an Importer Script for the feature model, stored as JSON file, and the assets in the form of 3D meshes, stored as FBX files, and textures, stored as PNG files.

In the following, we are focusing on the steps of product modeling, the transfer of product configurations, and the product configuration. While this demonstration paper just roughly goes through the whole process of ProConAR, the detailed description of the installation and usage is provided within the repositories.

Step 1: Product Modeling

In the first step of our process, we have to define the products which we want to configure in the environment. For that we need to design the parts of our product in the Asset Manager and model the possible product configurations within the Product Modeler as shown in Fig. 3.

Asset Modeler

Product Modeler At first, we use an Asset Manager like Blender 9 or AutoDesk 10 to separate the different product parts from each other and create a single model for each part (e.g. fridge of a kitchen). Moreover, we separate the 3D mesh as FBX from the texture as PNG to allow a reusing of the texture for different meshes. After that, we store both within our Unity product in the Assets/Inputs folder so that we can reference them with the corresponding Importer Script.

Next, we have to model the possible product configurations within our Product Configurator. For that, we create a new feature model that adds multiple features that can have the type of wrapper, product-part, or textures. While wrapper features are just used for structuring the model itself, the product-part 9 Homepage of Blender: https://www.blender.org 10 Homepage of AutoDesk: https://www.autodesk.com and texture features contain corresponding 3D mesh files or images. These files are referenced in the assets folder of the importer by choosing the same naming. For each feature of type product-part, we can set a brand, a price, and slots for left, right, and upper positioning. After adding all features to the feature model, we can export the model as a JSON file and store it within the Assets/Inputs folder so that it is useable within the Importer Script.

Step 2: Transfer of the Product Configuration

In the second step, we need to transfer the product configuration from the Product Modeler to the Product Configurator. The transfer process is shown in Fig. 4. In the last step, we have already shown the design and modeling of the product and its transfer to the Importer Script. This script is written for the Unity Game Engine and imports all files from the Assets/Inputs folder. After validating the files (e.g. looking if all references of the feature model are inside the folder), it bundles the JSON with the 3D meshes and textures by using the asset bundling11 which is a build-in technique of Unity to load non-code assets at runtime into an application. After the bundling process, the DB file is stored in the Assets/Bundles. This bundle can be uploaded to a web server, for which we used GitHub, and downloaded by the Product Configurator. The Product Configurator downloads the asset bundle and interprets it which is described in the next step.

Feature Model

3D Models Textures

WebApp

Product Modeler

Android Product Configurator

Unity

Importer Script

GitHub

Step 3: Configuration of Products

In the last step, we need to configure the product out of the asset file as shown in Fig. 5. In the beginning, the app loads the file from the webserver and splits into the JSON feature model and the assets in the form of 3D meshes and textures.

After that, the app deserializes the JSON with an external library12 and builds an internal structure of the model to validate violations at runtime. After that, the customer can scan the environment to detect horizontal surfaces on which the product parts will appear standing and vertical surfaces that will provide spatial boundaries of the environment (see Fig. 5 (a)). After that, the customer can tap on the environment and select the first product part he wants to add to the environment (see Fig. 5 (b)). From this point, the customer clicks on the plus buttons next to the placed product parts to extend the configuration and is informed with visual hints if an error (e.g. two stoves in one kitchen) duo to the placement has occurred (see Fig. 5 (c)). Moreover, the configurator validates at runtime if constraints of the underlying feature model or the user needs are violated. For that, we have developed our own feature model interpreter script. At the end of the configuration (see Fig. 5 (d)), the configurator check also violations which can not be displayed during the runtime.

Conclusion and Outlook

Mobile shopping apps have been used Augmented Reality (AR) in the last years to place their products in the environment of the customer. While this is possible with atomic 3D objects, there is is still a lack in the runtime configuration of 3D object compositions based on user needs and environmental constraints. For this, we previously developed an approach for model-based AR-assisted product configuration based on the concept of Dynamic Software Product Lines. Based on this approach, we have shown the tool-support ProConAR with a Product Modeler and a Product Configurator. For both, we have shown a solution architecture together with the most important parts of the technical implementation.

Research Directions

We want to extend our approach and our tool support so that the underlying concepts can be used domain-independent across VR and AR apps. For that, we improve the modeling of the user and the environment by separating them into distinct models as represented in our research paper. Based on that, we want to transfer our model-based approach into a model-driven framework that can generate code for different platforms. These platforms can be all kinds of mixed reality and allow a fluid usage of product configurations between the platforms. Furthermore, we want to extend the framework in various ways: First, we plan to configure parts of products in each other (e.g. changing the grips of a kitchen). This should support the modeling of product configuration with finer granularity and extend the possible use cases. Second, we want to add an intelligent obstacle detection based on machine learning (e.g. detecting sockets, water connections). This should improve the constraints space of the possible product configurations with the environment. Third, we add support for collaborative product configuration (e.g. changings of the product configuration by the customer and the kitchen salesman). This will allow a multi-user experience in the configuration of products by taking also a product validation based on different user feedback into account.

Fig. 1 .

 1 Fig. 1. ProConAR consists of a Product Modeler used by the Developer and a Product Configurator used by the User

Fig. 3 .

 3 Fig. 3. Creating of products with the Asset Modeler and modeling them with the Product Modeler

Fig. 4 .

 4 Fig. 4. Transfer of the product configuration from the Product Modeler to the Product Configurator by using the Feature Asset Bundle

  Scanning the Environment (b) Adding of Product Configuration (c) Validation of the Environment (d) Validation of the Product Configuration

Fig. 5 .

 5 Fig. 5. Configuration of the product by scanning the environment, adding product parts and validating the product configuration

Announcement of Amazon: https://techcrunch.com/2020/08/25/amazon-rollsout-a-new-ar-shopping-feature-for-viewing-multiple-items-at-once/

Unity Engine: http://www.unity.com

Source Code of the Product Modeler: https://github.com/sebastiangtts/ feature-modeler

Source Code of the Product Configurator: https://github.com/sebastiangtts/ ar-product-configurator

Asset Bundling of Unity: https://docs.unity3d.com/Manual/AssetBundlesIntro. html

Unity-Library for JSON Interpretations: https://assetstore.unity.com/ packages/\-tools/input-management/json-net-for-unity-11347

This work was partially supported by the German Research Foundation (DFG) within the Collaborative Research Center "On-The-Fly Computing" (CRC 901, Project Number: 160364472SFB901) 3 IKEA Place at Apple's AppStore: https://apps.apple.com/us/app/ikea-place/ id1279244498 4 VEKA Configurator AR at Apple's AppStore: https://apps.apple.com/us/app/ vµ-configurator-ar/id1450934980

Technical Implementation

In this section, we present the technical implementation of the Product Modeler 7 and the Product Configurator 8 . Both tools are built on top of well-accepted development techniques to allow the reusing for other researchers. The Product Modeler is built on an Angular web app and can run directly in the web browser. Moreover, we use an existing feature modeler [START_REF] Gottschalk | Hypothesis-driven Adaptation of Business Models based on Product Line Engineering[END_REF] to cover all possible feature restrictions and dependencies. The Product Configurator is built on ARCore, the AR SDK of Android. Moreover, we use the Unity SDK that provides solutions for most interactions with the environment like detecting obstacles or modifying 3D mesh models. For the transfer between both tools, we use the assets bundling


Sebastian Gottschalk
email: sebastian.gottschalk@uni-paderborn.de

Enes Yigitbas
email: enes.yigitbas@uni-paderborn.de

Eugen Schmidt
email: eschmidt@mail.uni-paderborn.de

Gregor Engels
email: gregor.engels@uni-paderborn.de

Model-based Product Configuration in Augmented Reality Applications

Keywords: Product Configuration, Augmented Reality, Runtime Adaptation, Dynamic Software Product Lines

Augmented Reality (AR) has recently found high attention in mobile shopping apps such as in domains like furniture or decoration.

Here, the developers of the apps focus on the positioning of atomic 3D objects in the physical environment. With this focus, they neglect the configuration of multi-faceted 3D object composition according to the user needs and environmental constraints. To tackle these challenges, we present a model-based approach to support AR-assisted product configuration based on the concept of Dynamic Software Product Lines. Our approach splits products (e.g. table) into parts (eg. tabletop, table legs, funnier) with their 3D objects and additional information (e.g. name, price). The possible products, which can be configured out of these parts, are stored in a feature model. At runtime, this feature model can be used to configure 3D object compositions out of the product parts and adapt to user needs and environmental constraints. The benefits of this approach are demonstrated by a case study of configuring modular kitchens with the help of a prototypical mobile-based implementation.

Introduction

Mobile shopping has become a big trend in the last years, as it allows purchasing from anywhere and at any time [START_REF] Ko | Modeling consumer adoption of mobile shopping for fashion products in Korea[END_REF][START_REF] Yang | Determinants of US consumer mobile shopping services adoption: implications for designing mobile shopping services[END_REF]. Because of the growing performance of mobile devices, Augmented Reality (AR) is becoming one key focus of these shopping apps [START_REF] Chatzopoulos | Mobile Augmented Reality Survey: From Where We Are to Where We Go[END_REF]. AR allows users to virtually try-out physical objects in their environment before purchasing them. Examples of these apps are IKEA Place 3 to determine the appearance of furniture in certain rooms, the configuration of products like windows in the VEKA Configurator AR App 4 , or the placement of multiple products in an environment as recently announced by Amazon 5 . With these apps, the users have the possibility to get more information about the products, be more certain about buying what they want, and have a greater product choice and variety [START_REF] Dacko | Enabling smart retail settings via mobile augmented reality shopping apps[END_REF]. For this, most of the existing solutions focus on the positioning of atomic 3D objects in the physical environment. With this focus, they neglect the configuration of multi-faceted 3D object composition. This 3D object composition, in turn, can support a runtime adaptation of parts of the product according to the product requirements (e.g. exclusion of different product parts to each other), the user needs (e.g. maximum price), and the environmental constraints (e.g. detection of obstacles). To overcome these limitations, we present a model-based AR-assisted product configuration that uses the concept of Dynamic Software Product Lines (DSPLs) [START_REF] Capilla | An overview of Dynamic Software Product Line architectures and techniques: Observations from research and industry[END_REF] to provide a runtime adaptation of product parts to the product requirements, user needs, and environmental constraints. An overview of the approach, which consists of a Design Stage and a Runtime Stage, can be seen in Fig. 1. At the Design Stage, we use the concept of reusable assets [START_REF] Apel | Feature-Oriented Software Product Lines[END_REF] to split the definition of products and their possible configurations from their actual representation in 3D objects. For this, the Developer uses the Product Modeler to model valid product configurations. These configurations are based on the Product Information (e.g. name, brand, price) which are used to create a Feature Model where each feature (e.g. tabletop, table leg) can be linked to a specific Product 3D Object (e.g. tabletop model, table leg model). This so-called Asset Feature Bundle is then transmitted to the Product Configurator at the Runtime Stage.

At the Runtime Stage, we use the MAPE-K architecture [START_REF] Kephart | The vision of autonomic computing[END_REF] to monitor the user needs in the form of user inputs, the actual product configuration, and the environment. The gathered information is validated and used to adapt to the product configuration in the physical environment. For this, the User uses the Product Configurator by defining his own needs (e.g. favorite brand, maximum price). After that, she starts the configuring process by selecting features (e.g. table ) and placing them in the environment. During the placement, the need of the User (e.g. price too high), the configuration of the Product (e.g. missing fridge in the kitchen), and the constraints of the Environment (e.g. collisions with other objects) are checked dynamically in the background.

With this paper, we present a novel model-based approach, which provides a twofold contribution to the research of AR-assisted product configurations: First, the separation of product modeling from 3D object implementation ensures a fast and flexible extension the configurator for new products. Second, the runtime configuration provides validation of product requirements and adaptation to user needs and environmental constraints. We demonstrate both benefits with a case study of a kitchen configurator.

The rest of the paper is structured as follows: Sect. 2 is providing the background of DSPLs which is used in our approach. Sect. 3 defines the requirements of our solution for the three models of the product, the customer, and the environment. Based on these requirements, Sect. 4 explains a solution concept that architecture is presented in Sect. 5. Sect. 6 shows the benefits of our approach based on the case study of a kitchen configurator. Sect. 7 presents the related work of our approach. Finally, we conclude our paper in Sect. 8.

Background

In this section, we explain the background on which we built our approach. We divide this background into Software Product Lines (SPLs) and Dynamic Software Product Lines (DSPLs).

Software Product Lines

Software Product Lines (SPLs) can be defined as "a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way" [START_REF] Clements | Software product lines: Practices and patterns[END_REF]. Here, a feature refers to a prominent characteristic of a product which can be, depending on the stage of development, a requirement, a component in an architecture, or a piece of code [START_REF] Benavides | Automated reasoning on feature models[END_REF]. To use these SPLs, a structure and an engineering process are needed. The structure of SPLs can be represented using hierarchical feature modeling which is shown in Fig. 2. Features can be mandatory (i.e. Cabinet) or optional (i.e. Ventilation) for the model instances. Moreover, there can be Or (at least one sub-feature is selected), and Xor (exactly one sub-feature is selected) relationships between a parent and a child feature. To refine the model instance, cross-tree constraints for requiring (i.e. Stove requires Ventilation) and excluding (i.e. Refrigerator and Build-In Refrigerator ) dependencies can be made. A big issue in SPL development is to find the right granularity for the features [START_REF] Kästner | Granularity in software product lines[END_REF]. While the classical feature model does not allow modelings like the usage of multiple feature instances and additional attributes, some approaches extend the modeling approach for these cases [START_REF] Czarnecki | Staged configuration through specialization and multilevel configuration of feature models[END_REF]. For this, they add these additional modelings to the meta-model of the feature model. In our approach, we use feature models to structure the possible product configuration. For this, we need to extend the meta-model of feature models with meta-data in the form of attributes, positioning, and links to 3D objects.

The engineering process for SPLs is shown in Fig. 3. The process can be divided into the Domain Engineering, which consists of the analysis of the domain and the development of reusable artifacts, and the Application Engineering, which uses these artifacts for the development of a specific software product for a user group. Moreover, the Problem Space describes the user perspective on the requirements of the software product, while the Solution Space covers the developer's perspective on the design and implementation of the software product. Based on this classification the process consists of the following four steps [START_REF] Apel | Feature-Oriented Software Product Lines[END_REF] We use the division of Domain Engineering and Application Engineering to configure different possible products from the same model. Moreover, we use the division of problem and solution space to separate the information about the product from their 3D objects.

Dynamic Software Product Lines (DSPLs)

Dynamic Software Product Lines (DSPLs) extend "the concept of conventional SPLs by enabling software-variant generation at runtime. The reason for this extension is that typical SPL approaches do not focus on dynamic aspects: the adaptation of their products is known and occurs during product line development" [START_REF] Bencomo | A View of the Dynamic Software Product Line Landscape[END_REF]. For this, adaptive systems can be used [START_REF] Hallsteinsen | Using Product Line Techniques to Build Adaptive Systems[END_REF].

One of the most prominent reference architectures for autonomic computing is the MAPE-K loop [START_REF] Kephart | The vision of autonomic computing[END_REF]. MAPE-K, see Fig. 4 for the example of a socket detector consists of a Managed Element and an Adaptation Manager. The Managed Element provides sensors to get corresponding data (e.g. camera screens) and effectors to change its state (e.g. display view). The Adaptation Manager monitors the sensors (e.g. screen the environment) and analyzes the gathered data (e.g. detect sockets). Based on that, an adaptation is planned (e.g. plan an overlay for the sockets) and executed (e.g. add the overlay). Moreover, the knowledge base stores knowledge which can support the different steps (e.g. socket library for image recognition). Depending on the use case, the adaptation can be divided into different dimensions of goal (type, evolution), cause (type), and mechanism (autonomy, type) [START_REF] Bashari | Dynamic Software Product Line Engineering: A Reference Framework[END_REF].

Fig. 4. Process of MAPE-K illustrated with the example of detecting and marking sockets in the environment

We use the MAPE-K loop to continuously monitor the user needs and environmental constraints for the product configuration. Moreover, the goal of the configurator will be of type self-configuring with static evolution. Moreover, the cause types are the user, product, and environment together of the manual component mechanism [START_REF] Bashari | Dynamic Software Product Line Engineering: A Reference Framework[END_REF].

Model Requirements

In this section, we describe the requirements on the models for model-based AR product configuration. Based on the context in the Runtime Stage, we divide the requirements and the resulting models into the primary sources of Product, User, and Environment.

-Product: Within the product model, also referred to as the feature model in the next sections, we store all information that is needed to configure the products out of their parts. For this, we use the modeling structure of feature models [START_REF] Clements | Software product lines: Practices and patterns[END_REF] to model each product part as a single feature. Therefore, each valid product configuration of the feature model is also a valid AR product configuration. We extend this model with product information and asset management. In the product information, we store additional attributes (e.g. name, price, brand) for each product part. In asset management, we link the product parts to specific assets (e.g. 3D object, texture) and give additional information about their usage (e.g. single product, part of product, texture of product). -User: Within the user model, we store all information that is needed to validate the user needs during the runtime configuration. Depending on the actual use case of the product configuration this could the maximum price which the user wants to pay for the configured product or the handedness to choose the side of the hinges.

-Environment: Within the environment model, we store all information that is needed to place the products in the physical environment during the runtime. We divide this information into the types of placement and obstacle management. In the placement management, we store a mesh structure of the environment that can be used to detect free places for the products and validates collisions with other objects. In the obstacle management, we store the positions of obstacles which can be needed by parts of the product (e.g. sockets, water supply).

Moreover, there can be dependencies between the different models. Because the product configuration is our main artifact, we consider just the combinations of Product-User and Product-Environment.

-Product-User: Between the product and the user, we specify dependencies that can exist between both models. Mostly these connections are between some product information and a specific user need. and For example, the product can have the dependency of a minimum height for a user or the user can prefer a specific brand of the product part. -Product-Environment: Between the product and the environment, we specify dependencies that can exist between both entities. Mostly these connections are between some product information and a specific constraint in the environment. For example, the product can have the dependency of a water supply or the environment sets a maximum height for products because of a window as an obstacle.

Solution Concept

In this section, we present a solution concept for model-based AR product configuration. For this, we first show how the model requirements for the product, user, and environment can be modeled. Based on that we apply the engineering process of SPLs to AR product configuration. In the end, we show how the runtime adaptation to the models can be done.

Modeling of Product, User and Environment Requirements

To create a model-based AR product configuration, we have to first create models for all parts which our approach should effect. Based on the model requirements in the last section, we create models for the user (UserModel ), the environment (EnvironmentModel ), and the product (FeatureModel ). The model of the user consists of simple attributes of the user (e.g. gender, maximum budget) which are characterized as important by the developer. Moreover, the modeled environment consists of a mesh structure of the physical environment together with positions of obstacles (e.g. windows, sockets) that need to be defined by the developer. Last, the model of the product consists of configuration parts that can be resolved to complete products. Because of the special interest of the product model in AR product configuration, we focus on this model within this section. The model of the product, see Fig. 5, is based on the concept of feature models [START_REF] Clements | Software product lines: Practices and patterns[END_REF]. For this, we implement all classes of feature models and extend the model with an AR specific extension. Each feature consists of a reference to a 3D object and an attribute if multiple instances (e.g. multiple cabins in one kitchen) can be created. Moreover, we added the classes of FeatureClass, Fea-tureAttribute, and Positioning together with extending CrossTreeRelationship. The FeatureClass, which is interpreted within the product configurator. Here, a Component relates to a 3D object which can be placed within the environment (e.g. table ). A ComponentPart is a 3D object which can not be placed in the environment alone but as a part of another Component (e.g. tabletop). A ComponentTexture is a texture that can be applied to an existing Component (e.g. wood). A ComponentWrapper is used to structure the features within the product modeler and is not interpreted within the product configuration (e.g. electrical appliances). The FeatureAttribute is used to add product information (e.g. name, price) to each feature. The Positioning restricts the placed of features to each other. The new class of Uses within CrossTreeDependencies can be used to link Components to ComponentTextures.

Moreover, the model of the product has relationships with the models for the user and the environment. Here, the FeatureAttributes are linked to the requirements of the user (e.g. specific brand or color). Moreover, the Feature[s] are linked to obstacles that are needed to use the feature (e.g. sockets or water supply).

Engineering Process of Model-based Product Configuration

The engineering process for AR product configuration can be seen in Fig. 6. The process can be divided into the Design Stage, which analyses the different product configuration parts and links them to the 3D objects, and the Runtime Stage, which analyses the needs of the user and environmental constraints to derive valid product configurations. Moreover, the Problem Space describes the user's perspective on the requirements of the product, while the Solution Space covers the developer's perspective on the implementation in the physical environment. Based on this classification, the process consists of the following four steps of Product Analysis, Product Implementation, Configuration Analysis, and Configuration Derivation.

Runtime Adaptation according to the Modeled Requirements

The runtime adaptation for AR product configuration can be seen in Fig. 7. The concept can be divided into the Mobile Application, which provides the sensors and effectors for the adaptation and the Configuration Manager, which structures the adaptation process.

The Mobile Application provides sensors to provide information for the monitoring and effectors to change the display according to the execution. In the Context Service the inputs of the Camera and User Input Management The Configuration Manager provides the adaptation logic to change the product configuration according to the requirements of the product, the user, and the environment. For this, the Monitor measures changes in the models of the configuration, the user, and the environment. While the user and configuration requirements can be directly used in the models, the environments need to be interpreted to create a mesh structure and detect the obstacles which should be considered. In Evaluate Conditions these changes are validated if the user needs, valid product configurations, or environmental constraints are violated. While the user and the configuration can be validated with the FeatureModel, the for the environment the positions in the configuration need to be compared to the structure in the EnvironmentModel. If all conditions are fulfilled, the Execute process the instructions to change the screen of the application are given. During the full configuration process, Knowledge is stored which can be used in the different steps. This Knowledge consists of the FeatureModel, which holds all Product Modeler and the Product Configurator. Second, we discuss the current limitations of the implementation.

Instantiation

To show the benefits of our approach, we implemented a prototype of the Product Modeler and the Product Configurator. While the Product Modeler is based on a web-based implementation of a feature modeler in [START_REF] Gottschalk | Hypothesis-driven Adaptation of Business Models based on Product Line Engineering[END_REF], the Product Configurator is based on the Unity framework6 . Although this section gives an overview of the implementation, our demonstration paper shows more technical details [START_REF] Gottschalk | ProConAR: A Tool Support for Model-based AR Product Configuration[END_REF]. The Product Modeler7 , see Fig. 9 for screenshots, can be used to create the FeatureModel of the product that should be configurated. For this, all dependencies and constraints of feature models [START_REF] Apel | Feature-Oriented Software Product Lines[END_REF] can be used within the configuration by using the feature modeling tool in [START_REF] Gottschalk | Hypothesis-driven Adaptation of Business Models based on Product Line Engineering[END_REF]. We extend this tool by adding meta-data for each feature (i.e. product part). In this meta-data, each feature can be linked to a 3D object, which is placed in a specific folder of Unity. Moreover, we add a price and a brand as feature attributes, which can be chosen to configure the product according to the customer needs. These feature attributes are also specified in our meta-model in Fig. 5. To place the product parts (e.g. cabin, stove) to each other, we allow the selection of valid other parts of the left, right, and upper slot of each product part. This, in turn, allows the flexible configuration of the kitchen in front of a single wall. After the full feature model is created, we need to bind the features to the corresponding 3D objects. For this, we export the whole model as a JSON file. After this, we use a developed importer script in Unity, which serialized the features and 3D objects in a single file. This file can be now uploaded to a web server. From this web server, the product configurator can download the Feature Asset Bundle and use it in the configuration process. With the Project Modeler it is possible to flexibly change the possible product configurations. These changes are automatically applied to all mobile clients in the form of Product Configurators. The Product Configurator8 , see Fig. 10 and 11 for screenshots, can be used to configure a product out of the feature model. For this, the user opens the app, which downloads the Feature Assets Bundle from the webserver. After that, the user scans the environment to detect where the different parts of the kitchen can be placed. This information is stored inside the EnvironmentModel of the approach. Initially, she clicks on the screen to place the first part of the kitchen (i.e. see mesh structure on the floor). Iteratively, she now can click on an existing part of the kitchen, which makes "+"-buttons appear to extend the configuration. By clicking on one of these buttons, a list of all valid product parts is shown for selection.

During the full configuration process, the app checks if product requirements, user needs, or environmental constraints are violated. As an example, the first screen of Fig. 11 shows a cabin which can not be placed because of an object collision in the environment. After the full configuration has been done, the user can go to the check out process. Here, conformance errors, which can not be Fig. 11. Product Configurator : Validation of the product configuration during the selection process and after the configuration is finished modeled graphically during the configuration, are validated. As an example, the second screen of Fig. 11 shows the error message that the sink requires an empty frame. Moreover, also requirements of the UserModel like the maximum price are applied here. With the Product Configurator it is possible to detect violated requirements of the product, the user, and the environment at runtime. Based on that, the violation can be resolved or conformance errors can be shown.

Discussion

With the implementation of the modular kitchen, we show the two main benefits of our approach. On the one hand, the separation of the modeling of products from their actual configuration, and, on the other hand, the adaptation of the configuration to the requirements of the product, the user and the environment. Nevertheless, the current implementation is just a prototype and has limitations according to the Model Management and Platform Compatibility.

For the Model Management, we develop a feature model where the configuration parts can be enriched with meta-data like product attributes and 3D objects. In the current prototype, we have not considered the user and the environment as explicit models. While for the user we fixed the data of the user at the beginning of the implementation, the EnvironmentModel is based on an internal model of AR Core9 . In the future, we want to improve the prototype so that all requirements of the product, the user, and the environment are based on distinct models and not internal models like used by AR Core. Moreover, it should be possible that changes in the models (e.g. adding new feature attributes) are also covered directly in the prototype and not need to be fixed at the beginning of the implementation.

For the Platform Compatibility, we develop the prototype based on the technologies of Android and Unity. While the adaptation logic is written for Angular's AR Core, the Asset Binding is built on a script in Unity. Here, the Asset Feature Bundle has also the limitation that the whole information is transmitted at the starting of the application which limits the scalability of the approach. Therefore, the current prototype has limitations in using other technology platforms. In the future, we want to modify the prototype so that different technology platforms can be used and interexchange. Moreover, we want to modify the Asset Feature Bundle so that assets are only transferred when they are needed at the runtime.

Related Work

In this section, we provide an overview of the related work of our approach. For this, we analyze relevant approaches considering the topics of Product Configuration, Augmented Reality, and Runtime Adaptation that are described and compared to our solution approach.

Product Configuration

Product configuration is a rich topic with a substantial amount of research behind it which is applied in various domains such as industrial applications, management, marketing, and computer science. In the following, we briefly focus on and describe product configuration approaches that rely on a model-based solution approach.

For model-based approaches, we use feature models to model the product features and business models of mobile applications [START_REF] Gottschalk | Intertwined Development of Business Model and Product Functions for Mobile Applications: A Twin Peak Feature Modeling Approach[END_REF]. We call our concept a Business Model Decision Line (BMDL). In this paper, we use our BMDL Feature Modeler [START_REF] Gottschalk | Hypothesis-driven Adaptation of Business Models based on Product Line Engineering[END_REF] as the basis of our AR Product Modeler. Furthermore, Bashari et al. present a reference framework for dynamic software product line engineering which also serves as an inspiration for extending our feature modeler towards a DSPL approach that supports dynamic product configuration in AR [START_REF] Bashari | Dynamic Software Product Line Engineering: A Reference Framework[END_REF]. For model-based approaches, one of the first things a product configuration system needs to do is validate the feature model itself. A lot of research has been conducted in the area of feature model validation [START_REF] Benavides | Automated analysis of feature models 20 years later: A literature review[END_REF], partially because feature models can be prone to errors (e.g. adding a new constraint might accidentally create a constraint deadlock or remove some valid products from the product line). One of the more advanced solutions in this area was proposed by Trinidad et al. who have developed a 3-step framework for automated feature model validation [START_REF] Trinidad | Automated error analysis for the agilization of feature modeling[END_REF]. Their approach can detect some situations where there is a discrepancy between what the feature model describes and the actual product line it is supposed to model (e.g. feature models that produce no valid products or features that cannot appear in any valid product).

Another related area of research is optimization of product configurations (i.e. using algorithms to scan through all valid product configurations to find one that is optimal concerning some criteria). Depending on the criteria used, optimization can bring benefits to both the manufacturer and the customer. For example, an algorithm can be used by the manufacturer to find configurations that are easiest to produce or would be fastest to deliver, while the customer can use an optimization system to find a configuration with the lowest price. For example, Yeh et al. focus specifically on using advanced algorithms to optimize the price of configurable products [START_REF] Yeh | Parallel genetic algorithms for product configuration management on pc cluster systems[END_REF]. However, using such advanced techniques is only needed in cases where the feature model is complex enough to induce a very large number of possible configurations that can only be analyzed by a highly efficient algorithm.

Augmented Reality

With the shift from mass manufacturing to mass customization, the interest in product configuration increases as more and more companies offer their customers the ability to tailor a product to their needs before manufacturing even begins. This customization step often relies on computer-based representations of the product, its variants, and sometimes the used environment. For this reason, technologies that offer computer-generated imagery, such as VR and especially AR, are being researched in this context. In the following, we discuss some of the relevant AR-based approaches for product configuration.

An early study by Gehring et al. explored the usage of a mobile AR configurator app as a means of on-the-fly customization at the point of sale [START_REF] Gehring | Mobile product customization[END_REF]. The app allowed the user to customize the color of a soap dispenser on their phone, either by picking a color directly or by calculating the most fitting color from the environment. Thus, the app exhibited a form of dynamic adaptivity to both user-based and environmental constraints. However, the focus of the study was not on using adaptive techniques for product configuration. Rather, the app was envisioned to be used in concert with a smart factory.

Moreover, the usage of AR in product customization is a related topic of this work. As a result, an additional related area of study in AR-based product configuration has emerged: virtual try-on. This discipline aims to use AR's capabilities to allow the customer to not only customize a wearable product but also to immediately try their design on themselves. This research field is in certain aspects similar to the idea of mobile product configuration, as both aim to enable the end-user to test and preview a product with the help of computer-generated imagery. Here, Eisert et al. developed an AR-based try-on system that they called Virtual Mirror [START_REF] Eisert | 3-d tracking of shoes for virtual mirror applications[END_REF]. This system was essentially an AR-based mirror that recorded the customer's feet and displayed the video output in real-time, horizontally flipped. The customer could then use a computer to select a shoe model and make various cosmetic changes such as color, materials, and embroidery.

Overall, approaches that focus purely on the try-on aspect, as exemplified by Eisert et al., can be considered as fulfilling the user-product requirement type due to their often minimal configuration capabilities [START_REF] Eisert | 3-d tracking of shoes for virtual mirror applications[END_REF]. Furthermore, they can be said to display a degree of adaptivity to the user, because they adapt their video output to the user's physical properties and movement. However, despite being AR-based, virtual try-on systems do not take their environment into account, as their AR capabilities are aimed at augmenting the image of the user, rather than the environment.

Runtime Adaptation

For enabling runtime adaptation in software systems, the research area of selfadaptive software systems [START_REF] Salehie | Self-adaptive software: Landscape and research challenges[END_REF] has emerged which provides extensive approaches to support the adaptation of software systems to changing context situations. As AR-based product configuration approaches have to monitor and adapt a viable product configuration regarding various aspects such as user and environmental requirements, we discuss relevant approaches based on runtime adaptation.

Here, especially adaptive UIs have been promoted as a solution for context variability due to their ability to automatically adapt to the context-of-use at runtime [START_REF] Akiki | Adaptive model-driven user interface development systems[END_REF]. RBUIS [START_REF] Akiki | Engineering adaptive model-driven user interfaces[END_REF] and Adapt-UI [START_REF] Yigitbas | Adapt-ui: an IDE supporting model-driven development of self-adaptive uis[END_REF] are two representative approaches that present methods, techniques, and tools for supporting the development of adaptive UIs. With regard to adaptive UIs, most existing approaches make use of a context model consisting of a user, platform, and environment model as described by us in [START_REF] Yigitbas | Integrated modeldriven development of self-adaptive user interfaces[END_REF]. Besides these approaches, there are also approaches that apply the idea of runtime adaptation for AR applications. While in [START_REF] Yigitbas | A context-aware virtual reality first aid training application[END_REF], we apply the idea of context-awareness for VR applications, in [START_REF] Krings | Development framework for context-aware augmented reality applications[END_REF] we propose a development framework for context-aware AR applications. Mostly, the before mentioned approaches are based on a runtime monitoring and adaptation loop which is characterized through the MAPE-K loop [START_REF] Kephart | The vision of autonomic computing[END_REF] which was also applied in this work. In contrast to the described approaches, our proposed solution in this paper combines the advantages of DSPLs with runtime adaptation to support model-based product configuration.

Moreover, some approaches use the concept of DSPLs to model the UI and the context of use (user, platform, environment) as feature models. Here, Gabillon et al. create a single feature model for the UI features and the contexts of use [START_REF] Gabillon | Designing an adaptive user interface according to software product line engineering[END_REF]. Moreover, they add logic rules to the UI features so that they can adapt to changes in the context and apply the concept of a case study of a desktop application. Sboui et al. model both as separate feature models and create a configuration model to link both feature models together [START_REF] Sboui | A ui-dspl approach for the development of context-adaptable user interfaces[END_REF]. They apply their approach to a case study of a mobile application. In contrast to the described approaches, our proposed solution in this paper combines the advantages of DSPLs with runtime adaptation to support model-based product configuration of AR products.

Conclusion

Augmented Reality (AR) has recently found high attention in mobile shopping apps such as in domains like furniture or decoration. While these apps focus on the displaying of atomic 3D objects, they neglect 3D object composition and their configuration according to the user and environmental requirements. To solve both issues, we develop a model-based AR-assisted product configuration approach based on the concept of Dynamic Software Product Lines. For this, we split the engineering process into a Design Stage and a Runtime Stage. While in the Design Stage we create a feature model where each product part is modeled as a feature can be linked to an asset, the Runtime Stage uses the feature model and assets to display a product configuration in the physical environment. Moreover, this product configuration can adapt to the requirements of the user and the environment. The benefits of this approach are demonstrated by a case study of configuring modular kitchens with the help of a prototypical mobile-based implementation.

Our future work is twofold and deals with the generalization of the introduced concepts: First, we want to improve the modeling of the user and the environment by separating them into distinct models. Second, we want to create a crossplatform solution for supporting model-based AR/VR product configuration. By combining both, our goal is to develop a model-driven framework for developing AR/VR product configurators.

Fig. 1 .

 1 Fig. 1. Overview of the approach, where the Developer can model products at the Design Stage which can be configured by the User at the Runtime Stage

Fig. 3 .

 3 Fig. 3. Engineering process of SPLs which splits the generic Domain Engineering from the specific Application Engineering

Fig. 5 .

 5 Fig. 5. Meta-Model of the FeatureModel with extensions for modeling the AR products and dependencies to the UserModel and the EnvironmentModel

-

  A. Product Analysis identifies the domain scope of the different products and develops a feature model out of them. In A.1. Product Scoping the possible products are identified, additional product information is collected and the different configuration parts are split up into reusable parts. In A.2. Product Scoping the possible product configurations are modeled as features of the feature model. Therefore, for each features the multiplicity, the feature class, the product information in the form of feature attributes and the possible positions are set (i.e. see section 4.1).-B. Product Implementation develops the reusable assets which are used in the configuration derivation. In B.1. Assets Modeling the 3D objects and textures are created and split up into the parts of possible product configurations of the feature model. In the B.

Fig. 6 .

 6 Fig. 6. Engineering process for an AR product configurator with a Design Stage to model the products and a Runtime Stage to configure the products

  are detected to update the Configuration Model, the User Model, and the Environment Model. These models can be monitored by the Context Monitoring. The effectors of the Display Service are triggered by the Display Adaptation. They are used to change the screen of the mobile application by Display [the] AR Configuration and Display [the] Validation Issues.

Fig. 9 .

 9 Fig. 9. Product Modeler : Creation of product parts and adding of meta-data inlcuding the 3D object for later AR representation

Fig. 10 .

 10 Fig. 10. Product Configurator : Configuring the product by selecting an existing product part and clicking on the "+"-button

  1. Assets Binding the created assets are linked to the feature model. So that the assets are inserted as a reference in the feature model (i.e. see section 4.1). -C. Configuration Analysis extracts the requirements of a single user based on user needs and possible product configurations. In C.1. User Input Validation the user input is validated against the current product configuration to detect violations that need to be resolved. In C.2. Product Selection Validation the product configuration is validated according to the underlying feature model. -D. Configuration Derivation is the matching of selected features and assets to place the product configuration in the environment. In the D.1. Environment Validation the configuration is validated against possible obstacles in the environment. Moreover, the Feature Selection Change of the C. Configuration Analysis can be changed. In the D.2. Product Configuration Derivation, the current Product Configuration is selected and can be exported for further usage (e.g. save for later, buy the product).

		Problem Space		Solution Space
	Design	A. Product Analysis		B. Product Implementation
	Stage			Mapping
	Product Knowledge	A.1. Product Scoping	A.2 Product Modeling		B.1. Asset Modeling	B.2. Asset Binding
	Product Configurator Requirements		Features			Asset Implementations
	Runtime Stage Customer Needs	C. Configuration Analysis C.2. Product C.1. User Input Selection Validation Validation	Feature Selection	D. Configuration Derivation D.1. D.2. Product Environment Configuration Validation Derivation
			Feature Selection Change
	Environment Contraints					Product Configuration

Announcement of Amazon: https://techcrunch.com/2020/08/25/amazon-rollsout-a-new-ar-shopping-feature-for-viewing-multiple-items-at-once/

Unity Framework: https://unity.com/

Source Code of the Product Modeler: https://github.com/sebastiangtts/ feature-modeler

Source Code of the Product Configurator: https://github.com/sebastiangtts/ ar-product-configurator

AR Core: https://developers.google.com/ar

This work was partially supported by the German Research Foundation (DFG) within the Collaborative Research Center "On-The-Fly Computing" (CRC 901, Project Number: 160364472SFB901) 3 IKEA Place App: https://apps.apple.com/us/app/ikea-place/id1279244498 4 VEKA Configurator AR App: https://apps.apple.com/us/app/vµconfigurator-ar/id1450934980

possible product configurations, and the Assets, which display these configurations in the form of 3D objects and textures. Moreover, the current state of the ConfigurationModel, the UserModel and the EnvironmentModel are stored from the Monitor to use them in the other steps.

Solution Architecture

In this section, we provide a solution architecture for model-based AR product configuration. Based on our definition of the engineering process in section 4.2, we provide a component-based architecture for the implementation. This architecture is divided into the Design Stage, which consists of the Product Modeler and the Asset Modeler, and the Runtime Stage, which consists of the Product Configurator.

In the Design Stage, the configuration parts of the products are modeled through a Feature Model Editor together with the attributes for additional product information. Moreover, the assets for the configuration parts are made available with the Asset Model Exporter. Both, the Feature Model and Assets, are connected so that each feature can consist of an asset with the type (e.g. product, part of product, texture Configurator. By choosing the Asset Feature Bundle as a loose coupling between both stages, the tools of each stage can be exchanged independently.

In the Runtime Stage, the Asset Manager is used to deserialize the Asset Feature Bundle into components. While the Assets can be directly used in the Configurator Manager, the Features need to be analyzed by the Feature Modeler Interpreter to derive the Feature Model. Moreover, the Configuration Manager, whose activities are explained in section 4.3, received the requirements of the user, the configuration, and the environment during the runtime. While the User Input of the UI Interface can be directly restricted to use the Configuration Model and User Model , the Environment needs to be analyzed by the Environment Interpreter to derive the Environment Model. This is done by analyzing the images of the camera. With all these requirements the Configuration Manager can control the UI Interface and provide the configuration to the Output.

Case Study

In this section, we show how the principles of the solution concept can be applied in a concrete usage scenario. For this, we present a use case in which a manufacturer of modular kitchens uses our approach to provide to their customers the ability to order customized kitchens. First, we show our implementation of the


Jan Van Den Bergh

Florian Heller

Wearable Touchscreens to Integrate Augmented Reality and Tablets for Work Instructions?

Keywords: Work Instructions, Augmented Reality, Tablet, User-Centered Design, Assembly, End-User Feedback

Manual assembly in high variety -low volume production is challenging for the operators as there might be small changes from one product to the next. The operators receive adapted instruction sets to be aware of these small differences, currently either printed or presented on a computer terminal. Augmented Reality (AR) is considered the future of work instruction display as it promises hands-free interaction, but interacting with AR interfaces can be problematic in production environments.

In this paper, we report on an exploratory study where we compare tablet interaction with three different ways of touch interaction for AR glasses: on the table, on the glasses, and the lower arm. While keeping the instructions the same in all conditions, we measured how placement affects the overall workload while performing an assembly task. Results indicate that combining additional wearable touch input with AR glasses is at least a promising direction for future research and, at best, an enabler for more widespread AR usage in manual assembly.

Introduction

Despite all automation achieved in manufacturing, a large part of the work is still performed by manual assembly operators, called just operators from now on. Especially in high variety, low volume production, they perform the majority of the work. In many cases, different models are assembled on the same line in mixed-model assembly systems (MMAS). This poses challenges for operators regarding cognitive load as these models might only differ in small details. Operators, thus, increasingly get (digital) work instructions that guide them during the assembly. However, given the challenging environment that a production line is, how can the operators interact best with these instructions.

Wearables -computing devices worn at specific places on the body (e.g., arms, legs, belt) or head -have been described as promising to support operators in their tasks [START_REF] Kong | Industrial wearable system: the human-centric empowering technology in industry 4.0[END_REF][START_REF] Lukowicz | WearIT@Work: Toward Real-World Industrial Wearable Computing[END_REF][START_REF] Stanford | Wearable computing goes live in industry[END_REF], as, e.g., head-mounted displays can superimpose Both authors contributed equally.

information directly onto the assembly [START_REF]Fundamentals of Wearable Computers and Augmented Reality[END_REF]. The issue of presenting the work instructions at the right time and offering a feedback channel for the operators to flag potential defects in the instructions [START_REF] Haug | Work instruction quality in industrial management[END_REF] makes an input capability for the AR interface a necessity. Body-based touch input can be given under various circumstances, such as standing, sitting, or kneeling [START_REF] Thomas | Where Does the Mouse Go? An Investigation into the Placement of a Body-Attached TouchPad Mouse for Wearable Computers[END_REF]. This paper discusses an exploratory study on how operators can interact with work instructions on the shop floor. As the work environment is likely to interfere with voice input and gestures, we focus on touch interaction, which is already supported by several such devices. We ran an experiment to evaluate the impact of the touch input location on the perceived workload while performing a primary assembly task. We presented the instructions on a tablet and an AR headset, and tested touch input on the tablet, the headset, or the lower arm.

Related Work

Augmented reality is frequently realized through HMDs. Koelle et al. investigated user attitudes towards these glasses and recommended being task-focused and using a least capabilities approach [START_REF] Koelle | Don't look at me that way!: Understanding user attitudes towards data glasses usage[END_REF]. In the case of operators, this implies that if they are not expected to contribute pictures or videos, HMDs without a camera may be recommended. People have concerns about being recorded and may not even notice indicator LEDs when present, as shown in earlier research with laptops [START_REF] Portnoff | Somebody's watching me?: Assessing the effectiveness of webcam indicator lights[END_REF].

While gesture interaction with an HMD without a camera is limited, such interaction may not be desirable anyway [START_REF] Werrlich | Design recommendations for hmd-based assembly training tasks[END_REF]. Also, other research found that hand gestures that can be detected by such a camera may not be desirable; e.g., Hsieh et al. [START_REF] Hsi Eh | Designing a willing-to-use-in-public hand gestural interaction technique for smart glasses[END_REF] found in their experiment on a gesture system for use in public space (using gloves) that most people naturally performed the gestures roughly around the lower torso. Most participants considered the gestures appropriate for use in the workplace (and other public conditions).

Nonetheless, a recent survey of 12 augmented reality glasses [START_REF] Syberfeldt | Augmented reality smart glasses in the smart factory: Product evaluation guidelines and review of available products[END_REF] showed that all but one have a camera built-in, although it can only take pictures on some devices. Control options mostly include a limited set of buttons and/or a touchpad on the glasses or a wired handheld device. Many reviewed AR glasses also offer voice or gesture control. Taking these studies together may imply that using other input than that offered by the devices may need to be explored. All devices should have this possibility as all glasses in the survey by Syberfeldt et al. [START_REF] Syberfeldt | Augmented reality smart glasses in the smart factory: Product evaluation guidelines and review of available products[END_REF] provide wireless connectivity options, which could be used to provide these alternative means of interaction.

Alternative Touch Interactions

To assess the potential of new multimodal interaction techniques on assembly lines, we performed an experiment where we compared various placements of visual output and touch input and their impact on the perceived workload. Visualizing assembly instructions on a screen, for example, is robust and straightforward, yet cumbersome for the worker if the assembly area is large and she has to carry the tablet with her, for example. Moving the visualization to an AR headset is promising as it allows the worker to look at the instructions at all times if needed. The instructions can also be laid over the real-world components to provide an even more seamless experience [START_REF]Fundamentals of Wearable Computers and Augmented Reality[END_REF]. Our goal is to provide a robust touch input surface with basic control of the worker's interface.

We evaluated how the visualization of the instructions (Fig. 1) and the placement of the control interface affects the perceived workload while assembling components. We wanted to investigate the effect of instruction placement (AR or tablet) and control (tablet, forearm, or head) on workload and performance.

Interface Description

Fig. 1 shows the user interface, as shown on the tablet or the AR-headset. The instructions take up most of the screen with the feedback and navigation controls placed vertically on the right side. Feedback regarding the current instruction step can be given at different levels of granularity: to draw attention to a specific step in the instructions, the user can flag that step by pressing the topmost button. This button is intended for fast interaction if the problem is obvious, or the user is unable to provide more information due to time constraints, situational disability, or limited motivation. The light-bulb button leads to the submenu where the user can provide more detailed information, such as adding dedicated flags for unclear description or unclear image, or an audio recording or a picture. These options would enable motivated operators to provide richer input during assembly.

We implemented our interface in Unity and ran it on an HTC Nexus 9 as a tablet and a Microsoft Hololens as AR glasses. Our wearable device was a 2.8inch touchscreen (Robopeak RPUSBDISP) attached to a Raspberry PI Zero W running Raspbian, and the control interface was implemented using Python and PyQt. The control interface sent its commands to the visualization application using OSC messages.

Experimental Setup and procedure

Similar to the experiment by Funk et al. [START_REF] Funk | Teach Me How! Interactive Assembly Instructions Using Demonstration and In-Situ Projection[END_REF], participants had to assemble LEGO Models according to given instructions. To level the difficulty of the assembly, we used models from the LEGO Racers Tiny Turbos series (8642 (hereafter referred to as Red ), 8644 (Yellow ), 8657 (Blue), 8658 (Black )). The models had between nine and 11 instruction steps, with the number of parts to be added in a single step ranging from one to seven (M=3.1, SD=1.6) and the total number of pieces to assemble ranging from 27 (Black ) to 38 (Red ) (M=32, SD=4.5). All instruction sets contain a phase where the assembled model has to be flipped around to add pieces to the bottom side.

We defined four conditions, shown in Fig. 2. We used the tablet as a baseline condition (A). In the remaining cases, we visualized the instructions on the ARheadset and ran the control interface on a tablet (B), and a wearable device fixed to the glasses (C), or the participant's lower non-dominant arm (D).

Due to the limited display resolution and field-of-view (FOV) of the ARheadset, the instructions covered a large part of the visible area. Participants could, therefore, minimize the instructions to the upper left corner of the FOV using a dedicated button in the control interface (Fig. 2,2). We deliberately did not fix the instructions to a particular physical location in the AR conditions, as this would basically mimic a dedicated screen at that position, which we already covered by the tablet condition.

We added requests to press specific buttons on the control interface to some of the instruction steps to simulate workers giving feedback (Fig. 2,1). This allows the participant to decide whether to react to this request immediately or finish the assembly of the components first and provides the same amount of triggers to all participants.

The order of the conditions and the order of the models to be assembled was randomized using Latin squares. After each condition, we asked the participants to fill out a pen-and-paper NASA TLX workload assessment questionnaire. We used the raw scores, frequently referred to as RTLX method [3].

Results

We recruited 10 participants (all male, mean age 31.1 years, SD=6.4) from our lab. We removed one trial for the blue and one for the black model from quantitative analysis, as the task completion times deviated more than two standard deviations from the mean.

Workload The mean TLX scores are reported in Fig. 3. We ran a repeatedmeasures ANOVA on the mean TLX scores with condition and model as fixed effects. User was modeled as random effect and the TLX scores were normally distributed. Condition had a significant effect on the average workload (F 3,30.03 = 10.32, p < .0001). A post-hoc Tukey HSD-test showed that the workload in condition C (AR glasses and interaction on glasses) was significantly higher than in all other conditions (p < 0.039), but there was no significant difference between the other pairs. The type of model to be assembled did not have a significant effect on the workload (F 3,30.03 = 2.75, p = .0603)

We ran a repeated-measures ANOVA on the mean TLX subscale ratings with condition as fixed effect and user as random effect. Pairwise post-hoc comparisons were made using Tukey HSD tests. We can observe the difficulties with condition C on the individual subscales of the TLX questionnaire. Condition had a significant effect on Mental Demand (F 3,33 = 5.5984, p = .0032) and Frustration (F 3,33 = 6.3473, p = .0016). Only condition C was significantly more mentally demanding than conditions A and B (p < .0444) and frustrating (p = .0007) than condition A.

Task Completion Time Similar to the workload, just having the tablet alone (condition A) leads to the shortest task completion times (Fig. 3 right). We performed a repeated-measures ANOVA on the log-transformed task completion times with condition as fixed effect and user as random effect. Condition had a significant effect on task completion time (F 3,33.01 = 3.3, p = .0323). A post-hoc pairwise comparison with a Tukey HSD-test showed only the difference between conditions C and A to be statistically significant (p = .032).

Qualitative Feedback and Observations

After having completed all conditions, we asked the participants which interface they preferred, and seven chose tablet only without the AR glasses. They justified their choice with the wellknown look-and-feel and that they could place it at a fixed position of their choice, making it easy to reach. One participant mentioned that he would prefer this interface even more than the original paper instructions because he does not have to look for the step he was at when he shifted his attention to the construction. Three participants said they preferred the combination of AR-Headset and the arm-worn wearable because the instructions are always in the field-of-view One major issue with the AR-visualization was the Microsoft HoloLens that we used. When justifying their choice for the tablet visualization, participants complained about the weight and the visual quality. This includes the low physical resolution and color depth, but also the low contrast and the fact that it is obstructing the view in general. Six of the seven participants who opted for the tablet as preferred interface mentioned one of the AR-based combinations as second choice with high chances of moving to first choice if the headset was more comfortable.

From the seven participants that opted for the AR visualization as a second choice, five also discussed which control interface they would like to use along with it. Three mentioned the wearable on the arm as of their interest and a request to make it more polished as they did not like the hardware's prototype appearance.

Overall, participants had trouble with the red model's instructions in the AR conditions because it contains many dark pieces that are difficult to perceive with the HoloLens due to low contrast. Two participants had one non-critical piece left over after assembling the red model which they forgot in step five of the assembly. With this model, most of the participants had to return a few steps, but all were successful in recovering from that mistake.

We observed that during simple assembly steps, participants overlooked feedback instructions. Participants quickly grasped which pieces were to assemble and how, and then hit the next button while returning to the instructions. In a real-world setting, the trigger to give feedback will not be external as in this experiment, but intrinsic to the worker. However, it indicates that additional information needs to be prominent to grab attention from the primary task.

Discussion

In this experiment, we gained insights on how AR visualization and various control interfaces influence how people perceive work on an assembly task. The assembly task we had the users perform was constrained to a small space in front of the user, which allowed them to place the tablet right next to the components and leave it there. This may not always be possible in practice; the components might be located on shelves from which the user has to collect them before the assembly. In some cases, the workspace is much larger than a desk, requiring movement. There might not be free space to place the tablet next to the working area. These factors might tip the preference more towards an AR headset or wearable displays that are more mobile and leave the hands free (most of the time).Nevertheless, more light-weight headsets with a good visual quality are required for a tool that can be worn throughout an 8-hour shift.

The screen placement only had a minor impact on both workload and task completion time compared to the placement of the feedback control panel. Once we move from a very local assembly task to larger items, this input device needs to be carried around, making the wearable control panel much more interesting. This also opens possible workarounds for the problem of the heavy headset, such as a combined input and output on a small touchscreen attached to the lower arm. To visualize assembly instructions, the screen needs to be bigger than the control interface. Smartphones or small tablets are already used to visualize instructions, and a similar size may be appropriate. Some aspects of the experiment limit generalizability of the results. Given the exploratory character, we opted for a limited amount of participants. This means an experiment with a larger sample size is needed. This experiment is preferably also carried out with assembly operators and AR glasses, rather than an AR headset.

On the practical level, we would pay more attention to the limited contrast of AR devices as it might affect readability more than visible on a regular screen. Ideally, the devices in the different conditions also have a similar level of refinement. Possible changes in the presentation include presenting the instructions on the wearable controller and making it easy to put it on a table. This could create a hybrid between the tablet and AR glasses.

Conclusion

We studied where assembly instructions should be presented and how interaction with a minimal feedback interface affects workload. Most participants preferred the tablet solution, but they also stated that a more comfortable AR setup, especially with wearable touch-interaction, might be preferred.

Wearable, easily removable, touch controllers, possibly integrating display of instructions in combination with less bulky AR glasses, seem promising. The latter are already available [START_REF] Syberfeldt | Augmented reality smart glasses in the smart factory: Product evaluation guidelines and review of available products[END_REF]. Before definitive conclusions can be drawn on which interface technologies are best suited in such settings, more refined prototypes and evaluation with a large group of operators are still needed.

Fig. 1 .

 1 Fig. 1. Left: the assembly instructions are shown with a minimal navigation interface that allows to go to the next/previous step, give feedback on the current step, and get additional help. Middle: The feedback menu allows to record audio, to flag an unclear description or an unclear picture, and to activate the camera with zoom (right).

Fig. 2 .

 2 Fig. 2. Experiment conditions used in the experiment (A.-D.). We added explicit feedback instructions (1) and AR interface minimization button (2).

Fig. 3 .

 3 Fig.3. Left: Mean RTLX by condition Right: Mean task completion time in minutes by condition (Fig.2). Both graphs indicate mean, quartiles, and whiskers. The only significant difference was between condition C and the other conditions for both workload (RTLX) and completion time.

Acknowledgements The work has been funded by the FlandersMake project OperatorKnowledge (grant number: HBC.2017.0395). The committee of ethics of Ghent University has approved the study (approval 2018/0439).


Ivan Jovanovikj

Stefan Sauer

Gregor Engels

Augmented and Virtual Reality Object Repository for Rapid Prototyping

Keywords: augmented reality, virtual reality, object repository, rapid prototyping

Augmented Reality (AR) and Virtual Reality (VR) are technologies on the rise and as their market grows, speeding up the process of developing new ideas becomes even more important. In achieving rapid development, rapid prototyping plays a very important role. To support rapid prototyping, this demo paper presents an approach that relies on an object repository. It enables scanning, editing, storing and publishing of virtual objects that can be reused in different augmented and virtual reality applications. To show the benefits of the approach, an exemplary scenario is illustrated by prototyping an interior design application.

Introduction

Augmented Reality (AR) and Virtual Reality (VR) have received grown interest by customers and media. The worldwide spending is predicted to over 20 billion USD in 2020 1 and up to 192 billion USD in the following three years 2 . Hence, the rapid development of new applications becomes even more important. Therefore, software designers and engineers should be able to quickly build AR/VR prototypes. However, developing such prototypes is not an easy task as the existing tools and SDKs are often highly technical and as such require much up-front learning effort [START_REF] Gandy | Designer's augmented reality toolkit, ten years later: Implications for new media authoring tools[END_REF][START_REF] Nebeling | Protoar: Rapid physical-digital prototyping of mobile augmented reality applications[END_REF]. Additionally, they are platform-dependent, making platform evaluations more complex. To support the designers in reusing existing objects in multiple prototypes and to gain a quick look at how the application might look and work, a large collection of different kinds of objects is needed. A fast way of building such a collection is translating physical objects into virtual ones. To speed up the prototyping of AR and VR applications, we combine an object repository with prototyping mechanisms. Therefore, in this demo paper, we present AVROsitory which is an Augmented and Virtual Reality Object Repository. Our approach utilizes the strengths of both object repositories and prototyping tools and comprises three main components: server (with an object repository), mobile client, and a web client component, as shown in Fig. 1.

Publishing

Storing/Prototyping Scanning/Edi5ng

Scanner Object Repository Fig. 1: Overview of the solution idea

The Scanning/Editing component is a mobile client which enables the designers to add new objects or edit existing ones. The real-world physical objects can be scanned and transferred into digital objects via Photogrammetry [START_REF] Foster | Integrating 3D Modeling, Photogrammetry and Design[END_REF]. It is also possible to scan physical objects with a smartphone and add them to the repository as virtual objects. Photogrammetry offers a quick solution for creating mock-ups, that can be used for prototypes, but as final AR/VR objects too. The Storing/Prototyping component, i.e., server, is a an object repository which enables reusability by providing a tagging system to classify the objects, thus allowing a quick search on existing objects. Furthermore, objects can be also classified by the level of detail quality so that depending on the usage scenario an object with a suitable quality can be provided. While in some cases an object must be in great detail, the same might be used in the background somewhere else, requiring less detail and fewer resources. Business logic can be also added to objects with a specific task by using the logic templates. Finally, the Publishing component is a web client which provides a mechanism (Unity Scripted Importer) to Unity IDE. As a result, a reusable Unity object (prefab) is added to the project's assets in Unity and it can be used for developing AR/VR prototypes or AR/VR applications. Furthermore, it provides also an option for creating custom importers for other IDEs.

The rest of the paper is structured as follows: In Section 2, the architecture of the repository is presented. To show the feasibility of our approach, in Section 3, we present an application example where the developed repository was used in a prototyping scenario. In Section 4, we briefly discuss the related work and in the end, Section 5 concludes the work and gives an outlook on future work.

Solution Concept

In this section, we present the solution concept which builds upon an object repository for prototyping and development of AR/VR applications. As shown in Fig. 2, our solution consists of three main components. The central component is the server which contains the object repository.

Before explaining the object repository, we shortly discuss the different types of supported objects. An object is defined as a 2D or 3D graphical model with optional business logic. It can depict small single real-world items as well as entire rooms. The business logic is rather basic as it does not contain any applicationspecific context. For example, an object representing a button would contain only a basic on-click listener without specific behavior. By evaluating existing AR applications publicly available, we classified the objects based on the user interaction into non-interactive, click/touch-triggered, and range-triggered.

A non-interactive object entails no business logic, e.g., a decorative object. On the contrary, a click/touch-triggered entails a business logic, e.g., a UI button. Finally, a range-triggered object is triggered according to a user's distance, e.g, a descriptive text appears when a user is in a given range. The object repository includes business logic for these object classes in terms of logic templates only, that can be written by the repository users, extending the existing solutions to their needs. So, the object repository stores graphics, business logic, and metadata. The objects are accessible via a REST-API which means that data is stored as entities, which can be retrieved, updated, and deleted via HTTP requests. The entities consist of metadata and binaries. Whereas the metadata is stored in a database, the binary objects are stored in the server's file system. The binary objects include original graphics, as well as converted binaries in formats distributed via the REST-API. The second component of the object repository is a mobile application which is the main client for designers. A mobile application was chosen, because of the wide availability of smartphones today, which makes prototyping easier and quicker. As the object repository is based on a server that offers an API, other client applications that run on Microsoft's HoloLens 3 , or photogrammetry equipment, e.g., tablets, can be added later on. The mobile client supports multiple tasks. As the client application synchronizes with the server, the users can display and browse the repository everywhere. This enables Fig. 3: Scanning of a coffee mug: Defining photogrammetry key points marked in green (left and middle); The imported object in the repository (right).

the users to preview their objects in their current environment. Furthermore, multiple objects can be combined into one composition thus allowing more complex mock-ups. The mobile client also enables users to create new objects and upload them to the server. The user can use photogrammetry and chroma-keying to create 3D and 2D graphics respectively. The objects can then be extended with metadata, like tags and quality levels. The last component is a web client that has similar functions to the mobile client, except that it does not allow any kind of scanning. However, it allows existing graphics to be uploaded. Its main purpose is to enable exporting objects to an IDE.

Application Example

This section presents an exemplary prototyping scenario of an interior design application. The application should enable users to decorate an empty office desk, to make design choices. Therefore, the application enables them to augment their desk with decorations and items from a catalog in the AR application. To achieve this, we firstly scan physical objects from an existing, real office desk and add them to the repository. These objects are then previewed, provided with business logic, if necessary, and in the further development process, replaced by high-quality 3D models. At the beginning, we have two 3D models of office chairs and one of a keyboard. Then, the three objects are added, all of them in good quality without any logic (classes). We begin by determining which catalog items we want to provide with our application. For simplicity, we support a basic set of coffee mugs, keyboard, monitor, and chairs. As our repository already contains chairs and a keyboard, we want to add models of coffee mugs and a monitor. The coffee mugs are scanned with photogrammetry, as shown in Fig. 3. As AVROsitory does not include an integrated photogrammetry solution, we use Scann3D4 , a third-party tool. As we are in an early development phase, it is sufficient for our monitor model to be two-dimensional. Therefore, we are using the build-in object creation by taking a photo. We also provide metadata for the object, as shown in Fig. 4a, for further reuse. So far, we have all objects in our repository that we need. We can thus proceed to test how our application would look like, when the user previews the decoration. To do that, we create a composition of our repository objects (the screenshot on the right in Fig. 4b). To enable interaction with the objects, we firstly sketch a button prototype on a board and we add it to the repository by simply taking a picture, as shown in the screenshots in Fig. 5. After minor editing, the button can be used in the example application. To really test the added objects, the programmers create a prototypical application using the objects. Firstly, they import the objects into Unity using the AVROsitory web client and add some business logic to our buttons, as shown in Fig. 6.

As the development is going on, we design high-quality models for our furniture, and we add these models to our repository as high-quality versions of our prototyping objects. Also, more buttons are needed and for this purpose, we can reuse the existing button logic in the repository. As we now have higher quality objects, we can replace the mock-ups in the actual application, as shown in Fig. 7.

Related Work

AR and VR applications are widely spread in various application domains (e.g., health [START_REF] Yigitbas | A context-aware virtual reality first aid training application[END_REF], education [START_REF] Yigitbas | Experiencing and programming the ENIAC in VR[END_REF], etc.). As described in previous work (e.g., [START_REF] Yigitbas | On the development of contextaware augmented reality applications[END_REF], [START_REF] Krings | Development framework for context-aware augmented reality applications[END_REF]), the prototyping and development of AR and VR applications is a cumbersome and challenging task. Multiple research projects and commercial tools are trying to make AR/VR application prototyping and development easier, quicker and overall more accessible. One can roughly categorize them into two types: repositories and authoring tools. Unity Asset Store5 and Google Poly6 are both repositories, albeit they serve different purposes. Poly is a sharing platform for digital art, mostly 3D models and its sole purpose is to enable designers to browse through the catalog and look for useful models. Unity Asset Store, on the other hand, distributes assets, whereby assets refer to whole components of programs, including art, audio and business logic. Furthermore, Unity Asset Store is neither limited to AR/VR assets nor does it specifically target them. In contrast, Poly is focused on AR/VR development, although it is not necessarily limited to these applications. So, object and asset repositories enable user to share and re-use their content over multiple projects. Nonetheless, they come with some pitfalls. If a repository is too restrictive, Poly for example, then its use is limited as it only allows plain 3D objects with no logic. Unity Asset Store contains almost every type of component that can be made in Unity Engine, which is why it is timeconsuming to find an asset that fits your needs. Then, there are authoring tools, like DART [START_REF] Macintyre | Dart: The designer's augmented reality toolkit[END_REF] and ProtoAR [START_REF] Nebeling | Protoar: Rapid physical-digital prototyping of mobile augmented reality applications[END_REF], both of them targeting AR. They do not provide content in terms of 3D objects, but rather support the prototyping and/or development of applications. While DART is based on Macromedia Director and thus acts as a development environment, ProtoAR serves as a prototyping tool only. As it only simulates AR functionality, ProtoAR does not fully support development. DART assist prototyping with its ability to record sensor data and play them back outside of the targeted environment. In contrast to ProtoAR, DART stronger supports development. DART's approach, however, is inflexible, because it ties the development to a specific tool for the whole process, from design prototyping to actual implementation. This approach might have been useful in the early 2000s, because of limited AR framework, today, however, it is too restrictive. The approach of ProtoAR, a tool for prototyping only, seems more appropriate. Designers can use ProtoAR to create mock-ups for their UI design, while developers can work with a tool set of their own choice. While this is more flexible, it does mean the designers cannot extend their prototype gradually, as at one point they have to switch their tool set.

Therefore, our solution, combines both approaches. It is an object repository, providing AR / VR specific objects, that can be used in any development environment. These objects also contain templates for logic, thus assisting the developers when prototyping or developing. This can be combined with a tool set for designers to create digital mock-ups from paper and previewing their design on an actual AR / VR device. The repository can be accessed via REST API, therefore the supported IDEs and devices can be extended.

Conclusion and Future Work

In this demo paper, we presented our approach for object round-tripping which supports the rapid prototyping of AR/VR applications. Designers can add new objects to the repository by using existing graphics or by creating new ones by taking pictures or via photogrammetry. Objects can be classified to support their reusability. Metadata, graphics, and business logic are stored separately, enabling easier and independent editing. Additionally, this enables storing each of those units to be stored either as a database for metadata or the file system for binary objects. With the case study, we have shown an exemplary use of the object repository and how the designers can benefit from the different provided features. While the concept is promising, several topics can be addressed by subsequent research. As the object repository is designed with extensibility in mind, using a platform-independent REST-API for all central functionalities, the approach can be extended to a variety of client devices. For example, a dedicated photogrammetry scanner or AR devices, like Microsoft's HoloLens can be added. They may increase the object quality without increasing the workload significantly. Additionally, the repository can be extended by object recognition to enable automatic tagging of objects.

Fig. 2 :

 2 Fig. 2: Solution architecture: A central server application with a web interface and an Android client.

( a )Fig. 4 :

 a4 Fig. 4: Cropping, importing, and previewing of repository objects

Fig. 5 :

 5 Fig. 5: Taking a picture in front of a monochromatic background (left) and using chroma keying to get a transparent background (middle and right).

Fig. 6 :Fig. 7 :

 67 Fig. 6: Using the web client (left) to export our objects into Unity (right)

idc.com https://bit.ly/3cI02HT

statista.com https://bit.ly/3jgLX6x

www.microsoft.com/hololens

Scann3D by SmartMobileVision. Paid application. Not related to this work.

https://assetstore.unity.com/

https://poly.google.com/

Acknowledgements

We would like to thank our student Niklas Junge who helped implementing the presented approach.