Jalil Boukhobza
email: boukhobza@univ-brest.fr

Pierre Olivier
email: pierre.olivier@univ-brest.fr

Loic Plassart
email: loic.plassart@treflevenet.fr

Hamza Ouarnoughi
email: hamza.ouarnoughi@univ-brest.fr

Ladjel Bellatreche
email: bellatreche@ensma.fr

Embedded Databases on Flash Memories: Performance and Lifetime Issues, the case of SQLite

Keywords: Embedded databases, NAND Flash memory, Flash file systems, SQLite, JFFS, UBIFS, Performance, Lifetime. I

Databases are more and more used in embedded system applications and especially in consumer electronics. This comes from the need to structure user and/or system data to be more efficiently managed and accessed. SQLite is one of the most used database applications. This paper presents a micro benchmarking methodology and results for SQLite database requests on embedded flash specific file systems. Indeed, flash file systems behavior are very specific to flash memory intricacies and the objective of this study is to highlight the interactions between flash memory, flash file systems, and SQLite based applications.

INTRODUCTION

Embedded systems actors are nowadays experiencing the golden age of Non Volatile Memories (NVM). Indeed, according to Market Research [START_REF]Advanced Solid State Non-Volatile Memory Market to Grow 69% Annually through 2015[END_REF], the NVM market will have an annual growth of 69% up to 2015. This tendency concerns several NVM technologies such as Ferroelectric RAM (FeRAM), Phase Change RAM (PCRAM), Magneto-resistive RAM (MRAM). Flash memory is however the most mature and disseminated NVM as its use is boosted by the evergrowing demand of smartphones and tablets market. In fact, mobile memory (including both NOR and NAND flash, DRAM and embedded multimedia cards) market has experienced a growth of 14% in 2012 (as compared to 2011) [START_REF]NAND Chip Market Growth Propelled by Smartphones and Media Tablets[END_REF]. Moreover, NAND flash memory revenues have hit a new record as of $5.6 billion (a growth of 17%) in the fourth quarter (as compared to the third quarter of 2012). Flash memory has just celebrated its 25 th birthday (1988-2013) after its creation in Toshiba labs and it is already shipped with almost 8 times more gigabytes than DRAM (in 2011). It became the process technology leader for memory fabrication and miniaturization.

Flash memory success is due to many attractive features such as good I/O performance, energy efficiency, shock resistance, small size and weight. These advantages come with some limitations/constraints the designers must deal with in order to maximize both lifetime and I/O performance. NAND flash memory is an EEPROM (Electrically Erasable and Programmable Read only Memory) on which one can perform operations at two different granularities. The smallest data unit used for read and/or write (program) operations is the page, while the data unit used for the erase operation is the block, with a block being composed of a given number of pages.

NAND flash memory constraints can be summarized as follows: (1) Write/erase granularity asymmetry: writes are performed on pages while erase operations are executed on blocks. (2) Erase-before-write rule: one of the most important constraints as one cannot modify data in-place. A costly erase operation must be achieved before data can be modified in case one needs to update data on the same location. [START_REF] Boukhobza | Flashing in the Cloud: Shedding Some Light on NAND Flash Memory Storage Systems[END_REF] Limited number of Write/Erase (W/E) cycles: the average number is between 5000 and 10 5 depending on the used flash memory technology. After the maximum number of erase cycles is achieved, a given memory cell becomes unusable. Finally, (4) the I/O performance for read and write (and erase) operations is asymmetric.

Embedded and critical systems typically use flash memories as storage support. In addition, the implementation of databases is becoming increasingly popular for that kind of systems to replace simple files for data management and thus meet the growing use and complexity of information management.

The purpose of the encompassing project from which this study is a part is threefold: [START_REF]Advanced Solid State Non-Volatile Memory Market to Grow 69% Annually through 2015[END_REF] to study the impact of unitary database requests on embedded flash memory storage systems from a performance and lifetime points of view (for instance: how many read, write, and erase operations are performed).

(2) To model the behavior of all the software stack impacted by such requests for different flash file systems (achieving comparative studies). Finally, (3) to optimize embedded database systems according to the results obtained in (1) and (2). The system optimizations can be declined in, at least, three different solutions: (a) optimize the existing file systems to better manage database requests. (b) If the first solution is unfeasible, one might think of developing a "database aware" flash file system. Indeed, current flash file systems are primarily designed to support simple file data management. (c) Another solution one can think about is the design of optimization techniques (such as buffers) that can be implemented at different levels such as the database management system or at lower operating system layers. This paper focuses on the first part of the study that is the impact of database requests on I/O performance and lifetime of embedded flash memory based storage systems.

II. BACKGROUND ON FLASH MEMORIES

A. Some Basics on Flash Memory

Flash memories are based on floating gate transistors and can be divided mainly into two types according to the logical gate used as the basic component: (1) NOR flash memories support random data access, are more reliable, have a lower density, and a higher cost (as compared to NAND flash memory). NOR flash memory is mainly used for storing executed code as a replacement of DRAM (e.g. in mid to low range mobile phones). ( 2) NAND flash memories are block addressed, offer a high storage density for a lower cost and are used as secondary storage. This paper only concerns NAND flash memory.

Three different NAND flash memory technologies exist:

(1) Single Level Cell (SLC), (2) Multi Level Cell (MLC) and (3) Triple Level Cell (TLC). In SLC, one bit is stored in each memory cell, while in MLC, two bits can be stored, and 3 bits for TLC (highest number of voltage states). From a bit density and cost per bit points of view, TLC is better than MLC that outperforms SLC. From a performance and reliability points of view, SLC performs better than MLC that is better than TLC. From the application point of view, TLC is used for low end media players, mobile GPS, and more generally non critical data applications that do not require frequent data updates. MLC and SLC are used for more data intensive appliances such as SSDs, mobile phones, and memory cards.

B. Flash Memory Management

To cope with the above-mentioned NAND flash memory constraints, some specific management mechanisms are implemented. [START_REF]Advanced Solid State Non-Volatile Memory Market to Grow 69% Annually through 2015[END_REF] In order to avoid costly in-place updates (block erase and data write operations), a logical-to-physical address mapping mechanism is used, allowing to perform outof-place data modifications (update data in another location and invalidate the first copy). ( 2) As the number of write/erase (w/e) cycles is limited and because of spatial and temporal data locality, some specific data blocks containing "hot" data can wear out quickly. To avoid this problem, some wear leveling mechanisms are implemented all with the mapping system in order to evenly distribute the erase operations over the whole memory surface. (3) Performing many update operations results in many invalidated pages/blocks that must be erased in order to be used. A garbage collector is generally used to perform this task.

Mapping mechanism, wear leveling and garbage collection services can be implemented either in hardware or in software.

In hardware, a Flash Translation Layer (FTL) is implemented to perform the aforementioned services [START_REF] Boukhobza | Flashing in the Cloud: Shedding Some Light on NAND Flash Memory Storage Systems[END_REF]. This is the case of some devices such as USB sticks, compact flash, and SSDs. Bare flash chips that can be found in most embedded systems such as smartphones, tablets, personal navigation devices, and video camcorders are generally managed by the operating system through some specific Flash File Systems (FFS). In addition to flash specific management services, FFS should achieve all traditional file system tasks: file and directory hierarchy organization, user access rights, etc.

C. Dedicated Flash File Systems

In embedded systems equipped with raw flash chips, NAND flash storage is mainly managed using dedicated Flash File Systems (FFS). Such hardware platforms only embed a simple NAND controller used to perform basic flash operations. The flash management is achieved at the FFS level, which is a software layer included in the operating system. Then, FFS can be viewed as a purely software flash management method. The Linux OS implements today's most popular FFS: JFFS2 [START_REF] Woodhouse | JFFS: The journalling flash file system[END_REF], YAFFS2 [START_REF] Wookey | YAFFS2: a NAND Flash File System[END_REF] and UBIFS [START_REF] Schierl | Abstract Specification of the UBIFS File System for Flash Memory[END_REF].

Figure 1 illustrates the FFS layer location inside the Linux NAND storage software stack. User space application access files using system calls. System calls are received by the Virtual File System (VFS), which role is to abstract the specifications of all the actual file systems supported by Linux. Moreover, at the VFS level, Linux maintains several caches in RAM in order to speed up files access. In particular, the Linux page cache (formerly buffer cache) is dedicated to file data buffering. VFS maps system calls to FFS functions. The FFS determines the NAND operations to perform, according to its state and algorithms. To access the flash chip, the FFS uses a NAND driver called the Memory Technology Device (MTD) layer. It is a generic driver for all kinds of NAND chips supported with Linux.

The above-mentioned FFSs present common features in their implementation. Files are divided into data nodes. When files are updated, old nodes are invalidated and new nodes are created. Invalid nodes are recycled through garbage collection which is generally performed asynchronously through the execution of a background kernel thread. As FFS implements out-of-place updates, nodes can be at a variable location on the flash media. To keep track of the nodes locations, FFS use indexing mechanisms which related metadata are also stored on flash.

In this study we chose to perform our tests on two FFS: JFFS2 and UBIFS. JFFS2 is a very mature and widely spread FFS, used for more than a decade (mainlined since Linux 2.4.10 in 2001). UBIFS, for its part, is a relatively recent FFS with a growing usage. UBIFS was created to address several of JFFS2 design issues strongly impacting file system performance [START_REF] Bityutskiy | JFFS3 design issues[END_REF].

III. EMBEDDED DATABASES

Today, the database functionality is needed to provide support for storage and manipulation of data in embedded systems [START_REF] Kataria | Commercial Embedded Database Systems: Analysis and Selection[END_REF]. Many different database management systems (DBMS) can be used to better meet the requirements of current embedded systems. In most cases, the implementation of embedded databases must rely on limited resources. Embedded DBMS must also exhibit robustness towards sudden power cuts. Among the existing solutions, three should particularly be considered: Berkeley DB, Firebird and SQLite. Berkeley DB [START_REF] Olson | Berkeley DB[END_REF] is a database engine compatible with several operating systems. Since version 2.0, Berkeley DB is available under both a free OSI-certified license and a commercial license. It comes in the form of a C-written library providing an API. Connectors exist for many programming languages such as C (native interface), C++ and Java. A Berkeley DB database is only composed of records whose format is freely determined by the calling program. There is no table concept, and the database cannot be used with a data manipulation language such as SQL. Berkeley DB is considered simple to use and supports concurrent accesses by multiple users.

Firebird project [START_REF]Firebird developers[END_REF] originally started as the Borland InterBase database. Firebird is an open source SQL relational database management system that runs on MS Windows, Linux and various UNIX flavors. Firebird is written in C++. A native API is provided to connect applications to Firebird databases. A Firebird database is operated using SQL language.

SQLite [START_REF]SQLite developers[END_REF] is an open source C library providing a relational database engine that can be operated by the SQL query language. SQLite is a lightweight relational database engine that can be integrated directly into an application. It is supposed to improve data storage and management potentials of embedded applications [START_REF] Li | Remote Monitoring Systems Based on Embedded Database[END_REF] [START_REF] Yue | Research of Embedded Database SQLite Application in Intelligent Remote Monitoring System[END_REF]. SQLite stores each database in one file, and each file consists of a given number of SQLite pages (the default value for the page size is 1KB). SQLite is widely applied for data management of embedded environment, such as smartphones, industrial control, etc. [START_REF] Qinlong | Study and Application of SQLite Embedded Database System Based on Windows CE[END_REF]. Today, SQLite is probably the most widely used DBMS for embedded applications and systems.

For the sake of this study we chose to concentrate on SQLite DBMS. This choice is motivated by the large adoption of SQLite. Moreover, the fact that SQLite is open source, in addition to the large available documentation are criteria that can help in explaining tests results. The developed methodology can however be applied to whatever DBMS on embedded system that supports the SQL language.

IV. BENCHMARKING METHODOLOGY AND TOOLS

A. Global Benchmarking Methodology

As discussed above, the objective of this study is threefold: (1) to evaluate (measure) the impact of unitary database requests on flash memory storage system from a performance and lifetime points of view. ( 2) to achieve comparative studies on different FFS and model the behavior of each of them for the tested SQL requests, and finally, (3) to optimize embedded database systems according to the obtained results.

For the sake of this paper, we focused on the first step of the study that is the performance evaluation and interpretation of the impact of SQL requests for embedded databases on flash memories based storage systems with a focus on two specific FFSs: JFFS2 and UBIFS.

Figure 2 describes the global methodology followed in our work. One can observe on the left hand side the tested platform. As discussed earlier, we relied on SQLite database engine for creating our databases and issuing SQL requests. SQLite has been installed on an embedded Linux platform executing a FFS on top of an embedded flash memory chip. A benchmarking tool relying on the SQLite API has been developed to automatically create the database, generate unitary SQL requests and measure the completion time of each request. In parallel to that, a tool, Flashmon [START_REF] Olivier | Flashmon v2 : Monitoring Raw Flash Memory Accesses for Embedded Linux[END_REF] [START_REF] Olivier | Flashmon v2 : Monitoring Raw Flash Memory Accesses for Embedded Linux[END_REF], that monitors all flash memory accesses (read, write, and erase operations), is executed in order to profile and trace SQL queries. Flashmon (described farther) gives a precise idea on the number of generated flash memory I/O accesses for a given workload. The novelty of our approach resides in using both quantitative and qualitative information in order to understand the system performance. By quantitative measures, we mean SQL request response times, while by qualitative measures; we mean the type and number of flash memory operations generated for our tests (number of reads, writes, and erasures). The use of both type of information gives more hints for understanding the interactions between the SQLite engine and the FFS (and more generally the embedded operating system, and the flash memory).

The results of our experimentations can allow to: (1) provide the developer with guidelines on how to better use the SQL database engine, which file system to use and how to tune it in order to get better performance. (2) Secondly to model performance of SQL requests with the objective to extract cost models in order to predict performance for a given workload. (3) Finally, to design optimization techniques based on the understanding of the performance behavior and the modeling step (cost models) [START_REF] Bellatreche | How to Exploit the Device Diversity and Database Interaction to Propose a Generic Cost Model?[END_REF]. The optimization can be applied on different levels: applicative layer (database engine itself) or the operating system layer (the flash management layer, FFS).

B. Metrics and Tools

In order to understand the performance of embedded databases, we mainly relied on two metrics: the response times (and throughput) of SQL requests, describing the system performance, and the number of generated flash memory operations: reads, writes, and erasures. This allows a better understanding of the performance in addition to a precise idea on the flash memory wear out through the metrics of number of erasures and number of writes.

1)

Embedded Database Benchmarking Tool: The tests consist in issueing SQL requests on a given table, and measuring the unitary response time of each request. The tested SQL requests are: the insertion, selection, join, and update of records. We first begin by generating a simple table that has two fields: a short integer that is the primary key and a character string. This string contained random data to avoid disturbance caused by the data compression performed at the FFS layer. Two parameters were varied for each SQL request: the number of records and their size (the table size). Each test was repeated many times to be sure of getting stable results. Response time was obtained with the gettimeofday system call (microsecond granularity). The database is located on a dedicated test partition of 50MB initially configured in JFFS2 format and then in UBIFS format to compare results. The partition is fully erased then formatted before each test to ensure a homogeneous initial state. Each set of experimentations for a given SQL request was preceeded by a BEGIN TRANSACTION instruction and ended by an END TRANSACTION instruction. It is recommended [START_REF]SQLite developers[END_REF] to use those two instructions especially in embedded systems, for instance, to prevent data corruption caused by power failures in the middle of a data-base transaction. This is implemented throughout the creation of journal log file that is specific to a given transaction and that is removed once the transactions succeeds. Return RT

Algorithm 1.Microbenchmarking

The database benchmark described in the preceeding algorithm was executed on UBIFS and JFFS2, for the insert, select, join (nested loop join), and update SQL queries with record sizes of 150, 300, 450, and 600 bytes, and a number of requests going from 100 to 5000 with an increment of 100.

2) The Flash Memory Monitor Flashmon: Flashmon [START_REF] Olivier | Flashmon v2 : Monitoring Raw Flash Memory Accesses for Embedded Linux[END_REF] is a Linux kernel module allowing to trace NAND flash I/O low-level operations on raw flash chips. Flashmon stands for flash monitor and traces the page read, page write and block erase operation performed on physical pages / blocks of the flash memory. The module is implemented at the MTD subsystem level (see Figure 1), it is then independent from the FFS and the NAND chip layer. Therefore, it can be used with all the previously evocated FFS, on all the NAND flash chips supported by the Linux MTD subsystem. Flashmon traces the operation type, time of arrival, physical address targeted, and also the name of the current task executed when the operation is traced. As MTD is completely synchronous, the traced task is responsible of the traced operation.

Flash management mechanisms such as dedicated FFS are rather complex due to the specific constraints exhibited by the memory. One simple I/O request from the applicative layer can end up in several read / write / erase flash operations according to the FFS state and algorithms as well as the flash memory state. So, in the context of this paper, performance evaluation of traditional metrics such as I/O response times or throughput are not sufficient and should be completed by some precise knowledge on the flash operations occurring during the benchmark. Flashmon helps embedded systems developers / researcher to extract and collect such information.

C. Hardware & Software test platform

The benchmarks were launched on the Armadeus APF27 development board [START_REF] Systems | APF27 Board Datasheet[END_REF]. It embeds an ARM9 based Freescale i.MX27 microprocessor clocked at 400 MHz, and 128 MB of RAM. The board contains 256 MB of Micron SLC NAND flash. Blocks in this flash chip are composed of 64 pages of 2 KB each. The chip datasheet [START_REF]MT29F2G16ABDHC-ET:D NAND Flash Memory Datasheet[END_REF] indicates that read, write, and erase operations' latencies are respectively 25 µs, 300 µs and 500 µs. From a software point of view, we used the 2.6.29.6 version of the Linux kernel, and SQLite 3.7.10.

V. RESULTS AND DISCUSSIONS

In this section, we try to analyze and discuss the results of the performance measures on the tested SQLite operations: "select", "insert", "join", and "update" operations.

A. Flash memory throughput calibration

In order to assess the raw throughputs offered by the flash chip and its management (driver and FFS), we ran a series of simple tests at various levels in the storage software stack (previously presented on Figure 1). We measured throughputs for (A) moving data from RAM to flash and (B) from flash to RAM in order to evaluate respectively flash write and read performance. These measurements were achieved at three levels: (1) the driver (MTD) level, under the FFS layer and close to the hardware layer; (2) the MTD applicative level: MTD programs allowing to access flash from user-space bypassing the FFS; and (3) the applicative level: a simple C program accessing files through the FFS. Results are depicted in Table 1. As one can see, the complexity induced by adding multiple software layers between the application performing flash I/O operations and the flash chip itself leads to an important performance.

B. The "insert" operation

For the insert operation, we created a database that was filled by inserting records in a loop via the SQLite API. We varied the number of inserted records and their size. The SQLite configuration used is the default one with the size of a SQLite page equal to 1KB.

From the throughput results shown on Fig. 3, one can observe that: (1) the throughput for both JFFS2 and UBIFS is low as compared to the throughput the flash memory can sustain for simple write operations (see previous section). This is due to meta-data management (file-system and SQLite) that is investigated farther in this section. (2) UBIFS performance is better than JFFS2 for most cases (except small 150 bytes records). The reasons behind such a performance difference are due to many file-system related factors that are highlighted in the following sections. We also can notice a large performance gap between small and large size records. This is related SQLite overhead of page management.

From the per-request response time analysis (see Table 2), one can draw three main observations (note that in addition to each request response time, we measured both "BEGIN TRANSCTION" and "END TRANSACTION" response times): (1) the insert operation response times are almost constant for a given request size, for both JFFS2 or UBIFS. [START_REF]NAND Chip Market Growth Propelled by Smartphones and Media Tablets[END_REF] The "BEGIN TRANSACTION" command at the beginning of each test takes almost a constant time to execute (around 32ms), whatever the request size, number or mounted filesystem. (3) The major performance difference between JFFS2 and UBIFS is noticed when the transaction is committed through the "END TRANSACTION" command. We noticed that the response time of this command is related to the size the inserted data (number and size of insert requests) and the file system. In fact, as we will see farther, this response time is related to the flush operation of the SQLite buffer to the flash memory.

In order to understand the difference between the behaviors of both tested file-systems, we performed some qualitative measures on the number of flash read and write accesses for the achieved tests. Fig. 4 shows the number of read I/O accesses per KB of data according to the number of insert requests for different record sizes (150, 300, 450, and 600).The curves show that for both reads and writes JFFS2 do more accesses to the flash memory. UBIFS performs a small fixed number of read operations that do not depend on the number of inserted data while for JFFS2 the number of performed reads strictly depends on the volume of inserted objects. Concerning the write operation, one can observe that JFFS2 performs more writes than UBIFS, this is due to two main characteristics: UBIFS maintains a proper buffer allowing to absorb more write operations, while JFFS2 does not (it is completely synchronous). The other reason behind such a behavior is that for JFFS2, we observed that the system performs interleaved reads and writes during the insert operation while UBIFS generates only writes. This may concern the JFFS2 metadata reading/checking. Finally, we also noticed an extremely interesting behavior related to SQLite. In fact, it maintains a very large buffer which prevents data from being flushed to the FFS underlying layer until the END TRANSACTION commit instruction is issued. Once the commit launched, all inserted objects are flushed to the flash memory, provided that the size of the inserted data is under a given threshold. In fact, above a given size of inserted data, flush operations from the SQLite buffer to the flash begin. This threshold was measured to be approximately 1.75MB. After this threshold flush operations to the flash memory are pipelined with the insert operations. We infer from this behavior that in Table 2, measured response times (except the first and last one) for the first requests (size > 1.75MB) are related to SQLite management of data in memory.

C. The "select" operation

In this section, we give some elements on the performance of the select operation. Before performing the tests, the database was filled through a set of insert requests. For each A first observation one can do when looking at Fig. 5 representing the throughput of the select SQL operation for different record sizes and request number, is that UBIFS slightly outperforms JFFS2 for the select operation for some measured record sizes. The throughput follows a logarithmic shape for both file-systems due to the constant overhead related to the log journal manipulation and other SQLite specific processing and memory usage.

Response time analysis (see Table 3) shows more stability (less value variations) for UBIFS response times as compared to JFFS2. In fact, for UBIFS, we can observe periodic small peaks (~2ms) which periodicity decreases according to the request size. The peaks have values that are approximately the double of the other stable values. For JFFS2, the amplitude of the peaks is much more significant as the value is around 42ms. The period of those peaks is also much larger and depend on the request size. In fact, this phenomenon (in JFFS2) is related to the Linux page cache read-ahead algorithm that prefetches chunks of 128KB (around 40ms, giving ~3MB/s throughput) of data, allowing future SQLite read accesses to be served from RAM rather than flash. This behavior is only observed for JFFS2 as in UBIFS the readahead mechanism is disabled by default [START_REF]UBIFS and Linux read-ahead[END_REF]. Read-ahead is mainly designed for hard disk drives, in which case system management allows asynchronous IO: while the system prefetches data, the CPU can execute other tasks taking benefit from I/O timeouts. As stated earlier the NAND driver (MTD) is fully synchronous, so in the best case read-ahead does not impact flash memory I/O performance. In the general case, read-ahead lead to a performance drop on flash storage, because all the prefetched data is not necessarily accessed in the future. This is the reason why it is disabled in UBIFS, and one of the explanations of the performance difference between the two tested FFS.

Even though UBIFS disables the read ahead mechanism of the Virtual File System of Linux, one can observe that the period of the peak response times is 4, knowing that each record (600 B for this example) is stored in a separate SQLite page (of 1KB), we can infer that the prefetch size is 4KB (2 flash pages). In fact, this is related to the I/O system page granularity on Linux which is 4KB. When observing the number of accesses (here I/O reads) in Fig. 6, one can clearly see the impact of the read-ahead prefetching algorithm. In fact, the peak values in Fig. 6 (for JFFS2 especially for 150 B objects) represent the case where a last prefetching (of 128KB) of data has been made but was not profitable as prefetched data were not used, thus generating more flash reads as compared to the needed data. In addition to the read-ahead related behavior, under JFFS2, the system performs more read operations especially for request number less than 1300 select operations. This is coherent with the I/O throughput observed in Fig. 5 but one could await a larger difference in the throughput when looking at the number of I/O reads generated for both file-systems. In fact, the flash memory I/O accesses times count for a given percentage of the total execution times but the SQLite processing and memory activity should not be ignored. They represent a large part of the execution time especially for small number of requests. For the example shown in Table 3 in case of JFFS2, measures show ~42ms response time each 64 requests. So if we consider a period of 64 requests, around 42ms are related to flash memory management while ~1*63ms are due to SQLite processing (memory and CPU). This proves the important overhead related to non-I/O operations.

D. The "join" operation

For the join operation requests, we created two tables from which we selected all elements of the second table that have an identifier (key) which value is equal to the one of the first table. This is done on a given number of objects corresponding to the number of requests. This join operation simply selects all the elements of the second table which identifier is smaller than the given number of requests. So the executed request looks like: "SELECT table2.val FROM table1, table2 WHERE table1.id = table2.id AND table2.id <= nb_request;" Fig. 7 shows the results for the measured throughput of the join operation relative to the set of selected objects. One can observe that for a high number of selected objects, UBIFS and JFFS2 give approximately similar results. Another observation one can do, is that we obtain better performances for join operations on 450 B objects than on 600 B objects for both file-systems. This is due to the internal fragmentation generated when the record size is 600 B, in this case, one SQLite file page (1 KB) contains only 1 record and the rest of the storage space is unused, while for record sizes of 450 B, two objects are put in 1 SQLite file page (filling 900 B of 1KB page as compare to 600 B).

The join operation performs only read I/O accesses to the flash memory layer. Fig. 8 shows the number of flash read operations performed per KB of requested data for both filesystems and different request size and number. The figure shows that for large request numbers, 600 B record request sizes give the worst performance for both file-systems due to internal fragmentation. For JFFS, we can observe the same phenomena as for the select requests, the read-ahead algorithm of the VFS page cache provokes some peaks that are visible especially for 150 B objects. The read-ahead mechanism is active whatever the record size and request number, but is graphically more visible when small portion of data are accessed. This is due to the fact that the overhead of the prefetch is proportionally higher.

One can also observe that for a small number of request sizes, UBIFS outperforms JFFS2, while for large numbers, performance are approximately equivalent as the number of flash memory reads per KB of accessed data is around 2 for both systems.

If we compare the join operation to the previous select query tests, one could question about the fact that the join gives far better performance. This is related to the manner with which the tests were performed. For the select operation, we issued one request per selected record while for the join, one query was issued to read the whole table objects. The difference between both cases is very important as the SQLite related processing overhead is much higher when issuing as many requests as there are records in the table. A single select on the same objects retrieved with the join operation gave better results which is normal. Indeed, for the join, the IDs of both tables are compared and the values of objects of the second table are retrieved while for the select, only the values of one table are read (no comparison).

E. The "update" operation

A first extremely important observation one can draw is the cost of the update operation as compared to the insert for both UBIFS and JFFS2. Updates are 2 times less performing than inserts for both JFFS2 and UBIFS. In order to update one object, the system needs to read the flash page containing the SQLite page which encapsulates the record to update. So to update the object, the system needs to rewrite all the data that are in the SQLite page.

We can also observe that UBIFS performs better than JFFS2, especially for large update sizes. This is mainly due to the additional buffering on UBIFS which prevents synchronizing the SQLite file page as frequently as JFFS2. Indeed, it collects the updates on the buffer before flushing to the flash memory. This behavior is clearly underlined by the number of write accesses described in Fig. 10. In addition to the additional read operations due to meta-data management of JFFS2, it performs an average of more than two times more write operations.

We can also see that for JFFS2, a larger number of erase operations are performed that can reach values of 0.07 block erasures per updated KB of data. Knowing that a block consists of 64*2KB pages, we can say that for updating 1KB of data for the 600B objects, we erase an equivalent of 4.48 flash pages (0.07*64), which is very significant. The values of the throughput and read/write I/O accesses are unstable, especially for JFFS2 due to the asynchronous garbage collector that is launched while the update test is running, generating additional flash read/write / erase operations and thus disturbing response times of the SQL queries.

VI. CONCLUSION AND FUTURE WORKS

This paper presents a set of results on measuring the performance of SQL requests targeting on-flash SQLite running on an embedded Linux operating system. As observed from the presented results, the performance behavior highly depends on the used FFS and can have a significant impact on the lifetime of the flash memory, for instance under JFFS2, the system can perform up to 0.07 to 0.08 erase operations for updating the equivalent of 1KB of data.

The performed tests revealed a very high disparity according to the varied parameters for the tested queries and the used FFS. The obtained results are closely related to the used FFS as it has a strong interaction with the SQL engine output due to the constraints of flash memories.

We can conclude that the performance of SQLite on embedded flash storage system depends on (1) the I/O load generated by the applicative layer (SQL requests) and its own buffering mechanisms; (B) the flash management algorithms (FFS) and (C) the state of the system: system / FFS caches state, but also flash memory state in terms of amount and location of valid / invalid / free pages. This was especially observed for update operations in which the garbage collector was launched after a given number of write operations were performed.

For future works, we consider investigating more SQLite complex requests as joins. We are also about to study real SQLite traces from typical embedded applications. Finally, we did not focus on the initial state of the flash memory in this paper. In fact we considered clean partitions at the beginning of each test. It would interesting to inject different initial states in terms of invalid/valid/clean pages and see how FFS deal with different configurations.

Figure 1 :

 1 Figure 1: I/O request software stack and Flashmon location in the Linux NAND storage hierarchy

Figure 2 :

 2 Figure 2: Embedded database performance evaluation methodology on flash based storage systems

Figure 3 :

 3 Figure 3: I/O throughput (KB/s) of the insert operations according to record size and number of requests

Figure 4 :

 4 Figure 4: Number of I/O accesses to the flash memory for insert operations according to record size and number of requests

Figure 6 :

 6 Figure 6: Number of I/O accesses to the flash memory for select operations according to record size and number of requests

Figure 5 :

 5 Figure 5: I/O throughput (KB/s) of the select request according to record size and number of requests

Figure 7 :

 7 Figure 7: I/O throughput (KB/s) of the join operations according to record size and number of requests

Figure 8 :

 8 Figure 8: Number of I/O accesses to the flash memory for join operations according to record size and number of requests

Figure 9 :

 9 Figure 9: I/O throughput (KB/s) of the update operations according to record size and number of requests

TABLE I

 I

	. MEASURED FLASH READ & WRITE THROUGHPUTS AT VARIOUS
	LEVELS IN THE FLASH STORAGE MANAGEMENT STACK
	Level	Write (MB/s)	Read (MB/s)
	MTD kernel level (driver)	3.36	5.94
	MTD userpace level (bypassing FFS)	2.33	3.80
	Applicative level (using FFS)	1.26	3.65

TABLE II .

 II SNAPSHOT OF RESPONSE TIMES (IN MS) OF INSERT OPERATIONS FOR DIFFERENT REQUEST SIZES ON JFFS2

		Record size (Bytes)/ resp. times in
			ms
	Query nb	150	300	600
	0 (BEGIN TR.)	32,75	35,02	34,29
	1	7,27	6,47	6,83
	11	0,66	0,63	0,91
	12	0,78	0,61	0,77
	13	0,76	0,67	0,78
	14	0,81	0,60	0,90
	15	0,66	0,60	0,79
	16	0,65	0,67	0,78
	17	0,65	0,83	0,92
	18	0,65	0,61	0,90
	19	0,74	0,67	0,78
	101 (END TR.)	20,86	39,07	116,84

TABLE III .

 III SNAPSHOT OF RESPONSE TIMES (IN MS) OF SELECT OPERATIONS FOR 600 BYTES OBJECTS ON JFFS2 AND UBIFS JFFS2 (req. Of 600 B) UBIFS (req, Of 600 B)

	Query nb	Time (ms)	Time (ms)
	0	30,52	33,25
	43	0,96	1,93
	44	1,06	0,93
	45	0,96	1,14
	46	42,65	1,17
	47	0,99	1,98
	48	1,07	1,18
	49	0,98	1,06
	50	0,96	1,04
	51	1,06	2,00
	52	0,96	1,19
	109	1,05	1,06
	110	42,14	1,17
	111	1,00	1,83
	112	0,98	1,24


Quentin Roy
email: quentin@quentinroy.fr

Camelia Zakaria

Simon Perrault
email: perrault.simon@gmail.com

Mathieu Nancel
email: mathieu.nancel@inria.fr

Wonjung Kim

Archan Misra
email: archanm@smu.edu.sg

Andy Cockburn
email: andrew.cockburn@canterbury.ac.nz

A Comparative Study of Pointing Techniques for Eyewear Using a Simulated Pedestrian Environment

Eyewear displays allow users to interact with virtual content displayed over real-world vision, in active situations like standing and walking. Pointing techniques for eyewear displays have been proposed, but their social acceptability, efficiency, and situation awareness remain to be assessed. Using a novel street-walking simulator, we conducted an empirical study of target acquisition while standing and walking under different levels of street crowdedness. We evaluated three phone-based eyewear pointing techniques: indirect touch on a touchscreen, and two in-air techniques using relative device rotations around forward and a downward axes. Direct touch on a phone, without eyewear, was used as a control condition. Results showed that indirect touch was the most efficient and socially acceptable technique, and that in-air pointing was inefficient when walking. Interestingly, the eyewear displays did not improve situation awareness compared to the control condition. We discuss implications for eyewear interaction design.

Introduction

Although introduced in the 1960s [START_REF] Sutherland | A Head-mounted Three Dimensional Display[END_REF], until recently see-through head-mounted displays (eyewear) were essentially dedicated to military and research environments [START_REF] Melzer | Head-mounted displays: designing for the user[END_REF]. Recently, however, there has been substantial commercial interest in developing eyewear technologies for public use, including Microsoft Hololens or Epson Moverio. While current devices have limitations, like narrow field-of-view, hardware is quickly improving, creating new possibilities for interaction in the office and home as well as during daily activities such as commuting. Indeed, academic and industry experts have suggested that eyewear displays might underpin a foundational change for the next generation of mobile interaction [START_REF] Costanza | Toward Subtle Intimate Interfaces for Mobile Devices Using an EMG Controller[END_REF][START_REF] Lucero | NotifEye: Using Interactive Glasses to Deal with Notifications While Walking in Public[END_REF][START_REF] Mustonen | Visual task performance using a monocular see-through head-mounted display (HMD) while walking[END_REF][START_REF] Lauber | In-your-face, Yet Unseen?: Improving Head-stabilized Warnings to Reduce Reaction Time[END_REF]. One potentially important advantage of eyewear displays is that they enable head-up interaction, which may enhance the user's situational awareness while engaged in concurrent activities, such as walking along a busy sidewalk. Current phones, in contrast, encourage a posture in which the head is bent down rather than looking outwards at the environment, causing poor environmental focus and attention, and raising significant safety concerns [START_REF] Basch | Pedestrian Behavior at Five Dangerous and Busy Manhattan Intersections[END_REF][START_REF] Nasar | Pedestrian injuries due to mobile phone use in public places[END_REF][START_REF] Ipsos | Distracted Walking Study: Topline Summary Findings[END_REF][START_REF] Mccullough | On Attention to Surroundings[END_REF][START_REF] Koh | Singapore trials LED lights on pavements: What other places are doing to keep smartphone zombies safe[END_REF].

Research on input methods for eyewear displays is in its infancy, especially when considering the interplay between the user's external urban activity and their internal (eyewear-driven) task. We focus on two-dimensional pointing as a basic interaction for input on eyewear displays. While other interaction modalities are being explored (for example gesture and voice), pointing remains fundamental in most vision-based human-computer interfaces. However, the design and evaluation of pointing techniques for eyewear displays poses particular challenges: their design must be properly adapted to mobility (e.g. efficient interaction while walking), and evaluations should account for environmental factors (e.g. navigating through a crowd), including impact on situation awareness and social acceptability.

As a first step in exploring the design space of eyewear pointing techniques, we examined practical solutions that can be readily adapted and implemented using today's off-the-shelf devices. We used a phone as an input device, a choice based on versatility (numerous sensors packaged in a small volume, allowing it to simulate handheld trackpads or in-air controllers for example), its ubiquitous ownership (smartglass users are likely to own and carry a phone) and its mobile pragmatics (for example bimanual techniques or bulky apparatus are impractical while walking).

We present an empirical study in which pointing techniques that have been proven useful in comparable contexts (like standing in front of an ultra-wall [START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF]) are adapted to phone input for eyewear displays-including variants of in-air pointing as well as using the phone as a hand-held trackpad. Direct touch on the phone, without eyewear, was used as control condition. We compare these techniques in three different environments: no simulator (while being stationary), simulated empty street and simulated busy street. Running such an experiment directly in a busy city street would put participants at risk, and is therefore ethically undesirable. Therefore, inspired by the work of Schwebel [START_REF] Schwebel | Validation of virtual reality as a tool to understand and prevent child pedestrian injury[END_REF], we developed a street simulator that enabled us to gain insights on the use of these techniques in pedestrian environments, while keeping our participants safe and preserving internal validity. Finally, we looked at three key metrics: perceived social acceptability, performance as a function of the simulated environment, and impact on situation awareness.

Our results demonstrate that (a) the trackpad technique was the most socially acceptable, most accurate, and fastest for eyewear, and (b) the in-air techniques (which are increasingly integrated in commercial AR products) tended to perform poorly and were subjectively unacceptable. Importantly, while a key expected benefit of eyewear is that their head-up view should improve situation awareness and safety while walking [START_REF] Mustonen | Visual task performance using a monocular see-through head-mounted display (HMD) while walking[END_REF][START_REF] Lucero | NotifEye: Using Interactive Glasses to Deal with Notifications While Walking in Public[END_REF][START_REF] Lauber | In-your-face, Yet Unseen?: Improving Head-stabilized Warnings to Reduce Reaction Time[END_REF], our results indicate that this may not be true: results indicate that situation awareness was worse when using the candidate techniques (i.e., with eyewear) than in the control condition without eyewear.

We make three specific research contributions: 1. empirical evidence of the perceived social acceptability of pointing techniques for eyewear, collected through interviews and web survey; 2. empirical evidence of the relative performance of eyewear pointing techniques in terms of speed, accuracy and ability to maintain situation awareness (for example avoid simulated pedestrian hazards); 3. demonstration of a Virtual Reality method for safely evaluating interaction techniques in a simulated pedestrian environment.

Background and Related Work

Two categories of previous research are briefly reviewed in the following subsections: first, general background research on pointing methods that might be adapted to eyewear displays, and second, research that focuses on interaction while engaged in other activities such as walking a busy street.

Pointing with Eyewear

Pointing to targets is an elemental component of interaction with graphically displayed content, and it is therefore important that efficient and acceptable pointing methods are developed for eyewear displays. While alternative selection methods that negate the need for pointing have been proposed-such as speech (e.g., [START_REF] Lumsden | A Paradigm Shift: Alternative Interaction Techniques for Use with Mobile & Wearable Devices[END_REF]) or hand-gestures (e.g., [START_REF] Mistry | WUW -Wear Ur World: A Wearable Gestural Interface[END_REF])-pointing-based methods offer substantial advantages due to their familiarity and learnability ('see and point versus learn and remember' [START_REF] Shneiderman | Direct manipulation for comprehensible, predictable and controllable user interfaces[END_REF]). Abundant novel or improved pointing techniques are represented in the HCI literature, and many could be adapted for eyewear. A key requirement for pedestrian pointing, however, is that the method can be operated while standing or walking. When eyewear displays are explicitly considered, the most commonly suggested pointing techniques are based on in-air pointing, in which the movement of the hand or a hand-held object is mapped to cursor movement (e.g., [START_REF] Ens | Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces[END_REF][START_REF] Hsieh | Designing a Willingto-Use-in-Public Hand Gestural Interaction Technique for Smart Glasses[END_REF][START_REF] Katsuragawa | Watchpoint: Freehand Pointing with a Smartwatch in a Ubiquitous Display Environment[END_REF]); a similar in-air pointing method is used with the Microsoft Hololens. Another approach is to use trackpad-like interaction, with a dedicated device held in the hand [START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF][START_REF] Mccallum | ARC-Pad: Absolute+Relative Cursor Positioning for Large Displays with a Mobile Touchscreen[END_REF], on the body [START_REF] Harrison | Skinput: Appropriating the Body As an Input Surface[END_REF][START_REF] Chen | uTrack: 3D Input Using Two Magnetic Sensors[END_REF][START_REF] Chan | FingerPad: Private and Subtle Interaction Using Fingertips[END_REF][START_REF] Saponas | PocketTouch: Through-fabric Capacitive Touch Input[END_REF][START_REF] Holleis | Evaluating Capacitive Touch Input on Clothes[END_REF] or the environment [START_REF] Harrison | OmniTouch: Wearable Multitouch Interaction Everywhere[END_REF][START_REF] Yang | Magic Finger: Alwaysavailable Input Through Finger Instrumentation[END_REF]. Finally, the use of eye-tracking or head-tracking is also possible [START_REF] Jalaliniya | Head and Eye Movement as Pointing Modalities for Eyewear Computers[END_REF][START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF].

All these pointing techniques are promising and most are valid candidates for eyewear pointing (providing the sensing mechanism can be made mobile). Research on large display interaction is also of interest as it often considers the users' need to stand or walk near the display [START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF][START_REF] Vogel | Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays[END_REF][START_REF] Liu | Gunslinger: Subtle Arms-down Midair Interaction[END_REF]. Only a few previous works specifically investigated pointing on eyewear. The work of Jalaliniya et al. on head and eye tracking [START_REF] Jalaliniya | Head and Eye Movement as Pointing Modalities for Eyewear Computers[END_REF], or Hsieh et al. on gloves [START_REF] Hsieh | Designing a Willingto-Use-in-Public Hand Gestural Interaction Technique for Smart Glasses[END_REF] are examples. However, as far as we know, none of these works comprehensively tackle social acceptability and formally investigate pointing performance under realistic urban movement, or used exotic, unrealistically bulky hardware. As a result, it is still unclear what is currently the best solution.

We focused on one-handed pointing techniques rather than bi-manual techniques because users often carry objects while walking. We used phones as input devices because they are readily available without requiring users to acquire a specialised input device; they also embed sensors that provide both touch (trackpad) and movement sensing (in-air controllers).

Interaction in Pedestrian Environments

The design of interaction techniques for use in pedestrian environments raises special challenges, including the need for the user to maintain situation awareness during interaction (to reduce safety concerns such as collisions with people or vehicles) and the need for the movements or actions required for interaction to be socially acceptable. In addition, there are also challenges for researchers in evaluating new technologies for pedestrian environments.

Situation awareness Several recent studies have highlighted evidence that the use of mobile phones in urban areas is elevating the risk of personal injury [START_REF] Basch | Pedestrian Behavior at Five Dangerous and Busy Manhattan Intersections[END_REF][START_REF] Nasar | Pedestrian injuries due to mobile phone use in public places[END_REF]. Rather than looking upwards and outwards at the environment, when interacting with a phone, the user's posture has the head bent down, causing poor environmental focus and divided attention [START_REF] Oulasvirta | Interaction in 4-second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI[END_REF][START_REF] Sawhney | Nomadic Radio: Speech and Audio Interaction for Contextual Messaging in Nomadic Environments[END_REF][START_REF] Lumsden | A Paradigm Shift: Alternative Interaction Techniques for Use with Mobile & Wearable Devices[END_REF]. This leads to the emergence of the "phone zombies" phenomenon: pedestrians who pay insufficient attention to their environment while looking at their phones, sometime walking into other people or traffic [START_REF] Basch | Pedestrian Behavior at Five Dangerous and Busy Manhattan Intersections[END_REF][START_REF] Nasar | Pedestrian injuries due to mobile phone use in public places[END_REF][START_REF] Ipsos | Distracted Walking Study: Topline Summary Findings[END_REF][START_REF] Mccullough | On Attention to Surroundings[END_REF]. In an attempt to ease these problems, cities such as Singapore and Melbourne have started to install LED strips on pavements at pedestrian crossings [START_REF] Koh | Singapore trials LED lights on pavements: What other places are doing to keep smartphone zombies safe[END_REF].

Rather than altering the environment, another approach to improving situation awareness is to alter the interface [START_REF] Lumsden | A Paradigm Shift: Alternative Interaction Techniques for Use with Mobile & Wearable Devices[END_REF], and to explore interaction mechanisms that are more fit for the challenges of pedestrian environments [START_REF] Lucero | NotifEye: Using Interactive Glasses to Deal with Notifications While Walking in Public[END_REF][START_REF] Zhao | Earpod: Eyes-free Menu Selection Using Touch Input and Reactive Audio Feedback[END_REF][START_REF] Lumsden | A Paradigm Shift: Alternative Interaction Techniques for Use with Mobile & Wearable Devices[END_REF][START_REF] Pirhonen | Gestural and Audio Metaphors As a Means of Control for Mobile Devices[END_REF], such as eyes-free interaction [START_REF] Zhao | Earpod: Eyes-free Menu Selection Using Touch Input and Reactive Audio Feedback[END_REF][START_REF] Jain | User Learning and Performance with Bezel Menus[END_REF] or the use of eyewear [START_REF] Lucero | NotifEye: Using Interactive Glasses to Deal with Notifications While Walking in Public[END_REF][START_REF] Hsieh | Designing a Willingto-Use-in-Public Hand Gestural Interaction Technique for Smart Glasses[END_REF]. Researchers have argued that eyewear displays allow more seamless integration between the display of information and the surrounding environment, and as a result, improve situation awareness [START_REF] Mustonen | Visual task performance using a monocular see-through head-mounted display (HMD) while walking[END_REF][START_REF] Lucero | NotifEye: Using Interactive Glasses to Deal with Notifications While Walking in Public[END_REF][START_REF] Lauber | In-your-face, Yet Unseen?: Improving Head-stabilized Warnings to Reduce Reaction Time[END_REF]. However, there is a lack of empirical study testing this assumption, possibly due to the risks associated with placing experimental participants in congested urban settings.

Social Acceptability Montero et al. define social acceptability as the combination of the user's social acceptance, which defines how comfortable a user is in executing a particular action, as well as spectators' social acceptance, which refers to the impression it makes on witnesses of such action [START_REF] Montero | Would You Do That?: Understanding Social Acceptance of Gestural Interfaces[END_REF]. Interacting with eyewear displays need to be socially acceptable for public performancethis is especially relevant for eyewear given the current scepticism toward such devices [START_REF] Koelle | All About Acceptability?: Identifying Factors for the Adoption of Data Glasses[END_REF]. While the social acceptability of actions may change as technologies become widespread [START_REF] Montero | Would You Do That?: Understanding Social Acceptance of Gestural Interfaces[END_REF], the likelihood of technology adoption is greatly improved if its interaction requirements are socially acceptable [START_REF] Goffman | The presentation of self in everyday life[END_REF][START_REF] Rico | Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability[END_REF]. Several factors are known to influence the social acceptability of actions, including movement duration (the shorter the better) [START_REF] Dobbelstein | Belt: An Unobtrusive Touch Input Device for Head-worn Displays[END_REF], and movement amplitude (small, discreet movements are better) [START_REF] Montero | Would You Do That?: Understanding Social Acceptance of Gestural Interfaces[END_REF][START_REF] Ronkainen | Tap Input As an Embedded Interaction Method for Mobile Devices[END_REF].

Evaluating interaction techniques for pedestrian environments

There are well known trade-offs between lab and field studies, with lab studies facilitating internal validity at the cost of external validity, and field studies the inverse [START_REF] Kjeldskov | Was It Worth the Hassle?: Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations[END_REF][START_REF] Hühn | On the Use of Virtual Environments for the Evaluation of Location-based Applications[END_REF].

Beyond concerns of internal validity, there are additional and important safety concerns that complicate the potential conduct of field studies in urban pedestrian environments [START_REF] Schwebel | Validation of virtual reality as a tool to understand and prevent child pedestrian injury[END_REF]. Consequently, researchers have examined the use of simulations to reproduce some of the realistic interaction context in safe settings. When the research focus is on the act of walking (e.g., to understand motor perturbations to interaction caused by pacing) treadmills have been used [START_REF] Bergstrom-Lehtovirta | The Effects of Walking Speed on Target Acquisition on a Touchscreen Interface[END_REF][START_REF] Barnard | An Empirical Comparison of Use-inmotion Evaluation Scenarios for Mobile Computing Devices[END_REF][START_REF] Ng | Comparing Evaluation Methods for Encumbrance and Walking on Interaction with Touchscreen Mobile Devices[END_REF]. And when the research focus is on environmental artefacts, video projection [START_REF] Lumsden | Investigating Microphone Efficacy for Facilitation of Mobile Speech-based Data Entry[END_REF] and virtual reality [START_REF] Hühn | On the Use of Virtual Environments for the Evaluation of Location-based Applications[END_REF][START_REF] Kim | Empath-d: Vr-based empathetic app design for accessibility[END_REF][START_REF] Schwebel | Distraction and pedestrian safety: How talking on the phone, texting, and listening to music impact crossing the street[END_REF][START_REF] Schwebel | Validation of virtual reality as a tool to understand and prevent child pedestrian injury[END_REF]] can be used. We chose the later approach. Notably, we were inspired by the work of Schwebel et al. who used a simulated street environment to investigate child safety at road crossings [START_REF] Schwebel | Distraction and pedestrian safety: How talking on the phone, texting, and listening to music impact crossing the street[END_REF]. Using a simulated environment not only replicates common pedestrian constraints, but also provides better control of parameters and allows us to put participants in simulated risky situations without physical risks.

Perceived Social Acceptability

To reiterate, social acceptability is a key issue in the design of interaction techniques for use in public settings. We structured our investigation of social acceptability in two parts: (1) semi-structured interviews, (2) a large-scale web survey. The goal of the study was to seek participants' perception of the social acceptability of the investigated interaction techniques. We focused on the user's social acceptance [START_REF] Montero | Would You Do That?: Understanding Social Acceptance of Gestural Interfaces[END_REF].

Table 1. Techniques included in our social acceptability study

In-Air Techniques Front Translation Translation of the hand on a plane facing the user [START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF][START_REF] Vogel | Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays[END_REF] Down Translation Translation of the hand on a plane parallel to the ground [START_REF] Katsuragawa | Watchpoint: Freehand Pointing with a Smartwatch in a Ubiquitous Display Environment[END_REF] Front Rotation

Rotation of the wrist and forearm as if laser-pointing on a plane in front of the user (Fig. 2 left) [START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF][START_REF] Vogel | Distant Freehand Pointing and Clicking on Very Large, High Resolution Displays[END_REF]

Down Rotation

Rotation of the wrist and forearm as if laser-pointing on the ground [START_REF] Katsuragawa | Watchpoint: Freehand Pointing with a Smartwatch in a Ubiquitous Display Environment[END_REF] Front Taps in-air taps on a plane parallel to the user (back-and-forth movements of the forearm and the index finger) [START_REF] Ens | Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces[END_REF] On-Body Touch Techniques Finger Touch The tip of the index is used as a trackpad controlled with the thumb [5] Palm Touch

The joined area of the four long fingers is used as a trackpad controlled with the thumb [6] Pocket Touch

The pocket area is used as a trackpad [START_REF] Saponas | PocketTouch: Through-fabric Capacitive Touch Input[END_REF] On-Device Touch Technique Trackpad A hand-held device is used as a touch surface (Fig. 2

right)

Our interview sessions were inspired by Rico and Brewster's methodology [START_REF] Rico | Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability[END_REF]: participants were asked to perform different gestures as if they were interacting with the device, and we gathered their feedback on the gestures' social acceptability (for public and private use). All interviews took place in a public setting within a local university campus.

For the web survey, participants watched online videos of the techniques, and were asked to rate their social acceptability. Videos are often used as a way to assess social acceptability [START_REF] Bailly | ShoeSense: A New Perspective on Gestural Interaction and Wearable Applications[END_REF][START_REF] Rico | Usable Gestures for Mobile Interfaces: Evaluating Social Acceptability[END_REF][START_REF] Chen | uTrack: 3D Input Using Two Magnetic Sensors[END_REF][START_REF] Serrano | Exploring the Use of Hand-to-face Input for Interacting with Head-worn Displays[END_REF]. The web survey was included to broaden participation in the study.

In our interview sessions, we examined a set of nine pointing techniques selected from previous literature (c.f. Table 1). Five of them were in-air gestures and one used a hand-held device as a trackpad. Though not the main focus of this work, we also included three body-touch techniques to learn about people's perception of less-common input methods.

After the interview sessions, we discarded the techniques with very poor rankings and kept Front Rotation, Down Rotation, Trackpad, Finger Touch and Pocket Touch for the web survey. Palm Touch was excluded because it was comparable to Trackpad. Front Rotation was slightly modified to allow movements from both the elbow and the wrist. Pocket Touch, which received polarized feedback in our interviews, was also modified so that the control area was shifted to the side of the thigh, further away from the genitals.

Participants and Procedure

For the interview sessions, we recruited eight participants (5 female), aged 22 to 45 years old (M = 32.1, SD = 6.5) from our students and university staff. Seven participants lived in Singapore and one in France. No compensation was offered. Each session lasted 45 minutes.

For each of the nine pointing techniques, we carried out the following procedure: (1) demonstrated the pointing movements for that technique and made sure its principles were understood, (2) asked the participant to perform the movements for approximately 30 seconds in a busy public area of our university campuses, (3) conducted a semi-structured interview focusing on their perception of the social acceptability of the techniques, at home or in the street. We finished the session by asking our participants to rank all techniques by order of social acceptability for usage in a private setting (e.g. home) and public setting (e.g. a street).

From our web survey, we gathered 56 responses (25 female, 1 preferred not to disclose) from 18 to 57 years old (M = 27.7, SD = 8.2, 7 preferred not to disclose). The web survey was advertised using our university's mailing lists. 50% of the participants were students (undergraduate and post-grads), 26% were IT Professionals and 5% in Academia and Research. They were mostly from South-East Asia (n = 41) and Europe (n = 8). The survey was divided into six parts, one dedicated to each technique, and a summary. In each survey part, participants were shown a short video of an actor walking in a street and demonstrating the use of the technique. Then they were asked to provide feedback on the perceived social acceptability of these techniques. Finally, they ranked techniques in order of perceived acceptability both for public and private contexts (1, most acceptable; 6, least acceptable).

Results and discussion

During the interview sessions and in the web survey, we asked participants about their perception of the social acceptability of the techniques in private and in public. We did not observe any statistically significant effect of the participants' continent of origin on the recorded answers.

Private Use In the interviews, Finger Touch was ranked as being the most socially acceptable technique (M = 1.8 /9), followed by Phone Touch (M = 3.3 /9), Pocket Touch and Palm Touch (both M = 3.8 /9). Among the in-air technique, Front Taps was ranked as the least socially acceptable (M = 4.6 /9). A Friedman test showed a significant effect of technique on average ranking (χ 2 (8) = 37.3, p < .00001), although Bonferroni corrected analysis showed no pairwise differences.

In general, the interview results suggested that for private use, smaller onbody or on-device actions were perceived as more socially acceptable than larger in-air movements of the device. On-body movements were perceived to be less tiring (2 interview participants) and as a result easier to use in private, and on-device actions were reported as being familiar [START_REF] Chan | FingerPad: Private and Subtle Interaction Using Fingertips[END_REF]. The in-air techniques were considered as "tiresome" (4) and "intrusive" [START_REF] Chan | FingerPad: Private and Subtle Interaction Using Fingertips[END_REF]. Our web survey results tended to confirm this trend: we found a significant main effect of technique on the average ranking for private use (χ 2 (5) = 108.3, p < .0001, see Fig. 3

for post-hoc comparisons).

Public Use Results for use in public spaces reflected those for private use. Among the on-body techniques, Finger Touch was ranked as being the most socially acceptable (M = 1.4 /9) followed by Trackpad (M = 2.9 /9), Palm Touch (M = 3.4 /9) and Pocket Touch (M = 5.0 /9). Among in-air techniques Down Rotation was ranked best (M = 4.6 /9) followed by Front Rotation (M = 4.6 /9), Down Translation (M = 6.5 /9), Front Taps (M = 7.5 /9) and Front Translation (M = 8.4 /9). A Friedman test showed a significant main effect of technique on average ranking (χ 2 (8) = 43.2, p < .00001).

Consistent with previous work [START_REF] Montero | Would You Do That?: Understanding Social Acceptance of Gestural Interfaces[END_REF][START_REF] Ronkainen | Tap Input As an Embedded Interaction Method for Mobile Devices[END_REF], participants expressed concerns with high amplitude movements. In particular, five interview participants reported that Front Translation exceeded their "personal space", and got "in the way of others". Participants also expressed strong concerns on Front Taps [START_REF] Ens | Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces[END_REF] that made them appear as though they were pointing at others, explaining the large ranking difference compared to a private setting. This is potentially important, as contemporary implementations such as Microsoft Hololens use this modality as a primary means for interaction. Finally, Pocket Touch was polarizing in our interviews: half of our participants expressed little social concern, the other half strongly opposed to what they perceived as sexually suggestive (2 participants, 1 male and 1 female, even entirely refused to perform the gesture in public as per protocol).

As suggested by one of the interview participants, in the web survey videos we moved the control area for Pocket Touch further to the outside thigh region. This improved the ranking of the technique compared to our interviews. The rest of our web survey results tend to confirm the trend observed during the interviews: we found a main effect of technique on the average ranking for public use (χ 2 (5) = 119.7, p < .0001, see Fig. 3.

Performance and situation awareness

We explored pointing techniques enabled by everyday devices and usable by pedestrians. We compared the three techniques presented in Fig. 2: Front Rotation, Down Rotation and Trackpad. Front Rotation requires positioning the phone flat (screen up), then pressing and holding the screen while rotating the wrist and forearm left, right, up, or down to move the cursor 6 , not unlike tilt techniques [START_REF] Rahman | Tilt techniques: Investigating the dexterity of wrist-based input[END_REF]. Trackpad requires sliding the thumb on the screen to move the cursor. Relaxed Rotation requires holding the phone sideways while keeping the arm down in a relaxed position; the cursor is then moved by pressing and holding on screen while rotating the wrist left, right, up, or down. We designed Relaxed Rotation to require movements comparable in amplitude to Down Rotation. As a result, we believe that it should be perceived to have similar social acceptability (Down Rotation was perceived as the most socially acceptable in-air technique in the previous study). In all three techniques, target acquisition could be performed either by tapping on the screen or pressing one of the volume buttons. In practice, and due to the different grasps, the volume buttons were only used with Relaxed Rotation.

We included direct touch pointing on the phone display as a control condition, with participants instructed to hold and interact with the phone using one hand. Eyewear was disabled and removed in this condition, so participants had to look down while acquiring the targets.

Except for the control condition, all techniques made use of a mobile phone as an indirect, eyes-free controller. The controller was always manipulated with only one hand because pedestrians often need their other hand for activities such as opening doors, carrying bags, etc. In all techniques but direct touch (control), the visual feedback was exclusively displayed on the smartglasses.

Our social acceptability study included several on-body techniques that we did not include in this experiment because we wanted to focus on currently pragmatic phone-based techniques. Furthermore, our pre-tests and pilots indicated that the Down Rotation technique was excessively hard to control, so we eliminated it. In-air translation-based techniques and Front Taps were also excluded due to their poor social-acceptability findings in the previous study.

We compared the remaining techniques under three different environments: No Simulator in which participants stood while performing pointing task; Empty Street, where participants walked in an empty street simulation with no red lights or pedestrians; and Crowded Street, where participants walked in a street simulation including traffic lights and pedestrians (Fig. 1).

We formulated the following hypotheses:

H 1 : Users achieve the fastest pointing with Phone because of their familiarity with traditional direct-touch pointing, H 2 : Users achieve the lowest walking speed and poorest situational awareness with Phone because they are required to look down (at the phone), H 3 : In the two street environments, users achieve faster pointing with Trackpad than with Front Rotation and Relaxed Rotation, because they are accustomed to trackpads and because the technique's input is arguably less sensitive to walking movements, H 4 : Users achieve the highest walking speed and best situational awareness using Trackpad because they are not required to look down, and Trackpad's input is arguably less sensitive to walking movements.

Street Simulation

Exploring safety or situation awareness in the wild implies putting participants at risk (e.g., within close vicinity to vehicle traffic), which is not ethically acceptable. Instead, inspired by previous works in social science [START_REF] Schwebel | Validation of virtual reality as a tool to understand and prevent child pedestrian injury[END_REF], we rely on a street simulator (see Fig. 1) to investigate the ability of users to maintain situational awareness while interacting with the eyewear device. Participants stood in front of a wide display, and their body movements were tracked using fiducial markers. Walking on the spot caused the camera to move forward at a speed that the participant could control (treadmills, often used in previous works [START_REF] Bergstrom-Lehtovirta | The Effects of Walking Speed on Target Acquisition on a Touchscreen Interface[END_REF][START_REF] Barnard | An Empirical Comparison of Use-inmotion Evaluation Scenarios for Mobile Computing Devices[END_REF][START_REF] Ng | Comparing Evaluation Methods for Encumbrance and Walking on Interaction with Touchscreen Mobile Devices[END_REF], do not allow pace control). Participants had to step sideways to avoid incoming pedestrians in the Crowded environment, and stop at red lights.

As realistic as it is, a simulation cannot be as externally valid as an inthe-wild experiment. The generalizability of our findings to real street scenarios remains for further work. Nevertheless, the method does require participants to remain aware of the situation and as a result provides actionable insights on situation awareness.

Street Elements Several factors influence a pedestrian's walking behavior, such as street layout, illumination, and other pedestrians. Previous work in social sciences have focused on distracted behavior in road-crossings [START_REF] Schwebel | Validation of virtual reality as a tool to understand and prevent child pedestrian injury[END_REF][START_REF] Schwebel | Distraction and pedestrian safety: How talking on the phone, texting, and listening to music impact crossing the street[END_REF][START_REF] Basch | Pedestrian Behavior at Five Dangerous and Busy Manhattan Intersections[END_REF]. However, Oulasvirta et al. observed that the most attention-taxing situations encountered by pedestrians are when they walk in busy streets [START_REF] Oulasvirta | Interaction in 4-second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI[END_REF]. After discussion and further observations of our own behavior in the street, we included incoming pedestrians and changing traffic lights.

We used a simple street layout: a series of blocks with the same length and walkway width, not unlike some North-American cities. We designed these blocks to appear shorter (in length) than usual, to increase the number of intersections encountered by the participants.

Layout and Traffic Lights Each street block was separated by a crosswalk and a traffic light. Traffic lights could have four different behaviors: Fixed Green, Fixed Red, Changing Green and Changing Red. Fixed Green remained green, Changing Green and Changing Red changed from one to the other when participants were 0.016 to 0.039 blocks away. Changing Green and Fixed Red switched to green after a wait time of 1 to 2.5 seconds. The ordering of Light behaviors were randomized, but we ensured that each behavior appeared at least once every four lights. Audio feedback of a car honk was played if participants jaywalked.

Pedestrian Behavior Simulated pedestrians walked towards the participants at a speed randomly assigned between 2.46 and 3.78 blocks per minutes. They walked in straight lines, stopped to avoid "bumping" into participants, and respected traffic lights. Audio feedback of a pedestrian shouting "hey!" was also played participants collided into them. In the Crowded Street condition, the street contained approximately 8 pedestrians per block (see companion video).

Steps and Position Tracking Fiducial markers [START_REF] Fiala | Designing Highly Reliable Fiducial Markers[END_REF] were attached on the participants' ankles to track stomping motions, as well as the participant's lateral position in front of the display. Our tracking algorithm enabled us to map the participants' simulated walking speed as a function of both their stomping pace and the vertical amplitude of their steps.

Vanishing Point Adaptation

The vanishing point of the scene was kept aligned with the participant's position in front of the display when they stepped sideways, as opposed to constantly fixed at the center of the display, to further support the realism and immersiveness of the simulation (see video figure).

Participants and Apparatus

Twelve right-handed participants were recruited from Singapore Management University's students and staff (7 female) aged 20 to 30 years old. Remuneration was the equivalent of 7.4 USD. All but one reported that they had used their phone while walking in the street at least once in the two days before the experiment.

The experimental software was run on an Epson Moverio BT-300 smart-glass, a Samsung S7 Edge smart-phone and two computers (one for the simulation, one for the devices). The simulation was run on a large TV monitor (75 inches diagonal, 1.65 × 0.93 meters) positioned 1 meter from the ground. Participants stood 1 meter from the display, and could move left and right in front of it.

Task

Participants were instructed to perform an ISO 9241-9 standard Fitts multidirectional pointing task as established by Soukoreff et al. [START_REF] Soukoreff | Towards a Standard for Pointing Device Evaluation, Perspectives on 27 Years of Fitts[END_REF] (see Fig. 5) using one of the four techniques (see Fig. 2). We chose a Fitts' Law task type for internal validity: controlling pointing distance and size simplifies comparison between techniques and with previous and future work. Except for the Phone condition, the display area on the eyewear appeared to be approximately 199.2 × 199.2 mm one meter away from the user (720 × 720 px), the radius of the targets layout (see Fig. 5) was 84.7 mm (306 px) and the radius of the targets was 10.5 mm (38 px). In the Phone condition, the total pointing area was 65.4 × 65.4 mm (1, 376 × 1, 376 px), the radius of the target layout was 27.9 mm (585 px) and the radius of the targets was 3.5 mm (73 px). In both cases, the ratio of the layout radius on the target radius remains constant ( 306 38 = 585 73 = 8), yielding the same Index of Difficulty [START_REF] Fitts | The information capacity of the human motor system in controlling the amplitude of movement[END_REF]. The extra space around the targets discouraged the use of edge pointing.

All three eyewear techniques (Front Rotation, Relaxed Rotation, and Trackpad) were indirect and relative. We defined transfer functions to map participant's input to cursor movements using Nancel et al.'s sigmoid function [START_REF] Nancel | Mid-Air Pointing on Ultra-Walls[END_REF]:

G t = G min + G max -G min 1 + e -λ(vt-V inf )
with v t and G t respectively the input speed and gain at time t and λ a constant. Reporting generalizable (typically, physical) display units is complicated with smart glasses: the pixels can be perceived as if they were at any distance from the user's eyes. Furthermore, since the virtual display is displayed as a flat surface facing the user, rather than a spherical one, angular units cannot be used consistently. For simplicity and generalizability, we report distance and speeds on the display as if the display was projected one meter away from the user's eyes. The 1280 × 720 pixel map of the Moverio BT-300 corresponds to a 354 × 199 mm area one meter away, so one pixel is 0.277 mm wide 7 . We tuned its parameters separately for each technique (see Table 2). In the No Simulator condition, participants executed the task while standing. In both street conditions, participants completed the task while navigating through the simulated street. Specifically in the Empty Street condition, participants were only required to walk on the spot. They were instructed to strive to maintain a natural walking speed. In the Crowded Street condition, they were asked to also avoid pedestrians and respect traffic rules, as in real world. If they failed to meet these rules, the simulation flashed red while the corresponding audio feedback was played (a man shooting "hey!" for pedestrians, and a car honk for the lights). We simplified this experiment by standardizing the walk to a straight path, under the hypothesis that it is reasonably similar to following a well-known path in term of cognitive load. This method was designed to simulate the most common external constraints encountered by pedestrians while enabling measures of pace, awareness, and interaction performance, as realistically as possible.

We finished the experiment with a short semi-structured interview during which participants provided subjective feedback. Before starting the experiment and as a training, participants were introduced to both conditions of the street simulator. During this training, we also asked participants to find what they thought was their usual walking speed and we recorded it for later comparison. Before each technique and under each environment, participants also had the opportunity to train themselves with the technique before starting.

Design

We used a 3 × 4 within-subjects design with the following factors and levels: environment {No Simulator, Crowded Street, Empty Street} and technique {Front Rotation, Relaxed Rotation, Trackpad, Phone}. To ensure consistency of the simulation across all participants, we generated four predefined crowded street configurations (including pedestrian position and speed, lights, etc.).

The experiment was divided into three parts, one for each environment condition. Each of these parts were divided into four blocks, each one dedicated to a technique. In accordance with ISO 9241-9 [START_REF] Soukoreff | Towards a Standard for Pointing Device Evaluation, Perspectives on 27 Years of Fitts[END_REF], each block started with the cursor centered and an initial unmeasured target selection, followed by four selections of each of the 9 targets in opposing order (see Fig. 5). environment and technique orders were counterbalanced using a Latin Square. In the street conditions, the simulation ran uninterrupted during a block, and restarted afterwards. For all trials, we measured selection time, wrong selections (clicks outside of the target), "walking" speed, number of pedestrian collisions, and jaywalking. Participants were allowed to take breaks between each in the simulation. We found a significant main effect of Street Environment (F Jaywalking and Collisions We only considered the Crowded Street scenario when investigating jaywalking and pedestrian collisions. Note that because of the differences in selection time (Fig. 6-top), we computed the average number of jaywalkings and pedestrian-collisions per block (a block is a unit of distance in the simulator). We did not found a significant effect of technique on jaywalking (p = .13, with 0.1 jaywalk/block on average), nor on collisions (p = .49, 0.659 collisions/block on average).

Subjective Assessments

We asked participants to rate the techniques on a 7level Likert-scale in term of (1) ease of use, (2) enjoyability, (3) effectiveness, (4) safety, (5) situation awareness, (6) ease to avoid pedestrians and (7) respect of traffic lights. A Friedman's test showed a significant main effect of technique on all questions: ease of use (χ 2 (3) = 23.4, p < .0001), enjoyability (χ 2 (3) = 18.5, p < .001), effectiveness (χ 2 (3) = 22.2, p < .001), safety (χ 2 (3) = 14.9, p < .01), situation awareness (χ 2 (3) = 12.2, p < .01), ease to avoid pedestrians (χ 2 (3) = 11.9, p < .01) and respect of traffic lights (χ 2 (3) = 9.9, p = .019).

Figure 7 shows these results. At the end of the experiment, participants also ranked the techniques from most-preferred to least-preferred, specifically in a crowded environment. Five participants ranked Trackpad best, while five others ranked it second-best. Accordingly, five participants preferred the Phone first, one participant secondbest, and another one third-best. One participant ranked Phone worst. Front Rotation was ranked second by three participants, while Relaxed Rotation was consistently ranked either third-best or least-preferred. Many participants felt that the Trackpad technique poses less danger on the streets, in comparison to frequent look-ups while interacting with the phone.

Ecological Validity Experiment

As an extra validation step, we ran an ecological validity experiment to challenge our simulation-bound results in a real street situation. For safety reasons, it was not ethically acceptable to use external participants. Four of the authors took part in the experiment to test the four pointing techniques in the (actual) wild. We used the same techniques and protocol, with two environment conditions: Wild and Inside, and 5 repetitions of each pointing target instead of 4 during the controlled experiment. In the Wild condition, the authors walked along a busy underground concourse. We measured an average 22.9 pedestrians /minute at this location and time, with a large variance. In the Inside condition, the authors performed the pointing tasks standing but without walking. We counterbalanced the order of the techniques using a Latin Square, and measured both selection times and selection errors. Due to the small population, we only report descriptive statistics, shown in Fig 8 . None of the four authors collided with a pedestrian. The results of this experiment show the same trend as our main experiment. Participants were generally faster and more accurate, which can be explained by a higher expertise with the techniques and by a lower pedestrian density.

Limitations

Techniques Some participants reported difficulties with Relaxed Rotation due to the width of the phone: it made it difficult to press-and-hold or click. Relaxed Rotation may perform differently with a more ergonomically adapted in-air controller. Two participants also reported visual fatigue. Hopefully, this issue can be resolved on future eyewear displays.

Generalizability Our street-walking simulator allows us to put participants in controlled situations resembling crowded streets. This novel methodology enables to gather preliminary insights on situation awareness without putting participants at risk. Concerns over generalizability to real-world situations are eased through our limited validity experiment, but further experimental validation is difficult due to risks of participant harm. When Schwebel et al. run into a similar problem, they argue that at least three indicators can still be considered: immersion, interactiveness, and realism [START_REF] Schwebel | Validation of virtual reality as a tool to understand and prevent child pedestrian injury[END_REF].

Though not as immersive as a virtual-reality "cave", we join Schwebel et al. in the argument that a large display area covering most of the participant's field of view provides sufficient immersion. At least four of our participants agreed and described our simulation as "immersive".

We think interactiveness is a strength of our simulator as it included pace and position control (using feet tracking) instead of less interactive controller like a joystick, or pure absence of control as when using treadmills.

On the realism side, our participants' opinions were more divided: three reported the simulation as "unrealistic" and four stated that it was "realistic". Criticisms mostly concerned the in-place stepping mechanism, although two participants described the speed control as being "natural". This is a trade-off for the interactiveness required by our experiment. When VR treadmills are commercialized, allowing pace control, they may provide a better alternative.

Allowing participants to change direction, like walking around a corner, would allow more complex paths to be walked and add interesting factors. Similarly, half of our participants observed that real-world pedestrians typically give way when a collision is about to occur (rather than stop in front of the participant in our simulation). This behavior can easily be added, though we needed participants to actively avoid the pedestrians.

Less trivially, the stakes of colliding into a pedestrian or jaywalking remain limited compared to real-life. Keeping participants safe is of course the main point of using a simulation, but recent approaches such as force-feedback [START_REF] Lopes | Muscle-propelled Force Feedback: Bringing Force Feedback to Mobile Devices[END_REF] could be put to use to produce physical sensations at no cost of safety.

Discussion

Participants were able to point faster and with fewer errors using the Phone technique. We therefore find support for H 1 . Contrary to our expectations, the better performance of Phone did not come at the cost of slower walking speed or worse situation awareness. Therefore, we reject H 2 and H 4 . This can be explained by a strong discrepancy in participants' prior experience between the control and candidate techniques.

Trackpad emerged as the best pointing technique for eyewear in terms of speed and error rate, in both the No Simulator and our street simulations. Therefore, we find support for H 3 . Trackpad was also perceived as more socially acceptable, easier to use and more enjoyable than the in-air techniques. These results could be influenced by the smaller movements or lower cognitive burden associated with highly familiar touch interaction.

The in-air techniques, Front Rotation and Relaxed Rotation, performed worse in every condition. Performance with these techniques was also more adversely affected by street crowding than the other techniques (selection times increased dramatically in the Street conditions). Despite the higher movement amplitude and the need to keep the forearm up, Front Rotation was found easier to use than Relaxed Rotation. This might be due to the additional joint involved (wrist + elbow vs. wrist only) [START_REF] Guiard | The Kinematic Chain as a Model for Human Asymmetrical Bimanual Cooperation[END_REF]. We observe significant differences in selection time for all environments, and increased errors in the Crowded Street compared to the No Simulator condition. The participants were also able to walk faster in Empty Street than in Crowded Street.

Interestingly, Trackpad and Phone performance were not significantly affected by the simulated Empty Street condition and were close to the No Simulator's. The two techniques did not significantly differ in term of situation awareness with our simulator. The two in-air techniques suffered substantially more from the simulator.

Though these results could only be safely obtained using a simulation, we believe they provide valuable insights sufficient to reliably recommend the use of the Trackpad for eyewear pointing by pedestrians (provided that it is implemented with an efficient transfer function 8 ).

Contrarily to previous assumptions [START_REF] Lumsden | A Paradigm Shift: Alternative Interaction Techniques for Use with Mobile & Wearable Devices[END_REF][START_REF] Mustonen | Visual task performance using a monocular see-through head-mounted display (HMD) while walking[END_REF][START_REF] Lucero | NotifEye: Using Interactive Glasses to Deal with Notifications While Walking in Public[END_REF][START_REF] Lauber | In-your-face, Yet Unseen?: Improving Head-stabilized Warnings to Reduce Reaction Time[END_REF] we were surprised that the use of eyewear did not improve situation awareness in our simulator; indeed, situation awareness with eyewear was worse than regular phone interaction with harder-to-use input techniques. User feedback was divided: five participants reported that it was easier to deal with divided attention using the smartglasses, while four others stated the opposite. One obvious caveat on this, however, is that people are highly familiar with current touchscreen interaction, and our participants' performance with eyewear conditions might improve with familiarization. In terms of pure input performance, as expected, Phone was superior.

Conclusion

This work contributed the first empirical study of eyewear pointing while mobile, taking into account environmental awareness and perceived social acceptability. In our street simulations or in a quiet building, participants were faster using a hand-held trackpad than with every investigated variant of in-air techniques. The trackpad was also perceived as the most socially acceptable technique.

Research on eyewear for pedestrians is still in its infancy. Our results indicate muted benefits regarding situation awareness. However, we remain confident that eyewear might expose situation awareness advantages, in particular for more passive tasks such as reading, and we plan to explore it as future work. Also, we would like to explore body-based techniques, like Finger-and Pocket Touch as they were deemed highly acceptable. But, there are also many other promising paths to investigate, as outlined in our related works.

We believe that our simulation-based method is promising, particularly given the ethical concerns associated with evaluation in-the-wild. While the generalizability of our simulation cannot be fully assessed, we argue it provides valuable insights on situation awareness. However, the generalizability of other aspects, such as path finding, might be easier to investigate in future work. Simulations will never be as externally valid as in-the-wild studies, but they require less resources and allows much greater control. Compared to other form of lab studies, we argue they do can provide opportune improvements of external validity.

Fig. 1 .

 1 Fig. 1. Illustrating the simulator: using a phone to point on eyewear while walking.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Pointing techniques explored during our experiment: Front Rotation (left), Relaxed Rotation (middle) and trackpad (right).

Fig. 4 .

 4 Fig. 4. Adaptation of the vanishing point. No adaptation (left) vs adaptation (right).

Fig. 5 .

 5 Fig. 5. Pointing task interface used during the experiment (black is transparent on the glasses). Each time a participant validates a target (currently the rightmost disk on the figure), a new one is highlighted until completion of the task. The superimposed arrow indicates the path to alternating targets following ISO 9241-9 standard procedure [53].

Fig. 7 .

 7 Fig. 7. Subjective Assessments of the techniques.

Fig. 8 .

 8 Fig. 8. Results of the ecological validity experiment. Error bars show .95 confidence intervals.

Table 2 .

 2 Parameters of the gain functions

	Technique	Gmin	Gmax	λ	V inf
	Trackpad	6.28 mm ⁄mm	18.84 mm ⁄mm	0.113 s ⁄mm	26.46 mm ⁄s
	Front Rot.	0.48 mm ⁄deg	2.9 mm ⁄deg	10.47 ms ⁄deg	143.24 deg ⁄s
	Relaxed Rot.	0.24 mm ⁄deg	3.38 mm ⁄deg	10.47 ms ⁄deg	137.51 deg ⁄s

  1,11 = 6.69, p < .05), technique (F 3,33 = 17.74, p < .0001), as well a environment × technique interaction (F 3,33 = 6.93, p < .001) on the Walking speed . Post-hoc comparison showed significant differences between two groups of techniques (p < .01, sec /block):

	Phone , 20.9	Trackpad 21.4	Front Rot. , 28.1	Relaxed Rot. 29.1

Our initial design used press-and-hold for clutching: the cursor was moving by default, but users could hold the screen to freeze it and reposition themselves. However, our pre-tests quickly revealed that it was counter-intuitive.

During initial tests, the trackpad shipped with the Epson Moverio glasses proved to be particularly cumbersome to use (in contrast with our transfer function).

Results

We ran two-way ANOVAs with two within-subjects factors (technique and environment) on Selection Time, Selection Error , Walking Speed , pedestrian collisions, and jaywalking. We applied Greenhouse-Geiser sphericity correction when needed, with adjusted p-values and degrees of freedom. We ran pairwise t-tests with Bonferroni correction applied to the p-values of post-hoc tests.

Selection Time As per ISO 9241-9 [START_REF] Soukoreff | Towards a Standard for Pointing Device Evaluation, Perspectives on 27 Years of Fitts[END_REF], Selection Time was measured as the time between two target selections (first excluded). We observed a significant main effect of technique on Selection Time (F 2.07,22.8 = 61.7, p < .0001). Post-hoc comparisons show significant differences between all pairs (all p < .0001), except Relaxed Rotation × Front Rotation (" " indicates significant differences): We found a significant difference between the No Simulator (M = 0.1 errors /trial) and Crowded Street (0.22) conditions (p < .001). Fig. 6 illustrates the results.

Walking Speed We draw comparisons of walking speed between techniques and between the two street conditions (Empty Street and Crowded Street). We measured the time it took each participant to walk through an entire block whilst