Gustave Monce
email: gustave.monce@labri.fr

Thomas Couturou
email: thomas.couturou@labri.fr

Yasmine Hamdaoui
email: yasmine.hamdaoui@labri.fr

Thomas Degueule
email: thomas.degueule@labri.fr

Jean-Rémy Falleri
email: falleri@labri.fr

Lightweight Syntactic API Usage Analysis with UCov

Keywords: Software and its engineering → Software libraries and repositories, Software maintenance tools, Software design engineering software library, API design, API usage, API documentation

come   L'archive ouverte pluridisciplinaire

INTRODUCTION

Application Programming Interfaces (APIs) govern the interactions between a library and client projects using it. In particular, they specify which elements of a library (e.g., types, methods, fields) can be accessed from the outside and the valid interactions with them. Designing an effective API is essential for library developers as it is the lens through which clients judge its usability and benefits and the main friction point when the library evolves [START_REF] Ochoa | Breaking bad? semantic versioning and impact of breaking changes in maven central: An external and differentiated replication study[END_REF][START_REF] Martin | What makes apis hard to learn? answers from developers[END_REF]. Despite its importance, defining the boundaries of an API is a challenging task, mainly due to the diverse mechanisms provided by programming languages and paradigms that either aim specifically at designing APIs or indirectly affect them as a side effect (e.g., visibilities, subclass restriction and sealing, name binding rules, overloading and overriding, polymorphism).

Given the complex interplay among those mechanisms, it is difficult for library maintainers to maintain an exhaustive and accurate mental model of their API and the interactions it enables. This complexity is compounded by the various styles libraries can adopt (e.g., frameworks and inversion of control, fluent interfaces) and the variety of ways that client code can interact with individual API elements. For instance, methods may either be invoked directly or overridden, and non-abstract classes may be instantiated, extended, referenced, etc. This impairs the maintainers' ability to fully understand how their APIs are-or could be!-utilized in the wild. As a result, libraries sometimes inadvertently allow some kinds of interactions that their developers did not intend. This led to best API design practices such as "minimize accessibility" or "forbid subclassing by default" that are widespread in different communities (e.g., in Java [START_REF] Bloch | Effective java[END_REF]). To minimize the likelihood of bad outcomes on the client side, library developers should restrict the possible interactions with their APIs to those that are intended and validated. Indeed, unintended interactions are a recipe for buggy client code and frustrated developers who may opt for an alternative library providing similar services with better support. However, there is no lightweight, effective way of extracting and maintaining a transparent, comprehensive API model, nor is there a way to validate the extent to which the interactions with an API are validated, documented, and representative of actual uses. Current approaches in the literature can successfully analyze the use of API symbols but do not account for the variety of possible interactions with an individual symbol [START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF][START_REF] Qiu | Understanding the API usage in Java[END_REF].

In this paper, we introduce a novel conceptual framework aimed at helping library maintainers better understand the boundaries of their APIs via the use of so-called syntactic usage models (SUMs). SUMs enable maintainers to reason about the interactions allowed by their APIs throughout their evolution. Based on the SUMs, we define syntactic usage footprints (SUFs), which measure the extent to which a given piece of code using the library (e.g., third-party clients, documentation samples, library tests) exercises its API and covers its possible uses. We show how different SUFs can be easily compared, enabling library maintainers to answer questions such as "do our tests validate the interactions found in third-party clients?" or "do the examples in our documentation align with in-the-wild uses of our API?" (Section 3).

We present UCov (Usage COVerage), an implementation of these models for the Java programming language that leverages static analysis to automatically infer them from Java source code (Section 4). We illustrate the benefits of UCov and the underlying models with an exploratory case study of three libraries implementing various styles of interaction: jsoup, a library for HTML manipulation; 1 Apache commons-cli, a library for implementing command-line interfaces;2 and spark, a simple web framework 3(Section 5). From our experiments with UCov, we lay out a research agenda for syntactic models and envision application scenarios in other areas (Section 6). [START_REF] Monce | Artifacts for "Lightweight Syntactic API Usage Analysis with UCov[END_REF] Overall, our syntactic models and UCov provide a novel means for library maintainers to understand the extent of possible interactions allowed by their APIs and to oversee their use in client code. By offering a robust conceptual framework for usage coverage and a first implementation for Java, we aim to empower library maintainers with the insights needed to improve their API's design, documentation, and tests, thereby reducing friction with client code and fostering long-term maintenance.

BACKGROUND AND MOTIVATION

Understanding how client code interacts with APIs is instrumental in prioritizing development and documentation efforts. Therefore, numerous studies have examined methods to extract usage data from client code to identify notable hotspots and coldspots, i.e., parts of the APIs that are over-or under-utilized [START_REF] Ashok | fine-grape: fine-grained api usage extractor-an approach and dataset to investigate api usage[END_REF][START_REF] Stylos | Improving API documentation using API usage information[END_REF][START_REF] Thummalapenta | Spotweb: Detecting framework hotspots and coldspots via mining open source code on the web[END_REF]. In a recent study, Qiu et al. delved into the usage of Java's standard library and third-party libraries across a corpus of over 5,000 projects, encompassing 150M+ lines of code [START_REF] Qiu | Understanding the API usage in Java[END_REF]. Various tools and studies adopt distinct methodologies for gathering usage data, whether from bytecode [START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF], source code and resolved ASTs [START_REF] De Roover | Multi-dimensional Exploration of API Usage[END_REF][START_REF] Lämmel | Large-scale, AST-based APIusage analysis of open-source Java projects[END_REF][START_REF] Qiu | Understanding the API usage in Java[END_REF], or other sources [START_REF] Stylos | Improving API documentation using API usage information[END_REF]. A unifying theme across these works and their underlying models of API usage is their focus on determining whether a specific API symbol is accessed in client code, rather than exploring how it is used. This emphasis stems from the primary objectives of these studies: assess the frequency, popularity, and coverage of API symbols in client code, rather than exploring the diverse uses of these symbols. Indeed, the way a symbol is used has no influence on its popularity. Extending, instantiating, or simply referencing a class all contribute equally to its popularity.

We argue that this approach to modeling library usage is too coarse-grained to allow library maintainers to understand fully the implications of the design of their API. For instance, let us consider a library designer providing a simple public class C and a client instantiating it with C c = new C(). Existing usage models would indicate that there is one exported symbol (C) and that this symbol is used in client code. Based on this information, library maintainers could conclude that the design of their API is satisfactory. However, this declaration does not prevent clients from subclassing the provided class with class ClientC extends C. The aforementioned models do not provide any means to distinguish this particular use from other uses in client code. This is a missed opportunity, as library maintainers could have reconsidered the API early on to prevent subclassing, disallowing clients from extending the class, and thus freeing themselves from supporting this scenario. The approaches mentioned above are oblivious to this distinction, while our work emphasizes it.

There is a wide variety of literature addressing the problem of mining and recommending API usage patterns and protocols [START_REF] Wang | Mining succinct and high-coverage api usage patterns from source code[END_REF][START_REF] Zhong | Mapo: Mining and recommending api usage patterns[END_REF]. These studies typically analyze library code and client code to infer automata and probabilistic graphs that represent legal sequences of API invocations and check whether client code complies with them. While these approaches go beyond our objectives and consider some kind of semantic relation between API elements, they are unable to differentiate between the different interactions allowed for a specific element. In contrast to previous works, our approach emphasizes the diverse ways a particular symbol may be utilized in client code, leveraging the syntactic usage models introduced in the next section.

SYNTACTIC API USAGE

Our analysis of API usage is built upon two distinct models: syntactic usage models, which are extracted from library code and represent the legal (possible) uses of an API (Section 3.1), and syntactic usage footprints, which are extracted from client code and represent the actual uses of an API (Section 3.2). From these two models, we derive various metrics, including a dedicated API coverage metric (Section 3.3). These models are formulated independently of any specific programming language and can be adapted for diverse analysis scenarios. However, we reference Java syntax and semantics throughout this section for illustrative purposes.

Syntactic Usage Model (SUM)

We consider that a library defines a set of symbols S (i.e., named entities), each of a particular kind and with a declaration that specifies its properties. For instance, the Java standard library defines the symbol public class ArrayList<E> of kind Class, the symbol public final static PrintStream out of kind Field, and the symbol public void println() of kind Method. These symbols can be exported, allowing client code to access them. Together, the exported symbols of a library form its API. Given a function exported : S → {⊤, ⊥} indicating whether a given symbol is exported (⊤) or not (⊥), derived from the language's semantics, the API of a library is the set A ⊆ S such that A = {𝑠 ∈ S : exported (𝑠) = ⊤}. In Java, for instance, visibilities and modules are the primary mechanisms for controlling symbol exports. As an ...

}

Depending on its properties, the same kind of symbol may be used in various ways in client code. Therefore, syntactic usage models employ a function uses : A → P (U), with U the set of all possible uses, that associates a given symbol with the set of its legal uses. For instance, following Java's semantics, a public class A can be Instantiated, Referenced, or Extended in client code. In contrast, a public abstract class B can only be Referenced or Extended, and a public class E extends Exception can even be Thrown. There is no universally correct definition of the uses function: it is the SUM designer's responsibility to specify the uses of interest that they would like to associate with each individual symbol. For instance, the use Referenced above may be refined for each kind of reference to a type (e.g., as a parameter, variable, or return type) if subsequent analyses require it. Finally, the syntactic usage model M of a library is the union of the possible uses for every exported symbol in its API M = 𝑠 ∈ A 𝑢𝑠𝑒𝑠 (𝑠). In the example of Listing 1, the SUM holds two exported symbols and five possible uses of the API, as shown in Table 2. Syntactic usage footprints materialize the diversity of ways a given API can be used in client code. For instance, Listing 2 depicts a classical interaction with the ArrayList API through instantiation and invocations. On the other hand, Listing 3 depicts a typical framework-like interaction with the same API through extension and overriding. This latter style is prevalent in frameworks such as web servers or batch processing frameworks (e.g., Spring, Hadoop, Spark) that heavily employ inversion of control and the Hollywood principle ("Don't call us, we'll call you"). As Table 1 shows, these two snippets yield two disjoint and complementary SUFs. Importantly, when looking at multiple clients, their SUFs can trivially be joined through set union and compared through set difference.

Usage Coverage and Metrics

The SUM gives the universe of legal uses to cover for a given API and the SUF pinpoints those that are actually covered in client code. The set of API symbols that are covered in a SUF is denoted C A = {𝑠 : ⟨𝑠, 𝑢, 𝑙⟩ ∈ F } and the set of covered API uses is denoted C M = {𝑢 : ⟨𝑠, 𝑢, 𝑙⟩ ∈ F }. Then, the API symbol coverage score of the SUF with respect to the API is

| C A | | A | and its API use coverage score is | C M | | M | .
Following these definitions, Listing 2 covers 50% of Listing 1's API uses and 100% of its symbols, so does Listing 3. Together, they cover 100% of the API uses. One can identify the exported symbols that are not covered in client code by exploring A\C A and the legal uses that are not realized by exploring M\C M . Interestingly, one may reproduce the popularity metrics used to identify hotspots and coldspots in the literature [START_REF] Qiu | Understanding the API usage in Java[END_REF][START_REF] Ashok | fine-grape: fine-grained api usage extractor-an approach and dataset to investigate api usage[END_REF] by counting the occurrences of ⟨𝑠, 𝑢, 𝑙⟩ ∈ F for a given symbol of interest 𝑠.

We distinguish between three levels of coverage for each individual API symbol. An API symbol 𝑠 ∈ A is fully covered if all of its uses are present in the SUF, i.e., if ∀𝑢 ∈ 𝑢𝑠𝑒𝑠 (𝑠), ∃⟨𝑠, 𝑢, 𝑙⟩ ∈ F . Conversely, an API symbol is not covered if none of its uses are present in the SUF, and it is partially covered if only some of its uses are present in the SUF.

UCOV: JAVA SYNTACTIC USAGE ANALYSIS

UCov is our proof-of-concept implementation of syntactic API usage analysis for Java libraries. 4 It parses and analyzes the source code of Java libraries to produce syntactic usage models and the source code of Java clients to produce syntactic usage footprints and their coverage. When implementing such a tool, one must decide on the symbols of interest to include in the API and the types of uses to represent, based on language semantics and analysis goals, as highlighted in Section 3. In this section, we discuss some of the choices we made while implementing UCov to support our case studies and the overall architecture of the tool, depicted in Figure 1.

Exported Symbols

SUM models employ a simple 𝑒𝑥𝑝𝑜𝑟𝑡𝑒𝑑 : S → {⊤, ⊥} function that indicates whether a given symbol 𝑠 is exported. From the library's code, UCov first extracts the list of all API symbols (namely, for Java, types, methods, and fields) uniquely identified by their fully qualified name (and their signature in the case of methods). Then, it distinguishes those that can be accessed from the outside, in client code, and those that cannot. Following the rules described in the Java 17 Language Specification [START_REF] Gosling | The java language specification: Java se 17 edition[END_REF], the following symbols can be accessed from client code: Public symbols: (transitively) public types, as well as public methods and fields within public types, can be accessed from anywhere without restriction; Protected symbols: protected fields and methods within effectively extensible public types can be accessed through subtyping or by code located in the same package. An effectively extensible type is one that is not final nor sealed and that, in the case of classes, possesses a public or protected constructor the subclass can access; Package-private symbols: package-private symbols (Java's default visibility in the absence of a visibility modifier) can be accessed by code located in the same package.

The only legal visibilities for a top-level type declaration are public and package-private, so API designers can only use the latter when they want to hide a type declaration from the outside. UCov's implementation of the exported function thus considers the public and protected symbols and discards the package-private ones, as client code intentionally using a package of the same name as the library's to access its package-private symbols is a pathological case that most likely breaches the API's intent.

Supported API Symbol Uses

Once the exported symbols of a library are identified, UCov maps each of these to a set of legal uses, based on its declaration specifics, 4 UCov is available at https://github.com/alien-tools/ucov. A snapshot of UCov alongside all artifacts discussed in this paper is available on Zenodo [START_REF] Monce | Artifacts for "Lightweight Syntactic API Usage Analysis with UCov[END_REF]. thereby implementing the uses function. Table 3 gives an excerpt of the uses we consider for each kind of symbol.

In a nutshell, UCov looks for every possible use of type references (as parameter types, return types, exceptions, etc.) and categorizes them all as Type reference uses. Classes can be instantiated if they provide a public (possibly default) constructor and are not abstract. They can be inherited unless declared final. Interfaces can be implemented by client classes or extended by other interfaces. Methods can be invoked (statically or dynamically) and overridden unless declared final. When polymorphism comes into play in client code and the static analysis cannot determine the dynamically invoked method beyond the static type of its receiver object, UCov registers an invocation to the corresponding method in the static type of the receiver object, as well as invocations to all super-methods with the same signature in its hierarchy. As a result, abstract methods in abstract classes and interfaces are considered covered whenever a more concrete implementation is invoked in client code. For instance, an invocation of ArrayList.size() in client code would cover List.size(). Finally, UCov distinguishes between read and write accesses to an exported field.

Note that these choices aim to reflect customary uses of Java APIs and are, to some extent, arbitrary. Another implementation may, for instance, distinguish between various references to a type or unify read and write accesses to a field under the same kind of use. These choices will vary depending on the task at hand and analysis objectives.

Implementation

UCov is implemented in Java and uses the Spoon framework [START_REF] Pawlak | Spoon: A Library for Implementing Analyses and Transformations of Java Source Code[END_REF] to parse Java code and create visitors that implement the necessary static analyses.

When supplied with library code, UCov uses Spoon to build a resolved AST and applies a dedicated visitor on types, methods, and fields to list the exported API symbols, thereby implementing the exported function (upper part of Figure 1). Then, it maps each of these symbols to the corresponding uses (×) according to the rules in Section 4.2, forming the final syntactic usage model.

When supplied with client code (library tests, library samples, code of third-party clients, or any arbitrary snippet using the library), UCov builds its resolved AST and visits it to identify every

Field write

Point.x = 2 java.awt.Point.x node that references, in one way or another, symbols of the API. For example, it analyzes method invocations, field accesses, and type references to determine if they correspond to any of the uses identified in the syntactic usage model. Table 3 lists an excerpt of the elements in client code that the static analyzer searches for and the information it collects for each. The resulting set of actual uses in client code constitutes its syntactic usage footprint.

EXPLORATORY CASE STUDY

In this section, we evaluate the capability of syntactic usage models and footprints, as well as their implementation for Java libraries in UCov, to produce meaningful information that assists maintainers in understanding the boundaries of their libraries and their usage in the wild. To this end, we follow an exploratory case study approach [START_REF] Ralph | Empirical standards for software engineering research[END_REF] to analyze three popular Java libraries that exhibit diverse interaction styles: jsoup, a library for HTML manipulation; Apache commons-cli, a library for implementing commandline interfaces; and spark, a simple web framework. We introduce our subject libraries in Section 5.1 and our methodology in Section 5.2, and then detail our results in Section 5.3. We direct the reader to our reproduction package to access the data, scripts, and analyses conducted in this section [START_REF] Monce | Artifacts for "Lightweight Syntactic API Usage Analysis with UCov[END_REF].

Subject Libraries

The implementation of syntactic usage models and footprints in UCov aims to explore the boundaries of APIs and their actual uses in client code. Every API is unique, and it is neither possible nor desirable to study all of them. Instead, we establish a set of criteria to select subject libraries that are diverse and representative of mature Java libraries.

Interaction styles. Based on our modeling of syntactic usage, we hypothesize that different interaction styles favor different kinds of uses in client code and yield different usage profiles. To explore the diversity of uses, we aim for libraries that expose diverse styles. We categorize interaction styles into three types that are widespread in practice: classical, framework, and fluent. Below, we provide an example of each and explain their significance in the context of API usage. Naturally, these styles are not mutually exclusive and a library may offer different styles to interact with the same features.

In the classical style, APIs expose a set of public classes that are then instantiated in client code to invoke their methods and access their services. Client code retains control of the execution flow. The first snippet of Listing 4 depicts a typical classical use of commonscli where a parser and options are instantiated and configured using their methods. The classical style is the most common and is especially popular in libraries that offer a collection of utilities (e.g., Google's Guava and Apache's Commons).

In the framework style, APIs implement inversion of control and the Hollywood principle to allow client code to extend and specialize types of the API (usually interfaces and abstract classes) by providing implementation code. The library retains control over the execution flow and only hands it to client code when needed. This style is prevalent in frameworks (e.g., Spring, Hadoop), hence its name. The second snippet of Listing 4 shows a typical frameworklike interaction with spark, where users configure routes for their application by passing lambda expressions that implement the functional interface Route and its unique method handle(Request, Response) which gives the implementation of endpoints.

In the fluent style, APIs heavily employ programming tricks such as method chaining and cascading, static methods, the Builder design pattern, and static imports to mimic the look and feel of a domain-specific language [START_REF] Fowler | Fluent interface[END_REF]. The third snippet of Listing 4 shows .data("query", "Java").userAgent("Mozilla") .cookie("auth", "token").timeout(3000).post(); a typical fluent interaction with jsoup that shows how the Builder pattern is used to configure a jsoup connection and load a document from a remote URL.

Client code. To make the analysis possible, we look for libraries that have sufficient amounts of client code available: third-party clients, tests, and documentation samples. This rules out immature libraries that lack sufficient documentation or clients, or that are not well-tested.

While we expect to find third-party clients and tests that can be parsed and analyzed with UCov, documentation and samples suffer from additional issues. Some libraries include their samples as proper compilable files in their source directory, while others include their samples in readme files (e.g., on GitHub) or dedicated documentation websites. These samples are typically simple snippets cleaned from the surrounding boilerplate code (imports, structure, type declarations, etc.) that often cannot be parsed on their own. When these cases arise, we create a Java file for each case, encapsulating the snippet within a main method and manually inserting the missing imports. For our subject libraries, with the help of the surrounding documentation, the missing imports are always unambiguous.

Selected libraries. To obtain high-quality libraries that meet these requirements, we start from the Duets dataset [START_REF] Durieux | Duets: A dataset of reproducible pairs of java library-clients[END_REF]. Duets contains 395 libraries and 2,874 clients extracted from GitHub that compile, can be executed, have passing test suites and a minimum of five stars. To identify libraries with documentation samples, we narrow our search to libraries with a readme file and an official documentation website. If these contain documentation samples, we collect them. Otherwise, we follow their hyperlinks and repeat the process. We also explore the source tree of each library to find samples stored alongside the library code. Finally, we handpick a set of three libraries that exhibit diverse interaction styles and have sufficient test cases, documentation samples, and third-party clients: jsoup (classical and fluent styles), commons-cli (classical and fluent styles), and spark (framework style). Table 4 provides some descriptive statistics for these libraries.

For these three libraries, we retrieve the following documentation samples:

• For commons-cli, its readme links to the official documentation, which includes a page showcasing some sample uses. 5This page contains partial code snippets without imports or structure. We manually reconstruct parseable Java files for these snippets. • For jsoup, we find a set of samples directly in its source code (src/main/java/org/jsoup/examples). Its readme file also leads to the official website and a cookbook that presents different sample snippets. 6 We reconstruct complete Java files with proper imports and structure for these samples. • For spark, its readme includes fully parseable sample code and links to the official documentation. 7 The documentation points to a list of user-authored tutorials, 8 which consist of parseable Java files, and a list of template projects, 9 which we also include.

Methodology

Our methodology aims to explore the interactions offered by various APIs and their actual uses in third-party clients, tests, and documentation samples. Since the number of clients per library stored in Duets is relatively low (31 for jsoup, 202 for commons-cli, and 8 for spark), and as we do not require clients to have passing test suites, we collect new clients for each of the libraries using GitHub's dependency graph. 10 We compile a list of every repository in the dependency graph that declares a dependency towards one to analyze all of these, so we apply the standard Cochran formula with a confidence level of 𝑐 = 95%, an error margin of 𝑒 = 5%, and a conservative proportion of 𝑝 = 0.5. We obtain sample sizes of 372, 372, and 373, which we draw at random from the corresponding client sets. These sets contain the third-party clients, which we analyze in the remainder of this section. We use UCov to construct the SUM of each library. Following the specification in Section 4, UCov extracts the list of all exported API symbols and maps them to their corresponding legal uses. The resulting models materialize the universe of interactions enabled by the libraries (Table 5). For each library, we classify client code as either tests, documentation samples, or third-party clients, and we run UCov on these three corpora to produce their SUF. Since SUFs can be easily joined through set union, we also construct a merged SUF for each library representing every use from all client code, denoted All in Table 5.

Analysis and Results

This section discusses the results obtained with UCov for the three subject libraries and their client code.

Syntactic usage models and footprints. Table 5 presents simple statistics for the SUMs and SUFs extracted from our three subject libraries. jsoup exposes the most extensive API, with 1,138 symbols and 2,941 legal uses in its SUM, followed by spark and commonscli. This is consistent with their overall size in lines of code (Table 4). The three libraries exhibit a similar ratio of exported API symbols to legal uses, averaging around 2.5 legal uses per symbol.

The coverage scores for uses are consistently lower than for API symbols. This is expected, as covering a use implies covering the corresponding symbol. However, the sizeable difference in coverage indicates that many of the interactions legally allowed in the APIs are not realized in either tests, documentation samples, or thirdparty clients, even when the corresponding symbols are known and used. Although the relatively low coverage of API symbols in client code has been extensively studied in the literature (e.g. [START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF][START_REF] Qiu | Understanding the API usage in Java[END_REF]), this suggests that symbol coverage does not fully reflect the extent of interactions permitted by APIs.

Across all libraries, tests achieve the highest coverage scores for symbols and uses, followed by third-party clients and samples. This suggests that the tests cover a sizeable subset of the APIs. According to Hyrum's law, "with a sufficient number of users of an API [. . .] all observable behaviors [. . .] will be depended on by somebody" [START_REF] Wright | Hyrum's law[END_REF], suggesting that third-party clients might eventually achieve the highest coverage score, assuming a sufficient number of clients. This has already been empirically verified for the libraries hosted in Maven Central [START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF]. Documentation samples are sparse, so naturally they cover much less than tests and third-party clients. spark stands out with a much better coverage score of its API by samples, thanks to the rich documentation on its official website.

Overall, commons-cli is the most focused library, with fewer exported symbols and possible uses, which results in better coverage of its symbols and uses in client code. Its API also exhibits repetitive patterns: client code often instantiates numerous command-line options and configures them similarly, resulting in frequent and uniform usage of identical symbols (averaging 216 instances per use, compared to 29 for jsoup and 50 for spark).

Library profiles. Figure 2a depicts the usage profiles of our three libraries. The profile of a library is the distribution of the legal uses in their SUMs, such that the proportion of each kind of use adds up to 1. It highlights which kinds of uses are allowed and which are the most frequent. The profiles are mostly similar, with some specificities for each library. In all cases, (virtual) method invocation and method overriding dominate the frequencies. This is expected as methods are by far the most frequent kind of symbol in the three libraries, with fewer types and fields exposed. commons-cli and jsoup share a very similar profile. Indeed, they both implement the classical and fluent styles in their interactions. Interestingly, spark exposes a larger number of methods that can be statically invoked and interfaces that can be extended and overridden in client code. These correspond to the primary interfaces and methods used in client code to configure routes (get(), post(), path(), etc.), as shown in Listing 4. This is consistent with its framework-like interaction style. Among the three libraries, method overriding is the least covered type of use by a large margin. This suggests that the APIs offer possibilities for extension and specialization that are not yet utilized in client code or, more likely, that many of the API methods could be closed for extension using the final keyword at the level of the method or its containing type. Until version 17 (2021) and the introduction of sealed classes, Java lacked the ability to restrict extension and overriding to a predetermined set of types. Without this possibility, library maintainers had to open types for extension and specialization to everyone, even when their intent was to allow extension and specialization in library code only.

Usage analysis.

Here again, commons-cli and jsoup expose a similar profile. However, jsoup's clients rely more on its fluent style, with fewer class instantiations and more frequent use of static invocations and builders to create objects. We also observe that although commonscli and jsoup expose some fields that can be read and written, as well as some types that can be extended or implemented, clients rarely use these interactions. The only uses of these interfaces in client code are through type references.

spark, on the other hand, exhibits a different profile with many static invocations, method overrides, and interface implementations. Indeed, as advocated in its documentation, the preferred method of declaring routes in spark is to pass a lambda expression that implements a single method in the Route interface, resulting in each route declaration involving one static invocation, one interface implementation, and one method override.

Digging deeper into the most popular uses for each library, we find that the SUFs accurately reflect the expected uses of each library, as documented in their samples. For commons-cli, the top three most popular interactions involve referencing and instantiating the symbols Option and Options, and invoking the method addOption, which developers use to build their command-line interface. For jsoup, the top three most popular interactions involve referencing the symbols Document, Element, and invoking the method Element.select(String), which developers use to navigate HTML documents. For spark, the top three most popular interactions involve implementing the interface Route and overriding its method Route.handle(), which developers use to declare routes, as well as referencing the types Request and Response, which are passed to and from the routes. We refer the reader to the reproduction package for a comprehensive list of the most and least popular API symbols and interactions.

Interestingly, we observe that jsoup exposes parts of its internal API publicly (org.jsoup.internal), most likely for technical reasons: isolating types in a package forces them to be public to allow other packages of the library to access them. While the maintainers take great care to discourage clients from using these APIs using source code comments ("Jsoup internal use only, please don't depend on this API."), we find several uses in the code of third-party clients. Perhaps more surprisingly, jsoup's official documentation also uses these internal APIs in one of its samples. 11 UCov allows library maintainers to identify these problematic cases at a glance.

Comparative analysis of third-party clients, tests, and samples. The SUF profiles of third-party clients, tests, and samples (not shown here for conciseness but available in the reproduction package) are very similar for each library. However, when looking at individual symbols and uses, we observe that each has its specificities. Figure 3 depicts the individual contributions to unique uses of third-party clients, tests, and samples, represented as UpSet plots to visualize the size of their intersections. This visualization enables library maintainers to immediately identify gaps in the coverage of client code by their tests and documentation.

The official samples of commons-cli 12 are particularly sparse compared to jsoup and spark: all the uses they document are also found in tests and third-party code. Conversely, we identify 176 uses that are common to tests and third-party clients but not documented (23% of legal uses). These include the deprecated OptionBuilder type, which is unsurprisingly not documented, but also symbols that are popular in client code such as org.apache.commons.cli. Option.isRequired(). As the API of commons-cli is well-focused, there is still a significant amount of uses that are common to all three. However, we observe that 52 uses found in third-party clients [START_REF] Lex | Upset: Visualization of intersecting sets[END_REF] depicting the common and unique uses between third-party clients, tests, and samples for jsoup, commons-cli, and spark are not exercised: for instance, the method org.apache.commons. cli.Option.setValueSeparator(char) is neither tested nor documented.

The samples of jsoup are more varied, but there is still a significant amount of uses that are undocumented. Among these 338 uses, we identify some popular interactions such as invoking the org.jsoup.nodes.Node.hasAttr(java.lang.String) method. Its tests are comprehensive, only a few uses are exclusive to thirdparty clients. Among these, we find clients extending and overriding the org.jsoup.Connection type to implement their own logic and innocuous cases such as invocations to the many toString() methods that are (predictably) untested and undocumented. Interestingly, some of the uses we find in third-party clients are also documented in its samples, but absent from the tests. This is, for instance, the case for the internal APIs of jsoup discussed previously.

Despite the extensive samples found in its readme and tutorials, spark still presents 39 uses that are common to clients and tests but undocumented. Even more surprisingly, we find some uses that are documented but neither tested nor used in third-party clients, such as invocations of the spark.Request.bodyAsBytes() method. In contrast with commons-cli and jsoup, we find some uses that are documented, tested, but for which we could not find any instance in client code: we did not find any static invocation of the method spark.Spark.modelAndView(Object, String) which is the documented way of instantiating a ModelAndView object. Similarly to jsoup, we find 27 API interactions that are documented in samples and used by third-party clients but remain untested. These include setting the listening port of the web server (spark. Spark.port(int)) and examining the underlying raw Java requests beneath spark's Request objects (spark.Request.raw()).

A significant part of the value of syntactic models resides in the intersection of the uses of different kinds of client code. UCov allows library maintainers to identify undocumented or untested interactions in their APIs and take appropriate action. Naturally, it is not practical to expect that all API interactions are covered by the library's tests or samples. Most developers use IDEs to discover features, and the associated JavaDoc is often sufficient. Besides, trivial APIs such as the toString() methods mentioned above may not necessarily require documentation. Nevertheless, we believe that UCov can help maintainers prioritize their documentation and testing efforts by focusing on the interactions that are commonly used in the code of third-party clients, and identify blind spots in their design.

DISCUSSION

Syntactic usage models and syntactic usage footprints offer a lightweight and intuitive way to analyze the interactions allowed by an API and their coverage in client code. While our case study primarily explores their ability to accurately represent different library profiles and identify unused, undocumented, and untested interactions, they can inform and benefit other scenarios.

Support for API design in programming languages. SUM models represent the interactions permitted by an API. When using them, it becomes clear that the capabilities provided by Java for designing APIs are limited. Library developers are often required to expose numerous legal interactions to client code for purely technical reasons that have little to do with API design. For example, although library developers can prevent extension and overriding for everyone by using the final modifier, they have only recently gained the ability to open extension to a predetermined set of implementers using sealed classes. On the other hand, it is not possible to fine-tune the interactions allowed in client code, such as allowing method overriding but not method invocation, which can violate the principles of inversion of control in framework-like libraries. Similarly, there are no mechanisms to restrict access to certain types, fields, and methods beyond the basic scoping mechanisms provided by visibilities. As a result, developers must sometimes declare types as package-private or even public for technical reasons and rely on code comments and naming conventions such as internal packages and @Internal annotations to warn their users. SUM models make all these issues explicit and could facilitate reflection on language design when adapted to other programming languages.

Compatibility and breaking changes. When libraries evolve, they sometimes introduce breaking changes that affect clients and force them to adapt their code. How breaking changes impact client code depends on the interactions with the broken API. For example, a class that evolves into an abstract class does not break client code that references it but does break client code that attempts to instantiate it. Simply knowing which API symbol the client code uses is insufficient to assess the impact of breaking changes. Some approaches attempt to measure the impact of potential breaking changes by analyzing client code. While some only rely on import declarations to determine if client code may be impacted [START_REF] Xavier | Historical and impact analysis of api breaking changes: A large-scale study[END_REF], more recent approaches gather usage information to improve the accuracy of impact detection [START_REF] Ochoa | Breaking bad? semantic versioning and impact of breaking changes in maven central: An external and differentiated replication study[END_REF][START_REF] Ochoa | Breakbot: Analyzing the impact of breaking changes to assist library evolution[END_REF]. SUF models provide accurate information regarding the uses made in client code and we believe they could improve the accuracy of these tools.

Another interesting direction is to utilize SUM models to automatically generate synthetic client code that thoroughly exercises every possible interaction with the API. This code would exhibit a complete footprint of the API and could be recompiled whenever the library is updated, allowing the compiler to automatically check if any of the interactions break, signaling a breaking change. Using the compiler as ground truth for breaking change detection would address the accuracy issues of existing detection tools [START_REF] Jezek | Api evolution and compatibility: A data corpus and tool evaluation[END_REF] and automatically cope with the evolution of programming language specifications and their implementation in compilers and virtual machines.

API evolution. Predicting the consequences of API changes can be challenging. When an API evolves, it can not only introduce breaking changes but also alter the way clients interact with it, even when the changes are backward-compatible. Because SUM models can be efficiently computed (e.g., during code review or continuous integration), maintainers can reason about the impact of their changes and refactorings by exploring the differences between pre-and post-change SUMs. This could allow maintainers to easily assess the impact of external contributions on the interactions permitted by their API and determine whether to incorporate the changes.

RELATED WORK

In this section, we discuss related work on the analysis and applications of API usage. Specifically, we review the tools and studies that analyze usage at the symbol level, the extensive literature on usage patterns and protocols, and their applications to API evolution and breaking changes analysis. API symbols analysis. Traditionally, the analysis of API usage has primarily been conducted at the level of individual symbols. Many studies analyze how API symbols are used to identify notable hotspots and coldspots, i.e., parts of the APIs that are over-or under-utilized [START_REF] Ashok | fine-grape: fine-grained api usage extractor-an approach and dataset to investigate api usage[END_REF][START_REF] Stylos | Improving API documentation using API usage information[END_REF][START_REF] Thummalapenta | Spotweb: Detecting framework hotspots and coldspots via mining open source code on the web[END_REF]. A recent study by Qiu et al. delves into the usage of Java's standard library and third-party libraries across a corpus of over 5,000 projects, encompassing 150M+ lines of code, and finds that usage follows a Zipf distribution [START_REF] Qiu | Understanding the API usage in Java[END_REF]. Another study by Harrand et al. examines 2.2M dependencies on Maven Central and confirms the observations of Hyrum's law: a small subset of the API attracts most usages in client code [START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF]. As illustrated by our exploratory case study, the coverage of uses is much smaller than that of symbols. Various tools and studies employ different methodologies for collecting usage data, whether from bytecode [START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF], source code and resolved ASTs [START_REF] De Roover | Multi-dimensional Exploration of API Usage[END_REF][START_REF] Lämmel | Large-scale, AST-based APIusage analysis of open-source Java projects[END_REF][START_REF] Qiu | Understanding the API usage in Java[END_REF], or other sources [START_REF] Stylos | Improving API documentation using API usage information[END_REF]. A unifying theme across these works and their underlying models of API usage is their focus on determining whether a specific API symbol is accessed in client code, rather than exploring how it is used. Syntactic models and UCov go beyond this level of granularity by distinguishing the various interactions allowed by individual symbols. As our case study shows, the coverage of uses in client code is much smaller than that of symbols.

Usage patterns and protocols. A variety of work in the literature addresses the problem of mining and recommending unordered or sequential API usage patterns and protocols to users [START_REF] Raymond | Synthesizing api usage examples[END_REF][START_REF] Thanh Nguyen | FOCUS: a recommender system for mining API function calls and usage patterns[END_REF][START_REF] Robillard | Automated API Property Inference Techniques[END_REF][START_REF] Wang | Mining succinct and high-coverage api usage patterns from source code[END_REF][START_REF] Zhong | Mapo: Mining and recommending api usage patterns[END_REF]. These studies typically analyze library code and client code to infer automata and probabilistic graphs that represent legal sequences of API invocations and check whether client code complies with them. Some even gather data from Q&A websites to infer and recommend usage patterns [START_REF] Uddin | Mining api usage scenarios from stack overflow[END_REF][START_REF] Zhang | Are code examples on an online q&a forum reliable? a study of api misuse on stack overflow[END_REF], while others empirically study these usages in the wild [START_REF] Zhong | An empirical study on API usages[END_REF]. These approaches consider sequences of API calls, often with temporal properties, and go beyond the goals of UCov. On the other hand, UCov still provides a more detailed view of interactions with API symbols, which could benefit these approaches. In our work, we leverage syntactic usage models and footprints to help developers understand which API uses are allowed and to help them improve their API design, eliminating unintended uses or documenting undocumented areas of their APIs. A promising area for future research is studying how syntactic usage models and footprints can contribute to extracting more fine-grained usage patterns and protocols.

API evolution breaking changes.

There is a wide range of literature on the nature and driving forces of API evolution [START_REF] Ketkar | Understanding type changes in java[END_REF][START_REF] Gaikovina Kula | An empirical study on the impact of refactoring activities on evolving client-used apis[END_REF][START_REF] Mcdonnell | An empirical study of api stability and adoption in the android ecosystem[END_REF], as well as on breaking changes and their impact [START_REF] Brito | Why and how java developers break apis[END_REF][START_REF] Ochoa | Breaking bad? semantic versioning and impact of breaking changes in maven central: An external and differentiated replication study[END_REF][START_REF] Xavier | Historical and impact analysis of api breaking changes: A large-scale study[END_REF]. Some studies focus on detecting breaking changes through API usage patterns [START_REF] Foo | Efficient static checking of library updates[END_REF][START_REF] Gao | APIfix: output-oriented program synthesis for combating breaking changes in libraries[END_REF][START_REF] Harrand | Api beauty is in the eye of the clients: 2.2 million maven dependencies reveal the spectrum of client-api usages[END_REF][START_REF] Lamothe | A Systematic Review of API Evolution Literature[END_REF][START_REF] Hoan | Graph-Based Mining of In-the-Wild, Fine-Grained, Semantic Code Change Patterns[END_REF][START_REF] Venturini | I Depended on You and You Broke Me: An Empirical Study of Manifesting Breaking Changes in Client Packages[END_REF][START_REF] Zaitsev | How libraries evolve: A survey of two industrial companies and an opensource community[END_REF][START_REF] Zhang | Has My Release Disobeyed Semantic Versioning? Static Detection Based on Semantic Differencing[END_REF]. A key enabler for these studies is the use of accurate models that can represent both the interactions allowed by an API and their uses in client code. The impact of a breaking change, for example, depends on both the symbol being used and the way it is used. We believe that the rich information provided by syntactic usage models and footprints can better inform breaking change detection tools [START_REF] Jezek | Api evolution and compatibility: A data corpus and tool evaluation[END_REF] and improve their accuracy in assessing the impact of breaking changes on client code. This information can also better inform studies that aim to understand API usage and evolution by refining the level of granularity at which APIs are considered [START_REF] Businge | Eclipse api usage: the good and the bad[END_REF][START_REF] Businge | How stable are eclipse application framework internal in[END_REF].

CONCLUSION

In this paper, we introduced syntactic usage models and footprints to support library maintainers in understanding the boundaries of their APIs and the interactions they allow. We presented an implementation of these models for the Java programming language in UCov, a static analysis tool that analyzes the source code of libraries to collect their exported symbols and allowed uses, and client code to collect fine-grained uses of the library.

Our exploratory case study of three popular Java libraries that exhibit diverse interaction styles (commons-cli, jsoup, and spark) showed that UCov provides valuable information regarding the usage profiles of libraries and their uses in documentation samples, tests, and third-party clients. Specifically, we showed how UCov can pinpoint undocumented and untested interactions, as well as misalignments between legal uses of the API and actual uses in client code.

For future work, we will explore the suitability of syntactic usage models and footprints to support graceful API evolution and the detection of breaking changes and their impact. We will also employ UCov for large-scale empirical analysis of the evolution of Java libraries and investigate the benefits of syntactic models in other programming languages and ecosystems.

Listing 2 :class

 2 Classical usage of the API of Listing 1 ArrayList<Integer> lst = new ArrayList<Integer>(); lst.add(42); lst.add(1337); Listing 3: Framework-like usage of the API of Listing 1 MyArrayList<E> extends ArrayList<E> { @Override public boolean add(E e) { ... } }

Figure 1 :

 1 Figure 1: UCov parses and analyses Java code to build SUM and SUF models. Each cross symbol (×) represents one possible use of an API symbol.

Listing 4 :

 4 Diverse API interaction styles exemplified using actual documentation samples from our subject libraries // Classical usage of commons-cli // https://commons.apache.org/proper/commons-cli/usage.html CommandLineParser parser = new DefaultParser(); Options options = new Options(); options.addOption("a", "all", false, "do not hide entries"); options.addOption("C", false, "list entries by columns"); try { CommandLine line = parser.parse(options, args); if (line.hasOption("block-size")) { ... } } catch (ParseException exp) { ... } // Framework-like usage of Spark // https://sparkjava.com/documentation#routes get("/", (request, response) → { ... }); post("/", (request, response) → { ... }); put("/", (request, response) → { ... }); delete("/", (request, response) → { ... }); options("/", (request, response) → { ... }); // Fluent-like usage of jsoup // https://jsoup.org/cookbook/input/load-document-from-url Document doc = Jsoup.connect("http://example.com")

  Figure 2b shows the usage profiles of the three libraries, defined as the distribution of actual uses in their SUFs (including third-party clients, tests, and samples). Comparing Figure 2a and Figure 2b reveals the discrepancies between what the APIs allow and what client code actually uses.

Figure 2 :

 2 Figure 2: Usage profiles of the three libraries commons-cli, jsoup, and spark

Figure 3 :

 3 Figure 3: UpSet plots [17] depicting the common and unique uses between third-party clients, tests, and samples for jsoup, commons-cli, and spark

Table 1 :

 1 Syntactic usage footprints for the API of Listing 1

	Client	Statement Symbol	Use
		ArrayList<Integer> lst	public class ArrayList<E>	Referenced
		new ArrayList<Integer>()	public class ArrayList<E>	Instantiated
	Listing 2	new ArrayList<Integer>()	public ArrayList<E>()	Invoked
		lst.add(42)	public boolean add(E e)	Invoked
		lst.add(1337)	public boolean add(E e)	Invoked
	Listing 3	class MyArrayList<E> extends ArrayList<E> @Override public boolean add(E e)	public class ArrayList<E> public boolean add(E e)	Extended Overridden
	illustrative example, consider the simplified excerpt of the JDK's
	java.util.ArrayList depicted in Listing 1. In this example, the
	set S holds three symbols: public class ArrayList<E>, public
	boolean add(E), and private int size while A holds the two
	first symbols only.
	Listing 1: A simplified excerpt

of java.util.ArrayList public class ArrayList<E> implements List<E> { public boolean add(E e) { ... } private int size;

Table 2 :

 2 Syntactic usage model of Listing 1

	Symbol Kind	Exported Uses
	public class ArrayList<E>	Class	✓	{Instantiated, Referenced, Extended }
	public boolean add(E e)	Method	✓	{Invoked, Overridden}
	private int size	Field	✗	∅
	3.2 Syntactic Usage Footprint (SUF)
	While the SUM holds the legal, possible uses of an API, the SUF
	materializes actual uses of the API in a given piece of client code.
	Formally, the footprint of a client on a SUM is the set of triples
	F = {⟨𝑠, 𝑢, 𝑙⟩ : 𝑠 ∈ A, 𝑢 ∈ uses(𝑠), 𝑙 ∈ L} where 𝑠 denotes the API

Table 3 :

 3 The kinds of uses considered in UCov, inspired and refined from the work of Qiu et al.[START_REF] Qiu | Understanding the API usage in Java[END_REF]

	Symbol Type Use	Example	Resolved API Symbol
	Type	Reference	String s	java.lang.String
			void f(Integer i)	java.lang.Integer
			Integer f()	java.lang.Integer
			void f()throws IOException	java.io.IOException
			catch (IOException e)	java.io.IOException
			List<String> l	java.lang.String
			etc.	etc.
	Class	Instantiation	new Integer(42)	java.lang.Integer
		Inheritance	class T extends Thread	java.lang.Thread
	Interface	Implementation	class R implements Runnable	java.lang.Runnable
			Runnable r = ()→{...}	java.lang.Runnable
		Extension	interface R extends Runnable	java.lang.Runnable
	Constructor	Invocation	new Integer(42)	java.lang.Integer(int)
	Method	Invocation	"a".length()	java.lang.String.length()
		Static invocation String.valueOf(42)	java.lang.String.valueOf(int)
		Overriding	@Override void run()	java.lang.Thread.run()
			Runnable r = ()→{...}	java.lang.Runnable.run()
	Field	Field read	Integer.MAX _ VALUE	java.lang.Integer.MAX _ VALUE

Table 4 :

 4 Descriptive statistics of the subject libraries, extracted from GitHub onOctober 30, 2023

		commons-cli	jsoup	spark
	Last release date	2021/10/29 2023/04/29 2020/10/08
	Version	1.5.0	1.16.1	2.9.3
	Since	2002	2010	2011
	Stars	309	10.4k	9.5k
	Commits	1,374	1,889	1,067
	Size (LoC)	6,307	27,817	11,298
	Contributors	46	97	100
	Clients	49k	131k	30k

Table 5 :

 5 Syntactic usage models extracted from commons-cli, jsoup, and spark, and syntactic usage footprints extracted from their third-party clients, tests, and samples. Symbols used are presented as % of API symbols and unique uses as % of legal uses.

				cli	jsoup	spark
	SUM		API symbols Legal uses	291 755	1,138 2,941	771 1,960
			All	205 (70%)	608 (53%) 301 (39%)
		Symbols	Clients	153 (53%)	282 (25%) 179 (23%)
		used	Tests	179 (62%)	586 (51%) 248 (32%)
			Samples	24 ( 8%)	47 ( 4%) 134 (17%)
			All	367 (49%) 1,028 (35%) 476 (24%)
	SUF	Unique	Clients	266 (35%)	456 (16%) 278 (14%)
		uses	Tests	315 (42%)	967 (33%) 400 (20%)
			Samples	38 (5%)	69 ( 2%) 211 (11%)
			All	79,284	29,979	23,882
		Total	Clients	74,453	15,246	20,827
		uses	Tests	4,690	14,511	1,605
			Samples	141	222	1450

of the libraries of interest. After discarding repositories that cannot be retrieved and forks, we obtain 11,159 clients for jsoup, 11,612 for commons-cli, and 12,530 for spark. Naturally, it is not necessary

https://jsoup.org/

https://commons.apache.org/proper/commons-cli/

https://sparkjava.com/

https://commons.apache.org/proper/commons-cli/usage.html

https://jsoup.org/cookbook/

http://sparkjava.com/documentation

https://sparkjava.com/tutorials

http://sparkjava.com/tutorials/application-structure, https://github.com/tipsy/ spark-basic-structure, https://github.com/perwendel/spark-template-engines

https://docs.github.com/en/code-security/supply-chain-security/ understanding-your-software-supply-chain/about-the-dependency-graph

https://github.com/jhy/jsoup/blob/84fd43766992401057b73f740acec4b82f1e3dd6/ src/main/java/org/jsoup/examples/HtmlToPlainText.java

https://commons.apache.org/proper/commons-cli/usage.html

ACKNOWLEDGMENTS

This work was partially funded by the French National Research Agency through grant ANR ALIEN (ANR-21-CE25-0007). We thank the anonymous reviewers for their precious comments.


Andreas Pastor

Pierre Lebreton

Toinon Vigier

Patrick Le Callet

Andréas Pastor

Comparison of conditions for omnidirectional video with spatial audio in

Keywords: subjective quality assessment, omnidirectional media format, 360 video, spatial audio, Ambisonic

INTRODUCTION

In recent years, 360°videos paired with spatial audio and experienced through head-mounted displays (HMDs) have earned significant engagement. These immersive experiences offer a new dimension to content consumption.

Subjective experiments exploring the quality of these media are essential to reveal the impact of these two modalities, i.e. audio and video, on the Quality of Experience (QoE). Methodologies for 360°video quality experiments have recently been standardized in ITU-T Rec. P.919 [START_REF]Subjective test methodologies for 360º video on head-mounted displays[END_REF], and previous works [START_REF] Singla | Quality assessment protocols for omnidirectional video quality evaluation[END_REF][START_REF] Elwardy | Comparison of acr methods for 360°video quality assessment subject to participants' experience with immersive media[END_REF][START_REF] Gutierrez | Subjective evaluation of visual quality and simulator sickness of short 360 videos: Itu-t rec. p. 919[END_REF][START_REF] Orduna | Methodology to assess quality, presence, empathy, attitude, and attention in 360-degree videos for immersive communications[END_REF] focusing exclusively on the video component. Spatial audio, e.g., ambisonics, brings a better sense of immersion in content by allowing users to look in a specific direction and hear audio that reflects their head movements and positions. Moreover, it enhances immersive experiences and user presence by fostering exploration and engagement within interactive contexts. Spatial audio is a guiding force for directing visual attention and aids in the cognitive analysis of scenes.

In this work, we reproduce part of the subjective quality assessment tests with trained assessors of [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF] on 360°video with Higher/4 th Order Ambisonic (HOA) audio. It is the first dataset with Mean Opinion Scores (MOS) to support perceptual quality research on immersive Audio-Visual content. In our paper, the focus is first to collect subjective data in other conditions for Audio-Visual experiment evaluation. Compared to the condition used in [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF], we use a consumer-grade setup where the 26-channel loudspeakers are replaced by Head Mounted Display (HMD) integrated headphones. We are getting closer to real use cases, and we can explore differences with a high-quality reference setup. Another difference is that we engage naive assessors for the task instead of the trained assessors. While trained assessors offer experience and expertise, they can be subject to developing a training bias and exploring the content in a non-ecological manner. Training used for [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF] can be found in [START_REF] Frans Fela | Assessor selection process for perceptual quality evaluation of 360 audiovisual content[END_REF] and includes training sequences evaluated during the subjective test.

Based on these data, we will demonstrate how rendering setup and subjective test methodologies affect the quality of subjective data in terms of discriminability [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF][START_REF] Nehmé | Comparison of subjective methods for quality assessment of 3D graphics in virtual reality[END_REF][START_REF] Pastor | Towards guidelines for subjective haptic quality assessment: A case study on quality assessment of compressed haptic signals[END_REF][START_REF] Pastor | Perceptual annotation of local distortions in videos: tools and datasets[END_REF][START_REF] Pastor | Comparison of subjective methodologies for local perception of distortion in videos and impact on objective metrics resolving power[END_REF]. This is an important aspect of codec development and standardization activities, as system performances must be evaluated and compared. To this aim, statistical testing that accounts for subjective data reliability is commonly used to compare differences between correlations, Root Mean Square Error (RMSE) [START_REF]Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models[END_REF], Spearman Rank Order correlation (SRCC) [START_REF] Naderi | Transformation of mean opinion scores to avoid misleading of ranked based statistical techniques[END_REF], Resolving Power [START_REF]Methodological framework for specifying accuracy and cross-calibration of video quality metrics[END_REF][START_REF]Method for specifying accuracy and crosscalibration of video quality metrics (VQM)[END_REF][START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF]. However, this is an afterthought, and the subjective data quality requirement and how much one shall invest in subjective testing to allow discriminating between different quality estimation models has only been weakly studied. To address this point, we will take commonly used audio and video quality models: VMAF [START_REF] Netflix | VMAF v0.6.1 Model[END_REF], ITU-T Rec. P.1203 [START_REF]Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport[END_REF][START_REF] Raake | A bitstream-based, scalable video-quality model for HTTP adaptive streaming: ITU-T P.1203.1[END_REF][START_REF] Robitza | HTTP Adaptive Streaming QoE Estimation with ITU-T Rec. P.1203 -Open Databases and Software[END_REF], ITU-T Rec. P.1204.3 [START_REF]Video quality assessment of streaming services over reliable transport for resolutions up to 4k with access to full bitstream information[END_REF][START_REF] Rao | Bitstream-based model standard for 4k/uhd: Itu-t p.1204.3 -model details, evaluation, analysis and open source implementation[END_REF], and ViSQOL [START_REF] Hines | ViSQOLAudio: An objective audio quality metric for low bitrate codecs[END_REF] and show how subjective data discriminability affects our ability to compare these metrics. Subjective data discriminability will be studied across different setups and test methodologies, giving a unique point of view on the relationship between experimental cost and conclusions that can be drawn from the data. Contributions are as follows:

• We compare a pristine and degraded audio setup regarding discriminability.

• We study the cost of an experiment and its relation with discriminability.

• We study metric resolving power as a function of subjective discriminability.

• We propose a method to apply the no-reference parameterbased audiovisual quality estimation model ITU-T P.1203 mode 0 to 360°video evaluation.

QOE CONDITIONS

This section describes the selected test material, the test environment for Omnidirectional Videos with Spatialized Audio playback, and the test methodologies selected for the subjective evaluation conditions. We decided to compare with the "Audiovisual" test from [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF], a test on 360°videos with HOA audio. We selected a subset of Sources (SRCs) and Process Video Sequences (PVS) of this dataset to compare in two new conditions.

Test material and environment

We choose from [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF] the following sequences: "CarWith-Chat", "DoorOpen", "HairDrying", "DogBarking", "River-Stream", "CrossRoadNight" and "HandballMatch". These sequences were selected for their video spatial and temporal information diversity and audio characteristics: source nature, location, and movement. All the test sequences are in YUV420 color space and Equirectangular Projection (ERP) format. The video decoding and display are operated via Unity. For Audio playback, we use Reaper, with the Sparta AmbiBIN plugin [START_REF] Mccormack | SPARTA & COM-PASS: Real-time implementations of linear and parametric spatial audio reproduction and processing methods[END_REF], to render a binaural audio stream from the HOA bitstreams.

In [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF], Visual stimuli were displayed in VR using a Samsung Odyssey+ Head Mounted Display (HMD) with a display resolution of 1440 × 1600 per eye, 110°horizontal Field of View (FOV). The HMD used during our experiments is the HTC Vive Pro eye, with the same 110°FOV and per-eye display resolution. In contrast with the original experiment, which used a 26-channel setup of 8040A Genelec loudspeakers, the audio is playback in the built-in HMD headphones. This difference will affect the perception of audio stimuli, as we will see in the analysis. But put us closer to a consumergrade experience. During the experiment, the subjects were seated on a swivel chair to allow them to turn freely. The start position of each 360°video was reset to the ERP center at the start of each viewing, irrespective of final world positions. It ensures that all observers start at the same position.

Subjective testing methodologies

In [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF], authors collected data using trained assessors. Their training procedure includes sequences evaluated during the subjective test part. More information about the assessors' training procedure can be found in [START_REF] Frans Fela | Assessor selection process for perceptual quality evaluation of 360 audiovisual content[END_REF]. In this first context, named "Reference condition", the subjective methodology was a modified version of Multiple Stimuli with Hidden Reference and Anchor (MUSHRA) to generalize from the SAMVIQ methodology [START_REF] Kozamernik | SAMVIQ-A new EBU methodology for video quality evaluations in multimedia[END_REF] for video, and MUSHRA [START_REF]Method for the subjective assessment of intermediate quality levels of coding systems[END_REF] for intermediate audio quality evaluation. In this paper, we want to reproduce with naive assessors the results obtained with trained assessors. Past works have focused on subjective methods comparison for mobile video viewing [START_REF] Tominaga | Performance comparisons of subjective quality assessment methods for mobile video[END_REF], 2D/3D videos [START_REF] Kawano | Performance comparison of subjective assessment methods for 3d video quality[END_REF], or 3D Graphics in Virtual Reality [START_REF] Nehmé | Comparison of subjective methods for quality assessment of 3D graphics in virtual reality[END_REF], and show that Degradation Category Rating (DCR) and Absolute Category Rating with Hidden Reference (ACR-HR) outperform SAMVIQ in the context of naive observers as it is longer and more complex to evaluate quality with it. These two methodologies are specified in ITU-T Rec. P.910 [START_REF]Subjective video quality assessment methods for multimedia applications[END_REF].

We design two contexts where naive assessors experience and rate audiovisual quality from HMD and build-in headphones: in the first context, an ACR-HR scale is used, namely "ACR-HR condition", and in the second context, a DCR scale, namely "DCR condition". For both conditions, we include a calibration phase before the start of the test to familiarize participants with the rating task and the degradation. We choose 3 PVS from "ChamberMusic" SRC for high, middle, and low audiovisual qualities.

In the ACR-HR condition, assessors are asked to rate the absolute quality of the PVS on a five-point ACR scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent). This test has 49 PVS, a hidden reference + 6 PVS per SRC: video degradations are selected from (6K QP22, 4K QP0, 4K QP28, 2K QP0, 2K QP34) and audio from (PCM, 64, 32, 16 kbps). PCM are the reference audio files at 1152 kbps/channel, 24bit, and 48kHz. The hidden references are 6K QP0 PCM sequences. We balance the PVS selection on the audiovisual quality scale, using prior knowledge from "Reference Condition". The median session duration is around 19 minutes for this condition.

In the DCR experiment, assessors use a five-point impairment scale (1: Very annoying, 2: Annoying, 3: Slightly annoying, 4: Perceptible but not annoying, 5: Imperceptible). The first stimulus is always the SRC. The second stimulus is the impaired PVS. Due to the explicit reference impacting test duration, only 5 SRCs are evaluated in these tests, with 6 PVS per SRC. "CrossRoadNight" and "HandballMatch" sequences are not evaluated. The average viewing duration is 24 min for the 30 PVS in the DCR condition. We recruit naive assessors aged 19 to 64 (mean=33.8, std=12.9) with different nationalities and educational backgrounds. Before the start of the test, we tested every assessor's vision in terms of visual acuity with Snellen charts and their color perception with the Ishihara test, and hearing acuity was self-reported. Assessors Table 1: Mean Content Ambiguity (CA) estimates from SUREAL MLE [START_REF] Li | Recover subjective quality scores from noisy measurements[END_REF] on data of the three conditions.

Conditions

360°AudioVisual test "Reference" 0.28 "DCR-Naives" 0.45 "ACR-Naives" 0.55 who did not meet the requirements were not recruited.

COMPARISON OF CONDITIONS

In this section, we compare the data collected in our two conditions with naive assessors against trained assessors of the "Reference condition".

Difficulty and subjectivity of the task

Advanced methods for outlier removal techniques like "MLE" and "MLE CO AP2" from SUREAL package 1 clean MOS.

The "MLE" algorithm jointly recovers subjective quality scores from noisy raw measurements, subjects' Bias and Inconsistency, and a Content Ambiguity estimate for each SRC. By modeling the noise, these methods help to understand the influence of assessors and SRCs on MOS. The reference papers of SUREAL [START_REF] Li | Recover subjective quality scores from noisy measurements[END_REF] show that the estimates of "MLE" and "MLE CO AP2", compared to methods using a threshold like BT.500 [START_REF]Methodology for the subjective assessment of the quality of television pictures[END_REF] and P.913 [START_REF]Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment[END_REF] are more interpretable and robust to spammer behaviors. The Content Ambiguity (CA) estimated by the "MLE" model indicates how subjective it is to annotate the quality of a particular sequence. By averaging estimated CAs across all SRCs of a test, we obtain a Mean CA score, reflecting the subjectivity of a QoE task. We can compare this score across test conditions and conclude if a task is more or less subjective for assessors. For the SAMVIQ scale in "Reference condition", we first scale the assessor's Opinion Score from 0-100 1 SUREAL: https://github.com/Netflix/sureal/tree/master/sureal Fig. 2: MOS discriminability evolution for the three 360°Au-dioVisual conditions, as a function of assessors' number and data collection duration. to 1-5 to fit the scale of ACR-HR and DCR. From table 1, we can see that the subjectivity of the task in our conditions with naive assessors using ACR or DCR scale is higher than for trained assessors. This shows how effective the training is on assessors using the SAMVIQ scale. Moreover, we can see that naive assessors using an ACR scale compared to a DCR scale face a more subjective task. This can be explained by the explicit reference in DCR easing the rating task.

Usage of the scale

In Figure 1, we present an analysis of the MOS obtained in the three test conditions. We fit a linear function (in red) and extract the slope and the intercept. We analyze these coefficients to see how assessors perceive the stimuli differently and how they use the rating scales. The black line translates the "one-to-one" relationship. In the Figure 1 left plot, we see a strong agreement between assessors in our conditions, with high correlation scores. Due to explicit references in the DCR condition helping detect differences, DMOS scores are, on average, lower for DCR than for the ACR scale. In the two other plots, the correlation is lower, translating that stimuli are perceived differently from naive assessors in consumer-grade conditions to trained assessors in a reference environment.

Discriminability analysis

In [START_REF] Randy F Fela | Perceptual evaluation on audio-visual dataset of 360 content[END_REF][START_REF] Nehmé | Comparison of subjective methods for quality assessment of 3D graphics in virtual reality[END_REF][START_REF] Pastor | Towards guidelines for subjective haptic quality assessment: A case study on quality assessment of compressed haptic signals[END_REF][START_REF] Pastor | Perceptual annotation of local distortions in videos: tools and datasets[END_REF], the authors suggested examining the MOS discriminability evolution with increasing numbers of assessors to show how well a subjective methodology can recover accurate MOS scores and compare subjective methodologies efficiency [START_REF] Pastor | Discriminability-Experimental Cost" tradeoff in subjective video quality assessment of codec: DCR with EVP rating scale versus ACR-HR[END_REF]. A two-sample Wilcoxon test is applied to all the possible pairs of MOS in a dataset to test the proportion of significantly different ones. We plot the evolution of this ratio in function of the assessors' number. The analysis is restricted to the 30 PVS DMOS common in the DCR-Naive, ACR-HR-Naive, and SAMVIQ-Trained conditions.

The results of the three 360°Audio-Visual conditions are presented in Figure 2 with a 95% confidence interval over 10,000 permutations. We can see that the most discriminative condition is the "Reference" one. The DCR-Naives condition can achieve similar discriminability with more assessors. 28 DCR-Naives assessors achieve the discriminability of 17 SAMVIQ-Trained assessors in "Reference" setup.

A factor that could explain the gap is the training followed by "Reference" condition assessors and the difference between the quality of the "Reference" and "Consumer-Grade" setups. "Reference" setup with a 26-channel loudspeaker Fig. 3: Audiovisual objective quality metrics Resolving Power [START_REF]Method for specifying accuracy and crosscalibration of video quality metrics (VQM)[END_REF] for the 3 conditions in the function of the discriminability Table 2: Objective metrics SRCC performance on data of the three audiovisual conditions: bold for best and italic for worst. Lastly, regarding experimental effort and data collection duration, ACR seems to be, at first, an efficient method to discriminate and obtain accurate MOS scores. But, we can see that using a DCR scale with an explicit reference per stimuli helps to achieve high discriminability in the long run. In the "Reference" condition, assessors are trained before the start of the data collection. This factor is not taken into account and can vary with the training process. It is, therefore, difficult to conclude on its benefit.

OBJECTIVE METRICS RESOLVING POWER

Metric resolving power has been defined in ITU-T Rec. J.149 [START_REF]Method for specifying accuracy and crosscalibration of video quality metrics (VQM)[END_REF], ITU-R Rec. BT.1676 [START_REF]Methodological framework for specifying accuracy and cross-calibration of video quality metrics[END_REF] and in [START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF]. It represents the lowest quality difference a metric can measure that has a statistical difference from the subjective ratings point of view. This characterizes the meaningfulness of quality differences measured by prediction models. However, resolving power depends on metric accuracy and also on the dataset quality that is used for evaluating the metric performance. This section will show how much performance can be claimed for different models depending on subjective data quality.

Five audiovisual quality prediction models are considered: vmaf visquol: VMAF [START_REF] Netflix | VMAF v0.6.1 Model[END_REF] video quality scores combined with ViSQOL [START_REF] Hines | ViSQOLAudio: An objective audio quality metric for low bitrate codecs[END_REF] audio quality scores using the linear combination defined in the audiovisual quality integration module of ITU-T Rec. P.1203.3 2 [19, 20, 34]. P1204m3 visquol: ITU-T P.12043 mode 3 bitstream video quality estimation model [START_REF]Video quality assessment of streaming services over reliable transport for resolutions up to 4k with access to full bitstream information[END_REF][START_REF] Rao | Bitstream-based model standard for 4k/uhd: Itu-t p.1204.3 -model details, evaluation, analysis and open source implementation[END_REF] combined with ViSQOL like vmaf visquol.

In addition, three parameter-based models based on ITU-T Rec. P.1203 mode 0 [START_REF]Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport[END_REF] are proposed. ITU-T Rec. P.1203 mode 0 only uses video bitrate, resolution, frame rate, and audio bitrate values to perform quality predictions. To apply this model to 360 videos, the headset FOV (110°) is used to scale resolution and bitrate values to represent what is seen in the headset. For example, a 6K (6144x3072) ERP with a 110°v iewport leads to 1878x1877 pixels seen into the HMD. Video bitrate is also scaled by (360×180)/(110×110) ≈ 5.35. Doing so, a per-viewport P.1203 mode 0 quality scores are computed and reported as P1203m0-O35v and P1203m0-O46v. O35 is the quality score after audiovisual and temporal pooling, while O46 is the final ITU-T P.1203 mode 0 prediction [START_REF]Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport -quality integration module[END_REF]. Finally, P1203m0-O22v visquol is proposed to replace the parameter-based audio quality estimation from the ITU model by ViSQOL.

Results are presented in Figure 3. For the three conditions, increasing the discriminability, hence the quality of subjective data improves the resolving power of quality metrics and shows the importance of having reliable MOS. Moreover, linking with results presented in table 2, P1203m0-O46v performs relatively well regarding SRCC across the different datasets. Its low resolving power also suggests it. Lower resolving power is better. Poorly performing metrics like VMAF VISQUOL on "Reference" conditions data and P1204m3 VISQUOL on ACR-Naives and DCR-Naives have accordingly high resolving power, and it is not changing much with higher discriminative subjective data. Finally, orders between metrics can change with increasing discriminability, as shown with P1203m0-O35v and P1203m0-O22v VISQUOL.

CONCLUSION

In this work, we compare the efficiency of subjective methodologies to assess the quality of 360°videos with HOA audio across different setups. We show that high discriminability obtained by trained assessors can be replicated with naive assessors and DCR methodology. Increasing discriminability enables to achieve of better resolving power of quality metrics. High-performing metrics tend to benefit more from high discriminability to express their best-resolving power. As well, the order of metrics can change across discriminability values. Finally, our proposal of a new parameter-based quality estimation model adapted to viewport resolution has shown competitive results with Full-Reference approaches.

For future audiovisual quality metrics development, we should consider more the discriminability of subjective data along with metrics performance evaluation.

Fig. 1 :

 1 Fig. 1: Comparison of Audiovisual conditions on rating scale usage by assessors.

https://github.com/itu-p1203/itu-p1203

https://github.com/Telecommunication-Telemedia-Assessment/bitstream mode3 p1204 3


Mathis Neuhauser

Sébastien Verrier

Sylvain Mangiarotti

Olivier Merlin

Beatriz Molero

Yann H Kerr

Multi-scale statistical properties of disaggregated satellite soil moisture product in Australia

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

anisotropic variant of multifractal analysis [START_REF] Lovejoy | Towards a new synthesis for atmospheric dynamics: Space-time cascades[END_REF]  "Two scale ranges" behavior of disaggregated soil moisture, possibly due to:

• Physical processes acting at different scales: evapotranspiration, infiltration [START_REF] Katul | On the spectrum of soil moisture from hourly to interannual scales: Spectrum of soil moisture content[END_REF] • Different surface areas between satellite measurements and disaggregated product  Two scale ranges on DisPATCh power spectrum (whether analysis was made on the entire period or a particular date)  Large scales (l > 10 km): follows satellite soil moisture behavior (SMOS)  Small scales (l < 10 km): different behavior (higher scaling exponent β)

•

  de Toulouse, CNES, CNRS, IRD, UPS, France mathis.neuhauser@cesbio.cnes.fr CASE STUDY AND DATA DisPATCh (Disaggregation based on Physical And Theoretical Scale Change; Merlin et al, 2008): • Deterministic algorithm that improves satellite soil moisture resolution from 40 to 1 km • Combines microwave soil moisture with optical data: vegetation index and surface temperature • Based on simplified radiative transfer model (estimation of soil evaporative efficiency) multi-scale variability of disaggregated satellite soil moisture product (DisPATCh), by applying spectral and multifractal analysis (in the framework of Universal Multifractal). Soil moisture has a strong impact on climate, hydrology and agronomy at different space scales, from the continent global scale to the local watershed. Actual technologies like SMOS satellite (Soil Moisture and Ocean Salinity; Kerr et al, 2010) allow a global study of soil moisture on the entire globe. However, these sensors measure soil moisture only on specific scales, often much larger than kilometer. To have access to kilometric variability, disaggregation algorithms have been developed. They consist in combining low resolution soil moisture with high resolution ancillary data. Validation of the small-scale soil moisture behavior comparing to other datasets: ground and airborne campaigns AACES (2010), NAFE (2006) • Restarting the disaggregation algorithm on the entire SMOS area (2000×2000 km²) • Comparison to an independent and complementary multi-scale analysis method:

•

  Non-linearity of the SEE=f(NDVI) relation affecting the scaling exponent• Dynamical behavior of synchronized aggregated processes[START_REF] Mangiarotti | Global modeling of aggregated and associated chaotic dynamics[END_REF]