Mobile devices are ubiquitous: nowadays most people own a mobile telephone. Because of this global presence, it is a target of interest for attackers. These attacks are delivered via malicious applications that can harm mobile device, for example by obtaining illegal control of it, gaining unauthorized access to the user's personal information or scamming a person with a ransom, among other malicious intents. Researchers in malware analysis put their effort to recognize these types of programs before they are installed on a user device. To do this, they perform experiments to automatically detect malware, for example with machine learning, where a learning phase is done over a set of already known malware and goodware. Afterwards, researchers make a performance evaluation over several new datasets, independent of the one in the learning phase. Depending on their choice of datasets, the evaluation of the experiments can yield acceptable results, or outstanding but overestimated results. Consequently, datasets with malware and benign samples are important elements to consider when designing an experiment.

Résumé

Les dispositifs mobiles sont ubiquitaires: aujourd'hui la majorité des gens possèdent un téléphone mobile. A cause de leur présence global, ces dispositifs sont une cible d'intérêt pour les attaquants. Ces attaques sont véhiculées au travers des applications malveillantes qui peuvent nuire aux dispositifs mobiles, par exemple en obtenant leur contrôle illégale, en gagnant accès aux informations personnelles de l'utilisateur, ou en menaçant les données à l'aide d'un rançongiciel -exemples parmi d'autres types de malveillances. Les chercheurs en analyse de malware travaillent à reconnaître ces types de programmes avant qu'ils soient installés sur un dispositif utilisateur. Pour faire cela, ils réalisent des expériences pour automatiquement détecter ces malware, par exemple avec de l'apprentissage automatique, où une phase d'apprentissage est réalisée sur un ensemble de malware et des applications bénignes déjà connues. Ensuite, les chercheurs réalisent une évaluation de la performance sur plusieurs nouveaux datasets, indépendants de ceux utilisés dans la phase d'apprentissage. Selon le dataset choisi, les résultats des expériences peuvent être acceptables ou bien exceptionnellement bons parce que surestimés. Par conséquent, les datasets de malware et applications bénignes sont des éléments importants à considérer quand nous élaborons une expérience.

Les chercheurs en détection de malware Android rencontre plusieurs obstacles pour créer un bon dataset expérimental. D'abord, beaucoup de magasins d'applications sont disponibles pour récupérer des applications. Leur façon de distribuer ces applications ne convient pas pour faire des expériences: il n'y a pas de méthode aisée pour extraire des applications et connaître la vérité terrain sur le fait d'être un malware ou non. De plus, les malware peuvent être sur ou sous-représentés dans un extrait du magasin d'applications. De ce fait, les chercheurs utilisent d'autres datasets expérimentaux, incluant la vérité terrain, que d'autres chercheurs ont partagés à la communauté scientifique. Ces datasets partagés deviennent peu à peu de vieux datasets avec de nombreuses citations. D'autres chercheurs ont aussi décidé de créer leurs propres datasets expérimentaux avec des applications provenant d'autres sources. Nous pouvons alors nous demander si tous ces datasets expérimentaux ont une influence sur les résultats des expériences conduites, et si c'est le cas, si une méthodologie de création est plus propice que d'autres.

Cette thèse présente, premièrement, une méthode pour évaluer la qualité des datasets basée sur un test statistique qui aide à comparer un dataset créé avec un grand ensemble d'applications par exemple issu d'un magasin d'applications. Nous montrons alors que les datasets historiques de la littérature sont de mauvaise qualité, ce qui justifie le besoin de créer des nouveaux datasets plus à jour. Deuxièmement, nous introduisons un algorithme pour mettre à jour des datasets mixtes de malware/goodware de mauvaise qualité afin de ressembler à un dataset cible qui ne peut pas être utilisé directement, e. g. un magasin d'applications. Nous évaluons les datasets mixtes mis à jour en utilisant un algorithme d'apprentissage automatique et nous montrons que la détection de malware sur notre dataset mis à jour devient un problème plus difficile à résoudre. Enfin, nous introduisons DaViz, un outil de visualisation de datasets pour explorer et comparer des datasets d'applications Android. Cet outil permet aux chercheurs de visualiser les biais dans les datasets de la littérature, et d'obtenir des informations utiles à leur propos.

Contributions

Cette thèse travaille sur deux axes: la qualité des datasets et la visualisation de ceux-ci. La thèse propose des contributions indépendantes sur ces deux axes bien que les contributions de visualisation des datasets aide à comprendre les enjeux de qualité.

Ainsi, la thèse débute par la présentation de l'état de l'art relatif aux datasets d'applications Android, que ce soit pour les goodware ou les malware. Nous analysons les datasets utilisés par la communauté scientifique et nous montrons que certains travaux souffrent d'irrégularités, par exemple sur l'équilibrage des dates des applications utilisés. Ces irrégularités confirment des travaux similaires conduits par Pendlebury etal et Allix etal. Cette première analyse n'étant basée que sur la lecture des méthodologies employées dans la littérature, la thèse poursuit l'investigation de ce problème en cherchant à représenter graphiquement ces irrégularités.

Une première contribution concerne donc la visualisation de datasets d'applications Android. Nous proposons une méthode intéractive pour visualiser et comparer des datasets, au travers d'un outil nommé DaViz. DaViz offre des fonctionnalités non rencontrées dans les outils existants: la possibilité de comparer plusieurs datasets, de sélectionner des sous parties interactivement, et de visualiser des caractéristiques grâce à trois représentations différentes. Le bénéfice attendu est de pouvoir réaliser des explorations de datasets inconnus ou de comparer deux à deux des datasets pour en saisir les différences notables. L'outil est exploité sur les datasets de la littérature (Drebin, AMD) mais aussi sur des extractions récentes de goodware et de malware.

Il confirme visuellement les irrégularités identifiées précédemment. Cette partie du travail de thèse nous a conduit à vouloir quantifier formellement ces différences, ce qui l'objet de la contribution suivante.

Une seconde contribution est dédiée à l'élaboration d'une méthode pour évaluer la qualité des datasets à l'aide d'un test statistique. Ce test permet de comparer un dataset créé manuellement avec un grand ensemble d'applications par exemple extrait d'un magasin d'applications. A l'aide de cette méthode statistique, nous montrons alors que les datasets historiques de la littérature sont de mauvaise qualité, ce qui justifie le besoin de créer des nouveaux datasets plus à jour. D'autres raisons, comme le manque de labels goodware/malware dans les sources possibles d'applications, par exemple le Google Play Store, nous a poussé à nous intéresser à la création de datasets dits "mixtes" i.e. contenant des goodware et des malwares. L'enjeu est alors de construire de tels datasets correctement labellisés, tout en étant proche de la réalité, c'est-à-dire de ce que l'on obtiendrait en échantillonnant un magasin d'application.

Une troisième contribution est dédiée à l'élaboration d'un algorithme qui travaille sur un dataset mixte mais biaisée, c'est-à-dire que certaines caractéristiques sont sur ou sous-représentées comparée à ce que l'on trouverait dans un magasin d'application. Notre algorithme va chercher à modifier autant que faire ce peut le dataset mixte afin de le faire ressembler au dataset dit "cible" par exemple un extrait du magasin d'application. L'intérêt de l'algorithme est de travailler sur un dataset mixte dont les labels sont connus i.e. la caractéristique "être un malware" est établie. Nous obtenons en un nombre optimal d'itérations un dataset dit "débiaisé" dont l'éloignement du dataset cible est contrôlé par un paramètre 𝛿.

Nos expérimentations montrent qu'il est difficile de modifier les très anciens datasets comme Drebin, vu leur éloignement du dataset cible. Pour les datasets mixtes que nous avons construit avec des sources plus récentes, la convergence est plus aisée. Les datasets débiaisés produits ont été évalués en les utilisant dans des expérimentations de détection basés sur des algorithmes classiques de machine learning. Ces expériences montrent qu'il est plus difficile de détecter les malware quand on utilise des datasets débiaisés. De plus, apprendre sur des datasets débiaisés donnent de meilleurs résultats lors de la détection sur les ensembles de tests.

Conclusion

Les résultats obtenus dans ces travaux de thèses ouvrent différentes perspectives. D'autres outils statistiques pour mesurer la qualité des datasets pourraient être employés afin de mieux quantifier la "distance" existant entre deux datasets. De plus, les méthodes employées sur les datasets d'applications Android pourraient être utilisées pour des datasets qui concernent les applications x86. L'outil de visualisation pourrait être étendu pour être capable de basculer de la visualisation d'ensembles à la visualisation d'une seule application: l'enjeu est alors de produire des vues internes des applications qui soient pertinentes pour l'analyste explorant un dataset inconnu.

Researchers in Android malware detection encounter several obstacles to create good experimental datasets. Many different markets are available to take applications from. Because of their nature, they are not suited for experiments: there is no clear and easy way to extract applications from them and to know the ground truth about being a malware or not. Additionally, malware could be underrepresented in a extract of the market. As a result, researchers use other experimental datasets, including their ground truth, that other researchers have shared to the research community. These shared datasets slowly became old and cited, and other researchers decide to craft their own experimental dataset from these shared datasets with applications from other sources. We wonder if these crafted experimental datasets have an influence over the results of conducted experiments, and if so, if some crafting methodology is more suitable than others. This thesis presents, first, a method to evaluate the quality of datasets based on a statistical test that helps to compare a crafted dataset against a large set of applications such as markets. We show that historical datasets of the literature are of low quality, which justifies the need to create new up-to-date datasets. Second, we introduce an algorithm to update mixed datasets of malware/goodware of low quality in order to resemble a target dataset that cannot be used directly, e. g. a market. We evaluate the updated mixed datasets using a machine learning algorithm and we show that the detection of malware in our up-to-date dataset becomes a more difficult problem to solve. Lastly, we introduce DaViz, a dataset visualization tool for exploring and comparing Android malware datasets, which enables researchers to visualize the biases in datasets of the literature, and obtain useful information from them.

Résumé substantiel en français Introduction

Les dispositifs mobiles sont ubiquitaires: aujourd'hui la majorité des gens possèdent un téléphone mobile. A cause de leur présence global, ces dispositifs sont une cible d'intérêt pour les attaquants. Ces attaques sont véhiculées au travers des applications malveillantes qui peuvent nuire aux dispositifs mobiles, par exemple en obtenant leur contrôle illégal, en gagnant accès aux informations personnelles de l'utilisateur, ou en menaçant les données à l'aide d'un rançongiciel -exemples parmi d'autres types de malveillances. Les chercheurs en analyse de malware travaillent à reconnaître ces types de programmes avant qu'ils soient installés sur un dispositif utilisateur. Pour faire cela, ils réalisent des expériences pour automatiquement détecter ces malware, par exemple avec de l'apprentissage automatique, où une phase d'apprentissage est réalisée sur un ensemble de malware et des applications bénignes déjà connues. Ensuite, les chercheurs réalisent une évaluation de la performance sur plusieurs nouveaux datasets, indépendants de ceux utilisés dans la phase d'apprentissage. Selon le dataset choisi, les résultats des expériences peuvent être acceptables ou bien exceptionnellement bons parce que surestimés. Par conséquent, les datasets de malware et applications bénignes sont des éléments importants à considérer quand nous élaborons une expérience.

Les chercheurs en détection de malware Android rencontre plusieurs obstacles pour créer un bon dataset expérimental. D'abord, beaucoup de magasins d'applications sont disponibles pour récupérer des applications. Leur façon de distribuer ces applications ne convient pas pour faire des expériences: il n'y a pas de méthode aisée pour extraire des applications et connaître la vérité terrain sur le fait d'être un malware ou non. De plus, les malware peuvent être sur ou sous-représentés dans un extrait du magasin d'applications. De ce fait, les chercheurs utilisent d'autres datasets expérimentaux, incluant la vérité terrain, que d'autres chercheurs ont partagés à la communauté scientifique. Ces datasets partagés deviennent peu à peu de vieux datasets avec de nombreuses citations. D'autres chercheurs ont aussi décidé de créer leurs propres datasets expérimentaux avec des applications provenant d'autres sources. Nous pouvons alors nous demander si tous ces datasets expérimentaux ont une influence sur les résultats des expériences conduites, et si c'est le cas, si une méthodologie de création est plus propice que d'autres.

List of Figures

Introduction 1

Mobile devices, primarily smartphones, have become ubiquitous: most people own at least one of them. They represent, according to statcounter 1 1: https://gs.statcounter.com/platformmarket-share/desktop-mobile-tablet as of June 2022, [START_REF] Lindorfer | ANDRUBIS -1,000,000 Apps Later: A View on Current Android Malware Behaviors[END_REF].77% of all types of computers used in the world nowadays. Among the different mobile operating systems (OS), Android is the most popular in the world with 72.12% of market share as of June 2022 2 .

2: https://gs.statcounter.com/osmarket-share/mobile/worldwide

These devices are used for many purposes, from everyday actions such as communication between persons and media consumption, to sensitive operations such as money transferring and business communication.

Because of the number of people using smartphones, and all the possible tasks that can be accomplished with them, the Android system has become a top target for attackers.

Of all the types of possible attacks, there are programs that may harm a system or trick its users to obtain a gain, without the user's knowledge and consent. These types of programs are called malware. Although there is no practical definition of what a "malware" is, there are definitions for types of malicious applications based on how they work. Among these types, we can mention: spyware, which sends personal information to the attackers without the user's knowledge; ransomware, which blocks the access of users to a system and demand them a ransom to grant access to it again, and bankbots, which steal login credentials for banking applications from users.

To protect users from such threats, experts in malware analysis take measures to detect these applications before they arrive to the end user.

To do this, they take applications from the places users tend to obtain them, such as Google Play, and conduct experiments to verify what these applications do. If analysts discover any malicious behavior in an application, they flag it as malware, and depending of this behavior, they can classify this malware into one of the numerous types there exists.

Analysts can operate manually, when a specific application is performing suspicious actions or upon request of users. Nevertheless, with the high number of applications on different online stores, it is impossible to rely on manual analysis for each new application or update of an application. This is why the research community has worked on automatic analysis procedures to extract relevant data from applications and eventually decide if an application is performing malicious activities.

Machine learning (ML) based algorithms have been used extensively in automating the attribution of maliciousness of applications [START_REF] Ucci | Survey of machine learning techniques for malware analysis[END_REF][START_REF] Liu | A Review of Android Malware Detection Approaches Based on Machine Learning[END_REF][START_REF] Pan | A Systematic Literature Review of Android Malware Detection Using Static Analysis[END_REF][START_REF] Qiu | A Survey of Android Malware Detection with Deep Neural Models[END_REF]. By extracting properties from applications, called "features" in the ML jargon, either statically (directly from their code) or dynamically (from executing them and registering their behavior), an ML algorithm can obtain outstanding results at detecting malicious applications. Academics contributing in this research area need to compare their results with previous ones published in the literature. As a consequence, they use datasets, literally "sets of data", as inputs to compare multiple detection algorithms.

The first seminal paper that released a dataset has been published by Arp et al. [START_REF] Arp | DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket[END_REF]. They released the Drebin dataset. At the time, it was the largest dataset containing malware samples available to the research community. Subsequent papers reused this dataset in order to provide incremental and reproducible results. In the meantime, whereas Google was releasing new versions of the Android OS, Google Play drastically increased in the number of applications. This effort of collecting and labeling malware samples is not trivial, making Drebin and AMD [START_REF] Wei | Deep Ground Truth Analysis of Current Android Malware[END_REF], another bigger Android malware dataset, an easy and fast way to obtain samples. As a consequence, few new datasets were published and released in the next ten years, and the old and widely cited Drebin dataset continued to be used. At the time of writing this thesis, Drebin is cited more than 2 000 times according to Google Scholar, which is an impressive score for only a single paper.

The reader should recall that the intent behind a dataset in a malware detection experiment is to mimic, in a laboratory, the conditions encountered in the reality. This "reality" is also called the "ground truth" in the literature, and we call it the "target" set of applications in this chapter. That is, for malware detection, mimicking the difficulty for an automatic program to discover malware hidden in Google Play, where thousands of applications are uploaded each day. We believe that it would be surprising that Drebin could mimic such a problem on the current Google Play, with recent sophisticated malware.

The context of this thesis is, thus, focused on the quality of the datasets of the literature and, if the quality is confirmed to be low -which is our intuition, on the elaboration of new methods to produce datasets of high quality, i. e., able to mimic the reality.

This thesis explores the creation and usage of experimental datasets for various usages, such as the typical malware detection problem. One of the issues we intend to solve is that experimental datasets are created, disregarding the target set of applications that the experiment aims. Thus, even though the experiment's results are excellent, its method would fail when operated on the target production environment.

The term "quality" for a dataset is undefined at this stage of writing, which we formalize in Chapter 4. For now, we explain what we believe are the problems of quality in datasets. We illustrate these problems with two hypotheses (that will be confirmed later in the manuscript):

▶ H1: Experiments with 99% accuracy are suspicious ▶ H2: Input datasets influence the results of the experiments H1: Experiments with 99% accuracy are suspicious Malware detectors using ML would predict a sample as a positive (i. e., is a malware), or negative (i. e., is a goodware). Since we know in advance the labels (i. e., which are malware and goodware), we can evaluate how well the detector predicted malware (true positives) and goodware (true negatives) correctly. Results from malware detectors using ML algorithms are usually given in the form of:

▶ accuracy (the concentration of true predictions among all predictions)

▶ precision (the concentration of true positives among all positive predictions) ▶ recall (the concentration of true positives among all true samples) ▶ F1-score (a harmonic mean of precision and recall) For all these scores given in percentage, 100% is a perfect score. With the increasing popularity of ML techniques, analysts have focused in developing better discriminants used by these machine learning algorithms to differentiate malware from benign programs. There are several studies that show very good results with over 99% [START_REF] Idrees | PIndroid: A novel Android malware detection system using ensemble learning methods[END_REF][START_REF] Billah | MalDozer: Automatic framework for android malware detection using deep learning[END_REF][START_REF] Vinod | A machine learning based approach to detect malicious android apps using discriminant system calls[END_REF] of detection [START_REF] Idrees | PIndroid: A novel Android malware detection system using ensemble learning methods[END_REF]: [START_REF] Idrees | PIndroid: A novel Android malware detection system using ensemble learning methods[END_REF], 'PIndroid: A novel Android malware detection system using ensemble learning methods' [START_REF] Billah | MalDozer: Automatic framework for android malware detection using deep learning[END_REF]: Karbab et al. (2018), 'MalDozer: Automatic framework for android malware detection using deep learning' [START_REF] Vinod | A machine learning based approach to detect malicious android apps using discriminant system calls[END_REF]: P. et al. (2019), 'A machine learning based approach to detect malicious android apps using discriminant system calls' accuracy. We can think that with these types of results, Android malware should no longer be a problem if these experiments were transferred from prototypes to real antiviruses. But unfortunately, this is still not the case: we observe that malicious programs are still discovered, for example, on the Google Play. Even if their choice of discriminant features is sound, we suspect that the way the chosen experimental dataset was created may have produced overestimated results.

H2: Input datasets influence the results of the experiments

During this thesis, two papers were published suggesting that input datasets influence experiments in Android malware detection. Allix et al. [START_REF] Allix | Are Your Training Datasets Yet Relevant?[END_REF] [10]: [START_REF] Allix | Are Your Training Datasets Yet Relevant?[END_REF], 'Are Your Training Datasets Yet Relevant?' discuss the influence of the applications' date in experimental datasets used in Android malware detection. The authors have found that the way researchers create their experimental datasets leads to biased results, in Android malware detection using ML. They compare a typical experiment configuration using ML to a more real-life scenario one, and the results of both experiments tended to differ. This indicates that experiments with the typical configuration are not suited to work against new malware just like in real-life, which is misleading despite having announced high detection accuracy in their results. Pendlebury et al. [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF] also discuss the We believe that this idea of "distribution resemblance", whether it is about date or malware proportions in experimental datasets, can be generalized to a wide variety of characteristics of Android applications in order to obtain a dataset that resembles more the target set of applications the experiment aims.

As a conclusion, this thesis focus on three main research questions:

▶ Is it possible to quantify the quality of a dataset of applications with respect to the target set of applications? This question suggests that a metric should be elaborated to compare a dataset with the "target" set of applications. ▶ Can other characteristics, besides the date and the ratio of malware/benign applications, be used to detect differences between datasets? Because inconsistencies have already been identified in previous works concerning the date of applications and the ratio of malware and benign applications of the dataset, this thesis intends to confirm and extend these works on more attributes than solely the date and malware/benign ratio. ▶ Given an arbitrary set of applications of low quality, can we investigate the reasons behind this low quality? This question is related to the fact that researchers may obtain datasets from different sources and eventually untrusted third parties without reliable information on the building process of this obtained dataset. In such a case, researchers would want to legitimately explore and assert the quality of the dataset.

Contributions

To answer the previous research questions, we propose three main contributions in this thesis:

1. We introduce a new tool to visually explore and compare datasets called DaViz [START_REF] Tomás Concepción | DaViz: Visualization for Android Malware Datasets[END_REF]. With this tool, researchers can inspect datasets to find interesting patterns about them, and also distinguish at a glance the differences between various datasets. 2. We give guidelines on how to calculate the size of an experimental dataset, and introduce the notion of "representativity" using combinations of characteristics we call "classes" [START_REF] Tomás Concepción | Debiasing Android Malware Datasets: How Can I Trust Your Results If Your Dataset Is Biased?[END_REF]. We also propose the use of a statistical test to verify that datasets are indeed different against a reference dataset chosen to study. 3. We then use these classes to create a "debiasing" method [START_REF] Tomás Concepción | Debiasing Android Malware Datasets: How Can I Trust Your Results If Your Dataset Is Biased?[END_REF] for datasets to resemble another reference dataset of choice. We use this method to create several "debiased" datasets that resemble more the "target" dataset than datasets from the literature. We use one of these "debiased" datasets in a malware detection experiment based on ML algorithms, showing that the detection performance is lower for old datasets.

Associated publications

Our work on visualization and DaViz have been presented in the RESSI 2022 conference [START_REF] Tomás Concepción | DaViz: Visualization for Android Malware Datasets[END_REF]. The method for debiasing datasets, alongside the code and the datasets used in the experiment, are available in the Dada dataset [START_REF] Concepcion | Debiased Android DAtasets[END_REF]. Lastly, the results from our debiasing experiments were

Outline

This dissertation is divided in two parts. The first part consists of this introduction and Chapter 2, that gives a background on the usage of datasets for experiments in Android malware.

The second part is dedicated to our contributions of this thesis. First, Chapter 3 explores a new visualization tool called DaViz. Next, Chapter 4 introduces and defines the notion of representativity of datasets. It also shows a procedure to differentiate datasets using statistical tests. Then, Chapter 5 presents a new method for generating "debiased" datasets to resemble a studied dataset that is much more difficult to use directly.

Finally, Chapter 6 summarizes the contributions of this thesis and gives perspectives for future works.

Research Android Datasets and

Where to Find Them 2 In order to start an experiment in malware research, malware samples, as well as goodware samples, must be collected to form an experimental dataset. Researchers have resorted to a variety of places to find malware samples: from VX (Virus eXchange) forums to research teams willing to share their samples to the community. The quality of the input dataset in an experiment is important, as it can have an influence in the results of the experiment. The way it is constructed, its size, and the sources, among other properties, need to be taken into account.

This chapter focuses on the study of Android datasets in malware research, the different uses of them, and the irregularities and biases encountered in their usage. In Section 2.1, we start by defining what a dataset is. After that, we explore the different platforms for software distribution in Android that users utilize to download and install applications, alongside the various research datasets used in security studies on Android malware. Finally, in Section 2.2, we review the biases that experiments suffer due to the construction of its experimental dataset, with a review of Android malware detection experiments to see from which irregularities they suffer, and lastly notice what missing type of bias experiments do not pay attention to.

Datasets used in the literature

In this section, we discuss the several datasets and repositories used in Android malware research.

We begin by defining the notion of dataset: the Cambridge dictionary defines a dataset as "a collection of separate sets of information that is treated as a single unit by a computer". For this thesis, an Android application dataset (that can be an "Android malware dataset" when it includes malware) fulfills at least one of the following criteria: ▶ A dataset is a collection of raw Android applications (.apk files) ▶ A dataset is a collection of results from analyses done to Android applications that includes a hash value for each application

In this matter, we can categorize the diverse datasets according to their main purpose. This section presents and discusses three types of datasets: markets, market archives, and specific datasets. Markets are repositories where users download applications. Market archives are application repositories that collect samples from markets. Specific datasets are ones created by researchers to aid in the creation of experimental datasets.

Markets

In this section, we start by defining what a market is, what they contain, what security mechanisms they offer to avoid the distribution of malicious applications, and the different markets in existence and those no longer available. At the end of the section, we conclude with the value of markets for researchers.

Markets are software distribution platforms where users can find and download applications, and developers can distribute their software. These types of datasets have a dynamic behavior in regards to how their applications are managed: new ones can be added increasing the size of the dataset, old ones could be removed and never found again, and newer versions replace the existing ones.

Google Play

The first Android application dataset to be considered is Google Play [START_REF]Google Play[END_REF] [15]: Google LLC (2022), Google Play (also known as Google Play Store, and previously known as Android Market). This is the main repository where Android users can officially download applications from. It has around 2.5 million applications [START_REF]Android and Google Play statistics[END_REF], [START_REF]Android and Google Play statistics[END_REF]: AppBrain (2022), Android and Google Play statistics which has peaked in 2018 [START_REF]Number of Android apps on Google Play[END_REF] to a little more than 3.7 million applications.

[17]: AppBrain (2018), Number of Android apps on Google Play

Previously, applications could be downloaded directly from the Google Play website, allowing the use of scraping methods to obtain as many applications as possible. This was done through the use of an API that Google changed at some point in time, which is not openly disclosed [START_REF] Viennot | A Measurement Study of Google Play[END_REF]. Nowadays, users use the official "Play Store" application to search, download and manage installed applications. Other alternative clients exits [19,[START_REF] Yeriomin | Yalp Store[END_REF], but they violate Google's terms of service.

[19]: Rahul Patel (2022), AuroraOSS [START_REF] Yeriomin | Yalp Store[END_REF]: Yeriomin (2016), Yalp Store

Although Google Play is the main platform for obtaining Android applications, its usage in research is limited. Firstly, because of the changing nature of the market, applications may be removed or updated, forbidding researchers to obtain samples from previous experiments and worsening reproducibility. And secondly, because of the lack of transparent API to interact with the platform, and limitations to download applications, it is difficult for researchers to get samples from the official market.

Third-party markets

Besides Google Play, other third-party markets exist that also offer software distribution to Android. They operate in a similar fashion to Google Play: they are backed by a company or organization, they have an application to contact the repository and download the applications. They also tend to have a different distribution model than the official market, or have certain regulations to what type of applications they offer.

One of those is SlideME [START_REF] Slideme | [END_REF]. It is one of the first third-party markets smartphones, but it can be installed in other devices. As recently as 2021, it hosted malware in the form of clones of other applications, in this case one for movie piracy [START_REF] Hager | Samsung's Galaxy Store is distributing apps that could infect phones with malware[END_REF]. Caterpillar also provides a specialized [START_REF] Hager | Samsung's Galaxy Store is distributing apps that could infect phones with malware[END_REF]: Hager (2021), Samsung's Galaxy Store is distributing apps that could infect phones with malware application store for their phones [START_REF]Toolbox App[END_REF]. Aptoide [START_REF] Aptoide | [END_REF] is another third-party applications also found in Google Play. F-Droid [START_REF] Limited | [END_REF] is a market special- [START_REF] Limited | [END_REF]: F-Droid Limited (2022), F-Droid ized in distributing free and open-source software (FOSS). It offers their own repository with its official application to download and manage the applications it contains. Since F-Droid offers FOSS, the source code of each application is provided in the description. This allows users and other external auditors to more easily verify the code for security flaws or for malicious behavior. The market itself builds and signs each application before distribution * . External repositories can be added to the client, which allows developers to distribute their software directly without passing through the F-Droid publishing process. AndroidOut [START_REF] Ltd | [END_REF] [28]: GLOBAL PTE LTD (2022), Android-Out is a web-based market where applications, some that can be found in Google Play, are curated by a community of people. According to their website [START_REF] Pte | Androidout corporate -Who we are -What we do Androidout[END_REF], they count with more than 2.3 million registered users are MiKandi [START_REF]MiKandi[END_REF] (which is defunct), 1mobile [START_REF][END_REF] and GetJar [33] (where [START_REF]MiKandi[END_REF]: MiKandi LLC (2022), MiKandi [START_REF][END_REF]: 1Mobile Market (2022), 1Mobile Market [33]: Laurs (2022), GetJar malware have been found [START_REF]Remove Android:Plankton [PUP] Virus from Android Device after Downloading Apps from GetJar[END_REF]). Several database services exists that gather information about markets. These are aimed for developers to provide data about these markets for business intelligence, but may also provide useful information for researchers that study these markets. Services like App-Brain [START_REF] Vogelzang | AppBrain[END_REF], MixRank [START_REF] Milliken | [END_REF], anroidrank.org [START_REF] Katrenic | ANDROIDRANK[END_REF], SimilarWeb [START_REF] Similarweb | Top Apps Ranking[END_REF], Tap- numerous indicators, like downloads, SDK usage, release dates, etc.

Regional third party markets

Besides those markets that can be accessed by everyone in the world, there are others aimed towards a specific group of people or region. These regional markets has emerged because of restrictions in their home countries, or their target audience's language. The Android market ecosystem in China is primarily ruled by regional markets, where they offer applications mainly in Chinese language. One of the reasons behind this is the restriction of Google services in China [START_REF] Quinn | Google services blocked in China[END_REF]. Therefore, Chinese [START_REF] Niayesh | How developers can make money in Iran's app market? AzerNews[END_REF]. Uptodown [START_REF] Domínguez | We are Uptodown[END_REF] is a Spanish-based marketplace founded in [START_REF] Niayesh | How developers can make money in Iran's app market? AzerNews[END_REF]: Niayesh (2017), How developers can make money in Iran's app market? focuses on them according to their "About" page, with a store application to access it.

Discussion

Researchers that aim to study markets face the dynamicity of these platforms. It is a problem in terms of reproducibility: since applications can be removed (or replaced by newer versions), experiments can no longer be reproduced with the same data. Besides this, none of these markets provide an API to interact with them, making it difficult to obtain applications and metadata from the sites. Malware required for experiments is very hard to find in markets, because these platforms tend to remove malicious applications once they find one. The quantity of markets available is another source of problem: because markets are different from each other, studies done on one market do not apply directly to others. And because of the difficulty to interact with markets to obtain applications, market metanalyses would take a lot of effort and time to perform. Nevertheless, since markets are the main focus of security for finding and mitigating the spread of malicious applications, they need to be considered when testing security solutions for them.

Market archives

In comparison to markets, market archives are software repositories that add applications from markets to their collection. They tend to use crawlers or rely on user submissions to obtain applications.

One of the first archives for Google Play available was PlayDrone [START_REF] Viennot | A Measurement Study of Google Play[END_REF]. The issue this archive tackled is crawling the Google Play. Google limits its access by requiring clients to use registered accounts and exclusively use the Google Play application to request Google's servers. To bypass this, PlayDrone uses real accounts in combination with the API they reverse engineered from the store application. The system is designed so that it can scale with more servers. Using this crawler, the authors have collected over one million applications, which they used to make a study of the market. An archive of their applications can be found [START_REF]Android Apps[END_REF] [48]: jakej (2022), Android Apps in the Internet Archive [START_REF]The Internet Archive[END_REF]. APKMirror [50] is an Android applications [START_REF]The Internet Archive[END_REF]: The Internet Archive (2022), The Internet Archive

[50]: Illogical Robot LLC (2022), APKMirror archive, with one of their primary purposes being to "provide an archive of popular applications along with changelogs and descriptions". It contains multiple versions of the applications it hosts, but only free of cost ones can be found. Users can download and install applications from the site, or use their installer on the phone. APKPure [START_REF] Apkpure | [END_REF], APKLinker [START_REF] Apklinker | [END_REF] [51]: APKPure, Inc. (2022), APKPure [START_REF] Apklinker | [END_REF]: APKLinker (2022), APKLinker and APK4Fun [START_REF]APK4Fun. APK4Fun[END_REF] are also archives that provide different versions of the [START_REF]APK4Fun. APK4Fun[END_REF]: APK4Fun (2022), APK4Fun cost free applications they hosts. As well as in markets, archive can hosts malware [START_REF] Golovin | Infected Android app store[END_REF]. Some application clients for these archives had malware [START_REF] Golovin | Infected Android app store[END_REF]: Golovin (2021), Infected Android app store found in them [START_REF] Golovin | Malicious code in APKPure app[END_REF].

Discussion

Market archives are more desirable to use for researchers because they maintain applications available for longer than markets, allowing better reproducibility of experiments. Another advantage is that we can recover missing applications from markets if the hash value is known. These market archives also provide certain metadata about the applications, such as the type of application, user reviews, and different versions of the same program. Note that, in a study aimed towards one or a set of markets, market archives cannot replace those. They can, however, be used to obtain applications previously in markets if these applications are unavailable. Despite all these advantages, researchers performing an experiment on one market archive, would obtain slightly different results from another archive or even a normal market. Particularly, there are no paid applications, and there are lot of clones of the same application, as they keep different versions of it. For these reason, using markets archives is not suitable for performing studies. This is why researchers use specific research datasets, which we explain in the next section.

Specific research datasets

These datasets are collections of applications made by researchers in Android malware. They tend to contain primarily malicious applications, as well as benign ones and metadata associated to the applications from results of analyses of these programs. This is the main difference between these datasets and the previous ones: these ones can contain malware deliberately, because it was used for an experiment or it was part of the design of the dataset to contain malware. These specific datasets are then shared among researchers to conduct experiments in Android malware, as malware is not easily found.

When we reviewed the literature datasets, we decided to divide these datasets in three main categories based on their usage goal:

▶ Malware archives: a collection of applications where we can find malware. ▶ Experimental datasets: collection of applications used in an experiment and made available by researchers conducting this study. ▶ Novelty datasets: collection fo applications made by researchers with particular properties to fulfill a specific need.

Malware archive datasets

These datasets are special archives maintained by researchers to primarily gather and share malware and eventually goodware (benign applications). They are different from market archives seen in the previous section because these datasets tend to require permission to access them, where market archives are more open to end users. We can further subdivide this category into collection archives and collaborative repositories.

Collection archives are repositories whose main aim is to gather applications and sharing them to the research community. Among them, we can cite VirusTotal [START_REF]Virus Total[END_REF], theZoo [START_REF] Nativ | theZoo -A Live Malware Repository web site[END_REF], VirusShare [START_REF] Virusshare | com -Because Sharing is Caring[END_REF] that gather all types [START_REF]Virus Total[END_REF] applications from Google Play and multiple other sources. Because goodware and malware are collected from these sources, AndroZoo provides additional metadata in order to distinguish them. These metadata is calculated using VirusTotal, where the number of detections by antiviruses and the date of analysis are provided. Even if these metadata cannot be considered a totally reliable ground truth, as antivirus detection may change over time, it has become the reference for experiments in Android malware analysis because of its size, its easy to use API for requesting applications and its ever evolving dataset, continuing to add more applications since 2016. Even if VirusTotal's detection accuracy is an independent problem that can be studied, we consider in is thesis that AndroZoo's metadata is reliable. Thus, in our contributions of this thesis, we use applications and metadata from AndroZoo.

Collaborative repositories have the aim to provide not only malware samples, but also a platform for researchers to share their samples with other members of the community. Such is the case of sites like MalShare [START_REF] Cutler | [END_REF], Koodous [START_REF]Koodous Project[END_REF], Malware Bazaar [START_REF]Malware Bazaar[END_REF], and Malpedia [START_REF] Daniel Plohmann | Malpedia: A Collaborative Effort to Inventorize the Malware Landscape[END_REF].

[62]: Silas Cutler (2013), MalShare

[63]: Koodous Team (2015), Koodous Project [START_REF]Malware Bazaar[END_REF]: abuse.ch (2022), Malware Bazaar Compared to markets, these collections are more useful for researchers, as well as being able to reference the samples to these repositories instead of hosting their experimental datasets. There is a trade-off with the ground truth of being malware with these repositories. On one side, smaller repositories have less malware samples but they are more accurate in the description of their malware samples, like in the case of Malpedia [START_REF] Daniel Plohmann | Malpedia: A Collaborative Effort to Inventorize the Malware Landscape[END_REF]. On the other hand, bigger repositories have more malware but their malware samples are less accurately described, like in the case of VirusShare [START_REF] Virusshare | com -Because Sharing is Caring[END_REF] [58]: VirusShare (2011), VirusShare.com -Because Sharing is Caring which relies in VirusTotal detections.

Experimental datasets These datasets where created for a particular experiment, and then shared to the research community. Particularly, researchers are interested in malware detection and family classification using machine learning techniques. Because there is theoretically a experimental dataset per experiment, thousands of datasets must be considered to make this section exhaustive. However, not all experiments share their experimental datasets, not even a list of hashes. We decided to take into account the datasets that are cited by others studies, because they are more discoverable than other ones, thus more prone to be used for other experiments.

The most significant of these experimental datasets is Drebin created by Arp et al. [START_REF] Arp | DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket[END_REF]. Their work focused on malware detection using machine used to evaluate a method for detecting similarities between Android applications. APIGraph [START_REF] Zhang | Enhancing State-of-the-art Classifiers with API Semantics to Detect Evolved Android Malware[END_REF], which is a framework to improve some "Android Adware and General Malware Dataset" used on an experiment of a new method that utilizes network traffic features to detect and classify malware, the CIC-InvesAndMal2019 [START_REF] Taheri | Extensible Android Malware Detection and Family Classification Using Network-Flows and API-Calls[END_REF] dataset that is a second part of the previous one from 2017 [START_REF] Habibi | Toward Developing a Systematic Approach to Generate Benchmark Android Malware Datasets and Classification[END_REF] with more features like permissions, intents, API calls and log files from dynamic analysis, and the CCCS-CIC-AndMal-2020 [START_REF] Rahali | DIDroid: Android Malware Classification and Characterization Using Deep Image Learning[END_REF] an Andriod malware family classification technique.

These datasets provide applications than can be used to reproduce their experiments, or as sources for researchers to take samples from and make their own. However, the use of experimental datasets come with some problems. One of the problems is that the calculation of the ground truth relies on VirusTotal. It have been shown that for Android, there is discrepancy between the expected detection rate of Android malware and the one from VirusTotal. Another problem is that static datasets tend to be useful for two to three years before they start getting old. This is the case for Drebin, which is widely used in the literature as a source of malware, despite containing samples from 2012. This influence the results of experiments using Drebin as we show in section 2.2.

Novelty datasets

The main motivation for the creation of these datasets is to provide one that has desirable properties not found in other research datasets. While experimental datasets may contain the metadata used in their experiments (such as the feature vectors for machine learning), in addition to the binaries of the applications, these novelty datasets are constructed explicitly to provide one with applications and metadata that the authors deemed rare or scarce.

For example, Genome [START_REF] Zhou | Dissecting Android Malware: Characterization and Evolution[END_REF] is a collection of manually analyzed Android is the most recent Android malware dataset, whose goal is provide one using malware reports, such as the ones provided by TrendMicro, ESET, Kaspersky and other, as the their ground truth, instead of using VirusTotal.

Discussion

Although there is a wide variety of datasets, very different from one another in terms of size and content, researchers construct their datasets with different properties (number of malware, number of goodware, date of the samples, etc) each time they perform a new experiment. On the contrary, all these malware archives, experimental and novelty datasets allow researches to either reproduce previous experiments, or take samples for their own experiment from a source that is recognized by other researchers in the field and that others can also take samples from. Using existing datasets is useful for the reproducibility of experiments, as these ones can be compared using the same data [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF] The variety of dataset results to be a problem too. Since there are too many datasets to choose from, researchers tend to select datasets like Drebin [START_REF] Arp | DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket[END_REF] and AMD [START_REF] Wei | Deep Ground Truth Analysis of Current Android Malware[END_REF], as they are widely used in many studies. It should be noted these datasets, Drebin and AMD, contain old samples, which may not resemble recent malware. This difference can be more properly appreciated using visual representation of the characteristics of their samples. A method to visually compare datasets is proposed in Chapter 3, where we compare these older malware dataset with newer ones.

Irregularities and biases in experimental datasets

Through the years, researchers created their experimental datasets and gathered applications by crawling the various markets, or by requesting access to other source or archive datasets. The choice of which Android applications should be included or excluded from an experimental dataset has an important impact on the measured performance of algorithms. An experiment can give bad results because of the nature of its input dataset. Conversely, it can artificially give good results with a specific dataset, but gives bad results with any other dataset. Both scenarios are undesirable effects that researchers would want to avoid.

In this section, we will study the impact of irregularities in experimental datasets and their influence in the experiments results. First, in Section 2.2.1 we categorize the different experimental scenarios in Android malware research. Then in Section 2.2.2, we describe the possible irregularities and biases that experiments may suffer due to its experimental dataset. Finally, in Section 2.2.3, we make a review of the literature and see that these irregularities still exist in experimental datasets.

Experimental scenarios

We have identified three types of scenarios typically experienced by security researchers. Each of these scenarios has various issues and therefore requires the use of datasets with different expected properties. applications have an API version lower than some threshold?". For this case, one needs to obtain a dataset that is representative of the market, where results of analysis on this dataset can be extrapolated to the whole market. Chapter 4 gives more details on how such a representative dataset should be extracted.

Markets profiling

Algorithm evaluation Some experiments aim to compute a characteristic of interest from an application. Examples of interesting characteristics include detecting malicious behavior in an application [START_REF] Grace | RiskRanker: Scalable and Accurate Zero-Day Android Malware Detection[END_REF], whether the sample is repackaged [START_REF] Gonzalez | Exploring Reverse Engineering Symptoms in Android Apps[END_REF], identifying its malware family [START_REF] Battista | Identification of Android Malware Families with Model Checking[END_REF], detecting similar applications [START_REF] Crussell | AnDarwin: Scalable Detection of Semantically Similar Android Applications[END_REF], whether it leaks personal information to the network [START_REF] Li | IccTA: Detecting Inter-Component Privacy Leaks in Android Apps[END_REF]. To answer these questions, researchers implement algo- rithms using various analysis techniques, and run them on a dataset to estimate their precision. This evaluation dataset should be representative (formalized later in Chapter 4) and labeled, i. e. the characteristic of interest for each application should be known beforehand. For any of the three kinds of experiments above, building an appropriate dataset is a non-trivial task. Before presenting our contribution for this task in Chapter 4, we have investigated the literature to see how researchers have built their datasets. Because the number of papers for the three experimental scenarios, market profiling, algorithm evaluation and machine learning is quite important, we have restricted our review to papers dedicated to machine learning scenarios. From these reviews, we extracted irregularities that we classified in the categories presented in the next section.

Machine Learning

Categories of irregularities

In this section we present the main categories of irregularities found in datasets that cause biases in experimental results. A bias is defined as a systematic error that influences the results of an experiment [START_REF] Dodge | The Oxford Dictionary of Statistical Terms[END_REF]. This error may be minimal, so that the experiment yields results very close to the real value. We could identify three main irregularities in input datasets: class imbalance, presence of clones and time incoherence.

Class imbalance

In supervised learning, a class is a property of an element that a machine learning algorithm uses to group different elements of the training set. It is also called category or label. In the case of malware detection, the possible classes of an application can be is a malware or is not a malware, and for malware family classification, each class correspond to the name of a family. Class imbalance occurs when a dataset contains significantly more units of one class than of others [START_REF] Gary | The effect of class distribution on classifier learning: an empirical study[END_REF]. An overrepresented class may push a learning algorithm towards preferring this class. For malware detection, if there are more samples of goodware, this has the effect of decreasing the recall but improving the precision [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF]. This problem has been addressed in the machine learning community [START_REF] Susan | The balancing trick: Optimized sampling of imbalanced datasets-A brief survey of the recent State of the Art[END_REF]: the overrepresented class is undersampled, i. e., some elements are removed while the underrepresented class is oversampled. The undersampling can be random or rely on a more sophisticated method, such as removing elements in dense clusters [START_REF] Tomek | Two modifications of CNN[END_REF]. The most popular [START_REF] Tomek | Two modifications of CNN[END_REF]: Tomek (1976), 'Two modifications of CNN' oversampling technique is SMOTE [START_REF] Nitesh | SMOTE: synthetic minority over-sampling technique[END_REF] that creates new instances from the interpolation of close samples. This technique would be interesting for machine learning experiments because it solves the class imbalance problem. Nevertheless, the new interpolated characteristics vector used in the experiments would not have any corresponding application. As a consequence, the application dataset cannot be provided by researchers, if is a problem for reproducibility of malware analysis experiments.

Besides, class balancing is mostly restricted to binary classes [START_REF] Susan | The balancing trick: Optimized sampling of imbalanced datasets-A brief survey of the recent State of the Art[END_REF], while we are interested in balancing more diverse classes.

In the case of a dataset evaluating a detection algorithm, class imbalance does not pose a threat to the validity of the results: since a representative dataset is needed for evaluation in this scenario, it does not need to be balanced, as the proportion of malware and goodware may be disproportionate but accurate to the reality of the set of applications the experiment is interested in.

Presence of clones Another irregularity is the presence of clones, where the abundance of several very similar apps in the training set prevents the algorithm from learning unique units [START_REF] Pfaff | Code Clones Considered Harmful? Quantifying and Exploiting the Effects of Code Clones in Static Malware Classifiers for JavaScript[END_REF]. Because experiments do to reduce redundancy in a dataset by extracting representatives of cloned units using mutual information and relative entropy.

Clones in a evaluation dataset to evaluate detection algorithm are admitted, since the samples are obtained in the proper way (as shown in Chapter 4), even if clones appear, they are representative of the applications in the target set of applications of interested for the experiment.

Time incoherence

This irregularity concerns the dates of the samples in the test and learning sets used in machine learning algorithms. It was shown that, when evaluating the accuracy of state-of-the-art algorithms, using a test set made of applications newer than those in the training set, the performance of these algorithms decreases [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF][START_REF] Fu | On the Deterioration of Learning-Based Malware Detectors for Android[END_REF].

[ Besides this, in the same study, the authors introduce a "spatial bias" which, according to them [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF], is a bias caused by the distribution of the class training and test sets. To remedy these biases, they propose three constraints that experiments should follow in order to avoid temporal and spatial biases:

▶ C1 "All the objects in the training must be strictly temporally precedent to the testing ones" [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF]: the samples in the training set must be older than those in the test set. ▶ C2 "In every testing slot of size Δ, all test objects must be from the same time window" [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF]: malware and goodware must come from the same time intervals. ▶ C3 "To have a realistic evaluation, the average percentage of malware in the testing must be as close as possible to the percentage of malware in the wild" [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF]: while the distribution of malware and goodware in the test set must approach as the one in-the-wild, it is possible to tune this distribution in the training set (to counter class imbalance).

Even if we apply these restrictions to build an experimental dataset, it is not guarantied that applications will behave the same way in the future. In [START_REF] Cai | Towards Sustainable Android Malware Detection[END_REF][START_REF] Cai | Assessing and Improving Malware Detection Sustainability through App Evolution Studies[END_REF] and [START_REF] Cai | Embracing Mobile App Evolution via Continuous Ecosystem Mining and Characterization[END_REF], Cai states that the main reason that existing Android malware detection techniques do not sustain well lies in their failure to account for the evolutionary dynamics of the Android ecosystem. Android malware classifiers may not be sustainable -they would need to be retrained constantly for later use, or their performance could downgrade over time. This is further explained on a recent study by Guerra-Manzanares et al. [START_REF] Guerra-Manzanares | Concept drift and cross-device behavior: Challenges and implications for effective android malware detection[END_REF]. A supplementary problem is one in [START_REF] Zhang | Enhancing State-of-the-art Classifiers with API Semantics to Detect Evolved Android Malware[END_REF],

[106]: Guerra- over time by making necessary updates to their detection models with evolving feature sets. For that purpose they maintain a pool of different detection models initialized with a set of labeled applications using various online learning algorithms.

Discussion

These three irregularities can be threats to the validity of experiments if present in the input dataset of an experiment. Researchers should take into account these irregularities, so that the results of their experiments give an appropriate result. In the case of an algorithm evaluation, we see that none have a significant impact on this type of experiment we are interested in. Nevertheless, a representative dataset must be used in order to ensure pertinent results when evaluating a detection algorithm. We show how to obtain a representative dataset in Chapter 4. Furthermore, we use the constraints C1, C2 and C3 proposed by Pendlebury et al. [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF] in our experiments in Chapter 5.

For machine learning, these irregularities are concerning and should be addressed.

Results of evaluation of the literature

We saw that researchers have plenty of sources to take applications and perform their experiments. Taking samples from heterogeneous sources may introduce irregularities into the built dataset. To confirm this hypothesis, we made a review of different experiments to see how they constructed their datasets.

We reviewed 28 articles published between 2016 and 2019 to evaluate if these irregularities may have an impact on past results. Table 2.1 presents the details of this review.

We extracted the following information from each article:

▶ Column "Paper" gives the name of the tool or the authors' names. ▶ Next, column "Goodware source" contains the datasets used in the experiment. Sometimes, several sets are used in a paper: we note these subsets D 𝑖 and we give their size in other columns accordingly. ▶ Next, columns "GS year" and "Nb Goodware" are the year range and number of goodware according to the article, if available. ▶ The next three colmuns give the same information for the used malware. ▶ Next, columns "Training set" and "Test set" correspond to the number of applications used in the respective sets. ▶ The notation "(M/G)" reports the Malware/Goodware for each experiment. The use of "𝑘-fold cross validation" will be specified in these columns. Any other special cases are reported in these two columns. Because experiments do not usually share the list of used applications, it is not possible to verify the presence of clones in their datasets. However, we can notice that 3 experiments use Drebin and Genome at the same time, and Genome is a subset of Drebin, thus having the algorithms trained and tested with several instances of the same APKs.

We notice that the test dataset is constructed from the input dataset in various ways. One common way is to perform a 𝑘-fold cross validation, where the input dataset is split into several subsets. If the input dataset is not a representative dataset of the studied set of applications, the subset of the 𝑘-fold split, and therefore the test set, will not be representative either. This invalidates results aiming to the studied set of applications.

But, since we want to avoid class imbalance in the training set, the class proportion would probably be not the same as the studied set of applications, so avoiding class imbalance and being representative are incompatible properties for a training set. On the other hand, if the input dataset is a representative dataset of a market, the classes/labels is a malware/is not a malware are most of the time unavailable at the time of creating the representative dataset. This problem of crafting a representative and labeled is test dataset for machine learning, which also works for evaluating an algorithm, is tackled in Chapter 5.

Discussion

We saw the types of scenarios researchers in security encounter, namely market profiling, algorithm evaluation and machine learning.

Specially for this last one, we saw the different irregularities that experimental datasets introduce to experiments, with proposed methods to mitigate these irregularities. Nevertheless, there is one bias that none of these experiments address: in the case of an experiment claiming that it works on a certain set of applications (e. g. a market), its test dataset must be either the whole set of applications or a sample of this set such that the results can be extrapolated to the whole set [START_REF] Arp | Dos and Don'ts of Machine Learning in Computer Security[END_REF] [134]: . The previously cited works do not solve the problem of researchers having to evaluate their algorithm on an up-to-date dataset. Of course, this problem is different than adapting a classifier to an evolving market -even if this problem is of independent interest. Researcher would primarily focus on getting a dataset that mimics the whole set of applications that they targets.

Additionally, the ground truth of the dataset should be known, for example with a malware/goodware label if the study concerns a detection algorithm. These labels can be missing in the source from which applications are picked. As a consequence, we propose in the following to define the notion of representativity of a dataset against the population it represents. A representative dataset has to be used to validate an experimental results at the time of the experiment. The representativity property is linked to a population at a period of time. Pendlebury et al. [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF] has recommendations for this irregularity related to time, namely time incoherence. A dataset that is representative at a date will not necessarily be so in the future because the population may change significantly. Such a representative dataset will help the researcher to assess results for real-world scenarios [START_REF] Allix | Are Your Training Datasets Yet Relevant?[END_REF]

Conclusion

In this chapter we explored the different sources of applications researchers use to create their experimental datasets. Markets are the prime source of recent, up-to-date applications. Although they are not free of malware, they provide a view of what to expect in the wild (at least for a specific market). In order to allow experiments be reproduced, a handful of researchers have provided their experimental datasets, which have also contributed to help other researchers obtain samples that would be otherwise not available. In addition to this, efforts to provide samples for researchers are still ongoing, with datasets like ViruShare, AndroZoo, Malpedia, and Malware Bazaar, aiding the reproducibility of experiments by hosting a large quantity of samples. We saw the differences between all these datasets in terms of size, dynamicity, availability, content, and experimental goal. Despite this, a more intuitive approach for comparing datasets could be performed with analysis data from their samples. This would allow to compare datasets based on their content, where we could find differences in the amount of applications with certain interesting properties. Such a comparison, and eventual exploration, of datasets can be done using visualization techniques. This can be done using our approach discussed in Chapter 3. We will see that old and new datasets are different in terms of certain properties, and that malware datasets alone are different from a population we fixed to study.

We saw the various experimental scenarios that researchers face, and the different irregularities that affect experimental results if these are present in the input dataset. While the landscape of research datasets seems to be booming, trying to provide ones with more and richer information, and with solutions to known irregularities, none of these studies seem to address the problem of representation, i. e. the usage of a dataset to represent another one. This representation could be useful to perform the following: instead of testing a machine learning model or an algorithm over a market, a "representative" dataset of a studied population (e. g. a market) could be used in place, allowing to obtain results using a sample that are very similar as if the test would have been performed to the population itself. A method for creating such a representative dataset is detailed in Chapter 4. Even so, samples from market do not have the properties available immediately to be used for testing. Such properties are mostly available in researches datasets that are biased, that is, they are not representative of the studied population. In order to use a dataset with samples from research datasets, it has to be debiased so that it resembles a representative dataset. We developed and test an algorithm to perform this procedure. We tested a debiased dataset produced by this technique against other established datasets, and results show that our debiased dataset have a better performance overall than the other datasets used in the test in Chapter 5.

Contributions

Visualization of Android Malware Datasets 3 With millions of Android malware samples available from many different datasets, researchers have a large amount of applications to perform malware detection and classification, specially with the help of machine learning. Others take advantage of newer samples to perform reverse engineering and study in detail the malicious behavior of these applications. Choosing a proper dataset for an experiment is not a trivial process, and the fact that there are various, and widely different datasets to get applications from make this task harder. Researchers would appreciate to benefit from an intuitive way to differentiate or examine Android malware datasets, which would help them obtain an overview of the contents of these datasets, and help them choose one with the desired properties, e. g. the right amount of a certain characteristic or set of characteristics, for their experiments.

Visualization tools can help to obtain meaningful information from the large amount of data contained in a dataset. Representation of datasets allow their exploration and comparison, taking advantage of our visual system to easily spot differences between the represented values. This is a first step that helps researchers to understand the content of a dataset and infer the differences between datasets.

In this chapter we introduce a new visualization tool called DaViz. This tool is aimed at providing visual representation of datasets for exploring and comparing them. This would allow users to obtain useful information for certain tasks associated with datasets, like understanding their composition and the differences between them. In Section 3.1 we start by discussing the other visualization methods in the literature and understand the limitations of these tools for the tasks we want to perform. Then, in Section 3.2, we propose our solution to these problems, DaViz. Finally, we visualize datasets from the literature in Section 3.3, where we highlight how different datasets can be between each other and through time. This work has been published in the RESSI 2022 conference [START_REF] Tomás Concepción | DaViz: Visualization for Android Malware Datasets[END_REF] [12]: Concepción Miranda et al. (2022), 'DaViz: Visualization for Android Malware Datasets' .

Motivation for Android malware datasets visualization

In order to take advantage of the abundance of applications available to study, researchers need to be able to navigate through large amount of applications in a dataset, often comparing datasets between them. In this section, we analyze the tasks involved when researchers examine datasets, then we inspect the literature in Android malware visualization for tools that aid in the achievement of these tasks.

Tasks and objectives for visualization

Let us imagine a situation where, in a research group or in an industry group, a dataset of unknown APKs is given to researchers working in malware analysis. A first step is to examine the content of this dataset, e. g. what types of applications can be found, what are their characteristics, whether there is any known malware/benign samples, etc. Once they recognize the content of a dataset, researchers can then compare it against other datasets. This comparison would allow them to discover similarities to other datasets and find correlations between them, or differences that would require a further study of the dataset in question.

From the previous remarks, we identify two main analysis tasks that researchers perform with datasets:

▶ an examination task that we call dataset exploration, ▶ a comparison task that we call dataset comparison.

We investigated how such tasks should be represented by following the methodology by Muzner [START_REF] Munzner | Visualization Analysis and Design[END_REF]. Using her methodology, we took [START_REF] Munzner | Visualization Analysis and Design[END_REF]: [START_REF] Munzner | Visualization Analysis and Design[END_REF], Visualization Analysis and Design into consideration the volume and type of data to visualize, and the type of tasks we found. With this knowledge, we examined the types of visualization techniques (e. g. charts, graphs), alongside their interactions (e. g. selection, filter, zoom) to proposed a visualization method suited to the data, the tasks and the possible interactions using the tool. During this phase of investigation using the methodology of Muzner, we extracted the following properties about the analysis tasks:

Dataset exploration The first task, dataset exploration, has a "discovery" and "exploration" goals according to the different goals defined by Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF]. By "discovering" datasets, users can gain new knowledge, verify hypothesis, or generate new sub datasets from the data at hand. For example, users can verify that more than half of the applications contain a specific set of characteristics. Users can also just "explore" the datasets when they do not know what they are looking for and where. This can be useful to users that look for outliers or unanticipated patterns in the data.

Dataset comparison

The second task is associated to a "comparison" goal according to Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF]: once users choose the datasets they are interested in, they can create charts to show differences between datasets. Multiple datasets can be compared at once, but a one by one comparison allows a more focused approach by concentrating on only two datasets at a time. Users can verify visually the difference of two datasets by comparing the proportion of applications with certain characteristics, or the presence of characteristics in either dataset (e. g. whether they characteristics A and B, A and not B, or B and not A).

Malware analysis visualization techniques

Now we examine the literature for visualization techniques for Android malware that fulfill both exploration and comparison tasks. The use of visualization for malware analysis is not new [START_REF] Yoo | Visualizing Windows Executable Viruses Using Self-Organizing Maps[END_REF], whether the results are [START_REF] Yoo | Visualizing Windows Executable Viruses Using Self-Organizing Maps[END_REF]: [START_REF] Yoo | Visualizing Windows Executable Viruses Using Self-Organizing Maps[END_REF], 'Visualizing Windows Executable Viruses Using Self-Organizing Maps' or for aiding malware analysts understand what an application does and how [START_REF] Quist | Visualizing compiled executables for malware analysis[END_REF]. We divide these methods into two groups: those for visualizing a single application and those for visualizing sets of applications.

Visualization for single applications

In this part we present techniques that show a visual representation of a single application in general.

2D image of an APK's file structure Jain et al. [START_REF] Jain | Enriching reverse engineering through visual exploration of Android binaries[END_REF] propose a method to visualize the program's .dex file into a color 2D image to help recognize patterns in the bytecode. Figure 3.1 shows five different applications (a to d) in form of figures using the method elaborated by Jain et al. [START_REF] Jain | Enriching reverse engineering through visual exploration of Android binaries[END_REF]. We see a representation of APK files as a 2D image, where the sections of the application are shown with different colors: the header of the file is shown as a red line in the top left corner of the image, followed by the index list of strings, types, prototypes, fields, methods and classes in different shades of green, and then the data section. In this last section we can find the string list, where string constants, class names and method names can be found. In Figure 3.1 it is encircled in red. Applications produced by DexGuard tend to have their string list before classes code in the data section, as shown in Figure 3.1 (a, b), whereas it should normally be after the classes code (like in Figure 3.1 (c, d, e)). These types of patterns can be easily detected by experts when searching for applications obfuscated using DexGuard. present a tool to visualize artifacts of an Android program (control flow graphs, system flow graphs, information flow graphs, or a combination of these) in order to help an analyst studying an unknown application decide if it violates confidentiality, integrity or availability. As an example of view proposed by the tool, we explain part of the case study of a confidentiality leak in an application. Part of the process involves the hypothesis that the application sends sensitive information in a message and sends it via the method Message.sendToTarget. To verify this, users can check the callers of this function to discover which other function is responsible for the possible information leak. Figure 3.2 is a "reverse call graph", where it shows all the functions that call sendToTarget. It the graph, we can see that this method is called by Camera.PreviewCallback.onPreviewFrame, an API overridden by the application. By inspecting the PreviewCallback.onPreviewFrame, users can see, in this example, that the preview image used by this method is sent using Message.sendToTarget, thus in part validating the hypothesis.

GroDDViewer: dynamic analysis replayer Lalande et al. [START_REF] Lalande | GroDDViewer: Dynamic Dual View of Android Malware[END_REF] propose GroDDViewer for representing the dynamic execution of an application's system flow graph with a replay option, alongside its method control flow. Figure 3.3 shows an example of the tool's interface analyzing a ransomware, in this case a Simplelocker sample. On the left side, we see the "system flow graph", a graph showing the interactions between We mention with a more brief description other tools used in Android malware analysis, since these tools do not present much interesting information about an application. propose a tool to better understand ICC and intents in an application. These tools use diverse techniques to deliver information for very specific tasks. Different parts of an application are exploited to obtain meaningful information about the studied application.

These techniques are useful for analysts as they provide a visual aid for the analysis of an application. Analysts can also extract significant information from a single program by using these tools. However, these methods can not be directly used to visualize a collection of applications at once.

Visualization of datasets

Instead of visualizing a single sample, other visualization tools focus on a set of programs. As far as we know, there are no tools dedicated for Android applications for this type of task, however there exists tools tested with x86 programs [START_REF] Saxe | Visualization of Shared System Call Sequence Relationships in Large Malware Corpora[END_REF][START_REF] Gove | SEEM: A Scalable Visualization for Comparing Multiple Large Sets of Attributes for Malware Analysis[END_REF]. Thus, we decided to review these tools in order to see if they meet the same goals than our tasks, and if some visualization method could be adopted. The following papers try to represent a collection of applications, by either presenting information of every element it has, by showing similarities between subsets of elements, or both. shows similarities between applications in a malware dataset, according to their sequence of system call sequences. Semantic sequences of system calls in logs are calculated, and then compared between the different applications in the dataset. Figure 3.4, taken from the study, shows an overview of the tool's interface with their different parts. On the top left we see various colored blocks besides the name of various samples, each block representing a sequence of system calls, with unique colors for different sequences. On the left side, a filter panel of behavioral traits such as "registry key modifications, network communications, and file drops" allows to highlight the samples on the right that contain this behavior. And on the right is a grid called "Sample Similarity Map", computed using principal component analysis and project in 2D using a Hilbert curve. The grid shows the samples grouped in such a way that similar samples are put together, and take identical colors, with known samples as circles, and unknown ones as squares. It allows users to inspect a dataset for applications with similar behavior based on their system call sequences, where they can visually recognize similar samples grouped together by their colors. Although this tool allows the exploration of a single dataset, it does not allow for comparison of various datasets at the same time. Despite the ability to compare the samples a dataset between them, the tool by Saxe et al. [START_REF] Saxe | Visualization of Shared System Call Sequence Relationships in Large Malware Corpora[END_REF] does not allow a comparison between sets of applications. And on the other hand, with the tool by Gove et al. [START_REF] Gove | SEEM: A Scalable Visualization for Comparing Multiple Large Sets of Attributes for Malware Analysis[END_REF], an exploratory task is not possible without having a comparison of one sample with a dataset.

Similarity in system call sequences

Discussion

Visualization tools for single applications provide analysts with visual aid when performing reverse engineering and manual malware detection and classification, but they lack the ability to visualize more than one application at a time. Visualization for sets of applications so far enable researchers to: compare samples between them, or compare a single sample to a dataset.

For now, the tools for comparing features are specialize on x86 applications. For the comparison and exploration of Android malware datasets, no tool performs these tasks simultaneously.

DaViz -Dataset Visualization for Android malware datasets

So far we saw in Section 3.1 the tools used to visualize Android applications and datasets. We realized that there are no tools in the state-of-the-art for exploring and comparing Android datasets. This section presents DaViz * , shown in Figure 3.6,a new visualization tool with the goal of aiding in the exploration and comparison of Android malware datasets. We know intuitively that researchers in malware analysis can identify differences in datasets when the characteristics of these are shown visually. This new knowledge can help researchers select better datasets fitted for their experiments than randomly selecting applications from different sources, or using an already made dataset.

Section 3.2.2, we explain the characteristics used as input data for visualization. Next, in section 3.

2.3, we explain the different charts used by

DaViz and what type of information can be obtained from them. After that, in section 3.2.4, we explain the interactivity features exploited by DaViz. Later, in section 3.2.5, we describe the technical implementation of DaViz. Finally, in section 3.3 a use case to exemplify an usage scenario of DaViz. chose the characteristics to display. Finally in the left, there is a "Current selection", which display the characteristics chosen by the user before creating the chart. Since there is currently no characteristic chosen, "Current selection" display "(none)"

Interface

Below is the chart section of the display, where the charts are created. We can see three datasets displayed at the same time:

▶ AndroZoo 30k 2020, an extract of 30,000 applications from Andro-Zoo in 2020, ▶ VirusShare 2018, a collection of all Android malware collected by this platform in 2019, ▶ and AMD, a collection of malware from 2010 to 2016. AndroZoo 30k 2020 is hidden, but can be unfolded by pressing the triangle button next to its name. Below is VirusShare 2018 showing three charts from left to right: ▶ a Venn diagram presenting the relationship of proportions of applications with the characteristics "Use reflection", "Use Accessi-bilityService", "DexClassLoader", and "LoadClass", ▶ a bar chart displaying the number of applications by MinSDK version, ▶ and a heatmap showing the proportion of applications that have "Use Java JNI", "Recieve SMS", "Send SMS" on abscissa, and "Use MMS", "Place calls", "Get CPU ABI", and "loadClass" on the ordinate.

And below this one is AMD, showing the same charts for this dataset.

On the top left corner there are four elements to change the current view. First, on the right, a dropdown list displaying "List View", which allows the user to select between this current view ("List View") and a "Compare View". Next, on the right, there are three icons:

▶ the left one allows to save a workspace with all the selections applied to the last chart, ▶ the middle one allows to switch between saved workspaces, ▶ and the right one allows to save the session, including their workspaces, for future use.

Input data: characteristics

The data we used to represent the datasets are characteristics: properties extracted from applications, e. g. their size, the number of classes they contain, whether they make a specific API call, etc. These characteristics are computed using Droidlysis [START_REF] Apvrille | Pre-filtering mobile malware with heuristic techniques[END_REF], a "property extractor for Android apps" † : for each application it outputs different boolean characteristics from patterns in the code, alongside other ones from the application's Manifest file (like permissions, intents, activities, application size, number of classes, etc.). Some characteristics, like the date and size, are shown in intervals parameterized by the user, like the date in the range of years, while the rest of characteristics with boolean attributes are shown in a characteristics tree, as we explain later. New characteristics can be added accordingly, by modifying Droidlysis or by using other analysis tools that output data about programs. Using this output data from Droidlysis, we can calculate the proportion of applications that have a certain characteristic in the dataset. With the computed proportions, we propose several visualization methods, using user interactivity.

Since there are 171 characteristics calculated using Droidlysis, we decided to arrange them in categories when the user is invited to select the set of characteristics to display. They are displayed as a tree, divided in categories, sub-categories, and finally the characteristics as shown in Figure 3.7. These categories are the following:

▶ "code": These are code-based characteristics that represent what the application can potentially do. • "code_exec": These are actions linked to external code execution, e. g. any "change" command (chmod, chown, chattr), load a DEX class, start activity via shell, etc. • "crypto": These are properties linked to the usage of cryptography, e. g. compute CRC32 and traces of encryption schemes usage (cipher method and key specification). • "exploit": These are traces of known vulnerabilities found in several CVEs specific to Android applications, e. g. CVE-2020-0069, CVE-2012-0025, among others. • "info_leak": These are methods to allow attackers obtain information collected by the application, e. g. usage of SSH, create a network socket, usage of HTTP GET, etc. • "native": These are properties linked to the usage of native code in the application, e. g. usage of Java JNI, load native libraries, use of BusyBox, etc. • "obfuscation": These are traces that indicate the use of methods to hinder the analysis of the application, e. g. use of reflection, use of APKProtect, usage of base64 strings, etc. • "payload": These are methods mostly used in the application's payload, e. g. send and recieve SMSs, stop/kill processes, usage of cryptominers, etc.

▶ "intelligence": These are information collection methods the application potentially can perform.

• "intel_device": These methods obtain information related to the current device the application runs in, e. g. IMEI, hardware information, bootloader's version, etc. • "intel_execution_environment": These methods obtain information related to the application's execution environment, e. g. emulation detection (like Andydroid, bluestacks, QEMU and other emulators), hardware sensors, check for debugger, etc. • "intel_network": These methods obtain information related to network connectivity of the phone: e. g. MAC address, WiFi SSID and RSSI, scan for WiFi hotspots, and cellular operator name. • "intel_user": These methods obtain information related to the user: e. g. call logs, IP address, phone email, SIM country, access to microphone, etc.

▶ "misc": These are other miscellaneous characteristics that could not fit in the last two categories.

• "integrity": These are actions the application does that modifies the system in one way or another, e. g. install an APK, mute microphone, place calls, bypass idling mechanisms, use adb internally, etc. • "interesting_behavior": These are actions the application performs that may be of interest in combination with other characteristics, e. g. IP addresses found in files, phone numbers found in files, call intent chooser, parse URI, etc. Users specify both abscissa and ordinate before creating the chart. In the case of characteristics such as size and the application's date year, users specify the range for each bar.

According to Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF], bar charts are used to lookup the values and compare them. In the dataset exploration task, users can identify which characteristics are present the most or the least in a dataset. On the other hand, in the dataset comparison task, users can compare which characteristics are more present in one dataset than the other, or equally present in both. These observations can lead to further investigation concerning the characteristics found to be, for example, more present in a goodware dataset than in a malware dataset. However, with bar charts, it is not possible to make the same analysis for proportion of applications with more than one characteristic simultaneously. Heatmaps These charts show the magnitude between two dimensions with a color hue. An example of a heatmap chart is given in Figure 3.9. This example shows that the dataset contains mostly applications that use the Java JNI, and inform Android which application binary interface (ABI) it uses to load the correct native program or library (in x86, ARM or MIPS). Each cell represents the proportion of applications that have the two corresponding characteristics in abscissa and ordinate at the same time in the dataset. A "third" dimension is used, the color hue, to represent a value corresponding to one element of the abscissa and one of the ordinate. Users can specify the different characteristics for the abscissa and ordinate, and lastly the value for each intersection (number of applications having both characteristics, the average size, etc.).

According to Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF], heatmaps are used to find clusters and outliers, but also to summarize the data. In the dataset exploration task, users can compare, within the dataset, the proportion of pairwise presence of several characteristics in applications at once. For the dataset comparison task, users can compare, between datasets, these pairwise proportions to identify a difference in presence of the chosen characteristics. For example, when looking for a pair of characteristics highly used at the same time, the heatmap will show a cell with a more intense color than the rest of the cells, which can be spotted easily. When this chart is used for dataset comparison, users take advantage of the color intensity to locate proportion differences with pairs of characteristics between datasets.

Although with heatmaps we can observe the proportion of applications with two characteristics in a dataset, we show in the following how it is possible to generalize these characteristics' relationships using Venn diagrams.

Venn (Euler) charts

In order to see the proportion of applications that have multiple characteristics, we have to treat each characteristic as a set of applications that present it. A common way to do this is through the use of Venn diagrams. They allow to see the set relationships between different characteristics as sets, where the size of each set corresponds to the number of applications with that characteristic. The methodology by Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF] does not cover the use of Venn diagrams. However, the study made by Alsallakh et al. [START_REF] Alsallakh | The State-of-the-Art of Set Visualization[END_REF] explores the uses of Venn diagrams to show and present sets. They propose different tasks that users may want to perform with these sets. These tasks can be related to elements, to sets and set relations, or element attributes. Among the tasks established by Alsallakh et al. [START_REF] Alsallakh | The State-of-the-Art of Set Visualization[END_REF], DaViz allows to perform the following tasks related to sets and set relations: Venn diagrams allow to identify the relationship of the chosen characteristics: if many of them are mutually exclusive, they will appear separate from each other. On the contrary, if they are highly present in a group of applications, the diagram will show them overlap each other, up to the point of perfectly fit in each other if there is a one to one correlation between (e. g. a group of applications all have these characteristics, but there are none without one of them). In the dataset exploration task, this helps to visually identify correlations between several characteristics. In the dataset comparison task, these correlations can be compared, identifying differences in the usage of a set of characteristics between datasets.

▶ (B2)

Interactivity

Interactivity is one of the key features of DaViz: once a chart is created, users can create more charts that will appear from left to right (or from top to bottom) of the first chart, and an interaction performing a selection on one of the charts will affect the charts that follows (on the bottom or on the right, depending of the chart's flow).

The intent behind this behavior is to allow users select sub-parts of the datasets based on the selection of characteristics in a chart. This will affect all the following charts of the flow, as they will only consider elements with the selected characteristics in the previous charts where the selection was made. For example, Figure 3.11 shows, on the left, an extract of F-Droid in 2021 with a Venn diagram followed by a bar chart on the bottom. The chart shows the proportion for 3,587 elements. When we select the set in the middle of the Venn diagram, the intersection of "Get SD card state" and "Get phone IMEI", the bar chart is updated, as seen on the right, to consider only elements with these two characteristics.

In this case, we get, as seen in the bar chart, a higher concentration of applications that use native methods, among a new total of 82 elements. This principle of interaction have been implemented for all three types of charts: bar charts, heat maps, and Venn diagrams.

We took inspiration from the study by Alsallakh et al. [START_REF] Alsallakh | The State-of-the-Art of Set Visualization[END_REF] to implement this behavior in DaViz. The study formalizes the tasks related to elements in Venn diagrams:

▶ (A1) Select elements that belong to a specific set. By clicking in the diagram, the user can select those elements, filtering out those that do not belong to the set. ▶ (A3) Select elements based on their set memberships. Users are allowed to select internal parts of intersections, without taking into account certain sets that are of the intersection. ▶ (A5) Filter out elements based on their set memberships. This is done by selecting sets or intersections in the diagram. The resulting selection appears in the next diagrams, as explained earlier.

Additionally, a specific interaction action have been implemented for Venn diagrams for displaying elements when they are not concerned by the characteristics, and thus not displayed. In Figure 3.10a, we can see that the "Exec command" is the biggest set (the characteristic is present the most in comparison to the others), with almost all other sets inside it, meaning that "Exec command" is always present in an application if these other are. We can see a difference in the intersection of "DexClassLoader" and "loadClass" between the two datasets: in AndroZoo 30k 2020 there is less use of both characteristics simultaneously than in VirusShare 2019. Notice that the sets' sizes represent the proportion between them in one dataset, which can be misleading for comparison: the sets' sizes cannot be visually compared because the number of considered applications is not the same. In order to fix this, users can include the whole dataset to the diagram. By selecting the scale option, the previous Venn diagram changes to Figure 3.10b. Now we can see that the "Exec command", although is the most present characteristic among the others selected, is not as present as in the malware dataset, where almost every application Android APK

.12: Diagram of the system implementation of DaViz use it. The user can conclude that the VirusShare dataset contains mostly malware that try to execute code more frequently compared to the general case of applications in AndroZoo, and further inspection may be performed.

As a summary, DaViz can perform a one-to-one comparison of datasets using bar charts, heat maps and Venn diagrams. DaViz's chained charts allow users to navigate datasets with different characteristics in each chart. Moreover, the selection of characteristics in the charts allows to concentrate the view on the selected characteristics in subsequent charts. This can help filter out applications of a dataset that do not apply to the criteria selected by the user, and consolidate those of interest for further inspection.

Implementation

.12 shows the diagram of the implementation for DaViz, showing the flow of data from Droidlysis analyses of APKs to the display of charts to users. We analyze each application of the datasets with Droidlysis, and store the results in a MariaDB relational database. The data is structured so that each application is represented by its SHA-256 hash value.

When users create their charts, the data has to be received as fast a possible. Delays in generating changes in the charts would impact the interactivity of the tool. To allow a more responsive response, the Elastic stack is used. It is composed of a data collection tool called Logstash and the Elasticsearch search engine. The former is used to transform the data from MariaDB to the data interpretation used by Elasticsearch. This last one handles the requests from the display client. Elasticsearch, on the other hand, is fast enough to allow rapid changes in the charts with short delays between a click on the chart and the corresponding reaction on it. 1 1: An earlier version of DaViz used MongoDB instead of Elasticsearch, and it suffered from performance issues.

A third part is the display client that the user will interact with, which is based on React to arrange the user interface and make requests to the Elasticsearch instance. It also uses the D3 library to create the charts and manage the interactions with them, like clicking on a bar or one of the intersections of the Venn diagram. Because analyses to malware must be performed in a secure environment, and to preserve the data from external attackers, all these instances are hosted in the High Security Laboratory (Laboratoire de haute sécurité or LHS) at INRIA in Rennes, France.

In the next section, we will explain our usage of interactivity to help users take advantage of the tool.

Discussion

With the variety of charts and characteristics available, users can gain knowledge from unknown datasets, by inspecting the different concentration of applications with selected characteristics in the dataset. Users can also examine the relationships between different characteristics inside a dataset. Additionally, the use of a fast implementation permit to change rapidly the views with new charts and filters, allowing users to explore continuously the data, which helps in the verification of hypotheses. This accomplishes our "Dataset exploration" goal in accordance with the methodology of Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF].

By looking at the graphs, we see the differences in the proportion of individual characteristics between datasets, and differences in the proportion of multiple characteristics between datasets. This information can be useful to researchers when they create their experimental datasets. On one hand, they can inspect datasets to see the relationship between the characteristics of interest and other ones. This would allow researchers to identify unexpected patterns about the presence of these relationships. On the other hand, researchers can compare their newly created dataset to another one, to see whether their dataset does not differ with the expected proportion of characteristics of another one. This accomplishes the "Datastet comparison" goal in accordance with the methodology of Munzner [START_REF] Munzner | Visualization Analysis and Design[END_REF].

With the variety of characteristics available, researchers can systematically explore a dataset. While doing this, they can also compare datasets using the same charts. The use of interactivity and fast implementations allow to rapidly make changes in the charts. A comparison between other datasets from the literature would further exemplify the utility of this new tool.

Visualization of datasets in the literature

With DaViz explained in the last section, we now show its use with a comparison of Android malware datasets. Several datasets from the literature were analyzed and imported into DaViz. In this section we are going to compare some of these datasets and see the differences between them. We start by comparing older and newer malware datasets to identify and discover key differences between them, and then comparing malware datasets from the literature with AndroZoo.

Differences between older and newer malware datasets

We are going to compare Drebin and AMD, particularly old but still widely used Android malware datasets, with applications dating from 2010 to 2012, to VirusShare 2018, a collection of all Android malware collected by the VirusShare platform in 2018. We will see the changes in the composition of these Android datasets malware dataset through time.

This first chart in Figure 3.13 shows the seven characteristics, automatically computed by DaViz, that have the biggest proportion difference between these datasets. At first glance, we see that around 60% of all applications VirusShare 2018 contains one of these characteristics, while Drebin never exceeds 20%. Among these, there is "Exec command", "Load DEX class", "Change command", "Load class". These are mainly code execution techniques used to hide the payload somewhere else, and then load it when the malware is ready to activate its malicious activity. On the other hand, few applications in Drebin have these characteristics. From this, we can assume that execution methods have changed over the years from the samples in Drebin to VirusShare 2018. And so, applications from Drebin are less complex compared to those in VirusShare 2018. As a consequence, we can suspect that studies using Drebin as an evaluation dataset will have significantly different results, compared to an evaluation using VirusShare 2018.

Let us examine the influence of "Change command" 2 , one of the charac-2: "Change command" refer to any of either chown, chmod, chgrp, chcon or chattr patterns found in the application's code teristics with the most concentration in VirusShare 2018, in these datasets: we click over "Change command" in the bar chart of Figure 3.13 and we obtain Figure 3.14. In this last figure, we can see that the number of applications in Drebin is reduced to 168. Indeed, it is only 3.02% of Drebin, compared to 66.64% of VirusShare 2018.. Nevertheless, with these few samples, we see the use of "Start activity via shell" and "Load native library" in almost the same proportions in Drebin as in VirusShare 2018 in relation to "Use Java JNI", but with more applications using these characteristics at the same time in VirusShare 2018 than in Drebin. We also see that "Load native library" and "Use Java JNI" are present simultaneously in both datasets. These observation indicate a shift in the behavior of applications to one using more native methods.

The same analysis is done with AMD and VirusShare 2018. Figure 3.15 shows again the seven characteristics, automatically computed by DaViz, with the biggest proportion difference between these datasets. Even if samples in AMD are older by two years, the figure presents the same characteristics as the previous comparison of Drebin and VirusShare 2018, just with a slightly different order. We can see that characteristics such as "Detect screen off", "Load DEX class", "Load class" and "URL in executable" are less used in VirusShare 2018 compared to AMD.

Again, we examine the influence of the "Change command" in these datasets as seen in Figure 3.16. In this case, "Change command" is present in only 2.21% of AMD, or 526 of samples presenting this characteristic, while this characteristic is present in 66.64% of VirusShare 2018. In both AMD and VirusShare 2018 we see that "Load native library" and "Use Java JNI" are both present, almost with the same proportions. Although "DexClassLoader" and "Start activity via shell" are present in AMD, we can see that there is more presence of these characteristics in VirusShare 2018. This evidences a change of behavior, where newer applications tend to use more native programs to hide their malicious behavior. Now we compare Drebin to AMD using previous figures. Comparing Figure 3.13 and Figure 3.15, AMD has more "Detect screen off", "Load DEX class", "Load class" and "URL in executable" than Drebin. And comparing Figure 3.14 and Figure 3.16, "Change command" is present in only 2.21% of AMD, which is lower in proportion compared to Drebin but with more samples (526 vs 168). We also see applications with "DexClassLoader" and "Load native library" in AMD, whereas in Drebin there are very few with "Load native library" or none with "DexClassLoader". Again, there is a trend of more recent malware to use native programs and loading external classes than in older malware samples.

We see a significant change in malware characteristics from Drebin to ViruShare 2018: newer applications tend to use more loading techniques than older ones, most probably to hide its malicious execution, with only few old applications that use these techniques. With AMD, these differences were also the same for some characteristics, as there are no significant changes between this dataset and Drebin. And although there are less applications in AMD and Drebin that use native libraries than VirusShare 2018, in all three datasets the use of JNI and library loading are present simultaneously.

Differences between literature datasets and AndroZoo

In this section, we intent to compare the same specific datasets, Drebin and AMD, with the "target" dataset that represent markets. Because markets cannot be easily retrieved, we use AndroZoo for this purpose. Indeed, as we said in the previous chapter, AndroZoo is a collection of applications from many sources, particularly from Google Play and Chinese markets. This dataset broadly represent applications found in the wild, that is, in environments where users can download them. As with any other market, malware can also be present in a AndroZoo sample, as they could be undetected at the moment AndroZoo collected them, so malware may appear in the sample. Drebin, being a malware dataset with applications dating between 2011 and 2012, should present a significant difference in certain characteristics with respect to AndroZoo.

Figure 3.17 shows the seven characteristics with the biggest proportion difference between Drebin and a 30 000 applications sample of AndroZoo in 2020. The biggest difference between these datasets is the lack of "Google Play services" in Drebin versus 60% of applications in Andro-Zoo using it. Another interesting but expected observation is the more prominent use of "Send SMS" in Drebin than in AndroZoo. In fact, SMS is one of the attack vectors of malware, as they use it to subscribe the victim to a premium rate service, or to make the victim pay a service for the attacker through SMS. The IMEI is a number used to identify phones in the GSM network, and can be used to blacklist a stolen phone from being used in a network. It can be also used to identify a phone regardless of the phone number. Malware, such as the Gemini [START_REF]Trojan:Android/Geinimi[END_REF] and [START_REF]Trojan:Android/Geinimi[END_REF]: F-Secure (2010), Trojan:Android/Geinimi GingerMaster [START_REF]Android/GingerMaster.A!tr[END_REF] families, tend to use this to identify victims and gather [START_REF]Android/GingerMaster.A!tr[END_REF]: FortiGuard Labs (2011), Android/GingerMaster.A!tr as much information about them as possible. We see that more than 60% of malware in Drebin gets the phone's IMEI compared to AndroZoo (around 30%). This indicate clearly identifiable characteristics of Drebin, such a the lack of Google Play services, the use of SMS, and the extraction of identifiable information from the phone.

In the case of a comparison between AMD and AndroZoo, some changes appear compared to Drebin. First, as wee see in Figure 3.18, samples in AMD collect the phone's number (40% of application with the "Get phone number" characteristic) in addition to the IMEI (63%). They also appears to collect MAC addresses (46% of applications with "Get MAC address" characteristic), as well as the SIM operator numeric name (43% of applications with "Get SIM operator" characteristic). Another interesting behavior is "Get installed packages" in more than 40% of applications in AMD. These information are not only of interest to malware authors. This feature can be exploited to profile users' behavior by legitimate application [START_REF] Luca | Leave My Apps Alone! A Study on How Android Developers Access Installed Apps on User's Device[END_REF], where it is mostly third party libraries that perform this type of data collection [START_REF] Pham | HideMyApp: Hiding the Presence of Sensitive Apps on Android[END_REF]. As seen previously, both Drebin and AMD have very few applications with "Native lib with JNI_OnLoad", "Exec command", "Load DEX class", "Change command", and "loadClass" compared to VirusShare 2018. Comparing these characteristics with AndroZoo, we see that, again, both Drebin (in Figure 3. 19) and AMD (in Figure 3.20) have less applications with these characteristics than AndroZoo. This is another notable difference between recent applications and older malware. If we examine a comparison between VirusShare 2018 and AndroZoo, Figure 3.21 shows us that almost all applications in VirusShare have these characteristics. More precisely, in Figure 3.22, we see that more than 80% of applications in VirusShare have "Native lib with JNI_OnLoad" and "Exec Command", and around 60% have "Load DEX class", "Change command" and "load-Class", while these characteristics are only present in around 20% of applications in AndroZoo. Indeed, there is an increase in usage of native methods in malware nowadays than ten years ago.

Discussion

Through the use of visualization we make a comparison between older and newer malware datasets, and between widely used malware datasets of the literature, Drebin and AMD, and AndroZoo. We see differences in terms of characteristics proportions such as "Native lib with JNI_OnLoad", "Exec command", "Load DEX class" and "Change command", which leads us to notice the evolution of malware towards the use of native code. We also made a comparison of a sample from AndroZoo against these three malware datasets from the literature. We see that malware in general tend not to use Google Play services; on the other hand, they tend to ask more frequently identifiable data such as the phone's IMEI, MAC address or SIM operator, besides the use of SMS.

Conclusion

In this chapter we presented DaViz, a new dataset visualization tool to explore and compare Android malware datasets. It uses different techniques such as filtering and reordering, that users can take advantage of to gain an insight of malware datasets. We presented the usage of Venn diagrams, bar charts, and heat maps to represent multiple values in different ways. We took advantage of interactivity to allow the user to perform two main analysis tasks: dataset exploration and comparison of datasets.

Then, we presented a comparison of Drebin and AMD, two old and recognized malware datasets from the literature, and a more recent malware dataset VirusShare 2018. We also presented a comparison of Drebin and AMD against an extract of AndroZoo. We found that the samples in the malware datasets are very different from AndroZoo. This is an expected result, because malware have, in general, a very different behavior than an average application found in markets. We also found that the malware datasets we inspected are different between each other. This confirms the works of Allix et al. [START_REF] Allix | Are Your Training Datasets Yet Relevant?[END_REF] and Pendlebury et al. [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF] over other characteristics than just the date.

DaViz gives an intuitive and visual difference between all these datasets. This tool has been presented at the RESSI 2022 conference [START_REF] Tomás Concepción | DaViz: Visualization for Android Malware Datasets[END_REF]. In the next chapter, we introduce a statistical procedure in order to qualify formally the differences we witnessed intuitevely with DaViz. Before performing experiments, researchers set the purpose of the study, i. e. malware detection, family classification or a market study, and the set of applications that the experiments are aimed for. They then create their experimental datasets by either using another one found in other experiments, or creating their own by gathering applications from various sources. We have seen that such a process can lead to irregularities in the dataset. These irregularities can be observed graphically as presented in Chapter 3. The term irregular suggests that we express a comparison against a real production environment, e. g. Google Play, called the "target" dataset. The expected consequence is that, if an experiment obtains good results with a regular dataset, then it should have the same good results on the larger dataset, that we call in this thesis the target dataset. Because of this, for the rest of this thesis, we introduce the notion of representativity between two datasets. An experimental dataset is said to be representative if it comes from this larger target dataset, and if it has similar proportions of characteristics compared to this larger target dataset.

Representativity in

In this section, we will define the notion of being representative: how we define representativity, how to sample such a dataset, and how we can measure a difference between datasets. In section 4.1, we start by defining what is a representative dataset with the new concept of 𝛿-representativity. Then, in section 4.2, we establish a method to create a representative dataset from a target set of applications. Later, we propose the usage of a statistical test to calculate if two datasets are statistically homogeneous to each other. And lastly, in section 4.3, we evaluate various datasets from the literature and compare them to a representative dataset of our choice.

Definition of a representative dataset

In statistics, population is a set of objects from which a study wants to be conducted [START_REF] William G Cochran | Sampling Techniques[END_REF]. In most science studies, such as medicine, psychol- [START_REF] William G Cochran | Sampling Techniques[END_REF]: Cochran (1977), Sampling Techniques ogy and biology, researchers pick a set of individuals from a defined population to study. In the case of research in Android applications, the population would be the set of programs from a defined source or sources. For example, a study of a new Android malware detection algorithm claims its solution works properly in Google Play. To verify this assumption, the researchers must test their new method with applications from Google Play.

Instead of performing a census of the population, which means to study all the individuals of a population, it is convenient to obtain a sample from the population. A sample, then, would be an subset of programs from a specified population. From here on, we refer to a sample as a "dataset", since it satisfies the characteristics of one as seen in the previous chapter. Obtaining a sample from the population have two main advantages:

1. Performing a census could be infeasible, as it would require too much time or resources that the researchers do not have. By taking a sample, it significantly reduces the number of individuals to study. 2. If the sample is representative of the population, the results from the experiment using this sample would be valid for the population.

In the case of evaluating a detection algorithm against applications from Google Play, the researchers then must get a sample from this market to perform their tests.

To formalize the definition a representative dataset, we consider a population P that contains 𝑁 applications. Then, a dataset D of size 𝑛 is representative of P if it satisfies the following properties:

1. 𝑛 is significantly smaller than 𝑁 2. evaluating the proportion p of a specific characteristic on D gives similar results as evaluation 𝑝 on P for the same characteristic 3. evaluating an algorithm on D gives similar results as performing the same evaluation on P

The individuals that compose a sample have characteristics inherent to them, and are of interest for a study. A characteristic is a property of an individual. In the case of Android applications, a characteristic can be properties such as the file size (evaluated as a integer), the number of classes in the code (an integer), or whether the applications contain a certain library (a boolean).

For a single characteristic, we could consider that some dataset D is representative of a population P for a single characteristic if the proportion of applications exhibiting is the same in D and P, (i. e. 𝑝 = p). Despite of this, there are cases where this is not possible: if 𝑝 = 1/𝑁, meaning that only one element in P has this characteristic, then, in order to have exactly the same proportion in D, the whole population have to be sampled. This would result in a contradiction of the first property about representative datasets. Therefore, a margin of error 𝛿 must be used, which is a difference between p and 𝑝 one allows to have considering that D is representative of P. And so, a dataset is representative of a population if it has the same proportion for a given characteristic up to a maximum difference of 𝛿.

The way 𝛿 is considered for a single characteristic can be replicated for multiple characteristics. Let us consider a set of 𝑑 characteristics {𝑐 1 , ..., 𝑐 𝑑 } that can take the values 𝑣 1 , ..., 𝑣 𝑑 respectively. This generates 𝑣 1 × ... × 𝑣 𝑑 different combinations. In the following, these combinations will be considered as classes, and K as the set of these classes. A class, then, is the set of applications that share the same values for the same set of characteristics. This term is not related to malware family names, which correspond to tags given to a set of malware that have the same behavior [START_REF] Tchakounté | Chapter 6 -Supervised Learning Based Detection of Malware on Android[END_REF]. Then, a dataset D could be considered representative of P for a set of classes K if the proportion of each class 𝑘 ∈ K is equal to P and D.

Sampling representative datasets

Once we have a definition for a representative dataset, we can talk about how to sample one from a target dataset. We take again the example of malware classification, where researchers have to take a sample from Google Play for doing the evaluation of their method. However, Google Play prevents user to crawl the platform to obtain information about a large quantity of applications, and so it is very difficult to obtain a complete snapshot of Google Play to get a representative sample from it. As a substitute, for the rest of this thesis, we use AndroZoo as a the population or our target dataset. As we explain in Chapter 2 (Section 2.1.3), this dataset is the largest dataset available with 19 553 370 (as of June 14, 2022), it is regularly updated with new applications from Google Play and other markets, and it keeps the applications they collect. Because of this, AndroZoo is the best option available to represent Google Play. This does not makes the assumptions of this work less relevant, as the methods proposed works beyond any specific dataset.

To obtain a representative sample of P of size 𝑛, a simple random sampling is performed [START_REF] William G Cochran | Sampling Techniques[END_REF]. This technique consist in drawing 𝑛 units out of the 𝑁 without replacement. Following definition 4.1.1, we fix a maximum difference 𝛿 that we accept. This will precondition 𝑛 to a minimum sample size in order to get an error less or equal than 𝛿. In statistics, studies quantify the probability of obtaining an error smaller than 𝛿. This is denoted as a confident level 𝐶. For example, "with a confident level 𝐶 = 95%, the error is smaller than 𝛿". Such estimations are performed because of the random sampling, as it may yield to differences in proportions bigger than 𝛿 just by chance. To minimize this, a minimum 𝑛 is choose based on 𝛿.

Sample size for one boolean characteristic

Once the maximum error 𝛿 and the confidence level 𝐶 is are defined in accordance to the experiment's requirements, a lower bound for 𝑛 can be calculated. The margin of error formula [START_REF] Richard | Stats: Data and Models[END_REF] with finite population correction [START_REF] William G Cochran | Sampling Techniques[END_REF] is used for this:

𝛿 ≤ 𝑧(𝐶) p(1 -p) 𝑛 𝑁 -𝑛 𝑁 -1
This formula is used to calculate the margin of error 𝛿 given the sample size 𝑛, the population size 𝑁, the sample proportion p and the standard deviation 𝑧(𝐶) at confident level 𝐶 for the standard normal distribution with mean 0 and standard deviation 1. The term p(1 -p) is maximized when p = 0.5, so we can bound this value by 0.5(1 -0.5) = 1 4 , namely:

𝛿 ≤ 𝑧(𝐶) p(1 -p) 𝑛 𝑁 -𝑛 𝑁 -1 ≤ 𝑧(𝐶) 𝑁 -𝑛 4𝑛(𝑁 -1)
Since we already know 𝛿, we can rearrange the inequality to bring 𝑛 to one side of the equation:

𝑛 ≥ 𝑁 4(𝑁 -1)𝛿 2 𝑧(𝐶) 2 + 1 -1 (4.1)
By using this formula, there are some light assumptions about the sample and population. First, we assume that 𝑛 is not to small and that it has at least 30 elements [START_REF] Robert V Hogg | Probability and Statistical Inference[END_REF]. Secondly, that the population proportion 𝑝 is not close to 0 or 1. In the case these assumptions fail, other confidence interval formulas can be used, like the Hoeffding inequality [START_REF] Bardenet | Concentration inequalities for sampling without replacement[END_REF].

[158]: [START_REF] Bardenet | Concentration inequalities for sampling without replacement[END_REF], 'Concentration inequalities for sampling without replacement'

Notice that the population size 𝑁 influence the sample size 𝑛 in the formula. Despite of this, if 𝑁 is relatively large, this influence is negligible. For example, the size of Google Play is estimated to be 2 664 893 applications as of June 06, 2022. For a representative sample of Google Play with 𝛿 = 0.01 and 𝐶 = 99%, the minimum required sample size is 𝑛 = 16 485, or only 0.62% of the total population. Surprisingly, for a representative dataset for AndroZoo, using the same 𝛿 and 𝐶, its size 𝑛 = 16 574, or 0.08%, a difference of 89 applications compared to the size of a representative sample of Google Play. Table 4.1 shows a summary of the sample size of a representative dataset for Google Play and AndroZoo respectively, for one or multiple characteristics. Next, we explain how to calculate the sample size of a representative dataset for multiple characteristics.

Sample size for multiple characteristics

The previous formula for calculating the minimum size for a representative dataset is only valid for a single boolean characteristic. This only applies for |K| = 2 classes, as a single boolean characteristic only has two possible values. When more than one boolean or non-boolean characteristics are considered, the number of classes increase, and so increases the sample's size, as more combinations have to be taken into account for the sample to be representative.

Just as in the case of a single boolean characteristic, the margin of error formula can be used with a slight modification. According to the Bonferroni correction [START_REF] Jean Dunn | Multiple Comparisons among Means[END_REF], representing a set of classes K with a representing one boolean characteristic with a confidence level 1 -(1 -𝐶)/(|K| -1). This correction implies the modification of the standard deviation 𝑧(𝐶) in Equation 4.1 to 𝑧(1 -1-𝐶 |K|-1 ). And so, the size of a representative dataset D of P with |K| classes with a margin of error 𝛿 and a confident level of 𝐶need to be at least:

𝑛 ≥ 𝑁 4(𝑁 -1)𝛿 2 𝑧 1 -1-𝐶 |K|-1 2 + 1 -1 (4.2)
Considering a set of classes with |K| = 36 864 (the set of classes of the considered characteristics as seen in Section 4.3.1), a margin of error 𝛿 = 0.015, and a confident level 𝐶 = 99%, the size of a representative sample of Google Play would be at least 𝑛 = 29 062, the size of one for AndroZoo 𝑛 = 29 339, and for an arbitrarily large population 𝑛 = 29 383.

Application to security experiments

Sampling applications from AndoZoo with the required value of 𝑛 gives a representative dataset for a chosen set of multiple characteristics. Unfortunately, two problems arise that limits this approach.

Problem 1: some characteristics may be hard-to-compute characteristics

We define as hard-to-compute, a characteristic that the researcher cannot compute with reliability, for example being a malware or not, containing a packer, etc. These characteristics may be missing from the population (for example Google Play) and recomputing them would require a manual analysis of the sampled applications.

Problem 2: underrepresentation of samples with the hard-to-compute characteristic Even if we suppose that the hard-to-compute characteristic could be calculated for the samples found up to now, sampling applications from a population would lead to new problems for the produced dataset. Indeed, the previous equation that have been presented in this chapter guaranty that the characteristics are homogeneous with the population for the given time. Let us imagine that a detection experiment is performed to Google Play at Google. The researcher's goal would be to create a dataset that resembles Google Play with the same malware/goodware proportion Google is facing, i. e., considering all the applications added to Google Play throughout all 10+ years this platform have been available. We represented this scenario in Figure 4.1. We see that, during this time:

▶ Goodware and malware are added to the store ▶ Applications continue to stay over the years ▶ Goodware are removed ▶ Malware may be detected and removed A researcher using our sampling method would obtain a representative dataset of Google Play at that time. We see that, for Google, the detection experiment would be to detect four malware out of nine applications, while for a researcher, the detection experiment would be to only detect one malware out of five applications. As a consequence, the drawn dataset would contain malware, but underrepresented compared to the reality of the detection problem. These two problems, a detection experiment at Google and a detection experiment for a researchers with representative dataset of Google Play, are indeed different.

Consequently, for these two reasons, we cannot push further a method based on simple random sampling of applications. As a fallback, we will switch to methods that consist in mixing applications that have different values for the hard-to-compute characteristics. For the malware/goodware example, it consists in mixing malware samples and goodware samples. The resulting dataset may be very heterogeneous from the population. In the next section, we propose a method, using a statistical test, to judge whether two datasets are different.

Evaluating existing datasets

In this section, our aim is to measure if a dataset D is statistically different from the population. This test will be based on the considered characteristics of applications. Before moving on the precise definition of the test, we need to discuss what happens if we include in our test all the possible characteristics of applications.

We recall the reader that our goal is to compare mixed datasets containing malware and goodware with the population. If the population is Google Play, there is little chance that our test answers that a mixed dataset is statistically the same as the population. Indeed, a lot of characteristics related to malware behavior would be different for the malware part of the dataset. On the contrary, the goodware part could be very similar to an extract of Google Play. These remarks hold only if Google Play does not contain too much malware, which is a reasonable hypothesis. As a result, comparing a set of malware with an extract of the Google Play is useless: there is a high chance that the characteristics are very different. As our mixed dataset of goodware/malware should have a significant amount of malware, comparing a mixed dataset to an extract of Google Play will give the same results: they will be different.

It comes from the previous remarks that we should more precisely discuss the considered characteristics. We can split the characteristics into two groups:

▶ Characteristics related to security: permissions, sensitive APIs, native libraries, etc. ▶ Characteristics not related to security: size, date, number of classes, etc.

The group of security related characteristics should be obviously different for malware compared to the population. If they are not, it means that malware are disguised perfectly as goodware and will not be spotted by any detection algorithm; this hypothesis is very uncertain, and we expect a difference. The group of non-security related characteristics should be similar to the population. For example, the date has no reason to be different in a malware dataset, in term of distribution, than in the population.

As a consequence, we propose to build a test of homogeneity considering the characteristics not related to security, letting other characteristics ignored because they are not homogeneous. Such a test would help to compare a mixed dataset to the population. Our goal is to compare the datasets of the literature, such as Drebin and AMD to the population. Later, in Chapter 5, our debiasing algorithm that modify datasets will also work on the group on non-security related characteristics, for the same reasons.

Non-security related characteristics to consider

When we choose these non-security related characteristics, it is important to notice that the number of combinations grows exponentially with the number of characteristics, and eventually grow bigger than the size of the dataset. With three boolean characteristics we have 8 classes, with four we have 16 classes, and with five there are 32 classes. As a result, we have to limit the number of these classes, because too much of them will lead to have classes with no application belonging to them. For numerical characteristics, they can be split into intervals so to have a value associated to each interval These intervals can be designed according to the study parameters. For example, one of those characteristics would be the file size, where the values are integers and this range can be split in intervals to designate "small", "medium", or "big" applications.

For this experiment, we decided to use the following characteristics that are not-security related: These characteristics were extracted using Droidlysis [START_REF] Apvrille | Pre-filtering mobile malware with heuristic techniques[END_REF] as we also explain in Chapter 3. These characteristics are widely different and easy to evaluate statically, lowering the computational resources needed to extract and use them in the following experiments. Although dynamic characteristics could be used, this requires executing the application. Some applications may not execute in emulated environments [START_REF] Alzaylaee | EMULATOR vs REAL PHONE: Android Malware Detection Using Machine Learning[END_REF], nor run on modern versions of the Android OS because of deprecated APIs or will just crash during execution [START_REF] Yang | Malware Detection in Adversarial Settings: Exploiting Feature Evolutions and Confusions in Android Apps[END_REF][START_REF] Lalande | Challenges for Reliable and Large Scale Evaluation of Android Malware Analysis[END_REF].

▶ APK size ( 4
With these 12 characteristics, there is a total of 36 864 classes: The APK size counts for four possible values for each interval, the year counts for nine values for the reason, and there are 10 boolean characteristics that count for two possible values each, so multiplying (4 × 9 × 2 10 ) = 36 864. These characteristics will be used to study the homogeneity of a number of datasets from the literature against an extract of the population, AndroZoo in 2020, with a statistical test that we explain in the next section.

Statistical test of homogeneity for datasets

In statistics, the 𝜒 2 statistical test for homogeneity [START_REF] Pearson | On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling[END_REF] is a method [START_REF] Pearson | On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling[END_REF]: Pearson (1900), 'X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling' to compare two samples and verify that these are not statistically homogeneous, i. e., they are not from the same population. To perform this test, we need a dataset D and a second one D 𝑟 , a sample drawn from a population P. Then, we establish a null hypothesis [START_REF] Richard | Stats: Data and Models[END_REF] (denoted 𝐻 0 ), a conjecture proposing that there is no effect or no difference in the possibilities established in it. In our case, 𝐻 0 is the following: D and D 𝑟 are drawn from the same population P (with replacement). Then we apply a formula (explained later in the section) using the proportion of applications of each class in K for D and D 𝑟 , and we obtain the 𝜒 2 test statistic for these two datasets.

The test result itself does not give a relevant indicator of how different or equal the datasets are. Instead, we look at the 𝑝-value of the result after performing the test. In plain terms, the 𝑝-value is the probability of obtaining a test result as the one obtained given that 𝐻 0 is true. Although it is a rather difficult concept to grasp even for statisticians, it is important to notice that a low 𝑝-value represents a statistically significant result, meaning that obtaining this result has a low chance of happening given that 𝐻 0 is true.

The 𝜒 2 test, as well as other statistical tests such as the Student's t-test or the Z-test, can only reject the null hypothesis but not confirm it.

Rejecting the null hypothesis means that we admit the dataset D is not homogeneous to D 𝑟 , and therefore dissimilar to the population P. To do so, a threshold to which the 𝑝-value is low enough to be significant is established. This threshold is typically set to be 0.05, but it can be lower, or higher depending of the experiment.

The test is applied as follows: we consider the finite set of classes K.

𝜒 2 = 𝑘∈K (𝑚 D,𝑘 -𝐸 D,𝑘,D 𝑟 ) 2 𝐸 D,𝑘,D 𝑟 + (𝑚 D 𝑟 ,𝑘 -𝐸 D 𝑟 ,𝑘,D ) 2 𝐸 D 𝑟 ,𝑘,D
This formula is related to the 𝜒 2 distribution, that is the generalization of all possible values of this formula for any given number of classes |K|.

The 𝑝-value can then be calculated using the cumulative distribution function (CDF) of the 𝜒 2 distribution. The CDF is parameterized by the number of degrees of freedom, in this case the number of classes minus one, i. e. |K| -1. The 𝑝-value can then be looked up in tables that are available in most statistics textbooks and statistics software (for example the Python library scipy).

In the following, we use the 𝜒 2 test to measure the difference between various Android malware datasets of the literature and a representative dataset of AndroZoo in 2020.

Studied Android datasets

To perform this test, several Android datasets from the literature were taken and analyzed with Droidlysis. These datasets are the following:

▶ 6 malware datasets: Drebin [START_REF] Arp | DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket[END_REF], AMD [START_REF] Wei | Deep Ground Truth Analysis of Current Android Malware[END_REF], as well as VS 2015, 2016, • AZ19 100k , 100 000 applications randomly drawn in 2019, and the subsets AZ19 100k in 2015, 2016, 2017, 2018 restricted to applications from each year; • AZ20 10k , AZ20 20k , and AZ20 30k three datasets randomly drawn from AndroZoo in 2020, containing respectively 10 000, 20 000 and 30 000 applications.

▶ 1 mix dataset: D mix composed of 5 560 malware from Drebin and 11 120 goodware from 2018 extracted from AndroZoo. D mix mimics the datasets used in previous machine learning papers [START_REF] Xu | ICCDetector: ICC-Based Malware Detection on Android[END_REF][START_REF] Feizollah | AndroDialysis: Analysis of Android Intent Effectiveness in Malware Detection[END_REF].

[110]: We perform a comparison between an extract of the population, in this case AndroZoo, called AZ20 30𝑘 with some datasets used in the literature (presented in Chapter 2). To illustrate the difference in class proportions between some of the datasets, Figure 4.2 shows the distribution of the 200 highest class proportions between AZ20 30𝑘 (in blue), the mix dataset D 𝑚𝑖𝑥 (in orange) and AMD (in green). The classes are sorted in descending order of size for AZ20 30𝑘 . The figure already shows the differences between these dataset. We can see that AMD has many classes with high proportions, between 4 % up to 11 %, than AZ20 30𝑘 and D 𝑚𝑖𝑥 . On the other hand, D 𝑚𝑖𝑥 has more classes with slightly higher proportions distributed in the center and the tail of the figure, but still different than AZ20 30𝑘 .

A further comparison is done by performing the 𝜒 2 test for homogeneity. Table 4.2 presents the dataset size, the number of classes represented only in AZ20 30𝑘 (and not in the studied dataset), the number of classes common to AZ20 30𝑘 and the studied dataset, the number of classes represented only in the studied dataset (and not in AZ20 30𝑘 ), the maximum and average 𝛿 of class proportions, the value of the 𝜒 2 test, the associated 𝑝-value, and the conclusion (or lack of) of the test between the studied dataset and AZ20 30𝑘 . Right away, last column shows that all datasets, except for the last extract of AndroZoo, have a 𝑝-value lower than the threshold. For AZ20 20𝑘 , this is consistent with the definition of representativity of Section 4.1.1, as we can see a low 𝛿 and a high 𝑝value where the test does not reject the null hypothesis of AZ20 20𝑘 and AZ20 30𝑘 are drawn from the same population. We also notice that AZ20 10𝑘 , even thought is a sample of AndroZoo, has a 𝑝-value that rejects the null hypothesis. This is mainly due to its size: AZ20 10𝑘 's size of 9 971 applications is lower compared to AZ20 20𝑘 which has the double. This affects the proportions for each class: since there are less applications to draw, proportions are prone to be overrepresented or underrepresented to a more larger extend. We can see this in the AZ20 10𝑘 's maximum 𝛿 for any class of 0.044 3 while for AZ20 10𝑘 this 𝛿 is 0.024 5, almost half of AZ20 10𝑘 's.

Conclusion

In this chapter, the notion of dataset representativity of a population was established using the definition of 𝛿-representativity. We presented how to sample a representative dataset using this notion, for either one or more characteristics. Then, we presented a comparison between a representative dataset of AndroZoo, extracted using the techniques established in this chapter, and other Android malware datasets of the literature.

It is clear that none of these datasets are representative of AndroZoo, our studied population, and only samples of this are. Thus, only by using a representative sample of AndroZoo, a study can claim that its results works for this population. However, samples from AndroZoo, or for most other markets, tend to lack the characteristics the studies use for their experiments. These characteristics are usually hard-to-compute, like, in the case of malware detection, being a malware. Note that we know that AndroZoo contain information about being a malware, which comes from VirusTotal. But this information cannot really be considered as ground truth, as VirusTotal contains false negatives and false positives. As a result, representative drawn samples cannot be used directly to perform tests.

To solve this problem, we propose to design a method to push a nonrepresentative labeled dataset to resemble a representative non-labeled dataset. This contribution is explained in the next chapter.

Debiasing Android Datasets 5 As we saw the differences between datasets, a question arises from these analyses: what are the consequences of using one dataset or another? Up to now, we explained how to sample a representative dataset from a population, and how to tell if a dataset is not representative of a population. In addition to this, in order to perform a malware detection evaluation, the dataset needs to be labeled, i. e. each sample have the metadata indicating if it is a malware or not. In contrast, a representative dataset is usually unlabeled because the characteristics have not been calculated yet, in particular the label being a malware or not is very difficult to obtain. However, the datasets in the literature are valuable for the reason of being labeled because they contain applications for which the characteristic of interest is already calculated. We would like to use these labeled datasets of the literature, while at the same time being representative of a population. We call such a generated dataset a debiased one, according to the population.

In this chapter, we propose an algorithm to obtain a representative and labeled dataset from a representative and unlabeled dateset, and a labeled dataset. In Section 5.1, we will start by proposing a new method for this goal, consisting of two algorithms. Then in Section 5.2, we present the generated debiased datasets we created from various datasets of the literature. Lastly, in Section 5.3, we use these debiased datasets in machine learning experiments, and see that they outperform old and new datasets from the literature: the malware detection becomes more difficult.

Debiasing algorithm

In this section we propose an algorithm to obtain a representative and labeled dataset from a representative and unlabeled dateset, and a labeled dataset.

Let us define the following datasets:

▶ a labeled dataset B that is of our interest because of the labels it contains ▶ a representative unlabeled dataset T of a population P ▶ and a large labeled dataset S.

We propose to design an algorithm to generate a representative dataset G that resembles T, and that is also labeled. To do this, the algorithm will add or remove applications from G, initially a copy of B, until the 𝛿 between G and T is less or equal to a predefined threshold.

We will be using the 𝛿-representativity, defined in Chapter 4, for measuring the similarity between two datasets. The margin of error 𝛿 will be assigned before executing the algorithm. We could have used the 𝜒 2 test that, as seen in the previous chapter, is used to tell if a dataset is not homogeneous to other, but it cannot be used to tell if the datasets are similar. In addition to this, the test results have no particular meaning on itself, and the 𝑝-value is cumbersome to interpret, while the notion of 𝛿-representativity is easier to grasp. Therefore, this new generated dataset G will be 𝛿-representative of T and also labeled, so that it can be used as a test set to evaluate an algorithm or a machine learning model on P.

To implement the previous idea, we will examine two algorithms. The first one is a constant-size algorithm, where it keeps the size of the initial dataset B . The second algorithm called optimal debiasing will optimize the result dataset G by allowing the modification of the initial dataset's size.

Recall of characteristics and classes

As seen in Chapter 4, a characteristic is a property of an element in a dataset. We represent applications using a set of boolean characteristics. We call a class 𝑘 a combination of specific values for each characteristic of this set of characteristics. For example, if we use two boolean characteristics 𝐴 = {𝑐 1 , 𝑐 2 }, each taking two values, we have the following four classes:

▶ 𝑘 1 = {0, 0} ▶ 𝑘 2 = {0, 1} ▶ 𝑘 3 = {1, 0} ▶ 𝑘 4 = {1, 1}
Applications sharing the same values for each characteristic in the set of characteristics are grouped in the same class. Finally, K is the set of all classes possible with this set of characteristics. In the previous example, K = {𝑘 1 , 𝑘 2 , 𝑘 3 , 𝑘 4 } for the set 𝐴.

Constant-size algorithm

Algorithm 1 produces a dataset G whose size is the same as its input B. For this, the algorithm take as input an initial dataset B, a representative dataset T of a population P, a large labeled dataset S and 𝛿. This algorithm consists in adding elements from the source S to the most underrepresented classes of B and to remove elements from S for the most overrepresented class. In this case, a class 𝑘 is underrepresented in G with respect of T if 𝑝 G,𝑘 -𝑝 T,𝑘 > 𝛿, and is overrepresented in G with respect of T if 𝑝 G,𝑘 -𝑝 T,𝑘 < -𝛿. Each time an element is added, another one is removed in order to keep the dataset's size constant. With this last constraint, the debiasing process may not be possible for one of following reasons: either no element can be added to an underrepresented class (because the source does not contain elements of this class); or it is not possible to obtain a proportion that lies in [𝑝 T,𝑘 -𝛿; 𝑝 T,𝑘 + 𝛿] for some class 𝑘 because 𝛿 is too small. Algorithm 1 is optimal in terms of number of modifications. * If a class 𝑘 𝑢 is underrepresented in B, elements are added to 𝑘 𝑢 to increase its proportion and make it correctly represented. If 𝑎 𝑘 𝑢 elements are added to this class while the size of D remains constant (i.e. |D| = |B|), the * Proofs are available in the appendix. ). By definition of the ceil function ⌈•⌉, ⌈|B|(𝑝 T,𝑘 -𝛿 -𝑝 B,𝑘 )⌉ is the lowest integer that verifies this inequality. So the minimal number of addition for an underrepresented class 𝑘 is ⌈|B|(𝑝 T,𝑘 -𝛿 -𝑝 B,𝑘 )⌉. If this number is negative, it means that the class is not underrepresented: in that case, no addition should be made. We denote, for any class 𝑘, the minimal number of addition as 𝑛 - B (𝑘) = max(0, ⌈|B|(𝑝 T,𝑘 -𝛿 -𝑝 B,𝑘 )⌉). By the same reasoning, we define the minimal number of deletions as 𝑛 + B (𝑘) = max(0, ⌈|B|(𝑝 B,𝑘 -𝑝 T,𝑘 -𝛿)⌉). In fact, we prove that Algorithm 1 makes exactly 2 max( 𝑘 𝑛 + B (𝑘), 𝑘 𝑛 - B (𝑘)) modifications between its input B and its output G.

K S = {𝑘 ∈ K | |S'(𝑘)| > 0} if 𝑝 G,

Optimal debiasing algorithm

Algorithm 1 can lead to suboptimal solutions. Algorithm 2 produces a dataset G without size constraint which is representative of T, with minimal modifications to B. This algorithm calls Algorithm 1 multiple times with different dataset sizes and maintains two variables: G 𝑏𝑒𝑠𝑡 is the best dataset found so far, i. e. the one that minimizes the number of modifications to B; and 𝑑 𝑚𝑖𝑛 is this distance between G 𝑏𝑒𝑠𝑡 and B. The algorithm starts with G 𝑏𝑒 𝑠𝑡 = "𝐼𝑚𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑒" and 𝑑 𝑚𝑖𝑛 = ∞ (line 1). First, it calls Algorithm 1 (line 4) with the initial dataset B. If this succeeds, 𝑑 𝑚𝑖𝑛 is updated with the appropriate number of additions and deletions, and G 𝑏𝑒 𝑠𝑡 becomes the result of Algorithm 1. Then, in lines 6 to 12, it tries to remove applications from the most overrepresented classes from B. Finally, in lines 14 to 25, it tries to add applications to the most underrepresented classes to B. For each modification to B, Algorithm While Algorithm 2 computes the closest dataset to the original one, it may happen that the solution found contains only a few applications from the original dataset. This is an indication that the original dataset was very biased.

If B has been debiased towards a target dataset T randomly drawn from a population P, with a tolerated difference of 𝛿 debiasing , we can estimate its error with the population P by taking into account the error 𝛿 sampling and confidence interval 𝐶 used to derive the size of T with Eq. (4.2): with confidence level 𝐶, the error between the debiased dataset and P is less than 𝛿 sampling + 𝛿 debiasing .

Generating debiased datasets

In

Characteristics used for debiasing

For our debiasing algorithm, we decided to use the non-security related characteristics as we explain in Chapter 4 Section 4.3.1. We want to homogenize a dataset to a target dataset in such a way that we leave discriminant characteristics for malware detection, i. e., security related ones, intact. Otherwise, if we debias using security related characteristics, we would obtain a dataset with malware indistinguishable from goodware, which would be artificially harder for a detection algorithm to make the difference between goodware and malware. Thus, the debiased dataset should be similar to the target dataset in relation to non-security related characteristics.

Debiasing datasets

The following details the use of our algorithm to produce representative datasets and in particular the choice of the base, reference and source datasets. Through these objectives, we discuss three main questions:

1. Is it always possible to debias a strongly biased dataset such as Drebin or VirusShare? 2. How many modifications (deletions/additions) are required? 3. Is a debiased malware dataset stable over time? Debiasing 100% goodware datasets We have randomly drawn a New AndrooZoo Extract (NAZE-18-G) and filtered it by relying on Virus Total to keep only goodware with a date no greater than 2018. This dataset should be close to a representative dataset of the Android app population in 2020. Indeed, by taking as reference dataset AZ20 30𝑘 and as source dataset AZ19 100𝑘 G, we successfully build a representative goodware dataset when 𝛿 reaches 0.001, as shown in Table 5.1. Indeed, for 𝛿 = 0.04 the dataset is already homogeneous: no addition/deletion is required, but the p-value is 0. Then, we decrease 𝛿 until the p-value becomes close to 1. For 𝛿 = 0.001, 19.53% of the resulting dataset are new applications and more than 50% of applications have been removed from the original one. Debiasing 100% malware datasets We were first interested in investigating whether it is possible to debias a small and old dataset (Drebin) on the one hand and a more recent and larger dataset on the other hand (malware from VirusShare between 2015 and 2018, namely VS 15, VS 16, VS 17, VS 17 and VS [START_REF]Google Play[END_REF][START_REF]Android and Google Play statistics[END_REF][START_REF]Number of Android apps on Google Play[END_REF][START_REF] Viennot | A Measurement Study of Google Play[END_REF]. We want the produced datasets to resemble AndroZoo, using an extract from 2020.

The debiasing results show us that Drebin is strongly biased as we need to add from 1.8% to 12.7% of new samples, because a lot of classes of the population are underrepresented or missing in Drebin. Even with these modifications, the resulting dataset, Drebin Debiased , is not homogeneous to the population. On the contrary, VirusShare can be debiased with few additions (less than 2.9%) but requires removing more than 90% of the dataset. In that case, the resulting dataset VS Debiased 15-18, obtains a 𝑝-value of 1. We conclude that the VirusShare dataset already contains enough material to be representative of AndroZoo for certain classes but that a lot of these classes are overrepresented.

As seen in the results, Algorithm 2 can generate a new dataset statistically indistinguishable from the target dataset with a margin of error 𝛿. However, the algorithm may fail (see grey cells in Table 5.1) when some classes are underrepresented in the source, as discussed in Section 5.1.2. This limitation can be overcome by providing a large and diverse enough source dataset.

On the other hand, when the algorithm succeeds, the generated datasets' size tends to be much smaller than the base dataset. This is due to a lot of overrepresented classes, from which the algorithm removes elements at line 9. This is especially the case for historical malware datasets.

The number of classes to consider may also limit the debiasing: intuitively, more characteristics would help to resemble the target dataset better, but it spreads the elements into more classes and makes it more difficult to find elements for certain classes.

Visual exploration of debiased datasets

Comparing using selected characteristics In Figure 5.1, we compare VS15-18 to AZ20 30𝑘 and see how much they differ. We see that there are very few applications with Google Play services in VS15-18, while there are more than 60% of applications with this characteristic in the AndroZoo sample. In contrast, we see, in Figure 5.2, when we compare VS Debiased 15-18 to AZ20 30𝑘 , Google Play services has increased, while the proportion of the other characteristics have not changed considerably.

Comparing using the seven characteristics with the most proportion difference This is the worst case visual comparison, i. e., other characteristics are more similar than the present ones. We hide all other characteristics that are similar and we show the most similar ones. In Figure 5.3, when we compare these VS15-18 and AZ20 30𝑘 , we see right away in the first column "Google Play services" as the characteristic with the most proportion difference between them. Next, we see most of the characteristics that are present in VS15-18 (more than 40%) in comparison to AZ20 30𝑘 (less than 30%). If we compare that with VS Debiased 15-18, as shown in Figure 5.4, we see that Google Play services appears at more than 40% of the applications in VS Debiased 15-18. On the other hand, all of the other characteristics more present in VS15-18 start to lower, up to the point that even "Exec command" is no longer present. Although we cannot visually confirm a total similarity between VS Debiased 15-18 and AZ20 30𝑘 , we can observe how much the debiased dataset is more similar than the original one. Indeed, it is an expected result because the amount of labeled source available is limited, so the algorithm cannot achieve a perfect proportion balance in the debiased dataset.

Building mixed datasets for machine learning

We finally produced two mixed datasets (goodware/malware) usable for training and testing machine learning algorithms. These datasets are built on top of the debiased malware datasets previously discussed. The goodware are taken from NAZE Debiased 18-G dataset. These two datasets are:

▶ DR-AG 𝐷𝑒𝑏 (standing for Drebin+AndroZoo Goodware), that mixes the debiased version of Drebin with 𝛿 = 0.01 and the goodware. Because the goal is to use mixed datasets for machine learning algorithms, we have to follow additional recommendations from Pendlebury et al. [START_REF] Pendlebury | TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time[END_REF]. We build our mixed sets by extracting random samples from goodware and malware and by following the recommendations C1 (APK of the training set should be older than the ones of the test set) and C3 (realistic malware-to-goodware ratio, i. e., 5% or 10%). In Table 5.2 we call "Pivot year" the year that delimitates the train and test sets. Note also that a consequence of C1 is that applications in the training and test sets are disjoint (which correspond to an added C4*: Train ∩ Test = ∅). For C2 (temporal window consistency) we do not enforce the constraint as we already balanced the classes on the year when performing the debiasing algorithms. Nevertheless, we can enforce them if needed, at the cost of reducing the output size of the training/tests sets. For example, the set VS-AG-C2 𝐷𝑒𝑏 has fewer samples than VS-AG 𝐷𝑒𝑏 . In the rest of the paper we only use the set that does not enforce C2 to decrease the quantity of results to discuss.

We recomputed the average 𝛿 of these mixed datasets: it is still quite low (worst case of 0.0012 for VS-AG 𝐷𝑒𝑏 ). The 𝑝-value falls to zero for some sets because applying the constraints of Pendlebury et al. drives the datasets away from the population. Nevertheless, we believe that respecting the time consistency for machine learning algorithms is of higher importance.

Finally, for evaluating these datasets when used for machine learning purposes, Table 5.2 includes additional datasets:

▶ VS-AG 𝐷𝑒𝑏,𝛿=0.4 a relaxed version of the debiasing of VS Debiased 15-18 with 𝛿 = 0.4. This dataset as a p-value of 0 but contains more samples in the training set. ▶ DR-AG: a mixed version of Drebin and goodware. ▶ VS-AG: a mixed version of VS 15-18 and goodware. ▶ ACT14 and ACT17: a mixed version of AndroCT [START_REF] Li | AndroCT: Ten Years of App Call Traces in Android[END_REF] with pivot year 2014 and 2017 respectively. ▶ AZL14 and AZL17: a mixed version of random samples extracted from AZ20 30𝑘 for which we took the goodware/malware label of AndroZoo as an oracle.

All these datasets will be used to compare the efficiency of machine learning algorithms when using our debiased datasets. In particular, AZL14 and AZL17 are the biggest extract of the population. AndroCT [START_REF] Li | AndroCT: Ten Years of App Call Traces in Android[END_REF] is a new recent datasets that cover ten years of applications: it will be used to observe its performances when learning with it and testing on AZL14/17.

Experiments with debiased datasets

We evaluate the performance of various machine learning classifiers depending on their training datasets to assess the contribution of debiased datasets. The datasets used in this experiments are those computed in the previous section, cf. Table 5.2.

Classifying with machine learning

The machine learning classifiers rely on a total of 262 characteristics. The first 12 characteristics are those used to debiased the datasets that are not related to security. Additionally, we included the characteristics related to security computed from two sources:

▶ Droidlysis that computes 168 booleans about the use of some APIs or behaviors (loading a DEX file, using cryptography, etc.); along with 34 permissions found in the Manifest file. ▶ FalDroid [START_REF] Fan | Android Malware Familial Classification and Representative Sample Selection via Frequent Subgraph Analysis[END_REF] that computes 48 characteristics representing score values related to graph-based features (with parameters 𝜖 = 0.8, 𝜃 = 0.1 [START_REF] Fan | Android Malware Familial Classification and Representative Sample Selection via Frequent Subgraph Analysis[END_REF]). As FalDroid is intended to classify families, We base this evaluation on the area under the receiver operating characteristic curve (AUC), a classical metric in machine learning. This metric can be interpreted as the probability that a classifier considers a randomly chosen goodware to be more probably benign than a randomly chosen malware. A perfect classifier has an AUC of 1 (every goodware is ranked higher than all malware), meaning that there exists a threshold to separate exactly goodware and malware. A random classifier has an AUC of 0.5. Besides, this metric does not depend on some threshold selection.

We evaluate various machine learning techniques: 𝑘-nearest neighbors [START_REF] Fix | Discriminatory analysis. Nonparametric discrimination: Consistency properties[END_REF] (neighborhood-based technique), decision trees [START_REF] Loh | Classification and regression trees[END_REF], random forest [START_REF] Tin | Random decision forests[END_REF] (bagging ensemble technique with decision trees), Gaussian naive Bayes [START_REF] David | Idiot's Bayes-not so stupid after all?[END_REF] (statistical model), and AdaBoost [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] (boosting ensemble models with decision trees). We used the scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF] implementation with default parameters. Since the decision trees, random forest and AdaBoost learning algorithms are stochastic, we took the average AUC over 25 seeds.

Results

Average and max AUC are presented in Tables 5 The datasets go by pair: there is one training set and one test set (cf. Table 5.2). In Table 5.3, we can see that generally learning on one training set yields the best results on the associated test set (i.e., the diagonal is in bold). The unique exception is that learning on AZL17 yields better results than learning on ACT17 when testing with ACT17. In fact, we can see that the AZL17 training set allows learning the best classifiers on ACT17 (max AUC: 0.77) and AZL17 (max AUC: 0.84), probably because it is a real randomly sampled dataset, and therefore it is less prone to bias. However, this training set was built using VirusTotal as an oracle, so it is, in general, not available for training.

Moreover, the most interesting results are the comparison of the performances of VS-AG and VS-AG 𝐷𝑒𝑏 on ACT17 and AZL17 test sets. We see that the debiased dataset always performs better than the original one. Another important result is that it is notably better to train on the debiased dataset VS-AG 𝐷𝑒𝑏 than on ACT17 (AndroCT [START_REF] Li | AndroCT: Ten Years of App Call Traces in Android[END_REF]), which is the most recent dataset we have. Finally, the debiased test dataset VS-AG 𝐷𝑒𝑏 is more difficult to predict than the test set VS-AG. It is a sign that the bias in the test set of VS-AG makes it easier for a model to correctly classify it, probably because some difficult cases are absent.

Finally, we can remark that the DR-AG training set yields poor results on all test sets, which is an expected result, as no applications in this dataset after 2015. We tried to confirm this result with an additional experiment with the pivot year of 2014, in Table 5.4. In this experiment, we evaluate if the debiased version of Drebin can obtain good results, even if it is old and quite small. The row in Table 5.4 concerning DR-AG 𝐷𝑒𝑏 shows that this dataset is far from being effective compared to ACT14. This is due to its size and the failing of debiasing when 𝛿 reaches 0.05 (cf. Table 5.1).

Discussion

This section discusses the validity of the datasets through time. We also compare these results with related works [START_REF] Cai | Embracing Mobile App Evolution via Continuous Ecosystem Mining and Characterization[END_REF][START_REF] Cai | A Longitudinal Study of Application Structure and Behaviors in Android[END_REF][START_REF] Cai | A study of run-time behavioral evolution of benign versus malicious apps in android[END_REF] that have studied this question and the relation with Android history. First, Cai et al. [START_REF] Cai | A Longitudinal Study of Application Structure and Behaviors in Android[END_REF] focus on the evolution of usage of the system's API over time, using a dedicated analysis framework [START_REF] Cai | Embracing Mobile App Evolution via Continuous Ecosystem Mining and Characterization[END_REF]. They conclude three main points: the decrease of usage of callbacks for methods of the app activities' lifecycle, a stable and small usage of inter-component communication, and a stable distribution of source/sink categories of callbacks. Second, Cai et al. [START_REF] Cai | A study of run-time behavioral evolution of benign versus malicious apps in android[END_REF] characterize this evolution for goodware and malware applications. Newer malware tend to access system APIs more often through third-party libraries than older malware. The use of activities and services has increased over time, while the use of ICC components has decreased. The diversity of callback categories has also increased.

Our experiments confirm this evolution at a higher level. We study the evolution of representativity of our debiased dataset over the years. More precisely, we debiased several extracts of VirusShare 2015, 2016, 2017 and 2018, and checked whether these debiased datasets stay representative of an evolving population over time. The target datasets are extracts of applications from AndroZoo 2019 (𝐴𝑍19 100𝑘 ) filtered on the corresponding period of time (T = 𝐴𝑍19 100𝑘 𝑖, with 𝑖 = 15, [START_REF]Android and Google Play statistics[END_REF][START_REF]Number of Android apps on Google Play[END_REF][START_REF] Viennot | A Measurement Study of Google Play[END_REF]. The source datasets are known datasets of malware such as Drebin and AMD, available at date 𝑖.

When the 𝑝-value confirms that the dataset is not statistically different, we tested the produced debiased version 𝑉𝑆 𝐷𝑒𝑏𝑖𝑎𝑠𝑒𝑑 𝑖 with the next years' population, i. e., 𝐴𝑍19 100𝑘 𝑖+1, 𝑖+2, .... The last column indicates contains 0 if the 𝑝-value is 0: no discussion can be done for such debiased datasets. The only debiased dataset of the parts of VirusShare is VS 18. As it is the last set of applications we have, we only conclude that this set can be used during 1 year.

Conclusion

In this chapter we defined a new method for creating a dataset that is both representative of a population and labeled. This method relies on two algorithms: a first one that aims to produce a debiased dateset with the same size as the input dataset. The second one takes the solution of the previous algorithm and improves it by allowing the addition or subtraction of applications to the debiased dataset. Then, we created several debiased datasets using our new method, using AndroZoo as the population. We used datasets from the literature as sources of applications, and a representative sample of AndroZoo as the dataset we want the target dataset to resemble. When using our algorithm for a security experiment, we explained that the set of characteristics to use should be those that are not related to security. Lastly, we used our new generated datasets in a machine learning malware detection approach.

We saw in the results that, despite the size of our debiased dataset (VS-AG 𝐷𝑒𝑏 ), it outperforms the others biased ones (like AndroCT) in almost every test. Learning with VS-AG 𝐷𝑒𝑏 gives the best results. Surprisingly, testing with VS-AG 𝐷𝑒𝑏 is not harder than with others, but it is very close. A possible explanation is the limited size of VS-AG 𝐷𝑒𝑏 test set. The other generated debiased dataset, DR-AG 𝐷𝑒𝑏 , as its original dataset is too biased, the algorithm reduces considerably its size. Consequently, this reduces efficiency in accuracy for malware detection using machine learning. This indicates that, by removing the differences of unrelated characteristics between the training and test set, the model can more easily identify the differences in characteristics more linked to the detection scheme, in this case being a malware. Besides the use in malware detection, these debiased datasets can be used for other types of experiments. Researchers can use these datasets to concentrate their efforts in relevant characteristics, knowing that their dataset is similar for unrelated ones to the population they are studying.

All materials used in the experiments (SHA256 hashes of samples composing the datasets and the code) are published in the Dada dataset [START_REF] Concepcion | Debiased Android DAtasets[END_REF].

Contributions of this thesis

In this thesis we described three main contributions targeting the usage of experimental datasets in Android malware detection.

First, we introduced DaViz, a visualization tool for exploring and comparing Android datasets. This new tool allows researchers to investigate and discover new insights about a dataset, such as unusual patterns and unexpected values. It also allows researchers to more easily identify differences between datasets, for example the difference of concentration of samples with a given set of characteristics between datasets. By exploiting our visual system, researchers can easily distinguish and locate trends in chart, and so identify differences between various characteristics. Using this tool, we observed that various old and recurrent Android malware datasets from the literature, namely Drebin and AMD, are very different to our reference dataset, an extract of AndroZoo. We also compared the Drebin and AMD datasets with a more recent VirusShare 2018 dataset. We saw specific differences in certain characteristics between the old and the new datasets. This visually confirms the evolutionary changes in malware.

For our second contribution, we focused on the representativity of datasets in relation to a studied population. We introduced the notion of representativity by defining 𝛿-representativity. We presented a method to sample a representative dataset using this notion, for either one or multiple characteristics. Hard-to-compute characteristics, such as being a malware, even if they could be calculated, may be over-or underrepresented in an extract of the population. Therefore, we need to use mixed datasets and evaluate their heterogeneity against the population. We presented a comparison between datasets of the literature and a representative dataset of AndroZoo, our studied population, using a statistical test. Using our sampling method, we saw that, for some cases, a sample of 15,000 application is statistically enough to get results with a low bias error. We showed that none of the datasets in the comparison, except an extract of AndroZoo, are representative of all AndroZoo, the studied population.

In our last contribution, we presented a method to transform a labeled nonrepresentative dataset into a labeled representative dataset that resembles a non-labeled representative dataset extracted from the studied population. We used this method to create several "debiased" datasets from various datasets of the literature. We also debiased several datasets from different years and checked their validity against reference datasets of the next year. Results revealed that debiased datasets are only valid at most one year, before they are no longer representative. This confirms that static datasets, the older they get, the less they resemble other more dynamic datasets, such as markets.

We also conducted an Android malware detection experiment using machine learning techniques with one of our debiased datasets. Results of the malware detection experiment show that our debiased dataset performs better than a bigger, more recent dataset like AndroCT, composed of 17 697 malware and 18 277 goodware samples ranging from 2010 to 2019.

Perspectives for future work

In this section we present our vision on how to improve and expand DaViz and our debiasing method.

Other types of characteristics can be included for visualization DaViz. For example, opcode frequencies, frequency of API sequences, code density, and others. These new characteristics may need modifications in Droidlysis or the use of new specialized extraction techniques to calculate them, including those that rely on dynamic analysis. This would expand the usage of our characteristics database to more machine learning experiments concerning other types of characteristics.

With these new characteristic, new challenges arise concerning how to manage these new characteristics. Because there are a lot of (potentially infinite) characteristics, users would have too much possibilities to explore datasets. This poses a problem in choosing which characteristic would be adequate to visualize. A solution would involve a recommendation system that would calculate the most pertinent charts and characteristics to explore a dataset. This way, users would choose their charts and characteristics based on the suggestions given by the system. Additional views could be implemented in DaViz to visualize individual applications on their own, such as those showing the control-flow graph (CFG), data-flow or information-flow graphs of the application's code or execution. This would allow a more detailed exploration of the applications that compose a dataset. For example, our visualization system would switch from a datasets view to an individual applications view. Additionally, the use of graphs could be extended not only to individual applications. Indeed, in malware family classification, some techniques use CFGs to classify samples into malware families, i. e. malicious applications with similar behavior. [START_REF] Zhou | Analysis of Android Malware Family Characteristic Based on Isomorphism of Sensitive API Call Graph[END_REF]. An implementation idea is to calculate and show a CFG of a dataset comprised of samples from a single family. This CFG would be calculated by obtaining the most commons nodes and edges of all the CFGs in the dataset. Additionally, this technique could be used to compare datasets composed of varieties of a family, which would allow to see similarities and differences of samples inside a family. These types of analyses would help researchers in malware family classification, as this topic is an ongoing research problem [START_REF] Joyce | MOTIF: A Malware Reference Dataset with Ground Truth Family Labels[END_REF]. So far we use the 𝜒 2 test to proof the statistical difference between two datasets. Unfortunately, this test cannot be used to proof the homogeneity of two datasets, as it only allows to proof that two datasets are statistically different. To evaluate the homogeneity of datasets, a distance measure could be used for this, e. g. using Kullback-Leibler divergence or Hellinger distance. In this case, a threshold must be established to set a boundary between similar and different when comparing two datasets. Beside this, other tests exists, just as the Kolmogorov-Smirnov test, that could be used besides the 𝜒 2 test. Because the data we use is primarily categorical data, as far as we know, there are not many distance measures available for this type of data. We believe that this work would require background knowledge in statistics to properly adjust existing tests and distances to categorical data.

Another scenario we consider as a possibility is the usage of our solutions, aimed toward the Android OS, for x86 programs. Of course, both DaViz and our debiasing method is platform agnostic, because they operate with characteristics rather than raw binaries. Therefore, with the extraction of characteristics from x86 and their importation to DaViz, it could be possible to explore and compare x86 malware datasets, in the same way it is done with Android datasets. Although the basic idea is easy to contemplate, x86 programs pose a problem in regard with the extraction of characteristics. In Android, the structure of an APK is known and it is possible to locate its Manifest file, native libraries and assets (i. e. images, documents, other code and data defined by the developer). The Manifest XML file contains important data such as permissions, intents, and others declared in it. Despite obfuscation techniques, it is possible to disassemble the code in an Android application. Thus, it is relatively easy to extract static information from an Android APK. In contrast, x86 executable formats do not follow this convention. In Portable Executable (PE) and Executable and Linkable Format (ELF) files, besides the header and the code section, no other valuable data is available for extraction. Because of this, there are less static characteristics that can be extracted from x86 programs. Despite this, it could be possible to extract more characteristics from dynamic analysis. Once characteristics are extracted, it is possible to debias an x86 dataset to another using our method. However, even with various experimental and malware datasets available in x86, there is no particular dataset of interest. In Android, markets are specially interesting because these are the main attack vector of malware distribution. In the case of x86, there is no market or "app store" of the same scale and usage as in Android. Rather, attack vectors in x86 includes download through phishing, P2P sharing, malicious websites, and USB drives. Datasets for x86 tend to be malware datasets where the ground truth is known [START_REF] Joyce | MOTIF: A Malware Reference Dataset with Ground Truth Family Labels[END_REF], but not a general repository of applications for users. Such a "reality" of applications for x86 and the generation of "general" datasets could be a subject of study.

Algorithm proofs A

For the sake of brevity and clarity, we introduce the notation Δ T 𝑝 G,𝑘 = 𝑝 G,𝑘 -𝑝 T,𝑘 . So a class 𝑘 is overrepresented w.r.t. T if Δ T 𝑝 G,𝑘 > 𝛿 and underrepresented if Δ T 𝑝 G,𝑘 < -𝛿.

Theorem A.0.1 Algorithm 1 always halts.

Proof. First, remark that if a class 𝑘 is correctly represented at some iteration, i.e., if |Δ T 𝑝 G,𝑘 | ≤ 𝛿, then it will be correctly represented in the next iteration. Besides, a class that is overrepresented in G cannot become underrepresented in the following iteration (and vice versa) without the algorithm halting. Both remarks are ensured by the "If" block on line 13 that halts the program by verifying whether a class with an added element is overrepresented or a class with a removed element is underrepresented.

Let's assume that this case never happens. At each iteration of the algorithm, there is at least one class 𝑘 such that |Δ T 𝑝 G,𝑘 | > 𝛿 that is modified in G: one element is added if 𝑘 is underrepresented and one element is removed if 𝑘 is overrepresented in G. So 𝑘 |Δ T 𝑝 G,𝑘 | is reduced by at least 1 |B| in the updated value of G. With enough iterations, 𝑘 |Δ T 𝑝 G,𝑘 | will so low that every class will be correctly represented. At that point, the algorithm halts.

Besides, a representative dataset always exists. This is necessary to prove the termination and the complexity of Algorithm 2. Proof. Assume such dataset D exists. Let us first prove that the return on line 9 cannot be reached. This line is reached if S is empty or if Δ T 𝑝 G,𝑘 𝑙 < -𝛿.

Consider the case where Δ T 𝑝 G,𝑘 𝑙 < -𝛿: we will prove by contradiction that this case is impossible when D exists. Since the algorithm never removes an element of an underrepresented class, at each iteration 𝑝 G,𝑘 𝑙 = B(𝑘 𝑙 )+𝑚 𝑛 where 𝑚 is the number of elements added to the class from S. If S(𝑘 𝑙 ) = 0, then 𝑝 G,𝑘 𝑙 is maximized, i.e., 𝑝 G,𝑘 𝑙 ≥ 𝑝 D,𝑘 𝑙 . So Δ T 𝑝 D,𝑘 𝑙 ≤ Δ T 𝑝 G,𝑘 𝑙 < -𝛿 which means that D is not representative of T. This is a contradiction: therefore this case is not possible.

Consider the case where S is empty and Δ T 𝑝 G,𝑘 𝑙 ≥ -𝛿. Once again, we will prove by contradiction that this case is impossible when D exists. The fact that Δ T 𝑝 G,𝑘 𝑙 ≥ -𝛿 implies that Δ T 𝑝 G,𝑘 ℎ ≥ 𝛿, otherwise the "While" condition would not have been true. Denote 𝑚 the number of completed iterations so far. Since one element of S is removed at each iteration, we conclude that initially |S| = 𝑚. We now prove that the "If" condition on line 13 cannot be true in Algorithm 1 when D exists. For algorithm 1 to return an error on line 14, the condition 𝑚𝑎𝑥 𝑘∈K |Δ T 𝑝 G,𝑘 | > 𝛿 must be true. Let us assume (without loss of generality) that there exists 𝑘 ′ such as Δ T 𝑝 G,𝑘 ′ > 𝛿 (the case Δ T 𝑝 G,𝑘 ′ < -𝛿 is similar).

Consider now the value of 𝑘 ℎ . Since 𝑘 ℎ ← arg max 𝑘∈K Δ T 𝑝 G,𝑘 , we can conclude that Δ T 𝑝 G,𝑘 ℎ ≥ Δ T 𝑝 G,𝑘 ′ > 𝛿. Denote 𝑛 = |B| and G ′ the dataset obtained from G by removing one element from 𝑘 ℎ and adding one element to 𝑘 𝑙 . The proportions of the classes of G ′ are:

𝑝 G ′ ,𝑘 =          𝑝 G,𝑘 -1 𝑛 if 𝑘 = 𝑘 ℎ 𝑝 G,𝑘 + 1 𝑛 if 𝑘 = 𝑘 𝑙 𝑝 G,𝑘 otherwise
Our goal is to prove that the "If" condition will never be true, i.e., that Δ T 𝑝 G ′ ,𝑘 ℎ ≥ -𝛿 and Δ T 𝑝 G ′ ,𝑘 𝑙 ≤ 𝛿.

Let 𝑎 1 and 𝑎 2 be two naturals such as 𝑝 G,𝑘 ℎ = 𝑎 This thesis presents, first, a method to evaluate the quality of datasets based on a statistical test that helps to compare a crafted dataset against a large set of applications such as markets. We show that historical datasets of the literature are of low quality, which justifies the need to create new up-to-date datasets. Second, we introduce an algorithm to update mixed datasets of malware/goodware of low quality in order to resemble a target dataset that cannot be used directly, e. g. a market. We evaluate the updated mixed datasets using a machine learning algorithm and we show that the detection of malware in our up-to-date dataset becomes a more difficult problem to solve. Lastly, we introduce DaViz, a dataset visualization tool for exploring and comparing Android malware datasets, which enables researchers to visualize the biases in datasets of the literature, and obtain useful information from them.

3. 1

 1 Five applications shown as images using the method by Jain et al. to recognize the usage of DexGuard in applications (a) and (b) (taken from[START_REF] Jain | Enriching reverse engineering through visual exploration of Android binaries[END_REF]) . . . . . . . . . . . . . . . . . . . . . . 3.2 Reverse call graph used by Santhanam et al. (taken from[START_REF] Ganesh | Interactive Visualization Toolbox to Detect Sophisticated Android Malware[END_REF]) . . . . . . . . . . . . . . . . . .3.3 The GroDDViewer[START_REF] Lalande | GroDDViewer: Dynamic Dual View of Android Malware[END_REF] interface showing the analysis of a Simplelocker sample . . . . . . . . 3.4 Screenshot of the tool's prototype proposed by Saxe et al. (taken from[START_REF] Saxe | Visualization of Shared System Call Sequence Relationships in Large Malware Corpora[END_REF]) . . . . . . . . . . .3.5 Screenshot of the SEEM interface byGove et al. (taken from [147]) . . . . . . . . . . . . . . . . . 3.6 Overview of the DaViz's interface in Firefox 104 . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Hierarchical visualization of characteristics divided in several categories. Selected characteristics are highlighted in green . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Example of a bar chart, showing for each bar the proportion, in percentage, of applications in the dataset with the given characteristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Example of a three by three heatmap, with the number of APK as color intensity . . . . . . . . 3.10a A Venn diagram in DaViz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10b The same Venn diagram with the whole dataset included . . . . . . . . . . . . . . . . . . . . . 3.11 Selection of elements with "Get SD card state" and "Get phone IMEI" on the F-Droid 2021 dataset 3.12 Diagram of the system implementation of DaViz . . . . . . . . . . . . . . . . . . . . . . . . . . 3.13 Top seven characteristics with the most proportion difference between Drebin and VirusShare 2018 3.14 Venn diagrams showing some of the top seven characteristics with the most proportion difference between Drebin and VirusShare 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.15 Top seven characteristics with the most proportion difference between AMD and VirusShare 2018 3.16 Venn diagrams showing some of the top seven characteristics with the most proportion difference between AMD and VirusShare 2018 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.17 Top seven characteristics with the most proportion difference between Drebin and a 30 000 sample of AndroZoo in 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.18 Top seven characteristics with the most proportion difference between AMD and a 30 000 sample of AndroZoo in 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.19 Venn diagrams showing the relationships between "Native lib with JNI_OnLoad", "Change command", "loadClass", "Exec command", "Load DEX class" in Drebin and a 30 000 sample of AndroZoo in 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.20 Venn diagrams showing the relationships between "Native lib with JNI_OnLoad", "Change command", "loadClass", "Exec command", "Load DEX class" in AMD and a 30 000 sample of AndroZoo in 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.21 Venn diagrams showing the relationships between "Native lib with JNI_OnLoad", "Change command", "loadClass", "Exec command", "Load DEX class" in VirusShare 2018 and a 30 000 sample of AndroZoo in 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.22 Top eight characteristics with the most proportion difference between VirusShare 2018 and a 30 000 sample of AndroZoo in 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Diagram of the evolution of Google Play and the sampling of a representative dataset. From Google's point of view, there are four malware samples to detect, while the sample taken on the right would only have one malware to detect. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Distribution of application classes for the top most 200 over 2 874 classes of AZ20 30𝑘 and compared with D 𝑚𝑖𝑥 and AMD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Comparison of VS15-18 to AZ20 30𝑘 for some of the selected characteristics . . . . . . . . . . . . 5.2 Comparison of VS Debiased 15-18 to AZ20 30𝑘 for some of the selected characteristics . . . . . . . . 5.3 Comparison of VS15-18 to AZ20 30𝑘 for the seven characteristics with the most proportion difference 5.4 Comparison of VS Debiased 15-18 to AZ20 30𝑘 for the seven characteristics with the most proportion difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Tables 2.1 Details on the dataset usage in detection experiments. GP = Google Play, VT = VirusTotal, GM = GM, DB = Drebin, AZ = AndroZoo, CT = Contagio, VS = VirusShare, PD = PlayDrone, MSL = MobiSec Lab, TZ = theZoo, MS = MalShare, N/A = Not applicable . . . . . . . . . . . . . . . . . 4.1 Sample size of representative datasets of Google Play and AndroZoo for one or multiple characteristics (𝛿 = 0.01 and 𝐶 = 99%). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 𝜒 2 test for homogeneity over AZ20 30𝑘 for a finite set of non-security characteristics . . . . . . . . 5.1 Results after debiasing datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Mixed datasets for machine learning training and evaluation . . . . . . . . . . . . . . . . . . . . 5.3 Mean and max AUC of machine learning classifiers depending on their training set and their test set when the pivot year is 2017. In bold face: best AUC on a test set . . . . . . . . . . . . . . . . . 5.4 Mean and max AUC of machine learning classifiers depending on their training set and their test set when the pivot year is 2014. In bold face: best AUC on a test set . . . . . . . . . . . . . . . . .

[ 5 ]

 5 : Arp et al. (2014), 'DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket'

[ 6 ]

 6 : Wei et al. (2017), 'Deep Ground Truth Analysis of Current Android Malware'

[ 11 ]

 11 : Pendlebury et al. (2019), 'TESSER-ACT: Eliminating Experimental Bias in Malware Classification across Space and Time' influence of date on Android malware detection experiments using ML. Results of their experiments show a decay in performance of state-ofthe-art ML Android malware detection algorithms, when applications in the test dataset are newer than in the training set. The authors also indicate the proportion of malware and benign application in the dataset as a way to influence the results of experiments.

[ 12 ]

 12 : Concepción Miranda et al. (2022), 'DaViz: Visualization for Android Malware Datasets'

[ 18 ]

 18 : Viennot et al. (2014), 'A Measurement Study of Google Play'

[ 34 ]

 34 : AndroidAdvices (2012), Remove Android:Plankton [PUP] Virus from Android Device after Downloading Apps from GetJar

[ 46 ]

 46 : Domínguez et al. (2022), We are Uptodown 2002. Since 2011 [47], it provides Android applications, and nowadays it [47]: Domínguez et al. (2011), Uptodown * FAQ -General | F-Droid

[ 18 ]

 18 : Viennot et al. (2014), 'A Measurement Study of Google Play'

[ 55 ]

 55 : Golovin et al. (2021), Malicious code in APKPure app

[ 65 ]

 65 : Plohmann et al. (2018), 'Malpedia: A Collaborative Effort to Inventorize the Malware Landscape'

[ 65 ]

 65 : Plohmann et al. (2018), 'Malpedia: A Collaborative Effort to Inventorize the Malware Landscape'

[ 71 ]

 71 : Taheri et al. (2019), 'Extensible Android Malware Detection and Family Classification Using Network-Flows and API-Calls'

[ 11 ]

 11 : Pendlebury et al. (2019), 'TESSER-ACT: Eliminating Experimental Bias in Malware Classification across Space and Time' .

[ 90 ]

 90 : Grace et al. (2012), 'RiskRanker: Scalable and Accurate Zero-Day Android Malware Detection'

[ 93 ]

 93 : Crussell et al. (2013), 'AnDarwin: Scalable Detection of Semantically Similar Android Applications'

[ 94 ]

 94 : Li et al. (2015), 'IccTA: Detecting Inter-Component Privacy Leaks in Android Apps'

[ 95 ]

 95 : Dodge et al. (2006), The Oxford Dictionary of Statistical Terms

[ 96 ]

 96 : Weiss et al. (2001), The effect of class distribution on classifier learning: an empirical study

[ 11 ]

 11 : Pendlebury et al. (2019), 'TESSER-ACT: Eliminating Experimental Bias in Malware Classification across Space and Time'

[ 97 ]

 97 : Susan et al. (2020), 'The balancing trick: Optimized sampling of imbalanced datasets-A brief survey of the recent State of the Art'

[ 99 ]

 99 : Chawla et al. (2002), 'SMOTE: synthetic minority over-sampling technique'

[ 97 ]

 97 : Susan et al. (2020), 'The balancing trick: Optimized sampling of imbalanced datasets-A brief survey of the recent State of the Art'

[ 100 ]

 100 : Pfaff et al. (2019), 'Code Clones Considered Harmful? Quantifying and Exploiting the Effects of Code Clones in Static Malware Classifiers for JavaScript' not usually share the list of used applications, it is not possible to check the presence of clones in their datasets. Pan et al. [101] show a method [101]: Pan et al. (2005), 'Finding Representative Set from Massive Data'

[ 103 ]

 103 : Cai et al. (2018), 'Towards Sustainable Android Malware Detection' [104]: Cai (2020), 'Assessing and Improving Malware Detection Sustainability through App Evolution Studies' [105]: Cai (2020), 'Embracing Mobile App Evolution via Continuous Ecosystem Mining and Characterization'

[ 10 ]

 10 : Allix et al. (2015), 'Are Your Training Datasets Yet Relevant?' .

Figure 3 . 1 :

 31 Figure 3.1: Five applications shown as images using the method by Jain et al. to recognize the usage of DexGuard in applications (a) and (b) (taken from [139])

[ 138 ]

 138 : Quist et al. (2009), 'Visualizing compiled executables for malware analysis'

[ 139 ]

 139 : Jain et al. (2015), 'Enriching reverse engineering through visual exploration of Android binaries'

Figure 3 . 2 :

 32 Figure 3.2: Reverse call graph used by Santhanam et al. (taken from [140])

[ 141 ]

 141 : Lalande et al. (2020), 'GroD-DViewer: Dynamic Dual View of Android Malware'

Figure 3 . 3 :

 33 Figure 3.3: The GroDDViewer [141] interface showing the analysis of a Simplelocker sample

Figure 3 . 4 :

 34 Figure 3.4: Screenshot of the tool's prototype proposed by Saxe et al. (taken from [146])

Figure 3 . 5 :

 35 Figure 3.5: Screenshot of the SEEM interface by Gove et al. (taken from [147])

Figure 3 .Figure 3 . 6 :

 336 Figure 3.6 presents an overview of DaViz's interface. On the top right of the figure, we see the name of the current's workspace, named as default "Workspace 1". Just below we see the user selection interface to create charts. On the left there is a dropdown list to select which graph to create: ▶ Venn diagram ▶ Bar chart ▶ Bar chart for categorical characteristics ▶ Heat mapsNext to it, a button "Select characteristics" for choosing the characteristics to display in the chart (see section 3.2.2), in this case for a Venn diagram. Next to this is a "Draw" button to create the chart once the user has

Figure 3 . 7 :

 37 Figure 3.7: Hierarchical visualization of characteristics divided in several categories. Selected characteristics are highlighted in green

Figure 3 . 8 :

 38 Figure 3.8: Example of a bar chart, showing for each bar the proportion, in percentage, of applications in the dataset with the given characteristic

Figure 3 . 9 :

 39 Figure 3.9: Example of a three by three heatmap, with the number of APK as color intensity

Figures 3 .

 3 10a and 3.10b show examples of Venn diagrams. Intersections between sets correspond to applications that have all the characteristics of the intersecting sets.

Figure 3 .Figure 3 .

 33 Figure 3.10a: A Venn diagram in DaViz

[ 149 ]

 149 : Alsallakh et al. (2016), 'The Stateof-the-Art of Set Visualization'

Figure 3 . 11 :

 311 Figure 3.11: Selection of elements with "Get SD card state" and "Get phone IMEI" on the F-Droid 2021 dataset

Figure 3 . 13 :Figure 3 . 14 :Figure 3 . 15 :Figure 3 . 16 :

 313314315316 Figure 3.13: Top seven characteristics with the most proportion difference between Drebin and VirusShare 2018

Figure 3 . 17 :

 317 Figure 3.17: Top seven characteristics with the most proportion difference between Drebin and a 30 000 sample of AndroZoo in 2020

Figure 3 . 18 :

 318 Figure 3.18: Top seven characteristics with the most proportion difference between AMD and a 30 000 sample of AndroZoo in 2020

[ 152 ]

 152 : Scoccia et al. (2020), 'Leave My Apps Alone! A Study on How Android Developers Access Installed Apps on User's Device'

[ 153 ]

 153 : Pham et al. (2019), 'HideMyApp: Hiding the Presence of Sensitive Apps on Android'

Figure 3 . 19 :

 319 Figure 3.19: Venn diagrams showing the relationships between "Native lib with JNI_OnLoad", "Change command", "loadClass", "Exec command", "Load DEX class" in Drebin and a 30 000 sample of AndroZoo in 2020

Figure 3 . 20 :

 320 Figure 3.20: Venn diagrams showing the relationships between "Native lib with JNI_OnLoad", "Change command", "loadClass", "Exec command", "Load DEX class" in AMD and a 30 000 sample of AndroZoo in 2020

Figure 3 . 21 :

 321 Figure 3.21: Venn diagrams showing the relationships between "Native lib with JNI_OnLoad", "Change command", "loadClass", "Exec command", "Load DEX class" in VirusShare 2018 and a 30 000 sample of AndroZoo in 2020

Figure 3 . 22 :

 322 Figure 3.22: Top eight characteristics with the most proportion difference between VirusShare 2018 and a 30 000 sample of AndroZoo in 2020

[ 12 ]

 12 : Concepción Miranda et al. (2022), 'DaViz: Visualization for Android Malware Datasets'

[ 156 ]

 156 : De Veaux et al. (2016), Stats: Data and Models

[ 157 ]

 157 : Hogg et al. (2015), Probability and Statistical Inference

[ 159 ]Figure 4 . 1 :

 15941 Figure 4.1: Diagram of the evolution of Google Play and the sampling of a representative dataset. From Google's point of view, there are four malware samples to detect, while the sample taken on the right would only have one malware to detect.

  We denote 𝑚 D,𝑘 (respectively 𝑚 D 𝑟 ,𝑘 ) the number of samples in D (respectively in D 𝑟 ) of class 𝑘 ∈ Kand 𝑝 P,𝑘 (respectively 𝑝 D,𝑘 and 𝑝 D 𝑟 ,𝑘 ) the proportion of elements of class 𝑘 in P (respectively in D, in D 𝑟 ). If the null hypothesis holds true, then the three proportions 𝑝 P,𝑘 , 𝑝 D,𝑘 and 𝑝 D 𝑟 ,𝑘 of class 𝑘 ∈ K should be equal or different only by a small margin. The proportion 𝑝 P,𝑘 can thus be estimated by 𝑚 D𝑟 ,𝑘 +𝑚 D,𝑘 |D 𝑟 |+|D| and the expected number of occurrences of class 𝑘 in D can be estimated by 𝐸 D,𝑘,D 𝑟 = |D| × 𝑚 D𝑟 ,𝑘 +𝑚 D,𝑘 |D 𝑟 |+|D| and in D 𝑟 by 𝐸 D 𝑟 ,𝑘,D = |D 𝑟 | × 𝑚 D𝑟 ,𝑘 +𝑚 D,𝑘 |D 𝑟 |+|D| . The 𝜒 2 value of two datasets D and D 𝑟 is defined as:

[ 5 ]

 5 : Arp et al. (2014), 'DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket' [6]: Wei et al. (2017), 'Deep Ground Truth Analysis of Current Android Malware' 2017 and 2018, containing all the malware collected by VirusShare for each of these years; ▶ 2 groups of datasets extracted from AndroZoo [61]: [61]: Allix et al. (2016), 'AndroZoo: Collecting Millions of Android Apps for the Research Community'

Algorithm 1 :

 1 Constant size debiasing algorithm Input: B, T, S, 𝛿 Output: Generated dataset G S' ← S; G ← B while 𝑚𝑎𝑥 𝑘∈K |𝑝 G,𝑘 -𝑝 T,𝑘 | > 𝛿 do 𝑘 ℎ ← arg max 𝑘∈K (𝑝 G,𝑘 -𝑝 T,𝑘 ) ⊲ most overrepresented class 𝑘 𝑙 ← arg min 𝑘∈K (𝑝 G,𝑘 -𝑝 T,𝑘 ) ⊲ most underrepresented class if |S'(𝑘 𝑙 )| = 0 then

Figure 5 . 1 :Figure 5 . 2 :

 5152 Figure 5.1: Comparison of VS15-18 to AZ20 30𝑘 for some of the selected characteristics

Figure 5 . 3 :Figure 5 . 4 :

 5354 Figure 5.3: Comparison of VS15-18 to AZ20 30𝑘 for the seven characteristics with the most proportion difference

  .3 (pivot year of 2017) and 5.4 (pivot year of 2014) where each line corresponds to a different training set and each column to a different test set. The mean AUC over the various classifiers helps to estimate the global quality of a training dataset for machine learning. The max AUC reflects what performances could be obtained with the best classifiers.

[ 14 ]

 14 : Concepcion Miranda et al. (2021), Dada: Debiased Android DAtasets Our debiasing method, alongside the definition of representativity in Chapter 4, were published in the IEEE Transactions on Information Forensics and Security journal [13]. [13]: Concepción Miranda et al. (2022), 'Debiasing Android Malware Datasets: How Can I Trust Your Results If Your Dataset Is Biased?' Epilogue Conclusion 6

[ 173 ]

 173 : Zhou et al. (2017), 'Analysis of Android Malware Family Characteristic Based on Isomorphism of Sensitive API Call Graph'

[ 174 ]

 174 : Joyce et al. (2022), 'MOTIF: A Malware Reference Dataset with Ground Truth Family Labels'

Theorem A.0. 2

 2 Let T be a dataset, 𝛿 > 0 and 𝑛 > 0 such as 𝑛 ≥ 1 𝛿 . There exists a dataset D of size 𝑛 such as D is 𝛿-representative of T.Proof. To prove the existence of such dataset of size 𝑛, we construct it.More precisely, we associate to each class 𝑘 a number of element 𝑏 𝑘 such as this class is 𝛿-representative of T, i.e.,𝑏 𝑘 𝑛 ∈ [𝑝 T,𝑘 -𝛿; 𝑝 T,𝑘 + 𝛿]. For each class 𝑘, define 𝑎 𝑘 such as 𝑎 𝑘 𝑛 < 𝑝 T,𝑘 ≤ 𝑎 𝑘 +1 𝑛 . Remark that 𝑝 T,𝑘 -𝛿 ≤ 𝑎 𝑘 𝑛 < 𝑝 T,𝑘 ≤ 𝑎 𝑘 +1 𝑛 ≤ 𝑝 T,𝑘 + 𝛿 since 𝛿 ≥ 1 𝑛 . As 𝑘 𝑝 T,𝑘 = 1, 𝑘 𝑎 𝑘 < 𝑛and 𝑘 (𝑎 𝑘 + 1) ≥ 𝑛. So there exist 𝑏 𝑘 such as for all class 𝑘 either 𝑏 𝑘 = 𝑎 𝑘 or 𝑏 𝑘 = 𝑎 𝑘 + 1 and that verify 𝑘 𝑏 𝑘 = 𝑛. Denote the dataset D such as 𝑝 D,𝑘 = 𝑏 𝑘 𝑛 . The size of D is 𝑘 𝑏 𝑘 = 𝑛 and D is 𝛿-representative of T since 𝑏 𝑘 𝑛 ∈ [𝑝 T,𝑘 -𝛿; 𝑝 T,𝑘 + 𝛿] for every class 𝑘. It proves the existence of such a dataset. Theorem A.0.3 If there exists a dataset D consisting of elements of B and S (i.e., D ⊆ B ∪ S) such as D is representative of T and |D| = |B|, then Algorithm 1 returns a representative dataset. If no such dataset exists, the algorithm returns "Impossible".

  Denote B D ⊆ B and S D ⊆ S such as D = B D ∪ S D . Due to the definition of 𝑘 ℎ and the fact that Δ T 𝑝 G,𝑘 ℎ ≥ 𝛿, it means that for each previous iterations one element has been removed from an overrepresented class. So there are at least 𝑚 + 1 elements of B that are absent in B D , i.e., |B D | ≤ |B| -𝑚. However, since |B| = |D| = |B D | + |S D |, it means that |S| ≥ 𝑚 + 1. This is in contradiction with the fact that |S| = 𝑚. So S cannot be empty. Finally, the return on line 9 cannot be reached when D exists.

Title:

  Profiling and Visualizing Android Malware DatasetsKeywords: Malware, Datasets, Bias, Visualization Abstract: Mobile devices are ubiquitous: nowadays most people own a mobile telephone. Because of this, it is a target of interest for attackers. Researchers in malware analysis put their effort to recognize these types of programs before they are installed on a user device. To do this, they perform experiments to automatically detect malware, for example with machine learning, where they use sets of already known malware and goodware. Depending on their choice of datasets, the evaluation of the experiments can yield acceptable results, or outstanding but overestimated results. Consequently, datasets with malware and benign samples are important elements to consider when designing an experiment.

  Aptoidemarket launched in 2011 with development starting in 2009. Their main repository offer applications for users to download and install. Besides the main store, this market has allowed the "decentralization" of the "stores": it has partnered with other companies to allow the redistribution of applications throughout different stores like Softsonic, Multilaser, and Aptoko. Amazon started distributing Android applications since 2011 in its Amazon Appstore[26]. It shares few of the most popular

[START_REF]Toolbox App[END_REF]

: Bullitt Mobile Ltd (2022), Toolbox App [25]: Aptoide S.A. (2022), [26]: Amazon.com, Inc. (2022), Amazon Appstore

  always a solution, as new samples are not always available as quickly as needed or in sufficient diversity. One way to limit this problem is to choose more semantic than syntactic features, as stated by Zhang et al. in[START_REF] Zhang | Enhancing State-of-the-art Classifiers with API Semantics to Detect Evolved Android Malware[END_REF]. Finally, in[START_REF] Xu | DroidEvolver: Self-Evolving Android Malware Detection System[END_REF], Xu et al. propose to enhance malware detection

	[67]: Zhang et al. (2020), 'Enhancing
	State-of-the-art Classifiers with API Se-
	mantics to Detect Evolved Android Mal-
	ware'
	[107]: Xu et al. (2019), 'DroidEvolver: Self-
	Evolving Android Malware Detection
	System'

Manzanares et al. (2022), 'Concept drift and cross-device behavior: Challenges and implications for effective android malware detection' [67]: Zhang et al. (2020), 'Enhancing State-of-the-art Classifiers with API Semantics to Detect Evolved Android Malware' where Zhang et al. observes that malware samples often implement the same functionalities over time but change their implementation because of the evolution of APIs or changes of third-party libraries. These changes allow such malware to avoid detection by older classifiers. Solutions are proposed to mitigate this phenomena. The first one consist of re-training a model, where a new test set is used to create a model using newer data. Regular re-training can solve this problem, however, it is not

Table 2 . 1 :

 21 Details on the dataset usage in detection experiments. GP = Google Play, VT = VirusTotal, GM = GM, DB = Drebin, AZ = AndroZoo, CT = Contagio, VS = VirusShare, PD = PlayDrone, MSL = MobiSec Lab, TZ = theZoo, MS = MalShare, N/A = Not applicable Finally, columns "Class imbalance" and "Time Incoherence" indicate whether the datasets used in the experiments suffer from these irregularities, introduced in Section 2.2.Out of the 28 articles, we found that 17 suffer from class imbalance: the goodware/malware ratio is often too high. Time incoherence happens in 15 articles: experimental datasets often contain recent applications from Google Play and older malware from Drebin or Genome. Additionally, 4 articles have a too old dataset age, compared to the publication date.

												Time
			Goodware	Goodware	Malware		Malware	Malware	Training set		Class Im-
	Paper	Goodware source										Test set (M/G)	Incoher-
			source year	size	source	source year	Size	(M/G)		balance
												ence
						GM		2010-2011	1 242	2,784/0
	MADAME [108]											X
						CT		2016		(Total)
						VS		2012-2015	1 923
	DroidDetector [109]	GP		2016	20 000	CT		2016	500	880/800		880/880	X
						GM		2010-2011	1 260
										15 561
	ICCDetector [110]	GP		2014	12 026	DB		2010-2012	5 264			1 203/526	X	X
										(mixed)
										10-fold cross-
	Androdialysis [111]	GP		2016	1 846	DB		2010-2012	5 560			5 560/1 846	X	X
										validation
						DB		2010-2012	5 560			DB/PD
	MaMaDroid [112]	PD	2013-2014	8 447	VS		2013	6 228	10-fold cross-		2013/PD	X
		GP		2016	2 843	"		2014	15 417	validation		2014/PD
						"		2015	5 314			2014/GP
						"		2016	2 974			2015/GP
												2016/GP
	StormDroid [113]	GP		2015	4 350	MSL		2015	2 000	(900+500+100)		(600+300+100)	X
						GM		2010-2011	1 260	1500		1000
						CT		2012-2015	360
		GP				CT		2012-2015	60	80% (from
	PIndroid [7]	AppBrain	up to 2016	445	DB		2010-2012	100	1300/445	20% (from 1300/445	X	X
		F-Droid				GM		2010-2011	1 000	total)		total)
		Getjar				VT		?	70
		Aptoid				TZ[57]		2016	70
		Mobango				MS[62]		?	25
						VS		2016	25
	Vinod et al. [9]	GP (D1, D2, D3, D4, D5) Baidu (D1, D2) Koodous(D1, D2)(1,514)	2016	3 130	DB (D1, D2) Koodous (D2) [114] (D4)		2010-2012 ? 2011-2015	1 514 1 000	10-fold cross-validation		D1 2 520/3 130 D3 2 474/2 474 D2 1 514/3 130	X	X
		1Mobile (D1, D2)				GM (D5)		2010-2011	1 206			D4 1 000/14 000
		9apps (D1, D2)										D5	1 206/1 206
						GM		2010-2011	1 000	Each malware dataset
	MalDozer [8]	GP		2016	38 000	DB		2010-2012	5 500	individually/37 627	X	X
						Own			20 000
						"Merged"			33 000
	IntelliAV [115]	VT	2011-2016	10 058	VT		2011-2016	9 664	9 664/10 058	X
		"		2017	2 898	"		2017	2 311	2 311/2 898
										10-fold cross validation with 9
										subsets of 50 000 samples, varying
										malware concentration from 2%,
	Martín et al. [116]	GP		2015	∼ 49 000	GP		2015	69 000	25%, 50% of the whole subset, and	X
										the number of antivirus alerts: 1-AV,
										2-AV, 4-AV (4-AV, 50% subset only
										contains 36 000)
						Comodo
		Comodo Cloud			920 (D1 Tr)	Cloud			914 (D1 Tr)	914/920 (D1)		500 (D1)
	HinDroid [117]			2017				2017
		Security Center			198 (D1 Te)	Security			302 (D1 Te)	15 000/15		10-fold CV
					15 000 (D2)	Center			15 000 (D2)	000 (D2)		(D2)
										10-fold CV Training-Test with date
	RevealDroid [118]	GP (AZ)		2016	24 600	VS		2016	22 592	separation (Training date < Test	X	X
						DB		2010-2012	5 538		date)
	Demontis et al. [119]	DB	2010-2012	121 329	DB Mod DB and		2010-2012 2010-2012,	5 615 10 500	10 runs, where 30 000 for training, and the rest for test	X	Too Old
						CT		2016
		GM (GP)	2010-2011	863	GM		2010-2011	1 260	10-fold CV, with same GW/MW
	McLaughlin et al. [120]											X
		McAfee Labs		2016 ?	3 627	McAfee Labs		2016 ?	2 475	ratio In training and test
		"		2016 ?	9 268	"		2016 ?	9 902
		Intel Security								2/3 of
	Alzaylaee et al. [121]			2016 ?	1 222	GM		2010-2011	1 222			1/3 of GW/MW	X
		(McAfee Labs)								GW/MW
										5 runs, where 60 000 units are
	Melis et al. [122]	DB	2010-2012	121 329	DB		2010-2012	5 615	randomly selected for training, And	X	Too Old
										the rest for testing
		GP1	2012-2014	5 000					CE1: 70% of DR	CE1: 30% of DR
		GP2	2012-2014	10 000	DR		2010-2012	5 560	+ GP1		+ GP1
	MKLDroid [123]	AndroidDrawer	2013-2016	2 399	VS		2013-2014	24 317	CE2: 70% of VS	CE2: 30% of VS	X	X
		(AD)	2013-2016	3 027	MYST		2015	3 000	+ GP2		+ GP2
		AnZhi (AZ)	2013-2016	2 481					CE3: DR + GP1	CE3: VS + GP2
		AppsApk (AA)	2013-2016	1 007					WEx: DR + VS		WEx: AD + AZ
		F-Droid (FD)	2013-2016	5 770					+ GP1 + GP2		+ AA + FD + SM
		SlideMe (SM)								MCLEx: 2 ⁄3 of		MCLEx: 1 ⁄3 of
										MYST + GP1		MYST + GP1
	Li et al. [124]	apkpure		2019	16 753	DB		2010-2011	5 560	5-fold CV	X	X
		360				AMD		2010-2016	16 753
		HKUST
	SMART [125]	GP		2015	5 600	DB		2010-2012	5 560	10-fold CV	X
	Xiao et al. [126]	GP		2016 ?	3 536	DB		2010-2013	3 567	10-fold CV	X
		GP (AZ)	2016-2017	5 346	VS, AZ		2016-2017	3 450	70% of older
	DroidCat [127]	"	2014-2015	6 545	VS, AZ		2014-2015	3 190	apps	30% of newer apps	X
		"	2012-2013	5 035	VS, AZ, GM, DB	2012-2013	9 084
		"	2009-2011	439	VS, AZ, GM, DB	2009-2011	1 254
	Rana et al. [128]	DB	2010-2012	5 560	DB		2010-2012	5 560	80% training and 20% testing	Too Old
										Everything divided in 10 parts,
	Ma et al. [129]	AZ		?	10 010	AMD		2010-2016	10 683			?	?
										Then 10-CV for each part
	Huang et al. [130]	GP		2018 ?	3 312	VS		?	3 312	10-fold CV	N/A
	Liu et al. [131]	DB	2010-2012	123 453	DB		2010-2012	5 560			X	Too Old
	HG-Learning [132]	Tencent Security Lab		2018 ? 2018 ?	83 784 13 313	Tencent Security Lab		2018 ? 2018 ?	4 433 106 912	912 83 784/106		13 313/4 433	X
		(Tr)				(Tr)
		" (Te)				" (Te)
								2010-2012
						DB
	BRIDEMAID [133]	GP		2016	9 804			2010-2011	2 794	2 974 applications tested	X	X
						GM
								2016
						CT

  Analyze inclusion relations, e. g., identify which sets are fully included in other sets, intersections or unions. ▶ (B3) Analyze inclusion hierarchies, e. g., identify if a set A is included in B, and B is then included in C, and so on. ▶ (B4) Analyze exclusion relations, e. g., if a set does not intersect with another, or if it does not have any intersection at all. ▶ (B5) Analyze intersection relations, e. g., if certain sets form intersections. ▶ (B7) Identify the sets that constitute a certain intersection. Users can hover the intersections and see which are the sets constitute it. ▶ (B8) Identify set intersections contained in a specific set. ▶ (B10) Analyze and compare set and intersection cardinalities. This is done by comparing the sizes shown in diagram.

Experimental Datasets 4 4.1 Definition of a representa- tive

	dataset . . . . . . . . .
	4.2 Sampling representative
	datasets . . . . . . . . . . .
	4.2.1 Sample size for one
	boolean characteristic . . .
	4.2.2 Sample size for multiple
	characteristics . . . . . . .
	4.2.3 Application to security
	experiments . . . . . . . .
	4.3 Evaluating existing
	datasets . . . . . . . . . . .
	4.3.1 Non-security related
	characteristics to consider
	4.3.2 Statistical test of homo-
	geneity for datasets . . . .
	4.3.3 Studied Android datasets
	4.4 Conclusion . . . . . . . . .

  Let 0 ≤ 𝛿 ≤ 1. A dataset D is 𝛿-representative of a population P for a finite set of classes K if ∀𝑘 ∈ K, |𝑝 D,𝑘 -𝑝 P,𝑘 | ≤ 𝛿, where 𝑝 D,𝑘 (resp. 𝑝 P,𝑘 ) is the proportion of class 𝑘 in D (resp. P).

Definition 4.1.1

Table 4 . 1 :

 41 Sample size of representative datasets of Google Play and AndroZoo for one or multiple characteristics (𝛿 = 0.01 and 𝐶 = 99%).

			Sample size	Sample size
			for one	for 36 864
	Dataset	Size	characteristic	characteristics
			(𝛿 = 0.01 and	(𝛿 = 0.015 and
			𝐶 = 99%)	𝐶 = 99%)
	Google Play	2 664 893	16 485	29 062
	AndroZoo	19 553 370	16 574	29 383

  Distribution of application classes for the top most 200 over 2 874 classes of AZ20 30𝑘 and compared with D 𝑚𝑖𝑥 and AMDAs proposed earlier in the chapter, we use AndroZoo as the population because it is one of the easiest source of Android applications that can be used to draw a large sample. The goal is to determine which datasets commonly used in the literature are not representative of AndroZoo. This dataset contains 19 553 370 applications as of June 15, 2022. Due to its size, it would be convenient to extract a representative sample as we explained before. Hence, we represent AndroZoo through AZ20 30𝑘 , a dataset of 30 000 randomly selected applications (corresponding to 𝛿 = 0.015 and 𝐶 = 99% for all 36 864 theoretical classes, see Section 4.2.2).

		12.0% 12.0%
			AZ2030k
		10.0%	Dmix
			AMD
	Proportion of apps	4.0% 6.0% 8.0%
		2.0%
		0.0%
		Classes
	Figure 4.2:

Xu et al. (2016), 'ICCDetector: ICC-Based Malware Detection on Android' [111]: Feizollah et al. (2017), 'AndroDialysis: Analysis of Android Intent Effectiveness in Malware Detection' ▶ DroidBench [163]: a collection of applications with source available [163]: Arzt et al. (2014), 'FlowDroid: Precise Context, Flow, Field, Object-Sensitive and Lifecycle-Aware Taint Analysis for Android Apps' designed to evaluate taint-analysis tools.

Table 4 . 2 :

 42 𝜒 2 test for homogeneity over AZ20 30𝑘 for a finite set of non-security characteristics

	Datasets	Dataset size	# of classes in common	# of classes only in the dataset	# of classes only in AZ20 30𝑘	Max 𝛿	Avg 𝛿 𝜒 2 value 𝑝-value	Conclusion
	AZ20 30𝑘	29 974	2 411	-	-	0	0	0
									Reject
	Drebin	5 304	181	141	2 230	0.1184 0.0007	27 902
									(C=99%)
									Reject
	AMD	23 258	128	16	2 283	0.0974 0.0007	39 771
									(C=99%)
	VirusShare								Reject
		28 896	250	65	2 161	0.1683 0.0007	44 537
	2015								(C=99%)
	VirusShare								Reject
		12 651	172	28	2 239	0.1595 0.0007	30 312
	2016								(C=99%)
	VirusShare								Reject
		9 945	234	39	2 177	0.0898 0.0006	22 050
	2017								(C=99%)
	VirusShare								Reject
		28 543	728	328	1 683	0.5057 0.0006	42 617
	2018								(C=99%)
	D 𝑚𝑖𝑥	16 424	982	447	1 429	0.0359 0.0004	17 852	Reject (C=99%)
									Reject
	AZ19 100𝑘	99 999	2 086	2 062	325	0.0320 0.0001	12 662
									(C=99%)
									Reject
	DroidBench	119	6	0	2 405	0.4773 0.0008	14 121
									(C=99%)
	AZ19 100𝑘								Reject
		5 794	648	291	1 763	0.0574 0.0005	15 390
	2015								(C=99%)
	AZ19 100𝑘								Reject
		27 516	1 183	921	1 228	0.0573 0.0004	30 725
	2016								(C=99%)
	AZ19 100𝑘								Reject
		6 524	688	274	1 723	0.0381 0.0004	12 378
	2017								(C=99%)
	AZ19 100𝑘								Reject
		24 549	1 216	547	1 195	0.0285 0.0003	14 754
	2018								(C=99%)
									Reject
	AZ20 10𝑘	9 971	1 370	423	1 041	0.0443 0.0003	6 296
									(C=99%)
	AZ20 20𝑘	19 927	1 361	397	1 050	0.0245 0.0001	3 560	Non-conclusive

5.1 Debiasing algorithm . . .

	5.1.1 Recall of characteristics
	and classes . . . . . . . . .
	5.1.2 Constant-size algorithm .
	5.1.3 Optimal debiasing algo-
	rithm . . . . . . . . . . . . .
	5.2 Generating debiased
	datasets . . . . . . . . . . .
	5.2.1 Characteristics used for
	debiasing . . . . . . . . . .
	5.2.2 Debiasing datasets . . . .
	5.2.3 Visual exploration of
	debiased datasets . . . . .
	5.2.4 Building mixed datasets
	for machine learning . . .
	5.3 Experiments with debi-
	ased datasets . . . . . . . .
	5.3.1 Classifying with machine
	learning . . . . . . . . . . .
	5.3.2 Results . . . . . . . . . . . .
	5.4 Conclusion . . . . . . . . .

  𝑘 𝑙 ≥ 𝑝 T,𝑘 𝑙 -𝛿 and K S ≠ ∅ then Remove one element of G(𝑘 ℎ ) chosen at random if 𝑝 G,𝑘 ℎ < 𝑝 T,𝑘 ℎ -𝛿 or 𝑝 G,𝑘 𝑙 > 𝑝 T,𝑘 𝑙 + 𝛿 then proportion of class 𝑘 in D is 𝑝 B,𝑘 + 𝑎 𝑘 |B| . A class is not underrepresented if its proportion is greater than 𝑝 T,𝑘 -𝛿, so we are looking for the smallest 𝑎 𝑘 such that 𝑝 B,𝑘 + 𝑎 𝑘 |B| ≥ 𝑝 T,𝑘 -𝛿, hence 𝑎 𝑘 ≥ |B|(𝑝 T,𝑘 -𝛿 -𝑝 B,𝑘

	8	𝑘 𝑙 ← arg min 𝑘∈K S 𝑝 G,𝑘 -𝑝 T,𝑘
	else return "Impossible";
	𝑟 ← one element of S'(𝑘 𝑙 ) chosen at random
	Remove 𝑟 from S' and add it to G
	return "Impossible"
	return G

T,𝑘 20 else break;

  1 is called to update 𝑑 𝑚𝑖𝑛 and G 𝑏𝑒 𝑠𝑡 .

		Algorithm 2: Optimal debiasing Algorithm
		Input: B, T, S, 𝛿
		Output: Generated dataset
	1 𝑑 𝑚𝑖𝑛 ← ∞; G 𝑏𝑒𝑠𝑡 ← "Impossible"
	2 if 𝐴𝑙 𝑔𝑜1(B, T, S, 𝛿) ≠ "Impossible" then 3 𝑑 𝑚𝑖𝑛 ← 2 max( 𝑘 𝑛 + B (𝑘), 𝑘 𝑛 -B (𝑘))
	4	G 𝑏𝑒 𝑠𝑡 ← 𝐴𝑙 𝑔𝑜1(B, T, S, 𝛿)
	5 G ← B
	6 for 𝑖 from 1 to 𝑑 𝑚𝑖𝑛 -1 do	⊲ Search smaller datasets
	7	if G = ∅ then break;
	8	𝑘 ℎ ← arg max 𝑘∈K (𝑝 G,𝑘 -𝑝 T,𝑘 )
	9 10	Remove one random element of class 𝑘 ℎ from G 𝑑 ← | |B| -|G| | + 2 max( 𝑘 𝑛 + G (𝑘), 𝑘 𝑛 -G (𝑘))
	11	if 𝑑 < 𝑑 𝑚𝑖𝑛 and 𝐴𝑙 𝑔𝑜1(G, T, S, 𝛿) ≠ "Impossible" then
	12	𝑑 𝑚𝑖𝑛 ← 𝑑; G 𝑏𝑒𝑠𝑡 ← 𝐴𝑙 𝑔𝑜1(G, T, S, 𝛿)
	13 G ← B
	14 for 𝑖 from 1 to 𝑑 𝑚𝑖𝑛 -1 do	⊲ Search larger datasets
	15	𝑘 𝑙 ← arg min 𝑘∈K (𝑝 G,𝑘 -𝑝 B,𝑘 )
	16	if |S(𝑘 𝑙 )| = 0 then
	17	K S = {𝑘 ∈ K | |S(𝑘)| > 0}
	23	𝑑 ← | |B| -|G| | + 2 max( 𝑘 𝑛 + G (𝑘), 𝑘 𝑛 -G (𝑘))
	26 return G 𝑏𝑒𝑠𝑡
		Theorem 5.1.1 Let B, S, T be three datasets and 𝛿 > 0. If there exists at least
		one 𝛿-representative dataset of T that is composed of elements of B and S, then
		Algorithm 2 produces such a dataset G such that the number of additions and
		deletions from the initial dataset B is minimal. If such a dataset does not exist,
		it returns "Impossible".
		Its worst-case temporal complexity is 𝑂 ( 1 𝛿 + |B|) 2 |K| .

18 if 𝑝 G,𝑘 𝑙 ≥ 𝑝 T,𝑘 -𝛿 and K S ≠ ∅ then 19 𝑘 𝑙 ← arg min 𝑘∈K S 𝑝 G,𝑘 -𝑝 21 𝑟 ← one element of S(𝑘 𝑙 ) chosen at random 22 Remove 𝑟 from S and add it to G 24 if 𝑑 < 𝑑 𝑚𝑖𝑛 and 𝐴𝑙 𝑔𝑜1(G, T, S, 𝛿) ≠ "Impossible" then 25 𝑑 𝑚𝑖𝑛 ← 𝑑; G 𝑏𝑒𝑠𝑡 ← 𝐴𝑙 𝑔𝑜1(G, T, S, 𝛿)

  this section, we are interested in producing datasets for experiments concerning Android malware detection. As identified in Chapter 2, three types of experimental scenarios can occur. The first scenario (market profiling) requires only unlabeled representative datasets that can be drawn randomly without requiring debiasing, as seen in Chapter 4. The two other cases, Algorithm evaluation and Machine learning, require labeled datasets with different proportions of malware/goodware samples. Therefore, we propose in Section 5.2.2 to debias datasets containing only malware or only goodware and mix them, in Section 5.2.4, to obtain mixed datasets directly usable for machine learning algorithms.

Table 5 . 1 :

 51 Results after debiasing datasets

	Base Dataset		Reference	Source	Diff	Del	Add		Debiased dataset G	Duration
	|B|		|T|	|S|	𝛿			|G|	Add ratio	𝜒 2	𝑝-value	validity
				Debiasing Goodware datasets
	NAZE-18-G	→	AZ20 30𝑘	AZ19 100𝑘 G					NAZE Debiased 18-G
	11 120		29 974	72 131	0.04	0	0	11 120	0.00	12 589 0
					0.02	438	50	10 732	0.47	12 058 0
					0.01	1 026 223	10 317	2.16	11 291 0
					0.005	1 885 632	9 867	6.41	9 726 0
					0.002 5	2 685 1 280	9 715	13.18	7 998 0
					0.001	5 808 1 289	6 601	19.53	5 104 0.999 44
				Debiasing Malware datasets
	Drebin	→	AZ20 30𝑘	VS 15-18, AMD					Drebin Debiased
	5 304		29 974	103 293	0.04	892	81	4 493	1.80	25 781 0
					0.02	3 634 116	1 786	6.49	20 178 0
					0.01	4 596 103	811	12.70	15 833 0
					0.005
	VS 15-18	→	AZ20 30𝑘	Drebin, AMD, AZ-17-18-M					VS Debiased 15-18
	80 035		29 974	32 562	0.04	66 475	21	13 581	0.15	16 709 0
					0.02	75 825	31	4 241	0.73	7 406 0
					0.01	78 074	43	2 004	2.15	4 349 1
					0.005	79 063	29	1 001	2.90	915 1
					0.002 5
				Debiasing the VirusShare dataset over time
	VS 15	→ AZ19 100𝑘 15	Drebin					VS Debiased 15
	28 896		5 792	5 304	0.04	13 877	4	15 023	0.03	29 475 0	0
					0.02	19 832	90	9 154	0.98	21 828 0	0
					0.01
	VS 16	→ AZ19 100𝑘 16	Drebin, VS 15					VS Debiased 16
	12 651		27 516	34 200	0.04	4 549	1	8 103	0.01	23 204 0	0
					0.02
	VS 17	→ AZ19 100𝑘 17	Drebin, VS 15-16, AMD					VS Debiased 17
	9 945		6 524	70 109	0.04	580	0	9 365	0.00	21 073 0	0
					0.02
	VS 18	→ AZ19 100𝑘 18	Drebin, VS 15-17, AMD					VS Debiased 18
	28 543		24 549	80 054	0.04	27 644	7	906	0.77	4 997 0.356 52	1
					0.02	28 189	7	361	1.94	1 809 1	1
					0.01

Table 5 . 2 :

 52 Mixed datasets for machine learning training and evaluation

	Malware	Goodware	Pivot	Set	MalGood		Constraints [11]	Mixed dataset
			year		prop. prop. C1 C2 C3 C4* Name		Size Avg 𝛿 𝑝-value
				Training 50% 50%	✓	×	✓	✓	DR-AG 𝐷𝑒𝑏	1	0.017	0
	Drebin Debiased (𝛿 = 0.01)	NAZE Debiased 18-G (𝛿 = 0.001)	2014	Test	10% 90%	✓ ✓ ✓ ×	✓ ✓	✓ ✓	DR-AG-C2 𝐷𝑒𝑏 DR-AG 𝐷𝑒𝑏		0.054 0.071	0 1
						✓ ✓	✓	✓	DR-AG-C2 𝐷𝑒𝑏		0.089	1
				Training 50% 50%	✓	×	✓	✓	VS-AG 𝐷𝑒𝑏	3	0.012	1
	VS Debiased 15-18 (𝛿 = 0.01)	NAZE Debiased 18-G (𝛿 = 0.001)	2017	Test	10% 90%	✓ ✓ ✓ ×	✓ ✓	✓ ✓	VS-AG-C2 𝐷𝑒𝑏 2 VS-AG 𝐷𝑒𝑏 1	0.013 0.057	1 0
						✓ ✓	✓	✓	VS-AG-C2 𝐷𝑒𝑏 1	0.057	0
	VS Debiased 15-18 (𝛿 = 0.04)	NAZE Debiased 18-G (𝛿 = 0.01)	2017	Training 50% 50% Test 10% 90%	✓ ✓	× ×	✓ ✓	✓ ✓	VS-AG 𝐷𝑒𝑏,𝛿=0.4 16 VS-AG 𝐷𝑒𝑏,𝛿=0.4 2	0.018 0.057	0 0
	Drebin	AZ19 100𝑘 G	2014 Training 50% 50%	✓	×	✓	✓	DR-AG	10	-	-
	VS 15-18	AZ19 100𝑘 G	2017	Training 50% 50% Test 10% 90%	✓ ✓	× ×	✓ ✓	✓ ✓	VS-AG VS-AG	133 6	--	--
	AndroCT [86]	2014 2017	Training 50% 50% Test 10% 90% Training 50% 50% Test 10% 90%	✓ ✓ ✓ ✓	× × × ×	✓ ✓ ✓ ✓	✓ ✓ ✓ ✓	ACT14 ACT14 ACT17 ACT17	19 12 26 4	0.065 0.057 0.054 0.093	0 0 0 0
	AZ20 30𝑘 with labels	2014 2017	Training 50% 50% Test 10% 90% Training 50% 50% Test 10% 90%	✓ ✓ ✓ ✓	× × × ×	✓ ✓ ✓ ✓	✓ ✓ ✓ ✓	AZL14 AZL14 AZL17 AZL17	4 8 9 2	0.015 0.057 0.019 0.059	0 0 0.894 0

Table 5 . 3 :

 53 Mean and max AUC of machine learning classifiers depending on their training set and their test set when the pivot year is 2017. In bold face: best AUC on a test set

	P P Train P	Test P P P P P	ACT17 Mean Max	AZL17 Mean Max	VS-AG Mean Max	VS-AG 𝐷𝑒𝑏 Mean Max
	ACT17	0.62	0.65	0.67	0.71	0.76	0.85	0.74	0.78
	AZL17	0.72	0.81	0.78	0.86	0.71	0.89	0.75	0.80
	VS-AG	0.62	0.68	0.66	0.73	0.96	0.97	0.74	0.84
	VS-AG 𝐷𝑒𝑏	0.70	0.74	0.71	0.75	0.86	0.89	0.83	0.87
	DR-AG	0.54	0.57	0.53	0.60	0.57	0.63	0.54	0.58

Table 5 . 4 :

 54 Mean and max AUC of machine learning classifiers depending on their training set and their test set when the pivot year is 2014. In bold face: best AUC on a test set the very limited size of the test set of DR-AG 𝐷𝑒𝑏 , these results are unreliable we configured the software with only two families (goodware/malware) and we extracted features related to control flow graph that manipulates sensitve APIs.

	P Train P P	P	Test P P P P	ACT14 Mean Max Mean Max Mean Max AZL14 DR-AG 𝐷𝑒𝑏
	ACT14	0.69	0.73	0.71	0.74	0.69* 0.83*
	AZL14	0.75	0.81	0.76	0.82 0.72* 0.99*
	DR-AG	0.57	0.58	0.56	0.58	0.48* 0.50*
	DR-AG 𝐷𝑒𝑏	0.62	0.66	0.61	0.66 0.72* 0.80*
	* due to

  1 𝑛 and 𝑝 D,𝑘 ℎ = 𝑎 2 𝑛 . Since Δ T 𝑝 G,𝑘 ℎ > 𝛿 and Δ T 𝑝 D,𝑘 ℎ ≤ 𝛿 (due to D being representative), we can conclude that 𝑝 G,𝑘 ℎ > 𝑝 D,𝑘 ℎ , i.e., 𝑎 1 > 𝑎 2 . Since 𝑝 G ′ ,𝑘 ℎ = 𝑝 G,𝑘 ℎ -1 𝑛 = 𝑎 1 -1𝑛and 𝑎 1 -1 ≥ 𝑎 2 , we conclude that 𝑝 G ′ ,𝑘 ℎ ≥ 𝑝 D,𝑘 ℎ and Δ T 𝑝 G ′ ,𝑘 ℎ ≥ Δ T 𝑝 D,𝑘 ℎ . Since D is itself representative of T, we know that Δ T 𝑝 D,𝑘 ℎ ≥ -𝛿 and therefore that Δ T 𝑝 G ′ ,𝑘 ℎ ≥ -𝛿.Let us show that there exists 𝑘 ′ 𝑙 such asΔ T 𝑝 G,𝑘 𝑙 < Δ T 𝑝 D,𝑘 ′ 𝑙 ≤ 𝛿. Since Δ T 𝑝 G,𝑘 ℎ > Δ T 𝑝 D,𝑘 ℎ and 𝑘 Δ T 𝑝 G,𝑘 = 𝑘 Δ T 𝑝D,𝑘 , we can conclude that there exists 𝑘 ′ 𝑙 such as Δ T 𝑝 G,𝑘 ′ 𝑙 < Δ T 𝑝 D,𝑘 ′ 𝑙 . Furthermore, 𝑘 ′ 𝑙 ∈ K 𝑆 because if S were empty it would not be possible for D to have more elements than G for this class. of their size, i.e., | |B| -|G| |. The number of modification between G and 𝐴𝑙 𝑔𝑜1(G) is 2 max( 𝑘 𝑛 + G (𝑘), 𝑘 𝑛 - G (𝑘)) (cf. the previous proof algorithm). So the number of modifications between B and 𝐴𝑙 𝑔𝑜1(G) is | |B| -|G| | + 2 max( 𝑘 𝑛 + G (𝑘), 𝑘 𝑛 - G (𝑘)). First, let us show that the algorithm always halts. This algorithm halts either if 𝑑 𝑚𝑖𝑛 ≠ ∞ (so both loops have a finite number of iterations) or if the source dataset lacks some element (lines 16 to 20). If a solution G exists, then at some point Algorithm 2 will call Algorithm 1 with a dataset whose size is the same as G (either at line 2 if |G| = |B|, at line 10 if |G| < |B| or at line 26 if |G| > |B|) and Algorithm 1 will find a solution (due to Theorem A.0.4). If at some point it is not possible to add an element to a underrepresented class because there is no such element in the source dataset, then the algorithm halts (lines 5 to 9). Second, we show that doing 𝑑 𝑚𝑖𝑛 -1 iterations per loop is sufficient (lines 6 and 15). Let G 𝑏𝑒𝑠𝑡 be the best dataset found so far, associated to a distance 𝑑 𝑚𝑖𝑛 , and a better solution G' associated to a distance 𝑑 ′ such as 𝑑 ′ < 𝑑 𝑚𝑖𝑛 . Remark that at each step, 𝑖 = | |B| -|G| | as one element is modified per iteration. Denote 𝑗 the iteration at which G' could be found. Since 𝑑 ′ = | |B| -|G| | + 2 max( 𝑘 𝑛 + G (𝑘), 𝑘 𝑛 - G (𝑘)) ≥ 𝑗 and 𝑑 ′ < 𝑑 𝑚𝑖𝑛 , we conclude that 𝑗 ≤ 𝑑 𝑚𝑖𝑛 and therefore that G' will be found by the algorithm. The worst-case temporal complexity of Algorithm 2 is 𝑂 ( 1 𝛿 + |B|) 2 |K| . Proof. Remark that for all datasets A and B, the number of additions and deletions required to obtain B from A is lower than or equal to |A| + |B|. Theorem A.0.2 shows that there exists a representative dataset D of size ⌈ 1 𝛿 ⌉. So the minimal distance 𝑑 𝑚𝑖𝑛 to reach a representative dataset D from the initial base dataset B is bounded by ⌈ 1 𝛿 ⌉ + |B|. So Algo. 2 will do at most 2(⌈ 1 𝛿 ⌉ + |B|) iterations. Each iteration calls Algo. 1, whose temporal complexity is 𝑂(|G| • |K|) where the size of G is at most |B| + 𝑑 𝑚𝑖𝑛 ≤ ⌈ 1 𝛿 ⌉ + 2|B|. So the worst-case temporal complexity of Algo. 2 is 𝑂 ( 1 𝛿 + |B|) 2 |K| . Profilage et Visualisation de Datasets d'Applications Android Malveillantes Mot clés : Malware, Datasets, Biais, Visualization Résumé : Les dispositifs mobiles sont ubiquitaires: aujourd'hui la majorité des gens possèdent un téléphone mobile. A cause de ce fait, ces dispositifs sont une cible d'intérêt pour les attaquants. Ces attaques sont véhiculées au travers des applications malveillantes qui peuvent nuire aux dispositifs mobiles. Les chercheurs en analyse de malware travaillent à reconnaître ces types de programmes avant qu'ils soient installés sur un dispositif utilisateur. Pour faire cela, ils réalisent des expériences pour automatiquement détecter ces malware, où ils utilisent des ensembles de malware et des applications bénignes déjà connues. Selon le dataset choisi, les résultats des expériences peuvent être acceptables ou bien exceptionnellement bons parce que surestimés. Par conséquent, les datasets de malware et applications bénignes sont des éléments importants à considérer quand nous élaborons une expérience. Cette thèse présente, premièrement, une méthode pour évaluer la qualité des datasets basée sur un test statis-tique qui aide à comparer un dataset créé avec un grand ensemble d'applications par exemple issu d'un magasin d'applications. Nous montrons alors que les datasets historiques de la littérature sont de mauvaise qualité, ce qui justifie le besoin de créer des nouveaux datasets plus à jour. Deuxièmement, nous introduisons un algorithme pour mettre à jour des datasets mixtes de malware/goodware de mauvaise qualité afin de ressembler à un dataset cible qui ne peut pas être utilisé directement, e. g. un magasin d'applications. Nous évaluons les datasets mixtes mis à jour en utilisant un algorithme d'apprentissage automatique et nous montrons que la détection de malware sur notre dataset mis à jour devient un problème plus difficile à résoudre. Enfin, nous introduisons DaViz, un outil de visualisation de datasets pour explorer et comparer des datasets d'applications Android. Cet outil permet aux chercheurs de visualiser les biais dans les datasets de la littérature, et d'obtenir des informations utiles à leur propos.

	Theorem A.0.5

Titre :

Debiasing Android Datasets

Bibliography

Here are the references in citation order.

Due to the definition of 𝑘 𝑙 , Δ T 𝑝 G,𝑘 𝑙 ≤ Δ T 𝑝 G,𝑘 ′ 𝑙 < Δ T 𝑝 D,𝑘 ′ 𝑙 ≤ 𝛿. Let 𝑏 1 , 𝑏 2 , 𝑏 3 , 𝑏 4 four naturals such as 𝑝 G,𝑘 𝑙 = 𝑏 1 𝑛 , 𝑝 T,𝑘 𝑙 = 𝑏 2 𝑛 , 𝑝 D,𝑘 ′ 𝑙 = 𝑏 3 𝑛 and 𝑝 T,𝑘 ′ 𝑙 = 𝑏 4 𝑛 . We can conclude of the previous inequalities that

𝑛 -𝑝 D,𝑘 𝑙 ≤ 𝛿. This proves that the "If" condition is never true if D exists.

Finally, if D exists, then Algorithm 1 returns a dataset. This dataset is representative because the "While" condition is true for the last iteration. When the algorithm outputs a dataset, it is representative and its size is the same as B. If such dataset does not exist, the algorithm cannot output one. Since the algorithm always halts, it must return "Impossible".

We denote 𝑑 𝐻 (A, B) the number of modifications between A and B. More precisely, we extend the Hamming distance to a distance between two sets by defining 𝑑 𝐻 (A, B) as

is the number of elements present in only one dataset, A or B. In other words, it is the minimal number of addition and deletions to transform A into B (and vice-versa). For any class 𝑘 that is overrepresented in B, 𝑛 + B (𝑘) is the minimal number of removal of elements of class 𝑘 such as 𝑘 is not overrepresented anymore. 𝑛 - B (𝑘) = max(0, ⌈-|B|(Δ T 𝑝 B,𝑘 + 𝛿)⌉) is similar for underrepresented classes.

To make a representative dataset from B, one must remove at least 𝑑 is equal to the number of additions and deletions between B and 𝐴𝑙 𝑔𝑜1(G). This number can be decomposed as the sum of the number of additions and deletions between B and G and between G and 𝐴𝑙 𝑔𝑜1(G).

The number of modifications between G and B is simply the difference