Edgar Haddad
email: edgar.haddad@orange-ftgroup.com

Nadine Malhouroux
email: nadine.malhouroux@orange-ftgroup.com

Patrice Pajusco
email: patrice.pajusco@telecom-bretagne.eu

Michel Ney
email: michel.ney@telecom-bretagne.eu

On the Frequency Dependence of UWB Indoor Channel Parameters: 3D Ray Tracing and Measurement

In this paper, measurements and 3D ray tracing predictions of multiple input multiple output (MIMO) ultra wideband (UWB) channel parameters in a residential area are presented. The studied UWB channel parameters, i.e. the delay spread and the azimuth and elevation angular spreads, are absolutely required in propagation modeling during the specification phase of future mobile MIMO communications systems. The evolution of the predicted parameters with the number and type of interactions of the simulated channel is also presented. Then, we focus on the frequency dependence of the predicted parameters and compare it with the measurement. Results show limited variations with increasing frequency. This implies that most of indoor normalized channel models can be extended and used for other frequencies.

I. INTRODUCTION

Wireless communications is one of the fastest growing fields in the communication engineering world. The increase in capacity of wireless communicating systems is still a constant goal in the mobile communication research area and many advanced radio techniques promising in terms of capacity have been deployed.

The evolutions of wireless systems based on a high spectral efficiency relate, among others, to two major topics: the improvement of the radio link by using MIMO configurations and the use of new frequency bands. For MIMO based systems, the system performance depends not only on the SNR, but also on the amount of spatial diversity the system can exploit [START_REF] Winters | On the Capacity of Radio Communication Systems with Diversity in a Rayleigh Fading Environment[END_REF][START_REF] Foschini | On limits of wireless communications in a fading environment when using multiple antennas[END_REF]. This spatial diversity relies on the delay and angular power profiles of the propagation channel and on their derived parameters like delay and angular spreads. As a result, a precise space-time propagation channel characterisation is required in order to evaluate the awaited profits of these techniques. 3D ray tracing based models can be used to predict the channel delay and angular profiles in a deterministic way [START_REF] Mantel | Measurement based verification of delay and angular spread raytracing predictions for use in urban mobile network planning[END_REF], and hence provide the required spatiotemporal channel characterization.

This paper presents a comparison of the measured and predicted delay and angular spreads of UWB indoor channel parameters with some focus on their frequency dependence. Section II gives definitions of the delay spread and angular spread. Section III presents the channel simulation tool and an analysis of the relevant phenomena. Section IV describes the measurement set-up performed in a typical residential flat. The comparison between measured and predicted channel parameters and the influence of the frequency are presented in section V. We conclude this work with an overview of the results in section VI.

II. DEFINITION OF DELAY AND ANGULAR SPREADS

The delay spread σ T characterizes the dispersion of the transmitted signal in time due to the time-dispersive influence of the channel. It is defined as the standard deviation of the delay values of the different paths, weighted proportional to the power in each corresponding path, i.e., t n P n are the delay and power of path n respectively.

The angular spread σ θ characterizes the broadening of the transmitted signal in azimuth and elevation due to the angulardispersive influence of the channel. The larger the angular spread, the higher is the space selecting fading. Note that the definition of angular spread is not as simple as that of delay spread, since angle is a periodic data. In this paper, we used the 3GPP definition of angular spread [START_REF]SCM-135: Spatial channel model text description[END_REF], which is the most commonly applied and it is calculated by: We note that since the angular spread should be independent of a linear shift in the angles of arrival and departure, it should be the minimum of σ over all .

) ( min ∆ = ∆ θ θ σ σ (2) ∑ ∑ - ∆ + = ∆

A. 3D Channel Simulation Tool

MATRIX is a 3D ray tracing software that calculates the ray paths between a given transmitter (Tx) and receiver (Rx). It is based on geometrical optics (GO) and the uniform geometrical theory of diffraction (UTD). The propagation phenomena taken into account are combinations of multiple transmissions, reflections and diffractions.

Since a detailed description of the 3D indoor environment is essential for a proper wave propagation modelling, we need to account for walls, floor, ceiling, windows and doors as well as some details such as furnitures. Plane facets with 3D dimensions describe indoor environment. Each facet is affected to a material which is characterized by its permittivity, conductivity and thickness. The electromagnetic properties of various building materials have been estimated in a previous study over the 2-16 GHz frequency band [START_REF] Tesserault | Multi-frequencies characterization of building materials: Angular and polarization analysis[END_REF]. Consequently, MATRIX can operate over a large frequency range. In addition, it takes into account the UWB antenna including polarization and radiation pattern.

As a result, for a given Tx-Rx configuration, the simulation tool provides a list of all the identified ray paths. Each ray is characterized by its complex propagation vector, absolute delay, direction of arrival (DOA) and direction of departure (DOD). It is therefore possible to easily estimate spatiotemporal channel characteristics such as delay spread and angular spread. Figure 1 shows an example of a 3D indoor scene with different ray paths found by MATRIX between the transmitter (Tx) and receiver (Rx).

B. Analysis of the Relevant Phenomena

It is known that ray tracing computation time increases rapidly with the number of reflection or diffraction phenomena. In order to reduce computation time, we have to compute useful paths only and ignore those without significant contribution to energy or any influence on the spatio-temporal channel parameters. For this purpose, we investigated on the evolution of both delay and angular spreads as a function of the number and type of phenomena taken into account. Also, different transmitter positions were considered in order to characterise the propagation channel in a residential environment.

Figure 2 represents the evolution of the delay spread and the azimuth and elevation angular spreads versus the type of added paths for 4 different transmitter positions. Each step on the horizontal scale corresponds to the addition of a new type of path in the simulation. Paths containing reflexions were added first, then paths containing diffractions. Finally, paths containing a mix of reflexions and diffractions were considered. Note that a maximum of 6 transmissions have been allowed for every path. One can observe that graphs representing the evolution of the delay spread and azimuth angular spread of the two links Tx1Rx1 et Tx2Rx1 have the steepest slope. This is due to the fact that both links can be considered in NLOS situation when a concrete thick wall is present between transmitters and receiver. As direct paths are strongly attenuated, rays that do not impinge at that wall will have some significant contribution at the receiver in comparison to the direct path.

D.P. 1R 2R 3R 4R 5R 6R 1D 2D 1R1D 2R1D 3R1D 4R1D 1R2D 2R2D
For most Tx positions, paths with six reflections or 2 diffractions or one diffraction and more than 3 reflections do not have any significant influence on the channel characteristics. For instance, they do not have any significant contribution to energy as their dynamic energy is 30 dB below the main path energy. Furthermore, the computation time required to find paths with a maximum of 6 reflections is 95 times larger than the time required to find paths with a maximum of 5 reflections. Also, it requires a computation time 160 times larger to find paths with one diffraction and maximum 4 reflections than to find paths with one diffraction and maximum 3 reflections. From the above observations, one can make some compromise between performance and computation time by only considering paths containing a maximum of 5 reflections or 2 diffractions, or one diffraction and maximum 3 reflections. Finally, a maximum number of 6 transmissions is allowed for each path. To evaluate the accuracy of our simulator, we compare hereafter the simulations to the measurements.

IV. DESCRIPTION OF THE MEASUREMENTS

A. Measurement Set Up

The measurement equipment relies on two UWB circular arrays at both sides of the transmission link. A monocone UWB specific antenna, designed by Orange Labs, was used. Its construction is very simple and inexpensive. It is matched from 3 to 12 GHz with a VSWR (Voltage Standing Wave Ratio) value less then 2. A UWB array structure was introduced in [START_REF] Pajusco | Novel array structure for space-time characterization of the UWB channel[END_REF], it combines a virtual uniform circular array (UCA) and a 5-axes monocone UWB antenna. It has been proved that the use of several antennas with particular orientations removes major drawbacks of classical uniform circular array. This setup offers some acceptable compromise between performance and complexity.

Propagation channel measurements were performed in frequency domain. The transmission channel was estimated with a VNA (Vector Network Analyser). Two LNA (Low Noise Amplifier) were used to improve the link budget. The MIMO configuration consists in measuring all links between all transmitting antennas toward all receiving antennas for each possible location. To reduce the experiment duration, only 3 antennas were measured at each side of the link. The radius of the UCA is 8 cm and the virtual location number was set to 60 to comply with Shannon sampling theory. The propagation channel was sampled over 4000 equally spaced frequencies between 2.5 and 12.5 GHz. It covers the FCCdefined frequency band for indoor UWB applications.

The whole measurement process was automated using specifically developed Matlab TM software. The schematic diagram of the measurements equipment is depicted in Fig. 3.

B. Environment

An extensive UWB MIMO channel measurement campaign was carried out in a residential flat built inside the Orange labs premise [START_REF] Pajusco | Techniques de caractérisation ULB doublement directionnelle[END_REF]. This flat (see Fig. 4) is furnished and consists of 2 bedrooms, a kitchen, a desk room and a large living room. Thirteen different transmitter locations were investigated. To allow a fair comparison between measurements and simulations, we accounted for the influence of antennas and MIMO configuration in our simulations. For this purpose, we generated the channel transfer functions of the sets of rays simulated with MATRIX, according to the MIMO circular array configuration and antenna characteristics.

A. Power Delay Profile and Angular Delay Profile

The power delay profile (PDP) and the azimuth and elevation angular profiles (PAP) of the different MIMO configuration links were computed from the corresponding transfer functions. For each MIMO link, the power delay profile is considered as the average of the power delay profiles of all single input single output (SISO) links between the various transmitting and receiving antennas of the corresponding MIMO link. The azimuth and elevation power angular profiles at the receiver side of a MIMO link were considered as the average of the azimuth and elevation angular profiles of all single input multiple output (SIMO) links between the various transmitting and receiving antennas of the corresponding MIMO links. The power angular profile of the different single input multiple output links were computed by carrying out a directional analysis of the channel at all intervals of time to find the direction of arrival of rays within that interval. At each interval of time, a dynamic threshold of useful energy has been fixed in order to help reducing secondary lobe effects during beamforming.

Figure 5 represents the measured and simulated power delay profile and azimuth and elevation power angular profile of the link Tx6Rx1 for a central frequency equal to 6 GHz with 1-GHz bandwidth. Main paths have been found, but the simulated power delay profile and angular power profiles contain less energy than the measured ones. This may be due to our hypothesis that considers all reflections as specular reflections and ignores scattered signals. This underestimation of the power profiles will lead to underestimate in most cases the spatio-temporal channel characteristics as well, like delay and angular spreads.

B. Delay Spread and Angular Spread

The delay spread was computed from each collected power delay profile (PDP) whereas the azimuth and elevation angular spreads were computed from the corresponding power angular profiles (PAP). The delay spread and the angular spreads were computed over 8 sub-bands of 1 GHz each. The central frequencies of the selected bands extend from 3 to 10 GHz with one GHz step. Then, we computed both the average value and the standard deviation for the 8 sub-bands. As a result, it was possible to analyse the effect of the central frequency on the characteristics of the UWB propagation channel. Figure 6 illustrates the average measured and simulated delay spread, azimuth and elevation angular spreads at the receiver side of all Tx positions of the figure 4 against central frequency. For each band, the value of the standard deviation of each parameter is represented by the length of the vertical line. A small offset is observed between simulations and measurements. As expected, the predicted delay spread and azimuth and elevation angular spread values tend to be lower than the measured ones. Their values are in agreement with those reported by others [START_REF] Tchoffo Talom | Modélisation déterministe du canal de propagation indoor dans un contexte Ultra Wide Band[END_REF][START_REF] Rautiainen | Measurements and 3D Ray Tracing Propagation Predictions of Channel Characteristics in Indoor Environments[END_REF]. The occurrence of lower predicted delay spread values can be attributed partially to the absence of scattered signals from small furniture in our simulations as scattering may add significant energy more distributed in space into simulations. Besides, it is difficult to model exactly all the environmental details. Concerning the frequency dependence of the parameters, we did not observe an obvious frequency dependence of the delay spread and the azimuth and elevation angular spreads. Their measured and predicted values are relatively stable over all frequencies from 3 to 10 GHz. A slight tendency to decrease with frequency is observed for the measured parameters. However, one can roughly consider their values as constant over all the frequency range. So an indoor channel model normalized for a frequency in the 3-10 GHz frequency band can be extended and used for other frequencies of this band.

VI. CONCLUSIONS

In this paper, we presented a comparison between measurements and 3D ray tracing predictions of MIMO UWB channel parameters in a residence. We showed also some specific interest on the frequency dependence of these parameters. The delay spread and the azimuth and elevation angular spread were computed over 8 sub-bands of 1 GHz each with central frequencies extending from 3 to 10 GHz.

A small offset is observed between measurements and simulations. Predicted values tend to be lower than measured ones. This may be due to the absence of scattered signals from small furniture in our simulations. The parameters did not present an obvious frequency dependence. Their values are relatively stable over all the frequency range. This implies that most of indoor normalized channel models can be extended for other frequencies in the 3-10 GHz frequency band.

Future works will focus on the comparison of MATRIX with other prediction tools and the addition of scattered waves to our simulations to have a better prediction of the UWB parameters.

  and are the power and angle of arrival or departure of path n respectively and ∆ is a linear shift angle.

Fig. 1

 1 Fig. 1 An example of a 3D indoor scene with different ray paths found between the transmitter (Tx) and receiver (Rx)

Fig. 2

 2 Fig. 2 Evolution of delay spread and azimuth and elevation angular spread vs. type of added paths: D.P. = Direct Path, R = Reflection, D = Diffraction

Fig. 3

 3 Fig. 3 Schematic diagram of the measurement equipment

Fig. 4 A

 4 Fig. 4 A representation of the residential flat

Fig. 5

 5 Fig. 5 Measured and simulated PDP and azimuth and elevation PAP of the link Tx6Rx1; the central frequency is 6 GHz with a bandwidth of 1 GHz

Fig. 6

 6 Fig. 6 Average measured and simulated RMS delay spread, azimuth and elevation angular spread and standard deviation of all Tx positions vs. central frequency


Jean-François Lalande
email: jean-francois.lalande@inria.fr

Valérie Viet

Triem Tong

Mourad Leslous

Pierre Graux

Challenges for Reliable and Large Scale Evaluation of Android Malware Analysis

INVITED TALK EXTENDED ABSTRACT

Since Android became the first smartphone operating system, malware developers have put large efforts to craft new threats uploaded to the Google Play store and other third market places. Companies and researchers now include in their activities the analysis of malware targeting smartphones. Most of the time, the problem that is addressed consists in deciding if an application should be considered as a malware or not. Nevertheless, once a malware is tagged as a malicious application, users that have been infected ask for more technical explanations about the threat they have been exposed to. Dissecting a malware requires a lot of efforts for a security analyst to be conducted and companies are in demand of new tools for automatizing the analysis.

From a research perspective, testing new ideas about malware analysis requires performing experiments on malware datasets. Compared to other operating systems, Android has fast development cycles with a new major release each year. A lot of malware samples do not run anymore when executed on new versions of Android. Experiments of the literature becomes quickly out of date and non reproducible when studying few samples. Thus, working on larger datasets, built at the time of writing, may give more consistent experimental results. New challenges come from using such datasets. First, as the behavior of the samples are unknown, the obtained results from the experiments are difficult to evaluate. Second, the experiment itself may require a large amount of time, depending of the quality of the automatization and the complexity of the analysis. Third, the protections that are put by developers in the malware decrease the quality of the results. This paper discusses these challenges and describes our efforts to build reliable and large scale experiments.

A. Lessons Learned from Manual Investigation

We started our research efforts in malware analysis with the help of several students that were asked to reverse known or unknown malware samples. Well known samples, such as SimpleLocker [START_REF] Lipovsky | RESET analyzes first android fileencrypting, TOR-enabled ransomware[END_REF] or DroidKungFu [START_REF] Jiang | Security alert: New sophisticated android malware droidkungfu found in alternative chinese app markets[END_REF], help to discover the basics of reverse engineering and to study complex attacks in a reasonable amount of time. Most of the time, picking randomly a sample identified as malicious by online platforms such as VirusTotal gives malware of low interest that send premium rated SMS or display unwanted ads.

Studying malware samples helped us to understand that malware developers now implement countermeasures to defeat both static and dynamic analysis tools. To illustrate and discuss these countermeasures, we built a small representative dataset, called kharon dataset [START_REF] Kiss | Kharon dataset: Android malware under a microscope[END_REF], that technically describes the way some samples try to escape analysis tools. These countermeasures can take different forms.

For defeating static analysis, malware developers use: malformed files raising an exception when parsed but keeping intact the application behavior; obfuscated payloads mixed with benign code which complexifies manual investigations; native code that hide parts of the payload; dynamic loading or packers that make the code unavailable without analyzing the runtime. These countermeasures make unreliable using solely static analysis techniques.

For defeating dynamic tools, malicious applications contain countermeasures for interacting and testing their environment: emulators can be detected because of the lack of hardware components or because emulation side-effects [START_REF] Vidas | Evading Android Runtime Analysis via Sandbox Detection[END_REF]; reconnaissance techniques can recognize known analysis tools; logic bombs may encapsulate the payload and prevent its execution [START_REF] Fratantonio | TriggerScope: Towards Detecting Logic Bombs in Android Applications[END_REF]; malicious code may require that the user uses the graphical interface of the application. Thus, any fully automated experiment on a dataset may suffer from these countermeasures and there are little chances to observe the dynamic behavior of a malware sample at runtime. For example, Tam et al. reported a successful stimulation of 2.78% to 3.08% of the malicious code related to SMS sending on their dataset [START_REF] Tam | CopperDroid: Automatic Reconstruction of Android Malware Behaviors[END_REF]. When targeting more general parts of the code (telephony abuse, introspection, contact access), we observed a covering of 20% at runtime, without any particular stimulation [START_REF] Abraham | GroddDroid: A gorilla for triggering malicious behaviors[END_REF]. These variations suggest that the chosen dataset may influence the experimental results and research efforts should be encouraged for elaborating evaluation datasets.

B. Existing Malware Datasets

Several papers have defined working datasets of malware in order to improve the repeatability of experiments. As pointed out by Wei et al. [START_REF] Wei | Deep Ground Truth Analysis of Current Android Malware[END_REF], the malware Genome project [START_REF] Zhou | Dissecting Android Malware: Characterization and Evolution[END_REF] is one of the first dataset to be largely used by the community which is now outdated and no longer supported.

In parallel of this academic dataset, a lot of researchers use randomly chosen malware from public or fee paying repositories such as Contagio Mobile, VirusTotal, Koodous. This approach has several advantages. It avoids building manually any contextual information for each sample, for example the family of the samples, the location of the malicious code, the triggering conditions, the required environment. Nevertheless, manipulating a large dataset without any knowledge about the samples makes difficult a precise evaluation of the obtained results. At least, conducting large scale experiments on thousands of malware may improve the confidence in the results, compared to an evaluation conducted on a small number of samples manually reversed.

Recently, new efforts have been done for building up-todate datasets with additional contextual information, as an improvement compared to random datasets. For example, AndroZoo [START_REF] Allix | AndroZoo: collecting millions of Android apps for the research community[END_REF] is a large dataset of three millions Android applications. In addition to the samples themselves, AndroZoo contains an analysis of pairs of repackaged [START_REF] Li | Understanding Android App Piggybacking: A Systematic Study of Malicious Code Grafting[END_REF] which helps to distinguish the original application and the repackaged one with the introduced payload. Another dataset, called AMD [START_REF] Wei | Deep Ground Truth Analysis of Current Android Malware[END_REF], has been built from 2010 to 2016 and contains 24,650 malware spread among 71 families. Each representative sample has been manually reversed and a graphical representation of the malicious parts of the code is given. Nevertheless, a structured and digital version of these results is not available yet. The AMD dataset is, by far, the first to structure the nature of the malicious code but we believe that more information is needed when performing experiments when the malware is run.

Using an existing dataset must be clearly linked with the reasons authors created it. For example, a dataset without enough precise attributes would not be usable for machine learning algorithms. Similarly, for an experiment that intends to execute the payload at runtime, the name of all methods that are malicious are required, especially when the code is repackaged in a benign application. The conditions to be met for executing the payload, i.e. the Android version, the events to trigger, the data expected by the malware, needs to be given for repeating experiments. Without such information, we believe that it is hazardous to compare experimental results on the same dataset as illustrated later in Section D and E.

C. Building Malware Datasets

Building a dataset from scratch is thus a difficult task. During our work on a specific dataset focused on Android malware that use native code, we had to deal with the problem of determining if a sample can be confirmed as malicious. We collected 2000 malware samples by downloading each day 20 recent samples from the Koodous repository and 30 random samples from the AndroZoo repository [START_REF] Allix | AndroZoo: collecting millions of Android apps for the research community[END_REF]. We identified 683 samples that use native calls in order to build our specialized dataset. Submitting these samples to VirusTotal is the usual method that enables to decide if these samples are real malware (TP for True Positive) and not regular applications submitted to the repositories (False Positive) [START_REF] Feng | Apposcopy: semanticsbased detection of Android malware through static analysis[END_REF][START_REF] Duan | Things You May Not Know About Android (Un)Packers: A Systematic Study based on Whole-System Emulation[END_REF][START_REF] Zheng | DroidAnalytics: A Signature Based Analytic System to Collect, Extract, Analyze and Associate Android Malware[END_REF][START_REF] Leslous | GPFinder: Tracking the Invisible in Android Malware[END_REF]. As shown in Figure 1, 48% of samples are not recognized by any antiviruses used by VirusTotal. Then, for a detection of at least 5 antivirus and more the curve is almost linear: there is no obvious threshold to decide that a sample has been recognized by enough antiviruses to classify it as a TP. We were expecting a drop of detection for a certain number of antiviruses, as represented by the clear curve. Additionally, these results may change with time, as the used antiviruses update frequently there data. We conclude from this experiment that using VirusTotal as an Oracle for confirming that a sample is malicious is not reliable, especially for recent samples.

D. Automatizing Dynamic Analysis

Large scale experiments require an automatization of the analysis process. For a static analysis, a distributed infrastructure can be employed, but for executing samples to observe its actions, the problem is more complex.

First of all, the setup of the execution environment is of primary importance. We believe that executing a sample in an emulator would lead to inconsistent results because of their ability to detect emulation artifacts [START_REF] Vidas | Evading Android Runtime Analysis via Sandbox Detection[END_REF]. Thus, a reliable environment requires a real smartphone connected to the internet and with plausible traces of usage. The version of the operating system may also influence the results and thus, repeating experiments may avoid compatibility issues and help to get a working malware execution. Additionally, because experimenting with a malware may damage the operating system, a fresh install should be restored between two experiments. Such a setup cannot be easily deployed in a cloud infrastructure which limits the scalability of experiments.

Second, proper inputs should feed the application that is analyzed [START_REF] Wong | IntelliDroid: A Targeted Input Generator for the Dynamic Analysis of Android Malware[END_REF]. In particular, the graphical interface should be manipulated carefully [START_REF] Abraham | GroddDroid: A gorilla for triggering malicious behaviors[END_REF][START_REF] Gianazza | PuppetDroid: A User-Centric UI Exerciser for Automatic Dynamic Analysis of Similar Android Applications[END_REF], external events such as SMS receiving or internal messages should be generated if needed [START_REF] Tam | CopperDroid: Automatic Reconstruction of Android Malware Behaviors[END_REF][START_REF] Wong | IntelliDroid: A Targeted Input Generator for the Dynamic Analysis of Android Malware[END_REF]. Fuzzing techniques [START_REF] Rasthofer | Making Malory Behave Maliciously: Targeted Fuzzing of Android Execution Environments[END_REF], symbolic or concolic executions [START_REF] Fratantonio | TriggerScope: Towards Detecting Logic Bombs in Android Applications[END_REF] are now actively investigated for reducing the number of inputs necessary to trigger the payload.

Finally, an oracle is required to evaluate the success of an experiment where a malware is executed. When using a prebuilt dataset where the malicious code is clearly identified, the oracle should monitor the successful execution of this part of the code. For a dataset that is built manually or if the malicious code is not clearly pointed out, the implementation of the oracle becomes a scientific challenge.

With these remarks in mind, we designed an execution platform, called the Kharon platform located in the High Security Laboratory of Inria Rennes. It is composed of two components: a static and dynamic analysis that are orchestrated by GroddDroid [START_REF] Tam | CopperDroid: Automatic Reconstruction of Android Malware Behaviors[END_REF][START_REF] Leslous | GPFinder: Tracking the Invisible in Android Malware[END_REF], and a kernel component, Blare [START_REF] Andriatsimandefitra | Designing information flow policies for Android's operating system[END_REF], that helps to monitor the actions of the malware 1 . We briefly discuss the performance of the platform in the next section.

E. Performances of Dynamic Analysis

Even if the performances are linked to the difficulties of the scalability of the experimental setup, some other figures from our experiments should be mentioned.

Executing unknown applications, even if in a polished environment that is as close as possible to a real smartphone, may result in an unexpected crash. When launching the application (or implemented component such as a service) without any stimulation, we observed 5% of crash for the AMD dataset [START_REF] Wei | Deep Ground Truth Analysis of Current Android Malware[END_REF] and 20% of crash for our specialized native dataset. Such results are of good quality, compared to the 76% of crashed apps reported by Yang et al. [START_REF] Yang | Malware Detection in Adversarial Settings: Exploiting Feature Evolutions and Confusions in Android Apps[END_REF]. Malware that have crashed are usually removed from experiments [START_REF] Yang | Malware Detection in Adversarial Settings: Exploiting Feature Evolutions and Confusions in Android Apps[END_REF] and never investigated, hiding potential dangerous samples.

We measured the analysis time when a sample is operated by the Kharon platform. The preparation of the smartphone requires 60s, the static analysis phase is of 7s on average and the dynamic analysis lasts 245s on average. In total, we need 312s for executing one suspicious method. Thus, for the AMD dataset where our 135 samples contain 99 suspicious locations on average, we need in the worst case 48 days of experiment when using one smartphone. Repeating experiments on Android versions from KitKat to Oreo, which covers 72% of devices according to Google 2 , would require 8 repeated experiments, requiring in total approximatively one year. Such a simple estimation illustrates the limits of dynamic analysis. In particular, locating the malicious code is of primary importance, before executing this code.

F. Conclusion

This paper has presented some of the difficulties related to experiments with Android malware. Currently, no dataset is enough mature for being used for comparison purpose and for different research objectives. Challenges remain unaddressed for building datasets with meta-data and execution environments that help to reproduce large scale experiments. Finally, processing samples with multiple smartphones, in a scalable way, remains a difficult challenge that requires rethinking the architecture of both the experiments and the internal components of Android.

Figure 1 .

 1 Figure 1. Ratio of recognized malware by at least x antivirus