domain: stat math

text:
Philippe Charpentier
email: philippe.charpentier@edf.fr

Themis Palpanas
email: themis@mi.parisdescartes.fr

Appliance Detection Using Very Low-Frequency Smart Meter Time Series

Keywords: Computing methodologies → Learning paradigms Appliance Detection, Smart Meter Data, Time Series Classification ACM Reference

In recent years, smart meters have been widely adopted by electricity suppliers to improve the management of the smart grid system. These meters usually collect energy consumption data at a very low frequency (every 30min), enabling utilities to bill customers more accurately. To provide more personalized recommendations, the next step is to detect the appliances owned by customers, which is a challenging problem, due to the very-low meter reading frequency. Even though the appliance detection problem can be cast as a time series classification problem, with many such classifiers having been proposed in the literature, no study has applied and compared them on this specific problem. This paper presents an in-depth evaluation and comparison of state-of-the-art time series classifiers applied to detecting the presence/absence of diverse appliances in very low-frequency smart meter data. We report results with five real datasets. We first study the impact of the detection quality of 13 different appliances using 30min sampled data, and we subsequently propose an analysis of the possible detection performance gain by using a higher meter reading frequency. The results indicate that the performance of current time series classifiers varies significantly. Some of them, namely deep learning-based classifiers, provide promising results in terms of accuracy (especially for certain appliances), even using 30min sampled data, and are scalable to the large smart meter time series collections of energy consumption data currently available to electricity suppliers.

INTRODUCTION

The energy sector is undergoing significant changes, primarily driven by the need for a more sustainable and secure energy supply. One way to better manage our consumption is to understand it better. In the last decade, electricity suppliers have installed millions of smart meters worldwide to improve their ability to manage the electrical grid [START_REF] Stanislav Chren | Smart grids deployments within EU projects: The role of smart meters[END_REF][START_REF] Milam | Smart meter deployment: US initiatives[END_REF]. These meters record detailed time-stamped data on electricity consumption, allowing both individual customers and businesses to better understand and rationalize their consumption [START_REF] Gouri | Smart metering and functionalities of smart meters in smart grid -a review[END_REF]. These data are also valuable for suppliers, as they can help them anticipate energy demand more accurately. Overall, the widespread adoption of smart meters plays a crucial role in transitioning toward a more sustainable and efficient energy system. We note that it has become essential for electricity suppliers to know which electrical appliances their customers own. This knowledge allows suppliers to better segment their customer base [START_REF] Allcott | Social norms and energy conservation[END_REF], and therefore to propose personalized offers and services that increase the customer satisfaction and retention. Furthermore, they can help customers rationalize their electricity consumption, therefore contributing to the energy transition. One way to gather this information is by asking customers directly through a consumption questionnaire. However, this method can be a significant investment in terms of time and resources, which customers may not accept, and is also prone to errors. Therefore, electricity suppliers need to find more efficient and non-intrusive ways of gathering this information, such as using advanced data analytics techniques to detect the appliances directly through the collected smart meters data [START_REF] Hart | Nonintrusive appliance load monitoring[END_REF].

Appliance detection has become a significant area of research, with various techniques employed to detect the presence of devices [START_REF] Liu | Non-intrusive energy estimation using random forest based multi-label classification and integer linear[END_REF][START_REF] Rossier | Near Real-Time Appliance Recognition Using Low Frequency Monitoring and Active Learning Methods[END_REF]. This problem is closely related to Non-Intrusive Load Monitoring (NILM), which aims to identify the power consumption, pattern, or on/off state activation of individual appliances using only the total consumption series [START_REF] Kaselimi | Towards Trustworthy Energy Disaggregation: A Review of Challenges, Methods, and Perspectives for Non-Intrusive Load Monitoring[END_REF]. While detecting an appliance can be seen as a step in NILM-based methods [START_REF] Aslan | An hybrid model for appliances classification based on time features[END_REF][START_REF] Kahl | Representation Learning for Appliance Recognition: A Comparison to Classical Machine Learning[END_REF][START_REF] Kahl | A Comprehensive Feature Study for Appliance Recognition on High Frequency Energy Data[END_REF][START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF][START_REF] Qu | A residual convolutional neural network with multi-block for appliance recognition in non-intrusive load identification[END_REF][START_REF] Seyed | Toward Non-Intrusive Load Monitoring via Multi-Label Classification[END_REF], and diverse approaches have been proposed in the literature [START_REF] Aslan | An hybrid model for appliances classification based on time features[END_REF][START_REF] Kahl | Representation Learning for Appliance Recognition: A Comparison to Classical Machine Learning[END_REF][START_REF] Kahl | A Comprehensive Feature Study for Appliance Recognition on High Frequency Energy Data[END_REF][START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF][START_REF] Qu | A residual convolutional neural network with multi-block for appliance recognition in non-intrusive load identification[END_REF][START_REF] Seyed | Toward Non-Intrusive Load Monitoring via Multi-Label Classification[END_REF], they differ from our objective. Indeed, these studies essentially focus on detecting when a specific appliance is "ON" rather than if a household owns a specific appliance, and the presence of a specific appliance is in several cases already known before applying these approaches. Moreover, the majority of the NILM studies rely on data sampled at ≥1Hz, and consequently use signature-based methods [START_REF] Liu | Non-intrusive energy estimation using random forest based multi-label classification and integer linear[END_REF][START_REF] Rossier | Near Real-Time Appliance Recognition Using Low Frequency Monitoring and Active Learning Methods[END_REF] that require either knowledge about how each appliance operates, or training on their individual power consumption. Nonetheless, most existing smart meter installations record consumption at a very low sampling frequency: once every 10 to 60 minutes (in some cases at an even lower frequency). This results in signals where the unique appliance pattern information has been smoothed-out, or lost. Figure 1 illustrates this loss of information. We observe that the dishwasher (shown on the left) and washing machine (shown on the right) signatures become increasingly hard to distinguish from one another as the sampling frequency drops. Therefore, it becomes infeasible to accurately detect appliances using signature-based methods for the sampling frequencies actually used in practice.

In this paper, we propose a benchmark of diverse state-of-the-art classification methods for the problem of appliance detection in very low-frequency electrical consumption time series. We conduct our experimental evaluation on five real smart meter datasets using different time series classifiers. We first focus on detecting appliances in very low-sampled smart meters data (30min level), as it is nowadays one of the standard sampling rates adopted by electricity suppliers. We then provide an in-depth analysis of the increasing detection quality using higher frequency smart meter readings: 15min, 10min, and 1 min. To our knowledge, this is the first study to perform an exhaustive comparison of 11 state-of-the-art methods on five diverse real datasets with 13 different types of appliances, for multiple sampling frequencies. The experimental evaluation demonstrates that current time series classifiers can accurately detect several appliances, even at the 30min resolution. Specifically, deep learning techniques are the most accurate and scalable when applied to large smart meter datasets. Moreover, we demonstrate that setting the smart meter reading frequency to 1min can greatly enhance appliance detection using time series classifiers.

Our contributions are summarized as follows.

• We describe a framework for comparing the performance of different time series classification methods for the appliance detection problem, and make this framework publicly available: https: //github.com/adrienpetralia/ApplianceDetectionBenchmark • We perform an extensive experimental evaluation using 5 diverse real datasets and 11 time series classifiers, including both traditional machine learning, as well as deep learning methods.

BACKGROUND AND RELATED WORK 2.1 Smart Meter Data

An electrical consumption load curve is defined as a univariate time series X = (𝒙 1 , ..., 𝒙 𝑇 ) of ordered elements 𝒙 𝑗 ∈ R 1 + following (𝑖 1 , ..., 𝑖 𝑇 ) time consumption indexes (i.e., timestamps). The sampling frequency is defined as the time difference between two records index Δ 𝑡 𝑖 𝑗 -𝑖 𝑗 -1 . Each element 𝒙 𝑗 , usually given in Watt, indicates either the actual power at time 𝑖 𝑗 or the average electric power called during the interval time Δ 𝑡 . The value can also be given in Watt-hour. In the literature, the definition of high and low-frequency smart meters data can differ [START_REF] Huber | Review on Deep Neural Networks Applied to Low-Frequency NILM[END_REF]. In this study, we refer to high-frequency data sampled at less than 1 second and lowfrequency data sampled between 1 second and 1min. Data sampled above 1min refers to very low-frequency smart meter data.

[Individual appliance load curve] By monitoring electric devices with individual meters, we can obtain the consumption load curve of each individual appliance in a household. However, instrumenting every appliance in the house is prohibitively expensive.

[Aggregate load curve] The main consumption power of a house is usually recorded by a smart meter device located on the electrical meter of the household. This aggregate signal is the addition of the power consumption of all individual appliances in the household.

Non-Intrusive Load Monitoring (NILM) and Appliance Detection

Non-Intrusive Load Monitoring (NILM) [START_REF] Hart | Nonintrusive appliance load monitoring[END_REF], also called load disaggregation, relies on identifying the individual power consumption, pattern, or on/off state activation of individual appliances using only the total aggregated load curve [START_REF] Kaselimi | Towards Trustworthy Energy Disaggregation: A Review of Challenges, Methods, and Perspectives for Non-Intrusive Load Monitoring[END_REF]. NILM was initially approached as a problem involving linear combinations, with algorithms aiming to estimate the proportion of total power consumption used by distinct active appliances at each time step [START_REF] Kaselimi | Towards Trustworthy Energy Disaggregation: A Review of Challenges, Methods, and Perspectives for Non-Intrusive Load Monitoring[END_REF]. Early research on this topic employed combinatorial optimization techniques [START_REF] Kaselimi | Towards Trustworthy Energy Disaggregation: A Review of Challenges, Methods, and Perspectives for Non-Intrusive Load Monitoring[END_REF]. Later, Hidden Markov Models became the dominant approach, and in the last few years, deep learning models have been the reference to perform disaggregation [START_REF] Huber | Review on Deep Neural Networks Applied to Low-Frequency NILM[END_REF][START_REF] Kaselimi | Towards Trustworthy Energy Disaggregation: A Review of Challenges, Methods, and Perspectives for Non-Intrusive Load Monitoring[END_REF][START_REF] Kelly | Neural NILM[END_REF][START_REF] Sykiotis | ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring[END_REF][START_REF] Yue | BERT4NILM: A Bidirectional Transformer Model for Non-Intrusive Load Monitoring[END_REF]. Furthermore, NILM approaches can be divided into supervised and unsupervised learning, depending on whether they usee labeled data for training the models. Supervised learning involves classifying detected events (appliances being switched on or off) by matching extracted features [START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Liu | Non-intrusive energy estimation using random forest based multi-label classification and integer linear[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF][START_REF] Seyed | Toward Non-Intrusive Load Monitoring via Multi-Label Classification[END_REF]. In contrast, unsupervised NILM methods detect events by analyzing feature similarities, or correlations without using labeled data [START_REF] Hart | Nonintrusive appliance load monitoring[END_REF][START_REF] Zhao | On a Training-Less Solution for Non-Intrusive Appliance Load Monitoring Using Graph Signal Processing[END_REF].

Since device recognition can be seen as a step of NILM-based methods, different approaches exist in the literature to detect appliances in load curves using high or low-frequency smart meter data [START_REF] Aslan | An hybrid model for appliances classification based on time features[END_REF][START_REF] Kahl | Representation Learning for Appliance Recognition: A Comparison to Classical Machine Learning[END_REF][START_REF] Kahl | A Comprehensive Feature Study for Appliance Recognition on High Frequency Energy Data[END_REF][START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF][START_REF] Qu | A residual convolutional neural network with multi-block for appliance recognition in non-intrusive load identification[END_REF][START_REF] Seyed | Toward Non-Intrusive Load Monitoring via Multi-Label Classification[END_REF]. However, numerous studies using pattern recognition at low frequency require knowledge about how each device operates. Few recent research studies [START_REF] Aslan | An hybrid model for appliances classification based on time features[END_REF][START_REF] Kahl | Representation Learning for Appliance Recognition: A Comparison to Classical Machine Learning[END_REF][START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF] used time series features, or deep learning representations, to detect events or appliance activation patterns. Despite the promising results demonstrated by these studies using modern machine learning approaches, we note that they are only applied to high-frequency data (i.e., data sampled at a minimum rate of 1 sample per second).

Studies on Very

Low-Frequency Data. Most NILM studies use high-frequency smart-meter data (seconds level at maximum), and only very few studies have been conducted using very-low sampling rates [START_REF] Eibl | Influence of Data Granularity on Smart Meter Privacy[END_REF][START_REF] Miyasawa | Energy disaggregation based on smart metering data via semi-binary nonnegative matrix factorization[END_REF][START_REF] Zhao | Nonintrusive load disaggregation solutions for very low-rate smart meter data[END_REF]. In [START_REF] Zhao | Nonintrusive load disaggregation solutions for very low-rate smart meter data[END_REF], the authors suggested three methods to estimate appliance consumption using hourly smart meter data. The first two methods are unsupervised and require knowledge about manufacturer appliance parameters. The third method is a supervised deep learning approach that requires disaggregate appliance load curves for training. In [START_REF] Eibl | Influence of Data Granularity on Smart Meter Privacy[END_REF], the authors proposed a data privacy-oriented study to assess the impact of Smart Meters sampling frequency on the detection of certain electrical appliances. They used an event detection approach as a feature extractor to train a classifier that identifies changes in power consumption. However, the experimental evaluation in this study was rather limited: the authors evaluated the classification performance on a single dataset, and used the same house for both training and testing. Overall, the few NILM studies that used low-frequency data focus on estimating the consumed power of each appliance, rather than detecting which appliances are present in the households.

Few papers in the literature [START_REF] Albert | Smart Meter Driven Segmentation: What Your Consumption Says About[END_REF]15] try to tackle the problem of detecting the devices owned by a household using very low-frequency sampled data. In [START_REF] Albert | Smart Meter Driven Segmentation: What Your Consumption Says About[END_REF], the authors used a Hidden Semi-Markov Model (HSMM) to extract appliance features from power consumption data. These features are then merged with external variables (such as temperature) and serve to train an AdaBoost classifier [START_REF] Robert E Schapire | Explaining adaboost[END_REF] to detect the presence of different appliances. In [15], the authors proposed a framework that uses a deep learning approach on subsequences of a long consumption load curve to detect the appliances present in the household. A majority vote gives the final device prediction, based on the individual predictions made on every examined subsequence. The study compares their method to [START_REF] Albert | Smart Meter Driven Segmentation: What Your Consumption Says About[END_REF], but not to any of the current state-of-the-art time series classifiers. In addition, only one public dataset at one sampling rate was considered.

Time Series Classification

Time series classification (TSC) [START_REF] Bagnall | The Great Time Series Classification Bake Off: An Experimental Evaluation of Recently Proposed Algorithms[END_REF][START_REF] Ismail Fawaz | Deep Learning for Time Series Classification: A Review[END_REF] is an important analysis task across several domains. Many studies have suggested different approaches to solve the TSC problem, ranging from the computation of similarity measures between time series [START_REF] Cover | Nearest neighbor pattern classification[END_REF] to the identification of discriminant patterns [START_REF] Hills | Classification of time series by shapelet transformation[END_REF]. In addition, benchmarks, such as the the UCR archive [START_REF] Hoang | The UCR Time Series Archive[END_REF], have been proposed, on which exaustive experimental studies have been conducted [START_REF] Bagnall | The Great Time Series Classification Bake Off: An Experimental Evaluation of Recently Proposed Algorithms[END_REF]. We discuss in more detail the current state-of-the-art time series classifiers in Section 3.

PROBLEM DEFINITION AND BENCHMARK 3.1 Problem Definition

In this work, we treat the appliance detection problem as a supervised binary classification problem. We aim to identify the presence/absence of a specified appliance's activation signature in a smart meter data series, independently of the number of activations of this appliance. The presence can be simply defined by the fact that the device is switched "ON" at least once. Formally, we define the problem as follows: know if appliance 𝑎 is activated at least once in X (i.e., was in an "ON" state, regardless of the time and number of activations).

Overview of Time Series Classifiers

We now provide an overview of the different approaches proposed in the literature to solve the TSC problem (refer to Figure 2). The objective is to compare the performance of these methods when applied to the appliance detection problem. Each classifier takes as input for training the univariate consumption time series (i.e., 1D signal) along with the ground-truth labels.

3.2.1

Nearest-Neighbor Classifier. 𝐾-Nearest-Neighbor (𝐾-NN) classifiers are the most simple and intuitive classifiers, based on the notion of time series similarity. Following a chosen distance measure, each new instance is classified by getting assigned the same label as the majority label of the 𝐾 closest samples in the training set (𝐾 = 1 in our experiments, i.e., we use 1-NN classifiers). The most popular distance measure is Euclidean distance, which allows comparing two instances point to point. However, this distance does not consider the possible distortions on the temporal axis. Dynamic Time Warping (DTW) [START_REF] Sakoe | Dynamic programming algorithm optimization for spoken word recognition[END_REF] is a distance measure to compute the similarity between two time series, where relevant patterns may evolve at different speeds. DTW suffers from a high computational cost, which makes it challenging to apply on large datasets.

Tree Based Classifier. Tree-based classifiers, like Random

Forest [START_REF] Breiman | Random Forests[END_REF], have exhibited promising results in classification tasks.

[Time Series Forest] TSF [START_REF] Deng | A Time Series Forest for Classification and Feature Extraction[END_REF] is a random forest-based classifier that uses as input features extracted from randomly sampled intervals of the raw data series. The algorithm first selects a number 𝑟 of intervals with a random start position and length; then, from each interval, three simple features are extracted: the mean, the standard deviation, and the slope. Finally, the 3𝑟 new features serve to train a classic random forest classifier. The number of intervals is set by default to √ 𝑇 , where 𝑇 is the length of the input time series, and the number of estimators for the decision tree is set to 200.

[Random Interval Spectral Ensemble] The RISE algorithm [START_REF] Lines | HIVE-COTE: The Hierarchical Vote Collective of Transformation-Based Ensembles for Time Series Classification[END_REF] is a random forest classifier based on spectral extraction features, rather than simple summary statistics for each interval. It computes the Fast Fourier Transform (FFT) and the Auto Correlation Function (ACF) for several randomly selected intervals. In contrast to TSF, the algorithm extracts only one interval from the raw series for each decision tree (set to 500 in our experiments), and the first tree is built using the features extracted from the entire series.

[DrCIF] The Diverse Representation Canonical Interval Forest Classifier (DrCIF) algorithm [START_REF] Middlehurst | The Interval (CIF) Classifier for Time Series Classification[END_REF] is an extension of the Canonical Interval Forest (CIF) classifier [START_REF] James Large | The Canonical Interval Forest (CIF) Classifier for Time Series Classification[END_REF], which itself uses the Canonical Time Series Characteristics (Catch22) [START_REF] Carl H Lubba | catch22: CAnonical Time-series CHaracteristics[END_REF]. Unlike the two previous tree-based methods, this algorithm is an interval-based time series classifier that looks for discriminative subseries before building the decision trees. As for TSF, the number of estimators is set to 200 in our experiments.

Dictionary

Based Classifier. Dictionary-based approaches, also called bag-of-words approaches, transform a time series into a sequence of symbols (letters usually) according to a chosen discretization technique. Using a sliding window of a specific size 𝑙, it is then possible to count the number of repeated patterns (i.e., symbolic words) to perform classification regarding the repetition frequency of similar patterns.

[BOSS] The Bag Of SFA Symbol (BOSS) [START_REF] Schäfer | The BOSS is concerned with time series classification in the presence of noise[END_REF] is a dictionary-based classifier that uses Symbolic-Fourier-Approximation (SFA) [START_REF] Schäfer | SFA: A symbolic fourier approximation and index for similarity search in high dimensional datasets[END_REF] as a discretization technique. It first extracts sub-sequences from the raw series using a predefined sliding window of length 𝑙. Then, each sub-series is discretized in a word of size 𝑤 of 𝛼 symbols using SFA and the Multiple Coefficient Binning algorithms [START_REF] Schäfer | The BOSS is concerned with time series classification in the presence of noise[END_REF] (𝑙 = 10, 𝑤 = 10, and 𝛼 = 2 in our experiments). This symbolic sentence (i.e., word arrangement) is then converted into a histogram by counting the frequency occurrence of each word. Finally, classification is performed using the histogram information.

[BOSS and cBOSS Ensembles] The BOSS ensemble [START_REF] Schäfer | The BOSS is concerned with time series classification in the presence of noise[END_REF] is a set of individual BOSS classifiers that use different discretization parameters 𝑤 and 𝑙. The parameter 𝑙 is defined as 𝑙 ∈ [10,𝑇 ] (𝑇 being the time series length), and values of 𝑤 ∈ {16, 14, 12, 10, 8}. The number of symbols, 𝛼, is set to the default value of 4. The algorithm keeps only individual BOSS classifiers that performed the best according to a validation test. The BOSS ensemble requires building and evaluating a large number of models, making it a time and memory-intensive classifier for large datasets. To address this complexity, a compact version (cBOSS) was introduced, that uses a restricted set of randomly chosen parameters for ensemble creation.

Deep Learning Based

Classifier. The interest in deep learning methods for time series classification has risen significantly in the past few years [START_REF] Ismail Fawaz | Deep Learning for Time Series Classification: A Review[END_REF][START_REF] Wang | Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline[END_REF]. These models have shown excellent performance, reaching the top of state-of-the-art.

[ConvNet] A Convolutional Neural Network (CNN) [START_REF] Keiron | An Introduction to Convolutional Neural Networks[END_REF] is a type of deep learning neural network widely used in image recognition that is specially designed to extract patterns through data with a grid-like structure, such as images, or time series. A CNN uses convolution, where a filter is applied on a sliding window over the time series. The ConvNet architecture proposed in [START_REF] Wang | Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline[END_REF] is composed of three stacked Convolutional blocks followed by global average pooling [START_REF] Lin | Network In Network[END_REF], and a Softmax activation function. Each Conv block comprises a convolutional layer followed by a batch normalization layer [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF], and a ReLU activation layer. The three block used the following 1D kernel sizes {8, 5, 3}.

[ResNet] The Residual Network (ResNet) architecture [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] was introduced to address the gradient vanishing problem encountered in large CNNs [START_REF] Simonyan | Very deep convolutional networks for largescale image recognition[END_REF]. A ResNet is formed by stacking several blocks and connecting them together using residual connections (i.e., identity mapping). For time series classification, a ResNet architecture has been proposed in [START_REF] Wang | Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline[END_REF], and has demonstrated a strong classification accuracy [START_REF] Boniol | DCAM: Dimension-Wise Class Activation Map for Multivariate Series Classification[END_REF]. It is the same architecture as the previously described ConvNet model, with adding residual connection between each Convolutional block.

[ResNet with Attention Mechanism] In [15], the authors proposed a an extension of the ResNet architecture to perform appliance detection. The model starts by extracting features using six convolution blocks with dilated convolution and residual connections, followed by two encoder/decoder modules that use a dot product attention mechanism. In this model, the dilated convolution (i.e., adding zeroes between the elements of the filter) aims to increase the receptive field of the kernels without increasing the number of parameters. After the feature extraction step, the classification step is performed using a multi-layer perceptron followed by a softmax activation function.

[InceptionTime] Inspired by inception-based networks in computer vision [START_REF] Szegedy | Going Deeper with Convolutions[END_REF], an ensemble of five neural networks using Inception modules has been proposed for time series classification [START_REF] Ismail Fawaz | InceptionTime: Finding AlexNet for time series classification[END_REF]. The model consists of five identical networks using residual connections and convolutional layers. One network uses 3 Inception modules that replace the traditional residual blocks that we can find in a ResNet architecture. Each Inception modules consist of a concatenation of convolutional layers using different size of filters. Specifically, each module results in the following layers. In the case of multivariate time series, a 1D convolutional bottleneck layer is used to reduce the number of dimensions of the time series .Then, the output is fed to three different 1D convolutional layers with different kernel sizes (10, 20, and 40) and one Max-Pooling layer with kernel size 3. The last step consists of concatenating the previous four layers along the channel dimension and applying a ReLu activation function to the output, followed by batch normalization. All the convolutional layers used in the module come with 32 filters and a stride parameter of 1.

Random Convolutional Kernel Features Classifiers.

The authors of [START_REF] Dempster | ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels[END_REF] proposed an approach based on convolution filters without learning any weights. Some variants of this model, based on the same principle, were later proposed in the literature.

[ROCKET] The RandOm Convolutional KErnel Transform (ROCKET) algorithm [START_REF] Dempster | ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels[END_REF] uses 1D convolutional kernels to extract relevant features. Instead of learning proper filter parameters using a gradient descent algorithm to detect relevant patterns, the method generates a large set of 𝐾 kernels with random length, weights, bias, dilation, and padding. After applying them, the maximum and the proportion of positive values are extracted as new features for each time series, resulting in a 2𝐾 features for each instance. Classification is then performed on these features, using a simple ridge classifier. By default, ROCKET uses 10000 random kernels.

[MiniRocket] MINImally RandOm Convolutional KErnel Transform (MiniRocket) [START_REF] Dempster | MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification[END_REF] is a version of ROCKET that reduces the random sampling space of the filter parameters, and keeps only the proportion of positive values as a new feature for each kernel. These modifications lead to a lower execution time complexity while maintaining similar performances.

[Arsenal] Arsenal [START_REF] Middlehurst | HIVE-COTE 2.0: a new meta ensemble for time series classification[END_REF] is an ensemble of multiple ROCKET classifiers that uses a restricted number of kernels compared to the original model. This method was proposed to the variance predicted by the classifier without changing the type of classifier.

Ensemble Models.

To reduce the variance in predictions, using a combination of models rather than a single one is a common technique. Ensemble models combining different approaches have been proposed to address the TSC problem. Several ensemble methods have been proposed in the literature, such as TS-CHIEF (Time Series Combination of Heterogeneous and Integrated Embedding Forest) [START_REF] Shifaz | TS-CHIEF: A Scalable and Accurate Forest Algorithm for Time Series Classification[END_REF] and HIVE-COTE (Hierarchical Vote Collective of Transformation-Based Ensembles) [START_REF] Middlehurst | HIVE-COTE 2.0: a new meta ensemble for time series classification[END_REF]. The first, is and ensemble of tree classifiers. The second is combining 4 different classifiers and use majority voting to provide the final prediction. However, these models suffer from a high execution time and cannot be applied to very long time series such as load curves.

Energy Consumption Datasets

Numerous energy consumption datasets exist in the literature [START_REF] Deepika | IEDL: Indian Energy with Low frequency for NILM[END_REF], and some of them have become references to conduct NILM studies [START_REF] Firth | REFIT Smart Home dataset[END_REF][START_REF] Kelly | The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes[END_REF][START_REF] Kolter | REDD : A Public Data Set for Energy Disaggregation Research[END_REF]. However, these datasets typically provide aggregated and appliance-level load curves for only a few houses at a highsampling frequency. Resampling them at a very low frequency leads to significant data reduction. In order to include a broader range of appliances and to align with existing literature, we include two NILM datasets in our experiments: UK-DALE [START_REF] Kelly | The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes[END_REF] and REFIT [START_REF] Firth | REFIT Smart Home dataset[END_REF]. We also include one public dataset providing 30min sampled aggregate load curves for a number of households [START_REF]CER Smart Metering Project -Electricity Customer Behaviour Trial[END_REF]. Moreover, we include two private datasets from EDF (the main french electricity supplier). In total, we consider five real diverse datasets in our experimental evaluation. These datasets are detailed below.

3.3.1 NILM Datasets. UKDALE and REFIT are two well-known high-frequency Smart Meters datasets used in NILM studies [START_REF] Sykiotis | ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring[END_REF][START_REF] Yue | BERT4NILM: A Bidirectional Transformer Model for Non-Intrusive Load Monitoring[END_REF].

[UK-DALE] The UK-DALE dataset [START_REF] Kelly | The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes[END_REF] contains data from 5 houses in the United Kingdom, and includes appliance-level load curves sampled every 6 seconds, as well as the whole-house aggregate data series sampled at 16kHz. Four houses were recorded for over a year and a half, while the 5th house was recorded for 655 days.

[REFIT] The REFIT project (Personalised Retrofit Decision Support Tools for UK Homes using Smart Home Technology) [START_REF] Firth | REFIT Smart Home dataset[END_REF] ran between 2013 and 2015. During this period, 20 houses in the United Kingdom were recorded after being monitored with smart meters and multiple sensors. This dataset provides aggregate and individual appliance load curves at 8-second sampling intervals.

CER Dataset.

The Commission for Energy Regulation of Ireland conducted a study to assess the performance of smart meters and their impact on consumer energy consumption [START_REF]CER Smart Metering Project -Electricity Customer Behaviour Trial[END_REF], recording the aggregate load curve consumption every 30min for over 5000 Irish homes and businesses. Pparticipants filled out a questionnaire on the household composition, the behavior of electricity consumption, and the type and number of appliances present in the home, or business. In this work, we use the residential sub-group of the study, i.e., 4225 households recorded from July 15, 2009, to January 1, 2011, for a total of 4225 series, of length 25728 data points each.

EDF Datasets.

To better understand its customers' base and electricity consumption behavior, Electricité De France (EDF) conducts surveys on customer samples. These customers consent to EDF to use their data and analyze their consumption behaviors, and only the aggregate power consumption of the house is recorded. Similar to the CER study, customers fill out a questionnaire with information on which appliances are present in their households, and on their consumption habits. Two EDF datasets from two different studies were used in our experiments.

[EDF Dataset 1] The first one contains 2611 load curves at 30min sampling frequency of one year of electricity recording consumption. Data were collected between September 2019 and September 2021 from 1553 different clients. The dataset consists of 2611 time series of length 17520 from 1553 different sources.

[EDF Dataset 2] The second dataset contains 5354 load curves at a 10min sampling frequency, recorded over a period of six months. Data were collected between January 2012 and January 2015 from 1260 clients. The dataset consists of 5354 time series of length 26208 from 1260 different sources.

EXPERIMENTAL SETUP

All experiments are performed on a high-performance computing cluster. The source code is in Python 3.7, and for each classifier we use the default parameters provided by the authors in the original papers. For non-deep-learning approaches, we use the sktime library [START_REF] Löning | sktime: A Interface for Machine Learning with Time Series[END_REF]. We perform each experiment on a server with 2 Intel Xeon Gold 6140 CPUs with 190 Go RAM. For deep-learning based models, we implement all the models using the 1.8.1 version of the PyTorch framework [START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF], and run experiments on a server with 2 NVidia V100 GPUs with 16Go RAM.

We consider all the classifiers presented in Section 3.2. We run each method five-time using different random train/validation/test splits, and we report the average of these runs. Note that the error bars shown in Figure 3, Figure 7, and Figure 8, correspond to the average variability of the classifiers through these five runs. Additionally, we set a 10-hour time limit per job. Only models that finished a run (training + inference) are considered. We note that the ResNet with Attention model was not evaluated using UKDALE and REFIT data due to the residual block's dilation convolution being incompatible with the small size of the time series of these datasets.

We make all code available online: https://github.com/adrienpet ralia/ApplianceDetectionBenchmark

Data Preprocessing

Since the datasets we employ in this study have been created using different sampling frequencies, we preprocess them for the experiments as explained below. The left part of Table 1 summarizes, for each dataset, the number of time series and the corresponding length, according to each sampling frequency.

NILM dataset preprocessing.

The REFIT and UKDALE datasets provide appliance level and total consumption load curves for a small number of houses: 5 and 20, respectively. Moreover, the electrical appliances in the houses are likely the same. Inspired by the data processing step in NILM studies [START_REF] Sykiotis | ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring[END_REF][START_REF] Yue | BERT4NILM: A Bidirectional Transformer Model for Non-Intrusive Load Monitoring[END_REF], we preprocess the For each experiment, we first resample the data to a specified sampling rate and fill in with linear interpolation the gaps of less than 1 hour. Then, we process the datasets by splitting each household's consumption load curve into smaller sub-sequences of one day, and by dropping those with missing values. The choice of the one day for the sub-sequence length provides an overall balance between positive (i.e., containing the device) and negative (i.e., not containing the device) samples. In contrast, slicing the entire consumption curve in weeks leads to very few negative samples for most appliance cases. This is because the appliances in these datasets are devices that are very frequently used (on average, once every two or three days). To assign the positive or negative label (i.e., appliance presence or not) to a sub-sequence, we use the corresponding disaggregated appliance load curve, allowing us to know if the appliance has been switched on at least once for a given day.

By preprocessing the UKDALE dataset, we noticed that the fourth house of the study could not be used for the experiments, since a single disaggregated load curve regrouped multiple appliances. Thus, we use only three houses for the training/validation set, whereas the one last house's sub-sequences are used for the test set. With the REFIT data, we use two randomly selected houses for the test set, while the other houses available are used for the train set.

CER and EDFs datasets preprocessing. The CER and EDFs datasets provide only the total aggregated load curve of each house.

As a consequence, it is impossible to know if an appliance is activated or not for a given day. Therefore, we cannot slice the time series into smaller subsequences as for the NILM datasets, and we provide as inputs to the classifier the full-length load curves. In addition, we process the load curves by linearly interpolating gaps of less than 1 hour and any time series with residual missing values are not retained. The appliance presence label is assigned using the provided questionnaire associated with each dataset. Finally, we do a 70%/10%/20% random split of the houses for the training, validation, and test sets, respectively.

Appliance Detection

Cases. We select different cases of device detection through all the datasets, including small and big appliances. The right part of Table 1 summarizes the selected appliance detection cases for all datasets. The REFIT and UKDALE datasets include mostly small appliances because, in these studies, only plugged devices were recorded. On the other hand, the CER and EDFs datasets provide information about larger appliances, directly connected to the electric meters, such as Water Heaters, Heaters, and Electric Vehicles.

The selected cases aim to determine if a specific device is present in a time series using binary detection. However, the "Convector/Heat Pump" case involves classifying the types of electric heaters, such as distinguishing between convectors and heat pumps.

In order to ensure that the classifiers are not biased during training, we maintain an equal balance of time series labeled with positive and negative samples. However, we note that the test set reflects the actual, imbalanced nature of the data, allowing us to evaluate the classifier's performance in a realistic scenario.

♯TS is the number of labeled time series used for each case, in which each class are balanced. IB Ratio indicates the imbalance level of the corresponding test sets (i.e., the percentage of positive instances in the number of instances).

Evaluation Measures

[Accuracy] When detecting appliance presence/absence, several classification cases may be unbalanced. Indeed, most people own a television or a washing machine but do not have an electric heating system or a swimming pool. However, using a model that only predicts the majority class may appear to perform well in these cases when using the classification accuracy (i.e., the ratio of well classified instances versus the total number of instances). Precision, Recall, and the harmonic average of both, called the F1-Score, are well-known measures, defined as follows: F1-score measures independently indicate the model's performance can be applied to one class only. In the case of a binary classification problem with data imbalance, these measures are typically applied only to the minority class. In our classification problem, the minority class varies depending on the specific device. Detecting an appliance (i.e., the positive class) could correspond either to the minority or the majority class. Thus, the F1-Score measure is not appropriate in our case. To account for this variability and provide an overall performance measure, we use the Macro F1-score to evaluate the performance of the classifiers. Formally, for 𝑁 class (in our case, 𝑁 = 2), the Macro F1-Score is defined as follows:

Macro F1-score = 1 𝑁 𝑁 ∑ 𝑖=1 F1-score 𝑖 [Time Performance]
Considering the computation time of classifiers is crucial for evaluating their effectiveness in real-world scenarios. We measure the time performance of the classifiers, considering the total time required for both training and inference.

RESULTS AND DISCUSSION

This section presents the results of our experimental evaluation. First, we normalize the different datasets to the same sampling frequency, i.e., 30min, to obtain overall results on all the cases. Then, we perform an experimental evaluation of the influence of sampling frequency on the detection quality of the classifiers. We also analyze the data size impact on the detection quality. Finally, we provide a discussion of the overall results.

Accuracy for 30min Sampling Frequency

The appliance detection results of the classifiers for the sampling frequency of 30min are summarized in Table 2. We observe that all classifiers return poor results for the UKDALE dataset. and explain these results in detail in Section 5.3.) Furthermore, we note that independently of the dataset, some appliances are easier to detect than others. The following sections provide an analysis of these results according to the type of appliances. Computer on other datasets is not as good, but is consistent with the number of time series provided. It can be explained by the fact that the pattern is hidden behind other appliance activation signatures in longer smart meters load curves, and thus, is hard for classifiers to detect.

5.1.2 Kitchen Appliances. First, detecting Kettle usage looks pretty challenging, with poor results obtained by all classifiers and a Macro F1-Score ≃ 0.45. Given that a kettle operates for relatively short periods, it is understandable that its activation may not be captured using 30min sampled data. Microwave oven and classic Oven are not well detected in the EDF datasets. However, the detection score obtained on REFIT by the best two classifiers is above 0.7, thanks to the larger amount of data available for this case in REFIT. Finally, the Cooker is well detected on the CER dataset.

Washer Appliances.

Classifiers achieve promising results detecting Dishwasher and Tumble Dryer through CER and EDF 2 datasets. The lower performance obtained with the EDF 1 datast is explained by the lower amount of labeled instances given for these cases. However, the low score results obtained on the three washer appliances for REFIT are not due to the amount of time series data. We believe that this poor detection score can be explained by the fact that these three devices are used in combination and have similar activation patterns; therefore, the classifiers cannot easily distinguish among them.

Heating Appliances.

The best detection scores are achieved for Water Heater on the EDF 1 and EDF 2 datasets. In France, water heaters refer mainly to devices that heat water from a hot tank, and usually operate during hours with high consumption power levels [START_REF] Hohne | A review of water heating technologies: An application to the South African context[END_REF]. The classifiers can effectively discern this type of pattern, even using 30min sampled data. The lower performance on the CER dataset can be attributed to the use of two types of water heaters in Ireland: instantaneous and tank-pumped. Instantaneous water heaters only operate on demand, resulting in high spikes of short duration. Using the same label for these two devices, which have different activation signatures, significantly impacts the performance of the classifiers. The results on heater detection are satisfying for the EDF datasets, and we assume that the score difference between EDF 1 and EDF 2 is mainly due to the span of the time period used for training the model. By providing a full year of electricity consumption, the model can more easily detect the heater pattern, since it trains with data during the high consumption levels of the winter season. The poor performance on the CER dataset for heater detection can be attributed to the fact that the heater label indicates the presence of a convector electric heater, which is typically used as a supplementary heat source in winter, rather than being the primary heat source for the home.

5.1.5 Other Appliances. Electric Vehicles are well detected on the EDF 1 dataset considering the restricted number of labeled instances that we have available. The lengthy recharging times of electric vehicles and the high power required, combined with the fact that recharging often occurs mainly during low-consumption night-time hours, can explain the good performance we observe.

Overall Classifier Results

Using 30min data. The overall results, shown in Table 2 and Figure 3, demonstrate that Inception-Time outperforms other classifiers when considering the average score and rank; InceptionTime is followed by ResNet, Arsenal, Con-vNet, MiniRocket and Rocket. Since the ResNet model enhanced with the attention mechanism was not evaluated in all the cases, we do not include it in the total average score shown in Figure 3. However, this classifier achieves relatively poor performance compared to the others (refer to Table 2). In light of these results, it is essential to note the difference in performance between the best and worst performing classifiers: convolutional-based classifiers, i.e., InceptionTime, ConvNet and ResNet, are the optimal choice for many detection cases. Figure 4 summarizes the average total running time (i.e., training and inference time together) for the 11 classifiers we studied. Taking into consideration the performance of the convolutional-based approaches (deep-and non deep-learning approaches), as well as their running time, we observe that this type of classifier is the most suitable for appliance detection using 30min sampled smart meter data. InceptionTime reaches a sligthly higher detection score, but at the cost of longer execution times. A balance between performance and efficiency is achieved by the ResNet and ConvNet classifiers.

Influence of Sampling Rate

In this part of the experimental evaluation, we analyze the improvement of the detection score of the different classifiers, as the smart meter sampling rate increases. We used the REFIT and EDF 2 All Classifier (Avg.)

Legend:

(h) Water Heater (EDF 2) (i) Heater (EDF 2) (k) Average All Cases datasets to perform these experiments, since these datasets provide data at a higher frequency than every 30min. Using the REFIT dataset, we performed experiments at four different sampling rates: 1min, 10min, 15min, and 30min. To obtain complementary results on bigger appliances that were not available with REFIT data, we also included appliance detection cases from the EDF 2 dataset. However, since this dataset offers data sampled at 10 min, we could only produce results for sampling rates: 10min, 15min, and 30min.

All the results are summarized in Figure 5. For clarity, we only illustrate the scores of the five best classifiers.

On average across all cases, the appliance detection accuracy decreases significantly (by almost 0.1) when the sampling rate drops from 1min to 30min. For the best classifier (InceptionTime), the average drop is 0.15.

However, it is interesting to note that not all appliances are significantly better detected using a higher sampling frequency. As expected, appliances that operate only for short periods, i.e., Microwave or Kettle, benefit the most when using higher smart meter frequencies. For example, the results in Figure 5 show that using 1min sampled data can significantly improve the Kettle detection. In this case, the best classifier, ResNet, achieves a 0.2 improvement in the detection score when the sampling rate increases from 30min to 1min. For the Microwave case, it is a 0.1 average gain score for all the classifiers using 1min sampled data.

Other appliances, such as Dishwasher, Desktop Computer, Television, Washing Machine and Water Heater, which typically operate for long periods, are better detected using higher sampling rates, as well. For example, using 1min level data, the Washing Machine is much more accurately detected than when using 30min data (refer to Figure 5(f)).

Influence of Data Size

In this last part, we analyze the impact of the number of distinct households on classifier performance. These experiments demonstrate that classifiers cannot effectively learn the patterns of an appliance using only a small number of households when the smart meter data sampling frequency is very low (this explains the poor results presented in Section 5.1 for the UK-DALE dataset). Furthermore, we demonstrate that the number of households is more important for training the machine learning models than the amount of data available for each household.

We compared the following two approaches for training: (i) randomly select a subset of the houses and use all the data from these houses to train the models, and (ii) select all houses and use a random subset of the time series from each house. We performed the experiments on the appliance detection cases using the REFIT dataset. Furthermore, in order to account for the impact of the smart meter reading on these results, we performed the experiments using 4 different sampling frequencies: 1min, 10min, 15min, and 30min. Figure 6 summarizes the results of these experiments: the graphs show the average performance of all classifiers1 for each sampling rate. The black line represents the score value averaged across all sampling rates.

We note that for every sampling rate and detection case, it is almost always preferable to use all the available households and a subset of their time series, rather than to use all time series from a subset of the households. Indeed, data from the same house is frequently characterized by the consumption patterns of the residents. Instead, using data from multiple households, enables the classifier to focus on and learn the actual activation patterns of the appliances. Interestingly, using a subset of the households, or a subset of the time series does not seem to significantly affect the detection accuracy for the Washing Machine and the Tumble Dryer. The Tumble Dryer is indeed not well detected in our experiments. However, the detection score of the Washing Machine seems to be more impacted by the sampling frequency rather than by the data size.

DISCUSSION

We now summarize the results of our evaluation. Figure 7 shows the average score for each classifier across all the experiments conducted in our study. The results show that the three deep learningbased methods are the most accurate overall. Among them, ResNet and ConvNet perform on average sligthly better than Inception-Time. However, as shown in Figure 8, the average score depends on the time series length. ResNet and ConvNet are better on average when using the short time series (REFIT and UKDALE datasets). InceptionTime is better on average when using long time series (CER and EDF datasets), because of InceptionTime's ability to capture long-lasting patterns through the use of a combination of differently-sized kernels. Nevertheless, as the confidence intervals indicate, there is no clear winner among the three deep-learning classifiers. Based on these findings, we recommend using either ResNet or ConvNet, since their time performance is one order of magnitude faster than InceptionTime (see Figure 4). Overall, the experiments show that for improving appliance detection, it is beneficial for electricity suppliers to collect data over extended periods of time, and at a finer time step than 30min. Indeed, a 15min step seems to be the minimum target in order to correctly detect a certain number of appliances. Furthermore, this study shows that further work is needed to more accurately detect appliances, even for data with 1min sampling frequency. Nevertheless, the lack of large electricity consumption public datasets that can be used to develop and train new algorithms is an important shortcoming. Having more good-quality data over long time intervals are necessary in order to allow for the development of more robust methods and further advancements in the field.

CONCLUSIONS

This paper presents a comprehensive evaluation of state-of-the-art time series classifiers applied to the appliance detection in very low-frequency smart meter data. We develop the first benchmark of time classifiers for appliance detection using five different real datasets of very electricity consumption with varying time series lengths. The results indicate that the performance of current time series classifiers varies significantly; only appliances that operate during long periods of time can be accurately detected using 30min sampled data. However, using 1min sampling data can drastically increase the detection accuracy of small appliances. deep learning-based classifiers have shown promising results in terms of particularly for tain appliances. Overall, this study provides a valuable contribution to electricity suppliers, as well as analysts and practitioners, order to help them choose the appropriate classifier for accurately detecting appliances in very smart meter data.

e-Energy ' 23 ,Figure 1 :

 231 Figure1: Comparisons of load curves containing a dishwasher and a washing machine at different sampling frequencies (1 second vs 1, 15, and 30min) One way to better manage our consumption is to understand it better. In the last decade, electricity suppliers have installed millions of smart meters worldwide to improve their ability to manage the electrical grid[START_REF] Stanislav Chren | Smart grids deployments within EU projects: The role of smart meters[END_REF][START_REF] Milam | Smart meter deployment: US initiatives[END_REF]. These meters record detailed time-stamped data on electricity consumption, allowing both individual customers and businesses to better understand and rationalize their consumption[START_REF] Gouri | Smart metering and functionalities of smart meters in smart grid -a review[END_REF]. These data are also valuable for suppliers, as they can help them anticipate energy demand more accurately. Overall, the widespread adoption of smart meters plays a crucial role in transitioning toward a more sustainable and efficient energy system.We note that it has become essential for electricity suppliers to know which electrical appliances their customers own. This knowledge allows suppliers to better segment their customer base[START_REF] Allcott | Social norms and energy conservation[END_REF], and therefore to propose personalized offers and services that increase the customer satisfaction and retention. Furthermore, they can help customers rationalize their electricity consumption, therefore contributing to the energy transition. One way to gather this information is by asking customers directly through a consumption questionnaire. However, this method can be a significant investment in terms of time and resources, which customers may not accept, and is also prone to errors. Therefore, electricity suppliers need to find more efficient and non-intrusive ways of gathering this information, such as using advanced data analytics techniques to detect the appliances directly through the collected smart meters data[START_REF] Hart | Nonintrusive appliance load monitoring[END_REF].Appliance detection has become a significant area of research, with various techniques employed to detect the presence of devices[START_REF] Liu | Non-intrusive energy estimation using random forest based multi-label classification and integer linear[END_REF][START_REF] Rossier | Near Real-Time Appliance Recognition Using Low Frequency Monitoring and Active Learning Methods[END_REF]. This problem is closely related to Non-Intrusive Load Monitoring (NILM), which aims to identify the power consumption, pattern, or on/off state activation of individual appliances using only the total consumption series[START_REF] Kaselimi | Towards Trustworthy Energy Disaggregation: A Review of Challenges, Methods, and Perspectives for Non-Intrusive Load Monitoring[END_REF]. While detecting an appliance can be seen as a step in NILM-based methods[START_REF] Aslan | An hybrid model for appliances classification based on time features[END_REF][START_REF] Kahl | Representation Learning for Appliance Recognition: A Comparison to Classical Machine Learning[END_REF][START_REF] Kahl | A Comprehensive Feature Study for Appliance Recognition on High Frequency Energy Data[END_REF][START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF][START_REF] Qu | A residual convolutional neural network with multi-block for appliance recognition in non-intrusive load identification[END_REF][START_REF] Seyed | Toward Non-Intrusive Load Monitoring via Multi-Label Classification[END_REF], and diverse approaches have been proposed in the literature[START_REF] Aslan | An hybrid model for appliances classification based on time features[END_REF][START_REF] Kahl | Representation Learning for Appliance Recognition: A Comparison to Classical Machine Learning[END_REF][START_REF] Kahl | A Comprehensive Feature Study for Appliance Recognition on High Frequency Energy Data[END_REF][START_REF] Laviron | Electricity Demand Activation Extraction: From Known to Unknown Signatures, Using Similarity Search[END_REF][START_REF] Paradiso | ANN-based appliance recognition from low-frequency energy monitoring data[END_REF][START_REF] Qu | A residual convolutional neural network with multi-block for appliance recognition in non-intrusive load identification[END_REF][START_REF] Seyed | Toward Non-Intrusive Load Monitoring via Multi-Label Classification[END_REF], they differ from our objective. Indeed,

Definition 3 . 1 (Figure 2 :

 312 Figure 2: Taxonomy of classifier considered in our benchmark (in blue: classifier used in the experimental evaluation).

F1-score = 2 .

 2 𝑃 .𝑅 𝑃 + 𝑅 , with: 𝑃 = 𝑇 𝑃 𝑇 𝑃 + 𝐹 𝑃 , 𝑅 = 𝑇 𝑃 𝑇 𝑃 + 𝐹 𝑁 with 𝑇 𝑃 = True Positive, 𝑇 𝑁 = True Negative, 𝐹 𝑃 = False Positive and 𝐹 𝑁 = False Negative. Nevertheless, precision (P), recall (R), and

  (We discuss K N N e u c li B O S S T S F R is e R o c k e t M in ir o c k e t A

Figure 3 :

 3 Figure 3: Average classifiers detection score through all the detection cases and all the datasets.

Figure 4 :

 4 Figure 4: Average running time per run (training + inference time) for all classifiers (log scale y-axis).

5. 1 . 1

 11 Tech Appliances. Desktop Computer and Television seem to be well detected in the REFIT dataset, with a Macro F1-Score above 0.7 for the best classifiers. The score obtained on Desktop

Figure 5 :

 5 Figure 5: Influence of sampling frequency on different appliance detection cases. The detection score is given for each classifier and detection case following the resampling frequency of the data. The black line shows the average score of all the classifiers.

Figure 6 :

 6 Figure 6: Results of the data imputation study using the REFIT dataset. For each appliance case, we can see on the left figure the evolution of the classification score according to the number of houses used, i.e., sources; on the right figure, we can see the evolution of the classification score according to the percentage of data used by houses.

K

  N N e u c li B O S S T S F R is e R o c k e t M in ir o c k e t A r s e n a l In c e p t io n C o n v N e t R

Figure 7 :

 7 Figure 7: Average classifier detection score through all the experiments realized in this study (including sampling frequency influence and data size influence experiments).

Figure 8 :

 8 Figure 8: Average score of the 3 best classifiers (ConvNet, ResNet and InceptionTime) according to the time series length (i.e., datasets).

Table 1 :

 1 Left side : datasets characteristics of time series, sampling frequency, time series length). Right side : selected appliance detection cases through the five datasets; for each case, the table summarizes the number of time series available (♯TS) and the imbalance degree of the test set for the case (IB Ratio). A slash indicate that no data are available for this case/dataset.

														Datasets
	Datasets Tot. TS		TS Length			Appliance case	REFIT	UKDALE		CER		EDF 1		EDF 2
			1min 10min 15min 30min			♯TS IB Ratio ♯TS IB Ratio ♯TS IB Ratio ♯TS IB Ratio ♯TS IB Ratio
	REFIT	9091	1440	144	96	48	Tech	Desktop Computer Television	5190 1134	0.56 0.92		⧸ ⧸		3286	⧸	0.47	1402	⧸	0.38	3740	⧸	0.62
								Cooker	⧸			⧸		1682		0.76		⧸			⧸
	UKDALE	4767	1440	144	96	48	Kitchen	Kettle Microwave	4790 7434	0.72 0.55	1222 1678		0.84 0.77		⧸ ⧸		324	⧸	0.91		⧸ ⧸
								Electric Oven	⧸			⧸			⧸		510		0.85	1152		0.91
	CER	4225	⧸	⧸	⧸	25728	Washer	Dishwasher Tumble Dryer Washing Machine	7798 3466 7422	0.44 0.22 0.54	2378 2830	⧸	0.32 0.38	2350 2214	⧸	0.66 0.68	224 1534	⧸	0.93 0.41	2846 3470	⧸	0.75 0.42
	EDF 1	2611	⧸	⧸	⧸	17520	Heating	Water Heater Electric Heater Convector/Heat Pump	⧸ ⧸ ⧸			⧸ ⧸ ⧸		3070 1348	⧸	0.56 0.19	1336 1624 506		0.66 0.58 0.69	548 1538	⧸	0.86 0.56
	EDF 2	1553	⧸	26208 17472	8736	Other	Electric Vehicule	⧸			⧸			⧸		140		0.3		⧸
	datasets by slicing the entire consumption curve of each household
	into smaller sub-sequences.

Table 2 :

 2 Results (average Macro F1-score for runs) for the 11 classifiers (as as the average score of all classifiers) evaluated through the appliance detection cases (best in bold and second best underlined). The "Appliance Average Score" row shows the average detection score for a specific device detection case if the appliance is available on multiple datasets. A slash indicates that the corresponding classifier failed to run on this case (time series length was not sufficiently large).

	Appliance	Dataset Arsenal Minirocket Rocket ConvNet ResNet ResNetAtt InceptionTime BOSS TSF Rise KNNeucli Avg. Score
		CER	0.618	0.617	0.606	0.602	0.614	0.530	0.608	0.516 0.580 0.586	0.491	0.579
		EDF 1	0.571	0.564	0.570	0.489	0.560	0.459	0.555	0.491 0.533 0.543	0.469	0.528
	Desktop Computer	EDF 2	0.603	0.576	0.582	0.579	0.620	0.514	0.601	0.519 0.570 0.592	0.520
		REFIT	0.697	0.683	0.674	0.715	0.740	⧸	0.623	0.542 0.525 0.600	0.548	0.635
	Appliance Average Score	0.622	0.610	0.608	0.596	0.634	⧸	0.597	0.517 0.552 0.580	0.507	0.578
	Television	REFIT	0.656	0.647	0.645	0.695	0.699	⧸	0.718	0.485 0.737 0.664	0.513	0.646
	Cooker	CER	0.680	0.673	0.676	0.661	0.689	0.541	0.710	0.526 0.566 0.584	0.440	0.613
	Kettle	REFIT UKDALE	0.368 0.540	0.376 0.502	0.381 0.522	0.522 0.428	0.477 0.432	⧸ ⧸	0.415 0.583	0.536 0.359 0.428 0.504 0.353 0.442	0.421 0.446	0.428 0.475
	Appliance Average Score	0.454	0.439	0.452	0.475	0.454	⧸	0.499	0.520 0.356 0.435	0.434	0.452
		REFIT	0.656	0.598	0.588	0.745	0.679	⧸	0.673	0.563 0.540 0.717	0.529	0.629
	Microwave	UKDALE	0.446	0.498	0.460	0.532	0.526	⧸	0.541	0.435 0.459 0.430	0.378	0.471
		EDF 1	0.480	0.471	0.475	0.534	0.510	0.409	0.474	0.454 0.400 0.429	0.457	0.463
	Appliance Average Score	0.527	0.522	0.508	0.604	0.572	⧸	0.563	0.484 0.466 0.525	0.455	0.521
	Oven	EDF 1 EDF 2	0.513 0.557	0.498 0.584	0.499 0.553	0.512 0.571	0.512 0.562	0.472 0.560	0.523 0.576	0.506 0.429 0.497 0.495 0.459 0.491	0.437 0.397	0.491 0.528
	Appliance Average Score	0.535	0.541	0.526	0.542	0.537	0.516	0.550	0.500 0.444 0.494	0.417	0.509
		REFIT	0.650	0.599	0.619	0.580	0.605	⧸	0.590	0.557 0.519 0.584	0.515	0.582
		UKDALE	0.458	0.465	0.465	0.419	0.380	⧸	0.384	0.399 0.429 0.554	0.525	0.448
	Dishwasher	CER	0.699	0.720	0.700	0.730	0.728	0.594	0.737	0.586 0.609 0.648	0.488	0.658
		EDF 1	0.454	0.441	0.450	0.528	0.522	0.383	0.535	0.430 0.418 0.421	0.211	0.436
		EDF 2	0.753	0.760	0.741	0.799	0.801	0.585	0.835	0.596 0.603 0.600	0.512	0.690
	Appliance Average Score	0.603	0.597	0.595	0.611	0.607	⧸	0.616	0.514 0.516 0.561	0.450	0.563
		REFIT	0.493	0.503	0.502	0.468	0.448	⧸	0.441	0.506 0.416 0.434	0.461	0.467
	Tumble Dryer	CER EDF 1	0.634 0.619	0.641 0.578	0.628 0.607	0.606 0.624	0.612 0.607	0.550 0.475	0.623 0.636	0.549 0.578 0.602 0.550 0.537 0.563	0.474 0.487	0.591 0.571
		EDF 2	0.733	0.714	0.714	0.757	0.769	0.475	0.769	0.560 0.593 0.681	0.493	0.660
	Appliance Average Score	0.620	0.609	0.613	0.614	0.609	⧸	0.617	0.541 0.531 0.570	0.479	0.572
	Washing Machine	REFIT UKDALE	0.605 0.475	0.572 0.505	0.592 0.478	0.581 0.535	0.586 0.530	⧸ ⧸	0.614 0.454	0.520 0.562 0.557 0.408 0.581 0.549	0.529 0.509	0.572 0.502
	Appliance Average Score	0.540	0.538	0.535	0.558	0.558	⧸	0.534	0.464 0.572 0.553	0.519	0.537
		CER	0.625	0.613	0.613	0.610	0.612	0.465	0.637	0.527 0.596 0.584	0.462	0.577
	Water Heater	EDF 1 EDF 2	0.835 0.733	0.821 0.685	0.827 0.724	0.814 0.731	0.828 0.685	0.768 0.591	0.841 0.759	0.670 0.713 0.805 0.658 0.580 0.666	0.591 0.617	0.774 0.675
	Appliance Average Score	0.731	0.706	0.721	0.718	0.708	0.608	0.746	0.618 0.630 0.685	0.557	0.675
		CER	0.522	0.532	0.514	0.533	0.508	0.477	0.565	0.459 0.492 0.527	0.397	0.502
	Heater	EDF 1	0.784	0.783	0.789	0.777	0.778	0.713	0.800	0.643 0.758 0.777	0.638	0.749
		EDF 2	0.591	0.566	0.578	0.626	0.637	0.527	0.648	0.497 0.591 0.605	0.451	0.574
	Appliance Average Score	0.603	0.597	0.595	0.659	0.607	0.572	0.616	0.514 0.516 0.561	0.450	0.609
	Type of Heater EDF 1	0.632	0.622	0.631	0.597	0.638	0.534	0.651	0.539 0.556 0.625	0.467	0.590
	Electric Vehicle EDF 1	0.689	0.730	0.670	0.681	0.699	0.553	0.720	0.541 0.456 0.725	0.556	0.638
	Classifiers Average Score	0.601	0.593	0.592	0.609	0.610	⧸	0.617	0.521 0.531 0.574	0.474	⧸
	Classifiers Average Rank	3.773	4.697	4.758	4.303	3.697	⧸	2.864	7.939 7.924 6.197	8.848	⧸

We average the performance of all classifiers listed in Table

2, except for ResNetAtt, which could not be used with the small length of the REFIT time series.