Roland Petcu
email: roland.petcu@gmail.com

Isabelle Bourdon
email: isabelle.bourdon.pro@gmail.com

Nicolas Giraudeau
email: nicolas.giraudeau@umontpellier.fr

TELEMEDICINE IN DENTISTRY, LESSONS TO BE LEARNED: A CASE STUDY Completed Research Paper

Keywords: telemedicine, adoption barriers, pilot project, healthcare information systems

Most of the telemedicine projects do not pass the pilot stage. A comprehensive evaluation of telemedicine pilot projects is needed in order to improve their success rate and to create a knowledge base for future projects. In this paper we address this issue through the study case on a pilot teledentistry project with 5 different implementation sites.

A literature review of teledentistry and telemedicine in general, points towards a rather one sided approach when evaluating these projects. The organizational aspect does not seem to be a priority in these evaluations while we consider its crucial impact on project's continuity and adoption. To support this assumption we use a healthcare information systems implementation perspective which allowed us to assess the adoption barriers for the teledentistry pilot. We make assumptions on how it could be used for telemedicine pilots and for telemedicine projects in general.

We have collected data using qualitative methods such as semi-structured interviews and nonparticipant observation of teledentistry activities and meetings related to the project. After drawing the dental teleconsultation process and observing that most of it turns around electronic medical records (EMR) we focused to identify all potential adoption barriers and linking them with the list of EMR adoption barriers by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF].

We have identified 23 adoption barriers for teledentristry: 4 in 5/5 implementation sites, 1 in 4/5 sites, 6 in 3/5 sites, 2 in 2/5 sites and 10 in 1/5 sites. Out of the 23 teledentistry adoption barriers, 17 have been linked with EMR adoption barriers. By adding the remaining 6 items to the EMR adoption barriers list and adapting the items to the telemedicine context we propose a telemedicine pilot project adoption barriers list.

Introduction

Telemedicine has been generally considered from its early beginnings as a solution for improving access to care, reducing healthcare costs and improving collaboration and coordination of care among healthcare professionals. This promise has yet to be fulfilled as most of the research on telemedicine projects is on pilot studies and very few reach to the routine use stage [START_REF] Jackson | Trends in telemedicine assessment indicate neglect of key criteria for predicting success[END_REF].

Teledentistry can be considered a new field of telemededicine. Teledentistry is developed to improve care quality, health outcomes, and reduced per-capita costs [START_REF] Daniel | Teledentistry: A Key Component in Access to Care[END_REF]) [START_REF] Golder | Practicing dentistry in the age of telemedicine[END_REF], especially in rural areas [START_REF] Bhambal | Teledentistry: potentials unexplored![END_REF] and for those who do not have easy access to dental care, such as elderly or prisoners [START_REF] Sanchez Dils | Teledentistry in the United States: a new horizon of dental care[END_REF]. Dentistry is one of the medical fields which registered a late start in telemedicine activities, with only a couple of pilot projects worldwide, which are fueled by the latest technological advances, these projects are considered as teledentistry. [START_REF] Mihailovic | Qualitative data analysis: An expanded sourcebook[END_REF] mentions that despite advances in the field of telemedicine, the field of teledentistry remains somewhat neglected, and that there is a need for more research to better understand its potential benefits and some of the barriers to its further development.

Nevertheless, the context under which telemedicine projects are implemented is decisive and previous research highlighted the slim chances of the telemedicine projects to pass the pilot experimental stage [START_REF] Jackson | Trends in telemedicine assessment indicate neglect of key criteria for predicting success[END_REF]. We argue that a thorough evaluation of these pilot projects is needed for teledentistry but also for telemedicine in general. Moreover, pilots have to be designed as scale-up implementation and adoption projects. Thus, the evaluation of pilots has to support decision-making, organisational and technical changes in order to set implementation and to increase the chances of success of large scale implementations. The focus of the evaluation should not be only on the medical and clinical aspects or a one sided point of view. We argue that the evaluation of telemedicine projects has to take into consideration key issues related to coordination, specificity of professional organization and project management. We support our assumptions by evidence driven from a case study focusing on a teledentistry pilot in France, which we will name "Dental project". We will describe key issues faced by the project and we will highlight the case with a theoretical framework on IT implementation in healthcare. We will conclude by outlining some of the future research directions in the field of telemedicine project implementation and evaluation.

2

Literature review

Before proceeding with the description of the applied methodology, we consider it important to present some of the literature considerations which led us to our approach. First used a wider angle namely, what has been done in similar telemedicine projects in general and then we focused specifically on teledentistry projects' evaluation.

Evaluation in telemedicine

Like in the teledentistry case, approaches in telemedicine assessment have considered a single evaluation perspective [START_REF] Ekeland | Effectiveness of telemedicine: A systematic review of reviews[END_REF], even though telemedicine often requires the creation of healthcare networks which involve governments, healthcare providers, general practitioners and patients in a multidisciplinary and cooperative context, implying different points of view [START_REF] Masella | Assessment models for telemedicine services in national health systems[END_REF]. The use of multiple assessment dimensions would imply the use of multiple collection and analysis methodologies which will identify the most useful and cost-effective criteria to design and assess a telemedicine application [START_REF] Masella | Assessment models for telemedicine services in national health systems[END_REF].

The local context should be included in the telemedicine evaluation as it could provide important insight and also improve generalizability of previous evidence to the local setting [START_REF] Mitchell | Integrating local data into hospital-based healthcare technology assessment: Two case studies[END_REF]; [START_REF] Romanow | Riding the Wave: Past Trends and Future Directions for Health IT Research[END_REF].

The telemedicine practice seems to be more influenced by the organizational aspects then the technology itself [START_REF] Sorensen | A qualittive description of telemedicine for acute stroke care in Norway: technology is not the issue[END_REF] which also points towards the needed multidimensional approach when assessing telemedicine projects. MAST [START_REF] Kidholm | A MODEL FOR ASSESSMENT OF TELEMEDICINE APPLICA-TIONS: MAST[END_REF] a three steps model for deciding whether or not to implement telemedicine services in healthcare systems and help decision makers in choosing the most efficient technologies to be used in the most cost-effective way could also provide some insight into how to properly assess pilot telemedicine projects [START_REF] Ekeland | ASSESSMENT OF MAST IN EUROPEAN PATIENT-CENTERED TELEMEDICINE PILOTS[END_REF]. However, although the model is comprehensive, some of the evaluated domains cannot/are difficult to assess: maturity of the application, legislation and reimbursement policies. Scientific standards and guidelines as basis of assessment seem to be rather difficult for some specific areas because such knowledge is not available. Moreover, the MAST model applied in pilot projects evaluation revealed some areas that are not covered such as: process studies, technological usability, responsible innovation, health literacy, behaviour change, caregiver perspectives and motivational issues of professionals [START_REF] Ekeland | ASSESSMENT OF MAST IN EUROPEAN PATIENT-CENTERED TELEMEDICINE PILOTS[END_REF].The required multidimensional approach into identifying the potential barriers to telemedicine/teledentistry adoption represents the starting point for choosing our methodology in the specific context of our study case. Nevertheless, these evaluations do not distinguish pilot and scaling-up projects while pilots are actually designed as a first step for making decisions related to scaling-up. Moreover, pilots for new health services make sense only if there is an "after plan", if the scaling up of the new service is considered in the design phase of the pilot [START_REF] Kohl | Scaling Up-A Conceptual and Operational Framework[END_REF]. The scaling up thinking provides important and concrete assessment areas for the pilot itself upon which the envisioned deployment can be revised and improved [START_REF] Paina | Understanding pathways for scaling up health services through the lens of complex adaptive systems[END_REF].

Evaluation in teledentistry

Teledentistry has highly diverse applications in both context and methods used ranging from teleeducation, tele-diagnostic, teleconsultation or a combination, to tele-monitoring, tele-support, teletreatment [START_REF] Marino | Teledentistry: a systematic review of the literature[END_REF].When evaluating teledentistry projects the literature is mostly medical, namely published in dental and telemedicine journals [START_REF] Marino | Teledentistry: a systematic review of the literature[END_REF], so it is with no surprise that the focus of these evaluations is oriented towards: clinical outcomes, efficacy, patient satisfaction, therapist satisfaction, healthcare utilization and costs. Although there is heterogeneity between studies in terms of design, settings and outcomes measured, there is a trend for supporting the efficacy and effectiveness of teledentistry [START_REF] Daniel | Teledentistry: A Systematic Review of Clinical Outcomes, Utilization and Costs[END_REF] which can also be a measure of the relatively new area of teledentistry [START_REF] Khan | Teledentistry in Practice: Literature Review[END_REF]. Very few research papers focus on identifying adoption barriers beyond the above mentioned criterion, as teledentistry implementation requires a full comprehension and consideration of the healthcare environment and also a commitment to completely integrate teledentistry within that environment [START_REF] Patel | Factors influencing the adoption and implementation of teledentistry in the UK, with a focus on orthodontics[END_REF]. We note that Rogers' diffusion of innovation theory is one of the few multidimensional models used to explain the challenges faced by teledentistry projects. The approach is valid, but not complete because it is difficult to measure what exactly causes the adoption of an innovation [START_REF] Damanpour | Organizational Complexity and Innovation: Developing and Testing Multiple Contingency Models[END_REF] and it can never account for all variables, especially in healthcare [START_REF] Plsek | The challenge of complexity in health care[END_REF]. So we needed to go further and since we have exhausted the existing literature on teledentistry, we have considered next the more general field of telemedicine, but on similar projects.

Theoretical background

The literature review has proven to offer a rather one sided or incomplete approach when evaluating pilot telemedicine projects so we wondered how are approached pilot projects in general, in the healthcare system. Since telemedicine is considered as a new healthcare service we focused our search on this area and we found impressive similarities. In establishing new services, pilots develop new roles, new patient pathways and new ways of working [START_REF] Bennett | A toolkit for incorporating genetics into mainstream medical services: Learning from service development pilots in England[END_REF] which are also present in telemedicine projects. The identified barriers to success of services point towards the importance of addressing organizational issues [START_REF] Bennett | A toolkit for incorporating genetics into mainstream medical services: Learning from service development pilots in England[END_REF].

Having as basis the definition of telemedicine, which implies the use of information and communication technologies, we focused next on healthcare information systems implementations. We assume that the evaluation of Electronic Medical Records (EMR) implementation can be used for evaluating the teledentistry pilot project and telemedicine projects in general because they are both related to technology-driven innovation in healthcare.

We considered the categorization adapted by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF] from [START_REF] Boonstra | Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions[END_REF] for EMR organizational adoption barriers. The initial list of adoption barriers is divided into 2 categories according to the phase in which they occur: initiation vs. implementation and into 7 categories according to their type: financial, technical, time, psychological, social, legal and organizational. The initiation and implementation phases used by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF], are those proposed by [START_REF] Rogers | The diffusion of innovations[END_REF] in the model of diffusion of innovation. We consider that in a pilot project both phases are present and moreover having in mind the scaling up of the project, both are equally important.

Research context and methodology

The specific context of the pilot Dental project

In France, the deployment of telemedicine activities falls under the legal framework of decree n° 2010-1229 from 19th of October 2010 which defines as telemedicine, "all medical acts realized remotely and by means of information and communication technologies" [START_REF] Simon | Le décret français de télémédecine: une garantie pour les médecins[END_REF]. Article R. 6316-1 also names the type of telemedicine acts: teleconsultation (TC), a doctor realizes a patient consultation remotely; tele-expertise, a doctor solicits advice from one or multiple doctors; telemonitoring which allows the doctor to remotely monitor the patient's status and take decisions regarding the patient's care; teleassistance which allows a doctor to remotely assist another doctor during a medical procedure.

According to the French legislation, teledentistry falls into the TC medical act. "Teledentistry is a combination of telecommunications and dentistry involving the exchange of clinical information and images over remote distances for dental consultation and treatment planning" [START_REF] Jampani | Applications of teledentistry: A literature review and update[END_REF].

In the Dental project, the participating establishments have been financed by the Regional Health Agency (RHA) after these have manifested their intention to participate in the pilot, following the agency's call for volunteer establishments. The financing included: the time the nurse spent on the experimentation, an amount/TC and the required hardware and software. The company Zeta was in charge of developing the Dental TC software for which hospital Epsilon has received specific funding. The oral TC experimentation was planned to be deployed in 16 establishments which can be categorized into 3 groups according to the type of subjects (elderly, mental and/or physical disabilities, inmates).

Establishments for the dependent elderly ( 12):  8 under the administration and supervision of hospital Alpha (1-8)  4 under the administration and supervision of hospital Beta (1-4) Specialized establishments for adult patients with severe mental and/or physical handicap (3):

 2 under the administration and supervision of association Gamma (1-2)  1 under the administration and supervision of association Delta (1) A penitentiary's medical service under the administration of hospital Epsilon (1). Hospital Alpha participated in the project following the hospital director's prior relationship with the Dental project manager and having a real wakeup call on the need to improve the oral health of the dependent elderly. Hospital Beta manifested its intention to take part in the pilot with no prior contact with the project manager. For both hospitals, Alpha & Beta, the oral TC was expected to improve the access to a dental consultation and as a consequence to improve the oral health of their patients which needed improvement according to a study published by the French Health Authority in 2010. Associations Gamma and Delta manifested their intention to take part in the pilot with no prior contact with the project manager and their participation in the project is related to the difficult access to a dental consultation for patients with severe mental and or physical disabilities. The hospital department of the prison is part of hospital Epsilon and most of the important specialties are represented, including dentistry. The dentistry service is provided by 2 dentists working in shifts, on a single dental unit. All inmates should pass a full medical exam when arriving in the institution. This medical exam has to include a dental consultation. Since for the dental treatments alone there is a delay of 2 months according to the auxiliary nurse, the dental consultations for the arriving inmates can't be realized under these conditions. The TC would provide a potential solution to this situation.

Methodology

We use exploratory case study methodology to investigate the teledentistry phenomena. Case research is known to be useful when the phenomenon is broad and complex, and when it couldn't be studied outside the context in which it occurs [START_REF] Benbasat | The case research strategy in studies of information systems[END_REF]. We have combined several qualitative data collection methods such as interviews, documentations and non participant observations to meetings and day to day activities. The description of the case study is mainly based on information collected through semi structured interviews with nurses, managers of health facilities, dentists and project manager from April 2014 until March 2016. Interview guides have been developed for each the previously mentioned persons. All interviews have been tape-recorded and then transcribed. The number of interviews followed the notion of saturation proposed by Glaser and Strauss [START_REF] Glaser | The Discovery of Grouded Theory[END_REF]. A total of 18 interviews were conducted over a period of 20 months in the 5 implementation sites. The average length of each interview is around 25 minutes. The interviews have been conducted in the early stage of the implementation, during the implementation phase for some and after the end of the pilot project. Non participant observations to project related meetings and day to day nurses' activities complete the qualitative study. For hospital Alpha sites, 3 non-participant observations have been made during days of TC activities in 3 of the 8 establishments, during 2 meetings with the project manager and 1 meeting with the RHA. For Beta1 site non-participant observations have been done during 1 training session for the second nurse, 1 meeting with the project manager and during 1 meeting with the RHA. For Gamma1 and Delta sites, non-participant observation has been done during 1 meeting with the RHA. For Epsilon site, non-participant observation has been done during 1 training session with the project manager.

To code the collected data, we used Miles and Huberman (1994) methods. We developed a database for each site containing interviews transcripts, notes, collected documents, sample description, chronological data, coding scheme and coded data. The coding scheme is based on the 7 categories of adoption barriers identified by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF] financial, technical, time, psychological, social, legal, organizational but not excluding other categories we might encounter and having in mind which are generally valid for telemedicine pilot projects and which are specific for the teledentistry pilot.

The teledentistry system

The Dental project is supported by the RHA and the public hospital. Following the agreement signature in December 2013, the planned launch was March 2014 targeting a total of 1500 TCs involving 800 subjects (100 prisoners, 100 frail people, and 600 elderly people) in 16 establishments: twelve long-term care facilities, three specialized facilities for frail people and a prison's medical department.

Our exploratory study was conducted on 12 of these sites. We will describe the different resources needed for the system to work and the actual oral TC process which led us towards the similarities between teledentistry and EMR.

The system's objective and involved personnel

The oral TC process for the Dental project is a store-and-forward; asynchronous type of telemedicine and it is composed of three steps: data acquisition/collection, data transfer and storage, data analysis. For the experimentation protocol, one step is considered as prerequisite since the patient, the family or legal guardian have to sign a consent form before undergoing the TC procedure. The consent form is in paper/physical format.

The teledentistry system, as envisioned by the project manager includes the following resources:

Intraoral camera: The Soprocare camera was selected for the pilot due to its fluorescence capabilities which allows tooth decay, dental plaque and gingival inflammation detection without using any dye solutions. The camera is connected to a laptop via the USB port.

Dental TC software: developed by company Zeta according to the requirements envisioned by the project manager. The software is client-server for the data and image acquisition part and web based for the data analysis/ dental diagnostic part. The server has been hosted on the hospital Epsilon's server in order to comply with French regulations for healthcare data hosting. The client software "Dental visit 1.0" is installed on a laptop along with the proper Soprocare camera driver and the Cisco AnyConnect Secure Mobility Client as VPN software. All client-server communications and data transfers are made through an encrypted and secured VPN connection. The client software can run both in online and offline scenarios, an internet connection being mandatory only for the client-server synchronization.

Actionable foot pedal: connected to the laptop via the USB port allows the nurse to start and stop recording with the intraoral camera without touching the laptop's keyboard.

Nurse or auxiliary nurse: the person administering the experimentation protocol has been trained for using the system and she/he is also responsible for collecting the patient's or family's consent form before any procedure related to the project can be accomplished. The training consists of 2-3 hours of theoretical and practical information about dental anatomy, Dental TC software presentation and a demonstration of how the teledentistry system works. The training also includes 1 (one) day of onsite assistance, when the project manager observes and assists the nurse during the protocol administration on actual patients. It is important to mention the fact that the nurses or auxiliary nurses do not receive this type of training as part of their education in nursing school and they are not used to performing this type of task.

Dentist: the dentists analyzing the images and putting the diagnostics have been trained on reading, analyzing and interpreting the images provided by the Soprocare camera. They were dentistry students in their final year of study and they were supervised by a hospital dentist, who is also the project manager for the Dental project.

The systems allows the nurse to create a patient file, to record images/video of the patients mouth and teeth and send them to the server in order to be analyzed by the dentist, who in return provides a diagnosis and a urgency level accompanied by a proposed treatment plan. The simplified oral TC process as observed in the field is presented in Figure 1.

6

Results and discussions

By analyzing and corroborating information extracted from the semi-structured interviews, the non participant observations and the Dental project's internal and external documents, we have created Table 1 which highlights the project's planned and achieved objectives and a timeline for the Dental project deployment for each site and sub sites where it was relevant. The first result that sets apart the Alpha site from any other site is the number of TCs, which is almost 14 times higher than in the Beta site, compared to the expected 4 times. Adding to this, the necessity to train a second nurse and no TCs realized in Beta2, Beta3 and Beta4 point towards a difficult implementation. Some of the reasons for this situation are mentioned in Table 2 in the "Beta" column.

Figure 1. Oral TC process

The second result to be noted is the difference between the planned timeline for activities and when they were actually realized, namely 1, 2, 8, 9 and 13 months late for the trainings; and 1, 5, 9, 10 and 13 months late for the first TC. The planned timeframe for completing the proposed number of TCs has been exceeded by 11 months and this because the RHA considered the pilot as finalized. None of the establishments reached the preset objective. The third important result is that for planning the project, a "one size fits all mentality" has been used even though the differences between site Alpha/Beta and Gamma/Delta and Epsilon were known to the project manager and the RHA as these are clearly defined in the Dental project's contract. This is highlighted by the financed nurse time and by the expected 80% general inclusion rate based on the number of beds. Taking into consideration the evaluation criteria considered by the RHA and the established timeline the pilot project should be considered a failure. The overly optimistic expectations, the short numbered and rather weak evaluation criteria for the pilot project could also point towards the lack of up scaling thinking during the design phase of the pilot, which according to [START_REF] Kohl | Scaling Up-A Conceptual and Operational Framework[END_REF] is close to preparing to fail. On the other hand, the fact that the quantitative evaluation does not provide as rich and accurate evaluation of what really happened during the pilot highlights the interest of using qualitative data for evaluating telemedicine pilot projects.

The fourth observation we could make is that the Dental TC software was designed and created according to the project manager's vision and executed by company Zeta. Although the nurses, the final users of the software could have been included in a requirements analysis starting with the design phase or for improvements proposals, this has not been the case.

When listing the adoption barriers for the dental TC system we have eliminated the sites: Beta2, Beta3, Beta4 and Gamma2 because they did not realize any TCs and the information we had so far was limited, namely that they were supposed to start after Beta1 and Gamma1 had finished the first visits for their patients. The list of barriers coded from our different sources is presented in Table 2, grouped under the same 7 categories used by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF]. Choosing to compare the dental TC system with an electronic medical records system might seem a stretch at first. Except for the specific intra-oral camera protocol for recording the videos of the patient's mouth, what is the dental project's system if not an electronic medical records system for the nurse? The nurse creates a patient file; inputs patient information; conducts and creates visits, shares information with the dentist and patients. Furthermore in all of the sites it comes as an addition to already existing EMR systems. Putting side by side the EMR adoption barriers and the dental TC barriers we can observe the fact that barriers from both initiation and implementation phases are present in the pilot dental TC project. Ta-ble3 represents the result of combining our Table 2 with the list of adoption barriers obtained by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF]. The result of combining the two tables represents our proposition for potential adoption barriers for telemedicine projects from a health information system implementation perspective. The initiation phase and implementation phase columns represent the list of adoption barriers adapted by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF] which we translated for the telemedicine context. The items coded during the study can be found in the pilot phase column of Table 3.

Table 3. Obtained list of telemedicine adoption barriers 7

Conclusions and limitations

We argued that telemedicine projects should be considered and evaluated as health information systems implementations and not only as new medical devices projects which was also stated by [START_REF] Hebert | Telehealth success: evaluation framework development[END_REF]and [START_REF] Klecun-Dabrowska | Evaluation and telehealth-an interpretative study[END_REF]. An adapted list of adoption barriers for information systems in healthcare by [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF] can also be applied to telemedicine projects. The added subcategories, in bold in Table 3, required to complete the picture of potential adoption barriers for pilot telemedicine projects can be placed into two categories: general or specific. General means that the item is valid for all telemedicine projects and specific means that it only applies to some specific projects similar to the Dental project.

For telemedicine pilot projects, a specific case by case implementation with a thorough requirements analysis before implementation, followed by a solid project management and a consistent feedback

Categories Telemedicine adoption barriers Pilot phase

Initiation phase √ Lack of leadership √ √ Adapted from [START_REF] Paré | Barriers to organizational adoption of EMR systems in family physician practices: A mixed-methods study in Canada[END_REF] from the stakeholders could anticipate and/or avoid most of the technical, organizational and time related adoption barriers. Telemedicine projects fall under the definition of "technochange" [START_REF] Markus | Technochange management: using IT to drive organizational change[END_REF] where both technology and change management are considered equally important for a successful project. The specific context of our study case could represent a bias, for example in our case the financial category is not so much represented because the establishments were financed by the RHA, which covered the direct costs of the pilot. However, the financial category becomes a valid point if there is no financing for the pilot and the establishments have to support the project from their own resources. The information gathered so far at the organizational level provides a rather one sided story which with extended interviews with other actors involved in the promotion and/or supervision of the project will provide a triangulation of the information and provide a more accurate image of the implementation scenarios encountered in the field. The dentist's side for the evaluation of the dental TC system has not been developed in our study case, since the Dental project's project manager is until today the only dentist to have used the system for establishing the diagnostics.

Table 1 :

 1 Planned and achieved project objectives and milestones

	Pilot Sites	Alpha		Beta		Gamma	Delta	Epsilon
	Establishment	1 -8	1		2-4	1	2	1	1
	Number of beds	506	140		216	40	55	45
	No. subjects targeted	400	100		100	30	30	40	100
	Number of visits	2	2		2	2	2	2	1
	Expected TC	800	200		200	60	60	80	100
	Realized TC	507	None	37	0	29	0	20	55
	Average time/TC	20-30'	45-60'		60'		60'	-
	Nurses planned	1	1		1	1	1	1	1
	Additional nurses			1
	Nurse time financed	10500		7000			3500		3500
	Planned trainings				February -March 2014
	Realized training	Apr 2014	May 2014	Nov 2014	-	May 2014	May 2014	Dec 2014	Apr 2015
	Timeline for TC				Mar -Oct 2014
	First TC	Apr 2014	Jan 2015	-	Aug 2014		Dec 2014	Apr 2015
	End of pilot(RHA)					Sep 2015
	Shared personnel	X			X
	Shared Equipment			X			X

Table 2 .

 2 List of coded adoption barriers for the teledentistry system by pilot site


Andersen Ang

Jeremy E Cohen

Nicolas Gillis

Accelerating Approximate Nonnegative Canonical Polyadic Decomposition using Extrapolation

On considère dans cet article le problème de la Décomposition Positive approchée Canonique Polyadique (aNCPD) d'un tenseur positif d'ordre trois. Ce problème consiste à minimiser T -(U ⊗V ⊗W )Ir 2

F part rapport à des matrices positives U , V et W . Nous proposons un algorithme de descente par blocs de coordonnées utilisant une extrapolation à la Nesterov, de la forme

). Les résultats expérimentaux prouvent l'efficacité de l'algorithme proposé par rapport aux autres approches de descente par blocs pour des données simulées mal conditionnées.

Introduction

Given a nonnegative three dimensional tensor T ∈ R I×J×K + , assume one wants to compute the approximation of T by a nonnegative tensor G = (U ⊗ V ⊗ W )I r of given small nonnegative rank r, where G ijk = r q=1 U iq V jq W kq . Using the Frobenius norm as an error metric, the problem becomes :

min U ≥0,V ≥0,W ≥0 f (U, V, W ) = 1 2 T -(U ⊗ V ⊗ W )I r 2 F , (1)
where the inequalities are meant element-wise.

Problem [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF] is often referred to as the approximate Nonnegative Canonical Polyadic Decomposition (aNCPD) of T , as long as r is smaller than the rank of T . This problem is the extension of the low-rank nonnegative approximation problem for matrices, namely nonnegative matrix factorization (NMF). Solving aNCPD is an important issue for mining information out of tensors collected in a wide variety of applications ; see [START_REF] Kolda | Tensor decompositions and applications[END_REF] and references therein. Note that (1) is well-posed. In fact, a best low-rank approximation for tensors may not exist, but because the nonnegativity constraints are lower bounds for the parameters, this prevents degeneracy and the best nonnegative low-rank approximation always exists [START_REF] Lim | Nonnegative approximations of nonnegative tensors[END_REF][START_REF] Qi | Uniqueness of nonnegative tensor approximations[END_REF].

Contribution In this paper, we are interested in algorithms featuring both low-cost iterations and fast convergence speed. Such fast and cheap algorithms are the backbone of modern machine learning, where both the data and the number of parameters to estimate can be very large. To this end, we first study some existing Block-Coordinate Descent (BCD) algorithms, and present an extrapolated BCD algorithm based on Hierarchical Alternating Least Squares (HALS). To be specific, we adopt the numerical scheme of [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF], which has found to be able to significantly improve the convergence speed on the problem of NMF. Experimental results showcase that our approach is faster at producing accurate results than state-ofthe-art BCD algorithms in all tested scenarios.

Existing block-coordinate descent algorithms for aNCPD

Let us first mention some existing algorithms to compute aNCPD. Problem (1) is non-convex and admits no closed-form solution. A baseline method called ANLS solves aNCPD by alternating minimization on blocs {U, V, W } as detailled below. However since computing the gradient of f is simple, and projecting on the nonnegative orthant only requires clipping, some algorithms have been proposed based on projected first-order methods [START_REF] Huang | A flexible and efficient algorithmic framework for constrained matrix and tensor factorization[END_REF][START_REF] Xu | A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion[END_REF]. Compression using Tucker models has also been explored [START_REF] Cohen | Fast decomposition of large nonnegative tensors[END_REF].

As mentioned in the introduction, in what follows, we study algorithms with i) a low cost per iteration, and ii) a high convergence speed. First-order methods are typically slow to converge despite having low complexity. However, it is possible to accelerate significantly first-order methods using extrapolation, which is our main contribution described in Section 3 below. Note that other accelerations are of interest but outside the scope of this paper, such as online, mini-batch or randomized algorithms. Furthermore, we turn towards a particular set of first-order algorithms coined as Block-Coordinate Descent (BCD), where only a subset of the parameters are updated at each iteration.

Two types of blocks for BCD

To compute approximate unconstrained tensor decomposition models such as the Canonical Polyadic Decomposition or the Tucker decomposition [START_REF] Kolda | Tensor decompositions and applications[END_REF], the Alternating Least Squares (ALS) algorithm is considered a baseline algorithm. In what follows, we first describe the well-known Alternating Nonnegative Least Squares algorithm (ANLS), which is an adaptation of ALS for aNCPD.

A block-coordinate framework : ANLS

In ANLS, the cost function f is minimized over each factor matrix U, V and W alternatively until convergence. For factor matrix U , we solve min

U ≥0 T [1] -U (V W ) T 2 F (2)
where matrix T [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF] is the first mode unfolding of tensor T as defined in [START_REF] Cohen | About notations in multiway array processing[END_REF] and is the Khatri-Rao product. Note that (2) is simply an equivalent rewriting of (1), with fixed V and W .

It appears now clearly that f is quadratic with respect to each block U , V and W . In other words, problem (2) is a nonnegative least squares problem in a matrix format. Clearly, ANLS is a BCD algorithm where blocks are {U, V, W }, in this order.

Solving ANLS with BCD : HALS

There are two categories of algorithms to compute NLS. A first category of algorithms produces the exact solution of the NLS in a finite number of steps, such as the historical active set solution by Lawson and Hanson [START_REF] Lawson | Solving least squares problems[END_REF]. The number of step is however not known and can be quite large. Also, each iteration in the active set algorithm requires solving a least squares problem, which can be costly. A second category asymptotically converges towards the optimal solution. These algorithms are only interesting if they converge in a shorter time than the exact ones. Among those, Hierarchical Alternating Least Squares is state of the art [START_REF] Gillis | Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization[END_REF] when dealing with several, alternated NLS problems.

In this paper, we call HALS-NLS the BCD algorithm solving the NLS problem that alternates on the columns of U in (2) while fixing the others. The solution for the ith column u i of U requires to solve min ui≥0

T [1] -U \i (V \i W \i ) T -u i (v i w i ) T 2 F (3)
and has the following closed-form solution

u i = T [1] -U \i (V \i W \i ) T (v i w i ) v i 2 2 w i 2 2 + (4)
since (3) boils down to the coordinate-wise minimization of second order polynomial functions over a half space. In (3), matrix M \i stands for a matrix M with the i-th column removed, while stands for the Kronecker product. Operator [x] + clips negative values in x to zero. The NLS problem is solved by alternatively using update rule (4) on the columns of the matrix being updated in the outer ANLS algorithm until convergence.

In what follows, we refer to the whole procedure "ANLS with NLS solved by HALS-NLS" as HALS.

Several important remarks can be done at this stage :

-HALS can be seen itself as a BCD algorithm to solve [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF], where blocs are

{u 1 , u 2 , . . . , u r , u 1 , . . . , u r , v 1 , v 2 , . . . , w 1 , w 2 , . . .}.
-Because updating the columns of U requires the computation of T [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF] (V W ) which is costly, it is computationally interesting to update the columns of U several times before switching to the columns of

V . -The product (V W ) T (v i w i ) can be computed effi- ciently as (V T v i * W T w i )
where * is the element-wise product.

Extrapolation for ANLS

Let us now discuss the proposed extrapolation scheme for ANLS, which is an extension of the one proposed in [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF].

In each step of the regular ANLS, we update a block of parameters using a NLS solver. For instance, when updating U , we compute U k+1 = NLS(V k , W k ; U k ) where U k is the initialization for U in an NLS solver such as HALS-NLS.

Extrapolated ANLS (E-ANLS) involves pairing variables Ũ , Ṽ and W initialized equal to U , V and W . At each iteration, e.g., for U , we compute

Update U k+1 = NLS( Ṽ k , W k ; Ũ k ) (5)
Extrapolate

Ũ k+1 = U k+1 + β k (U k+1 -U k ), (6)
where k is the current iteration index and β k is the extrapolation parameter. Similar updates are carried out for V and W . Intuitively, extrapolation makes use of the geometry of the cost function to predict estimates for a future iteration using previous estimates. In a convex optimization setting, Nesterov provided an optimal way to compute β k for steps [START_REF] Huang | A flexible and efficient algorithmic framework for constrained matrix and tensor factorization[END_REF] in terms of convergence speed. He showed that for this choice of β k , the error of the iterates of accelerated projected gradient methods converge in O( 1 k 2 ) instead of O( 1 k ) [START_REF] Nesterov | Introductory lectures on convex optimization : A basic course[END_REF]. However, as aNCPD ( 1) is non-convex, and because we are using extrapolation with the same β k for different blocks, there is a priori no closed-form solution to update β k . Thus, the critical part of the update is the parameter β k , which we now discuss.

Update of β

The extrapolation scheme [START_REF] Ang | Accelerating nonnegative matrix factorization algorithms using extrapolation[END_REF] has 5 variables, two of them indexed by the iteration index k : β k , βk , γ, γ and η, where β k ∈ [0, 1] is the extrapolation variable, βk ∈ [0, 1] is the ceiling parameter of β k . Parameters γ (η) control the way β grows (decays). Lastly γ is the growth parameter for βk . We grow or decay β k after each iteration as follows : let e k denote the error at iteration k after the extrapolation, then

β k+1 = min{γβ k , β} if e k+1 ≤ e k , β k /η else. (7)
It is important to note that extrapolation may increase the objective function value (e k ) due to a bad parameter β k . If this occurs, we abandon the extrapolation at that iteration and simply set the paring variable Ũ k+1 = U k+1 . In other words, when the error increases, we perform restart [START_REF] Candes | Adaptive restart for accelerated gradient schemes[END_REF]. In addition, an increase of e k means the current β k is too large, we then decrease β k as [START_REF] Kolda | Tensor decompositions and applications[END_REF]. To avoid β k+1 to grow back to the large value which caused the increase of the error, we update βk by ( 8)

βk+1 = min{γ βk , 1} if e k+1 ≤ e k and βk < 1, β k else. ( 8
)
The relations between the parameters are

∀k ≥ 0, 0 ≤ β k ≤ βk ≤ 1 < γ ≤ γ ≤ η.

An existing line-search approach for ANLS

It is often reported in the literature on computing approximate CPD that ALS is a viable algorithm only when using "line search" [START_REF] Bro | Multi-way Analysis in the Food Industry : Models, Algorithms, and Applications[END_REF]. Although it is not stated explicitly in that reference, what is actually done is extrapolation 1 . It is also reported that "line search" speeds up NLS for aNCPD considerably. The scheme is the following heuristic : 10) where h(k) is a recursive function so that h(k + 1) = h(k) if the error has not increased for more than four iterations, h(k + 1) = 1 + h(k) otherwise, and h(1) = 3. Also, it is reported that for constrained optimization, "line-search" should be discarded in the first few iterations (4 in tests below) because of instability issues. The main difference between this approach, which we refer to as "Bro" after the name of its author, and the extrapolation scheme we propose is that on top of being based on the Nesterov acceleration, the variables after our scheme are always nonnegative (only the pairing variables may have negative entries).

Update U k+ 1 2 = N LS(V k , W k ; U k ) (9) Extrapolate U k+1 = U k+ 1 2 + (k 1 h(k) -1)(U k+ 1 2 -U k )(

Experiments

Let us now compare various accelerated BCD techniques for aNCPD, namely Projected gradient (PG), HALS (baseline),

1. Line-search as usually understood in smooth optimization has been studied for approximate CPD in [START_REF] Rajih | Enhanced line search : A novel method to accelerate parafac[END_REF]

Median curves

Computational Time (sec) HALS with extrapolation using Bro's β tuning procedure, E-HALS (proposed), Accelerated Projected Gradient [START_REF] Xu | A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion[END_REF] (APG).

Notice that the HALS-based algorithms haves inner loops within the HALS and the PG-based algorithms have no inner loop. For fair comparisons we also consider PG and APG with inner loops, namely PG-r and APG-r. We compare these methods in three scenarios described below. In all tests, the rank r is small enough for uniqueness of factor matrices to hold up to trivial permutations and scaling.

Set up We form the ground truth tensor T true by sampling elements of matrices U true ,V true and W true from the uniform distribution U[0, 1]. To make the problem more difficult, we increase the condition number of the mode-1 matrix U by replacing U true (:, 1) with 0.01U true (:, 1) + 0.99U true (:, 2) (using Matlab array notations) and we add i.i.d. Gaussian noise to the tensor with variance σ 2 = 10 -4 . Note that some values in the noisy tensor may be negative due to the noise. We run all the algorithms under the same number of iteration (500) with maximum run time of 10 seconds, same number of inner loops (50) and random initialization U ini ,V ini ,W ini sampled from U[0, 1].

For E-HALS, we set β0 = 1, β 0 = 0.4, γ = 1.1, γ = 1.001 and η = 2 after calibration. We repeat the whole process 20 times. We perform the following three tests :

1. Low rank, balanced sizes We set I = J = K = 50, R = 10. Fig. 1 shows the cost function values for each algorithm against iteration and computational time. Table 1 provides the median reconstruction error of each mode compared to ground truth : we define RE final as the relative error between U k and U true in percent after column normalization and matching. The unpredictable behavior of Bro (error may increase significantly) shows that it is not a priori well-designed for aNCPD, even though its final error is sometimes on par with other algorithms. APG-r performs better than HALS, while E-HALS is faster than the other methods.

2. Low rank, balanced sizes, ill-conditioned We use the same setting as in the previous test. However here the condition number of factor matrix U is severely increased using U = U (I r + 1 r ), where I r is identity matrix of size r and 1 r is r-by-r all-1 matrix. Due to U being very ill-conditioned, method Bro di- verges (it therefore does not show on Fig. 2). On the other hand, E-HALS is not only the fastest method, see Fig. 2, but also provides much more accurate factor estimates, see Table 1. Unexpectedly, E-HALS seems to perform even better in this second test than in the previous one.

3. Medium rank, unbalanced sizes Here I = 150, J = 10 3 , K = 35 and R = 20. Fig. 3 shows similar results to the previous tests : E-HALS outperforms all other BCD approaches.

Conclusion

In this paper, we presented an algorithm to compute approximate Nonnegative Canonical Polyadic Decomposition that employs extrapolation to accelerate the convergence of a blockcoordinate scheme. The extrapolation is in the form of

U k+1 = U k + β k (U k -U k-1 )
where k is the iteration number and β k ≥ 0 is the extrapolation parameter. Experiment results show that the proposed scheme can significantly speed up the algorithm and is much faster than benchmark algorithms in several ill-conditionned settings. Future works will further improve on speed and adapt the proposed method to other decompositions such as approximate nonnegative Tucker. Also, we will adapt other acceleration methods desgigned in the unconstrained case (here we only compare with Bro [START_REF] Bro | Multi-way Analysis in the Food Industry : Models, Algorithms, and Applications[END_REF]), such as the one in [START_REF] Mitchell | Nesterov acceleration of alternating least squares for canonical tensor decomposition[END_REF].

FIGURE 1 -

 1 FIGURE 1 -The error curves e k -e min of various algorithms, where e min is the lowest e among all trials of all algorithms.

FIGURE 2 -FIGURE 3 -

 23 FIGURE 2 -Median value of e k -e min of HALS, E-HALS and APG-r for test

TABLE 1 -

 1 Median RE final in % of U, V, W , (* means ≥ 40)

	Algo	Test 1	Test 2	Test 3
	PGD	15, 1.8, 1.8	4.3, *, *	*, *, *
	PGD-r	0.2, 1.8, 1.6 4.1, *, *	*, *, *
	HALS	0.2, 1.6, 1.6 2.2, 22, 23	4.7, 5.2, 5.2
	E-HALS 0.2, 1.8, 1.8 0.04,0.3, 0.3 0.4,0.8,0.8
	Bro	0.2, 1.5, 1.5 0.2, 2.4, 2.4	*, *, *
	APG	0.5, 3.0, 2.9 4.3, *, *	*, *, *
	APG-r	0.2, 1.3, 1.2 2.7, *, 28	10,11,11

* NG acknowledges of the Fonds de la Recherche Scientifique -FNRS and the Fonds Wetenschappelijk Onderzoek -Vlanderen (FWO) under EOS Project no O005318F-RG47, and of the European Research Council (ERC starting grant no 679515).


Yewan Wang

David Nörtershäuser

Stéphane Le Masson

Jean-Marc Menaud

An Empirical Study of Power Characterization Approaches for Servers

Keywords: Server power model, Inlet temperature, IPMI, Red-Fish

Data centers are energy-hungry facilities. Emerging studies have proposed energy-aware solutions for reducing the power consumption of data centers. Power consumption characterization of servers is an essential part to realize power-aware adaption strategies. Traditional methods adopt accuracy and secure direct measurements by using physical instruments such as wattmeters. Recently, watt-meter free solutions are adopted widely as an economical replacement. These solutions provide power consumption information by making use of self-resources without additional instruments. There are two commonly adopted solutions: 1) standard specifications that provide interface with integrated sensors, such as Intelligent Platform Management Interface (IPMI) and Redfish; 2) Power models based on system activity related indicators. The energy-aware scheduling decisions are made based on the power values obtained, but few works give information about the correctness of the power values while discussing the results or drawing conclusions. In this study, we try to fill up this missing part by evaluating some commonly used, economical ways in obtaining power values. We compare and discuss the reliability, advantages and limitations for the CPU-utilization based power models. The findings highlight the challenges in realizing accurate and reliable power models. We also evaluate the reliability of IPMI and RedFish, in order to give references in choosing appropriate power characterization solutions.

I. INTRODUCTION

Cloud data centers are computer facilities formed by hundreds or even thousands of servers. With the increasing demand of cloud services, energy efficiency of servers in data center has become a significant issue. Reliable power characterization approaches are essential for supporting energy-aware solutions. High accuracy power analyses are capable of providing accurate power consumption data. However, limitations like high cost, weak support of integration with Unix systems (which are widely adopted among server platforms) make it unrealistic by using power analyzers in a data center environment. Therefore, some alternative economical solutions that make use of available resources without extra investment are widely adopted. The reliability of these solutions need to be evaluated thoughtfully before applying to different real situations. In this paper, we investigate experimentally two kinds of alternative solutions: 1) Power models based on system activity indicators, such as usage of physical components and Performance Monitoring Counters (PMC); 2) Industrial standard specifications based on integrated sensors, such as: IPMI, Redfish.

The paper is organized as follows: In Section II, we present some previous studies about building power models for PC or server systems. In Section III, we evaluate empirically the power models based on CPU utilization. The server is stressed with an industrial test tool. Principal components like CPU, memory and storage are stressed at different load levels. Then, in Section IV we evaluate the precision of Redfish and IPMI applied in a new series of IBM servers. The measures of power consumption recorded by Redfish and IPMI are compared with a high-accuracy power analyzer. Conclusions and perspectives are given in Section V.

The major contributions of this paper are:

• We evaluate power models based on CPU-utilization. Thermal influence of inlet temperature is especially discussed. • We give some suggestions in building accuracy power models according to experiments. • We discuss the reliability of power data provided by IPMI and Redfish based APIs.

II. POWER MODELS: BACKGROUND AND RELATED WORK

Modeling power consumption of servers is an active area of research. Power models takes advantages of indirect measurements to predict the power consumption of the elements in an IT system, from single components like processors, to the entire machine. Comparing to physical power analyzers, power models have several advantages. In general, it provides an economical way via making use of self-resources. In deeper insight, power models are capable of linking the energy usage data with system activities. That makes it possible to isolate the energy consumed by single process or Virtual Machine (VM). In this case, performance bottlenecks, inefficiency of algorithm are possible to be identified and optimized in a more dedicated way. In practical, power models are easy to integrated with server system, they can be used to help with some power management strategies, such as VM Migration, shut down technologies, etc. Therefore, it is not surprising that there has been a considerable amount of work in the area of power modeling.

A. CPU-Utilization based method

In early stage, power models adopt the utilization of CPU as the only input. One of the most notable study is conducted by Fan et al. [START_REF] Fan | Power provisioning for a warehouse-sized computer[END_REF], whose study has shown that the power consumption of servers can be accurately represented by CPU utilization by using simple linear relationship. The error is validated less than 5% for dynamic system activities. Economou et al. [START_REF] Economou | Fullsystem power analysis and modeling for server environments[END_REF] introduced a method called Mantis to model full-system power consumption. The model is built with linear regression based on component utilization metrics: CPU Utilization, off-chip memory accesses, disk and network I/O rates. The model achieves an overall error range from 0% to 15% for two different server systems. Especially, the blade model has errors less than 5% for all cases. After that, with the evolution of manufacturing, server architecture becomes more complex, the accuracy of models based on CPU utilization has been questioned in many ways. Orgerie et al. [START_REF] Orgerie | Demystifying energy consumption in grids and clouds[END_REF] argued that the relationship between the CPU consumption and its load is not linear. The results of their experiments showed that even applying the same CPU load, they observed three different power consumption values. Hence, they concluded that it is indeed not possible to get a linear function between CPU utilization and power consumption. Zhang et al. [START_REF] Zhang | A high-level energy consumption model for heterogeneous data centers[END_REF] validated the linear model for 392 published results, which composed of different kind of servers. They use R-squared values to evaluate their model. R-squared is also known as coefficient of determination. The value of R-squared ranges from 0 to 1, describes the goodness of prediction, the higher the better. The authors show that, among 395 published results, 6.5% (25 kinds of servers) have the R-squared values less than 0.95, which means the CPU utilization is not always correlated significantly with server power usage.

B. PMC based method

Furthermore, researchers try to build power estimation models with performance monitoring counters. PMCs record and store the counts of system-related activities. The principal of models based on PMCs is the selection of several PMCs, which have good correlation with power. The models can be then illustrated by linear, non-linear regression formula, or even by neural network. PMCs based power models usually have better accuracy in comparing with single indicators based model.

Da Costa et al [START_REF] Costa | Methodology of measurement for energy consumption of applications[END_REF] evaluate the power consumption of a PC by using performance counters, then extend the conception to predict the power consumption of single applications. Training data is collected by running several applications and synthetic benchmarks. A small number of optimal variables combinations within 165 different counters are selected for each synthetic benchmark, which has the best regression result with the real power consumption measurements. Then, a global model for the entire PC is derived by including and analyzing the possible candidate variables. The global model has Rsquared values greater than 0.94 for all the cases when applied respectively to each benchmark. Even though they did not evaluate the global model with some real applications, their results confirm the feasibility by using PMC to predict the power consumption of the IT systems.

Bircher et al. [START_REF] Bircher | Complete system power estimation using processor performance events[END_REF] propose a method to create power models for six subsystems (CPU, memory, chipset, I/O, disk and GPU) by using performance counters within processors. They chose several performance events which are highly correlated to power consumption in subsystem including memory, chipset, IO, disk and processor. Resistors are connected in series with power source to capture the power consumption for each subsystem. Their models are validated by a wide range of workloads and achieved an average error less than 9% for each subsystem. However, they did not mention the model accuracy for the entire system.

Witkowski et al. [START_REF] Witkowski | Practical power consumption estimation for real life hpc applications[END_REF] present a practical approach to estimation power consumption of applications in High Performance Computing (HPC) environment. Their models are represented as regression functions by using only a few variables related to CPU, motherboard and memory. Variables will be included in the model once the coefficient of determination increases. Some of the original variables are transformed to increase model accuracy. When validated with the same synthetic workloads during training phase, their model reports an average error between 1% and 4% comparing to real measurements. However, the average error is increased to a range of 3% -7% when tested with a real HPC application.

Some state-of-the-art power models provide platformspecific solutions [START_REF] Lewis | Run-time energy consumption estimation based on workload in server systems[END_REF] [START_REF] Basmadjian | Evaluating and modeling power consumption of multi-core processors[END_REF], which makes the model more accurate and adaptive in current situation. However, the method used to build the model is also less general and portable, limited to specific conditions. Some other researchers suggest advanced machine learning techniques to improve accuracy of PMC-based models for general use. Some of them point that, the accuracy of model can be greatly increased by removing some irregular outliers of measurements [START_REF] Witkowski | Practical power consumption estimation for real life hpc applications[END_REF] [START_REF] Costa | Methodology of measurement for energy consumption of applications[END_REF].

Cupertino et al. [START_REF] Cupertino | Towards a generic power estimator[END_REF] propose to use Artificial Neural Network (ANN), one of the computational intelligence technologies to improve the model accuracy. They compare the Mean Absolut Percentage Error (MAPE) between an ANN model and a traditional capacitive model, and show that the ANN can decrease the MAPE from 5.45% to 1.86%.

Wang et al. [START_REF] Wang | Span: A software power analyzer for multicore computer systems[END_REF] point out that for a given processor, the usage of PMCs is limited by the available event counters and the maximum number of PMCs that can be read simultaneously. Even more, power models with less PMC can be more flexible and applicable. Based on this fact, the authors have then proposed a power model with only Instruction Per Cycle (IPC) and frequency as inputs. In order to improve the accuracy, running benchmarks will be divided into different categories based on IPC values, then they build the models separately for each category. The authors also develop a tool SPAN to realize run-time power profiling and correlate power dissipation to source code functions. Their power model is validated by using two benchmarks from SPEC2008Cjvm, and achieve absolute error rate of 5.17% and 4.46% respectively. Tool "SPAN" achieve accuracy as high as 97% on average by running FT benchmark from NAS Parallel benchmark suite and synthetic workloads.

Mair et al. [START_REF] Mair | Pmc-based power modelling with workload classification on multicore systems[END_REF] present their power estimation model called W-Classifier. The model classifies different workloads into 5 categories by using some power-dominant PMCs: INT, FPU, FPU/cache mixed, INT/cache mixed and memory/idle. They validate W-Classifier with OpenMP multi-threaded benchmarks from NAS Parallel Benchmark suite on all 16 cores. They find that W-Classifier has an average MAE of 6.95% for all benchmarks, while traditional multi-variable model achieves an average MAPE of 40.74%. However, authors admit that W-classifier has difficulty to estimate the power consumption of benchmarks with large range of power variation. They have then proposed to improve the model by adding more kinds of classification categories as further work.

C. Challenges

Obtaining accurate results of consumption behavior at the whole system level or individual component level is not straightforward. The difficulties include but are not limited to the following raisons: 1) Diversity. Physical architecture of server differs very much between manufactures and becomes more complicated from generation to generation, with the emerging of new features. The availability of PMCs differs among different machines [START_REF] Witkowski | Practical power consumption estimation for real life hpc applications[END_REF]. The problem of the diversity makes the power models less portable between heterogeneous servers in Data centers. 2) Evolution. Evolution of system is somehow rapid and random. Some indicators used to build the original model would be no longer exist with the evolution of computing system. For example, four years after the introduce of Mantis [START_REF] Economou | Fullsystem power analysis and modeling for server environments[END_REF] (mentioned in 2.1), John C et al [START_REF] Mccullough | Evaluating the effectiveness of model-based power characterization[END_REF] have noticed that, some of the original indicators used by Mantis on longer exist in current systems. 3) Hidden system behaviors. Some component incorporates make optimization without exposing to any of the existing counters, which makes some device behaviors invisible to OS [START_REF] Mccullough | Evaluating the effectiveness of model-based power characterization[END_REF]. High precision will be difficult to achieve without being aware of these changes that affect power draw. 4) Variability. Previous studies have given quantitative evidences about the significant variability between identical designed CPU and servers [START_REF] Kistowski | Variations in cpu power consumption[END_REF]- [START_REF] Mehdi | Your cluster is not power homogeneous: Take care when designing green schedulers![END_REF]. Parameter variation, also known as Process, Voltage and Temperature (PVT) variation [START_REF] Humenay | Impact of parameter variations on multi-core chips[END_REF], is responsible for most of the variations. Among them, process and voltage variations are caused by imperfections from fabrication process. Temperature variation comes from fluctuating environments. These variations are inevitable and influential to power consumption. One of our previous study has demonstrated the power consumption variation between identical servers leads from thermal effects [START_REF] Wang | Potential effects on server power metering and modeling[END_REF]. These observations bound the potential accuracy achievable concerning all kinds of power modeling approaches.

All the evidences listed here highlights the challenges in building reliable power models for servers, especially for the modern ones.

III. POWER MODELS BASED ON CPU-UTILIZATION

In this section, we evaluate experimentally the effectiveness of using CPU-utilization as the indicator for building power models. Influence of inlet temperature variation has also been demonstrated by using a test use case.

A. CPU-Utilization based models

The model is built for a Gigabyte mw50-sv0 server, equipped with one Xeon E5-2609v3 processor. Several workloads from Server Efficiency Rating Tool (SERT) [START_REF] Lange | The design and development of the server efficiency rating tool (sert)[END_REF] are chosen as the test suite. SERT is an industrial standard rating tool for evaluating energy-efficiency for server systems, developed by SPEC committee. It contains a number of micro-workloads, called worklets that exercise different components of the Server Under Test (SUT) at different target load levels. (see Table I for details). In this experiment, ten worklets composed of both CPU and memory intensive types are chosen to stress the server. The test suit includes six CPU-intensive tests (Compress, CryptoAES, LU, SOR, Sort and SHA256), one CPU and memory hybrid test (SSJ), and two memoryintensive tests (Flood3 and Capacity3). Consumption at idle state is also measured. Details about the worklets used can be found in Table I [START_REF] Committee | Server efficiency rating tool public design document (latest version)[END_REF].

The power consumption data is collected by Yokogawa WT330, a high-accuracy power analyzer, with maximum measurement error less than 1%. CPU utilization is collected by redirecting the information from directory /proc/cpuinfo. Data collection frequency is set at 1Hz. At the end of test, box plot from matplotlib [START_REF] Hunter | Matplotlib: A 2d graphics environment[END_REF] is used to interpreted the dispersion of power for a given CPU utilization. Box plot is widely used for displaying statistic distribution, a simplified manner in comparison to a histogram or density plot. For a normal distribution, 50% of the data is within the box. Two short lines beyond the box represent for the minimum and maximum values within 99.3% of the data. Outliers represent for the remaining 0.7% data. The test result is shown in Figure 1. We can see from the figure that the models built from CPUutilization are not reliable enough. For a fixed CPU-utilization, the server power spreads out between a considerable range, especially for the utilization at 100%. Outliers are caused by the quick changes of system loads. The power ranges are relatively lower at 10%, 50%, 70% and 90%, because of a lack of experimental data at these load levels.

We illustrate then the relationship between CPU-utilization and server power for each worklet by marking with different colors. The result in Figure 2 shows that, for each worklet, server power has a narrow distribution for a specific CPU utilization, which means power consumption can be precisely modeled by CPU utilization during a single workload execution. Moreover, for the worklets from the same "CPU intensive" category, power is proportional to CPU utilization with different parameters. For instance, worklet LU has an obvious different way in increasing power comparing with the other CPU intensive worklets. As further work, we will try to improve the accuracy for CPU utilization based power models by classifying the workloads into different groups with PMCs. The results of the power consumption of server under different ambient temperatures can be seen in Figure 3. Server power increases with the rise of inlet temperatures. The increment of power is contributed mainly by fans and leakage current of CPU [START_REF] Wang | Potential effects on server power metering and modeling[END_REF]. Unlike power consumption, CPU utilization Taking the data sets of worklet SSL as a use case. Function (1) describes a baseline model proposed by [START_REF] Fan | Power provisioning for a warehouse-sized computer[END_REF]. Data set is collected at 22 • C inlet temperature. Estimated power is simply represented by a linear function by using the power values at idle and full load. Beyond the baseline model, Delta(T ) is derived by analyzing the relationship between inlet temperature and power increment for the whole data sets. Delta(T ) can be interpreted by a quadratic equation as shown in function [START_REF] Economou | Fullsystem power analysis and modeling for server environments[END_REF]. Finally new power model is built by adding Delta(T ) to the baseline model, as shown by [START_REF] Orgerie | Demystifying energy consumption in grids and clouds[END_REF].

P estimated = P idle + U cpu% (P 100% -P idle ) (1)
Delta(T ) = a 0 + a 1 T + a 2 T 2 (2)
P estimated = P idle + U cpu% (P 100% -P idle ) + Delta(T ) (3)
Within the formulas, U cpu% represents CPU utilization in percentage and T is inlet temperature. P idle and P 100% are the average powers (Watt) when server running at idle (U cpu% = 0) and full load (U cpu% = 100). The models are trained and validated with the same data set by using cross validation from function cross_val_score from scikitlearn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF], cross validation (cv) generator is set to 4 to realize a 4-fold cross validation. The average MAPEs after CV for model ( 1 [START_REF]Intel. Intelligent platform management interface: Ipmi adopters list[END_REF]. IPMI is designed to realize system-management independently without passing through OS. Administrators are allowed using IPMI to manage the machine locally or remotely regardless of its state (on or off). Monitoring system status is one of the functionality of IPMI. IPMI can communicate with Baseboard Management Controller (BMC) to retrieve data of certain hardware components (temperature probe, Fans, power supplies, etc.). BMC is a specialized micro controller embedded on the motherboard by the vendors. There are several open source tools supporting IPMI protocol, such as ipmitool, freeipmi, OpenIPMI, etc. Then, with the massive growth in size and complex of Data centers, traditional IPMI is not sufficient to manage the modern scalable data centers anymore. Hence, In 2010, Distributed Management Task Force (DMTF) proposed Redfish to overcome the limitations of IPMI in terms of scalability, performance, simplicity and interoperability [START_REF]Validation of redfish: The scalable platform management standard[END_REF]. In comparison with IPMI, Redfish is a standard API adopts HTTPS protocol, which is considered more secure than UDP protocol (adopted by IPMI). In addition, Redfish use human readable technologies like JSON and OData, which makes the operations such as request and response more user friendly. However, the work for Redfish hasn't finished and is still a Work in Progress on the website of DMTF [START_REF]Redfish api[END_REF].

A. IPMI or Redfish: Which is more accurate?

The difference between IPMI and Redfish makes us curious about their actual capabilities of power characterizations. Before implementing specific tool into energy-aware projects, it is better to well aware of the accuracy and limitations of the tool. Therefore, we design and conduct an experiment to evaluate accuracy of power consumption readings from IPMI and Redfish. Their readings are recorded and compared in real time with a high accuracy power analyzer. The experiment is conducted in a prototype modular server from Lenovo Skylake series. Both IPMI and RedFish interfaces are supported. The server equipped with two Xeon Gold 6142@2.6GHz processors, 390GB RAM and 500G SSD. Server is stressed with the same test suite SERT as mentioned in previous experiments, with a total execution time of about 2 hours. The power usage data from IPMI are retrieved by an open source API tool freeipmi [START_REF] Team | Freeipmi -home[END_REF]. And readings of Redfish are provided by Redfish REST API, through the integrated XClarity Controller introduced by Lenovo. High accuracy power analyzer Yokogawa WT330 is place between server power supply unit and wall plug to measure and record power consumption data as reference (with maximum measurement error less than 1%). A docker container is developed to redirect and synchronize the readings from three channels to local database by using Network Time Protocol (NTP). Sampling frequency is set to 2 Hz.

We calculate MAPEs for both IPMI and Redfish within three power ranges. The results shown in Table III show that, sensors integrated in servers seem to have different precisions across different power ranges. Otherwise, power readings retrieved from Redfish have better results comparing to IPMI for each ranges. Power consumption has denser variations during lower ranges. After several experiments, we find that, the data collected from Redfish and IPMI is not refreshed instantly. A latency of about 200 ms is observed, which means the power value recorded may come from 200ms ago. The accuracy can be questioned when power varies quickly.

The work presented in this paper give references in choosing power consumption characterization solutions. We suggest that, before applying available economical solutions in an energy-aware scheduling or distribution task, check the precision of the solution and make sure that accuracy is enough for supporting the decision making. V. CONCLUSION

In this paper, we present a deep evaluation about the power models based on CPU utilization. The influence of inlet temperature on models has been especially discussed. According to the analysis, one regression formula by using CPU utilization as the only indicator is not adequate for building reliable power models. First of all, Workloads have different behaviors by using CPU and other hardware resources in server platforms. Therefore, power is observed to have high dispersion for a fixed CPU utilization, especially at full workload (CPU utilization = 100%). At the same time, we also find that, power is well proportional to CPU utilization within the execution of one single workload. Hence, applying workload classifications could be an effective way to improve model accuracy. Moreover, inlet temperature can cause surprising influence on model accuracy. The model reliability can be questioned without including inlet temperature data. In a use case, after including inlet temperature data, we have greatly improved the precision of model outputs while stressing server under three different ambient temperatures. Using industrial specifications, such as IPMI and Redfish is another popular way to get power consumption data for some modern HPC servers. The experiment results show that, the precision of both IPMI and Redfish differs from different power ranges, the higher the better. We blame the loss of precision to the latency during request. Comparing to IPMI, Redfish is observed to have less latency in our experiments.

Figure 1 .

 1 Figure 1. Distribution of power under different CPU-utilization.

Figure 2 .

 2 Figure 2. Relationships between CPU-utilization and server power of different worklets.

Figure 3 .

 3 Figure 3. Server power under three different ambient temperatures.

Figure 4 .

 4 Figure 4. CPU utilization under different inlet temperatures.

  ) and (3) shown in Table II demonstrate the effectiveness of the updated model (3). The MAPE of baseline model (1) increases dramatically with the inlet temperature rises from 22 • C to 45 • C.

TABLE I .

 I TEST SUITE INFORMATION

	Worklet	Components	Description	Load Levels
	Compress	CPU	Compress and decompress data	100%, 75%, 50%, 25%
	CryptoAES	CPU	Encrypts and decrypts data	100%, 75%, 50%, 25%
	LU	CPU	LU factorization of dense matrix operations	100%, 75%, 50%, 25%
	SOR	CPU	Jacobi Successive Over-relaxation workload	100%, 75%, 50%, 25%
	Sort	CPU	Sorts randomized 64-bit integer array	100%, 75%, 50%, 25%
	SHA256	CPU	SHA256 hashing transformation and encryption/decryption	100%, 75%, 50%, 25%
	SSJ	CPU/Cache/Memory	simulates Online Transaction Processing (OLTP) operations	100%, 87.5%, 75%, 62.5%, 50%, 37.5%, 25%, 12.5%
	Flood	Memory	Measures memory bandwidth across four common and important array operations	Full, Half
	Capacity	Memory	Exercises Java's XML Validation	Base, Max
	Idle	System	No load on SUT	None

TABLE II .

 II MAPE OF MODELS AT DIFFERENT INLET TEMPERATURE.

	T Inlet ( • C) Model without Delta (T • )	22 5.6 %	30 9.2 %	45 16.8 %
	Model with Delta (T • )	4.6 %	4 %	3.2 %
	IV. INDUSTRIAL STANDARD SPECIFICATIONS
	EVALUATIONS: IPMI & REDFISH
	IPMI and Redfish are usually available in modern high
	performance servers. They can be used to monitor system
	state information such as power consumption, inlet and exhaust
	temperatures through specific interfaces. IPMI represents for
	Intelligent Platform Management Interface, created by Intel,
	Dell, HP and NEC in 1998. It is a standardized hardware
	management interface and has been widely implemented on
	more 200 server vendors nowadays

TABLE III .

 III MAPE OF REDFISH & IPMI BETWEEN DIFFERENT POWER RANGES.

	Power Ranges (Watt)	Redfish MAPEs (%)	IPMI MAPEs (%)
	From 0 to 199	4.1	5.5
	From 200 to 399	4.0	4.4
	From 400 to 600	1.8	2.3
	From 0 to 600	2.9	3.7