Chi Dung Tran
email: chidung.tran@univ-valenciennes.fr

Houcine Ezzedine
email: houcine.ezzedine@univ-valenciennes.fr

Christophe Kolski
email: christophe.kolski@univ-valenciennes.fr

A generic and configurable electronic informer to assist the evaluation of agent-based interactive systems

The evaluation of user interactive systems has been an active subject of research for many years. Many methods have been proposed but most existing evaluation methods do not take the specific architecture of an agent-based interactive system into account and nor do they focus on the coupling between the architecture and evaluation phase. In this article, we propose an agent-based architecture of interactive systems that is considered as being mixed (it is both functional and structural). Based on this architecture, we propose a generic and configurable model of an evaluation tool, called "electronic informer", designed to assist evaluators in analyzing and evaluating interactive systems with such architecture.

Introduction

Nowadays, in spite of the existence of several methodologies for the development of interactive systems, designing, developing and assessing, in terms of utility and usability [START_REF] Bastien | Evaluating a user interface with ergonomic criteria[END_REF][START_REF] Nielsen | Usability Engineering[END_REF][START_REF] Shneiderman | Designing the user interface : strategies for effective human-computer interaction[END_REF] an agentbased interactive system is still a difficult task. It is therefore necessary to provide methods, models and evaluation tools to make it easier. The section 2 of this paper presents a brief state of the art concerning the architectures for traditional interactive systems as well as for agent-based interactive systems. The section 3 proposes an architecture that is both functional and structural. By using this architecture as a basis, in the section 4, we propose a generic and configurable model of an evaluation tool called "electronic informer"; its aims at assisting the evaluation of interactive applications of this architecture. The last section is used for our experiment and the conclusion.

Interactive system architectures

Architecture of an interactive system supplies the designer with a generic structure from which he/she can build an interactive application. It is a set of structures that include: components, the outside visible properties of these components and the relations between them [START_REF] Coutaz | Architecture logicielle conceptuelle des systèmes interactifs, chapter 7 of «Analyse et Conception de l'Interaction Homme-Machine dans les systèmes d'information[END_REF]. Researchers have proposed several architecture models over the past twenty years. These architecture models recommend the same principles, based on the separation between the functional core of system (application) and the human-machine interface. This separation makes modifications easier; it allows modifying the interfaces without affecting the application. Two main types of architecture can be singled out: (1) functional models, such as Seeheim [START_REF] Pfaff | User interface management system[END_REF], Arch [START_REF] Bass | The Arch Model: Seeheim revisited[END_REF]) and (2) structural models, such as PAC proposed by J. Coutaz [START_REF] Coutaz | PAC, an Object-Oriented Model for Dialog Design[END_REF], and its variations, AMF [START_REF] Tarpin-Bernard | AMF : un modèle d'architecture multi-agents multi-facettes[END_REF] or MVC [START_REF] Goldberg | Smaltalk-80, the interactive programming environnement[END_REF]) and its variations.

The functional model split an interactive system into several functional components. For example, the Seeheim model is made up of 3 logical components (Presentation, Dialogue Controller, Application Interface); the Arch model defines a functional breakdown of an interactive system into 5 components in which: both the presentation and interaction components are a decomposition of the presentation of the Seeheim model, the functional kernel component, the domain adapter component and the dialogue controller component. The functional models enable to separate system analysis-design difficulties by decomposing into different modules. Nevertheless, the functional models show some disadvantages. First, they do not define the internal structure of modules and the communication between them. Second, Arch and Seeheim provide canonical functional structures with big grain, the functionalities are mixed in the too macroscopic components (Tarpin-Bernard 1999); they are useful as a structural framework for a design or a rough analysis of the functional decomposition of an interactive system [START_REF] Trabelsi | Architecture modelling and evaluation of agent-based interactive systems[END_REF]). These decompositions are generally not enough to complex applications. These inconveniences make functional systems inadaptable to complex applications in general and to industrial supervision systems in particular.

The structural models aim at a finer breakdown by using structural components, and in particular those said to be distributed or agent approaches, suggest grouping the functions together into one entity, the agent. The agents of this type of architecture are then organized in a hierarchical manner according to principles of composition or communication. For example, a MVC agent is made up of three facets: Model, View and the Controller. The PAC model defines an agent using three facets: the Presentation, the Abstraction and the Control. The decomposition into many autonomous and cooperative entities enables to accelerate the feedback of system to the user. This advantage is very useful to the supervision applications that can be highly interactive; in consequence, the intensive dialogue between the user and the application can slow the system down. In spite of this advantage, the structural models have also following disadvantage: the number and the role of agents are not made clear. Moreover, the designer of a supervision application can have difficulty in following the global interface because the interface is distributed in many facets "Presentation" of agents. The functional models can solve this problem because they provide only one "Presentation" component to represent the interface.

Our approach is intended to be mixed as its principles borrow from both types of model; it is both functional and structural.

Agent-based architecture proposed as a mixed model

The mixed model is a combination of these two above mentioned models in order to exploit advantages of each of them. The agent-based architecture proposed (Grislin-Le [START_REF] Strugeon | Agents intelligents en interaction hommemachine dans les systèmes d'information[END_REF] has to be considered as a such mixed model. We suggest using a division into three functional components recommended in the Seeheim model which we have called respectively: interface with the application (connected to the application), dialogue controller, and interface or presentation (this component is directly linked to the user). Each of these components can be broken down further in a structural approach in the form of agents. These components are built like three multi-agent systems and they are considered as working in parallel, at least, at a theoretical point of view. The application agents, manipulating the field concepts of the application, cannot be directly accessed by the user. One of their roles is to ensure the real time transmission of the information necessary for the other agents to perform their task. The interface agents are in direct contact with the user (they can be seen by the user) through the associated user interface events. These agents co-ordinate between them in order to intercept the user commands and to form a presentation that allows the user to gain an overall understanding of the current state of the application. The control agents in the Dialogue Controller component provide services for both the application and the interface agents in order to guarantee coherency in the exchanges emanating from the application towards the user, and vice versa. Their role, in particular, is to link the two other components together by distributing the user commands to the application agents concerned, and by distributing the application feedback towards the interface agents concerned. All these agents communicate amongst themselves in order to answer the user actions. Each agent of this architecture associates with a set of services that are the actions that this agent can execute. The communication between agents is realized by the invocation between services of agents. This architecture as well as its events (user interface events, services) has been formally described before proposing a generic and configurable "electronic informer", designed to assist evaluators in analyzing and evaluating interactive systems of this architecture.

Proposition of an electronic informer adapted to agent-based interactive systems

An "electronic informer" is a software tool that ensures the automatic collection, in a real situation, of users' actions and their repercussions on the system. The collection of information is done in a discreet and transparent way for the user, who must not at any time feel hampered by the presence of the informer. This is an advantage of such a tool. Objective data collected through interactions can be treated, analyzed and shown in a synthetic shape to the evaluator. This facilitates the analysis of the results. At this moment, many evaluation tools have been proposed but they have some disadvantages and limitations: (a) The current evaluation tools do not take into account the specific architecture of an agent-based interactive system [START_REF] Trabelsi | Architecture modelling and evaluation of agent-based interactive systems[END_REF]) and there are rarely propositions concerning the coupling between the architecture and evaluation phase. (b) The current electronic informers often contain some stages like collecting interaction data, and then retrieving these data to realize some analysis such as counts, summary statistics, detecting patterns, etc., and finally visualizing analysis results in a synthetic shape to the evaluator. For example, the tool WET [START_REF] Etgen | What does getting WET (Web Event-logging Tool) Mean for Web Usability? User Experience Engineering Division AT&T Labs[END_REF] collects only interaction data between user and the interface without any later analysis, the tools RemUSINE [START_REF] Paterno | RemUSINE: a bridge between empirical and model-based evaluation when evaluators and users are distant[END_REF], WebRemUSINE (Paganelli and Paternò 2002) realize some analysis such as calculations, summary statistics concerning tasks executed and Web pages visited by the user and then, then visualize analysis results to the evaluator in terms of diagrams, WebQuilt represent sequences of visited Web pages in terms of an interactive directed graph (Hong et al.2001). After tools show analysis results, in order to identify problems of the user interface and propose useful suggestions for improvement, the evaluator must interpret these analysis results by himself without any indications or assistances. As consequence, evaluation results depend on the ability and experience of the evaluator very much. (c) The current tools working with guidelines often read source code of the user interface to determine that whether their static presentations (color or size of the text fonts, position or size of the button...) violate a set of predefined guidelines. They do not take into account of interactive behaviors of the application in terms of interactions between user and interface or between components of the application for the evaluation. (d) Some current "feedback quality agents" are only softwares used to gather data about what were happening in the application whenever it crashes. The "feedback quality agents" only gather technical information about the context, state of the application when it had problem such as OS Version, Processor Type, Display Type, register, functions were called just before the failure, etc. And then, these data are sent to the development team to help them identify problems, the cause of the crash more readily and then, improve the future version of the application. "Feedback quality agents" can permit the user to report what he/she was doing when the failure appears to the development team. These tools only take into account of the context and state of the application when it had problem, as consequences, assistances in evaluating of interactive applications are limited compared other tools like electronic informers or tools working with guidelines.

We propose a generic and configurable model of an "electronic informer" to remedy these limitations. The first version of an electronic informer has ever been studied and developed [START_REF] Trabelsi | Architecture modelling and evaluation of agent-based interactive systems[END_REF]). However, it is not a generic tool but only a specific one. This first specific tool aims at evaluating a specific agentoriented applicative application that is intended to manage the passenger information on a public transport system in a project called SART [START_REF] Sart | Système d'Aide à la Régulation de Trafic du réseau de transport valenciennois et de ses pôles d'échanges[END_REF]. It cannot be used to evaluate other agent-oriented applications because it depends on the number of agents, the structure and the contents of such systems. Furthermore, it also shows some inconveniences and shortcomings. We solve such problems by proposing and developing a generic and configurable model of an "electronic informer" made up of 7 main modules (Figure 2). Our tool takes into account of the architectural specificities of interactive applications of the proposed architecture. In particular, our tool collects interactions between the user and the interface agents in terms of occurred user interface events and interactions between agents themselves in terms of executed services to evaluate interactive applications. After analyzing collected data, this tool provides the evaluator with an open list of determined criteria which can be ergonomic criteria or quality attributes. These criteria are associated to analysis results to give to the evaluator clear indications and assistances in interpreting analysis results in order to identify problems of the application based on these criteria and then, he/she can propose useful suggestions to the designer for necessary modifications to improve the application. These associations should be automated as much as possible in the future version of this tool.

Module 1 (M1): This module can run in background to collect events that appear (occurred user interface events and executed services) from all agents of the concerned interactive system and save them into a database which will be exploited by other modules. This module 1 and evaluated application can be in the same or other place. That means the evaluation can be done remotely. The module 1 and the other modules do not communicate directly each other. The data collection and its treatment are separated. As a result, the module 1 can be modified without affect the other modules and vice versa.

M2: this module enables the evaluator to associate events in intermediate level (user interface events and services) with each task. Several events in intermediate level can be realized to obtain a certain task. For example, the user interface events Image_Vehicule_Click, TabDriver_click, TextBoxMessage_OnChange and buttonOK_Click can be associated with the task "Send a message to the driver/voyagers" of the system intended to supervise the passenger information on a public transport system in the SART project.

Fig. 2. Generic and configurable model of the electronic informer proposed

M3: processing collected data of a chosen agent (interface, control and application agents) or all the agents in a certain period of time and showing results in forms understandable for the evaluator. Here are examples of calculations and statistics: response time for interactions between services; time for a certain user interface event (time for typing a text box…); time for completing a service and a task; the percentage of services accomplished and furthermore, of tasks accomplished, the error's percentage, the percentage of services and furthermore, of tasks achieved per unit of time, the ratio of failure or success for interactions between services, the ration of failure or success for each or all the tasks, the ration of appearance of each user interface event of a certain interface agent, the percent-age of execution for each service of a certain agent, and so on. The results are showed in table or graph form (figure 3,4).

M4: generating Petri Nets (PNs) to describe activity process of agents and users in the system from collected data. Indeed, it describes process of interactions between services of different agents and process of activity of user (in terms of user interface events) to realize a certain task. We call them "observed" PNs. Generated PNs brings evaluators visual views of all real activities of the user and the concerned system and can be used for comparison purposes later. The left part of the figure 5 and the figure 6 illustrates generated PNs describing the activity process to realize the task "Send a message to stations" of the application that supervises the vehicles of urban common transport in the SART project. The generated PNs are described by PNML (Petri Net Markup Language) and are opened by the tool Renew version 2.1. In the figures 5 and 6, eviuM,N-I stands for user interface event M of the interface agent N, sM,N-I(A) stands for service M of the interface (application) agent N. M5 (figures 5 and 6): comparing PNs generated above with theoretical PNs that system designer has intended. This comparison assists the evaluators in detecting use errors; for ex., one can perceive that the user has done useless manipulations or chosen a non-optimal way to realize a task. These comparisons can also be used to assist the evaluator in learning habits of users or in evaluating and comparing the ability of different users to use a system. For ex., the fig. 5 tells the evaluator about the way chosen by the user to realize the task among three ways predicted by the system designer. The fig. 6(A) let the evaluator know that the way chosen by the user 1 is optimal but the one chosen by the user 2 is not. The fig. 6(B) let the evaluator perceive two problems: (1) the user 3 realized unsuccessfully the task because of an error of the service s2,3-I. This service must invoke the service s1,2-A but in fact, the service s1,2-A is not executed. One can also perceive this problem by comparing generated PNs of user 2 with theoretical PNs that system designer has intended. (2) the user 4 executed an useless manipulation through the UI event button_Cancel_Click (red arrow on the figure). As a result, the UI event Window_Properties_Station_Hidden was triggered and the user 4 had to execute the manipulation through the event Image_Station_Click again. M6: the evaluator must interpret analysis results to evaluate agent-based user interactive systems as mentioned above. M6 is responsible to assist the evaluator in doing that. As a result, the evaluator can criticize concerned system and give useful suggestions to the designer for necessary modifications to improve it. M6 is not an expert system but a tool to assist the evaluator in evaluating concerned system by providing him/her with an open list of criteria which can be ergonomic criteria or quality attributes. These criteria must respect some principles: (1) they aim at evaluating 3 different aspects of agent-based interactive systems: user interface, user ability to manipulate systems, some non-functional properties such as response time, reliability, so on. (2) These criteria must be evaluated completely or partially using the data collected and processed by the other modules. In other words, the evaluator has to be able to interpret the data collected and processed by the other modules to evaluate the systems based on these criteria. In the future, the electronic informer can be combined with other methods to evaluate more enough and exactly such criteria. (3) The list of criteria is open and modifiable. The evaluator has to be able to add new criteria or modify the old criteria. He/she can use the criteria from available sources or determine the criteria by himself/herself. (4) The criteria can be generic or specific for the evaluated application. For example, the criteria such as "response time", "complexity", "immediate feedback" are generic and can be used to evaluate all the agent-based interactive systems but the criteria such as "Does the regulator find easily the necessary vehicle?", "Does the regulator find easily the necessary station?" are specific for the agent-based interactive system in the SART project. These specific criteria influence much the satisfaction of user of this system. The figure 7 illustrates this module. Each criterion is composed from 4 parts: title; definition; interpretation (explain to the evaluator how to use the collected data as well as its analysis results to evaluate this criterion); evaluating this criterion (according to this criterion, the evaluator enters his/her critiques for the system and suggestions to fix it). Because of lack of place, it is not possible to describe in detail this module in the paper. M7: this module enables the evaluator to configure electronic informer to evaluate different agent-oriented systems by entering the specific data of the evaluated system, for example, the Specification of Agents that describes agents, associated services and so on., the tasks that the user can realize to reach his/her goal and some other configuration parameters.

Conclusion

We have presented a brief state of the art concerning interactive system architectures, and proposed a mixed architecture as well as a generic and configurable model for assisting the evaluation for agent-oriented interactive systems. An electronic informer has been proposed and its constitutive modules have been described. We intend to combine this "electronic informer" method with other methods (questionnaire, interview…). That needs to combine data collected from "the electronic informer" and data collected from the other methods to evaluate more efficiently such systems. An experiment is planned during the first semester of 2008 at the laboratory LAMIH, University of Valenciennes and Hainaut-Cambrésis, France with about 5-10 participants. This experiment is the final stage of the SART project and it will show the advantages as well as inconveniences of this tool to help us determine the improvements in the future.

Fig. 1 .

 1 Fig. 1. Our agent-based architecture

Fig. 3 .

 3 Fig. 3. One of screen pages of M3: UI events occurred in the interface agents in table form

Fig. 4 .

 4 Fig. 4. Two screen pages of M3. (A) The results of the realizations of tasks in graph form. (B) Some calculations (the ratio of successful tasks, average response time, confrontation between real and predicted time to realize tasks, and so on).

Fig. 5 .

 5 Fig. 5. Comparing generated PNs of user 1 (left side) with PNs intended by designer (right side) to realize the task "Send a message to stations"

Fig. 6 .

 6 Fig. 6. Confrontation between PNs to realize the task "Send a message to stations" (A) Comparing generated PNs of user 1 (left side) with PNs of user 2 (right side) (B) Comparing generated PNs of user 3 (left side) with PNs of user 4 (right side)

Fig. 7 .

 7 Fig. 7. Screen page of the module 6

ACKNOWLEDGMENTS

The present research work has been partially supported by the "Ministère de l'Education Nationale, de la Recherche et de la Technologie», the «Région Nord Pas-de-Calais», the FEDER (MIAOU, EUCUE, SART), the ANR ADEME (Viatic.Mobilité), the PREDIM (MouverPerso).


Lucille Gallardo
email: lucille.gallardo@gmail.com

Lara Gautier
email: lara.gautier@umontreal.ca

Fanny Chabrol
email: fanny.chabrol@ird.fr

Valery Ridde
email: valery.ridde@ird.fr

Private consulting firms' intervention in public health policymaking: an exploratory review

Keywords: consulting firms, consultancy, public policy, epidemics, health, public administration, policymaking

Context: While there is ample research in the social sciences on the role of private consulting firms in public policy, there is little information about their intervention in managing public health crises and epidemics. The COVID-19 pandemic revealed how much public administrations across the globe have been using these firms. The purpose of this exploratory review of scientific literature is to identify research on the involvement of these firms in governing epidemics and health crises since 2000.

Methods: This review investigates the following question: what research evidence about the role of these firms is there, and what research methods and analytical categories are used?

Following the stages of the PRISMA methods, we identified 24 references since 2000.

Findings: We classified authors' analyses of the role played by those firms using three analytical categories: the management approach, the consultocracy phenomenon, and the phenomenon of elite hybridization. Only two references were explicitly related to the work of consulting firms in the context of epidemics (e.g., . The others focused on public health reforms. This finding confirms the scarcity of research evidence on the role played by consulting firms in the management of epidemics.

Conclusions: This review reports on a blind spot of the scientific literature and calls for additional empirical research.

Introduction

There is a large body of research in the social sciences on the role of private consulting firms in public policy [START_REF] Empson | The Oxford Handbook of Professional Service Firms[END_REF][START_REF] Hurl | Professional Service Firms and Politics in a Global Era: Public Policy, Private Expertise[END_REF][START_REF] Saint-Martin | Le consulting et l'Etat : une analyse comparée de l'offre et de la demande[END_REF]. Several studies reported the increase and standardization of their intervention with national and international public organizations for more than 30 years [START_REF] Saint-Martin | Les consultants et la réforme managérialiste de l'État en France et en Grande-Bretagne : vers l'émergence d'une « consultocratie[END_REF][START_REF] Bezes | État, experts et savoirs néo-managériaux[END_REF][START_REF] Henry | Un entrepreneur de réforme de l'État : Henri Fayol (1841-1925)[END_REF][START_REF] Belorgey | Une « consultocratie » hospitalière ?[END_REF][START_REF] Gautier | L'adoption de la réforme du système de santé au Mali : rhétorique et contradictions autour d'un prétendu retour de la santé communautaire[END_REF][START_REF] Kipping | America First: How Consultants Got into the Public Sector[END_REF]. Among this work, research in the health sector focuses on the intervention of private consulting firms in the framework of national health sector reforms, such as those of the National Health Service in the United Kingdom since 1974 [START_REF] Kirkpatrick | Management Consulting in Health[END_REF] or the 2009 hospital reform in France (Belorgey, 2010;[START_REF] Belorgey | Une « consultocratie » hospitalière ?[END_REF]. The issue of epidemics, and more generally of public health crises, remains little studied in these writings. However, the COVID-19 pandemic has been an opportunity to highlight the use of private consulting firms by many administrations worldwide (Aron and Miguel-Aguirre, 2021;[START_REF] Castonguay | Au coeur de la bataille pour sauver le Québec. Un récit exclusif des dessous de la crise[END_REF][START_REF] Collington | Britain's public sector is paying the price for the government's consultancy habit[END_REF][START_REF] Petitjean | Covid-19 : ces consultants au coeur de la 'défaillance organisée' de l'Etat[END_REF][START_REF] Rice | What a Spectacularly Ill-Advised Idea Says About McKinsey[END_REF][START_REF] Vogelpohl | Pandemic consulting. How private consultants leverage public crisis management[END_REF]. In France, for example, the controversy about the use of private expertise by the French central administration date from the beginning of the COVID-19 health crisis [START_REF] Braun | Use of consultancies for vaccine rollout sparks controversy in France[END_REF][START_REF] Cohen | Quand McKinsey conseille la stratégie vaccinale: comment les cabinets de conseils se sont imposés dans la gestion de l'État[END_REF][START_REF] Girard | Coronavirus : le cabinet privé qui pilote la stratégie de dépistage de la France[END_REF]. The publication of the report of the French Senate Committee of Inquiry (Commission d'enquête sur l'influence croissante des cabinets de conseil privés sur les politiques publiques, 2022) revived the controversy. A recent journalistic investigation showed a similar phenomenon in Quebec where one of the most important transnational consulting firm has worked with the central administration to manage the COVID-19 pandemic [START_REF] Gerbet | Dans l'ombre, la firme McKinsey était au coeur de la gestion de la pandémie au Québec[END_REF].

The objective of this exploratory study is to review the academic knowledge on the worldwide intervention of consulting firms in public policy during epidemics and public health crises that have occurred since 2000. We unravel the theoretical approaches and methods used by authors to study the role of private firms.

Method

Scoping reviews (Arksey and O'Malley, 2005;[START_REF] Dagenais | Guide de soutien au développement d'une communauté de chercheurs, praticiens, et décideurs compétents pour réaliser des revues de portée (scoping review)[END_REF][START_REF] Peters | Updated methodological guidance for the conduct of scoping reviews[END_REF] produce information on an issue within a reasonably short timeframe; they include research works regardless of the methods employed by their authors [START_REF] Munn | What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences[END_REF][START_REF] Saul | A time-responsive tool for informing policy making: rapid realist review[END_REF]. We followed the stages referenced in the PRISMA-ScR model [START_REF] Tricco | PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation[END_REF]. We presented them in the review protocol [START_REF] Gallardo | Consulting firms in the public management of epidemics. A scoping-review protocol[END_REF].

We built the search strategy (Table 1) around three themes: keywords referring to consulting firms, keywords referring to epidemics and public health crises, and, finally, keywords involving national and international public administrations. After several search attempts, we realized the paucity of works on epidemics and health crises. We thus decided to review works on public health policy more broadly, so as not to miss some pertinent research. We identified six inclusion criteria. References must be:

-Related to consulting firms, that are private firms (both cross-industry multinationals and specialized companies with national or even local roots), with for profit ambition, which sellexpertise services to public actors.

-Related to epidemics or health crisis issues which have emerged from the 2000s.

-Published between January 1 st , 2000, and December 31 th , 2021. This time frame is justified because of the important growth of private consulting firms since the 2000s.

-Published in English, French, Spanish, Italian and Portuguese.

-Original research papers using qualitative, quantitative, and mixed methods.

-Peer-reviewed journal articles and books/book chapters.

We excluded non-academic documentation, that is any reports and working documents also named "grey literature" which are not subject of a peer-reviewed evaluation (e.g., think tank or international and national organization reports). This choice is linked with the objectives of the scoping-review that are to make a state of academic knowledge about our research object.

Based on these pre-established inclusion and exclusion criteria [START_REF] Gallardo | Consulting firms in the public management of epidemics. A scoping-review protocol[END_REF], we used a step-by-step approach that is summarized in the PRISMA-ScR flow diagram (Figure 1). Of the 964 initial references identified in five databases (Embase, PubMed, Scopus, Web of Science, and Bielefield Academic Search Engine), we retained 15 items through sorting and double proofreading. We decided to use Bielefield Academic Search Engine because this European database integrates non-anglophone academic publications which are not always referenced by mainstream databases. Through analyzing bibliographies, we manually added nine references found outside the databases, leading to a total of 24 references. We grouped the data using an extraction table (Appendix 1). The MMAT grid (Appendix 2)

was used to appraise the quality of study methods included in systematic and scoping reviews [START_REF] Hong | Mixed Methods Appraisal Tool (MMAT). User guide[END_REF]. This assessment was done by one member of the team.

The analysis of the selected articles and the identification of analytical categories (see Results section) was carried out by the first author, and subsequently refined by each of the different co-authors, and discussed in group discussions by the whole team.

Results

Within the 24 references selected, there are 20 peer-reviewed journal articles and four book chapters. The articles have been published in peer-reviewed journals specializing in health, management, or public administration, and in generalist interdisciplinary or disciplinary journals (Figure 2) income country. Two references investigating the work of consulting firms with international organizations are not included on the map [START_REF] Momani | Professional management consultants in transnational governance[END_REF][START_REF] Tchiombiano | Santé publique, approche privée : Le Fonds mondial et la participation des acteurs privés à la santé mondiale. Face à face[END_REF]. The majority of the first authors were affiliated or associated with North American and Western

European universities (n = 18), including almost half of them being affiliated with British universities. A total of nine women and ten men were the first authors. Of the multi-author publications (n = 14), some were the result of repeated collaborations (Begley and Sheard, 2019;Begley and Sheard, 2021;[START_REF] Kirkpatrick | The impact of management consultants on public service efficiency[END_REF][START_REF] Sturdy | The management consultancy effect: Demand inflation and its consequences in the sourcing of external knowledge[END_REF]. Only two references related specifically to the work of consulting firms in the context of the COVID-19 pandemic (Atkinson et al., 2020;[START_REF] Untermaier-Kerléo | Un regard juridique sur le recours par l'État à un cabinet de conseil pour l'élaboration de sa stratégie vaccinale[END_REF]. The others focus on health policy analysis and public health reforms. This finding confirms the scarcity of work on the issue of epidemics before the emergence of COVID-19.

Methods of investigation and analysis

Most studies used qualitative research methods (n = 21), such as analysis of archives and interview administration (n = 4), ethnographic observations combined with interviews (n = 8).

Interviews only were also used (n = 4) in combination with grey literature (n = 1). Grey literature as the only source of data was also used (n = 4). Two studies employed a longitudinal quantitative method, while one research used a mixed quantitative and qualitative methodology that combined ethnographic and prosopographic surveys, which is a method of statistical analysis of the biographies of a social group. Methods for data analysis included thematic content analysis [START_REF] Paillé | Chapitre 11 -L'analyse thématique[END_REF] and semantic or qualitative discourse analysis [START_REF] Seignour | Méthode d'analyse des discours. L'exemple de l'allocution d'un dirigeant d'entreprise publique[END_REF]. The selected studies had some limitations. In particular, the MMAT tool (Figure 4) identified a lack of transparency on the methods (n = 4).

Analytical categories to study relationship between the state and consulting firms

Through our analysis of the 24 references, we identified three analytical categories used by the authors to study their object, which we conceptualized as follows: the management approach, the consultocracy phenomenon, and the phenomenon of elite hybridization (see Table 2 for details). By analytical categories, we mean ways of considering the relationship between public and private actions, the market and the state, and the processes of production and implementation of public policies [START_REF] Hassenteufel | Chapitre 5 -L'action publique comme construction collective d'acteurs en interaction[END_REF]Weible and Sabatier, 2017). These categories determine the ways consulting firm's roles are analyzed by the authors. They also reveal the theoretical approach, the conceptual tools underlying analyses, and the authors'

First analytical category: the management approach

Authors who used the management approach considered the intervention of consulting firms with public institutions as a resource or expertise to improve public policy. This approach is based on the idea that management pertains to technical expertise that improves public or private organizations. These works aimed to uncover the conditions or 'factors of success' [START_REF] Wye | What do external consultants from private and not-for-profit companies offer healthcare commissioners? A qualitative study of knowledge exchange[END_REF] of the intervention of consultants with public administrations, without questioning it. Three studies adopted this approach. For example, they apply the concepts of 'change management' [START_REF] Kuna | Paradoxical Processes Impeding Public Management Reform Implementation: Perspectives of Consultants[END_REF] or 'knowledge management' [START_REF] Radnor | The role of management consultancy in implementing operations management in the public sector[END_REF] to define de role of consultants considered as beneficial for public administrations. These concepts are not specific to public policy, but they are also applied to the corporate world. Two studies in this category reflected the perspective of the consultants and their experiences of collaborating with public health institutions. This is the case, for example, of the work of of the consultant's intervention with the health administration: the consultants' ability to 'solve problems', the 'co-production of solutions', and the consultants' ability to produce a joint analysis of the results of their intervention (Wye et al., 2015: 7).

Second analytical category: the consultocracy phenomenon

The consultocracy phenomenon brings together works that considered the role of consulting firms in public health in terms of influence and power. These studies used other research that considered the crucial role of these private actors in governing contemporary societies [START_REF] Guttman | The Shadow Government: The Government's Multi-Billion-Dollar Giveaway of Its Decision-Making Powers to Private Management Consultants[END_REF][START_REF] Howlett | Making the invisible public service visible? Exploring data on the supply of policy and management consultancies in Canada[END_REF][START_REF] Lapsley | Transforming the public sector: management consultants as agents of change[END_REF]. We owe the term 'consultocracy' to the two political scientists Christopher Hood and Michael Jackson. They used this concept to qualify the involvement of management consultants in the public administrations of several English-speaking states [START_REF] Hood | Administrative Argument[END_REF]. While the selected studies did not systematically refer specifically to this concept, all of them critically considered the role of private consulting firms in contemporary democracies. The action of these firms would participate in the phenomenon of state withdrawal. Of the selected studies, nine adopted this analytical category; the most common discipline of their authors was the management sciences.

The concepts used by these authors reflected this idea of the preponderant influence of firms on public policy. Authors of works in this category employed the following concepts and relevant expressions: 'multidimensional power' [START_REF] Horrocks | Experts' and E-government. Power, influence and the capture of a policy domain in the UK[END_REF], 'performative seduction' [START_REF] Ford | Performative seduction: how management consultants influence practices of leadership[END_REF], 'strategic actor' [START_REF] Lalonde | Dramaturgical awareness of consultants through the rhetoric and rituals of cooperation[END_REF], or 'agency' [START_REF] Lalonde | Information asymmetry in process consultation: An empirical research on leader-client/consultant relationship in healthcare organizations[END_REF] to analyze the multiple forms of power used by firms to influence public agents and administrations. For example, Lalonde and Adler analyzed the evolution of information asymmetries in the consulting/client relationship within health organizations in Canada. The authors showed how this asymmetry evolved between the actors according to the stage of the consultation process, even though the said asymmetry remained more favorable to consultants who reportedly took turns in the roles of 'doctor, facilitator, and expert' (Lalonde and Adler, 2015: 194). Through the case study of the institutionalization of the concept of talent management within the National Health Service (NHS) in the United Kingdom in the early 2000s, Ford and Harding showed the 'hidden power' of management consultants, which proceeded in two stages: first, the creation of this expertise as a 'product' for the firms' clients, and second, the practices of 'seductive performativity' of consultants in the 'closed doors' of administrations (Ford and Harding, 2021: 230). Consulting firms' strategy of making themselves indispensable was present in other studies [START_REF] Lalonde | Dramaturgical awareness of consultants through the rhetoric and rituals of cooperation[END_REF]. It was reflected in Momani's analyses of consulting firms' work with different international organizations. The author showed that firms positioned themselves as the only holders and analyzers of global knowledge and data related to the areas of interest they were solicited for.

The promotion of a technical expertise presented to public actors by consulting firms as unique and essential was one of the elements used by the author to explain their growing influence.

Such expertise was then presented as being at the origin of the sparking phenomena of empowerment of consulting firms in different sectors and public organizations.

Finally, two studies focused on the impact of consulting firms' intervention on the effectiveness/efficiency or the performance of public health policy [START_REF] Kirkpatrick | The impact of management consultants on public service efficiency[END_REF][START_REF] Sturdy | The management consultancy effect: Demand inflation and its consequences in the sourcing of external knowledge[END_REF]. Using longitudinal quantitative methods, authors of these studies showed the correlation between the increase in the use of consulting firms and the adoption of 'commercialization practices' which refers to the externalization of non-core clinical services and private financing (Sturdy et al., 2020: 492). The authors of these works also showed 'that higher levels of spending on management consultants have had a statistically negative effect on two separate indicators of efficiency ('operational efficiency' and 'adjusted cost efficiency')'

in the hospitals studied (Kirkpatrick et al., 2019: 90). The increased dependence of public organizations on consulting firms was also highlighted. However, unlike the other references included in this category, public servants were not considered passive actors merely exposed to the presence and intervention of firms. Despite this nuance, most of these studies highlighted the increasing dependence of public administrations on consulting firms, as well as a form of submission, if not impotence, to the firms' practices and power. Authors of selected papers highlighted the major role these firms played in public health policies. These studies also showed how consulting firms participated in implementing New Public Management practices in public health administrations and public health policies, as well as in normalizing an austerity economics and state's diminishing role (Hurl, 2018: 58) in the various contexts studied.

Third analytical category: the phenomenon of elite hybridization

The studies corresponding to this third category analyzed the making of public policy by using a relational perspective, which breaks the boundaries between the state and the market. To do that, authors of these studies emphasized the trajectories of public policy actors, their changing positionality between the public and private worlds, and the proximity of their social paths and positions. Consulting firms and administrations are considered as interdependent spaces where hybrid elites are positioned at the crossroads of these two worlds.

The nine studies mobilized concepts that underline this relational perspective. Concepts and relevant expressions associated with this category included, for example, the notion of 'hospital intermediaries' to define consultants1 who are, with others actors, positioned between administrations and caregivers and who play a role of translators and transmitters of norms (Belorgey, 2010). The notions of 'management state', 'loss of autonomy of bureaucracies' [START_REF] Frederic | Le mandarin, le gestionnaire et le consultant. Le tournant néolibéral de la politique hospitalière[END_REF], 'governance partners' [START_REF] Kirkpatrick | Management Consulting in Health[END_REF], or 'deprofessionalization of public officials' [START_REF] Belorgey | Une « consultocratie » hospitalière ?[END_REF]) also illustrated the blurring of the boundaries between the private and public worlds.

The first type of studies were history studies. Begley, Sheard, and Weiss focused on tracing the history of consulting firms working with health authorities. Their study showed the anteriority of relations between consulting firms and public administrations of the advent of so-called neoliberal policies in the 1980s. The authors were particularly interested in the relationship between Mc Kinsey & Company and the British NHS. Their analyses revealed the structural interdependencies between the two organizations as early as the 1940s (Begley and Sheard, 2021) but also the sharing by NHS agents of a vision of management consulting as conducive of public service improvement [START_REF] Weiss | Management Consultancy and the British State: A Historical Analysis since 1960. Management Consultancy and the British State: A Historical Analysis Since[END_REF]. Weiss thus considered the firms' consultants under the service of the 'administrative power'; hence the author suggested that private advice was a 'legitimate source of expertise' and objectivity (Weiss, 2018: 137). Weiss showed that these collaborations contributed to the rise of a modern 'hybrid state', 'where the lines of public and private sectors are blurred, and agents of both sectors act in the delivery of public services' (Weiss, 2018: 15).

This category includes a second strand of work that investigated consulting firms in the wider space of actors involved in public health policymaking. These studies examined the practices and the role of consulting firms in the competition and alliance relationship with other actors involved. To do that, Belorgey analyzed the position of consultants among the 'galaxy of hospital intermediaries' involved in the implementation of the 2009 French hospital reform. He showed how their 'composite [professional] capital' tended to make them special, and positioned them at a 'high degree of reticular centrality' compared to other intermediaries like department heads, hospital directors, health executives or engineers (Belorgey, 2020: 418).

Belorgey analyzed the consulting firms' expertise as a key resource to legitimize themselves in hospital reforms processes. In an article co-written with Pierru a few years earlier, the authors used this approach by showing how consulting firms participated in the implementation of the New Public Management and acted as 'essential actors in the reorganization of the health system' (Belorgey and Pierru, 2017: 45). The authors nevertheless nuanced the consultants' role in two ways. On the one hand, Belorgey and Pierru showed that the consultants remained 'at the service of the programming elites' within public administrations. On the other hand, the authors highlighted how the omnipresence of consultants was a 'manifestation of the growing hybridization of political, technocratic, and economic elites' (Belorgey and Pierru, 2017: 57).

In relation to the same 2009 French hospital reform, Pierru also showed how the extensive use of consulting firms is part of a larger phenomenon of 'bureaucratization', which is characterized by the hybridization of private and public organizational models. The author positioned this phenomenon in the broader 'neoliberal' process of the strengthening of a 'managerialized state' (Frédéric Pierru, 2012: 34). While strengthening administrative and management teams, such process did not, however, completely remove 'the autonomy of the medical field' (Frédéric Pierru, 2012: 50). Building from the adoption of the accrual method of accounting in the Norwegian Health Administration in the early 2000s, Kaarbøe and Robbestad considered private consulting firms as 'change agents' stakeholders in competitive and alliance positions with other agents (like politicians, agents of the Norwegian Ministry of Health, directors of health care departments, other consultants etc.) to legitimize their expertise. The consultants of these firms were 'institutionalized' actors, positioned 'at the center of the executive machinery' because of their double proximity with agents at national and local levels (Kaarbøe and Robbestad, 2016: 402). Similarly, Kirkpatrick and colleagues considered consultants, not as providers outside public policy but rather as 'partners in government' [START_REF] Kirkpatrick | Management Consulting in Health[END_REF] 2). For this purpose, the authors studied the specific phenomenon of 'revolving door', i.e., the professional roundtrips between the private and public worlds. By analyzing the social proximity between consulting firm' consultants and public administration agents, they showed how their collaborations 'have contributed to the privatization agenda for public services' and also to the transformation of consultants 'into somewhat less 'private' and more 'public' actors' (Kirkpatrick et al., 2016: 530). In other words, they have contributed to implement the business approach to public policy, which is now relatively common in public administrations worldwide.

threat to democracy. Authors of these studies made a normative judgment on the intervention of these firms and essentialized the boundaries between the public and private domains. For example, Authors who adopt the management approach considered New Public Management through the action of consulting firms as "the best way to govern well" (Peters, 2019: 411). As 'government sciences', management sciences adopt a normative and prescriptive dimension and aim to inform and guide public policy [START_REF] Ihl | Pour une sociologie historique des sciences de gouvernement[END_REF][START_REF] Ihl | Sciences de gouvernement[END_REF]. In addition, the concepts used by scholars in management sciences were also used by management consultants, thus recalling the fuzzy boundaries between management studies and the professional world of management [START_REF] Boussard | Sociologie de la gestion: Les faiseurs de performances[END_REF].

This normative dimension was also present in political science studies that mobilized the consultocracy phenomenon. Authors using this analytical category tended to consider the action of consulting firms as a takeover of the public administration. Yet the ways they conceived public problems often owed to the research paradigm of symbolic interactionist, an approach derived from American sociology which is based on the observation of inter-individual relations [START_REF] Breton | L'interactionnisme symbolique[END_REF]. These studies seldom considered how more structural dynamics (political reforms, people's social position, etc.) participated in the decision to use consulting firms.

The references pertaining to the category of elite hybridization were at the crossroads of several subfields of social science research such as the sociology of the elites, public policy analysis, and the historical sociology of the state [START_REF] Bezes | Sociologie de l'État et politiques publiques[END_REF]. Several of these subfields are close to French political sociology works, which have criticized normative and essentializing bias described above (Halpern et al., 2018). While criticism could be made of the neoliberal orientation of contemporary public policy, the rigorous empirical research methods used, and the analysis of the key actors' trajectories made it possible to be as close as possible to reality (Olivier de [START_REF] De Sardan | La violence faite aux données[END_REF]. For this reason, the analytical category of elite hybridization provided a promising heuristic approach to study the intervention of consulting firms in public health policy. Indeed, analyzing individual trajectories enables scholars to closely follow the professional circulations of actors. Such analysis also helps reveal the connections built throughout these trajectories within diverse public policy arenas. This analysis of individual career trajectories can also be found in global health works, which explain the action of 'diffusion entrepreneurs' in the wide spread of health financing policies in sub-Saharan Africa that are in line with the New Public Management [START_REF] Deville | Les mutuelles de santé subventionnées comme instruments de la Couverture Maladie Universelle au Sénégal[END_REF][START_REF] Gautier | From ideas to policymaking: the political economy of the diffusion of performance-based financing at the global, continental, and national levels[END_REF]. For example, in the case of health systems performance-based financing, the initial training (in economics and medical sciences) and 'founding' experiences with certain international organizations that piloted the premises of this policy (Médecins sans frontières and Belgian and Dutch private consultants' firms, in particular) was key. This experience helped the entrepreneurs to consolidate the policy's underlying systems of representations of transnational actors that will commit to defend this financing policy on the long-run. Building from their own systems of representations, they develop the theoretical foundations of this policy and use their knowledge, experiences, and professional networks to diffuse the policy, by going through different institutions (universities, international organizations, NGOs, consulting firms, and private training agencies) [START_REF] Gautier | From ideas to policymaking: the political economy of the diffusion of performance-based financing at the global, continental, and national levels[END_REF]. These transfers from the public to the private sector are subsequently reproduced by the 'African experts' who are trained and convinced of the relevance of this policy. These 'national' experts themselves become diffusion entrepreneurs.

In addition, the phenomena of 'revolving door' occur in professions other than those of consulting, especially in the pharmaceutical industry [START_REF] Gautier | Health for all" and the challenges for pharmaceutical policies: A critical interpretive over 40 years[END_REF], European lobbying [START_REF] Laurens | Les courtiers du capitalisme: milieux d'affaires et bureaucrates à Bruxelles[END_REF], or business lawyers [START_REF] France | Sphère publique, intérêts privés. Enquête sur un grand brouillage[END_REF], to name a few. In the case of the pharmaceutical industry and the regulation of pharmaceutical products in the United States of America, professionals frequently move from pharmaceutical companies to public regulatory agencies in the pharmaceutical sector (the Food and Drug Administration, in particular), and vice versa. Moving across these spheres of government facilitates the dissemination of the idea that the pharmaceutical industry is indeed good for U.S. economic interests, and that it should therefore be regulated as little as possible [START_REF] Davis | Unhealthy Pharmaceutical Regulation: Innovation[END_REF].

The diachronic perspective which we identified in most of the studies in the elite hybridization category, finally invited us to question the novelty of the intervention of consulting firms in public health policy. It also invited us to ask how this phenomenon has emerged and has evolved over time, depending upon diverse and dynamic political spaces.

Conclusion

This exploratory review identified three categories to analyze the intervention of private consulting firms in public health management. However, the scarcity of work in the field of epidemics and public health crises led us to formulate the hypothesis that this field remains a blind spot of academic research, which should be filled. The complexity of carrying out empirical studies on this topic often considered controversial and affecting strategic and commercial issues certainly explains such scarcity [START_REF] Daho | Les territoires du secret: Confidentialité et enquête dans les mondes pluriels de la sécurité[END_REF].

Bibliography

Figure 1 .

 1 Figure 1. PRISMA-SrC flow diagram[START_REF] Page | The PRISMA 2020 statement: an updated guideline for reporting systematic reviews[END_REF]

Figure 2 .

 2 Figure 2. Scientific journals of the selected articles

Figure 3 .

 3 Figure 3. Geographical distribution of selected case studies

Figure 4 .

 4 Figure 4. Quality appraisal of the selected studies' methods

  can be used to answer questions The qualitative approach is appropriate Data collection methods are adequate The results are adequately derived from the data The interpretation of the results is sufficiently substantiated Consistency between source, collection, analysis and… Representative sample Appropriate Effects and Intervention Measures Effects data are complete Confounding factors are taken into account The rationale for the method is adequate The integration of the various components is justified Integration is adequately interpreted Differences between methods are justifiedThe components adhere to the quality criteria of the or subfields. Three references did not fit into the proposed analytical categories because of their research methods, their authors' discipline, or the research question.

  Radnor and O'Mahoney[START_REF] Radnor | The role of management consultancy in implementing operations management in the public sector[END_REF] who analyzed the differences in methods and functioning used by the consultants compared to those of the agents of public administrations. The authors produced recommendations to improve the intervention of management firms in the public domain. The prescriptive aspect of this approach is also reflected in the work of Wye and colleagues[START_REF] Wye | What do external consultants from private and not-for-profit companies offer healthcare commissioners? A qualitative study of knowledge exchange[END_REF], based on the 2012 case study of the Health and Social Care Act in the United Kingdom. They identify three 'success factors'

Table 1 .

 1 Search strategy

			Scopus -Web of Science -Embase -PubMed
	Concepts	Consulting firms	Epidemic/infectious and tropical diseases	International and national (semi) public institutions
			health ; epidemi ; pandemic ;
			outbreak ; infectious disease ;
			communicable disease; tropical
			disease ; virus ; covid ;
			coronavirus; h1n1 ; aids; ebola ;
			dengue ; sars ; influenza ;
	Keywords	service firm ; consulting firm ; management consult ; big four ; professional service firm AND NOT financial service firm	measles ; chikungunya ; hepatitis ; opioid ; malaria ; gastroenteritis ; fever ; ulcer ; rabies ; trachoma ; syphilis ; leishmaniasis ; dracunculiasis ; trematodosis ; onchocerciasis ; schistosomiasis ; geohelminthiasis ; filariasis ;
			echinococcosis ; fascioliasis ;
			cysticercosis ; chagas ;
			trypanosomiasis ; yaws ; leprosy ;	s
			zika ; tuberculosis ; meningitis ;	directorate for health and food safety ; european centre for disease
			polio; poliomyelitis; cholera ;	prevention and control ; ecdc ; dg santé ; direction générale de la
			encephalitis	santé et de la sécurité alimentaire
			Bielefield Academic Search Engine
	Concepts	Consulting firms	Epidemic/infectious and tropical diseases	International and national (semi) public institutions
		service firm ;	health ; epidemi ; pandemic ;	state ; governance ; government ; polic ; ministry of health ; public
		consulting firm ;	outbreak ; disease	sector; international agenc ; ppp
	Keywords	management consult ;
		big four ; professional
		service firm

state ; governance ; government ; polic ; ministry of health ; public sector; international agenc ; ppp ; care international ; the global fund to fight aids tuberculosis and malaria ; global fund ; gfatm ; gavi ; the vaccine alliance ; global polio eradication programme ; global alliance for tb drug development, gatbdd ; international aids vaccine initiative ; iavi ; international aids society ; international federation of red cross and red crescent societies ; ifrc ; joint united nations programme on hiv/aids ; unaids ; world health organization ; who ; médecins sans frontières ; msf ; the partnership for maternal newborn and child health ; pmnch ; roll back malaria partnership ; rbm ; special programme for research and training in tropical diseases ; safe injections global network ; sign ; stop tb partnership ; save the children ; unitaid ; united nations international children's fund ; unicef ; world bank ; medicines for malaria venture ; united states centers for disease control and prevention ; us cdc ; the us president's emergency plan for aids relief ; pepfar ; eu public health ; european commission'

Table 2 .

 2 The three analytical categories

	Non applicable

'multidimensional power'; 'performative seduction'; 'strategic actor'; 'agency', 'moral economy of austerity', 'cost/efficiency' 'hospital intermediaries'; '"management state'; 'loss of autonomy of bureaucracies'; 'governance partners'; 'de-

  FranceT he involvement of cabinets in public action is not new. T hey participate in the acclimation of the NPM (analyze 4 periods).HPST Law: cabinets are involved in the very development of reforms and not only in their implementation. Produces a " selfconsolidation logic". Consultants as " intermediaries of public action or brokers" at the crossroads of know-how & techniques, cumulation of caps and capital that makes them a " composite legitimacy" and unavoidable. Nuance their " political" weight: they remain under the dome of " programming elites" and " decisive actors in the de-differentiation of the public sphere".

	15 20	Weiss, T chiombia	2018 Book chapter. 2019 Face à face.					Chapter 3: Reorganising, 1970s:	Switzerland	Quali: archives,	Socio-history	70's: health	UK (NHS)	Use of Mc Kinsey in NHS reform of 74 for	Elite hybridization
		A.E. no, S.		T itle of the Regards sur la					T he 1974 National Health Service			interviews and grey	policy reform	management consulting (not present in
				book: santé					Reorganisation and McKinsey &			literature.	administration), to give perception of
	RefId First	Year Quote Management					Company			Country of	Methods	Concepts/	Subject and year Country of the objectivity. Sharing in administration a Main results	Analytical
		Author		consultancy and								publication	theories	case studied vision of the power of management to	category
				the British										improve public services. Consultants > at the
				State. A										service of the administrative power. Reform
				historical										that has led to the establishment of
				analysis since										management and consultants as a legitimate
				1960.										source of expertise in the field.
												References excluding scoping
	RefId First	Year Quote							Country of	Study design	Concepts/theories	T ype/year	Geographic	Main results	Approach
		Author		Newspaper	Volum	Num T itle		publication	Methods	epidemic or	scope
					e									time scope
	16	Untermaier-	2021 La semaine					Un regard juridique sur le recours	France	Legal texts	ND	2020: Public	France	T he law allows outsourcing to consulting	Other
		Kerléo, E.		juridique					par l'État à un cabinet de conseil				management of	firms when it is not a matter of sovereign
									pour l'élaboration de sa stratégie				Covid-19	functions and " essential tasks" of public
									vaccinale					officials. Revolving-up situations can be
														pursued around argument " illegal take of
														interest".
	17	Hurl, C.	2018 Innovation:	31	1		Operationalising austerity: T he	ND		Quali: grey litt &	" Moral Economy	2011:	Canada	Firms' techniques and programs that	Consultocracy
				T he European					role of transnational professional			interviews	of Austerity"	Evaluation of	" contrary potential policy options" for local
				Journal of					service firms in local government				municipal public	governments and produce a fragmentation of
				Social Sciences					restructuring					services	institutional knowledge and a division of
								3	Performative seduction: how management consultants influence leadership practices	International Quali: Discourse Analysis	T heory of performativity (including 1998-2008: Health policy reform health)	UK (NHS) labour that leads to a concentration of "Secret influence" of cabinets in two stages: expertise. Cabinets, increasingly, are traffic 1. constitution of their knowledge into essential products. 2. seductive agents of the moral economy of austerity.	Consultocracy
	5 18	Horrocks, I. N. Belorgey,	2009 Information communication & society "nouveau Enquête sur le sous pression. book: L'hôpital T itle of the 2010 Book chapters. /	12	/	1	Experts' and E-government Power, influence and the capture of a policy domain in the UK choses. » Les méthodes de acceptent un certain nombre de Chapter 4: « Il faut que les gens ambivalents faux croyants : des intermédiaires Chapter 3: Convertis, résistants, France UK	Quali: grey literature and interviews. Quali: ethnography	"Multidimensional " approach to power reforms intermediaries of actors and reforms Consultants as 2000's: hospital 60's and end of 2000: Health policy reform France	UK (NHS) of these intermediaries: their objective is to "E-Governement" as a lever for the taking consulting firms in the public sector. Concept On the methods and techniques of persuasion of power and institutionalisation of are in a position of competition. Chapter 4: and distinguish themselves from each other, public institutions at the highest level. consultants) have in common to legitimise "performativity" in the hidden corridors of Chapter 3: intermediaries (including Elite hybridization Consultocracy
				management				3	Ward Managers' Attitudes towards persuasion des intermédiaires	International Quali: interviews,	ND	1990: Hospital	of the "loop of power" at the origin of this Interactions between heads of services and 2 make the reform project acceptable (via UK (NHS)	Other
				public"					External Consultants in Ashworth,			observations	reform	institutionalisation and of the external groups produces criticism on both strategic work (close to manipulation) +
									a Special Hospital, 1992-1994				"consultocracy". sides (not specialists, no response to their communication + practices and techniques of
	6	Kaarbøe,	2016 Journal of		12		3	T he valuation of fixed assets in	UK		Quali: archives and	T ranslation of	2001-2008:	Netherlands " lead of change"). Consultants are in position of change agents expectations, relation to complacent	Elite hybridization
	19	K. Belorgey,	Accounting and 2017 Les T ribunes de 55	2		Norwegian health care: An Une « consultocratie »	France	interviews Quali: ethnography	models, public Socio-historical 2004-2009: Hospital reform	via 3 dimensions that allow the hierarchy + logic of self-promotion of Elite hybridization
		N.		Organizational la sante					account of change agent struggle hospitalière ?		& interviews.	sector arenas of approach. Public Hospital	implementation of their approach: lobbying external groups).
	14	Change Sturdy, A.J. 2020 Public					T he management consultancy		USA		Quanti:	action	accounting & Process approach: Reform (HPST ) 2009-2013:	UK (NHS)	of the central admin, audit of health Show the logic of self-feeding or inflation of	Consultocracy
				Administration				effect: Demand inflation and its			longitudinal	framing theory "vicious circle of intermediaries	Hospital reform	structures and education/training private council demand by the administration
	7	Kirkpatric	2019 Policy and		47		1	T he impact of management consequences in the sourcing of	UK		Quanti:	Cost/efficiency addiction"	2009/2012:	UK (NHS)	Strong association between consultancy + correlation between high level of recourse	Consultocracy
		k, I.		Politics					consultants on public service external knowledge				longitudinal	approach	Health policy	spending and organizational (in)efficiency. to cabinets and adoption of so-called
									efficiency					reform	Use of consultants not statistically associated "marketing" practices/policies. T he
														with improving the effectiveness of public inflation of the use of firms has negative
														action effects on the performance of services/admin
														clients.
														29
	Arksey H and O'Malley L (2005) Scoping studies: towards a methodological framework.
		International Journal of Social Research Methodology 8(1). Routledge: 19-32.
	Aron M and Miguel-Aguirre C (2021) Des milliards dépensés pour se substituer à l'Etat :
		enquête	sur		la			République	des		consultants.	Available	at:
		https://www.nouvelobs.com/politique/20210627.OBS45815/des-milliards-depenses-
		pour-se-substituer-a-l-etat-enquete-sur-la-republique-des-consultants.html (accessed 17
		November 2021).
	Atkinson P, Gobat N, Lant S, et al. (2020) Understanding the policy dynamics of COVID-19
		in the UK: Early findings from interviews with policy makers and health care
		professionals. Social Science & Medicine 266: 113423.
	Begley P and Sheard S (2019) McKinsey and the 'Tripartite monster': The role of management
		consultants in the 1974 NHS reorganisation. Medical History 63(4): 390-410.
	Begley P and Sheard S (2021) From "honeymoon period" to "stable marriage": The rise of
		management consultants in british health policymaking. Bulletin of the History of
		Medicine 95(2): 227-255.
	Belorgey N (2010) L'hôpital sous pression. Paris: La Découverte.

In this research, "consultants" could be professionals from consulting firms but also individuals with different expertises who propose their services to hospital managers.

IRD, for her advice and support in the formulation of the protocol of the scoping review. We would also like to thank Heather Hickey for translating a first version of the paper, Flore-Apolline Roy for producing figure 2 and the LIEPP team for the proofreading of this work. The authors disclosed receipt of the following financial support for the research: This work was supported by French National Research Agency as part of the "Investissements d'Avenir" program LIEPP, GRIP and the Paris University IdEx. [Grants number: ANR-11-LABX-0091; ANR-11-IDEX-0005-02; ANR-18-IDEX-0001]

Conflict of Interest statement:

The authors declare no conflict of interests in relation to the contents presented in the manuscript.

References excluding categories

The work of Atkinson and colleagues (Atkinson et al., 2020) focused on the receptiveness of the UK's national COVID-19 policy by healthcare professionals and local health officials. The authors showed how these groups acting on the front lines had fuzzy interactions with centralized public policies. Atkinson and colleagues also investigated these groups' perception of private consulting firms. The role of consulting firms in public policy was less prominent in this study. The study by [START_REF] Richman | Ward Managers' Attitudes towards External Consultants in Ashworth, a Special Hospital, 1992-1994[END_REF] reflected on the experience of using management consulting firms and a task force of healthcare professionals by the administration of the Special Hospital of Ashworth in England between 1992 and 1994.

The authors analyzed the strong criticism of the head of the institution's services of these two groups (the consulting firms 'consultants and the task force of healthcare professionals).

Nevertheless, their work was overly descriptive; it did not make it possible to identify the conceptual and analytical orientations.

Finally, starting from the controversy over the massive use of private consulting firms during the public management of COVID-19 in France, Untermaier-Kerléo explored the legal texts allowing the use of consulting firms. The author identified two 'legal risks related to outsourcing': administrative 'loss of competence' and conflict of interest (Untermaier-Kerléo, 2021: 3). The study also was very descriptive and difficult to link to any of the above-mentioned approaches.

Discussion

The analytical categories used to study the role of consulting firms in public health action, in addition to being diverse, depended on the disciplinary or sub-disciplinary traditions of their authors. For most of the work in management sciences, consulting firms were seen as either a technical device at the service of what is perceived as an improvement of public policy or as a as intended?

3.1. Is there an adequate rationale for using a mixed methods design to address the research question?