Nathalie Fabbe-Costes
email: fabbe-costes@univ-amu.fr

The power of co-construction and dialogue to interact with industry in strategic logistics, OM and SCM research projects

Keywords: research project management, industry-academia interactions, strategics logistics, OM and SCM

The purpose of this paper is to study industry-academia interactions in strategic logistics, OM and SCM research projects. The paper is based on an in-depth retrospective analysis of eight research projects performed in collaboration with industry. Methodology combines single and cross-case analysis. Results point out the importance of industry-academia dialogue and co-construction at each step of a research project. They provide some guidelines to improve the management of such projects and the quality of their outputs for both parties. We conclude by suggesting some issues to deepen dialogue and enrich co-construction in strategic logistics, OM and SCM research projects.

Introduction

Most academics consider that research in logistics, OM and SCM is useful for companies, and almost every research work sells its so-called managerial contribution. But few academic papers seem to rely on genuine industry-academic collaboration. One should not conclude too quickly from a literature review based on academic journals that such research does not exist or is rare (referred as the "white space" in Frankel et al. 2005). This only shows that despite the call for a greater research variety, in particular more field research [START_REF] Dehoratius | Field research in operations and supply chain management[END_REF], logistics, OM and SCM literature remains mainly positivistic and rationalist [START_REF] Näslund | Logistics needs qualitative research -especially action research[END_REF], and, looking for more scientific rigor, seems to have lost its connection with practice [START_REF] Touboulic | A relational, transformative and engaged approach to sustainable supply chain management: The potential of action research[END_REF].

Nevertheless, for politic, economic and pragmatic reasons, this kind of research is developing. Most funded calls for research (ex: H2020) put pressure on having companies in consortia and producing more value for industry and society. Companies look for external expertise (alternative from consultancy) and diversify the partners participating in their open innovation processes. Academics look for "problems" with practical relevance combined with funding… But collaborating with industry is not so easy and many academics experience difficulties related to the conflicting logics behind such type of collaboration (some of them being identified in action research, e.g. [START_REF] Coughlan | Action research for operations management[END_REF].

In line with the theme of the call for abstracts of EUROMA 2016, the purpose of the paper is to answer the following question: how can we improve industry-academic interactions to gain both in managerial, societal, and academic relevance and value, i.e. being useful and creating value for companies and society, as well as being valuable from an academic point of view with "publishable" material and innovative theoretical results?

Surprisingly, there are very few papers in logistics, OM or SCM literature that detail the way academics and practitioners interact, or should interact, with 'industry' (in a broad sense of this term, i.e. not restricted to industrial companies) in research projects. The ones that do are mainly papers that discuss or are based on action research (e.g. Naslund et al., 2010;[START_REF] Touboulic | A relational, transformative and engaged approach to sustainable supply chain management: The potential of action research[END_REF] and case study methodology (e.g. [START_REF] Stuart | Effective case research in operations management: a process perspective[END_REF][START_REF] Voss | Case research in operations management[END_REF][START_REF] Dubois | Systematic combining: an abductive approach to case research[END_REF]. In most cases, the papers describe some aspects of the interactions with respect to the context of the research (to clarify the relationship with the field in the selected methodology), to explain how the researcher(s) gathered the data (to give credibility to data quality), or to attest of difficulties in the research process. To our knowledge, the nature of reported interactions is mainly 'technical' and does not reveal the dynamic of interactions along the research project.

As a member of a research centre specialized in strategic logistics, OM and SCM, with a tradition of research performed in close collaboration with industry, and having a long experience in such research projects, the purpose of this paper is to report about industry-academia interactions in achieved research projects and to discuss about the importance of industry-academia dialogue and co-construction. We first present the research design for this paper. Then, we detail the results of the retrospective analysis of eight research projects. We discuss the importance of industry-academia dialogue and co-construction before concluding the paper.

Research design -description of the eight research projects

The paper is based on a reflexive thinking considering past research works conducted in interaction with industry. Among the research projects we have been involved in, we selected finished projects that were, since the beginning, mixing academic and industry outcomes. Another criterion was to be able to report about the overall research project process. Eight projects were eligible from 2000 to 2015. All the research projects were related to strategic logistics, OM or SCM, i.e. concerned the relationships between logistics, OM and/or SCM and the overall strategy of companies or organisations.

Being part of the projects since the beginning of the discussion with industry partners, and having the traceability of the overall processes, it was thus possible to think back about: the construction of each project, goals and expected results for companies vs. researchers (ex ante), the overall management of the project (including the making of research choices), difficulties encountered during the research process, results for the companies vs. researchers (ex post), potential for academic publication (and effective published papers), experience gained about how to manage industryacademic interactions.

Each research project is thus, for this article, considered as a case study and is analysed retrospectively. For each research project, we also considered its characteristics (main topic, duration, number of researchers, funding…) and the explicit or implicit research choices (ontology, epistemology, methodology, reasoning mode). We combined single and cross-case analysis to end with results. Table 1 and 2 give an overview of the eight cases, showing a variety of research projects. For confidentiality reasons, the names of the companies, all having leadership positions, are not mentioned. (2) HSDM indicates if the projects, as far as we know, helped for strategic decision-making (HSDM).

Before giving an overview of the 8 projects, it must be explain why we chose projects performed after 2000. Following the calls for more rigorous logistics and OM research at the end of the 1990's (e.g. [START_REF] Ellram | The use of case study method in logistics research[END_REF], for case study in logistics), from 2000 we designed more carefully the research methodology of industry-academia research projects. The aim was clearly to improve robustness of research results and to have collected data with a quality that should not stop us from publishing in academic journals.

From the 8 projects, 5 were initiated by industry, 3 by academia. Six projects involved only one company and two project more than one: P1 was a jointed project with two companies and P6 concerned a group of companies (members of the same professional association). In both case (P1 and P6), the companies the industry partners before launching the project. The research projects were in interaction with different kind of supply chain (SC) members: 4 with LSPs (P1, P3, P4, P8), 2 with industrial companies (P5, P7), one with a wholesaler-distributor (P2), and P6 with a mix of industry, logistics, distribution companies.

Most of the projects mixed logistics, OM and/or SCM issues. As mentioned by Larson and Halldorsson (2004, p.18) "The unclear conceptual boundaries of SCM make it difficult to design educational and research programmes in SCM without large overlap with other fields such as logistics, marketing, operations management and purchasing". It is thus not easy to delineate the precise domain of each project.

What was clear is that every project had strong strategic issues. Most of the projects (5 out of 8) had a prospective / scanning orientation (P1, P5, P6, P7, P8), and had a direct connection with companies' strategy formulation (P1, P2, P3, P4, P7). Most projects permitted to think both about relationships between logistics strategy (lS) and strategic Logistics (sL) [START_REF] Fabbe-Costes | The 'white space' of logistics research a look at the role of methods usage[END_REF].

Two projects (P2, P8) were short ones (3 months), most projects were longer than one year (P1, P3, P4, P6, P7), P5 being intermediary (8 months).

Among the 8 projects, one can be considered as action-research (P1), 3 as 'classic' qualitative and/or quantitative research projects (P3, P4, P6), and the 4 remaining as knowledge sharing (P2, P5, P7, P8) with a mix of research production (state-of-the art, results transfer and prospective).

Concerning outputs, it can be noticed from table 1 and 2 that the 8 projects produced outputs generally in line with expected results. Some project can be considered as being more 'valuable' for companies, others for researchers. Almost every project (except P7 and P8 that are the more recent ones) leads directly or indirectly to publications (conference papers -most of them at Euroma or Nofoma conferences -and articles in peer-reviewed OM and SCM academic journals), sometimes not immediately after the research, not using the collected data and not directly related to expected results.

Looking back at the outputs for researchers, this type of projects have long-lasting effects and strong impact the overall researchers' agenda, probably because of the richness of the structured and, more importantly, the unstructured data researchers collect and the many ideas and questions raised during the course of the project. Most of the 'revelatory' data [START_REF] Yin | Case Study Research. Design and Methods[END_REF] could not be published, which is somehow frustrating, but enriched researchers' knowledge of strategic logistics, OM and SCM thinking and practices, orienting further research topics.

Findings related to industry-academia interactions

The in-depth analysis of the 8 research processes provides insights about industryacademic interactions in logistics, OM and SCM strategizing and organizing research projects [START_REF] Whittington | The work of strategizing and organizing: for a practice perspective[END_REF]. Table 3 presents main results for the single case analysis. Important to be able to communicate about complex concepts (sometimes adapting language) Useful to make company validate the 'production'

Important to dialogue about the understanding of key concepts (e.g. scanning, eco-system, innovation…)

Table 4 presents main results of the cross case analysis pointing out the results concerning dialogue and co-constructions at each step of a research project as well as the success factors facilitating both. As mentioned before, for the 8 selected projects, we were involved from the beginning of the contract negotiation, we participated to methodology design, data collection and analysis, and we were associated to every key decision in the research processes.

Even if each project was initiated by one of the partners (see table 1), the general problematic revealed to be of strategic relevance both for academics and practitioners. The discussions during the first steps of project researches helped in refining the research design in order to fit with the objectives of each party. In some projects, they were not fully aligned which was not a problem since what is shared is the project.

Working on refining together the research design, we often experienced the "unclear conceptual boundaries" between logistics, OM and SCM. The statement of Larson and Halldorsson (ib.) remains valid today. And this is true for every keyword of a research project (e.g. integration for P.3 and P4, scanning for P6 and P8, sustainable logistics and sustainable SCM for P5, P6 and P7). When researchers and practitioners work together it is worth checking they understand each other. If this is not done at the beginning of the project, the problem may occur later with incidence on the quality of the research. In-depth discussion of concepts (e.g. in P3, P8) fosters the sharing of visions improving interactions during the project and quality of its outputs. It is important, in particular when researchers initiate a project (P3, P4, P6), that the theoretical aspects do not hinder dialogue with companies. Each has to go towards the other.

When companies come to academia, in particular for those who are not used to work with researchers, they often look for 'best practices', ready-to-use or fast appropriable solutions. Underlining logic is there are 'best practice' (cf. ontology). Considering that strategic logistics, OM and SM projects needs a systemic approach and a reference to complexity thinking (in particular for scanning and prospective projects), some discussion were needed to check that the vision of industry partners was compatible (there is no 'one best way', one fits all approach). For those who were not so convinced at the beginning, but dared to make the project, they were happy to end with a more complex view of the initial problematic and news questions.

Interacting with companies, in particular when they are waiting for 'results' (e.g. P1), can lead to adopt 'weak' methodology (less robust than expected) to reach results quickly, but ending with data difficult to reuse for publications. Time can be an important issue vis-à-vis research quality. Academics' time horizons are generally much longer than practitioners, resulting sometimes in conflicts during the execution of the project. Common decision-making about such critical aspect of projects can prevent from dissatisfaction of both parties, leading sometimes to decide redirecting the project.

To interact about "strategic logistics and SCM" projects, it is necessary to have access to top managers (key informants). Since these projects include confidential information sharing, company needs to trust researchers; ethical behaviour is a key issue. Previous experiences with the partners were a success factor for P2, P3, P4, P5 and P7. Time is needed to convince partners to open the doors and share information.

For most projects, we worked under a contract on behalf of our university, specifying, among other points confidentiality and intellectual properties issues. What was considered at the beginning as a constraint, an administrative task and a waist of time, revealed much more important and useful that expected. The time (particularly long for some projects) spent to discuss about duration of the project, the expected results for both parties, the methodological choices, the different steps in the project and if the 'door is open' to redirection of the project, was not a "lost time". It was also important to state about what could be done afterwards in term of publications (an important point nowadays for researchers). Quality of collected data (P1), confidentiality issues (P2, P4, P7, P8) sometimes prevent from publishing results or data related to research project made in interaction with companies. However, publishing is not the unique value added of research projects performed in close collaboration with industry. More important sometimes is to have access to revelatory data (P5), to feed one's scanning process (P7), to test the relevance of knowledge for companies (P8).

Finally, looking back at the overall panel of projects we analysed, we realized how interconnected these projected were because of the topics (e.g. P3 and P4 with SCI) or because of the partnerships (e.g. P1 and P3). P1 led to P2; P4 was built upon P3; P6 incited companies to launch P7 and P8. The related partners acknowledged how important dialogue and co-construction were to build confidence and create a sense of intimacy, as well as to develop a know-how working together.

Discussion / contribution

The results of our case analysis give some highlights about industry-academia interactions in research projects. They provide some guidelines to improve the management of such projects and the quality of their outputs for both parties. The results suggest that 1) the co-construction of such type of research project is a critical key success factor; and that 2) to support this process, more industry-academic dialogue leads to more valuable results for both parties.

Co-construction helps partners to clarify and align expected outcomes, to co-design the research process (specifying the managers' and researchers' roles) and to collaborate in the research execution. The dialogue supporting the co-construction permits to show off controversial points (e.g. having fast results/having time to collect good set of data), or ontological and epistemological mismatches between researchers/managers visions, both being source of potential conflicts (e.g. about the 'reality' of best practices).

The dialogue can help researchers to refine the reasoning mode and the methodological choices, improving the feasibility and the rigor of the project (a key point for publishing action research, [START_REF] Näslund | Action research in supply chain management: A framework for relevant and rigorous research[END_REF]. Adopting a systemic approach, we design a framework of interactions between the key topics of dialogue identified in our set of research projects.

The paper provides guidelines about how to manage industry-academy interactions in strategic logistics, OM and SCM research projects. The research identifies a list of points (see table 3 and4) partners launching a common research project could consider and suggests to take time to dialogue, using the dialogic principle [START_REF] Shotter | Dialogism and polyphony in organizing theorizing in organisation studies: Action guiding anticipations and the continuous creation of novelty[END_REF], to coconstruct the project… not a wasted time considering differences between academic and practitioner (Bartuneck and Rynes, 2014).

In line with the underlying constructivism epistemology of our research [START_REF] Avenier | Shaping a Constructivist View of Organizational Design Science[END_REF], our paper does not give any 'best practice' but proposes principles for practicing research.

Conclusion

Some limits can be mentioned. We have not selected projects that failed at an early stage. Long negotiations for two important projects with leading companies broke down when the two parties were unable to agree on the resources (finance and personmonths), intellectual properties and confidentiality issues). Some lessons from these 'failures' could enrich results.

We did not include supervised PhD projects with high interaction with companies during the period 2000-2015. Since it is the PhD student who was interacting, it would be necessary to collect his/her point of view to be able to include these cases. This could be done when expanding the paper. Some more work is needed to develop a comprehensive framework and to discuss it in the light of previous work in organisation studies, expanding the discussion section of this paper.

Table 1 -

 1 About the eight cases of industry-academia research project

	N°	Research	Origin of	Objective for company	Objective for	Characteristics
		project	interaction	(ex ante)	researcher (ex ante)	of the project
		partner(s)	Domain
	P1	Two LSPs	Companies	Understand the	Build a research-	From 03-2004
		(LSP1 and		dynamic of health	based logistics/SCM-	to 06-2005
		LSP2)	Logistics	distribution in France,	oriented prospective	3 mains steps
			SCM	prospective analysis to	methodology	with collective
		Three		support LSPs' strategic		decision
		academics		expansion in this		Qualitative
				market		method
	P2	Health	Company	Answer the questions:	Deepen the study of	From 09-2005
		wholesaler		Is logistics for us a	logistics as a core	to 11-2005
		distributor	Logistics	"core competence"?	competence	1 step
				How can we build a	Study logistics /	Workshop
		One		strategic approach	strategy interactive	with top
		academic		based on logistics?	loop in their case	managers
	P3	PSL1	Researcher	Get a better	Explore a LSP's	From 10-2006
				understanding about	points of view about	to 06-2008
		One	SCM	SCI, clarify LSP's	its roles in SCI,	2 main steps
		academic		strategy and role in	compare to its	Qualitative
				SCI, understand	clients' point of view	method (2
				clients' view about	(alignment within	waves of
				LSP's role in SCI	dyads?)	interviews)
	P4	PSL	Researcher	Knowledge transfer	Explore a LSP's	From 10-2009
				about SCI, clarify their	points of view about	to 11-2010
		LSP3's	SCM	vision about their role	its role in SCI	1 main step
		role in SCI		in SCI and their	Confirm results of	Quantitative
		Two		strategy vis-à-vis their	P3, explore new	method +
		academics		clients	perspectives	focus group
	P5	Industrial	Company	Challenge their lean-	Discover a new	From 02-2010
		luxury		agile management	"industry" (for us)	to 09-2010
		company	OM-SCM	approach	Confront theoretical	2 steps
				Get prospective ideas	concepts and	Two
		Three		about SSCM, eco-	previous research	workshops
		academics		design and	results to a specific	and focus
				remanufacturing	context	groups
	P6	Group of	Researcher	Answer the questions:	Build a "general"	From 01-2012
		companies		Is a collective (inter-	SSC scanning	to 07-2013
		in	Sustainable	organisational)	framework, compare	3 steps
		consumer	SCM,	scanning process	scanning of different	Meetings, two
		goods	logistics	feasible? How to	actors in a SC,	focus groups,
			and	prioritise collective	question inter-	debriefing
		Two	transport	scanning targets?	organisational	with
		academics			scanning feasibility	companies
	P7	Industrial	Company	Challenge their strategy	Participate to	From 01-2013
		Company		related to sustainable	strategic thinking of a	to… (ongoing)
			SCM-OM-	development (related to	leading group	Prepare topics
		One	logistics	logistics, OM and SCM	Better understanding	and participate
		academic		issues)	of inter-functional	to a mixed
				Share knowledge about	interfaces in strategic	stakeholder
				SSCM issues	SSCM developments	panel every
					Strategic case study	year
	P8	LSP4	Company	Knowledge transfer	Translate theoretical	12-2015 to
				about logistics an SCM	concepts and	01-2016
		Two	SCM-OM	scanning activities	previous research	Conference +
		academics		Improve managers'	results to a specific	workshop with
				scanning behaviour	company	top managers

Table 2 -

 2 Ex post outputs of the research projects

	Project Output for practice	Output for academia	Publications (1)
	P1	Better understanding of	A how-how about making	No immediate publication
		health distribution	Logistics-SCM oriented	(confidentiality)
		dynamic and the potential	prospective studies	Generic results, combined
		role for LSPs	Discover important	with other experiences were
		Clarification of the	methodological issues	reused in 2 conf. papers
		business models they	The basis for a so-called	(2007), 1 article (2008), they
		could explore (alone or	"logistics intelligence	inspired SSCM scanning
		within an alliance)	process" (LIP) we reused	conf. paper (2009) + 2 book
		HSDM (2)	afterwards	chapters (2010, 2011) + 1
				article (2011) and project P6
	P2	Making company's top	Confirm some hypothesis	No publication about this
		managers understand the	related to health SC strategic	case (confidentiality)
		importance of its	changes (P1)
		organizational logistics	Refine what is a logistics	Generic results reused in a
		competence	core competence	book chapter (2007)
		A trigger to develop	Confirm how creative is the
		strategic logistics	logistics / strategic loop
	P3	Realize they had not a	In-depth understanding of	Results published in 1 conf.
		unified approach of SCI in	the paradoxical roles a LSP	paper (2007)
		the company	can play in SCI	+
		Clarify their vision of what	Clarifying the relationships	was the trigger for 2 conf.
		could be their SCI strategy	between LSP and its client	papers (2008), 2 articles
		and their role vis-à-vis	in SCI	(2008, 2009) and for project
		their clients' SCI	Distinguish between being	P4
		(HSDM)	integrated in a client SC and
			integrating the client SC
	P4	Realize they had not a	Confirm P3 results	Results published in 2 conf
		unified approach of SCI in	Open doors to news ideas	papers (2010) and 1 article
		the company	Projects P3+P4 permit to	(2011)
		A topic discussed at the	build a typology of LSP's
		highest level of decision	roles in SCI
		(executive committee)
	P5	New avenues for a	Have a 'revelatory' access to	No publication
		systemic approach of eco-	a world-class company	(confidentiality)
		design, after-sales services	Share very strategic	Inspired P7 and P8
		and reverse logistics	information, good case study
	P6	A collaborative research	Validation of the SSC	Results published in 2 conf
		project that led to test a	scanning framework	papers (2012, 2013) and 1
		"pilot" (up to end 2013)	Good qualitative data	article (2014)
		that turned into is an	Valuable strategic
		effective "shared-service"	information sharing
	P7	Benefit from clarification	Learn from the many voices	No publication
		of new concepts, synthesis	in the panel (not all being	(confidentiality)
		of state-of-the-art,	"scientific" ones)
		emerging research results,	Catch tendencies, new ideas
		prospective views	Share very up-to-date
			knowledge and signals
	P8	Incite managers to develop	Reuse of LIP (P1) +	No publication
		an open-minded and	scanning framework (P7)	(confidentiality)
		'complex' approach of	Know-how in knowledge
		scanning	transfer and production of
			appropriable knowledge by
			managers

Nota: (1) List of publications is available upon request. Research reports linked to the projects are not mentioned.

Table 3 -

 3 About the industry-academia interactions during projects

	Project Main difficulties / key	Lessons for researchers	About dialogue
		success factor		Co-construction
	P1	Strategic objectives of	Learning process about how to	No experience of working
		companies were not	interact with "unknown"	together
		clearly communicated	companies in action research	At the beginning, a lack of
		(considered as too	projects	trust between the 2
		confidential)	Time for the project (short) had	companies and between
		Difficult to decide about	impact on methodology (and	companies and academics
		research orientation at	data quality)	=> dialogue was not easy
		each step (companies were	It is necessary to design ex ante	Importance of time to
		reluctant to make choices)	a robust methodology to have	tame each other and dare
		Quality of results	publishable data	co-construct step by step
		convinced industry partner
	P2	The company's objective	Credibility is important	Company attended a
		was clear and focused	Working with top managers is	public presentation of
		Role of academia was	challenging but permits to have	some of the results of P1
		clear but challenging	access to key strategic data	=> researcher was
				credible and dialogue easy
	P3	Integration, SCI, were not	Clarifying concepts, getting sure	Being partners in P1,
		clear concepts for LSP1	that objectives are aligned is not	facilitated interactions and
		=> it was difficult to "sell"	a lost of time	the access to key
		the project	Access to key informant is a key	informants within LSP1
		After discussion, the main	success in qualitative research,	and its clients
		objective of both parties	in particular for topics related to	Since LSP1 "bought" the
		was to gain in-depth	strategic issues	project, no-construction
		understanding of what is	Robust, clear and detailed	Interesting dialogue about
		SCI, perceived as a	methodology helps "selling" a	the "surprising" results
		complex phenomena =>	project	(for both industry and
		ended aligned		academia)
	P4	Having contact with key	A project not time consuming	LSP3 was a teaching
		managers in the company	for the company is quite easy to	partner in master program
		was helpful to "sell" the	sell	=> dialogue was easy
		project	Importance of debriefing	Company was not so
		Demand was only to	strategic issues raised by results	interested in the results =>
		spread questionnaires to a	with the top management	no co-construction but
		select number of persons		challenging discussion
				about results
	P5	Confidentiality was the	It is important to overcome	To sign a strict contract
		main problem for	frustration not to be able to	concerning confidentiality
		researchers (impossible to	publish data	helps partners to be in an
		publish nor communicate	Understand that value of a	"open" knowledge sharing
		anything)	project can be in the experience	perspective
	P6	Coherent group of leading	Robust focus group method	Clear research objective
		companies developing	produced very good data	helps maintaining the
		collaborative sustainable	Frontier between research and	research focus of the
		logistics projects => easy	consulting needs to be regularly	researchers' role
		to manage	reaffirmed
	P7	Clear role for researchers:	A panel is not the best situation	Polyphony in a panel is
		stakeholder among others	to an in-depth case study	rich and raise lots of ideas
		in a stakeholder panel	The panel experience is	and questions, it helps
		A panel, it is not a	stimulating but topics are not	'surfing over the wave' of
		'scientific' context	deeply studied => to develop	new trends and tendencies
	P8	Important to get sure that
		we shared the vision of the
		expected outcome (not
		best practices)
		Clear demand from the
		chief executive

Table 4 -

 4 About the dialogue and co-construction during projects

	Project phases About dialogue content	About co-construction	Influencing factors
	First contact	Why launching and	At this stage partners	Reputation of partners
	up to "go"	participating in the	can influence research	Capacity to communicate
	decision	project?	orientation to align the	clearly about the project
		What about the topic, the	project with their	Ability to convince about
		potential methodology and	expectations, think	the project interest
		the expected outputs	about potential	Level of experience and
		Clarify the underlining	difficulties with	trust with the partner(s)
		concepts, theories	fieldwork, raise	Previous projects done
		Exchange can help	potential problems (e.g.	together
		finalizing the project,	related to	Inter-personal previous
		sometimes giving another	confidentiality, public	experience
		orientation	publications of results)	Easy access to decision-
				makers of both partners
	Contracting	Be sure that objectives are	Do not buy projects	Previous projects done
		clear (prevent from	unquestioningly, except	together (e.g. vis-à-vis
		misunderstandings)	if it fits your plans.	confidentiality, ethics)
		Clarify the details of the	The objective is to	Previous experience in
		research process: steps,	design a feasible and	contracting for research
		duration, work-packages,	valuable project for	projects
		deliveries, support and	both parties, getting	Templates available to
		finance from partners, role	sure that the project can	limit comings and goings
		of industry partners vs.	feed each partner's	Support of jurists to
		researchers, expected	perspective (not	analyse contractual issues
		results and publications	necessary aligned)	(e.g. risks)
	Making the	Data and knowledge	Depending on contract	Access to and availability
	research	sharing	agreement, precise	of key managers and
		Traceability of the	definition of field	senior researchers during
		research execution	research choices	the project
		Encountered difficulties,	Execution of research	Degree of partners'
		how to overcome them	together	'interest' in the results
		Negotiate about 'quick	Making common /	Research skills of partners
		production' of results	controversial analysis	Research know-how
	Direction -	Negotiate about	Taking together the	Research know-how
	redirection	unexpected but necessary	decision about new	Strategic milestones for
		redirections	orientations	partners (deadlines to
		Making decision about	Analysing their impacts	produce results, to take
		time constraints and delays		decisions)
	Results	Importance of debriefing	Some results cannot be	Critical vision of results
	analysis	strategic issues raised by	obtained without	Ability to stand back and
		results with the top	working hand-in-hand	think about what each
		management	Results can be	partner learned
		What are the "surprising"	challenged by partners
		results (for both industry
		and academia)?
	Concluding	Debriefing: what about	Making a common	Interest for research
	the project	satisfaction of partners?	return from experience	process and production
		What about the project?	about the project	Long-term vision of
		Think about 'next' step?	Learning process	partnering in research


Tom Cornebize

Arnaud Legrand

Franz C Heinrich

Fast and Faithful Performance Prediction of MPI Applications: the HPL Case Study

2 62 Resources (node hours) 12,000 62 q q q q q q q q q q q q q q 0 5000 10000 15000 20000 1e+05 2e+05 3e+05 4e+05 5e+05

Matrix rank

Gflop/s q Reality Prediction Figure 1: HPL performance on Dahu can be predicted within a few percents of reality using a stochastic heterogeneous model for dgemm and a simple model for all other kernels and the network Abstract-Finely tuning MPI applications (number of processes, granularity, collective operation algorithms, topology and process placement) is critical to obtain good performance on supercomputers. With a rising cost of modern supercomputers, running parallel applications at scale solely to optimize their performance is extremely expensive. Having inexpensive but faithful predictions of expected performance could be a great help for researchers and system administrators. The methodology we propose captures the complexity of adaptive applications by emulating the MPI code while skipping insignificant parts. We demonstrate its capability with High Performance Linpack (HPL), the benchmark used to rank supercomputers in the TOP500 and which requires a careful tuning. We explain (1) how we both extended the SimGrid's SMPI simulator and slightly modified the open-source version of HPL to allow a fast emulation on a single commodity server at the scale of a supercomputer and (2) how to model the different components (network, BLAS, . . . ) of the system. We show that a careful modeling of both spatial and temporal node variability allows us to obtain predictions within a few percents of real experiments (see Figure 1).

I. INTRODUCTION

Today, machines with 100,000 cores and more are common and several machines beyond the 1,000,000 cores mark are already in production. This high density of computation units requires a diligent optimization of application parameters, such as problem size, process organization or choice of algorithm, as these have a huge impact on load distribution and network utilization. Scientific application developers and users often spend a considerable amount of time and effort running their application at different scales solely to tweak parameters and therefore optimize their performance. Whenever actual performance does not match expectations, it can be very difficult to understand whether the mismatch originates from application misunderstanding or machine misconfiguration. This expenditure of time, combined with the power consumption that often reaches several MW, makes it financially expensive to test-run applications. Similar difficulties are encountered when (co-)designing supercomputers for specific applications. A large part of this tuning work could be simplified if a generic and faithful performance prediction tool was available. This article presents a decisive step in this direction.

In this article, we explain how to predict the performance of High-Performance Linpack (HPL) on a supercomputer with the SimGrid/SMPI [START_REF] Casanova | Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms[END_REF], [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF] simulator. HPL is used to rank twice a year the world's largest and fastest machines in the TOP500 [START_REF] Meuer | The TOP500: History, Trends, and Future Directions in High Performance Computing[END_REF] and requires both a careful tuning and a sizable input to get near to the peak performance. We detail how we obtained faithful models for several key functions (e.g., dgemm and dtrsm) and managed to reduce the memory consumption from more than a hundred terabytes to several gigabytes. This allowed us to emulate HPL on a single commodity server within 1-3 days scenarios similar to the run that was carried out on the Stampede cluster (hosted at TACC) in 2013 for the TOP500 (see Table I). More importantly, we show that despite the genericity of our approach (we only modify a few dozens of lines of HPL), our simulation allows to predict the performance of HPL on a recent cluster (running a thousand MPI ranks) within a few percent of reality (see Figure 1). This article is organized as follows: Section II presents the main characteristics of the HPL application and provides information on how the runs are conducted on modern supercomputers. Section III discusses related work and explains why emulation (or online simulation) is the only sensible approach when studying an application as complex as HPL. In Section IV, we briefly present the simulator we used for this work, SimGrid/SMPI, followed by an extensive discussion in Section V about the optimizations on all levels (i.e., simulator, application, operating system) that are necessary to make a large-scale run tractable. The scalability of our approach is evaluated in Section V-C. In Section VI, we explain how the different resources of the supercomputer (network, compute kernels) can be modeled to predict the performance of HPL faithfully. We particularly discuss the usefulness and relevance of different degrees of modeling complexity. In Section VII, we compare simulation results with real experiments and illustrate the importance of modeling both spatial and temporal variability. Section VIII concludes this article by discussing perspectives and future work.

II. CONTEXT A. High-Performance Linpack

In this work, we use the freely-available referenceimplementation of HPL [START_REF] Petitet | HPLa portable implementation of the High-Performance Linpack benchmark for distributed-memory computers[END_REF], which relies on MPI. HPL implements a matrix factorization based on a right-looking variant of the LU factorization with row partial pivoting and allows multiple look-ahead depths. The principle of the factorization is depicted in Figure 2. It consists of a series of panel factorizations followed by an update of the trailing sub-matrix. HPL uses a two-dimensional block-cyclic data distribution of A and implements several custom MPI collective communication algorithms to efficiently overlap communications with computations. The main parameters of HPL are:

• N is the order of the square matrix A.

• NB is the "blocking factor", i.e., the granularity at which HPL operates when panels are distributed or worked on. • P and Q denote the number of process rows and the number of process columns, respectively. • RFACT determines the panel factorization algorithm. Possible values are Crout, left-or right-looking. • SWAP specifies the swapping algorithm used while pivoting.

Two algorithms are available: one based on binary exchange (along a virtual tree topology) and the other one based on a spread-and-roll (with a higher number of parallel communications). HPL also provides a panel-size threshold triggering a switch from one variant to the other. • BCAST sets the algorithm used to broadcast a panel of columns over the process columns. Legacy versions of the MPI standard only supported non-blocking point-topoint communications, which is why HPL ships with in total 6 self-implemented variants to overlap the time spent waiting for an incoming panel with updates to the trailing matrix: ring, ring-modified, 2-ring, 2-ring-modified, long, and long-modified. The modified versions guarantee that the process right after the root (i.e., the process that will become the root in the next iteration) receives data first and does not further participate in the broadcast. This process can thereby start working on the panel as soon as possible. The ring and 2-ring versions each broadcast along the corresponding virtual topologies while the long version is a spread and roll algorithm where messages are chopped into Q pieces. This generally leads to better bandwidth exploitation. The ring and 2-ring variants rely on MPI_Iprobe, meaning they return control if no message has been fully received yet, hence facilitating partial overlap of communication with computations. In HPL 2.1 and 2.2, this capability has been deactivated for the long and longmodified algorithms. A comment in the source code states that some machines apparently get stuck when there are too many ongoing messages. • DEPTH controls how many iterations of the outer loop can overlap with each other.

The sequential complexity of this factorization is

flop(N ) = 2 3 N 3 + 2N 2 + O(N )
where N is the order of the matrix to factorize. The time complexity can be approximated by

T (N ) ≈ 2 3 N 3 + 2N 2 P • Q • w + Θ((P + Q) • N 2 ),
where w is the flop rate of a single node and the second term corresponds to the communication overhead which is influenced by the network capacity and the previously listed parameters (RFACT, SWAP, BCAST, DEPTH, . . . ) and is very difficult to predict.

B. Typical Runs on a Supercomputer

Although the TOP500 reports precise information about the core count, the peak performance and the effective performance, it provides almost no information on how (software versions, HPL parameters, etc.) this performance was achieved. Some colleagues agreed to provide us with the HPL configuration they used and the output they submitted for ranking (see Table II). In June 2013, the Stampede supercomputer at TACC was ranked 6th in the TOP500 by achieving 5168.1 TFlop s -1 . In November 2017, the Theta supercomputer at ANL was ranked 18th with a performance of 5884.6 TFlop s -1 but required a 28-hour run on the whole machine. Finally, we ran HPL ourselves on a Grid'5000 cluster named Dahu whose software stack could be fully controlled. The performance typically achieved by supercomputers (Rmax) needs to be compared to the much larger peak performance (Rpeak). This difference can be attributed to the node usage, to the MPI library, to the network topology that may be unable to deal with the intense communication workload, to load imbalance among nodes (e.g., due to a defect, system noise, . . . ), to the algorithmic structure of HPL, etc. All these factors make it difficult to know precisely what performance to expect without running the application at scale. It is clear that due to the level of complexity of both HPL and the underlying hardware, simple performance models (analytic expressions based on N, P, Q and estimations of platform characteristics as presented in Section II-A) may be able to provide trends but can by no means accurately predict the performance for each configuration (e.g., consider the exact effect of HPL's six different broadcast algorithms on network contention). Additionally, these expressions do not allow engineers to improve the performance through actively identifying performance bottlenecks. For complex optimizations such as partially non-blocking collective communication algorithms intertwined with computations, a very faithful modeling of both the application and the platform is required. Our goal in this article is to simulate systems at the scale of Stampede. Given the scale of this scenario (3,785 steps on 6,006 nodes in two hours), detailed simulations quickly become intractable without significant effort.

III. RELATED WORK

A first approach for estimating the performance of applications like HPL is statistical modeling of the application as a whole [START_REF] Luszczek | Reducing the time to tune parallel dense linear algebra routines with partial execution and performance modeling[END_REF]. By running the application several times with small and medium problem sizes (of a few iterations of large problem sizes) and using simple linear regressions, it is possible to predict its makespan for larger sizes with an error of only a few percents and a relatively low cost. Unfortunately, the predictions are limited to the same application configuration and studying the influence of the number of rows and columns of the virtual grid or of the broadcast algorithms requires a new model and new (costly) runs using the whole target machine. Furthermore, this approach does not allow to study what-if scenarios (e.g., to evaluate what would happen if the network bandwidth was increased or if node heterogeneity was decreased) that are particularly useful when investigating potential performance improvements.

Simulation provides the details and flexibility missing to such black-box modeling approach. Performance prediction of MPI applications through simulation has been widely studied over the last decades but two approaches can be distinguished in the literature: offline and online simulation.

With the most common approach, offline simulation, a trace of the application is first obtained on a real platform. This trace comprises sequences of MPI operations and CPU bursts and is given as an input to a simulator that implements performance models for the CPUs and the network to derive predictions. Researchers interested in finding out how their application reacts to changes to the underlying platform can replay the trace on commodity hardware at will with different platform models. Most HPC simulators available today, notably BigSim [START_REF] Zheng | BigSim: A Parallel Simulator for Performance Prediction of Extremely Large Parallel Machines[END_REF], Dimemas [START_REF] Badia | Dimemas: Predicting MPI Applications Behaviour in Grid Environments[END_REF] and CODES [START_REF] Mubarak | Enabling parallel simulation of large-scale HPC network systems[END_REF], rely on this approach. The main limitation of this approach comes from the trace acquisition requirement. Not only is a large machine required but the compressed trace of a few iterations (out of several thousands) of HPL typically reaches a few hundred MB, making this approach quickly impractical [START_REF] Casanova | Simulation of MPI applications with time-independent traces[END_REF]. Worse, tracing an application provides only information about its behavior at the time of the run: slight modifications (e.g., to communication patterns) may make the trace inaccurate. The behavior of simple applications (e.g., stencil) can be extrapolated from small-scale traces [START_REF] Wu | ScalaExtrap: Trace-based communication extrapolation for SPMD programs[END_REF], [START_REF] Carrington | Inferring large-scale computation behavior via trace extrapolation[END_REF] but this fails if the execution is non-deterministic, e.g., whenever the application relies on non-blocking communication patterns, which is unfortunately the case for HPL.

The second approach discussed in the literature is online simulation. Here, the application is executed (emulated) on top of a simulator that is responsible to determine when each process is run. This approach allows researchers to study directly the behavior of MPI applications but only a few recent simulators such as SST Macro [START_REF] Janssen | A simulator for large-scale parallel architectures[END_REF], SimGrid/SMPI [START_REF] Casanova | Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms[END_REF] and the closed-source xSim [START_REF] Engelmann | Scaling To A Million Cores And Beyond: Using Light-Weight Simulation to Understand The Challenges Ahead On The Road To Exascale[END_REF] support it. To the best of our knowledge, only SST Macro and SimGrid/SMPI are mature enough to faithfully emulate HPL. In this work, we decided to rely on SimGrid as its performance models and its emulation capabilities seemed quite solid but the developments we propose would a priori also be possible with SST. Note that the HPL emulation we describe in Section V should not be confused with the application skeletonization [START_REF] Bird | Performance modelling of magnetohydrodynamics codes[END_REF] commonly used with SST. Skeletons are code extractions of the most important parts of a complex application whereas we only modify a few dozens of lines of HPL before emulating it with SMPI. Finally, it is important to understand that the approach we propose is intended to help studies at the level of the whole machine and application, not the influence of microarchitectural details as intended by MUSA [START_REF] Grass | Musa: A multi-level simulation approach for next-generation hpc machines[END_REF].

IV. SIMGRID/SMPI IN A NUTSHELL

SimGrid [START_REF] Casanova | Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms[END_REF] is a flexible and open-source simulation framework that was originally designed in 2000 to study scheduling heuristics tailored to heterogeneous grid computing environments but has later been extended to study cloud and HPC infrastructures. The main development goal for SimGrid has been to provide validated performance models particularly for scenarios making heavy use of the network. Such a validation usually consists of comparing simulation predictions with results from real experiments to confirm or debunk network and application models. SMPI, a simulator based on SimGrid, has been developed and used to simulate unmodified MPI applications written in C/C++ or FORTRAN [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF]. The complex network optimizations done in real MPI implementations need to be considered when predicting the performance of MPI applications. For instance, the "eager" and "rendez-vous" protocols are selected based on the message size, with each protocol having its own synchronization semantics, which strongly impact performance. SMPI supports different performance modes through a generalization of the LogGPS model. Another difficult issue is to model network topologies and contention. SMPI relies on SimGrid's communication models where each ongoing communication is represented as a whole (as opposed to single packets) by a flow. Assuming steady-state, contention between active communications can then be modeled as a bandwidth sharing problem that accounts for non-trivial phenomena (e.g., crosstraffic interference [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF]). If needed, communications that start or end trigger a re-computation of the bandwidth share. In this fluid model, the time to simulate a message passing through the network is independent of its size, which is advantageous for large-scale applications frequently sending large messages and orders of magnitude faster than packet-level simulation. SimGrid does not model transient phenomena incurred by the network protocol but accounts for network topology and heterogeneity. Special attention to the modeling of collective communication algorithms has also been paid in SMPI, but this is of little significance in this article as HPL ships with its own implementation of collective operations.

SMPI maps every MPI rank of the application onto a lightweight simulation thread. These threads are then run one at a time, i.e., in mutual exclusion. Every time a thread enters an MPI call, SMPI takes control and the time that was spent computing (isolated from the other threads) since the previous MPI call is injected into the simulator as a virtual delay. This time may be scaled up or down depending on the speed of the simulated machine with respect to the simulation machine. Recent results report consistent performance predictions within a few percent for standard benchmarks on small-scale clusters (up to 12 × 12 cores [START_REF] Heinrich | Predicting the Energy Consumption of MPI Applications at Scale Using a Single Node[END_REF] and up to 128 × 1 cores [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF]). In this article, we validate this approach at a much larger scale with HPL, whose emulation comes with at least two challenges:

• The time-complexity of the algorithm is Θ(N 3 ) and Θ(N 2 ) communications are performed, with N being very large. The execution on the Stampede cluster took roughly two hours on 6006 compute nodes. Using only a single node, a naive emulation of HPL at the scale of the Stampede run would take about 500 days if perfect scaling was reached. • The tremendous memory consumption and amount of memory accesses need to be drastically reduced.

V. EMULATING HPL AT LARGE SCALE

We now present the changes to SimGrid and HPL that were required for a scalable simulation. We provide only a brief presentation of our modifications and refer the reader interested in details to a previous report [START_REF] Cornebize | Capacity Planning of Supercomputers: Simulating MPI Applications at Scale[END_REF], [START_REF] Cornebize | Laboratory Notebook hosted on GitHub[END_REF] and a companion repository [START_REF] Cornebize | Fast and Faithful Performance Prediction of MPI applications: the HPL use case study (experiment artifact)[END_REF]. For our experiments in this section, we used a single core from nodes of the Nova cluster provided by the Grid'5000 testbed [START_REF] Balouek | Adding virtualization capabilities to the Grid'5000 testbed[END_REF] (32 GB of RAM, two 8-core Intel Xeon E5-2620 v4 CPUs, Debian Stretch OS (Linux 4.9)). Kernel modeling q q FALSE TRUE q q q q q q q q q q q q 0 500 1000 1500 0 10000 20000 30000 40000

Matrix order

Time [seconds]
Simulation time (P=Q=8, i.e., 64 MPI processes) q q q q q q q q q q q q 0 25 50 A. Speeding Up the Emulation 1) Compute Kernel Modeling: HPL heavily relies on BLAS kernels such as dgemm (for matrix-matrix multiplication) or dtrsm (for solving an A • x = b equation). The analysis of an HPL simulation with 64 processes and a very small matrix of order 30,000 showed that about 96 % of the time is spent in these two kernels. Since the output of these kernels does not influence the control flow, simulation time can be reduced by substituting dgemm and dtrsm function calls with a performance model of the respective kernel. Skipping kernels renders the content of some variables invalid but in simulation, only the behavior of the application and not the correctness of computation results are of concern. Figure 3(a) shows an example of this macro-based mechanism that allows us to keep HPL code modifications to an absolute minimum. The (1.029e-11) value represents the inverse of the flop rate for this compute kernel and was obtained through calibration. The estimated time of the kernel is calculated based on the given parameters and passed on to smpi_execute_benched that advances the clock of the executing rank by this estimate. The effect on the simulation time for a small scenario is depicted in Figure 3(b). This modification speeds up the simulation by orders of magnitude. The precision of the simulation will be investigated in more details in the next sections but it can already be observed that this simple kernel model leads to a sound, albeit slightly more optimistic, estimation of the performance.

In addition to the main compute kernels, we identified seven other BLAS functions through profiling as computationally expensive enough to justify a specific handling: dgemv, dswap, daxpy, dscal, dtrsv, dger and idamax. Similarly, a significant amount of time was spent in fifteen functions implemented in HPL: HPL_dlaswp*N, HPL_dlaswp*T, HPL_dlacpy and HPL_dlatcpy. All these functions are called during the LU factorization and hence impact the performance measured by HPL; however, because of the removal of the dgemm and dtrsm computations, they all operate on bogus data and hence also produce bogus data. To study their impact on the overall performance, we handled them similarly to dgemm and dtrsm, through performance models and macro substitution, which speeds up the simulation by an additional factor of 3 to 4 on small (N = 30,000) and even more on large scenarios.

2) Specific Adjustments: HPL uses pseudo-randomly generated matrices that are setup every time HPL is executed. This initialization, just like the factorization correctness verification at the end of the run, is not considered in the reported performance and can therefore be safely skipped. Note that HPL implements an LU factorization with partial pivoting, which requires a special treatment of the idamax function that returns the index of the first element equaling the maximum absolute value. Although we ignored the cost of this function as well, we set its return value to a random (but controlled) value to make the simulation unbiased (but fully deterministic). We confirmed that this modification was harmless in terms of performance prediction (see [START_REF] Cornebize | Laboratory Notebook hosted on GitHub[END_REF] for details).

B. Scaling Down Memory Consumption

The largest two allocated data structures in HPL are the input matrix A (with a size of typically several GB per process) and the panel which contains information about the sub-matrix currently being factorized. This sub-matrix typically occupies a few hundred MB per process. Unfortunately, when emulating an application with SMPI, all MPI processes are run within the same simulation process on a single node and the memory consumption of the simulation can therefore quickly reach several TB of RAM. Yet, as we no longer operate on real data, storing the whole input matrix A is needless. However, since only a minimal portion of the code was modified, some functions may still read or write some parts of the matrix. It is thus not possible to simply remove the memory allocations of large data structures. SMPI provides the SMPI_SHARED_MALLOC (SMPI_SHARED_FREE) macro to replace calls to malloc (free). They indicate that some data structures can safely be shared between processes and that the data they contain is not critical for the execution (e.g., an input matrix) and that it may even be overwritten. SMPI_SHARED_MALLOC works as follows (see Figure 4): a single block of physical memory (of default size 1 MB) for the whole execution is allocated and shared by all MPI processes. A range of virtual addresses corresponding to a specified size is reserved and cyclically mapped onto the previously obtained physical address. This mechanism allows most applications to obtain a nearly constant memory footprint, regardless of the size of the actual allocations.

Although using the default SHARED_MALLOC mechanism works flawlessly for A, a more careful strategy needs to be used for the panel, which is an intricate data structure with both ints (accounting for matrix indices, error codes, MPI tags, and pivoting information) and doubles (corresponding to a copy of a sub-matrix of A). To optimize data transfers, HPL flattens this structure into a single allocation of doubles (see Figure 5(a)). Using a fully shared memory allocation for the panel therefore virtual physical Figure 4: SMPI shared malloc mechanism: large area of virtual memory are mapped onto the same physical pages.

matrix parts indices matrix parts can be shared can be shared must not be shared leads to index corruption that results in classic invalid memory accesses. Since ints and doubles are stored in non-contiguous parts of this flat allocation, it is therefore essential to have a mechanism that preserves the process-specific content. We have thus introduced the SMPI_PARTIAL_SHARED_MALLOC macro that allows us to specify which ranges of the allocation should be preserved (i.e., are private to each process) and which ones may be corrupted (i.e., are shared between processes). For a matrix of order 40, 000 and 64 MPI processes, memory consumption decreases with this approach from about 13.5 GB to less than 40 MB.

Another HPL specific optimization is related to the systematic allocation and deallocation of panels in each iteration, with the size of the panel strictly decreasing from iteration to iteration. As we explained above, the partial sharing of panels requires many calls to mmap and introduces an overhead that makes these repeated allocations / frees a bottleneck. Since the very first allocation can fit all subsequent panels, we modified this allocation mechanism so that SMPI can reuse panels as much as possible from an iteration to an other (see Figure 5(b)). Even for a very small matrix of order 40, 000 and 64 MPI processes, the simulation time decreases from 20.5 sec to 16.5 sec. The number of page faults decreased from 2 million to 0.2 million, confirming the devastating effect these allocations/deallocations would have at scale.

The last three optimizations we describe are not specific to HPL. We leveraged the information on which memory area is private, shared or partially shared to improve the overall performance. By making SMPI internally aware of the memory's visibility, it can now avoid calling memcopy when large messages containing shared segments are sent from one MPI rank to another. For fully private or partially shared segments, SMPI identifies and copies only those parts that are process-dependent (private) into the corresponding buffers on the receiver side. HPL simulation times and memory consumption were considerably improved in our experiments q q q q q q q q q q q q q q q q 512 1,024 Matrix rank Simulation time (hours) q q q q q q q q q q q q q q q q 512 1,024 Matrix rank Memory consumption (gigabytes) Number of processes q q q q 512 1,024 2,048 4,096 because the panel is the most frequently transferred data structure but only a small part of it is actually private. As explained above, SMPI maps MPI processes to threads of a single process, effectively folding them into the same address space. Consequently, global variables in the MPI application are shared between threads unless these variables are privatized and the simulated MPI ranks thus isolated from each other. Several technical solutions are possible to handle this issue [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF]. The default strategy in SMPI consists in making a copy of the data segment (containing all global variables) per MPI rank at startup and, when context switching to another rank, to remap the data segment via mmap to the private copy of that rank. SMPI also implements another mechanism relying on the dlopen function that allows to load several times the data segment in memory and to avoid costly calls to mmap (and subsequent cache flush) when context switching. For a matrix of order 80,000 and 32 MPI processes, the number of minor page faults drops from 4,412,047 (with mmap) to 6880 (with dlopen), which results in a reduction of system time from 10.64 sec (out of 51.47 sec) to 2.12 sec.

Finally, for larger matrix orders (i.e., N larger than a few hundred thousands), the performance of the simulation quickly deteriorates as the memory consumption rises rapidly. Indeed, folding the memory reduces the physical memory usage. The virtual memory, on the other hand, is still allocated for every process since the allocation calls are still executed. Without a reduction of allocated virtual addresses, the page table rapidly becomes too large for a single node. Thankfully, the x86-64 architecture supports several page sizes, such as the huge pages in Linux. Typically, these pages are around 2 MiB (instead of 4 KiB), which reduces drastically the page table size. For example, for a matrix of order N = 4, 000, 000, it shrinks from 250 GB to 0.488 GB.

C. Scalability Evaluation

The main goal of the previous optimizations is to reduce the complexity from Θ(N 3 ) + Θ(N 2 • P • Q) to something more reasonable. The Θ(N 3 ) was removed by skipping most computations. Ideally, since there are N/N B iterations (steps), the complexity of simulating one step should be decreased to something independent of N . SimGrid's fluid models, used to simulate communications, do not depend on N . Therefore, the time to simulate a step of HPL should mostly depend on P and Q. Yet, some memory operations on the panel that are related to pivoting are intertwined in HPL with collective communications, meaning that it is impossible to get rid of the O(N ) complexity without modifying HPL more profoundly.

To evaluate the efficiency of our proposal, we conduct a first evaluation on a non-existing but Stampede resembling platform comprising 4,096 nodes interconnected through a fattree topology. We run simulations with 512, 1024, 2048 or 4096 MPI ranks and with matrices of orders 5 × 10 5 , 1 × 10 6 , 2 × 10 6 or 4 × 10 6 . All other HPL parameters are similar to the ones of the original Stampede scenario. The impact of the matrix order on total makespan and memory is illustrated in Figure 6. With all previously described optimizations enabled, the longest simulation took close to 47 hours and consumed 16 GB of memory whereas the shortest one took 20 minutes and 282 MB of memory. These optimizations also enabled us to simulated the configuration used for the Stampede cluster in 2013 for the TOP500 ranking in less than 62 hours and using 19 GB on a single node of a commodity cluster (see Table I).

VI. MODELING HPL KERNELS AND COMMUNICATIONS

As explained in Section V, HPL spends most of its computation time in a dozen specific functions for which a performance model has to be designed. Most compute kernels have several parameters from which a very simple model can generally easily be identified (e.g., proportional to the product of the parameters) but refinements including the individual contribution of each parameter as well as the spatial and temporal variability of the operation are also possible. Likewise, communications between two nodes are mostly linear in message size but the actual performance can wildly vary depending on the range of the message size as MPI switches from one protocol to another whenever needed (see Figure 7). In this section we first introduce some notations to describe the complexity of the models we have investigated. We then briefly compare the prediction of these models with individual measurements of both computations and communications to illustrate the importance of the model complexity. We finally show in Section VII that using simpler modeling options leads to a serious inaccuracy of the simulation.

A. Modeling Notations

We denote as T the duration of an operation with parameters M , N , K (in the case of the dgemm operation, these parameters describe the geometry of the input matrices). We first consider the three following modeling options:

• Modeling option M -0: For simple and stable compute kernels, the duration can be modeled as a constant duration independent of the input parameters, i.e., T ∼ α, where α is estimated through the sample average of the duration of the operation (or simply 0 if the kernel is negligible) . • Modeling option M-1: A simple combination of the parameters (e.g., S = M.N.K) may be the primary factor driving the performance of the operation. Then T ∼ α.S (+β) and α and β can be estimated through a classical least-square linear regression. • Modeling option M-2: When the behavior of the operation is complex or requires a faithful modeling over the full range of input parameters, a full polynomial model is required, i.e., T ∼ α.M.N.K + β.M.N + γ.N.P + . . . Again, the α, β, γ, . . . can be estimated through a classical least-square linear regression. There are two situations where more elaborate variations need to be considered:

• Whenever the platform is slightly heterogeneous (spatial variability), the previous models should be built for each host individually. This modeling option is denoted M H . • The behavior of the operation may be mostly linear but only for specific parameter ranges. This is for example the case for networking operations or for computing nodes on Stampede where Intel's Math Kernel Library (MKL) uses the Xeon Phi accelerator only when the input is large enough to compensate for the data transfer. In such situations, the models considered will be piece-wise linear, e.g., T ∼

     if M < θ 1 α 1 .M + β 1 else if M < θ 2 α 2 .M + β 2 ... ,
where the θ, α, β should all be estimated. This kind of model is denoted M . All previous models can be fit with relatively simple linear regressions or maximum likelihood learning methods. However, an important hypothesis underlying all these methods is the homoscedasticity, i.e., that the variability is independent on the parameters.

The residual (temporal) variability may be an important phenomenon to account for, as "system noise" is known to be detrimental to the overall performance of parallel applications like HPL. We thus consider different modeling options for this temporal variability:

• Noise option N-0 (no noise): This is the simplest option.

It consists in injecting the value predicted by the model • Noise option N -1 (homoscedastic): The simplest probability family to model variability is the normal distribution, hence T ∼ M(M, N, K) + N (0, σ 2 ), where σ 2 is the sample variance of the model residuals. • Noise option N-2 (heteroscedastic): The conditional variance of the residuals (i.e., σ 2 given M, N, K) is modeled by a polynomial function of the input parameters. Finally, even the sophisticated normal distribution from N-2 may be too simple to describe the noise observed on real platforms where it may be common for a same parameter set to have a few operations being one order of magnitude slower than all the other ones. In this case, a reasonable option consists of modeling noise with a mixture of normal distributions whose parameters π 1 , . . . , π k should be estimated. We denote this kind of model as N . Likewise, the per-host estimations are denoted by N H . Linear regression is a standard tool in R [START_REF] Core | R: A Language and Environment for Statistical Computing[END_REF] or in python/statsmodels [START_REF] Seabold | Statsmodels: Econometric and statistical modeling with python[END_REF] and models M H -0, 1, 2 are thus easy to fit assuming N-1. Model M -0 (piece-wise constant) assuming N -1 can easily be fit using the cubist [START_REF] Kuhn | Cubist: Rule-And Instance-Based Regression Modeling[END_REF] library. We could not find any implementation allowing to fit M -1 (piece-wise linear, possibly discontinuous) assuming N -1 so we implemented a method based on model trees [START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF]Chapter 9] [26] in a python library (pytree). When the noise is heteroscedastic (N -2), it is sometimes possible to fall back to the previous methods if a careful sampling method is employed (by sampling more on highly variable areas and fitting the average). Finally, for even more complex noise modeling options (N -), some custom Expectation Maximization algorithms are available (e.g., flexmix [START_REF] Grün | FlexMix version 2: Finite mixtures with concomitant variables and varying and constant parameters[END_REF] for M -1 N -1).

B. Modeling MPI communications

Prior to this work, the standard way of accounting for protocol changes in SMPI was to estimate breakpoints visually and to conduct a linear regression for each range. The expected duration was then used directly in the simulation with no particular effort with respect to the temporal variability (M -1 N -0). Yet, as illustrated in Figure 7, the variability of high speed networks is quite particular. We therefore diligently estimated all the parameters of M -1 N -1, where each message size range is automatically estimated with pytree, as well as the 2 to 4 modes of the Gaussian mixture for each range. Such temporal variability could explain some (overall bad) performance since they generally get amplified by broadcast and pipelined communication patterns.

C. Modeling dgemm

HPL spends the most time in the dgemm kernel. We therefore evaluated the previous modeling alternatives: M -{1, 2} N-{0, 1, 2} and M H -{1, 2} N H -{0, 1, 2}. The M and N families were not investigated as nothing in our observations called for such complexity on classical multi-core machines. Figure 8 illustrates various models and their respective quality for the dgemm function. In these figures, the performance of dgemm is evaluated by calling dgemm with randomized sizes over all the cores of each node (to reproduce experimental conditions similar to the one of HPL). The first observation (Figure 8(a)) is that a few nodes exhibit quite a different behavior (each color and each regression line under model M H -1 corresponds to a different cpu, whereas the black dotted line corresponds to model M -1 over all the nodes). These nodes will systematically be slightly slower than other nodes and accounting for this spatial heterogeneity is likely to be rather important for HPL. Second, we took care of covering a wide variety of combinations for M , N , and K and it can be observed that M.N.K is not sufficient to describe correctly the performance of dgemm. Indeed, for M.N.K ≈ 4.5 × 10 9 some duration are systematically higher regardless of the node. This happens for some particular (e.g., tall and skinny) matrix geometries, which strongly suggests using the full polynomial model. Figure 8(b) depicts the performance (red dots) of a given node as well as the prediction using a simple linear model (M H -1, black line), a full polynomial model (M H -2, blue dots) and a full polynomial model with heteroscedastic noise (M H -2 N H -2, orange dots). A close inspection reveals that all experimental variability is actually very well explained by both the polynomial model (better fit for particular parameter combinations) and some temporal variability.

D. Modeling Other BLAS and HPL Kernels

Four other BLAS kernels and a few other very small HPL compute kernels (often related to memory management) are deeply intertwined with collective operations to allow HPL to be as efficient as possible. Although the total duration of these kernels is extremely small compared to the total execution time, they may perturb collective communication by introducing late sends and receives. The behavior of one of these kernels is illustrated in Figure 8(c). This kind of data can only be obtained by running HPL for a small input matrix over each node individually. Again, for all these kernels a single parameter combination explains most of the performance and there is some variability from one node to another (one blue regression line per CPU) but it remains quite limited (black dotted line for the platform as a whole), especially since these kernels are very short and infrequently called compared to dgemm. Finally, since variability significantly increases with the value of the input parameters, a N -2 model is clearly required. The blue dots in Figure 8(c) represent the outcome of a M -1 N -2 model and are hardly distinguishable from the real behavior. Similar results can be obtained with this category of model for all other kernels.

VII. VALIDATION AT SCALE A. Experimental Setup

To evaluate the soundness of our approach, we compare several real executions of HPL with simulations using the previous models. We used the Dahu cluster from the Grid'5000 testbed. It has 32 nodes connected through a single switch by 100 Gbit s -1 Omnipath links. Each node has two Intel Xeon Gold 6130 CPU with 16 cores per CPU and we disabled hyperthreading. We used HPL version 2.2 compiled with GCC version 6.3.0. We also used the libraries OpenMPI version 2.0.2 and OpenBLAS version 0.3.1. HPL executions were done using a block size of 128 and a matrix of varying size (from 50,000 to 500,000). We used one single-threaded MPI rank per core and a look-ahead depth of 1. Finally, we used the increasing-2-ring broadcast with the Crout panel factorization algorithms.

Although this machine is much smaller than top supercomputers, faithfully simulating an HPL execution with such settings is already quite challenging.

• We used one rank per core to obtain a higher number (1024) of MPI process. This is more difficult than simulating one rank per node, as [START_REF] Casanova | Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms[END_REF] this increases the amount of data transferred through MPI and (2) the performance is then subject to memory interference and network heterogeneity (we used a different model for local and remote communications). • We used a smaller block size than commonly used, which leads to a higher number of iterations and hence more complex communication patterns. • We used relatively small input matrices, which reduces the makespan and makes good predictions harder to obtain.

B. Comparing Simulations with Real Executions

Our first evaluation consists in comparing the traces of the simulations with reality. We instrumented HPL to collect the start and end timestamps of each kernel and MPI call. We limited the execution to 256 ranks and the first five iterations. A first qualitative validation can be done by visually comparing the Gantt charts of the simulations with reality (see Figure 9). Calls to dgemm are depicted in yellow, MPI_Send in red, MPI_Recv in blue. Although the structure is similar in all The charts show that, at this scale, using a deterministic or a stochastic model for the network has no noticeable impact on HPL simulation. However, having a more complex model for the kernels leads to much more realistic traces. The variability in the computation durations leads to an increase of the time spent in communications and an overall slightly longer execution. In HPL, computation variability directly translates to late senders/receivers that destroy the efficiency of collective operations.

We now provide a more quantitative comparison using the whole cluster and varying matrix sizes, focusing on the GFlop s -1 rate reported by HPL (see Figure 10). The real executions are depicted in black, for each matrix size we performed 8 runs of HPL, to illustrate the temporal variability of the performance. The line (a), on the top, is our first attempt to simulate HPL. The simulation was done with a simple model: M -1 for the kernels and M -1 for the network with no noise (N-0) in both cases. This model overestimates HPL performance by more than 30 %. We initially thought that the network model was too optimistic, however, switching to a stochastic multi-modal network model (N -1, the line (b)), does not significantly improve the prediction precision.

Figure 8 shows that there is an important heterogeneity in the cluster. For this reason, we started using a M H -1/N -0

(1) spatial var.

(2) temporal var. We found that adding the temporal variability noise (N -2 for all kernels, N H -2 for dgemm) is the key ingredient to obtain the last bit of realism. The prediction (the line (e)) is now extremely close to reality as it slightly underestimates the performance by less than 5 % and even as little as 1 % for the larger matrices. Adding back temporal variability to the network model (N -1, line (f)) still has no significant effect but this can be explained by the fact that HPL mostly communicates very large amounts of data in bulk. Network temporal variability is however very important aspect to model for applications that are more latency-bound.

As illustrated in Figure 10 (rightmost labels), we would like to stress again that accurate predictions require a careful modeling of both spatial and temporal variability. Overall, although this was not particularly foreseeable and could have been different with an other application, only the dgemm kernel needs to be carefully modeled with a M H -2-N H -2. In Figure 1, we used a simple M -1-N -0 for the other BLAS kernels and simply skipped every other kernel. A detailed modeling of all (BLAS and HPL) kernels is possible but a minimal calibration of the dgemm kernel over a representative set of nodes is thus sufficient to consistently predict the performance of HPL on this machine within a few percent of reality.

C. Comparing With the Stampede Qualification Run

Each node of the Stampede cluster comprises two 8-core Intel Xeon E5-2680 8C 2.7 GHz CPUs and one 61-core Intel Xeon Phi SE10P (KNC) 1.1 GHz accelerator that is roughly three times more powerful than the two CPUs. The HPL output submitted to the TOP500 (Table II) does not indicate how the KNC was used. However, because of the values assigned to P and Q, we are certain that there were only one MPI rank per node. For this reason, it is likely that the KNC was used as an accelerator, which is effortless with Intel's Math Kernel Library (MKL) as it supports automatic offloading for selected BLAS functions. While we cannot know precisely which MKL version was used in 2013, we measured the default version (version 11.1.1) that was used on Stampede in the beginning of 2017 and built precise M -1 N -1 models of dgemm and dtrsm. Likewise, using measurements of the performance of MPI obtained in 2017, we built a precise M -1 N-0 model of the fat-tree interconnect. All HPL input parameters were set to exactly the values used in the TOP500 qualification run and were coherent with the submitted output.

Surprisingly, despite our efforts, our predictions forecast a significantly lower performance (≈ 20-30%) than the one reported for the top500, even when considering an ideal network (no contention, no latency, no noise) and perfectly stable and homogeneous nodes where Xeon Phi would operate at peak performance. Unfortunately, when tracking down the cause of this discrepancy, we eventually realized that despite the reported output which mentioned HPLinpack 2.1 from October 26, 2012 by Petitet et al., the TOP500 performance had not been obtained with the open-source version of HPL but with a specifically optimized binary version from Intel employing a custom broadcast algorithm relying on nonblocking sends, and which was a priori specifically modified for Stampede. We refer to a previous work [START_REF] Cornebize | Emulating High Performance Linpack on a Commodity Server at the Scale of a Supercomputer[END_REF] for a detailed discussion on this investigation, which explains why this failure to predict similar performance cannot be considered as an invalidation of our approach. Validation experiments therefore cannot rely on the lacunar information provided by the TOP500 and cannot make the economy of a large-scale controlled (software environment, source code) run.

VIII. CONCLUSIONS

Studying and tuning HPC applications at scale can be very time-and resource-consuming. We believe that being capable of precisely predicting an application's performance on a given platform is useful for application developers and users (e.g., to evaluate the scalability of the application on a given machine) and will become invaluable in the future as it can for example aid compute centers with the decision of whether the envisioned technology of a new machine works best for a given application or if an upgrade of the current machine should be considered. Simulation is often an effective approach in this context and SimGrid/SMPI has previously been successfully validated in several small-scale studies with simple HPC benchmarks [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF], [START_REF] Heinrich | Predicting the Energy Consumption of MPI Applications at Scale Using a Single Node[END_REF]. In this article, we proposed and evaluated extensions to SMPI that allowed us to emulate HPL at the scale of a supercomputer. Our application of choice, HPL, is particularly challenging both in terms of tuning and simulation as it implements its own set of non-blocking collective operations (from which the users has to choose) that rely on MPI_Iprobe in order to overlap with computations.

More specifically, we explain how to simulate on a single commodity node scenarios at the scale of the TOP500 qualification runs. Although simulation is generally longer than a real execution, it is significantly less resource consuming. This emulation employs several non-trivial operating-system level optimizations (memory mapping, dynamic library loading, huge pages) that have since been integrated into the last version of SimGrid/SMPI. Unlike the common skeleton approach [START_REF] Bird | Performance modelling of magnetohydrodynamics codes[END_REF], we do not rely on a simplified version of HPL but only apply minor modifications (HPL comprises 16K lines of ANSI C over 149 files, our modifications only changed 14 files with 286 line insertions and 18 deletions) to the original source code to fully capture its whole complexity.

We also explain how different levels of detail of modeling of the performance of compute kernels and of the MPI library influence the quality of the prediction. In particular, we show that a careful modeling (including both spatial and temporal variability) of the main compute kernel (dgemm, which is rather inexpensive from a benchmarking perspective) allows to predict performance within a few percents of real experiments with 1,024 ranks on a controlled cluster of Grid'5000. Validation experiments involving larger platforms are underway but, as briefly explained with the Stampede study, cannot be done simply from the information reported in the TOP500 and require access to a well-controlled supercomputer.

As we explained, a careful modeling of networking and computing resources is crucial but requires a diligent combination of statistical libraries. We intend to automate our benchmarking and modeling approach as much as possible using a generic and unified Bayesian estimation framework like STAN [START_REF] Carpenter | Stan: A probabilistic programming language[END_REF]. The MCMC sampling technique used in STAN would be particularly helpful to inject uncertainty (in particular the platform heterogeneity which should be estimated from only a few sample nodes) in the simulation. This integration effort is underway. As another future work, building on the effort of SimGrid developers on supporting the emulation of a wide variety of applications with SMPI [START_REF] Developers | Continuous integration of ECP Proxy Apps, CORAL and Trinity-Nersc benchmarks with SimGrid/SMPI[END_REF], we also intend to conduct similar studies with other HPC benchmarks (e.g., HPCG [START_REF] Dongarra | HPCG benchmark : a new metric for ranking high performance computing systems[END_REF] or HPGMG [START_REF] Adams | Hpgmg 1.0: A benchmark for ranking high performance computing systems[END_REF]), real applications (e.g., BigDFT [START_REF] Genovese | Daubechies wavelets as a basis set for density functional pseudopotential calculations[END_REF]) and larger infrastructures.

Figure 2 :

 2 Figure 2: Overview of High Performance Linpack

#

  define HPL_dgemm(layout, TransA, TransB, M, N, K, \ alpha, A, lda, B, ldb, beta, C, ldc) ({ \ double size = ((double)M)*((double)N)*((double)K); \ double expected_time = 1.029e-11*size + 1.981e-12; \ smpi_execute_benched(expected_time); \ }) (a) Non-intrusive macro replacement with a very simple computation model.

  i.e., 64 MPI processes) (b) Gain in terms of simulation time.

Figure 3 :

 3 Figure 3: Replacing the calls to computationally expensive functions by a model allows to emulate HPL at a larger scale.

  (a) Structure of the panel in HPL. initial buffer current buffer (b) Reusing panel allocation from an iteration to another.

Figure 5 :

 5 Figure 5: Panel structure and allocation strategy.

Figure 6 :

 6 Figure 6: Time complexity and memory consumption are linear in the number of processes but remain mildly quadratic with matrix rank.

Figure 7 :

 7 Figure 7: Illustrating piecewise linearity and temporal variability of high-speed communications on two systems.

Figure 8 :

 8 Figure 8: Illustrating the realism of modeling for BLAS and HPL functions.

Figure 9 :

 9 Figure 9: Gantt charts of HPL first iterations in simulation

  (f) model (e) with network∼ M -1N -1 (e) model (a) with kernels∼ M-1N -2 dgemm∼ MH -2NH -2 (d) model (a) with dgemm∼ MH -2N -0 (c) model (a) with dgemm∼ MH -1N -0 (a) kernels∼ M-1N -0 network∼ M -1N -0 (b) model (a) with network∼ M -1N -1 (f) model (e) with network∼ M -1N -1 (e) model (a) with kernels∼ M-1N -2 dgemm∼ MH -2NH -2 (d) model (a) with dgemm∼ MH -2N -0 (c) model (a) with dgemm∼ MH -1N -0 (a) kernels∼ M-1N -0 network∼ M -1N -0 (b) model (a) with network∼ M -1N -1 reality reality reality reality reality reality

Figure 10 :

 10 Figure 10: HPL performance: predictions vs. reality

Table I :

 I For large matrices, simulation is orders of magnitude less expensive than real executions

			Reality	Simulation
		#nodes / #processes	32 / 1024	1 / 1
	Dahu	Memory Duration (hours)	2 TB 1	9 GB 5
		Resources (node hours) 32	5
	Stampede	#nodes / #processes Memory Duration (hours)	6006 / 6006 1 / 1 120 TB 19 GB

Table II :

 II Typical runs of HPL

		Stampede@TACC	Theta@ANL	Dahu@G5K
	Rpeak	8520.1 TFlop s -1	9627.2 TFlop s -1	62.26 TFlop s -1
	N	3,875,000	8,360,352	500,000
	NB	1024	336	128
	P × Q	77×78	32×101	32×32
	RFACT	Crout	Left	Right
	SWAP	Binary-exch.	Binary-exch.	Binary-exch.
	BCAST	Long modified	2 Ring modified	2 Ring
	DEPTH	0	0	1
	Rmax	5168.1 TFlop s -1	5884.6 TFlop s -1	24.55 TFlop s -1
	Duration	2 hours	28 hours	1 hour
	Memory	120 TB	559 TB	2 TB
	MPI ranks	1/node	1/node	1/core

Acknowledgments

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

We warmly thank our TACC colleagues for their support in this study and providing us with as much information as they could. We thank our Cray colleagues for providing us with information about the TOP500 run of Theta, a resource of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357.

Last, we warmly thank the SimGrid developers for their help in integrating our contributions as well as for their early feedback on this work before submission.