Fabien Hermenier
email: fabien.hermenier@inria.fr

Robert Ricci
email: ricci@cs.utah.edu

How To Build a Better Testbed: Lessons From a Decade of Network Experiments on Emulab

The Emulab network testbed provides an environment in which researchers and educators can evaluate networked systems. Available to the public since 2000, Emulab is used by thousands of experimenters at hundreds of institutions around the world, and the research conducted on it has lead to hundreds of publications. The original Emulab facility at the University of Utah has been replicated at dozens of other sites. The physical design of the Emulab facility, and many other testbeds like it, has been based on the facility operators' expectations regarding user needs and behavior. If operators' assumptions are incorrect, the resulting facility can exhibit inefficient use patterns and sub-optimal resource allocation. Our study, the first of its kind, gains insight into the needs and behaviors of networking researchers by analyzing more than 500,000 topologies from 13,000 experiments submitted to Emulab. Using this dataset, we re-visit the assumptions that went into the physical design of the Emulab facility and consider improvements to it. Through extensive simulations with real workloads, we evaluate alternative testbeds designs for their ability to improve testbed utilization and reduce hardware costs.

Introduction

Network emulation testbeds [START_REF] Benzel | Experience with DETER: a testbed for security research[END_REF][START_REF] Bolze | Grid'5000: A large scale and highly reconfigurable experimental grid testbed[END_REF][START_REF] Miyachi | StarBED and SpringOS: Large-scale general purpose network testbed and supporting software[END_REF][START_REF] White | An integrated experimental environment for distributed systems and networks[END_REF][START_REF]Other Emulab testbeds[END_REF] have become environments of choice for evaluating network research. Their ability to provide highly customizable network topologies make them suitable for hosting a wide variety of experiments, including security, networking, and distributed systems research. One of the oldest and largest, the Utah Emulab facility [START_REF] Eide | An experimentation workbench for replayable networking research[END_REF][START_REF] Hibler | Large-scale virtualization in the emulab network testbed[END_REF][START_REF] White | An integrated experimental environment for distributed systems and networks[END_REF], is currently built from more than 600 nodes having more than 3,000 network interfaces. It gives experimenters full administrator access to their allotted physical nodes and the ability to shape characteristics of dedicated network links. We estimate that worldwide, at least ten thousand experimenters, a substantial fraction of the network research community, use testbeds that are derived from, or similar to, the Utah Emulab facility, and are thus affected by the decisions that have gone into its design. In this paper, we examine user requests submitted to Emulab to gain insight into experimenters' behavior, and we use this information to inform the physical design of future testbed facilities.

The goals of network testbeds differ from general-purpose clusters, and thus traditional cluster and data center designs cannot be applied directly to this domain. Clusters typically concentrate on maximizing host-based metrics (such as compute power or I/O operations) with the network playing only a supporting role. In a network testbed, these priorities are reversed: the network is the main object of study, and the physical infrastructure of the testbed must support embedding of a wide variety of experimenters' requested topologies. It is important to enable embeddings that avoid artifacts due to under-provisioned network links or devices. In most clusters, communications patterns on the network are not explicit; that is, any node may wish to communicate with any other node, and the existence and bandwidths of particular communication paths are not defined a priori. As a result, cluster networks must be provisioned based on measured load or built with full bisection bandwidth. In contrast, the networks in emulation testbeds are explicit, and it is possible to directly study whether the network substrate is capable of meeting the demands that are placed on it. Testbeds also have relaxed requirements for high-availability-downtime due to failure of non-redundant components represents an inconvenience, rather than a large loss of revenue.

Thus, while datacenter networks have received extensive study [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF][START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF][START_REF] Leiserson | Fat-trees: Universal networks for hardware-efficient supercomputing[END_REF], a different approach is needed for testbed networks. Facilities such as the Utah Emulab [START_REF] White | An integrated experimental environment for distributed systems and networks[END_REF], DETER [START_REF] Benzel | Experience with DETER: a testbed for security research[END_REF], and StarBed [START_REF] Miyachi | StarBED and SpringOS: Large-scale general purpose network testbed and supporting software[END_REF] have independently designed their physical network topologies subject to budgetary constraints and assumptions regarding experimenters' needs. Such decisions impact the effectiveness of the testbed. Budgetary constraints necessitate tradeoffs between factors such as the number of nodes in a testbed (affecting the number and size of experiments that can be supported) and the degree of connectivity between them (affecting the types of topologies that can be instantiated). If the decisions made during testbed design are mismatched with user behavior, the result is a testbed that either does not meet user needs or is unnecessarily expensive.

Emulation testbeds are now mature enough to provide us with a large dataset showing how experimenters use them in practice; by studying this dataset, we can learn how to build better testbeds. To this end, we have analyzed the network topologies submitted to the Utah Emulab site, the largest and most widely-used emulation testbed, over a period of eight years: from 2003 to 2011. 1 This dataset consists of over 500,000 network topologies submitted by 500 projects as they ran 13,000 experiments. From our analysis, we form several hypotheses about the most effective ways to build future testbeds, and test these hypotheses with simulations using real workloads.

We begin in Section 2 with a description of the Emulab testbed and the dataset used for our study. In Section 3, we analyze the properties of experimenters' submitted topologies, including their size and edge-connectivity. Section 4 considers key decisions in the physical design of a network emulation testbed, taking into account the cost implications of these choices. Section 5 evaluates the effectiveness of different testbed designs through extensive simulation using real user-submitted topologies. Finally, Section 6 presents the high-level conclusions that we draw from the study.

The Emulab Testbed

To run an experiment on Emulab, a user submits a description written in a variant of the ns-2 simulator language [START_REF] Fall | The ns Manual[END_REF]. Included in this description is a virtual topology defining the nodes, links, and LANs2 on which the experiment is to be run. Emulab realizes this virtual topology on top of the hosts and switches in its physical topology by loading operating systems, configuring software, and creating VLANs. 3One of Emulab's primary goals is scientific fidelity: it only "accepts" experiments for which it has sufficient capacity. Bottlenecks in the physical topology (such as links between switches) should not lead to artifacts in experiments. Because Emulab is a multi-user facility, this means that interference between simultaneous experiments must be minimized and resources already allocated to other users must be taken into account. Emulab uses assign [START_REF] Ricci | A solver for the network testbed mapping problem[END_REF], a randomized solver, to find mappings from experimenters' virtual topologies onto the testbed's physical infrastructure. assign ensures that the mappings it produces do not violate constraints such as bandwidth on bottleneck links, and attempts to maximize the potential for future mappings by avoiding scarce resources when possible. Most experiments run on Emulab use "bare hardware" with a one-to-one mapping between nodes in the virtual topology and physical hosts. For the purposes of this paper, we refer to these as "isolated experiments." Emulab also supports "multiplexed experiments," in which multiple virtual machines can be placed on each physical host. Multiplexed experiments are less popular than isolated ones because Emulab users typically want high fidelity and low-level access to hardware. The primary use of multiplexing is to run experiments that are too large to be instantiated one-to-one on the testbed's limited physical resources.

Emulab's physical topology has two networks: each node connects to both. The first is a "control" network, over which testbed control tasks such a disk loading and network filesystem mounts occur. This network connects to the Internet, and is used for remote access to the nodes. The second is the "experimental" network: this network is isolated from the outside and is re-configured as needed to create the experimenter's virtual topology. In this paper, we focus on the experimental network, as it is the one in which the experimenter's topology is embedded, and which must be sufficiently provisioned to ensure scientific fidelity. Emulab supports "delay" nodes for traffic shaping on the experimental network: when an experiment requests a link with non-zero latency or with a bandwidth value not natively supported by Ethernet, Emulab inserts a node acting as a transparent bridge. These nodes bridge two interfaces together using FreeBSD's dummynet [START_REF] Rizzo | Dummynet: a simple approach to the evaluation of network protocols[END_REF] to delay packets or limit their bandwidth. Because delaying a single link requires only two interfaces, physical nodes having an even number of interfaces can delay multiple links.

The Emulab facility at Utah has grown organically over the past twelve years: from an initial set of 10 nodes, it has now grown to approximately 600, connected by thirteen Ethernet switches. Nodes have been added over time in large, homogeneous batches, and old nodes are rarely removed from service. Most nodes have four connections to the experimental network. The rationale behind the use of homogeneous node groups is to achieve a high degree of flexibility: the more interchangeable nodes are, the more freedom there is in resource assignment. Old nodes are kept under the assumption that since network researchers are interested primarily in the network, most can tolerate older, low-powered hosts. Emulab's physical topology is tree-like: because there is no need for redundancy, there is only a single path between any pair of switches. It is built primarily with large modular switches (supporting up to 384 ports per chassis), because full bisection bandwidth can be provided for modest cost within one switch. Emulab does not attempt to provide full bisection bandwidth across its entire physical topology due to the high cost of doing so. Instead, it relies on assign to map experiments in such a way as to minimize the use of inter-switch links, and rejects experiments that cannot fit within these limits. Switch interconnects are generally built from interfaces one order of magnitude faster than the node interfaces. For example, switches providing 100 Mbps Ethernet to nodes are interconnected using four to eight 1 Gbps ports, and gigabit switches are interconnected using four 10 Gbps ports. Older Emulab nodes have all interfaces connected to a single switch, while newer ones are "striped" across multiple switches as described in Section 4.

The Utah Emulab is one of the oldest and largest facilities of its kind, and its design has influenced many other facilities. Other testbeds built using the Emulab software [START_REF]Other Emulab testbeds[END_REF], including DETER [START_REF] Benzel | Experience with DETER: a testbed for security research[END_REF], use a similar physical topology. Unrelated testbeds, such as StarBed [START_REF] Miyachi | StarBED and SpringOS: Large-scale general purpose network testbed and supporting software[END_REF] and Grid'500 [START_REF] Bolze | Grid'5000: A large scale and highly reconfigurable experimental grid testbed[END_REF], also use fundamentally similar physical topologies.

Description of the dataset

Each time a user creates, deletes, or modifies an experiment, Emulab records the action in a database and puts the resulting topology in an archive file. We extract from these records a raw dataset containing 619,504 topologies submitted by experimenters since 2003. These topologies are grouped into 22,266 different experiments: in Emulab, a single experiment may be instantiated on the testbed many times, because users typically release resources when not in active use. The experiments belong to 522 projects. A project is a group of experimenters collaborating on the same research topic, from the same lab, or in the same class.

Not every topology is relevant to our study, so we reduced the raw dataset in several ways. First, some topologies request only a single node or no network links. Such topologies are used by experimenters who are not studying the network, or to test the deployment of specific applications or operating system images in preparation for larger experiments. They are not relevant to the design of the testbed's physical network, so we ignore them for this study. Second, Emulab provides users with the ability to create or modify experiments without instantiating them on the physical hardware, and we excluded such topologies. Third, the operators of the testbed have several internal projects. As such projects are not representative of general research needs, we have excluded them. Finally, some experiments use resources that are not part of the main cluster testbed, such as wireless nodes or wide-area hosts embedded in campus or backbone networks [START_REF] Duerig | Designing a federated testbed as a distributed system[END_REF]. While important, these types of experiments have little effect on the design of the cluster network, so we removed them from our study. Table 1 details the filtering performed to produce a working dataset composed of 477 projects, 13,057 experiments, and 504,226 topologies. We begin our study by analyzing the topologies submitted by experimenters, concentrating on the properties that directly affect testbed design: the topologies' sizes, the types of nodes they request, and their connectivity.

Most experiments are small

Figure 1 shows the distribution of experiment sizes in the dataset. The first fact evident from this figure is that most experiments are small: 80% of the isolated experiments use fewer than 20 nodes. There is a small, but significant, tail of large experiments: 4% (more than 20,000 topologies) requested 100 nodes or more. The large topologies belonged to 514 different experiments in 68 different projects, so while large experiments represent a small fraction of all experiments, 14% of projects run at least one. We believe this is because many projects conduct many small experiments in preparation for a few large ones. For example, they may run several experiments to develop and debug a distributed system before evaluating it at scale. Only 5% of experiments are multiplexed, but these tend to be large: 20% have 200 nodes or more. As a result, multiplexed nodes actually represent a sizable fraction (31%) of all node requests. When a user submits an experiment to Emulab, it is "accepted" if there are enough nodes and network resources free to instantiate it. Otherwise, it is "rejected." Figure 2 shows the percentage of experiments that were accepted as a function of the experiment's size. We first observe the acceptance rate for isolated topologies is fairly low, even for small topologies: only 50% of submissions with 20 nodes are accepted, despite the fact that such a topology only uses 3% of the testbed. This shows that the testbed is used heavily, and frequently does not provide sufficient free resources to admit even small experiments. In order to cope with resource scarcity, some users turn to multiplexed experiments, deploying many VMs on the same physical node. Because they use fewer physical nodes, these experiments have a much higher acceptance rate, and even allow experiments larger than the physical testbed. Other users reduce the sizes of their experiments to increase the chance they will be accepted.

Attractive physical nodes are the bottleneck resource

The Utah Emulab facility has gone through several major hardware expansions over its lifetime. Two such expansions occurred during the time covered by our dataset: in June 2005, 160 pc3000 nodes were added, approximately doubling the size of the testbed and bringing the first Gigabit Ethernet to Emulab. In early 2010, 160 d710 nodes were added, giving Emulab its first multi-core processors and greatly expanding its number of Gigabit network interfaces.

One objective of the expansions was to support the deployment of larger experiments. As we can see in Figure 3, the expansion in 2005 was somewhat successful in meeting this goal: for nearly two years, the average experiment size grew slowly and reached almost double the pre-expansion size. Eventually, however, the modern nodes attracted more users, forcing experiment sizes back down. The expansion in 2010 was less successful in this regard for three reasons. First, this expansion increased the testbed size by only 50%, in contrast with the doubling in 2005. Second, the rollout of the new nodes was gradual, with a period of limited release before they were made available to all users. Third, Emulab had long ago reached a saturation point, with utilization of over 90% being common, so the new nodes did little to alleviate this pressure: they simply allowed more experiments to be accepted.

Emulab allows users to request specific node types (such as pc3000 or d710), or to simply request nodes of any available type. Figure 4 shows how these requests have evolved over time. The Utah Emulab facility has kept all old nodes in service after expansions, with two goals in mind: to keep the testbed as large as possible, and to support reproduction of results published using these nodes [START_REF] Eide | An experimentation workbench for replayable networking research[END_REF]. However, can be seen in the graph, specific requests for old nodes types have dropped near zero. Requests for "any" node type have declined slowly but steadily, presumably because users do not wish to be allocated the oldest nodes, which are now over a decade old. We conclude that the cause of the high rejection rates seen in Section 3.1 is not simply lack of available nodes, but more specifically a lack of attractive nodes.

Most requests use few interfaces

Emulab was built with fairly homogeneous node connectivity: with the exception of some d710s, most nodes have four interfaces available for experiments. The rationale behind this design was maximum flexibility: large blocks of identical nodes greatly simplify resource allocation. However, as we can see in Figure 5, requests using many interfaces are uncommon-the majority of requested nodes have only a single interface. This is in keeping with typical networks containing many edge hosts, which are typically not multihomed. The result, however, is that most interfaces in Emulab go unused most of the time. It is interesting to note that the popularity of 4-interface nodes grows up to experiments of size 100, then drops off sharply. Topologies at or right under 100 tend to be large trees, while topologies larger than 100 nodes are dominated by LANs, which require only one interface per node. It is also worth noting that users have clearly adapted to the physical constraints of the testbed: nearly all user requests have four or fewer interfaces. While some topologies requested five or more interfaces, the number was negligible, and is not shown in the figure. Anecdotal evidence suggests that if higher-degree nodes were available, some users would take advantage of them, but this effect cannot be measured from our dataset.

Figure 6 shows that delay nodes represent a significant proportion of the requests: up to 50 nodes, delay nodes represent 10% to 40% of the total. This not only shows that experimenters are interested in traffic shaping, but also the necessity of having numerous physical nodes with at least two network interfaces to act as delay nodes. We also observe that use of delay nodes decreases for large experiments. We believe this to be caused by two factors. First, as seen in Section 3.1, larger experiments are harder to instantiate, and removing delay nodes is one way of keeping the experiment size down. Second, multiplexed experiments are more common at large sizes, and these experiments perform traffic shaping on the hosts rather than using delay nodes. LANs are of particular interest to testbed design: because they require full bisection bandwidth, they are much more difficult to instantiate than topologies comprised of links alone. Figure 7 shows the percentage of experiments that include at least one LAN. Figure 8 depicts the distribution of the sizes of those LANs. The most striking feature of Figure 7 is the fact that LANs dominate large experiments. Even at smaller sizes, they are quite common: 40% of 8-node experiments use at least one LAN. LANs are indeed primary components in many real networks, and experimenters who want representative topologies must consider these components when designing their experiments. While some experiments consist solely of a single LAN containing all nodes, this is not the dominant use case for LANs: only 20% of the experiments that contain them take this form. Finally, we observe from Figure 8 that a majority of LANs are small in practice: only 10% of LANs interconnect 20 or more nodes. This is explained by the small average experiment size.

Key considerations for network testbed design

Having seen the types of topologies that users submit to Emulab, we now turn our attention to the design of the physical facility: a well-designed facility should be matched to the types of experiments that are submitted to it.

A cost model for physical nodes

We begin with a model for the cost of a testbed's hardware. Of course, minimizing cost may not be the only factor influencing a testbed's design. It may be desirable, for example, to support certain rare but important types of experiments, even if the monetary cost of doing so is high. Nevertheless, it is important to understand the effect of design decisions on facility cost so that these decisions can be made with full awareness of their effect on the testbed's budget. In addition, spending limited money to support certain types of experiments may limit the type or quantity of other experiments that the testbed can support.

Our cost model takes into account all factors involved in the purchase of new nodes, including not only the nodes themselves, but also network interfaces, switches, interswitch connectivity, infrastructure servers, racks, cabling, and power distribution. Prices for these items were determined from current market rates. An important consideration is that while nodes can be purchased individually, most supporting hardware must be purchased in larger units: for example, one rack may hold 20 servers, and a modular switch requires the purchase of a chassis plus some number of line cards. The result is that the average cost per node is not a simple linear function of the number of nodes, but instead exhibits a "sawtooth" pattern as seen in Figure 9(a). This figure also shows the effect of varying the number of interfaces per node: adding more interfaces both increases the cost (due to the expense of switch ports) and shortens the period of the sawtooth (as additional switches must be purchased more often).

Clear "sweet spots" are present in the graph, though their effects become less pronounced as the number of nodes increases. On a small scale, connectivity is not a large factor in average node cost: when purchasing 10 nodes, a node with four links costs only 14% more than a node with one link. At this scale, a strongly-connected testbed is warranted. At larger scales, the amortization of infrastructure costs results in a price more strongly influenced by connectivity: when purchasing 100 nodes, the difference between these two configurations widens to 30%. With a $1M budget, choosing one link rather than four allows the purchase of 100 additional nodes. As a result, large testbeds must plan their connectivity carefully to avoid spending money on unneeded interfaces.

A critical parameter to the cost model is the type and size of switches. Figure 9(b) compares two types of switches: a modular switch supporting up to 284 ports with 24-port modules, and a fixed-configuration switch supporting 48 ports. All switch ports that connect to nodes are 1 Gbps, and ports used for interconnects to other switches are 10 Gbps. The large switches have four 10 Gbps interfaces (leaving 264 ports available for the nodes), and the small switches have two. 4 Using smaller switches results in significant cost savings: 30% on average, and for a budget of $1M, a testbed built with small switches can acquire 70 additional nodes. The penalty is that the "islands" of full bisection bandwidth are much smaller, putting more demand on the interconnects between switches. As a result, a small-switch testbed will be unable to host large LANs and some strongly-connected topologies. A traditional way to increase interswitch bandwidth is to purchase faster interconnects between the switches. This strategy, however, quickly becomes expensive. Because of their large number of network interfaces and explicit communications channels, an alternative strategy is available to network testbeds. This strategy is to stripe nodes' links across multiple switches; rather than connecting all of a node's interfaces to the same switch (as shown in Figure 11(a)), a node's interfaces are distributed across different switches, as shown in Figure 11(b). In a traditional datacenter network, a node may be connected to two switches for redundancy. Interface striping in a network testbed involves more interfaces per node and has a different purpose: it serves to align the switch connectivity with communication patterns. Intuitively, the motivation for connecting two interfaces to the same switch, rather than different switches, is to take advantage of the high bisection bandwidth within the switch. Two interfaces on the same node, however, rarely have a need to exchange traffic among themselves. It is much more likely that they will be connected to interfaces on other nodes, and striping ensures that, for any pair of nodes, there exists at least one switch on which both nodes have an interface. The result is a greatly decreased dependence on interswitch links and the ability to create large LANs, potentially connecting every node in the testbed. A disadvantage of striping is that it limits the number of nodes in the testbed to the maximum available switch size. Another disadvantage is that heterogeneous interface counts become more complicated to support, as they disrupt the symmetry of the striping.

We ran a simulation to discover how much interswitch bandwidth is used in practice on Emulab. We used the physical topology of Emulab's 160 pc3000 nodes, which are connected in an unstriped manner to two large switches connected by an 8 Gbps trunk. Our simulation re-played the user requests for the pc3000 nodes: after mapping each user topology to the testbed using Emulab's assign program, we examined the amount of bandwidth allocated on the interswitch trunks. The original submission time for each experiment was preserved, and we assumed that each experiment stayed instantiated on the testbed for 24 hours.

Figure 12 shows the distribution of allocated interswitch bandwidth after each experiment is instantiated. We observe the usage is significantly below the maximum capacity of 8 Gbps: the maximum usage is 1.1 Gbps, and only 3% of instantiations result in consumption of more than 1 Gbps. The domain features of network testbeds explain this surprising result. Emulab relies on large switches with hundreds of ports to connect physical nodes. With 80% of submitted topologies being composed of fewer than 20 nodes, the probability of being able to fit a requested topology within a single switch are high. Furthermore, assign optimizes each topology mapping to keep the interswitch bandwidth usage as low as possible. This result suggests that it is worth investigating testbeds built with smaller switches. While such testbeds would rely more heavily on interswitch links, and possibly be unable to support some large experiments, the large switch topology is clearly over-provisioned in this respect. Since small switches typically cost much less than large ones, the savings would allow the testbed to add more nodes.

Evaluating testbed designs

We now combine the lessons of the two previous sections to evaluate new testbed designs: using information about users' requests and the costs of alternative topologies, we run simulations to find the topologies that are most effective on real workloads. The workload used for these simulations is a subset of the entire Emulab dataset: we use the 14,873 topologies that specifically requested Emulab's pc3000 nodes. We chose this subset to limit the runtime of the simulation, and to capture what could be considered a "sub testbed" of the full Emulab topology. This workload is processed through a FIFO batch queue and resource mappings are produced by Emulab's assign program. If resources are available, the experiment in the front of the queue is instantiated on the testbed, and those resources are marked as unavailable; otherwise, the queue blocks until sufficient resources are available. After instantiation, each experiment stays on the testbed for 24 hours of simulated time before terminating. We submit all experiments simultaneously at beginning of the simulation, and measure the total amount of simulated time until all experiments have been completed. We also measure the rejection rate: that is, the number of virtual topologies that cannot be mapped to the physical testbed under any conditions. This occurs when the physical topology being evaluated does not contain enough physical nodes, a sufficient distribution of interfaces, or satisfactory inter-switch bandwidth. This measure is important, since rejecting a large number of topologies would keep the completion time low, but indicate that the testbed does not meet the needs of at least some experimenters.

Heterogeneous node connectivity

As discussed in Section 3.3, most nodes in virtual topologies have few network interfaces, and the cost model in Section 4.1 shows us that it is possible to trade off fewer physical interfaces for more physical nodes. Choosing a distribution for network interfaces is a critical decision for improving the testbed's hosting capacity, but it is important to balance cost against variability in the workload. We evaluated the suitability of seven simulated testbeds which differ in the number of interfaces on each physical node. Each testbed was generated from the cost model using a $500,000 budget. Each has a different proportion of nodes having either two or four Gigabit Ethernet links on the experiment network. Interfaces are not striped across switches, and the inter-switch bandwidth capacity is set to 40 Gb (modeling four trunked 10 Gb ports). The configuration having only 4-link nodes is similar to that used on the Emulab facility at Utah. Table 2 details the results. As expected, the number of physical nodes in the testbed increases with the proportion of nodes that have two links. For example, with 20% of nodes having two links, the testbed is 5 nodes larger than when all nodes are 4-link nodes. Around a proportion of 80%, 4-link nodes become scarce resources and experiments start to be rejected for requesting too many nodes with more than two links. When we increase the percentage of 2-link nodes to 90%, no additional topologies are rejected, but the completion time is increased by 5.8%. The small number of 4-link nodes is sufficient to map the same set of topologies, but because these nodes are scarcer, some topologies must wait a long time for them to become free, blocking the queue. In the extreme case, a testbed comprised of only 2-link nodes rejects nearly one quarter of all experiments.

We also observe that increasing the number of nodes with two interfaces reduces the average interswitch bandwidth usage. This is explained by the increasing number of physical nodes attached to the same switch. When all nodes have four links, a 264-port switch connects 66 nodes; when all nodes have two links, 132 nodes fit onto each switch. Thus, the probability that a topology is able to fit within a single switch increases with the percentage of 2-link nodes.

In this experiment, the testbed with a 60%/40% split between two-and four-link nodes is the most efficient: the 18 additional nodes afforded by this configuration reduces the completion time by 10.8% while still being able to host every topology in the workload.

Switch connectivity

We now evaluate the alternate switch topologies from Section 4.3: striping the interfaces of each node across more than one switch, and building a testbed out of small 48-port switches. We compare to the best design from the previous evaluation, a heterogeneous testbed with a 60%/40% split. As before, all testbeds are built with a budget of $500,000. Table 3 shows the impact of a striped physical topology using two switches, with each node having two interfaces on each switch. Neither configuration rejects any experiments, and the difference in total completion time is negligible. The striped configuration uses much less interswtich bandwidth, giving it a clear advantage-only 7.7% as much on average, and 26% at the maximum. However, both configurations result in interswitch bandwidth comfortably below even a single 10 Gb link, meaning that from a practical perspective, either is capable of handing the workload in our simulation. Striping still has the potential to be advantageous if the submitted topologies are more strongly connected than those typically seen on Emulab. It can also provide a benefit if the cost of interconnects is high, as is the case when using the latest, and therefore most expensive, generation of Ethernet technology. Finally, striped connectivity is also meaningful when the ratio of node ports to interswitch ports is high. This can occur when when using large, high-density switches or small fixed-configuration switches with a limited number of "uplink" ports.

It can be difficult in practice to map virtual topologies to striped physical facilities. We repeated our previous simulation with five interfaces per node, necessitating an asymmetric striping pattern-each node had two interfaces on one switch and three on the other. The result of this experiment was a marked increase in the interswitch bandwidth used, to 800Mb. This can be explained by the fact that the irregularity introduced by asymmetric striping makes the mapping problem harder, and Emulab's solver, assign has not been tuned for this case. Making effective use of an asymmetric striped topology would require a new mapping algorithm.

We next turn our attention to the size of the switches used to build the testbed. As we have seen in Section 4.3, connectivity in Emulab experiments tends to be sparse, so small, cheap, switches may be an economical alternative to the large switches used in the Utah Emulab facility. The first two rows of Table 4 show simulations for two testbeds using 264-port and 48-port switches, respectively. As expected, the cheaper 48-port switches allows us to acquire more physical nodes, and the resulting testbed is 27% larger. However, we observe that 138 topologies (1%) were rejected as being unable to be fit on this testbed. An analysis of the rejected topologies confirms they were composed of LANs with more than 30 virtual nodes; these LANs fit on a single switch in the large-switch configuration. 85 of the rejections (60% of the total) were due to actual capacity limitations, while the remaining 57 are due to the fact that assign sometimes misses valid solutions due to its randomized, heuristic nature.

The greater number of nodes in the small-switch testbed allows the workload to complete 28.5% faster. The cost of providing a network capable of hosting sizeable LANs is an increase in the completion time. Viewed from this perspective, when spending $500,000, the use of large switches allows us to host 1% more topologies, but at the cost of increasing the completion time of the whole workload by 40%.

In order to estimate the monetary cost of supporting this 1% of experiments, we ran another simulation, shown in the third row of Table 4. This time, we used small switches, but limited the number of nodes to 148, the number in the large-switch testbed. The result was a cheaper testbed, costing only $390,000. As with the previous simulation, the use of 48-port switches restricts the testbed's hosting capacity, and 142 topologies were rejected. 5 Comparing this configuration to the large-switch testbed, we see that the completion times are similar. 6 As both testbeds have the same number of nodes, this confirms that the network is the bottleneck that causes some experiments to be rejected. The usage of small switches reduces the testbed cost by $108,000. Thus, we can estimate that supporting these 142 experiments (0.95% of the total) raises the testbed cost by 28%. Put another way, the "easiest" 99% of experiments cost on average $33.5 each, while the "hardest" 1% cost $760.7 each. Note that these experiment costs should be interpreted as illustrative, rather than true costs: since our simulations only cover 6 years (less than the useful life of a node) and use a simple model for experiment duration, they should not be considered definitive costs.

The conclusion we draw from these simulations is that testbed design should take into consideration the relative costs of supporting certain types of experiments, and the research value of such experiments (in particular, large LANs) should be weighed against their increased infrastructure costs.

Conclusions

From our analysis of user topologies and simulations of alternative physical topologies, we draw the following conclusions:

While experiments requiring few nodes dominate testbed usage, there is also a significant contingent of users who want to deploy experiments that are quite large. The latter class of experimenters are likely limited by the number of nodes available on a shared-use testbed. The fraction of projects using at least one large experiment is greater than the fraction of experiments that are large, suggesting that projects use many small experiments to prepare for a relatively small number of large trials.

A physical testbed topology need not have full bisection bandwidth; on the contrary, a testbed is able to meet its users' needs with a surprisingly small amount of bandwidth between switches. This makes building testbeds out of small, cheap, switches more attractive than expected. The exception to the low edge-connectivity rule is large LANs, which are employed by some experimenters. This implies that a testbed built from small switches may find value in one large switch to support these LANs.

Striped topologies reduce the need for inter-switch connectivity. Though it is possible to build a testbed with modest bisection bandwidth even without striping, a striped configuration aligns the physical topology with common communication patterns, and can reduce the need for inter-switch bandwidth even further. The tradeoff is a more difficult mapping problem, requiring further work to use this configuration with maximum efficiency.

Nodes with few links are prevalent in users' requests. The result is that a homogeneous testbed, with an equal number of physical interfaces per node, makes inefficient use of resources. By varying the number of interfaces per host, it is possible to build a larger testbed, a cheaper testbed, or one with a significant number of high-degree nodes.

The savings from heterogeneous node connectivity vary with the scale of the testbed. For large testbeds, many costs amortize, and the cost of network ports becomes a dominant factor. Giving some nodes fewer interfaces results in substantial cost benefits. For small testbeds, this effect is much less pronounced, and a homogeneous testbed is preferable for its flexibility.

Some experiments require a more expensive testbed than others, so care should be taken when planning a testbed to decide how valuable these experiments are. In our simulations, we found that if Emulab were to intentionally exclude the 1% most "difficult" experiments, it could save 25% of hardware costs or could build a 32% larger testbed.

While users know what topologies they want, they also quickly learn the limitations of the testbed, and tend to adapt their expectations to what is realistic to instantiate on it. For example, experimenters rarely request more network interfaces than Emulab nodes have. This is in keeping with earlier studies finding similar effects on the Internet infrastructure: the degree of connectivity of Internet routers is heavily influenced by available commercial offerings [START_REF] Li | A first-principles approach to understanding the Internet's router-level topology[END_REF]. Experimenters also decrease the sizes of their experiments when nodes are scarce.

Multiplexed experiments are used primarily to support very large topologies. Though multiplexed experiments represent only 5% of submitted topologies, their large average size means that they account for 31% of all nodes allocated by Emulab, making them an important resource. The implication is that most users prefer "bare hardware," but are willing to tolerate virtualization in order to run large experiments. Testbeds aiming to appeal to the widest possible userbase should support non-virtualized experimentation, those aiming to support large experiments should support virtualization, and for maximum flexibility, it is desirable to support both.

In the Utah Emulab facility, demand always exceeds capacity, and expansions to the testbed hardware are met with increased usage. As a result of Emulab's "first come, first serve" policy, users tend towards small experiments. In order to support larger experiments, a testbed would need to be either significantly over-provisioned (and thus expensive and inefficient), or to implement allocation policies (such as advance scheduling) that favor large experiments.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Distribution of the number of nodes in experiments (note log-scale X axis)

Fig. 3 .Fig. 4 .

 34 Fig. 3. Average isolated topology size; shaded region is the 95% confidence interval

Fig. 5 .Fig. 6 .

 56 Fig. 5. Network interfaces per node

Fig. 7 .Fig. 8 .

 78 Fig. 7. Percentage of topologies having LANs

Fig. 9 .

 9 Fig.9. Cost per node to build a testbed using a budget of up to $1M

Fig. 11 .

 11 Fig. 11. Two network topologies connecting eight nodes, each with two gigabit interfaces

Fig. 12 .

 12 Fig. 12. Interswitch bandwidth between the switches connecting the pc3000 nodes

Table 1 .

 1 Refinement performed on the raw dataset to form the working dataset

		Projects	Experiments	Topologies
	Raw dataset	522	22,266	619,504
	-empty topologies			553
	-single node topologies			27,983
	-unconnected nodes			24,454
	-non-instantiated topologies			32,260
	-internal projects	9	1,936	16,004
	-non-cluster resources			14,024
	Working dataset	477	13,057	504,226
	3 Properties of experimenters' topologies

Table 2 .

 2 Simulation results for seven generated testbeds that differ in the proportion of twointerface and four-interface nodes. The "0%" case is similar to the topology of the Utah facility

	2-link nodes	Phys. nodes	Interswitch Bw. avg. max.	Rejections	Time (days)
	0%	130	1.46 Gb	9 Gb	0 (0.0%)	1564
	20%	135	1.41 Gb	10.1 Gb	0 (0%)	1510
	40%	144	1.27 Gb	8 Gb	0 (0%)	1421
	60%	148	1.11 Gb	8.1 Gb	0 (0%)	1394
	80%	159	223 Mb	3 Gb	33 (0.2%)	1405
	90%	161	30 Mb	4 Gb	33 (0.2%)	1487
	100%	165	12 Mb	1 Gb	3,532 (23.8%)	995

Table 3 .

 3 Simulation results for two generated testbed topologies, with and without interfaces striped across the switches

	Network configuration	Interswitch bw avg. max	Rejections	Time (days)
	Unstriped	1.11 Gb	8.1 Gb	0	1394
	Striped	85 Mb	2.1 Gb	0	1392

Table 4 .

 4 Simulation results for testbeds using an 60%/40% split of two-and four-link nodes. 264port switches and 48-port switches are connected through 40 GB and 20 GB uplinks, respectively

	Switches	Nodes	Interswitch bw avg. max	Rejections	Time (days)	Cost
	264 ports	148	1.11 Gb	8.1 Gb	0 (0%)	1394	$498,796
	48 ports	186	1.06 Gb	8.1 Gb	138 (0.9%)	996	$498,354
	48 ports	148	883 Mb	9 Gb	142 (0.9%)	1314	$390,268

Topologies from Emulab's first years,

2000-2002, were not saved in enough detail for analysis.

For Emulab's purposes, a LAN is defined as a clique of nodes with full bisection bandwidth.

In this paper, we concentrate on Emulab's most heavily used resource: a "cluster" testbed made up of PCs and Ethernet switches. Emulab also supports a wide range of other physical devices, including wireless links, network processors, wide-area hosts and links, and Layer 1 switches.

These switch configurations, and their prices, are based on current HP ProCurve products.

The difference with the previous experiment is explained by the nondeterminism of assign.

The slightly higher time for the large-switch testbed is explained by the fact that it runs more experiments.

Heterogeneous node connectivity enables larger testbeds

We now expand our model to consider configurations in which nodes have heterogeneous connectivity: some nodes have more interfaces than others. To do so, we ran a series of simulations. In each simulation, a set of experimenters' virtual topologies are mapped onto a physical testbed of infinite size. Nodes in testbed have differing numbers of physical interfaces, and each virtual node is mapped to a physical node having the minimum number of interfaces needed to satisfy it. We look at the resulting distribution of physical interfaces used by the mapping; this gives us a rough guideline of how many nodes of each type to include in a heterogeneous testbed. Figure 10(a) shows these distributions for three different testbeds: one having one to four interfaces per node, another having one to six, and a third having one to eight. On average, 67% of the physical nodes used have one interface, 20% have two interfaces, and very few nodes with more interfaces are needed. This correlates with our earlier statistics in Section 3.3, showing that nodes with few interfaces dominate testbed use.

We then construct testbeds using our previous cost model with different nodes having different numbers of interfaces. We use the distribution from Figure 10(a) to select the proportion of the physical nodes having each number of interfaces. Within a fixed budget, more nodes can be acquired, as the majority will only have a few links. Figure 10(b) compares the size of a homogeneous testbed with testbeds built using the heterogeneous distributions from Figure 10(a). The heterogeneous testbeds enable us to acquire 25% more nodes using the same budget. The gain is slightly reduced when the testbed is composed of nodes having more than four links. However, the loss is not significant as the number of high-interface nodes is small. Nodes with six or eight experimental links may be still useful in Emulab to support future users' needs.

Alternatives for switch connectivity

Because providing full bisection bandwidth across the entire network is prohibitively expensive, bandwidth between switches (interswitch bandwidth) becomes a critical resource in distributed infrastructure [START_REF] Al-Fares | A scalable, commodity data center network architecture[END_REF][START_REF] Guo | BCube: a high performance, server-centric network architecture for modular data centers[END_REF]. In a network testbed, it is important to avoid oversubscription of these bottleneck links, which can lead to capacity artifacts and harm


Valérie Fiegenwald
email: valerie.fiegenwald@grenoble-inp.fr

Celine Cholez
email: celine.cholez@upmf-grenoble.fr

Thomas Reverdy
email: thomas.reverdy@upmf-grenoble.fr

Sub-theme 15: Transboundary Risks and Crisis Management Management of transboundary risks in a low-volume industry: the role of boundary objects and boundary spanners

Introduction

Today's industrial companies face a paradox: they must be innovative in order to hold onto their markets and at the same time perform from an operational point of view. This requirement sees them embarking on improvement approaches such as Lean management to reduce organizational slacks. The organization becomes leaner in its processes but routines are less and less present in their standard operations, meaning that day-to-day uncertainties increase. This situation exposes them to industrial crises.

This article looks at the management of industrial crises in a company making bespoke equipment for the energy sector (low volumes, high customisation) and which has for a year now faced a series of serious quality incidents resulting in production stoppages which potentially impact the functioning of the equipment. These incidents have generated a feeling of uncertainty internally as to control and the reliability of the production process.

Analysis of these incidents reveals failures in the articulation of the different stages of the process which are segmented into major, relatively partitioned, functional sectors. This article therefore contributes to thinking on transboundary risks, taking into account internal boundaries in an organization.

We can talk about "transboundary risks" because of the diversity of venues where deviations occur which contribute in the end to a failure, the possible propagation of failures throughout the process and finally, the necessary work of re-articulation carried out by those involved to rectify such incidents.

Purpose of the paper

Recent theories on risks [START_REF] Schulman | The Negotiated Order of Organizational Reliability[END_REF][START_REF] Barton | Overcoming dysfunctional momentum: Organizational safety as a social achievement[END_REF][START_REF] Carroll | Naturalistic Decision Making and Organizational Learning in Nuclear Power Plants: Negotiating Meaning Between Managers and Problem Investigation Teams[END_REF], and resilience [START_REF] Tillement | Organizational resilience at the limit: projects integrating new technologies in complex operational systems[END_REF][START_REF] Hollnagel | Risk + barriers = safety?[END_REF], present flexibility, "DIY" and improvisation [START_REF] Weick | The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster[END_REF] as conditions which allow organizations to better face up to risks and unforeseen events. In the case of transboundary risks, this improvisation requires the intervention of "astute individuals" who are able to promote the circulation of information, and "fruitful interactions" between organizationally distant actors [START_REF] Kapucu | Interagency communication networks during emergencies[END_REF][START_REF] Adrot | Interagir pour improviser en situation de crise[END_REF]. The roles of boundary spanners and boundary objects are discussed here, in the causes behind and rectification of incidents. In the case presented here, coordination breakdowns between departments have for a long time been compensated for by boundary spanners who repair deviations, retrieve information and facilitate the circulation of information, drawings and objects to enable production continuity.

They act as an interface between disjointed universes. It is they who detect a large number of errors and incoherencies. In this sense they are a strong pillar of the organization's resilience.

However, the case that we present questions the limits of resilience based only on the flexibility of those actors: we show that when studied in accidental situations in particular, this flexibility can lead, in crisis cases linked to an accumulation of incidents, to the actors involved losing control. A disorganised environment that has too many "cognitive attractors1 " wears actors down through a series of minor urgent tasks. This phenomenon is described by [START_REF] Lahlou | Attracteurs cognitifs et travail de bureau[END_REF] about intellectual workers (R&D). In this paper we focus on the cognitive overflows to which boundary spanners are subjected when trying to resolve transboundary unexpected disruptions.

Theoretical background

Our paper fits into a systemic approach of risks [START_REF] Perrow | The limits of safety: the enhancement of a theory of accidents[END_REF], linked to the organizational complexity. According to this framework, the organization is both a source of risks and of reliability; it is an "open system" whose functioning is based on the exchanges between its different components.

The division of work creates uncertainties, disruptions and unforeseen events

The problems of coordination and partitioning between activities are conventional problems addressed by organization theory. Many studies have focused on the sharing of knowledge [START_REF] Nonaka | A dynamic theory of organizational knowledge creation[END_REF] and have demonstrated that specialisation linked to the division of labour implies the development of different perspectives on the organization of operational modes [START_REF] Bechky | Sharing Meaning Across Occupational Communities: The Transformation of Understanding on a Production Floor[END_REF]. Other studies have looked at the power struggles (Crozier and Friedberg 1977, vol. 436) between functional departments. Finally, authors such as [START_REF] Strauss | Work and the Division of Labor[END_REF]) have looked at discontinuity problems linked to the division of labour. Activities are segmented, which can create disruption, incoherence and loss of information but also uncertainty as to task-related responsibilities.

Disruptions are components of the organizations' daily life, reliability is to cope with them

Disruptions and their informal arrangements have for a long time been seen as a problem to be eliminated. More recently, new approaches of risks present irregular variations and degradation of expected working conditions as a component of the organizations' daily life. This theoretical change is partly due to the works developed within the framework of the studies about the organizational resilience (Weick 2004;[START_REF] Barton | Overcoming dysfunctional momentum: Organizational safety as a social achievement[END_REF][START_REF] Hollnagel | Epilogue: Resilience engineering precepts. Resilience Engineering-Concepts and Precepts[END_REF]. It can be seen as a reversal of the classical perspectives about the control of risks because it means that reliability is not the absence of unforeseen events and variations, but the ability for an organization to take in charge "the irregular variations, disruptions and degradation of expected working conditions" [START_REF] Hollnagel | Epilogue: Resilience engineering precepts. Resilience Engineering-Concepts and Precepts[END_REF] and to cope with unanticipated dangers and uncertainties [START_REF] Douglas | Risk and culture: An essay on the selection of technical and environmental dangers[END_REF]. This means it is necessary to study the daily work activity and the way the different members of the organization take in charge problems they meet in the course of their normal operations; to identify the mechanism they are able to develop to rebuild some order to avoid the accident. The comparison between the observation of the "normally disturbed activity" and the analysis of situations of completely unforeseen events, when sensemaking breaks down [START_REF] Weick | Collective Mind in Organizations: Heedful Interrelating on Flight Decks[END_REF] is particularly interesting to identify conditions of organizational resilience [START_REF] Tillement | Etude du rôle des dynamiques intra et inter -groupes professionnels dans la maîtrise des risques[END_REF]. The question arises as to the way people hold on foreseen or unforeseen events.

This approach suggests paying attention to collective learning processes and highlighting the importance of flexibility

According to [START_REF] Weick | Managing the unexpected: Assuring high performance in an age of complexity[END_REF], two factors can ensure a relative reliability:

-a collective vigilance defined as the ability for a group to detect and anticipate errors thanks to a relative reluctance to simplification and an operational matter; -a resilience ability that presupposes a sense of DIY and wisdom in actions and interactions.

Flexibility, ability to interpret weak signals and to immediately correct errors and to learn from these errors are the keys of reliability. In crisis situations, actors have to rapidly adapt and improvise [START_REF] Weick | The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster[END_REF]. Improvisation corresponds to an adaptation process [START_REF] Vera | Improvisation and innovative performance in teams[END_REF] during which individuals must "make with" available resources (e Cunha, da [START_REF] Cunha | Organizational improvisation: What, when, how and why[END_REF], and combine them in an innovative manner [START_REF] Rerup | Houston, we Have a Problem[END_REF]) in a quasi simultaneousness of decision and action [START_REF] Moorman | Organizational Improvisation and Organizational Memory[END_REF]. Improvisation is often associated with the concept of "DIY". It is, however, different from "DIY" because of the very short timelines between decision and action. Improvisation can raise several problems: it can create complexity, subject actors to emotionally strong pressures [START_REF] Chédotel | L'improvisation organizationnelle[END_REF] and not necessarily produce a coherent, collective action. Thus [START_REF] Adrot | Interagir pour improviser en situation de crise[END_REF] look at "how to improvise together in a crisis event" within an organization.

Crossing boundaries to re-articulate work and build a multi-situated viewpoint

In the course of normal activity as facing unforeseen events, organization break-downs always need to be reduced. According to Strauss, alignment is always necessary to ensure business continuity and what Strauss calls the work of articulation, which must accommodate the different actors whilst the action is underway. Articulation requires negotiations and arrangements. Actors will align their definitions of the situation, or at least make them compatible around a shared objective. This articulation work can be the responsibility of individuals (boundary spanners) or objects (boundary objects) which allow meaning and language to be shared, along with the alignment of practices, learning and people's understanding of other actors' roles. [START_REF] Star | Institutional ecology,'translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology[END_REF] developed the concept of boundary object to analyse the nature of cooperative work in the absence of a consensus. Boundary objects are the ingredients of action, a mean of "representing, learning about and transforming knowledge to resolve the consequences that exist at a given boundary" [START_REF] Carlile | A pragmatic view of knowledge and boundaries: Boundary objects in new product development[END_REF]Carlile 2004). Adaptability o f the object and its relative interpretative flexibility promotes various buy-ins, transformations and adaptations by social groups who are prepared to cooperate in a permanent to-ing and fro-ing between the consensual form and its deviations. They "provide a lingua franca for exchanges" and facilitate cooperation [START_REF] Star | Institutional ecology,'translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology[END_REF].

The notion of boundary spanning individual was developed by [START_REF] Tushman | Characteristics and External Orientations of Boundary Spanning Individuals[END_REF] as part of development projects. These individuals are described as an effective resource for collecting and transferring information between organizational boundaries. The phenomenon of boundary-crossing is often informal and confers power and status upon the boundary spanner. The role of the boundary spanner is to (re)construct alignment [START_REF] Cholez | Assessing organizational resilience: an interactionist approach[END_REF]. Its activities of rectification and informal coordination have been identified by crisis specialists as factors of resilience. According to [START_REF] Adrot | Interagir pour improviser en situation de crise[END_REF], in a highly uncertain situation, the boundary spanner contributes to coherence between parties of the organization through generation of fruitful interactions between actors giving them both an action framework and room for manoeuvre.

Boundary spanners and boundary objects could organize the emergence and the achievement of a multi-situated attention that guarantees a better understanding of the complexity of the situations. Opposing to Perrow who associates complexity with increase of the risks, Weick asserts that to reduce risks, the organizations should complicate themselves to encourage redundancies. Multi-disciplinary groups, cross-department teams should be encouraged to get people used to interact and communicate [START_REF] Brion | La coordination par la vigilance collective réciproque[END_REF].

The approach taken

The field study was carried out within a plant of a global powerhouse in electronics and electrical engineering employing around 500 persons including 200 operators. The plant works to order, manufacturing customised equipment meeting the needs of each customer. This means great variation in terms of design and assembly. Production operations involve manual assembly. There is very little automatic equipment involved in the production process (handcraft industry).

There are two distinct production lines which assemble their own products. Each line has its own quality team which is responsible for the final control and validation of products before shipment. In addition, the plant has a centralised quality team responsible for employees' training and respect of assembly procedures throughout the process for both lines.

Our research is based first of all on operational immersion in the company under study.

Through 40 training sessions in Lean Manufacturing and running different workgroups on assembly errors and assembly documents, we noted and analysed data concerning both quality problems and dysfunctions regarding articulations between departments. We went deeper into this data collection through semi-directive interviews on the roles of different actors in the management of quality problems and interactions with other departments in crisis situations. We questioned key actors from the different departments in problem solving and crisis management (Technical, Design Office, industrial, logistics, production and quality) who occupy different levels in the hierarchy.

We also experienced 3 weeks' total immersion on a production line to identify in detail all the barriers to control, redundancies and control breakdowns which can lead to quality errors.

The data collected within this framework come from our own observations and interviews with assembly operators and team leaders.

In parallel, we also performed detailed analyses of fifty or so formal quality reports, describing quality incidents which had occurred in 2009 and 2010.

Main findings

We should first of all note that the equipments manufactured in the studied plant are extremely complex and sensitive what entails numerous constraints for assembly operators.

The least speck of dust can, for example, damage the product. An assembly error can have serious implications in terms of safety.

The studied company works in project mode. It develops and produces an entirely customised piece of equipment at the customer's request which requires specific design before manufacture can begin.

This process is described in the following diagram: The different steps in the process correspond to different teams, located at the same production site. A project manager for each order follows the project from the offer to the on site installation and is the official customer interlocutor. He is also in charge of the coordination between the different teams but this coordination is essentially done via documents (offer, customer specifications, minutes of meetings, drawings, etc.). The project manager is thus more focused on external coordination.

The complexity of the product and the high level of customisation for each customer bring new uncertainties for each activity of the process for each project. Actually, customer requirements induce design specificities for each project. These specificities lead to specific assembly drawings, that means little standard assembly procedures and little routine for the operators. Customer requirements can also concern components' quality, for example use of specific screws which can easily be confused.

I-FACTUAL FINDINGS

A-Generation of incidents

The operational immersion and the analysis of quality reports show that the company is regularly confronted to disruptions that alter the process flow. We can say that the company has a "normally disturbed activity"; disruptions are actually components of its daily life.

We can talk about "transboundary risks" first of all because of the diversity of venues where deviations occur which contribute in the end to a failure. Each process (offer, design, logistics, manufacturing, installation) is a potential non-conformity generator. This phenomenon is amplified by the singularity of customer specificities for each project. For example, errors can occur in the offer and be detected during the specific design phase.

Design errors can also occur and be detected during the manufacturing phase.

Secondly, we talk about transboundary risks because of the possible propagation of failures throughout the process. These defects may actually propagate beyond the borders of the stages because they are not systematically detected. There are control checkpoints but they seem to be somewhat porous. Errors are detected when they become blocking for a department. Thus it is during the assembly phase that many errors are identified. Indeed, an analysis of quality reports showed that 70% of problems detected during assembly were generated beforehand, and were therefore detected outside the department they stemmed from. Of the remaining 30%, cause analysis shows that half may be due to incomplete or unclear information, in particular in assembly documents.

Finally, the term transboundary refers to the necessary work of re-articulation carried out by those involved to rectify such incidents.

B-Their management throughout the process: late detection and rectification

Like in every organization rectifications occur and avoid accident and crisis. When analyzing the management of failures and disruptions we observe two steps in the solving process: local and transboundary.

The local solving is undertaken by the operator/team leader pairing. This occurs upstream from the standard process for dealing with non-conformities and is an exercise to qualify the problem to determine whether it can be rectified or not, i.e. whether they are going to be able to manage it on their own. The second step in the solving process is a transboundary management of the incident. The team leader will then manage the problem by contacting other actors, but not necessarily the Quality team, although a fault detected by an assembly operator is supposed to be passed on to Quality via the team leader.

The team leader has different action possibilities which result in different types of interventions. Table 1 shows the action range of the team leader who will for example do informal searches for missing information especially concerning misunderstanding or unclearness of documents. In this case he will address directly to the technical support, design teams or industrial teams. Another intervention is the pressure on upstream departments to rectify the failure. For example in case of missing parts, he will urge the prefabrication line to the problem as quickly as possible.

Actually, the team leader will attempt to reduce the number of incidents which have to be reported to the Quality department.

Intervention types Examples

Search for missing information in other

Table 1 : Intervention of the Team Leader

Team leaders act by their permanent involvement as fire-fighters at the heart of the de rearticulation work which is necessary to the process. Why, then, do these actors not pass on all the information? First of all, the team leader does not have a global overview of the project and may sometimes underestimate the potential impacts of the problem on the organization, as well as the consequences of his isolated corrective action. Then, he may hesitate to report a problem because he fears delays to processing or even a production stoppage.

C-Incidents that are not rectified

In spite of all the filters put in place by the organization and the permanent involvement of team leaders, certain problems are not rectified.

Consequences for the company can be serious with regards to costs, lead times and corporate

image. An incident that has not been rectified can correspond to an incident that stops production, an incident detected during on site installation or material in use, or to a personal accident.

These incidents are not much different from those that are rectified

Of course some non-conformities could not have been rectified, because they are undetectable before on site installation. A conception error in the global station architecture for example will be blocking only for on site assembly. But when we look at them, we realise that most are not different from those that are rectified (constituent defects, documentary defects, assembly errors, unavailable or unsuitable tools), the only difference being that they have not been detected. Indeed, we observe that detection of non-conformities is often a "chance" discovery, i.e. outside the framework of formalised controls. The company is therefore relying on the vigilance of actors.

These incidents can be due to rectifications

Furthermore, informal corrections are local and occasional adjustments which do not guarantee fundamental resolution of the problem and may even lead to undesirable consequences or deviations in practices because of the application of a new method with local but not transversally effective solutions. This can occur for example when a team leader requests an isolated technical derogation on a constituent or method. The technical services involved may accept the derogation for the particular case. Generalizing this principle may not be suited for other cases and applying it could lead to a series of other incidents.

Incidents can lead to a disaster when propagating to the customer

A crisis will emerge through an accumulation of these incidents which propagate along the process: the long period between the generation of a defect and its detection may mean that several products have been assembled and therefore potentially impacted. The response of the company is then to send experts on site to repair the defective material, analyze the root causes and the origin of the failure to identify potential risks on materials assembled in the meantime between generation and detection, in order to finally verify this material.

D-Faced with a crisis, mobilisation of Manufacturing Quality Assurance

In an attempt to resolve the crisis the company is experiencing, the Manufacturing Quality Assurance department was asked to construct collective and transversal mechanisms for incident analysis. It coordinates the implementation of corrective and preventative actions by creating areas of exchange between actors from different departments who work directly on the shopfloor, around a board.

The department also developed transversal working groups to make people from different department exchange around major transboundary issues. Another exchange area is the technical trainings for operators and especially for non-operators who can here experience the assembly constraints and stakes. The department also conducts Quality and Lean trainings for all employees. These massive training campaigns were privileged exchange moments between people with different backgrounds and helped aligning them around common objectives. All these multi-disciplinary methods and tools are set up by the Quality department in order to rebuild the multi-situated view necessary to understand complex situations and to achieve organizational reliability.

II-ANALYSIS

A-Boundary objects are not reliable and do not frame the actions of actors

We saw above that documents should ensure work continuity and articulation. However they often do not and can even be a trap. Actually the multiplicity of documents which emanate from different sources and whose update procedures are complex and less reliable often trigger errors. Assembly documents actually lack harmonisation and coherence.

They could be boundary objects. However, they do not facilitate the transfer of information between departments. Indeed, the large number of assembly documents and their different origins create problems of coherence for final users. The documents are issued at different points in time during the product life cycle, also from different departments, what compromises overall coherence. Furthermore, document updating is not handled effectively by the organization because of the diversity of modification channels. Attempts at simplifying assembly documents have for the moment proved unproductive. The Quality department actually tried to conduct working groups on documents harmonization and simplification. Each department is convinced of the usefulness of the documents it issues and is not prepared to discuss the issue. The tendency actually seems to be the creation of even more documents (kit list, safety data sheet, etc.).

B -The boundary spanner is overflowed in work which makes him less vigilant

In our case study, the team leader plays the role of boundary spanner. It is he who literally crosses interfaces to manage day-to-day problems in production. His role is that of a firefighter who is in fact always reacting to emergency situations. He is extremely flexible and adaptable: he is a "DIY" specialist. The responsibility scope of the team leader is wideranging. He supervises an average team of twenty operators. This is four times more than recommended by Lean theories. Three team leaders cover the whole assembly of one product.

He is permanently solicited from all sides and is therefore completely overrun with work, which of course reduces his vigilance and increases his fatigue and stress. He also has pressures in terms of deadlines because it is he who manages the production schedule of his workshop and is the guarantor of deadlines being respected. Thus, as described by (e Cunha, da Cunha, and Kamoche 1999), management of unforeseen situations with severe time constraints can be a strong source of anxiety for those involved. The flexibility of those actors can lead to their becoming submerged. This situation also results in a general feeling of fatigue. However, it is in their day-to-day management of unforeseen circumstances and improvisation that they find their legitimacy. They have a certain power coming from their "undercover" problem solving capacities. But they are also acting alone because they lack legitimacy when facing the different support services. Indeed, their requests are often qualified by these latter as comfort modifications and refused.

C-The boundary spanner does not necessarily have the capacities to detect all deviations

He can only imagine failure scenarios according to past experience, which by definition is limited. His capacity to represent failure scenarios and maintain his vigilance is limited. His capacities of anticipation are also limited.

The team leader acts mainly in emergency situation, coping with the daily failures of its workshop. Solving urgent problems does however remain primarily superficial because the team leader has neither the time to analyse the problems to find the deep root causes, nor the necessary legitimacy to put in place long term solutions to transboundary problems at an organizational level. As described by [START_REF] Pina | Organizational Improvisation and Leadership-A Field Study in Two Computer-Mediated Settings[END_REF] he can fall in an "opportunity trap" meaning that he does not manage to reuse knowledge acquired during the exploratory process of organizational improvisation. Analysis and implementation of long term actions is not his job, but that of Quality and Manufacturing Quality Assurance team leaders. The boundary spanners do not have the legitimacy or the negotiating capacity to make their voices heard in terms of their perception of risks, and in particular when it comes to asking another department to reduce discrepancies relating to "risk-free" operations. The qualifier "comfort modification" by the designer of the modification requests from the production shows this different perception of risks and responsibilities, and the associated negotiated relationship.

However, the isolated management of dysfunctions stops information on those problems getting back to the appropriate departments. As a result, the dysfunction is very likely to recur. Thus, the team leader, who is an essential part of the organizational resilience can also limit organizational learning.

D-How to re-establish order and coherence in crisis management -use of third parties to ensure more stabilised coordination

Flexibility is an essential component of crisis management. However, as above-mentioned, flexibility also be counterproductive and does not allow sustainable problem solving.

That means that a minimal formalisation, as well as coherent and centralized articulation work are necessary.

Manufacturing Quality Assurance is well placed to play a role in the articulation work. The department has the posture of an outside party with a global and objective view of the situation, as well as the required legitimacy to coordinate action. It organises confrontations of points of view, creates negotiation and arrangement platforms between actors through crisis meetings or workgroups and tables long-term solutions through the creation of transboundary objects such as the unforeseen situation management board. The success of these methods is due to the transversal approach coordinated and animated by a neutral actor, the Quality department. Through its third party position, the department has the legitimacy to bring all actors together in order to find solutions. Moreover it coordinates actions and follows their executions. Doing this, the department has really become a boundary spanner.

Discussion

Flexibility and "DIY" are presented in recent theories on risks as essential components of resilience. The case we have presented in this paper shows that the cleverness of the team leader, his ability to cross the boundaries to find information and to negotiate arrangements allow to solve many problems in the short term. Most of the daily degradations of the productive activity are corrected; the team leaders and their teams achieve the recovery of a relative order in the process and avoid most of the products' accident that could occur. The team leaders gains power and a relative legitimacy for this and are considered as fire-fighters vital for the productive process.

But, this case also shows that an excess of flexibility can be counter-productive and mean that problem solving cannot be made long term. We discuss here the limits of this type of resilience in the case of crises resulting from a succession of incidents which occur over a long period of time. It would therefore appear that a minimum of formalisation is required, as well as coherent and unified articulation, which is recognised by all. Even though the definition of strict standards may not seem suitable, an action framework and legitimate coordination seem to be necessary to find long-term solutions to crises. As described by [START_REF] Adrot | Interagir pour improviser en situation de crise[END_REF], individual improvisation is not desirable in the management of transboundary crises. Organizational improvisation requires coordination between all actors involved.

This case also allows examining the hypothesis which considers that the observation of the good recovery of the disruption, performed by the teams in the daily activity, can predict the resilience of the organization. an observation focused on the daily work of the productive teams could conclude to their relative resilience. But the study of the incidents suggests that this resilience is limited and can be a cause of disruptions and unawareness within the organization. Then, it shows the importance to have a scale of observation (temporally and organizationally) large enough to be able to identify some distinctions between local and transboundary resolutions, short term and long term learning.

When we study the scale of the interactions between the productive teams and the other departments of the organization, we meet some other limitations of the resilience of the existing system based on boundary spanners individuals: limitations linked to their ability to run beyond the competitions between what can be named some organizational territories organizational jurisdictions [START_REF] Bechky | Sharing Meaning Across Occupational Communities: The Transformation of Understanding on a Production Floor[END_REF]. Indeed, the notion of boundary evokes coordination problems and breakdowns in understanding that can occur inside and outside the organization. A negotiation exercise is therefore required to effectively accomplish collective activities. This articulation work is crucial, in particular during crisis periods. The question of who takes responsibility for it however is to do with issues linked to legitimacy.

The analysis of the overflow of the documentary process and of the team leaders reintroduces the issues of legitimacy as a condition of their effective action through boundaries [START_REF] Strauss | Work and the Division of Labor[END_REF]. The literature has focused on the skills of communication of the boundary spanning who should speak several languages to facilitate the circulation of information across boundaries. In the case that we have presented, the team leaders also try to re-articulate the activities with many actions that can consist in a re-alignment of the tasks of the different services involved. But if the emergency of the situation allows these temporary arrangements, it does not mean that the team-leaders are legitimate enough to be heard when they ask for permanent improvements linked to the infrastructure of the process (Star 2002;[START_REF] Star | The structure of ill-structured solutions: heterogeneous problem-solving, boundary objects and distributed artificial intelligence[END_REF] .

Comparing the responses of the team leaders and of the Manufacturing Quality Assurance, we suggest a new dimension of the boundary spanner's activity, i.e. the creation of negotiation zones.

Finally, we consider that this case suggests the fertility to associate an approach based on the observation of daily activity and situations [START_REF] Weick | Managing the unexpected: Assuring high performance in an age of complexity[END_REF]) and a study the interactions between the different groups that compose the system that is studied [START_REF] Tillement | Etude du rôle des dynamiques intra et inter -groupes professionnels dans la maîtrise des risques[END_REF].

Conclusion

The notion of boundary evokes coordination problems and breakdowns in understanding that can occur inside and outside the organization.

A negotiation exercise is therefore required to effectively accomplish collective activities.

This articulation work is crucial, in particular during crisis periods. The question of who takes responsibility for it however is to do with issues linked to legitimacy.

Flexibility and DIY are presented in recent theories on risks as essential components of resilience. We have shown the limits of this type of resilience in the case of crises resulting from a succession of incidents which occur over a long period of time. Thus, flexibility can result in actors who play the boundary spanner role becoming submerged, making it counterproductive. Even though the definition of strict standards may not seem suitable, an action framework and legitimate coordination seem to be necessary to find long-term solutions to crises.

Management of this form of "accumulative" crisis seems to be an interesting avenue of exploration for future research on resilience in distributed action situations.

Figure 1 :

 1 Figure 1: Project delivery process

  to an upstream department Anticipation request to the upstream assembly team

Prefabrication, Preassembly, Assembly & Testing Production Engineering, Special design (SOB) Order Management Logistics, Sourcing GIS parts Packing & Shipping (ext. Company)

					Project delivery Project delivery
	Acquisitio n Go/No Go decision	Start project	Entry order clarified			Approval of detailed planning		Material and resource s at site	Construction/ Installation completed	Commissioning completed	Provisional Customer Acceptance
					Steelwork	Civil work	Construction Planning / Execution
	Offer		Set -up	Basic Design	Detail Engineering	Manufacturing Dispatch	Installation	Commissioning / SAT	PAC
				Approval Basic desig	Approval Typical	Equipme ord	FA
							Manufacturing,

Table 2 :

 2 Boundary objects

	Boundary objects	Update issuing department	Update channel	Shopfloor update
			Line 1 : email to the
			line supervisors,	Line 1: Document
	Technical drawings	Technical support	industrials, document preparation assistant	preparation assistant Line 2: Line
			Line 2 : document	supervisors
			management software
				Line 1: Document
				preparation assistant
	Primary engineering drawings	Primary engineering	Project folder + email	Line 2 : Line supervisors +
				Document preparation
				assistant
				Line 1: Document
	Part lists	Technical support +Primary engineering	ERP + email	preparation assistant Line 2: Line
				supervisors
			Oral information to the
	Working procedures	Industrial support	operators	Industrial support
			(unsystematic)
	Control procedures	Product Quality Assurance	email from the Product AQ to the System AQ, and from the System AQ to the line supervisors	Line 1: Document preparation assistant Line 2: Line supervisors
	Following sheets	Product Quality Assurance	email from the Product AQ to the System AQ, and from the System AQ to the line supervisors	Line 1: Document preparation assistant Line 2: Line supervisors
	Standard preparation manual	Product Quality Assurance	email from the Product AQ to the System AQ, and from the System AQ to the line supervisors	Line 1: Document preparation assistant Line 2: Line supervisors
	Picking lists (assembly kits)	Production, Logistics	Not followed	Not followed

They are issued by different departments for the use of the production department and are distributed via several channels. Examples of boundary objects are given in table 2.

An attractor is a set of material and immaterial elements which potentially participate in a given activity and which are simultaneously present from the subject's point of view.