FROM OBSCURITY TO PROMINENCE: HOW DRIP IRRIGATION CONQUERED THE WORLD

Over the last 60 years, drip irrigation has been envisioned according to a multiplicity of lenses: a way to extend agricultural land, to increase agricultural production and productivity, to modernize irrigation and use water efficiently and, more recently, to preserve water and contribute to poverty alleviation. This imagery has been progressively shaped first by public agricultural and irrigation engineers and now mostly by the private interests of large equipment manufacturers who manage to enroll policy makers and development operators. These narratives reinforce each other and this explains why drip irrigation "conquered the world": the technology appears to be able to address global challenges, hence shaping a better future.

Introduction

Over the last 60 years, drip irrigation has evolved from an experimental technology tested in a few research stations in a limited number of countries to a truly global phenomenon. This chapter recounts the journey of how drip irrigation has become one of the most popular technologies in the field of irrigation and agriculture for professionals and the wider public alike. This, the chapter will argue, partly hinges on the ability of drip irrigation to lend and reinvent itself in line with continually evolving dominant environment and development discourses. This "malleability" was not a given and emerged progressively as ever more people and organizations became interested and involved in the use and promotion of drip irrigation. These actors now form a loosely-bound but wide-reaching coalition through which drip irrigation artifacts (plastic pipes and emitters) are attributed inherent characteristics such as that of efficiency, productivity and modernity. The malleability of the technology and its systemic nature, in turn, allows multiple actors (including farmers; see chapter 15 and 16 for instance) to engage with it, adapt it to their context of use and specific needs, thus giving weight to far reaching claims of environmental preservation, agricultural modernization and poverty alleviation. This chapters' objective is not to assess whether those claims are "real" (or not) but to understand how they have come into being; neither do we aim at a detailed ethnography of the way drip irrigation systems have evolved over time. Rather, we recognize their diversity but use the term drip irrigation to designate a concept (the frequent application of small quantities of water directly at the root zone of crops) and a suite of objects through which this concept is put into practice (a system of perforated plastic pipes, emitters and ancillary devices).

The chapter adopts a diachronic and geographic perspective and identifies "significant trends" regarding the way drip irrigation has been, and still is, envisioned. This is done on the basis of an extensive review of documents published since the 1970s (grey literature, journal articles, policy documents), an analysis of wide audience media pieces (news articles, blog posts, public statements) and key informant interviews with irrigation professionals who have been active in the field for several decades. The analysis also benefitted from the multiple interactions the author had with colleagues when preparing this edited volume.

The chapter starts by showing how data on drip irrigation, and the way they are discussed and presented, contributes to establishing drip irrigation expansion as a global imperative, notably pursued through far-reaching public subsidy schemes (section 1). The chapter then turns towards identifying the main discourses that have been associated with drip irrigation since the 1960s, and how these play out today in different regions/countries of the world (see the color section for a spatial illustration). Section 3 highlights that such drip irrigation discourses are not "deincarnated" but have been actively shaped by multiple actors whose influence has waxed and waned with time. We notably show the importance of "mythical figures" to bring an "expert technology" out in the open, and the fact that agricultural and irrigation engineers working in public organizations have progressively lost ground vis-à-vis private manufacturing companies in shaping the debate around drip irrigation. A short conclusion recaps the findings.

Uncertain data but a general consensus to promote drip irrigation

As pointed out by [START_REF] Venot | Beyond the promises of technology: A review of the discourses and actors who make drip irrigation[END_REF], a first challenge in attempts to understand the dynamics of drip irrigation relates to the availability and reliability of data. Most current drip irrigation development indeed takes place outside large scale public irrigation systems (whose extent is relatively well known) and is largely driven by private initiatives, which makes monitoring more difficult for governmental and international organizations. 1 The first multi-country assessment of the extent of drip irrigation use dates back to 1975 [START_REF] Shoji | Drip irrigation[END_REF]. The International Commission for Irrigation and Drainage (ICID) conducted its first world-wide micro-irrigation survey in 1982 and has regularly updated it since (data is obtained from national irrigation and drainage committees). These surveys, however, remain largely incomplete (only 25 countries were listed in 1982 and 45 countries are listed in the most recent 2012 survey). 2 The surveys as well as the literature highlight that large scale use of drip irrigation started in the 1970s in countries such as Israel, the United States, and South Africa. Countries in the south of the European Union joined the trend in the 1980s [START_REF] Bucks | Historical developments in microirrigation[END_REF]Reinders, 2000). Since the mid 1990s, most drip irrigation development is occurring in emerging economies; notably on the southern shores of the Mediterranean Sea, Latin America and Mexico, and above all China and India (which together are estimated to account for more than one third of the entire area irrigated with drip worldwide; see box 1 for a brief description of trends in China).

Box 1. China: A (not so) hidden giant?

In 2012, ICID evaluated at 1,6 million hectares the drip irrigated area in China, and some manufacturers do not hesitate talking of a growth rate of 20% a year, 3 but it remains very difficult to assess the extent to which drip irrigation is used in China, especially for non Chinese scholars.

What is sure, however, is that China has emerged over the last few decades as a drip irrigation power house, both in terms of manufacturing capacity (alongside Israel and India) and drip irrigated area (alongside India). This has notably happened because of the strong support of the Chinese government whose priority is to support an agricultural boom in the country but is increasingly concerned by the unsustainable rate of groundwater abstraction in some of the most productive regions of the country (the north-western plains). The commitment of the Chinese government to promote drip irrigation has to be understood in this context and was clearly illustrated by the No. 1 document of the Chinese government of 2011. This high level policy document that identifies the government priorities focused on irrigation and identified "waterefficient irrigation technology" as a way to pursue the policy goal of "water conservancy" (USDA, 2011; [START_REF] Burnham | The human dimensions of water saving irrigation: lessons learned from Chinese smallholder farmers[END_REF]. This priority notably translated in the form of subsidies to farmers for purchasing drip irrigation equipment (of more than 50% of the equipment cost), as well as low interest loans and tax reductions or exemptions for manufacturing companies.

If drip irrigation systems have long been used by smallholders in 'traditional' Chinese greenhouses for the production of vegetables, it is now mainly used for cotton cultivation, most noticeably in the semi-autonomous arid region of Xinjiang in the extreme west of China, but also for sugarcane and maize production. Apart from the Xinjiang region, the semi-arid provinces of the northwestern plains (such as Gansu, Shaanxi, Ningxia) are a hot-spot of drip irrigation use as well. It is no surprise that the major equipment manufacturers are located there.

Drip irrigation was introduced by governmental research institutes in the 1970s but it is only in the 1990s that the drip equipment manufacturing sector got structured. The three biggest equipment manufacturing companies were set up in the late 1990s (Xinjiang Tianye Water Saving Irrigation and Dayu Conserving Water Group in 1999 and Gansu Yasheng Industrial Group in 1998); since the mid 2000, they are registered on the Chinese stock market. In addition to these giants, there are probably over 500 small and medium scale manufacturing companies. Taken together, the manufacturing capacity of Chinese companies would reach about 2 billion meters of drip line a year (that is, if a spacing of one meter is considered, the equivalent of 200,000 hectares/year) (Tianye, on its website, announces that the total cumulative irrigated area that could be equipped with the equipment it manufactured reaches 4 million hectares in Chinathis includes other equipment than drip irrigation such as sprinklers and pivot http://www.tianyejieshui.com.cn). Chinese manufacturing companies conduct most of their business in China but some also try to enter the international marketnotably through acquiring foreign companies. On the other hand, many large non-Chinese drip irrigation companies have offices and distributors in China. They tend to highlight the quality and 'high-tech' nature of their products as most Chinese companies still manufacture relatively simple (and cheap) products. Entering the Chinese market for international actors does not seem to be an easy task, as can be illustrated by the history of Netafim in the country. The company first entered China in 1994; in 2000s it partnered with a Chinese company and built two plants there that were shut down in 2008. Only in 2016 did Netafim open a new plant in Ningxia, hoping to benefit from the booming wine industry of the region. The company is also looking at partnering with Chinese fertilizer companies to expand its business, communicating on the potential of drip irrigation in terms of input optimization through fertigation (the use of soluble fertilizers) (https://www.netafim.com/news-item/231841; http://www.ft.com/intl/cms/s/2/24e45b40-3b00-11e2-bb32-00144feabdc0.html#axzz303cS9mWm) (see also see [START_REF] Guerra | John Deere driving east to sustain growth: John Deere Water Technologies[END_REF] for an overview of the irrigation industry sector, in relation to John Deere Water investments).

In 2012, and still according to ICID, India, China, Spain, the USA, Italy, Korea, Brazil, South Africa, Iran, and Mexico were, in that order, the 10 countries with the largest area under drip irrigation (ICID, 2012). Middle-eastern countries (United Arab Emirates, Israel and Jordan) had the largest drip irrigated area when expressed as a percentage of their national irrigated area (ICID, 2012) (see Figure 1). The statistics as well as scholarly analysis further show that drip irrigation is mostly used for vegetable crops (in open fields or greenhouses), fruit trees (including vine), as well as industrial crops such as cotton and sugarcane [START_REF] Ayars | Introduction[END_REF][START_REF] Narayanamoorthy | Impact assessment of drip irrigation in India: The case of sugarcane[END_REF]. Indeed, countries in which such crops are common are also the countries in which drip irrigated areas are largest (see the list above).

Although data are few and unreliable, there is a striking commonality in how the data that are available become part of larger narratives. For scholars, governments and the wider public, the tremendous growth observed over the last few decades (though it is not supported by consistent statistics) serves as proof of the multiple benefits that drip irrigation can bring to farmers and the society as a whole (indeed, why would it spread so quickly if it was not good?). Most scholars and policy documents further highlight, however, that drip irrigation still only represents a marginal share of all irrigation worldwide (about 5%; ICID, 2012), especially in emerging economies and developing countries (see GoI, 2004;[START_REF] Narayanamoorthy | Drip and sprinkler irrigation in India: Benefits, potential and future directions[END_REF]on India and Zhu et al., 2013or Zou al., 2012 on China). They also stress that drip irrigation is concentrated in a limited number of countries (when taken together, the USA, Spain, China and India account for two thirds of all drip irrigated areas) and lament the fact that its benefits are still largely eluding smallholder farmers notably because of high investment costs [START_REF] Hillel | Adaptation of modern irrigation methods to research priorities of developing countries[END_REF][START_REF] Postel | Drip irrigation for small farmers: a new initiative to alleviate hunger and poverty[END_REF].

6 Figure 1: A worldwide map of the importance of drip irrigation: data shortcomings Contrasting the tremendous expansion with the limited actual extent of drip irrigation use is not a neutral way of telling the drip irrigation story. It serves as a justification for massive public investment and lending schemes, in support of the promises of drip irrigation for a better future (and that even if a large share of the world's irrigated food cropsrice, wheat, maize are little adapted to existing drip irrigation systems). Many governments have engaged in aggressive policies to increase the share of the irrigated area under drip irrigation.

Public policies often take the form of subsidy programs that are, depending on the country, said to contribute to broader agricultural development and/or water resources management policies. They are also invariably justified on the ground that high initial investment costs still constitute a major barrier for farmers to adopt drip irrigation (drip-kits put aside), especially in emerging economies and developing countries [START_REF] Postel | Drip irrigation for small farmers: a new initiative to alleviate hunger and poverty[END_REF][START_REF] Ayars | Introduction[END_REF].

The map in the color section spatially represents the main discursive frameworks that underpin public policy and international development programs promoting drip irrigation around the world (the level of analysis is the country; the map is based on a review of available policy and development projects documents). In India, for instance, federal level financing schemes to promote drip irrigation have been launched in the early 1980s and have remained un-interrupted since then, culminating in the National Mission on Micro-Irrigation (NMMI) in 2010 [START_REF] Venot | Beyond the promises of technology: A review of the discourses and actors who make drip irrigation[END_REF]chapter 18). Drip irrigation was first seen as a way to promote a "modern" (i.e. productive) agricultural sector; over the last two decades, however, the government primarily justified subsidy schemes on the basis that extending the area under drip irrigation (a technology that allows efficient water use) would alleviate the current pressure over groundwater resources, which are unsustainably tapped in many parts of India. Subsidizing drip irrigation is also presented as a way to include smallholders (who have limited financial resources) in the on-going efforts to build a modern and intensive agricultural sector (see Chapter 18 for further discussion on a subsidy scheme adopted in one particular state of India, Gujarat). In the same way, in Morocco, drip irrigation has been at least partly subsidized since 1986 onwards [START_REF] Venot | Beyond the promises of technology: A review of the discourses and actors who make drip irrigation[END_REF]. The recent national plan for water saving in irrigation (PNEEI) as well as the Green Morocco Plan (the national agricultural policy) have led to a massive conversion and extension of areas under drip irrigation. In Morocco, subsidies schemes are said to serve the objectives of preserving groundwater resources and modernizing low performing public irrigation schemes (see chapters 5, 6 and 10, this volume).

Among the 10 countries with the largest drip irrigated areas in the world (see above), Mexico (since 2003; see chapter 9), Iran (since 1990), and China also subsidize the installation of drip irrigation systems. In Mexico and Iran, subsidizing drip irrigation is part of broader irrigation modernization projects and is also said to contribute to water preservation (for Iran) and agricultural intensification (for Mexico). In China, the primary stated objective to subsidize drip irrigation is "water conservancy" (see box 1). Other countries such as Chile, Iraq, Oman, Pakistan, Peru (through the irrigation sub-sector project, which started in 1996), the UEA, Syria, Tunisia, and Turkey provide direct subsidies to the installation of on-farm drip irrigation systems. 4 Broadly speaking, governments in South America generally justify subsidizing drip irrigation as a way to trigger a shift towards a modern and intensive agricultural sector; in the Mediterranean region and the Middle-East, the "(ground)water conservation argument" is more prominent. In both regions, and notably in countries with large canal-based irrigated schemes, subsidizing drip irrigation is seen as a way to modernize low performing irrigation systems, which also receive significant attention from international development organizations (this is also the case in Egypt where drip irrigation is seen as a way to extend irrigation in desert areas while improving the management of irrigation systems in the Nile delta).

Support to drip irrigation can also take the form of tax exemption (e.g. Uzbekistan); low-interest loans on drip irrigation equipment (e.g. Iran, Syria, Brazil, micro-credit in sub-Saharan Africa); and subsidized seedlings for crops seen as particularly adapted to drip irrigation (such as date palm trees in Saudi Arabia, where the government also buys dates at a fixed price). 5 Agricultural policies and water laws may also indirectly favor the use of drip irrigation, as for instance is the case in Peru where getting water rights is tied to using "efficient technologies" (see chapter 8). Finally, in the least developed countries, support to drip irrigation does not take the form of direct subsidy schemes but of projects funded and implemented by public agencies and the third sector (NGOs, development agencies, private foundations, and national government; see chapters 12,13,14).

The multiple and reinforcing framing of drip irrigation

The map in the color section provides a snapshot of current discursive framings of drip irrigation, taking countries as the unit of analysis. However, the way drip irrigation has been seen, perceived and discussed in each of these countries has evolved with time, in relation to global discourses in the inter-related sectors of irrigation, agriculture and water management. The current "multiplicity of drip" was not a given when the first large scale experiments took place after the second world war; it emerged with time and several periods can be identified since the 1960s.

The 1960s and 1970s (when experimentations with drip irrigation were increasingly taking place in the United States, Europe, Israel but also in India) are the decades of the Green Revolution, a term that refers to a tremendous increase in the production and productivity of agricultural crops. At the time, it is the yield increase allowed by drip irrigation (thanks to better matching crop water requirements) 6 together with the facts that the technology made it possible to extend irrigation to areas that could not be otherwise cultivated as well as to use low quality water that attracted most attention. That drip irrigation allowed extending cultivation in arid regions (with sandy and little fertile soils) led to the fact that the technology became closely associated with the attractive image of a 'blooming desert' (see, picture 9 and 10 in the color plate). This was recently re-highlighted when Daniel Hillel was awarded the World food Prize in 2012, and also underpins current drip irrigation dynamics in the coastal area of Latin America (see chapter 8 and the map in the color plate). At the time, the fact that drip irrigation allowed optimizing fertilizer application (through fertigation, which consists of applying soluble fertilizers together with irrigation water) and saving labour (in developed economies) was the object of much attention too. In short, in the 1970s, drip irrigation was mostly discussed in relation to agricultural intensification. This is still the case, as highlighted above, in developed economies, many South American countries as well as China.

In the mid to late 1980s, things started to change. These were the times when irrigation performance started to be questioned and there were increasing calls for irrigation "modernization", intended as "a process of technical and managerial upgrading combined with institutional reforms, with the objective to improve resource utilization (labour, water, environment, economy) and water delivery service to farmers" [START_REF] Facon | Modernization of irrigation system operations[END_REF]. In some contexts, the modernization debate led to talk of pressurizing open canal systems or building pressurized systems from scratch (US, Europe but also the Middle East, most noticeably Israel and Jordan and the United Arab Emirates) in order to increase irrigation efficiency (that is, limit water losses through evapotranspiration and open conveyance systems). The consequences of shifting from canal to pressurized drip irrigation have been particularly intense in Spain where the government supported massive irrigation modernization programs (see for instance, Lopez-Gunn et al., 2012). In policy discussion, the high efficiency of drip irrigation as a modern technology was often contrasted with the wastefulness of traditional irrigation methods, and it was this efficiency that took centre stage (see van der Kooij et al., 2014 and chapter 5 for critical reflection on the notion of efficiency and the uses made of it). Such framing has now "lost" in importance at global level but still plays out in countries with large publicly managed irrigation systems such as Turkey, Egypt, Pakistan, Mexico, Brazil, and countries of Central Asia.

The most significant change may have happened in the early 1990s. It can be traced back to the 1992 Earth Summit held in Rio de Janeiro, which established "sustainable development" (a term first coined in WCED, 1987) as a global policy goal and dominant development discourse. The fact that the global debate on sustainable development influenced the way drip irrigation was perceived is clearly reflected in the 5 th micro-irrigation congress, the first of the series to have a title that reflects environmental concerns: "Microirrigation for a changing world: Conserving resources/preserving the environment" (ICID, 1995). Drip irrigation became to be seen through the lens of the environment, and with it the potential water savings produce became its most important feature; irrigation modernization programs started to be assessed and evaluated through this lens, most notably in Spain (see above).

The 1990s are also the decade during which irrigation largely fell out of the favor of the international community. Returns to public irrigation investments had been notoriously disappointing, whereas the negative consequences of large infrastructural works (notably in dams and canals) started raising critical questions at international level. In addition, the rapid rise in the popularity of neo-liberal ideologies (after the collapse of the Soviet Union and the fall of the Berlin Wall had reduced the appeal of and belief in socialist forms of government) made it increasingly difficult to defend spending so much public money on the construction, operation and maintenance of public irrigation systems. In the late 1990s and early 2000s, the water issue started to be framed as a tension between the environment and agriculture in a context of growing scarcity and competition over water. Agriculture was pointed out as being the single main water user and irrigation as wasting a significant share of it, as many observers emphasized the low efficiency of most irrigation worldwide (for instance, [START_REF] Postel | Last Oasis. Facing Water Scarcity[END_REF]Gleick, 2003). The question became: how to save water in agriculture so that it can be allocated to other usesincluding the environment. Despite a growing debate on the water saving potential of drip irrigation in absolute terms (see for instance, [START_REF] Seckler | The new era of water resources management: from ''Dry'' to ''Wet'' water savings[END_REF][START_REF] Perry | Efficient Irrigation, inefficient communication, flawed recommendations[END_REF][START_REF] Molle | River basin closure: processes, implications and responses[END_REF] as well as chapter 5, this volume), the alleged high efficiency of drip irrigation continued to be a defining element of the discourse.

Rather than being framed from an engineering point of view (as had been the case in the 1980s when modernizing irrigation was hot on the agenda), the question of efficiency was primarily discussed in relation to the alleged potential that drip irrigation held in terms of preserving water resources from overexploitation. This happened at a time when groundwater use in irrigation skyrocketed worldwide (especially in the developing economies of India, China and the southern shore of the Mediterranean) leading to declining water tables (see [START_REF] Shah | Water for Food, Water for Life: A Comprehensive Assessment of Water Management in Agriculture[END_REF] for an analysis of groundwater development trends in the agricultural sector). This focus on the potential that drip irrigation would hold in terms of preserving (ground) water is very much to the fore in North African and Middle Eastern countries as well as China and India (see chapter 5 for a critique of this argument, on the basis of a case study in Morocco).

The mid 2000s, characterized by a renewed concern for food production and food security (notably heightened by the 2008 food crisis) constitute another turning point. Bringing together two distinct narratives, that of agricultural intensification (prominent in the 1970s) and of water use efficiency (prominent in the 1980s), drip irrigation became the material embodiment of a broad agricultural development discourse stressing the need for "more crop per drop", which combines calls for intensifying agriculture with the need to preserve natural resources as well as a technology for adaptation to climate change (UNFCC, 2006;[START_REF] Clements | Technologies for Climate Change Adaptation-Agriculture Sector[END_REF]ADB;2014). 7 In recent years, in relation with the emergence of the new water-food-energy nexus vocabulary (see [START_REF] Allouche | Technical veil, hidden politics: Interrogating the power linkages behind the nexus[END_REF] drip irrigation is also increasingly framed as an energy saving technology, notably in India. The energy saving argument is closely linked to the water saving idea that underpins many drip irrigation promotion initiatives. Indeed, by supplying water more efficiently to crops, drip irrigation would allow pumping less water per unit area (see chapter 5), hence using less energy. 8 This is a contentious issue as other scholars highlight that the need to pressurize water to use drip irrigation is a reason for the booming energy consumption in agriculture (see, for instance, Rodriguez-Diaz et al., 2011 in the case of Spain).

At the same time, a parallel discourse the basis of which had been laid out as early as 1988 when [START_REF] Hillel | Adaptation of modern irrigation methods to research priorities of developing countries[END_REF] highlighted that most drip irrigation development eluded small farmers in the developing world, started to take ground. This discourse centered on the potential of drip irrigation as a poverty alleviation tool. This new "feature" of drip irrigation is related to what [START_REF] Postel | Drip irrigation for small farmers: a new initiative to alleviate hunger and poverty[END_REF] identify as the existence of "a new spectrum of drip systems [that] can form the backbone of a second green revolution, this one aimed specifically at poor farmers in sub-Saharan Africa, Asia, and Latin America". The late 1990s and early 2000s have indeed been marked by significant efforts from Non Governmental Organizations (supported by national and international aid agencies), social enterprises, international agricultural research centers, and private irrigation equipment manufacturers to design and disseminate systems that would soon become to be known under the generic term of "drip-kit". Drip-kits are widely presented by those promoting them as being affordable, small-size, and infinitively expandable. These are all features that make them (seem) particularly suited for meeting the specific needs of smallholders in the developing world. Their small size, low cost and aptitude for irrigating vegetables, in addition, makes it easy to align them with goals of women's empowerment, based on the idea that female farmers can make use of the technology in their kitchen gardens (see among others, [START_REF] Polak | Out of Poverty -What works when traditional approaches fail[END_REF]; see also chapters 11, 12, 13 and 14 as well as picture 12 and 13 in the color plate). 9

If the main discursive justification of NGOs disseminating these drip-kits is to support smallholder farming and contribute to transforming it into an entrepreneurial venture, large scale equipment manufacturers promote these smallholder drip irrigation systems as part of a broader Corporate Social Responsibility rhetoric 10 but also in an attempt to tap into an important market; the so called "Bottom of the Pyramid" (see [START_REF] Venot | A success of some sort: Social enterprises and drip irrigation in the developing world[END_REF] for a critical analysis of these processes). The poverty alleviation potential of drip irrigation currently underpins efforts to promote the technology in most of sub-Saharan Africa, some countries of South and Central America (Bolivia, El Salvador, Guatemala, Haiti, Honduras, Nicaragua, etc. ) and others in South and south-east Asia (Bangladesh, Nepal, Myanmar, Cambodia, and Indonesia).

What is striking in this short historical review is that, over time, and at global level, drip irrigation has increasingly come to be discussed in relation to water resources management (and more broadly poverty and the environment) and less and less for its agricultural benefits (higher yields, input optimization, labour savings) that had made its early fame and are still the main reasons why farmers start using it. But what is also central to explain how drip irrigation "conquered the world" is the fact that one story does not replace the other, rather they "add-up", reinforce each other, with the emphasis shifting depending on context as shown in Figure 1, thus defining a true "panda" (see the introduction of the book). The following section highlights that the discourses and (policy and development) initiatives promoting drip irrigation are not deincarnated; they have been actively shaped and the trends described also reflect changes in the actors who are the most influential in shaping the debate around drip irrigation.

A wide reaching and multi faceted advocacy coalition

The "Mythical" figures One of the reasons drip irrigation "conquered the world" is linked to the fact that the technology found powerful advocates in a number of people from different backgrounds, pursuing different goals, and embedded in different social networks. To use the term of Madeleine Akrich (Akrich et al.,1998a,b) , these individuals acted as "spoke persons" and created "interessement" in an ever widening coalition of actors who supported the dissemination of drip irrigation. These individuals came to personify some of the key discursive regimes of drip irrigation; hence lending it weight.

Most authors agree that the concept of drip irrigation (that is the slow and frequent application of water to the root zone of crops) is as old as agriculture (for instance [START_REF] Goyal | Sustainable micro irrigation -Principles and practices[END_REF] and trace back "modern day" drip irrigation to experiments with underground clay pipes that took place in the second half of the 19 th century (1860s-1880s) in Germany and the United States [START_REF] Ayars | Introduction[END_REF]. The introduction of perforated pipes in the 1920s in Germany constituted a significant breakthrough as did the improvements in plastic technology: during the Second World War, perforated plastic pipes were used for vegetables production in greenhouses in the United Kingdom, France and Germany [START_REF] Ayars | Introduction[END_REF][START_REF] Goyal | Sustainable micro irrigation -Principles and practices[END_REF]. Experiments in the 1950s followed in the United States and Israel and led to patents on emitters. This is when the personalization of (the history of) drip irrigation really begins.

Simcha Blass, a water engineer also known for having played a key role in the early years of Mekorot (the national water agency of Israel) and the elaboration of a "fantasy plan" to irrigate the Negev in the late 1930s (a plan that would lay the basis for supplying water to Jewish settlements in the area and also formed the start of the future National Water Carrier) 11 is commonly attributed (together with his son, Yeshayahu) the first patent of a coiled in-line emitter (allowing the slowing of water flow and avoiding clogging) [START_REF] Shoji | Drip irrigation[END_REF][START_REF] Ayars | Introduction[END_REF]. That the role of Simcha Blass in drip irrigation development has attracted significant attention and has found its ways in collective memories can partly be explained by the fact that the contract he signed with the Kibbutz Hatzerim led to the establishment of the NETAFIM company. This company would later become the largest manufacturer of drip irrigation equipment worldwide (see below) and pursues very aggressive communication campaigns. He became to be seen as an example of an Israeli innovator-entrepreneur, who allowed for making the Israeli desert "bloom". In the same vein, Daniel Hillel has received recent attention for the role he played in the development of drip irrigation, first in the Negev desert of Israel in the 1960s and in its dissemination in other countries of the world during the following decades. A soil and water scientist who has been affiliated to several Israeli and American universities, he closely worked with international organizations such as the United Nations and the World Bank. He was awarded the 2012 World Food Prize on account of "his role in conceiving and implementing a radically new mode of bringing water to crops in arid and dry land regions -known as "microirrigation"." (http://www.worldfoodprize.org). Daniel Hillel impersonates this ambition to make "the desert bloom" but also an aspiration for making agricultural transformation and modernization benefit smallholders in developing countrieswhich has been most noticeably associated to other individuals since the 1970s. Richard D. Chapin, the founder of a small irrigation equipment manufacturing firm in the United States, Chapin Watermatics, is also presented as an individual having been instrumental in developing drip irrigation. He is notably attributed with the development of drip-tape (thin plastic tubing) and early experiment with plastic mulch in the 1950s [START_REF] Ayars | Introduction[END_REF]; but also with designing and disseminating (through a dedicated foundation) the first small-scale drip irrigation systems specifically targeting farmers in developing countries in the 1970s, at the request of the NGO Catholic Relief Services [START_REF] Keller | Gardening with low-cost drip irrigation in Kenya: For health and profit[END_REF][START_REF] Venot | Beyond the promises of technology: A review of the discourses and actors who make drip irrigation[END_REF]chapter 11). In the late 1990s and early 2000s, the poverty alleviation potential of drip irrigation has also been impersonated by yet another American entrepreneur, Paul Polak, founder of International Development Enterprises (iDE) and by an Israeli researcher, Dov Pasternak. The former attracted attention less for his role in developing drip irrigation systems per se and more for his unapologetic call to adopt a "business approach to poverty alleviation". He advocated a system based on the sale of affordable products to farmers. The enterprise would make a small margin on each sale (hence being a profitable venture,e in itself); that poor farmers bought the product acted as a proof of the added value of the latter and of the fact that farmers derived economic benefits from it.. [START_REF] Polak | Out of Poverty -What works when traditional approaches fail[END_REF][START_REF] Polak | The business solution to poverty. Designing products and services for three billions new customers[END_REF]see Venot, 2016 for a critical analysis). The latter has been pivotal in harnessing support from development aid agencies and private manufacturers of drip irrigation equipment for the dissemination of "drip kits", notably in sub-Saharan Africa (see Wanvoeke et al., 2015).

Through these individuals and their work, a picture of the different types of actors who have played and still play an active role in shaping the imagery of drip irrigation worldwide emerges too. The supportive coalition for drip irrigation includes irrigation and agricultural engineers (and their professional organizations); public research centers and universities; private companies; NGOs and social enterprises; and international or national development agencies. When tracing the history of drip irrigation, what becomes clear too is that drip irrigation is far from the "neutral" technology that is often described. On the contrary it has long been, and still is, clearly associated with particular political and ideological movements and agendas (the Kibbutzim in the early years of Israel; the dream to make the desert bloom in the same country but also others such as the USA, Peru and Algeria (see chapters 8 and 16); the "will to improve" that drives development agencies; or the search of modernity through technology and entrepreneurship as articulated by social enterprises).

Public irrigation research: Highlighting efficiency and productivity

Less publicized but also central in establishing drip irrigation as a technology that would become to be associated with notions such as efficiency, productivity and input optimization is an "army" of agricultural and irrigation engineers working in research and development organizations, universities of developed countries, as well as extension services and agricultural ministries. ICID, the main professional body for irrigation practitioners and researchers, constituted the main platform through with international experiences with drip irrigation were shared. Even though most work in the sector still related to canal based irrigation, an increasing number of actors became involved in drip irrigation research and development during the 1970s-1980s (in 1985, the contributions of more than a 100 individuals were already acknowledged; [START_REF] Bucks | Historical developments in microirrigation[END_REF].

The increasing attention towards drip irrigation among irrigation professionals is illustrated by the existence of a series of International Microirrigation Congresses, the first of which was organized in Israel in 1971. From the mid 1970s to the mid 1990s, engineers and researchers from the US Department of Agriculture-Agricultural Research Services (USDA-ARS) appear to have been particularly active in these events, as shown by the fact that 3 of the successive 4 international micro-irrigation congresses were held in the United States in 1974States in , 1985States in and 1995States in (in 1988, the congress was held in Australia). 12 Such congresses played a key role in establishing the technical and scientific legitimacy of drip irrigation as a technology allowing to (1) use water (more) efficiently than canal irrigation; (2) save labour in intensive agricultural systems; (3) and improve yields (both in quantity and quality (see, for instance, ICID, 1995). At the time, the debate and discussions revolved on identifying and establishing the conditions for drip irrigation to fulfill a potential that had been demonstrated in experimental plots as well as closely monitored farmers field. It involved significant expert-led refinement of the technological artifacts and discussion around the elaboration of (quality) standards (see box 2) and certification by professional authorized bodies.

The fifth international microirrigation congress organized in 1995 marks the height of this era of irrigation engineering research. At the time, research focused on (1) determining water requirement and irrigation scheduling; (2) development in nutrient management, fertigation methods, and water quality issues, (3) system design and uniformity; (4) the development of decision support tools and simulation models [START_REF] Phene | Research trends in microirrigation[END_REF]. From 1995 onwards, the influence of ICID as a global knowledge broker on drip irrigation progressively waned. Three subsequent international micro-irrigation congresses were organized in South Africa (2000), Malaysia (2008) and Iran (2011). These attracted less interest and attention than the 1995 congress (a much lower number of papers were presented during these meetings), but provided room for discussing drip irrigation in the context of emerging and developing economies.

That ICID became less pivotal perhaps can be attributed to the fact that drip irrigation became increasingly discussed in relation to water rather than agriculture while its members are first and foremost ageing (male) agricultural and irrigation engineers. It is also linked to a widespread feeling among these individuals that the potential of drip irrigation (in terms of efficiency, productivity, and scope of use) had been clearly demonstrated and that little interesting work (read: technical oriented research) remained to be done, as well as to dwindling investment (at least in Europe and the United States) in training, research and education in the field of irrigation engineering.

Box 2. An elusive search for standards? At a time where certification schemes (built on a sets of ideas and standards; see box in Chapter 8 for instance) are gaining in importance in the agricultural and water sectors, the drip irrigation sub-sector stands aside and is conspicuous by the near absence of standards.

The issue of elaborating standards for drip irrigation systems has, however, long been debated among irrigation professionals as illustrated by the existence of an ISO subcommittee on "irrigation and drainage equipment and systems" since the late 1970s [START_REF] Baudequin | Is standardization a solution to improve the sustainability of irrigated agriculture? Improved irrigation technologies and methods: Research, development and testing[END_REF]. 13 The "proponents" of elaborating standards argue that the latter can act as a guarantee of "quality" (material and installation), which they see as a prerequisite to fulfill the potential of the technology (be it in terms of water saving or yield improvement). The "opponents" to elaborating standards, on the other hand, stress that, given the systemic nature of drip irrigation systems (they are, by essence, networks), this is an impossible task. Indeed, standards could not possibly account for the diversity of on-farm situations and would act as impediments to innovation by users (see, for instance, [START_REF] Solomon | Standards development for microirrigation[END_REF]. The opponents to "strict standards" see the boom of drip irrigation development as a proof that the latter are not needed. They however also stress that many systems observed function in a sub-optimal way, and attribute this to the fact (1) that the systems are not designed or used according to "good engineering practice" or (2) that the actual discharge of the drippers differ from the nominal value displayed by the manufacturer. These discussions over standard development mostly take place among private manufacturers, industry associations whose raison d'être is to further develop and promote irrigation businesses (such as the US-based irrigation association and the EU based European Irrigation Association), national and international standard agencies, and public testing laboratories. Farmers (who use drip irrigation systems and are presented by irrigation professionals as being the ultimate beneficiaries of standard development) are rarely contributing to these debates and meant to be represented by research laboratories -though the latter appear to be far from farmers' realities (see box on field-measurement performance level in Chapter 5).

Against this backdrop, and since its creation, the ISO sub-committee has endorsed over 30 international standards. A similar number of standards have also been developed by the American National Standards Institute (ANSI) and the European Committee for Standardization (ECS). Several countries such as Morocco and India also have their own standards, which need to be followed by farmers if these want to benefit from the public subsidies.

Most existing standards at international level are "design standards" for specific pieces of equipment (notably ancillary devices such as filters or pumps) and "procedural standards" that set the conditions under which irrigation equipment shall be tested. Most commonly, drip irrigation pipes and emitters are tested for (1) the degree of variation the material has when compared to manufacturing specifications (notably in terms of the relation between water flow and pressure), and (2) the level of uniformity across emitters. How this relates to water use efficiency, the quantity of water used or yield levels is never discussedthese are indeed seen as beyond the realm of the tests and solely attributed to farmers' practices rather than to the equipment.

There is a conspicuous lack of "performance standards" and, despite the name of the subcommittee, no "system level" standard according to which systems need to be designed and their performance measured (again, this is 'justified' by the fact these are seen as depending on designers and farmers practices rather than to the equipment). 14 In such conditions, it is difficult to hold private manufacturers accountable to farmers or policy makers when promises of improved efficiency or productivity are not met. They can, as a result, continue to promote their products on the basis of their water savings and efficiency potential irrespective of whether this potential is ever realized. Very much aware of this discrepancy between efficiency claims and the ways the irrigation equipment works in farmers' fields, and in a context in which they are sidelined from the processes of standard development, several testing laboratories (among which the IRSTEA-France, the INCTEI-Brazil, the UCLM-Spain, and CIT-USA; see picture 4 in the color plate for an illustration of a drip irrigation test bench) have formed the International Network of Irrigation Testing Laboratories (INITL). They aspire to conduct independent cross tests of drip irrigation equipment to form an alternative informal certification mechanism among peers. 15 Yet, the laboratories are limited in terms of the tests they can conduct and are also in a delicate situation as they derive part of their funding from private manufacturers (when the latter need a certification of their material to qualify for public tenders notably), who may (threaten to) sue them if they do not comply with internationally standardized tests.

Jean-Philippe Venot et Bruno Molle

When private companies gain in importance

The lower influence of networks such as ICID and public research organizations happened as the private sector, partly structured through industry associations, played an increasingly pivotal role in developing drip irrigation systems and supporting their dissemination through very aggressive marketing policiesincluding lobbying national governments and organizations such as the EU so that they would consider subsidizing so called "water efficient" technologies.

The first drip irrigation manufacturing companies were set up in the 1960s (for instance, Chapin Watermatic in 1960 and Netafim in 1965). In the 1980s and 1990s, several companies that had been active in the field of agriculture and/or irrigation (such as Jain irrigation in India and Toro in the USA 16 ) specifically started manufacturing drip irrigation equipment. In countries such as Cyprus, Greece, France, Italy and Spain a multitude of smaller scale manufacturers also emerged.

From the 2000s onwards, two different trends can be identified. First, the emergence of a local manufacturing capacity and private companies in countries such as Turkey, Jordan, Morocco and evidently Korea and China, which have seen a tremendous increase in the use of drip irrigation. Second, a tendency towards a consolidation of the drip irrigation equipment market, which goes hand in hand with a financialization of the sector (see box 3). Through a series of acquisition (Chapin Watermatic in 2006 and the Israeli Naandan in two stages, in 2007 and 2012), Jain irrigation for instance emerges as a second power house alongside the Israeli company Netafim (both companies are valued at more than 1 billion US$). Alongside this trend, the profile of the decision makers of these companies has also evolved. While the directors and CEO of the "early age" drip irrigation equipment companies had often been trained as agriculture and/or irrigation engineers, this is less and less the case today; they rather have MBA training or the like. This has come together with a change in the way drip irrigation is discussed: there is far less attention to the technical aspects of the technology (the technical potential of drip irrigation is seen as a given) and much more emphasis on overarching claims of water preservation and agricultural productivity that are difficult to actually grasp and assess.

Box 3. Global finance and drip irrigation

That drip irrigation has entered the world of "big-business" is evidenced by the publication of reports on market trends and potential, which are sold for hefty sums (for instance Sustainable Asset Management, 2004;Transparency Market research, 2013).

Maybe more significant is the fact that pension funds such as the London based private equity firm, PERMIRA, acquired a majority share of 61% of Netafim in 2011 for $850 millions (Kibbutz Hatzerim and Kibbutz Magal remain minority share holders with 33% and 6% respectively) and justified this investment on the ground that "Netafim [was] the undisputed market leader and […] ideally positioned to be the market maker in one of the most attractive segments of the agriculture industry worldwide" [START_REF] Permira | Permira funds complete acquisition of Netafim[END_REF]. In February 2013, PERMIRA further reported on its webpage that "Through the cycle, Netafim has delivered strong double digit revenue growth and it targets a similar growth rate over the coming years. It will continue to pursue a growth strategy built around enhancing its leadership position as a market maker in drip irrigation. Specific growth opportunities include a significant expansion in emerging market businesses, particularly India and Latin America, entrance into the Chinese market and further penetration of large commodity crops." For the CEO of Netafim, "having the backing of an international investor like the Permira funds [was] invaluable" to accelerate the company's international growth [START_REF] Permira | Investing in growth-Annual review 2011[END_REF]. This is not the only example of the involvement of global finance in the drip irrigation equipment sector. FIMI, one of the largest Israeli private equity funds also acquired John Deere Water in 2014 (which had itself been set up in 2006 and acquired Plastro and T-Systems, two other drip irrigation companies, in 2008).

What is striking in the two cases here is the terminology used. The focus is on economic growth and profit making; the technologylet alone the farmers-are conspicuously absent. There are tentacular implications to such trend. Netafim has 28 subsidiaries and 16 manufacturing plants worldwide, and employs 4,000 workers, some of which commented during interviews that the sales target for regional managers increased after the acquisition of PERMIRA to support the double digit growth objective of the pension fund, while field workers benefit-sharing schemes were cut. More surprising, maybe, is the fact that Dutch civil servants (including contributors to this book) are part of the story. Indeed, PERMIRA is financed by other funds among which ABP (the Dutch pension fund for civil servants) through Alpinvest.

Jeroen Vos and Jean-Philippe Venot

Conclusion

Over the last 60 years, drip irrigation has been envisioned according to a multiplicity of lenses, which tend to reinforce each other in a never ending process. It was first seen as a way to extend agricultural land and a means to increase agricultural production. It then became associated with the idea of irrigation modernization and improved efficiency and productivity. Agricultural and irrigation engineers working in public and/or international agencies started losing their influence from the mid 1990s onwards as drip irrigation travelled from experimental sites to farmers' fields, the political sphere, and global water arenas. In the latter, and in line with the dominant discourse of sustainable development, drip irrigation started to be discussed as a technology to preserve water resources and later as an adaptation strategy to climate change, including by and for the benefits of small farmers in the developing world. As the technology travelled from the realm of self-proclaimed irrigation experts and moved into a broader public sphere but also the world of big businesses and international water arenas, caution regarding the prospects it held largely gave way to definitive claims on the potential of the technology. This is partly because of the ever higher political and economic stakes linked to the promotion of drip irrigation.

What is striking in this "traveling process" is that new sets of characteristics ascribed to the technology do not replace earlier ones; they rather add up to form a mythical artifact, which we called a panda or a hydra in the introduction. As such, drip irrigation has the characteristics of a boundary object [START_REF] Star | Institutional ecology. ''Translations'' and boundary objects: Amateurs and professionals in Berkeley's museum of vertebrate zoology[END_REF], that is, an artifact that is multivalent in character, can be cast in different ways to speak to various social worlds and communities of practices, and embodies multiple discourses in the process acting to mediate between different actors.

A multiplicity of drip irrigation narratives co-exist, which, when taken together, make the technology seem capable of addressing a number of current challenges in order to contribute to the shaping of a better future. Beyond the technical attributes commonly associated to the artifact (efficiency, modernity, productivity), it is this widely shared positive imagery that explains the attraction that drip irrigation exerts among a wide variety of actors. public subsidies. Whether the material and the system respect the standards is assessed by certified companies and engineers (the equipment of all major drip irrigation manufacturers is eligible to the subsidy). 15 Many of these laboratories are not certified by their national standard organizations, let alone by ISO, due to the costs this would imply for them in regard to a rather limited "market" for testing drip irrigation equipment. 16 TORO is specialized in turf & landscape management. It entered the irrigation equipment sector in 1962 (turf and landscape) and expanded into drip irrigation for agriculture in the mid 1990s through acquisition. Rainbird, set up in 1933, is another major player in the irrigation equipment sectorspecialized on sprinklers.

The map is a simplification of a complex reality; it only represents the main discursive framings of drip irrigation at country level. For each country only one or two main discourses are identified, even though other ideas of drip irrigation may also exist. Within a country, different discourses may play out differently across regions. Finally, the map may give the impression that different discursive regimes are neatly bounded; this is not the case as different discourses and imageries overlap and reinforce each others (see chapter 1). Author: Jean-Philippe Venot

Acknowledgements

The author would like to thank the four reviewers and his fellow co-editors for their valuable comments on earlier versions of this chapter. I also greatly acknowledge the insights from the many people interviewed in the course of this research project who shed light on the multiple aspects and dynamics of drip irrigation worldwide.

Endnotes

1 In countries where the installation of drip irrigation is subsidized (see below), consistent monitoring of subsidy schemes at national level could be a proxy to assess the extent of drip irrigation development. It would, however, overlook the fact that many farmers (often smallholders) purchase drip irrigation equipment on the market, without resorting to any subsidy scheme (see, for instance, chapters 5, 7 and 16 in this volume). Further, volumes of subsidy or sales are do not always reflect actual use by farmers, as demonstrated in chapter 13).

2 FAO Aquastat gives data on "localized irrigation" (meant to include drip and sprinkler irrigation) for 85 out of 199 countries, with some records dating back to over two decades (FAO, 2013;[START_REF] Venot | Beyond the promises of technology: A review of the discourses and actors who make drip irrigation[END_REF]. 3 http://www.breakingisraelnews.com/21745/china-buys-israeli-smart-irrigation-tech-company-20-million/ 4 In Algeria drip irrigation was heavily subsidized in the early 2000s but subsidies have been progressively cut down. 5 From the 1970s onwards, Saudi Arabia pursued an elusive objective of food sufficiency and, in order to do so, heavily subsidized its agriculture sector, notably wheat production under pivot irrigation [START_REF] Elhadj | Camels Don't Fly, Deserts Don't Bloom: an Assessment of Saudi Arabia's Experiment in Desert Agriculture[END_REF]. Since the early 2000s, wheat subsidies have been phased out in part due to concern over groundwater over abstraction and the government is now prioritizing vegetables under greenhouses, using drip irrigation (FAO/GIEWS, 2015). 6 Many experiments consisted in assessing the yield-response of different crops under drip. 7 IPCC (2014) identifies "efficient irrigation" as an adaptation strategy, without mentioned drip irrigation explicitly. 8 This concern over the energy use of (drip) irrigation has led some organizations to test associating drip irrigation systems with solar pumping together. 9 Interestingly, small drip irrigation systems such as drip kits are also promoted in the United States, though in very specific contexts, that of the reservations of Native Americans. There, drip kits are promoted on the ground they could contribute to better health through improved diets and physical activity (vegetable gardening) in a context of very high incidence of diabetes. 10 Netafim, the largest manufacturer of drip irrigation equipment worldwide for instance widely advertise that (1) by selling drip irrigation systems, the company contributes to preserving water resources worldwide and that (2) by partnering with NGOs, research centers and international development organizations in projects promoting drip kits, it contributes to poverty alleviation worldwide (see http://www.netafim.com/corporate--responsibility). 11 See http://www.mekorot.co.il/ as well as [START_REF] Siegel | Let there be water: Israel's solution for a water-starved world[END_REF]. These early plans to develop the Negev desert were part of a Zionist strategy that aimed at influencing the drawing of the boundaries of the future State of Israel. The National Water Carrier plays a key role in the Israeli identity as does drip irrigation repeatedly presented as one of the innovation Israel "brought to the world". 12 Between 1979 and 1997, ICID had a dedicated working group on "micro-irrigation", which would then be renamed "on-farm working group" in 1998 (www.icid.org). Irrigation and agriculture engineers from Israel, South Africa, several European countries as well as India were also particularly active during these two decades. 13 Currently, the ISO/TC23/SC18 counts 11 participating and 18 observing members (www.iso.org) among which the countries where drip irrigated areas are the largest and the irrigation industry well developed (among the 10 countries with the largest drip irrigated areas, only South Africa and Mexico do not contribute). [START_REF] Solomon | Standards development for microirrigation[END_REF] highlight that Israel, France, the U.S.A, Canada, Spain, and Italy are the most active countries. 14 A noticeable exception of "performance standard" is a US standard on the performance of drip irrigation tape. Due to the opposition of Israeli manufacturers to the idea of standardizing tape, there is no equivalent ISO standard (American and Israeli manufacturers are competitors on the drip irrigation tape market). The difficulty to develop (material) performance standards at international level can also be illustrated by the 'journey' of a proposition that aimed at developing a standard on drippers' sensitivity to clogging, which, after ten years, has been registered as a "technical report" rather than a standard (hence is not bounding). In terms of system standards, the ASABE (American Society of Agricultural and Biological Engineers) has devised an "Engineering Practice" which specifies practical engineering methods for field evaluation of existing microirrigation systems. India and Morocco have their own (national level) quality and dimensioning standards that farmers need to adhere to if they want to benefit from


Giang Do

Khiem Le

Quang Pham
email: phamhq@i2r.a-star.edu.sg

Trungtin Nguyen

Thanh-Nam Doan

Thanh-Binh Nguyen

Chenghao Liu

Savitha Ramasam

Xiaoli Li

Steven Hoi

Trungtin Nguyen

Binh T Nguyen

Savitha Ramasamy

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Recent years have witnessed tremendous successes of the Transformer model [START_REF] Vaswani | Attention is All you Need[END_REF] and its variants across a wide range of tasks, ranging from natural language and speech processing (Bao et al., 2022b;[START_REF] Gulati | Conformer: Convolution-augmented Transformer for Speech Recognition[END_REF], computer vision [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF][START_REF] Ruiz | Scaling Vision with Sparse Mixture of Experts[END_REF]Bao et al., 2022a), reinforcement learning [START_REF] Chow | A Mixture-of-Expert Approach to RL-based Dialogue Management[END_REF], to life sciences [START_REF] Rives | Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences[END_REF]. Since then, scaling up to larger models has become the prevailing approach for advancing the state-of-the-art in pre-training and finetuning tasks. However, training such large models comes with a high computational cost [START_REF] Lin | A survey of transformers[END_REF]; therefore, there is a growing need to develop efficient strategies that facilitate the training of large language models (LLMs) (Fedus et al., 2022a). One of the most effective strategies thus far is the Sparse Mixture-of-Experts (SMoE) [START_REF] Vaswani | Attention is All you Need[END_REF]Fedus et al., 2022b), which utilizes routers to direct each input token to a small subset of network parameters (experts). SMoE improves both efficiency and performance compared to dense training approaches [START_REF] Lewis | BASE Layers: Simplifying Training of Large, Sparse Models[END_REF][START_REF] Artetxe | Efficient Large Scale Language Modeling with Mixtures of Experts[END_REF][START_REF] Zhang | MoEfication: Transformer Feed-forward Layers are Mixtures of Experts[END_REF][START_REF] Du | GLaM: Efficient Scaling of Language Models with Mixture-of-Experts[END_REF]. Despite the encouraging results, SMoE has been found to encounter the issue of representation collapse, where all experts converge to similar representations [START_REF] Chi | On the Representation Collapse of Sparse Mixture of Experts[END_REF][START_REF] Chen | Towards Understanding the Mixtureof-Experts Layer in Deep Learning[END_REF]. This problem arises due to the learning process of the router encouraging experts to cluster around a centroid [START_REF] Chi | On the Representation Collapse of Sparse Mixture of Experts[END_REF]. Therefore, significant efforts have been dedicated to addressing the representation collapse issue while preserving the simplicity and efficiency of SMoE training. To this end, one of the most effective strategies is freezing the router, as demonstrated by SMoE-Dropout [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF], where a randomly initialized router remains fixed throughout the training process. Additionally, [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF] found that progressively increasing the number of selected experts can be beneficial. However, we argue that such a naive strategy exhibits two key limitations. Firstly, the random routing policy may be sub-optimal, which hinders the overall training process. Secondly, we will show in Section 3.3 that a fixed router will restrict the model's representation capabilities, necessitating a progressive increase in the number of chosen experts to achieve satisfactory performance. Therefore, SMoE-Dropout inherently suffers from limited representation and does not offer efficiency gains during inference. This work introduces a novel approach called HyperRouter to address the trade-off between fixed and trainable routers in SMoE training. Instead of employing a random but fixed router, HyperRouter utilizes a random but fixed hypernetwork [START_REF] Ha | Hy-perNetworks[END_REF] to generate the router's parameters based on a trainable router embedding. By doing so, HyperRouter can improve the routing policy throughout training while mitigating the representation collapse issues. To demonstrate its effectiveness, we conduct extensive evaluations on various NLP tasks, comparing HyperRouter to several state-of-the-art SMoE routing strategies. Moreover, HyperRouter achieves the same performance threshold with fewer experts (compute) during inference, thereby significantly enhancing the efficiency of deploying LLMs in real-world applications. Fig. 1 shows that HyperRouter consistently outperforms other competitors when the same number of experts is used during inference.

Related Work

Existing Routing Mechanism for SMoE. One of the most important components for training SMoE is the expert routing strategy, which specifies experts to process input tokens. There are two common classes of token-expert assignment algorithms for SMoE training: (1) letting tokens select the top-k experts and (2) letting experts select the topk tokens. For the first approach, (Fedus et al., 2022b, Switch Transformer), (Lepikhin et al., 2021, GShard), (Zuo et al., 2022, THOR), (Lewis et al., 2021, BASE), (Clark et al., 2022, S-BASE), and SMoE-Dropout [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF] are representative methods. Meanwhile, [START_REF] Zhou | Mixture-of-Experts with Expert Choice Routing[END_REF] introduced expert choice that enables selecting different experts for each token and demonstrates the potential of the second approach.

On the Representation Collapse of SMoE. A major research focus is how to improve the tokenexpert assignment to avoid the representation collapse issue where: (i) all the inputs are routed to the same expert [START_REF] Zuo | Taming Sparsely Activated Transformer with Stochastic Experts[END_REF] or (ii) all experts converge to similar representations [START_REF] Chi | On the Representation Collapse of Sparse Mixture of Experts[END_REF][START_REF] Chen | Towards Understanding the Mixtureof-Experts Layer in Deep Learning[END_REF]. Such issues result in poor specialization among experts, parameter redundancy, and limited performance gains. Existing works address the issue (i) by employing various ad hoc-heuristics, e.g., adding Gaussian noise [START_REF] Vaswani | Attention is All you Need[END_REF], limiting the maximum number of inputs that can be routed to an expert [START_REF] Lepikhin | GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding[END_REF], imposing a load balancing loss (Fedus et al., 2022b), using linear assignment [START_REF] Lewis | BASE Layers: Simplifying Training of Large, Sparse Models[END_REF], or eliminating the necessity for router networks ( [START_REF] Zuo | Taming Sparsely Activated Transformer with Stochastic Experts[END_REF]) employed a consistent regularized loss for stochastic expert assignment, [START_REF] Roller | Hash Layers For Large Sparse Models[END_REF] incorporated deterministic hashing assignments). To resolve the issue (ii), [START_REF] Chi | On the Representation Collapse of Sparse Mixture of Experts[END_REF] proposed an X-MOE layer to improve the routing algorithm via dimension reduction, ℓ 2 normalization, and gating temperature. Furthermore, [START_REF] Dai | StableMoE: Stable Routing Strategy for Mixture of Experts[END_REF] proposed Stable-MoE with two training stages to reduce the router's fluctuations. The work related to ours the most is SMoE-Dropout [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF], which is considered a randomly initialized and fixed router to route tokens. Thus, the routing policy is stable during training and deterministic during inference. Our work improves upon SMoE-Dropout by extending the fixed router to a hypernetwork to further improve the routing policy during training while alleviating the collapsing issues.

Methodology

This section describes SMoE training and details of the proposed HyperRouter method.

SMoE Training

We first describe SMoE training of LLMs, which consists of a router R(•) with parameter W r and N expert networks {E i (•)} N i=1 with parameters W e i . We follow the most common implementation (Fedus et al., 2022b) to use a linear network as the router and split the feedforward networks in LLMs into N separated experts. Switch Transformer. Given an input token x with its representation h ∈ R d , the SMoE output y is calculated by routing h only to k-most suitable experts determined by the router, i.e.,

y = N j=1 R(h) j • E j (h) and
(1)

R(h) = TopK(σ(W r × h), k),
where the TopK(•, k) function keeps the largest k

HyperRouter

We describe our proposed HyperRouter strategy that balances between a random but fixed router and a router optimized during training. HyperRouter employs a fixed hypernetwork [START_REF] Ha | Hy-perNetworks[END_REF]) H(•) to dynamically generate the router's parameters conditioned on a trainable router embedding e. Specifically, HyperRouter obtains the output y as:

y = N j=1 R(h) j • E j (h) and R(h) = TopK(σ(W r × h), k) and W r = H(e),
where e is a low dimensional trainable vector associated with the current layer. All parameters are jointly optimized to minimize the task loss. Lastly, like SMoE-Dropout, HyperRouter also gradually increases the number of activated experts through-out training, starting at k = 2 and ending at k = N . Fig. 2 visually illustrates our proposed HyperRouter method.

A Deeper Analysis of HyperRouter

We now investigate the representation capabilities of HyperRouter compared to the naive SMoE and SMoE-Dropout. Due to space constraints, we only present the main results here and provide the detailed calculations in Appendix B. First, the naive SMoE jointly trains the router and expert parameters, leading to entanglement and eventually the representation collapse issue [START_REF] Zuo | Taming Sparsely Activated Transformer with Stochastic Experts[END_REF]. SMoE-Dropout proposed to fix the router parameter W r , which we will show to result in a restricted representation. To do so, we need to calculate the Jacobian with respect to h, which characterizes how the feature h is involved during training.

Jacobian of SMoE and SMoE-Dropout. Let

S SMOE j (h) = σ(W r ×h) j , S Hyper j (h) = σ(H(e)×
h) j , 1 ji be the indicator function, and W r,i be the icolumn of W r , the Jacobians for SMoE and SMoE-Dropout are calculated as:

∇ h L =J ⊤ 1 ∇ y L + J ⊤ 2 ∇ y L =J ⊤ 1 ∇ y L + k i=1 c i W r,i , where (2)
c i = k j=1 S SMOE j (h) 1 ji -S SMOE i (h) × E j (h) ⊤ ∇ y L .
Here, the Jacobian ∇ h L is a combination of two terms related to J 1 and J 2 . The J 1 component represents how h contributes to the output y and is the same for all methods. The second component related to J 2 characterizes learning better experts' representation of h and is the main difference. Since SMoE-Dropout fixes W r , its J2 = J ⊤ 2 ∇ y L term is expressed as a linear combination of columns in W r , which lies in a lower dimensional subspace because W r ∈ R k×d with k ≪ dnumber of experts is usually smaller than the feature dimension. Thus, SMoE-Dropout alleviates the entanglement between the router and experts at the cost of restricting the experts' representation capabilities.

Jacobian of HyperRouter.

By a similar calculation, the Jacobian of HyperRouter is shown in equation 3.

Given that e is trainable, HyperRouter's Jacobian is not expressed as a simple linear combination in a much lower-dimensional subspace. In contrast, HyperRouter maps the feature to a subspace whose dimension can be controlled by e, which we can easily set to be greater than k. Furthermore, by fixing the hypernetwork, the Jacobian does not change freely as in SMoE, which helps alleviate the entanglement. Overall, HyperRouter can alleviate the representation collapse while not sacrificing the representation capabilities. Jacobian of HyperRouter:

∇ h L =J ⊤ 1 ∇ y L + J ⊤ 2 ∇ y L =J ⊤ 1 ∇ y L + k i=1 c i H i (e), where (3)
J 1 = k j=1 S Hyper j (h)∇ h (E j (h)) , c i = k j=1 S Hyper j (h) 1 ji -S Hyper i (h) × E j (h) ⊤ ∇ y L .

Experiments

We conduct experiments to investigate the following hypotheses. First, HyperRouter improves efficiency compared to existing routing strategies, requiring less computation to achieve similar performance thresholds during evaluation. Second, with the same amount of experts used during inference, HyperRouter can achieve better performances compared to other competitors. [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF] to train the small and medium models for 400K iterations. We only train the large models for 100K iterations due to resource constraints. Second, finetuning on the SST-2 [START_REF] Socher | Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank[END_REF], SST-5 [START_REF] Socher | Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank[END_REF], IMDB [START_REF] Maas | Learning Word Vectors for Sentiment Analysis[END_REF], and BANKING77 [START_REF] Casanueva | Efficient intent detection with dual sentence encoders[END_REF]) datasets using the model pre-trained on enwik8. Similar to [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF], we also perform dense finetuning (always choosing all experts) for several epochs. In all experiments, we report the evaluation metrics on the test set when using different numbers of experts for routing methods. More implementation details and additional results are provided in the Appendix.

Experiment Setting

Most of our experiments follow

Pre-training Result

Tab

Finetuning Result

Tab. 3 reports the results of the finetuning experiment on the SST-2, SST-5, IMDB, and BANKING77 datasets. Although we perform dense finetuning, we also report the results of using only half the experts during evaluation. Overall, we observe consistent accuracy gains from HyperRouter compared to other baselines on all datasets. Notably, on SST-2 and IMDB datasets, HyperRouter with only eight experts could substantially outperform other strategies when using all experts. We now investigate the distributional output from the routers (softmax of router's output) trained with different strategies. Such outputs determine how tokens are assigned to experts. We hypothesize that high-entropy distributions are not preferred since they are closer to being uniform, indicating the router's low confidence in choosing experts for the current token. In the extreme, the router outputs a uniform distribution and assigns tokens to all experts, eventually causing the collapse issue. Therefore, a router with a lower entropy is preferred since it confidently assigns tokens to a few experts, which can improve experts' specialization. To this end, we report the entropy of each router in the small TransformerXL trained on the enwik8 dataset in Tab. 4. The entropy is calculated on all samples in the test set using the model obtained after training. The result clearly shows that HyperRouter achieved much lower entropy than SMoE and SMoE-Dropout. Due to space constraints, we will provide additional visualizations and discussions in Appendix D.

Router Analysis

Conclusion

In

A Additional Figures

Fig. 3 illustrates the conceptual difference between the traditional SMoE (Fedus et al., 2022b), SMoE-Dropout [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF], and our HyperRouter. In summary, SMoE uses a trainable router to send tokens to experts. On the other hand, SMoE-Dropout fixed a randomly initialized router and gradually increased the number of experts chosen throughout training. Lastly, HyperRouter improves upon SMoE-Dropout by replacing the router with a fixed hypernetwork and trainable router embedding.

B Derivation of the Jacobian

This Section details the calculations of the Jacobians presented in Section. 3.3. The Jacobian of SMoE and SMoE-Dropout is similar to the Jacobian of HyperRouter. Therefore, only the later detailed calculation is shown. Recall that HyperRouter obtains the output y as follows:

y = N j=1 R(h) j • E j (h), R(h) =TopK(σ(H(e) × h), k).
Here, σ(•) is the standard Softmax function given by:

σ(H(e) × h) j = exp((H(e) × h) j ) N i=1 exp((H(e) × h) i ) ≡ S Hyper j (h), ∀j = 1, . . . , N.
Without loss of generality and for simplicity, by rearranging the index of the top k-experts, HyperRouter can be rewritten as follows:

y = k j=1 S Hyper j (h) • E j (h), where S Hyper j (h) = exp((H(e) × h) j ) k i=1 exp((H(e) × h) i )
, ∀j = 1, . . . , k. The Jacobian matrix J of the output y with respect to h can be decomposed into two terms as follows:

∇ h y =∇ h   k j=1 S Hyper j (h) • E j (h)   = k j=1 S Hyper j (h)∇ h (E j (h)) + k j=1 ∇ h S Hyper j (h) E j (h) = k j=1 S Hyper j (h)∇ h (E j (h)) + k j=1 k i=1 S Hyper j (h) 1 ji -S Hyper i (h) × E j (h)H i (e) ⊤ ≡ J 1 + J 2 (4)
(since e is independent of h). Note that the last equation in ( 4) is obtained by using the chain rule, the logarithmic derivative, and the inner product as follows:

∂s j ∂z i = s j • ∂ ∂z i log(s j ) = s j • (1 ji -s i ),
where

s j = exp(z j ) k i=1 exp(z i ) , ∀j = 1, . . . , k.
It is worth mentioning that the first term J 1 means to produce a better token representation given the current activation router S Hyper j (h), while the second term J 2 represents learning a better gating function for the appropriate activation score router S Hyper j (h). After the back-propagation, the gradi- ent of the loss function L is obtained from the two paths mentioned above and is written as follows

∇ h L =∇ y L × ∇ h y = ∇ h y ⊤ × ∇ y L =(J 1 + J 2 ) ⊤ × ∇ y L (using (4)) =J ⊤ 1 ∇ y L + J ⊤ 2 ∇ y L.
(5) Finally, by expanding the second term as follows, we obtain the desired result:

J ⊤ 2 ∇ y L = k i=1 k j=1 S Hyper j (h) 1 ji -S Hyper i (h) × E j (h) ⊤ ∇ y L H i (e) = k i=1 c i H i (e), where (6)
c i = k j=1 S Hyper j (h) 1 ji -S Hyper i (h) × E j (h) ⊤ ∇ y L .

C Additional Experiments

This section provides the implementation details of our experiments in Sec. 4.

C.1 General Setting

Our experiments are conducted based on the publicly available SMoE-Dropout [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF]) implementation1 . Moreover, the pre-training experiments were conducted on a single A40 or A100 GPU, while the finetuning experiments were conducted on a single GeForce RTX 3090 GPU. We also emphasize that parallel training on multiple GPUs might yield different results.

Model architecture. The small TransformerXL variant [START_REF] Chen | Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers[END_REF] consists of 4 Transformer decoder layers with an input dimension of 256. Each layer consists of a self-attention layer with 8 attention heads, followed by a feedforward network with an inner dimension of 512. The dropout ratio is kept at 0.1. We split the feedforward network into 16 experts with the same dimension. For the HyperRouter , we initialize the embeddings with size 256 and employ a 2-layer perceptron with an inner dimension of 256 as the hypernetwork. The ReLU function is used as the activation function for the hypernetwork. Totally our HyperRouter introduces an additional 1024 trainable parameters in the TransformerXL model.

It is worth noting that the parameter overhead is fixed as we scale up to larger transformer models.

For the medium and large variants, we scale the model to eight and twelves layers, respectively.

C.2 Pre-training Experiments

Tab. A1 provides the implementation details for pre-training our TransformerXL small and medium on enwik8 and WikiText-103. The Trans-formerXL large network was trained in the same maner, but only for 100K iterations. [START_REF] Pham | Learning fast and slow for online time series forecasting[END_REF] can further reduce this cost while improving the performance, which we will leave for the future work. This Section provides the full details of the routers' entropy and visualizes its distributional output. This is supplementary to Section 4.4. Tab. A5 reports the mean and standard deviations of the entropy at each router. This table is the full version of Tab. A5. We can see that all methods have rather low standard deviation, indicating that the differences are significant. We also provide an illustrative example of the routers' outputs using a randomly picked sample on the test set in Fig. 4. Here we can clearly see that the policies from SMoE and SMoE-Dropout are much closer to uniform while HyperRouter 's policies are much sharper and have lower entropy. We emphasize that this example is not cherry-picked since we already calculated the averaged entropy on all samples in Tab. A5. Overall, this result shows insights into how HyperRouter can perform better than other state-of-the-art SMoE strategies.

D Routing Visualization

E Future Work

Our HyperRouter opens several promising venues for future research. Particularly, we believe that investigating two HyperRouter components: (i) fixed hypernetwork, and (ii) trainable embedding can yield further performance and efficiency gains. Potentials directions include incorporating regularization such as ℓ 2 -penalty or dropout [START_REF] Peng | A comparative study on regularization strategies for embedding-based neural networks[END_REF] or using better hypernetwork initialization techniques [START_REF] Chang | Principled Weight Initialization for Hypernetworks[END_REF]. Furthermore, the current implementation uses a hypernetwork for each transformer layer, and it generates all parameters of the router. Sharing hypernetworks among layers or generating the router coordinate-wise can offer knowledge sharing [START_REF] Yin | Mitigating forgetting in online continual learning with neuron calibration[END_REF][START_REF] Pham | Learning fast and slow for online time series forecasting[END_REF], which can further improve the result.

Figure 1 :

 1 Figure 1: Perplexity (log-scaled) on the WikiText-103 dataset with varying numbers of experts used for inference. All methods have the same FLOPs.

Figure 2 :

 2 Figure 2: An illustration of our HyperRouter that dynamically generates the router's parameters from a fixed hypernetwork. Yellow modules including the router embedding and experts are trainable while the gray module , the hypernetwork, is frozen. Best viewed in colors. values of a given vector while setting the remaining values to zero; σ(•) is the standard Softmax function. In practice, only a few experts are activated for the current token (k ≪ N ) for efficient training. SMoE-Dropout. SMoE-Dropout (Chen et al., 2023) is a state-of-the-art strategy that addresses the representation collapse issues (Sec. 2) by simply fixing a randomly initialized router to improve the token routing consistency. Furthermore, SMoE-Dropout gradually increases the number of experts chosen throughout training, starting with k = 2 and ending at k = N . During the evaluation, SMoE-Dropout proposes to use half or all experts to balance its efficiency and performance.

Figure 3 :

 3 Figure 3: An illustrative comparison among SMoE, SMoE-Dropout, and our HyperRouter . The input token is a representation vector (could be output from the previous layer). Yellow modules are trainable. Gray modules are frozen, indicated by a lock symbol ( ). Best viewed in colors.

Figure 4 :

 4 Figure 4: Visualization of the distribution of the routers' output.

Table 1 :

 1 Bit-per-character and Perplexity on the enwik8 and WikiText-103 test sets, respectively. Lower is better. k denotes the number of experts chosen during inference. The best results are in bold.

	Chen et al. (2023)
	and consider a small TransformerXL (Dai et al.,
	2019) with four layers in our experiments due to
	the limited resources. We also investigate the scal-
	ability by considering larger variants of the Trans-
	formerXL with eight or twelve layers. We compare
	our HyperRouter with several state-of-the-art rout-
	ing strategies: (i) SMoE (Fedus et al., 2022b) -train-
	ing with trainable routers; (ii) THOR (Zhou et al.,
	2022) -replacing routers with a stochastic experts
	selection strategy and a consistency regularizer;
	(iii) SMoE-Dropout (Chen et al., 2023) -using a
	random router and gradually increasing the number
	of selected experts during training. We also include

two traditional baselines: (i) Dense -the standard training of transformers where no routing mechanisms are implemented; and (ii) Dense+Dropout similar to Dense but with Dropout

[START_REF] Srivastava | Dropout: A Simple Way to Prevent Neural Networks from Overfitting[END_REF]

inserted in the fully connected layers. All

Table 2 :

 2 Bit-per-character on the enwik8 test set using the TransformerXL medium and large models. Lower is better. k denotes the number of experts chosen during inference, and SD denotes the SMoE-Dropout method. The best results are in bold.

		Medium TransformerXL	Large TransformerXL
	k	SMOE	SD	HyperRouter SMOE	SD	HyperRouter
	1	2.170 3.741	1.457	1.370 1.766	1.215
	2	1.395 1.642	1.201	1.216 1.298	1.181
	4	1.260 1.257	1.156	1.189 1.225	1.174
	8	1.230 1.167	1.137	1.182 1.197	1.171
	16 1.226 1.141	1.132	1.181 1.183	1.170

Table 3 :

 3 Transfer performance (Accuracy) on the SST-2, SST-5, IMDB, and BANKING77 datasets. Higher is better. k denotes the number of experts chosen during inference. The best results are in bold.

		SST-2	SST-5	IMDB	BANKING77
	Dense	82.18	39.72	88.30	84.54
	Dense+Dropout	81.94	39.95	88.37	84.34
	SMoE-Dropout (k=8)	81.37	41.17	87.78	87.40
	HyperRouter (k=8)	82.87	41.26	88.43	87.66
	SMoE (k=16)	79.98	39.45	88.16	86.52
	SMoE-Dropout (k=16)	81.83	41.30	87.81	87.89
	HyperRouter (k=16)	83.33	41.80	88.46	88.41
	HyperRouter performs competitively with SMoE-
	Dropout while only using half of the experts during
	evaluation.
	Tab. 2 reports the BPC on the enwik8 dataset using
	the medium and large TransformerXL. Notably, we
	only train the large model 100K iterations due to re-
	source constraints. With larger backbone networks,
	we observe the gap between our HyperRouter and
	the baselines becomes more significant, indicating
	our HyperRouter enjoys good scalability with the
	model complexity. Lastly, we observe that SMoE-
	Dropout does not achieve satisfactory results with
	the large backbone when only trained with 100K
	and is outperformed by the naive SMoE. This result
	shows that SMoE-Dropout requires significant re-
	sources to achieve good performance. On the other
	hand, our HyperRouter consistently outperforms
	both baselines, regardless of the backbone size and
	number of experts activated.

outperforms SMoE-Dropout when using only one expert, reducing BPC from 3.02 to 1.48 on enwik8, and perplexity from 560.93 to 65.17 on WikiText-103. Overall, with eight experts or less,

Table 4 :

 4 Average entropy of the distribution of the routers on the enwik8 dataset. Lower is better.

	Method	Router 1	Router 2	Router 3	Router 4
	SMoE	2.5164	2.2126	2.2424	2.2583
	SMoE-Dropout	2.5849	2.1624	2.2579	2.2690
	HyperRouter	1.4393	0.8894	1.1269	1.3477

Table A1 :

 A1 Implementation details for pre-training experimentson enwik8 and WikiText-103 datasets.

	C.3 Finetuning Experiments
	To conduct finetuning experiments, we use the
	same model architecture as in pre-training. Tab.
	A2 provides the implementation details used for
	finetuning experiments on four different datasets.
	Dataset	Input length Batch size Optimizer	Lr	# Iterations
	enwik8	512	22	Adam	2.5e-4	400000
	WikiText-103	512	22	Adam	2.5e-4	400000

Table A2 :

 A2 Implementation details for finetuning experiments on four different datasets.

	Dataset	Input length Batch size Optimizer	Lr	# Epochs
	SST-2	512	16	Adam	1e-4	3
	SST-5	512	16	Adam	1e-4	3
	IMDB	512	4	Adam	1e-4	3
	BANKING77	512	16	Adam	1e-4	15
	C.4 Inference Efficiency
	We discuss an implementation detail that makes our
	HyperRouter have the same efficiency during in-
	ference as SMoE and SMoE-Dropout. Particularly,
	before performing inference on any input samples,
	we use the hypernetwork to generate the router
	for each layer and then store them. During infer-
	ence, HyperRouter does not need to re-generate
	routers. Therefore, HyperRouter enjoys the same
	inference complexity as SMoE and SMoE-Dropout
	because their routers have the same dimensionality.
	Tab. A3 reports the number of FLOPs of different
	methods during inference on the enwik8 dataset
	according to a different number of experts chosen
	(k). Note that the FLOPs do not scale linearly with
	k since SMoE training is only applied to the fully
	connected layers, while other layers (multi-head
	self-attention, layer normalization, etc.) remain the
	same across baselines. In the extreme, using only
	one expert only incurs 45% of the FLOPs compared
	to using all 16 experts.

Table A3 :

 A3 Inference FLOPS (10 10 ) on the enwik8 dataset, k denotes the number of experts used during inference.

	k Dense Dense+Dropout THOR SMoE SMoE-Dropout HyperRouter
	1	-	-	-	3.47	3.47	3.47
	2	-	-	-	4.00	4.00	4.00
	4	-	-	-	4.54	4.54	4.54
	8	-	-	-	5.61	5.61	5.61
	16 7.76	7.76	7.66	7.66	7.66	7.66

C.5 Parameter Comparison

Tab. A4 provides the number of parameters in different components of SMoE-Dropout and HyperRouter . There are three categories: (i) trainable parameters which are the transformer

Table A4 :

 A4 Number of parameters in different components of SMoE-Dropout and HyperRouter during training. Blue parameters are trainable, red parameters are frozen, and underline parameter are dynamically generated in each iteration. HyperRouter is neglectable. Moreover, the number of frozen parameters (hypernetworks) is quite small compared to the transformer backbone (5.7%). Investigating into sharing the hypernetworks across layers or generating the routers coordinately

	Method	Router Emb. Router HyperNetwork Transformer
	SMoE-Dropout	-	16.448	-	19, 505.350
	HyperRouter	1.024	16.448	1, 112.576	19, 505.350
	backbone and the router embeddings; (ii) frozen
	parameters which are the routers SMoE-Dropout
	and hypernetworks in HyperRouter ; (iii) dy-
	namic parameters that are dynamically gener-
	ated in each iteration, such as the routers in
	HyperRouter . Overall, the additional trainable
	parameters in

Table A5 :

 A5 Average entropy of the distribution of the routers' output on the enwik8 dataset. Lower is better.

	Method	Router 1	Router 2	Router 3	Router 4
	SMoE	2.5164	2.2126	2.2424	2.2583
		± 0.17	± 0.31	± 0.30	± 0.25
	SMoE-Dropout	2.5849	2.1624	2.2579	2.2690
		± 0.11	± 0.37	± 0.28	± 0.27
	HyperRouter	1.4393	0.8894	1.1269	1.3477
		± 0.42	± 0.46	± 0.46	± 0.50

https://github.com/VITA-Group/ Random-MoE-as-Dropout

Acknowledgement

This research/project is supported by the National Research Foundation, Singapore, under its AI Singapore Programme (AISG Award No: AISG2-RP-2021-027).

Ethics Statement

Despite encouraging results, training large-scale LLMs is inevitably costly and requires extensive computational resources, which need to be properly managed. Moreover, our work used data collected on the web, which has been known to suffer from gender and racial biases and requires additional efforts to mitigate its negative impacts. Lastly, our study is a promising step towards facilitating the development of new LLMs, which still requires careful regularization to avoid potential misuses in harmful applications.


ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Le vers dans la pomme d'Adam

  Violences sexuelles et reprises de pouvoir Le vers dans la pomme d'Adam Émilie Notéris Édition électronique URL : https://journals.openedition.org/fixxion/2289 DOI : 10.4000/fixxion.2289 ISSN : 2295-9106 Éditeur Ghent University Référence électronique Émilie Notéris, « Le vers dans la pomme d'Adam », Revue critique de fixxion française contemporaine [En ligne], 24 | 2022, mis en ligne le 15 juin 2022, consulté le 15 juin 2022. URL : http:// journals.openedition.org/fixxion/2289 ; DOI : https://doi.org/10.4000/fixxion.2289 Ce document a été généré automatiquement le 15 juin 2022. Les contenus de la Revue critique de fixxion française contemporaine sont mis à disposition selon les termes de la licence Creative Commons Attribution -Pas d'Utilisation Commerciale -Pas de Modification 4.0 International.

  Le vers dans la pomme d'AdamRevue critique de fixxion française contemporaine, 24 | 2022

		Le vers dans la pomme d'Adam
		Elle décline l'étoile jaunie
		Il écrit sa traumabiographie
		Le larynx écrasé
		Pour elle,
		La poésie arrêtée
		Pour lui,
		La philosophie recommencée
		As-tu bien reçu le massage ?
		Message et massage
		Personnels et contrariés
		Responsabilité diminuée
		Inventaire des effets
		Déviation cervicale trotskyste Émilie Notéris Une série d'histoires tristes
		Violée par le père
		Rejetée par la mère
		Tuée par le mari
		Normal et supérieur
	1	À Mexico, il se prend pour Guillaume Tell Comme une femme aime un homme extérieur
		Show and Tell, dit-elle Comme un homme aime une femme intérieure
		Il tire droit devant
		Manque sa cible, dit-il Mais de quatre-vingt à quatre-vingt-
		Manque le verre, son reflet à elle dedans Cinq étages newyorkais plus bas
		Tous les verres qu'il n'aura pas manqué de boire, Le gratte-ciel touche la terre du doute raisonnable
		À sa santé Enfant perdue d'une famille illustre, arrachée à Cuba
		Dégradée, abîmée Jetée dans le vide du doute escamotable
		À ses déboires, Nourriture chinoise et champagne
		Benzédrine, amphétamines et téquila Sans amour c'est sa chair qu'il empoigne
		Ne la manque pas, Pour la précipiter trente-cinq étages newyorkais plus bas
		Elle, ne lui manque pas Que terre où elle se vautre encore
		Ivre de jeunes-qui et d'écriture-quoi Dans les musées aux murs blancs de Pandore
		Trouver le terme adéquat À l'odeur refroidie du passage à tabac
		Il a viré le second sexe Elle est la vie il est le froid
		Pour étudier le premier sexe Autoportrait en sang avant l'heure
		Le premier texte Prémédité face à un art sans humeurs
		Je ne serais jamais devenu écrivain
		Une Star calibre .380 Où est Ana Mendieta ?
		Si elle n'était pas morte Où est Hélène Legotien ?
		Ce sont ses mots à lui qui rapportent Où est Joan Vollmer ?
		Posés dans la balance, face à sa vie
		À elle, perdue Art, philosophie et poésie
		Une vie de littérature et de philosophie Toujours l'histoire des grands hommes enterre
		Qui n'aura pas été vécue L'histoire des femmes volontaires
		La poésie documentaire ne saurait se contenter
		De cinquante-et-un D'être règlement-taire et enchantée
		À quatre-vingt Elle a trop de comptes à régler
		De Mexico à Paris Avec la fausse vérité
		Toujours la même litanie
		Revue critique de fixxion française contemporaine, 24 | 2022