Jeremy E Martin
email: jeremy.martin@ens-lyon.fr

Pierre-Olivier Antoine

Vincent Perrier

Jean- Loup Welcomme

Gregoire Metais

Laurent Marivaux

A large crocodyloid from the Oligocene of the Bugti Hills, Pakistan

Cenozoic continental deposits of the Bugti Hills crop out South of the Sulaiman Range in Balochistan, Pakistan, and they have been renowned since [START_REF] Vickary | Geological report on a portion of the Beloochistan Hills[END_REF] for their exceptionally rich vertebrate-bearing Miocene localities. This led G.E. Pilgrim (Geological Survey of India) then C. Foster-Cooper (University of Cambridge) to conduct expeditions in the beginning of the 20 th century. More recent expeditions in the late 1990s have reported the occurrence of successive fossiliferous horizons spanning the Eocene-Pleistocene interval (Welcomme et al., 1997[START_REF] Welcomme | Himalayan Forelands: palaeontological evidence for Oligocene detrital deposits in the Bugti Hills (Balochistan, Pakistan)[END_REF]Welcomme and Ginsburg, 1997). Fossil-yielding series around Dera Bugti document marine, coastal, then fluvio-lacustrine depositional environments during the Eocene, the earliest Oligocene, and the late Oligocene-Pleistocene, respectively [START_REF] Welcomme | Himalayan Forelands: palaeontological evidence for Oligocene detrital deposits in the Bugti Hills (Balochistan, Pakistan)[END_REF]Métais et al., 2009b;Roddaz et al., 2011;[START_REF] Antoine | Mammalian Neogene biostratigraphy of the Sulaiman Province, Pakistan[END_REF]. The most species-rich intervals, at least for mammals, are Oligocene and Miocene in age [START_REF] Marivaux | Première découverte de Cricetidae (Rodentia, Mammalia) oligocènes dans le synclinal Sud de Gandoï (Bugti Hills, Balouchistan, Pakistan)[END_REF][START_REF] Marivaux | A fossil lemur from the Oligocene of Pakistan[END_REF](Marivaux et al., , 2002a(Marivaux et al., , 2002b[START_REF] Marivaux | Anthropoid primates from the Oligocene of Pakistan (Bugti Hills): data on early anthropoid evolution and biogeography[END_REF][START_REF] Antoine | A new rhinoceros from the lower Miocene of the Bugti Hills, Baluchistan, Pakistan: the oldest elasmotheriine[END_REF][START_REF] Antoine | First record of Paleogene Elephantoidea (Proboscidea, Mammalia) from the Bugti Hills of Pakistan[END_REF][START_REF] Antoine | New remains of the baluchithere Paraceratherium bugtiense (Pilgrim, 1910) from the Late/latest Oligocene of the Bugti Hills, Balochistan, Pakistan[END_REF][START_REF] Antoine | A revision of Aceratherium blanfordi Lydekker, 1884 (Mammalia: Rhinocerotidae) from the early Miocene of Pakistan: postcranials as a key[END_REF][START_REF] Antoine | Mammalian Neogene biostratigraphy of the Sulaiman Province, Pakistan[END_REF][START_REF] Métais | New artiodactyl ruminant mammal from the late Oligocene of Pakistan[END_REF][START_REF] Métais | New remains of the enigmatic cetartiodactyl Bugtitherium grandincisivum Pilgrim 1908, from the upper Oligocene of the Bugti Hills (Balochistan, Pakistan)[END_REF]Métais et al., , 2009a[START_REF] Métais | A new assemblage of stem pecoran ruminants from the Oligocene Chitarwata Formation, Bugti Hills, Baluchistan, Pakistan: Paleoenvironmental and paleobiogeographic implications[END_REF][START_REF] Marivaux | Diatomyid and baluchimyine rodents from the Oligocene of Pakistan (Bugti Hills, Balochistan): systematic and paleobiogeographic implications[END_REF][START_REF] Orliac | Listriodon guptai Pilgrim, 1926 (Mammalia, Suidae) from the early Miocene of the Bugti Hills, Balochistan, Pakistan: new insights into early Listriodontinae evolution and biogeography[END_REF][START_REF] Orliac | Suoidea (Mammalia, Cetartiodatyla) from the early Oligocene of the Bugti Hills, Balochistan, Pakistan[END_REF]. As for crocodylians, fossil remains were initially described from the Bugti Hills by [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF][START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF], but their precise stratigraphic provenance has not been recorded. Most of those fossils are quite fragmentary and they consist of large tomistomine skull fragments [START_REF] Martin | The taxonomic content of the genus Gavialis from the Siwalik Hills of India and Pakistan[END_REF] formerly referred to species of Gavialis [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF][START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF]. Together with these remains, Crocodilus bugtiensis [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF] was erected from a left maxilla and associated skull fragments of remarkable size found at Pishi Nala. Since its formal description four years later [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF], the taxon has been briefly recorded from several levels in the Bugti Hills (Welcomme et al., 1997) but no detailed account has been given ever since. Here, we describe partial, yet diagnostic, mandibular elements of a large crocodyloid recovered from upper Oligocene deposits of the Bugti Hills, which are morphologically compatible with the skull elements described by [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF]. This material was collected from a well-identified Oligocene horizon within the framework of the field campaigns that took place in Baluchistan between 1995 and 2000.

We also report a partial mandible collected in the 1920s from Baluchistan but with no provenance data. Finally, we mention the presence of toothmarks on associated megaherbivore remains. GEOLOGICAL SETTING [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] states that the holotype specimen IM E221 comes from the lowest beds in the series in Pishi Nala, Bugti Hills. Unfortunately, this information remains ambiguous and depending on the occurrence and current understanding of the systematics of co-occurring ruminant mammals, one may consider its provenance as lower Oligocene or upper Oligocene-lower Miocene [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF][START_REF] Antoine | Mammalian Neogene biostratigraphy of the Sulaiman Province, Pakistan[END_REF][START_REF] Métais | A new assemblage of stem pecoran ruminants from the Oligocene Chitarwata Formation, Bugti Hills, Baluchistan, Pakistan: Paleoenvironmental and paleobiogeographic implications[END_REF]. For this reason, we consider IM E221 as Oligocene in age. On the other hand, the age and provenance of UM-DB-LCJ1-02 is clearer. This specimen was later collected in the Bugti Hills (Baluchistan, Pakistan) by the members of the "Mission Paléontologique Française au Baloutchistan" (MPFB), in deposits situated in the south of the Zin Anticline, Lundo Chur section, and upper Oligocene in age (see [START_REF] Welcomme | Himalayan Forelands: palaeontological evidence for Oligocene detrital deposits in the Bugti Hills (Balochistan, Pakistan)[END_REF][START_REF] Antoine | First record of Paleogene Elephantoidea (Proboscidea, Mammalia) from the Bugti Hills of Pakistan[END_REF][START_REF] Antoine | New remains of the baluchithere Paraceratherium bugtiense (Pilgrim, 1910) from the Late/latest Oligocene of the Bugti Hills, Balochistan, Pakistan[END_REF]Métais et al., 2009b;[START_REF] Orliac | Suoidea (Mammalia, Cetartiodatyla) from the early Oligocene of the Bugti Hills, Balochistan, Pakistan[END_REF]. UM-DB-LCJ1-02 originates from the J1 horizon (lower elephantoid-bearing horizon; [START_REF] Antoine | First record of Paleogene Elephantoidea (Proboscidea, Mammalia) from the Bugti Hills of Pakistan[END_REF][START_REF] Antoine | New remains of the baluchithere Paraceratherium bugtiense (Pilgrim, 1910) from the Late/latest Oligocene of the Bugti Hills, Balochistan, Pakistan[END_REF] of the Lundo Chur section, about 25 km south of Dera Bugti [START_REF] Welcomme | Himalayan Forelands: palaeontological evidence for Oligocene detrital deposits in the Bugti Hills (Balochistan, Pakistan)[END_REF]Métais et al., 2009b).

Institutional

SYSTEMATIC PALEONTOLOGY

4 CROCODYLIA [START_REF] Gmelin | Systema Naturae[END_REF]CROCODYLOIDEA Fitzinger, 1826 Family incertae sedis ASTORGOSUCHUS, gen. nov. [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF] Diagnosis-As for the type and only known species Etymology-From the Greek 'Astorgos' meaning merciless or inexorable; and 'souchos', from the Greek name of the crocodile-headed god in Egyptian mythology (Sobek); referring to the inexorable fate of animals being seized in such powerful jaws.

Type species-Crocodilus bugtiensis

ASTORGOSUCHUS BUGTIENSIS, combinatio nova (Figs. 1234) Holotype-IM E221, parts of a single cranium including a nearly complete left maxilla and connected fragments of a premaxilla, nasal and lacrimal [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] and Fig. 1). [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] reports this specimen as originating from Pishi Nala in the Bugti Hills.

Referred Material-UM-DB-LCJ1-02, a mandibular symphysis including the dentaries and splenials (Figs. 2,3). NHMUK R.5266 (Fig. 4) was presented to NHMUK in 1925 by Clive Foster-Cooper and is labeled as originating from the Miocene of Baluchistan.

However, its precise origin (locality) and age (horizon) remain unknown.

Locality and Horizon-Oligocene of the Bugti Hills, Baluchistan, Pakistan.

Diagnosis-Astorgosuchus bugtiensis is characterized by the following unique combination of skull characters as noted by [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] (autapomorphies are denoted with *): total skull length larger than 80 cm for mature individuals; short-snouted skull as indicated by a rostrum only twice as long as wide; broad-headed skull as indicated by a rostrum at level of the largest maxillary alveolus as wide as the width in front of the orbits; largest maxillary alveolus is the fifth; presence of a deep notch lateral to the premaxillary-maxillary suture; presence of a deep notch located between the sixth and seventh maxillary alveoli for reception of the 11 th dentary tooth; enlarged external nares extending far posteriorly, almost reaching the level of the enlarged fifth maxillary alveolus*; nasals contributing to the external nares; presence of swollen and rugose prominences on the dorsal surface of the lacrimal (not contributing to the orbital margin)*, premaxilla and anterior part of the nasals. Astorgosuchus bugtiensis also displays a unique combination of mandibular characters among Crocodyloidea including: a mandibular symphysis encompassing the first seven dentary alveoli*; fused splenials penetrating in the mandibular rostrum to the level of the sixth dentary alveolus* and elevated contiguous large fourth and diminutive fifth dentary alveoli*. Differs from crocodylines, mekosuchines and the basal crocodyloid Asiatosuchus in having splenials involved in the mandibular symphysis; differs from tomistomines in having a shorter mandibular rostrum.

Description-The two newly reported specimens (NHMUK R.5266 and UM-DB-LCJ1-02) comprise the fused right and left anterior portions of mandibular rami (Figs. 234).

In NHMUK R.5266, the left mandibular ramus is preserved to the level of the ninth alveolus and the associated splenial is flanked on the entire preserved medial surface of the dentary ramus. However, the symphysial surface is somehow damaged and the extent to which the splenial contributes to it, is difficult to precisely ascertain (Fig. 4A-C). The relative dimensions and positions of dentary alveoli are identical to those of UM-DB-LCJ1-02. For those reasons, our description focuses on UM-DB-LCJ1-02.

UM-DB-LCJ1-02 preserves each dentary from the first to the seventh dentary alveoli.

Several cracks run through the specimen, the anteriormost edge of the left dentary is slightly worn out, some of the ventral surface of the right dentary is eroded and teeth have been broken away with only roots remaining in some alveolar positions. The specimen has not undergone deformation and the left and right rami are symmetrical. The lateral surface of the dentary is profusely ornamented with deep circular foramina. They are sparsely distributed and slightly larger on the ventral surface.

The robust mandibular rostrum is slightly longer than wide, giving to it a general spoon-shape appearance when viewed dorsally (Figs. 23). Anterior to the largest alveoli, the outline of the rostrum forms an isosceles triangle in dorsal view, while the lateral margins behind the largest alveoli are parallel to each other. The dentaries meet medially from the level of the first to the fifth alveoli. As observed in occlusal view, the mandibular symphysis is strongly serrated and it forms a median ridge, which stops at the level of the adjoining anterior splenial processes. The splenials penetrate in the mandibular rostrum with the shape of an inverted Y-process, their anterior two-thirds running nearly parallel to the outer mandibular margin. Here, the occlusal surface of the splenials is faintly concave. The portion of the splenials contributing to the mandibular rostrum extends posteriorly to a point level the sixth and seventh dentary alveoli. The splenials do not contribute to the ventral surface of the mandible. As seen in posterior view, their dorsal margin forms a shelf in the mandibular rostrum. Below this shelf, the splenials enclose the small meckelian foramen, which opens posteroventrally (Fig. 3C,D).

Alveolar dimensions are presented in Table 1. All preserved dentary alveoli are nearly circular in outline ( Figs 2A, B and3A, B). The first alveolus is facing anterodorsally and seems to have hosted a procumbent tooth. The first right alveolus hosts a broken tooth root, which is markedly curved. The second alveolus is slightly smaller than the first one and is separated from it by less than a centimeter. It is also anterodorsally directed. A two-centimeter diastema separates the second from the third alveolus. The third alveolus opens on the anterior everted surface of the large fourth alveolus, both alveoli being contiguous but not confluent. The fourth alveolus is the largest of all preserved alveoli and it projects in a dorsolateral direction. Its lingual alveolar wall bulges along the occlusal surface of the mandibular rostrum. The fifth alveolus is much smaller and is also contiguous with the fourth alveolus, its dorsal margin protruding at the same level as the fourth alveolus. The sixth and seventh alveoli open at the level of the main occlusal surface of the dentary, and are about the same size. They are separated from each other with the seventh alveolus possessing a distinct outline along the lateral border of the dentary. Occlusal pits occur on the lateral margin of the dentary between the sixth and seventh alveoli. The medial margin of the dentary tooth row is pierced by numerous foramina of variable dimensions. The largest (about 13 mm in diameter) are observed at the level of the diastema between the second and third alveoli as well as medial to the fourth through sixth alveoli. A single tooth crown is preserved in the left third dentary alveolus of NHMUK R.5266 (Fig. 4). Despite a circular cross section, the crown is heavily worn and no feature of the enamel can be described. DISCUSSION [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF][START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] was the first to describe and name several crocodylian species from the Bugti Hills, including a number of longirostrine forms [START_REF] Martin | The taxonomic content of the genus Gavialis from the Siwalik Hills of India and Pakistan[END_REF]. With these remains, [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF] erected Crocodilus bugtiensis, which was later described [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] and represented by a portion of the rostrum (IM E221) from Pishi Nala near Dera Bugti. Although no overlapping material exists to relate the skull material described by [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF] with the mandibular material described above, we consider all these specimens to represent a single species for the following reasons: first, IM E 221, NHMUK R.5266 and UM-DB-LCJ1-02 are all of large dimensions (i.e. skull length estimated over 80 cm; Appendix 1) and all retain characters of the Crocodyloidea, as explained in the paragraph below. The skull (IM E221) as well as the two mandibles described here independently point to an animal with a remarkably broad cranium [START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF]; Appendix 1). The morphology of the maxillary and dentary tooth rows shares the same pattern of occlusion. Among crocodylians, at the level of the premaxillary-maxillary suture, a marked notch on the lateral margin of the rostrum always accommodate an enlarged dentary tooth of the anterior part of the lower jaw.

Compatible with the deeply excavated premaxillary-maxillary notch, the mandibular specimens described herein show a distinctly enlarged fourth dentary tooth that was occluding at this level. Finally, the enlarged fifth maxillary tooth occludes laterally at the level of the sixth-seventh dentary alveoli, where an occlusal pit is well visible in NHMUK R.5266.

In light of these observations, we view all these three specimens as conspecific. The generic name 'Crocodilus' has traditionally been applied to a variety of fossil crocodylians with a "generalized non-alligatorid-like" morphology [START_REF] Delfino | First European evidence for transcontinental dispersal of Crocodylus (late Neogene of southern Italy)[END_REF]. The new taxon from the Bugti Hills cannot be assigned to the genus Crocodylus or to any members of the Crocodylinae because of the significant posterior extent of the external nares and because its splenial is involved in the mandibular symphysis. For these reasons, we propose the name Astorgosuchus bugtiensis as a new combination for the Bugti crocodylian represented by IM E221, NHMUK R.5266 and UM-DB-LCJ1-02.

The fragmentary nature of Astorgosuchus bugtiensis does not facilitate a clear taxonomic attribution beyond Crocodyloidea. Although we did not examine the holotype firsthand, we view IM E221 as a large crocodyloid as evidenced by its enlarged fifth maxillary alveolus (Pilgrim, 1912, plate XXVI). The presence of an enlarged fourth dentary alveolus is plesiomorphic and observed in alligatorines, caimanines, and crocodyloids. The most notable character is the participation of the splenial in a relatively short, mandibular symphysis, which extends posteriorly to a point level with the seventh dentary alveoli. This character is not encountered in basal eusuchians or alligatoroids nor in any short-snouted basal eusuchians. On the contrary, deep splenial participation into the mandibular symphysis is observed in longirostrine forms, such as tomistomines and gavialids, where the mandibular symphysis is extensive and encompasses most of the mandibular length, a condition not approached in A. bugtiensis. In alligatoroids, the mandibular symphysis extends posteriorly at the level of the fourth dentary alveolus. In crocodyloids, the symphysis is slightly more elongate, extending posteriorly to a point level with the fifth alveolus. In some extinct crocodyloids such as species of the mekosuchine Kambara from Australia (Salisbury and Willis, 1996) or the basal crocodyloids 'Crocodilus' depressifrons from Europe [START_REF] Delfino | A reassessment of the morphology and taxonomic status of 'Crocodylus' depressifrons (Crocodylia, Crocodyloidea) based on the Early Eocene remains from Belgium[END_REF] and Asiatosuchus (Mook, 1940;Berg, 1966), the symphysis extends to the level of the sixth or seventh alveoli. In Kambara and 'C.' depressifrons, however, the splenials do not participate in the mandibular symphysis, although they approach it. Mook (1940, p. 2) describes the splenial in Asiatosuchus grangeri as almost approaching the symphysis but not actually doing so. The taxon most closely resembling Astorgosuchus bugtiensis is Asiatosuchus germanicus where the splenials are slightly involved in the symphysis (Berg, 1966, p.57), but admittedly, Astorgosuchus bugtiensis displays a stronger splenial involvement into the symphysis, reaching the sixth dentary alveoli. Although the precise placement of Astorgosuchus bugtiensis in a phylogenetic framework is beyond the scope of this study, we view Astorgosuchus bugtiensis as a probable basal crocodyloid close to Asiatosuchus germanicus in light of the following shared characters: mandibular symphysis extending to the level of the seventh alveolus; participation of the splenial into the mandibular symphysis; diastema between second and third alveoli and enlarged fourth alveolus.

Astorgosuchus bugtiensis differs from Asiatosuchus germanicus in the relative dimensions of the fourth alveolus relative to the much smaller fifth dentary alveolus, and in its more extensive participation of the splenial in the mandibular symphysis. Astorgosuchus bugtiensis is recovered in much younger strata (Oligocene) than Asiatosuchus germanicus, which is Lutetian in age (i.e., middle Eocene). Although the affinities of the Asiatosuchus-like taxa remain unclear [START_REF] Delfino | A reassessment of the morphology and taxonomic status of 'Crocodylus' depressifrons (Crocodylia, Crocodyloidea) based on the Early Eocene remains from Belgium[END_REF], Astorgosuchus bugtiensis will require more complete specimens for its affinities to be assessed.

Total body length (TBL) of Astorgosuchus bugtiensis is estimated using mandibular width at the level of the largest dentary alveolus. A regression equation using skull and mandibular measurements of 13 individuals of the genus Crocodylus (Appendix 1) allowed to derive a skull length for A. bugtiensis of 80 cm for UM-DB-LCJ1-02 and of 91 cm for NHMUK R.5266. Measurements taken on a large series of Crocodylus porosus have demonstrated that the skull becomes shorter relative to the body length during ontogeny (Fukuda et al., 2013 and references therein). For this reason, the total length versus head length ratio (TL/HL) changes from 1:7 to well over 1:8 for C. porosus individuals with a skull length around 60 cm [START_REF] Fukuda | Estimation of total length from head length of saltwater crocodiles (Crocodylus porosus) in the Northern Territory, Australia[END_REF]. Using a 1:8 ratio would result in a minimum TBL estimate for A. bugtiensis of 6.4 meters for UM-DB-LCJ1-02 and of 7.3 meters for NHMUK R.5266. A TL/HL of 1:8.8 was established for the longest C. porosus ever measured, which possesses a skull length of 70 cm [START_REF] Britton | Here be a dragon: exceptional size in a saltwater crocodile (Crocodylus porosus) from the Philippines[END_REF], which is significantly smaller than the skull length estimated above for A. bugtiensis. Here, applying this ratio gives a TBL of 7 meters for UM-DB-LCJ1-02 and of 8 meters for NHMUK R.5266. Estimates should be taken with caution but one can reasonably consider probable that the largest representative of A. bugtiensis approached or surpassed 8 meters in TBL (Fig. S1).

The estimated size of adult A. bugtiensis falls in the range of other large fossil Crocodylus [START_REF] Brochu | A giant crocodile from the Plio-Pleistocene of Kenya, the phylogenetic relationships of Neogene African crocodylines, and the antiquity of Crocodylus in Africa[END_REF][START_REF] Delfino | A giant crocodile in the Dubois Collection from the Pleistocene of Kali Gedeh (Java)[END_REF]. The fossil record offers repeated instances of gigantism among semi-aquatic crocodylomorphs (Table 2), and based on cranial material, the largest exemplars could be estimated to surpass TBL of 10 meters [START_REF] Head | Systematics and body size of the gigantic, enigmatic crocodyloid Rhamphosuchus crassidens, and the faunal history of Siwalik Group (Miocene) crocodylians[END_REF][START_REF] Sereno | The giant crocodyliform Sarcosuchus from the Cretaceous of Africa[END_REF]Aguilar et al., 2006;[START_REF] Riff | The world's largest gharials Gryposuchus: description of G. croizati n. sp. (Crocodylia, Gavialidae) from the Upper Miocene Urumaco Formation, Venezuela[END_REF].

Astorgosuchus bugtiensis was retrieved in the same fluvio-lacustrine deposits as the large tomistomines 'Gavialis' curvirostris and 'Gavialis' breviceps [START_REF] Pilgrim | The Tertiary and post-Tertiary freshwater deposits of Baluchistan and Sind with notices of new vertebrates[END_REF][START_REF] Pilgrim | The vertebrate fauna of the Gaj series in the Bugti Hills and the Punjab. Memoirs oft he Geological Survey of India[END_REF][START_REF] Martin | The taxonomic content of the genus Gavialis from the Siwalik Hills of India and Pakistan[END_REF]. Both of the later taxa co-occur with several megaherbivores, such as rhinocerotoids and chalicotheriids among perissodactyls, anthracotheriid artiodactyls and proboscideans [START_REF] Antoine | First record of Paleogene Elephantoidea (Proboscidea, Mammalia) from the Bugti Hills of Pakistan[END_REF][START_REF] Antoine | New remains of the baluchithere Paraceratherium bugtiense (Pilgrim, 1910) from the Late/latest Oligocene of the Bugti Hills, Balochistan, Pakistan[END_REF][START_REF] Antoine | Mammalian Neogene biostratigraphy of the Sulaiman Province, Pakistan[END_REF]Métais et al., 2009a;[START_REF] Lihoreau | Enamel microstructure defines a major Paleogene hippopotamoid clade: the Merycopotamini (Artiodactyla, Hippopotamoidea)[END_REF]. Indeed, predator-prey interactions due to crocodylians are recorded on a wide array of mandibular and postcranial remains of rhinocerotoids, including the gigantic rhinocerotoid Paraceratherium bugtiense. The jaw of a juvenile specimen referred to as P. bugtiense has several large conical unhealed bisected tooth marks (Figure 5B) unambiguously left by a giant crocodylian (compare with [START_REF] Njau | A diagnosis of crocodile feeding traces on larger mammal bone, with fossil examples from the Plio-Pleistocene Olduva Basin, Tanzania[END_REF], as for several postcranial remains of P. bugtiense, rhinocerotoids, and chalicotheriids (Figure 5), which would support the conspicuous nature of such predator-prey interactions at those times. Nevertheless, it is not possible to differentiate between tooth marks inflicted by A. bugtiensis or by other large tomistomines due to their similar tooth morphology. Presumably, these different crocodylians inhabited the same streams but did not share the same microhabitats as is possibly the case today in India with gharials and muggers [START_REF] Da Silva | Mugger crocodile Crocodylus palustris[END_REF] or as previously discussed in various Mesozoic and Cenozoic aquatic habitats where both mesorostrine and longirostrine forms have been described (e.g. [START_REF] Salas-Gismondi | A Miocene hyperdiverse crocodylian community reveals peculiar trophic dynamics in proto-Amazonian mega-wetlands[END_REF][START_REF] Martin | Cranial anatomy of Pholidosaurus purbeckensis from the Lower Cretaceous of France and its bearing on pholidosaurid affinities[END_REF][START_REF] Souza | New crocodilian remains from the Solimões Formation (lower Eocene-Pliocene), state of Acre, Southwestern Brazilian Amazonia[END_REF]. Whatever the predator-prey interactions, this ecosystem is remarkable for the gigantic sizes attained by several of its organisms calling for further investigation of the underlying evolutionary mechanisms accounting for increase in body size.

  Abbreviations-IM, Indian Museum, Calcutta, India; NHMUK, Natural History Museum, London, United Kingdom; UM-DB-LC, Université de Montpellier-Dera Bugti, Lundo Chur locality.

Figure captions FIGURE 1 .

 captions1 Figure captions

FIGURE 2 .

 2 FIGURE 2. Photographs and associated drawings of the mandibular rostrum of

TABLE 1 .

 1 Table 2 for measurements). Alveolar dimensions (in mm) of the dentary Astorgosuchus bugtiensis (UM-DB-LCJ1-02). Abbreviations: a , mesiodistal length; b , labiolingual length; d , damaged alveolar border.

	Dentary	left a	left b	right a	right b
	alveolus
	#1	34 d	33 d	33	30
	#2	22	21	24 d	?
	#3	?	?	25	23
	#4	46	42	44	44
	#5	24	23	23	20
	#6	23	20	22	17
	#7	23	20	19	19

TABLE 2 .

 2 Maximum TBL of crocodylomorphs surpassing 7 meters stratigraphically sorted from oldest to youngest.

	Age	Taxon	estimated TBL Reference
	Early Cretaceous Chalawan thailandicus	7-8m	Martin et al. 2014
	Aptian-Albian	Sarcosuchus imperator	11-12m	Sereno et al. 2001
	Cenomanian	Stomatosuchus inermis	10m	Stromer, 1925
	Campanian	Deinosuchus sp.	9.1m	Erickson and Brochu, 1999
	early Eocene	Phosphatosaurus gavialoides 9m	Buffetaut, 1979
	late Oligocene	Astorgosuchus bugtiensis	8m	this study
	late Miocene	Mourasuchus mirandai	11.5m	Aguilera et al. 2006
	Miocene	Purussaurus brasiliensis	12.5m	Aureliano et al. 2015
	Mio-Pliocene	Gryposuchus croizati	10.15m	Riff and Aguilera, 2008
	Mio-Pliocene	Rhamphosuchus crassidens	8-11m	Head, 2001
	Pleistocene	Crocodylus thorbjarnarsoni	7.6m	Brochu and Storrs, 2012
	Pleistocene	Kali Gedeh Crocodylus	7m	Delfino and De Vos, 2013

ACKNOWLEDGMENTS

JEM thanks L. Steel who provided all necessary support during his stay at NHMUK. This research received support from the SYNTHESYS Project http://www.synthesys.info/ which is


Agatha Bedi

Julien Dalle

Martin Perrot

Michel Benoit

Mangrove restauration: biomimetic artificial

Keywords:

come

2) Mangrove forest vulnerable to erosion and strong wave action due to tree felling.

3) Deployment of ROOT in intertidal zone to restore hydrodynamics and sediment flow.

INTRODUCTION

Mangroves are productive habitats that provide a multitude of ecosystem services critical to mangrove dependent communities and to the local biodiversity in tropical and subtropical regions. Today, with the intensifying consequences of climate change and human activities, they play an essential role in protecting coastal communities by serving as natural buffers against storm surges, erosion due to current and wave actions, and flooding. Despite this, mangrove belts are still being depleted in an alarming rate. In response, massive propagule planting has become a popular approach in restoration programs. However, it has generally yielded a low ecological survival rate due to inappropriate planting methods and site-species selection and disregard to disturbed hydrological and sediment conditions due to aquaculture activities.

Mangroves, specifically their root system, play an essential role in facilitating sediment deposition and retention. These functions have given birth to the concept of the ROOT model, an innovative solution that replaces labour intensive hand planting through hydrological and ecological restoration for mangrove recovery. Many results can be drawn from these experiments. For instance, we note that using 2 modules with different porosity values (90 and 70%) maximizes energy dissipation and minimizes energy reflection. This is the best compromise for efficient wave attenuation without enhancing scouring effects at the front of the structure (due to limited reflection).

NEXT STEPS

  Mimicry of physical and hydrodynamic functions of pencil and prop roots to  Attenuate wave action  Restore tidal flow  Secure sediment supply  Foster mangrove growth  Serve as a wildlife habitat and nursery Sustainability  Side-uses for local community (sustainable aquaculture, blue carbon projects)  Environment-friendly material (locally made low pH and porous cement)  Laboratory tested in partnership with universities and research institutes Monitoring and Evaluation Large-scale Deployment two-fold wave dissipation 1) Healthy natural mangrove forest before deforestation.

  integration of ROOT models in restored mangrove forest. Determination of drag coefficients : Smooth Vertical Cylinders: • If 𝜏 and h are held constant, CD increases when D decreases (ie « roots » number grows) • If D and h are held constant, CD increases when 𝜏 decreases ( the density of cylinders increases) • If 𝜏 and D are constant, CD increases when h decreases Roughness and Tortuosity effects*: • CD increases significantly if roughness of cylinders is increased • CD seems to increase with tortuosity but difficult to compare with different tests in terms of equivalent porosity * Only 1 test conducted for each effect 1. The design and production of a bio-inspired wave attenuation device for mangrove environments is realistic and feasible. 2. Several characteristics, such as spatially-varying porosity, roughness, tortuosity, length of the global structure,… can be adjusted to optimize solution efficiency, i.e. maximizing energy dissipation and minimizing energy transmission through the device. 3. Additional tests need to be done with numerical tools to support design work and investigate deeper some effects. 4. Pilot operation should be launched, once the device is finalized, to check assess ROOT' functional performance at the field scale on : 1. Wave attenuation, sediment deposition, ecological functionalities and services 2. Improvement of spontaneous mangrove resilience on degraded areas 3. Societal benefits and local The ROOT model is inspired by the different aerial mangrove root systems of the genus, Rhizophora and Sonneratia. A series of the modules will be linked together and arranged in a certain pattern in degraded intertidal zones to foster natural regeneration of mangroves. Determine key structural components that characterize mangrove tree roots by generating waves in a tank containing an array of vertical and circular cylinders representing mangrove tree roots Duration: 6 months Materials and methods: Hexapod tests (geometric scale 1/2.5) Use: Experimental characterization of an array of cylinders undergoing forced horizontal motion at a given frequency and identification of environmental parameters defining the environment for a numeric model. Factors: diameter (20-63 mm), roughness (smooth or rough), tortuosity of cylinders (straight or tortuous) and porosity of array of cylinders (70-90%) Configurations: 8 configuration types and 18 tests in total, considering 3 water depths Types of water height (cm): 3 For each test, 21 frequency values (Hz) x 3 or 4 amplitudes of motion (mm) Wave Flume (geometric scale 1/16) Use: Measurement of progressive wave attenuation across different arrangements of of an array of vertical cylinders undergoing regular wave motion to obtain transmission and reflection coefficients of the modules. Data will be used for calibration and validation of the numeric models. Factors: model porosity, diameter (0.005-0.010 m) of cylinders Configurations: 6 types and 180 number of tests Types of water height (cm): 2 Other factors: 3 wave height values (mm), 5 wave periods (sec) Numerical Simulations Use: calculations based on deterministic 1DH wave propagation approach for small scale tests, and then 2DH spectral wave model SWAN for large scale tests. Factors: representative porosity of the array of cylinders, roughness of cylinders through the specification of a drag coefficient CD.

•Fig. 1 :

 1 Fig. 1: Response Amplitude Operator (RAO) of free surface elevation at the first sloshing mode of the tank. Porosity 𝜏 = 80%, water depth h = 235 mm, diameter D = 32 mm. Note roughness effect results in highest dissipation followed by porosity and tortuosity effects.

  Wave flume Fig. 2: examples of reflection, transmission and dissipation coefficients measured during the wave flume experiments.


Julien Seznec

Andrea Locatelli

Alexandra Carpentier

Alessandro Lazaric

Michal Valko

Rotting bandits are not harder than stochastic ones

, which suffers regret O(K 1/3 T 2/3 ). FEWA also matches known bounds for the stochastic bandit setting, thus showing that the rotting bandits are not harder. Finally, we report simulations confirming the theoretical improvements of FEWA.

Introduction

The multi-arm bandit framework [START_REF] Bubeck | Regret analysis of stochastic and nonstochastic multi-armed bandit problems[END_REF][START_REF] Lattimore | Bandit algorithms[END_REF] formalizes the exploration-exploitation dilemma in online learning, where an agent has to trade off the exploration of the environment to gather information and the exploitation of the current knowledge to maximize reward. In the stochastic setting [START_REF] William | On the likelihood that one unknown probability exceeds another in view of the evidence of two samples[END_REF] Proceedings of the 22 nd International Conference on Artificial Intelligence and Statistics (AISTATS) 2019, Naha, Okinawa, Japan. PMLR: Volume 89. Copyright 2019 by the author(s). Auer et al., 2002a), each arm is characterized by a stationary reward distribution. Whenever an arm is pulled, an i.i.d. sample from the corresponding distribution is observed. Despite the extensive algorithmic and theoretical study of this setting, the stationarity assumption is often too restrictive in practice, e.g., the preferences of users may change over time. The adversarial setting [START_REF] Auer | The nonstochastic multi-armed bandit problem[END_REF] addresses this limitation by removing any assumption on how the rewards are generated and learning agents should be able to perform well for any arbitrary sequence of rewards. While algorithms such as Exp3 [START_REF] Auer | The nonstochastic multi-armed bandit problem[END_REF] are guaranteed to achieve small regret in this setting, their behavior is conservative as all arms are repeatedly explored to avoid incurring too much regret because of unexpected changes in arms' values. This behavior results in unsatisfactory performance in practice, where arms' values, while non-stationary, are far from being adversarial. [START_REF] Garivier | On upperconfidence-bound policies for switching bandit problems[END_REF] proposed a variation of the stochastic setting, where the distribution of each arm is piecewise stationary. Similarly, [START_REF] Besbes | Stochastic multi-armed bandit problem with non-stationary rewards[END_REF] introduced an adversarial setting where the total amount of change in arms' values is bounded. These settings fall into the restless bandit scenario, where the arms' value evolves independently from the decisions of the agent. On the other hand, for rested bandits, the value of an arm changes only when it is pulled. For instance, the value of a service may deteriorate only when it is actually used, e.g., if a recommender system shows always the same item to the users, they may get bored [START_REF] Warlop | Fighting boredom in recommender systems with linear reinforcement learning[END_REF]. Similarly, a student can master a frequently taught topic in an intelligent tutoring system and extra learning on that topic would be less effective. A particularly interesting case is represented by the rotting bandits, where the value of an arm may decrease whenever pulled. [START_REF] Heidari | Tight policy regret bounds for improving and decaying bandits[END_REF] studied this problem when rewards are deterministic (i.e., no noise) and showed how a greedy policy (i.e., selecting the arm that returned the largest reward the last time it was pulled) is optimal up to a small constant factor depending on the number of arms K and the largest per-round decay in the arms' value L. [START_REF] Bouneffouf | Multi-armed bandit problem with known trend[END_REF] considered the stochastic setting when the dynamics of the rewards is known up to a constant factor. Finally, [START_REF] Levine | Rotting bandits[END_REF] considered both non-parametric and parametric noisy rotting bandits, for which they derive algorithms with regret guarantees. In the non-parametric case, where the decrease in reward is neither constrained nor known, they introduce the sliding-window average (wSWA) algorithm, which is shown to achieve a regret to the optimal policy of order O(K 1/3 T 2/3 ), where T is the number of rounds in the experiment.

In this paper, we study the non-parametric rotting setting of [START_REF] Levine | Rotting bandits[END_REF] and introduce Filtering on Expanding Window Average (FEWA) algorithm, a novel method that constructs moving average estimates of increasing windows to identify the arms that are more likely to perform well if pulled once more. Under the assumption that the reward decays are bounded, we show that FEWA achieves a regret of O( √ KT ), thus significantly improving over wSWA and matching the minimax rate of stochastic bandits up to a logarithmic factor. This shows that learning with non-increasing rewards is not more difficult than in the stationary case. Furthermore, when rewards are constant, we recover standard problem-dependent regret guarantees (up to constants), while in the rotting bandit scenario with no noise, the regret reduces to the one of [START_REF] Heidari | Tight policy regret bounds for improving and decaying bandits[END_REF]. Numerical simulations confirm our theoretical results and show the superiority of FEWA over wSWA.

Preliminaries

We consider a rotting bandit scenario similar to the one of [START_REF] Levine | Rotting bandits[END_REF]. At each round t, an agent chooses an arm i(t) ∈ K {1, ..., K} and receives a noisy reward r i(t),t . The reward associated to each arm i is a σ 2 -sub-Gaussian r.v. with expected value of µ i (n), which depends on the number of times n it was pulled before; µ i (0) is the initial expected value. 1 Let H t i(s), r i(s),s , ∀s < t be the sequence of arms pulled and rewards observed until round t, then

r i(t),t µ i(t) (N i(t),t ) + ε t with E[εt|Ht] = 0 and ∀λ ∈ R, E e λεt ≤ e σλ 2 2 ,
where N i,t t-1 s=1 I{i(t) = i} is the number of times arm i is pulled before round t. We use r i (n) to denote the random reward of arm i when pulled for the n+1-th time, i.e., r i(t),t = r i(t) (N i(t),t ). We introduce a nonparametric rotting assumption with bounded decay.

Assumption 1. The reward functions µ i are nonincreasing with bounded decays -L ≤ µ i (n + 1) -1 Our definition slightly differs from the one of [START_REF] Levine | Rotting bandits[END_REF]. We use µi(n) for the expected value of arm i after n pulls instead of when it is pulled for the n-th time.

µ i (n) ≤ 0. The initial expected value is bounded as µ i (0) ∈ [0, L]. We refer to this set of functions as L L .

The learning problem A learning policy π is a function from the history of observations to arms, i.e., π(H t ) ∈ K. In the following, we often use π(t) π(H t ). The performance of a policy π is measured by the (expected) rewards accumulated over time,

J T (π) T t=1 µ π(t) N π(t),t .
Since π depends on the (random) history observed over time, J T (π) is also random. We define the expected cumulative reward as J T (π) E J T (π) . We now restate a characterization of the optimal (oracle) policy.

Proposition 1 [START_REF] Heidari | Tight policy regret bounds for improving and decaying bandits[END_REF]. If the expected value of each arm {µ i (n)} i,n is known, the policy π maximizing the expected cumulative reward J T (π) is greedy at each round, i.e.,

π (t) = arg max i µ i (N i,t ).
(1)

We denote by J J T (π ) = J T (π ), the cumulative reward of the optimal policy.

The objective of a learning algorithm is to implement a policy π with performance as close to π 's as possible. We define the (random) regret as

R T (π) J -J T (π).
(2)

Notice that the regret is measured against an optimal allocation over arms rather than a fixed-arm policy as it is a case in adversarial and stochastic bandits. Therefore, even the adversarial algorithms that one could think of applying in our setting (e.g., Exp3 of Auer et al., 2002a) are not known to provide any guarantee for our definition of regret. On the other hand, for constant µ i (n)-s, our problem and definition of regret reduce to the one of standard stochastic bandits.

Let N i,T be the (deterministic) number of rounds that arm i is pulled by the oracle policy π up to round T (excluded). Similarly, for a policy π, let N π i,T be the (random) number pulls of arm i. The cumulative reward can be rewritten as

J T (π) = T t=1 i∈K I {π(t)=i} µ i N π i,t = i∈K N π i,T -1 s=0 µ i (s).
Then, we can conveniently rewrite the regret as

R T (π) = i∈K   N i,T -1 s=0 µ i (s) - N π i,T -1 s=0 µ i (s)   = i∈up N i,T -1 s=N π i,T µ i (s) - i∈op N π i,T -1 s=N i,T µ i (s), (3)
where we define up

i ∈ K|N i,T > N π i,T and like- wise op i ∈ K|N i,T < N π i,T
as the sets of arms that are respectively under-pulled and over-pulled by π w.r.t. the optimal policy.

Known regret bounds

We report existing regret bounds for two special cases. We start with the minimax regret lower bound for stochastic bandits.

Proposition 2. (Auer et al., 2002b, Thm. 5.1) For any learning policy π and any horizon T , there exists a stochastic stationary problem

µ i (n) µ i i with K σ-sub-Gaussian arms such that π suffers a regret E[R T (π)] ≥ σ 10 min √ KT , T .
where the expectation is w.r.t. both the randomization over rewards and algorithm's internal randomization. [START_REF] Heidari | Tight policy regret bounds for improving and decaying bandits[END_REF] derived regret lower and upper bounds for deterministic rotting bandits (i.e., σ = 0).

Proposition 3. (Heidari et al., 2016, Thm. 3) For any learning policy π, there exists a deterministic rotting bandits (i.e., σ = 0) satisfying Assumption 1 with bounded decay L such that π suffers an expected regret

E[R T (π)] ≥ L 2 (K -1).
Let π σ0 be the greedy policy that selects at each round the arm with the largest reward observed so far, i.e., π σ0 (t) arg max i (µ i (N i,t -1)). For any deterministic rotting bandits (i.e., σ = 0) satisfying Assumption 1 with bounded decay L, π σ0 suffers an expected regret

E[R T (π σ0 )] ≤ L(K -1).
Any problem in the two settings above is a rotting problem with parameters (σ, L). Therefore, the performance of any algorithm on the general rotting problem is also bounded by these two lower bounds.

3 FEWA: Filtering on expanding window average Since the expected rewards µ i change over time, the main difficulty in the non-parametric rotting bandits is that we cannot rely on all samples observed until Algorithm 1 FEWA Input: σ, K, α, δ 0 ← 1 1: pull each arm once, collect reward, and initialize N i,K ← 1 2: for t ← K + 1, K + 2, . . . do 3:

δ t ← δ 0 /(t α ) 4: h ← 1 {initialize bandwidth} 5: K 1 ← K {initialize with all the arms} 6: i(t) ← none 7: while i(t) is none do 8: K h+1 ← Filter(K h , h, δ t ) 9: h ← h + 1 10: if ∃i ∈ K h such that N i,t =
N i(t),t ← N i(t),t-1 + 1 16:
N j,t ← N j,t-1 , ∀j = i(t) 17: end for round t to predict which arm is likely to return the highest reward in the future. In fact, the older a sample, the less representative it is for future rewards. This suggests constructing estimates using the more recent samples. Nonetheless, discarding older rewards reduces the number of samples used in the estimates, thus increasing their variance. In Alg. 1 we introduce FEWA (or π F ) that at each round t, relies on estimates using windows of increasing length to filter out arms that are suboptimal with high probability and then pulls the least pulled arm among the remaining arms.

We first describe the subroutine Filter in Alg. 2, which receives a set of active arms K h , a window h, and a confidence parameter δ as input and returns an updated set of arms K h+1 . For each arm i that has been pulled n times, the algorithm constructs an estimate µ h i (n) that averages the h ≤ n most recent rewards observed from i. The subroutine Filter discards all the arms whose mean estimate (built with window h) from K h is lower than the empirically best arm by more than twice a threshold c(h, δ t ) constructed by standard Hoeffding's concentration inequality (see Prop. 4).

The Filter subroutine is used in FEWA to incrementally refine the set of active arms, starting with a window of size 1, until the condition at Line 10 is met. As a result, K h+1 only contains arms that passed the filter for all windows from 1 up to h. Notice that it is important to start filtering arms from a small window and to keep refining the previous set of active arms. In fact, the estimates constructed using a small window use recent rewards, which are closer to the future value of an arm. As a result, if there is enough evidence that Algorithm 2 Filter

Input: K h , h, δ t 1: c(h, δ t ) ← (2σ 2 /h) log (1/δ t ) 2: for i ∈ K h do 3: µ h i (N i,t ) ← 1 h h j=1 r i (N i,t -j) 4: end for 5: µ h max,t ← max i∈K h µ h i (N i,t ) 6: for i ∈ K h do 7: ∆ i ← µ h max,t -µ h i (N i,t ) 8: if ∆ i ≤ 2c(h, δ t ) then 9:
add i to K h+1 10:

end if 11: end for Output: K h+1 an arm is suboptimal already at a small window h, it should be directly discarded. On the other hand, a suboptimal arm may pass the filter for small windows as the threshold c(h, δ t ) is large for small h (i.e., as few samples are used in constructing µ h i (N i,t ), the estimation error may be high). Thus, FEWA keeps refining K h for larger windows in the attempt of constructing more accurate estimates and discard more suboptimal arms. This process stops when we reach a window as large as the number of samples for at least one arm in the active set K h (i.e., Line 10). At this point, increasing h would not bring any additional evidence that could refine K h further (recall that µ h i (N i,t ) is not defined for h > N i,t ). Finally, FEWA selects the active arm i(t) whose number of samples matches the current window, i.e., the least pulled arm in K h . The set of available rewards and the number of pulls are then updated accordingly.

Runtime and memory usage At each round t, FEWA needs to store and update up to t averages per-arm. Since moving from an average computed on window h to h + 1 can be done incrementally at a cost O(1), the worst-case time and memory complexity per round is O(Kt), which amounts to a total O(KT2 ) cost. This is not practical for large T . 2 We have a fix.

In App. E we detail EFF-FEWA, an efficient variant of FEWA. EFF-FEWA is built around two main ideas.3 First, at any round t we can avoid calling Filter for all possible windows h starting from 1 with an increment of 1. In fact, the confidence interval c(h, δ t ) decreases as 1/ √ h and we could select windows h with an exponential increment so that confidence intervals between two consecutive calls to Filter have a constant ratio. In practice, we replace the window increment (Line 9 of FEWA) by a geometric window h 2 j . This modification alone is not enough to reduce the computation. While we reduce the number of estimates that we construct, updating µ h i from h = 2 j to h = 2 j+1 still requires spanning over past samples, thus leading to the same O(Kt) complexity in the worst-case. In order to reduce the overall complexity, we avoid recomputing µ h i at each call of Filter and by replacing it with precomputed estimates. Whenever N i,t = 2 j for some j, we create an estimate s c i,j by averaging all the last N i,t samples. These estimates are then used whenever Filter is called with h = 2 j . Instead of updating s c i,j at each new sample, we create an associated pending estimate s p i,j which averages all the more recent samples. More formally, let t be the round when N i,t = 2 j , then s p i,j is initialized at 0 and it then stores the average of all the samples observed from t to t , when N i,t = 2 j+1 (i.e., s p i,j is averaging at most 2 j samples). At this point, the 2 j samples averaged in s c i,j are outdated and they are replaced by the new average s p i,j , which is then reinitialized to 0. The sporadic update of the precomputed estimates and the small number of them drastically reduces per-round time and space complexity to O(K log t). Furthermore, EFF-FEWA preservers the same regret guarantees as FEWA. In the worst case, s c i,j may not cover the last 2 j-1 -1 samples. Nonetheless, the precomputed estimates with smaller windows (i.e., j < j) are updated more frequently, thus effectively covering the 2 j-1 -1 samples "missed" by s c i,j . As a result, the active sets returned by Filter are still accurate enough to derive regret guarantees that are only a constant factor worse than FEWA (App. E).

Regret Analysis

We first give problem-independent regret bound for FEWA and sketch its proof in Sect. 4.1. Then, we derive problem-dependent guarantees in Sect. 4.2.

Theorem 1. For any rotting bandit scenario with means {µ i (n)} i,n satisfying Asm. 1 with bounded decay L and any horizon T , FEWA run with α = 5, i.e., δ t = 1/t 5 , suffers an expected regret4 of

E[R T (π F )] ≤ 13σ( √ KT + K) log(T ) + 2KL.
Comparison to [START_REF] Levine | Rotting bandits[END_REF] The regret of wSWA is bounded by O(µ

1/3 max K 1/3 T 2/3
) for rotting functions with range in [0, µ max ]. In our setting, we do not restrict rewards to stay positive but we bound the per-round decay by L, thus leading to rotting functions with range in [-LT, L]. As a result, when applying wSWA to our setting, we should set µ max = L(T + 1), which leads to O(T ) regret, thus showing that according to its original analysis, wSWA may not be able to learn in our general setting. On the other hand, we could use FEWA in the setting of [START_REF] Levine | Rotting bandits[END_REF] by setting L = µ max as the largest drop that could occur. In this case, FEWA suffers a regret of O( √ KT ), thus significantly improving over wSWA. The improvement is mostly due to the fact that FEWA exploits filters using moving averages with increasing windows to discard arms that are suboptimal w.h.p. Since this process is done at each round, FEWA smoothly tracks changes in the value of each arm, so that if an arm becomes worse later on, other arms would be recovered and pulled again. On the other hand, wSWA relies on a fixed exploratory phase where all arms are pulled in a round-robin way and the tracking is performed using averages constructed with a fixed window. Moreover, FEWA is anytime, while the fixed exploratory phase of wSWA requires either to know T or to resort to a doubling trick, which often performs poorly in practice.

Comparison to deterministic rotting bandits

For σ = 0, our upper bound reduces to KL, thus matching the prior (upper and lower) bound of [START_REF] Heidari | Tight policy regret bounds for improving and decaying bandits[END_REF] for deterministic rotting bandits. Moreover, the additive decomposition of regret shows that there is no coupling between the stochastic problem and the rotting problem as terms depending on the noise level σ are separated from the terms depending on the rotting level L, while in wSWA these are coupled by a L 1/3 σ 2/3 factor in the leading term.

Comparison to stochastic bandits The regret of FEWA matches the worst-case optimal regret bound of the standard stochastic bandits (i.e., µ i (n)s are constant) up to a logarithmic factor. Whether an algorithm can achieve O( √ KT ) regret bound is an open question. On one hand, FEWA needs confidence bounds to hold for different windows at the same time, which requires an additional union bound and thus larger confidence intervals w.r.t. UCB1. On the other hand, our worst-case analysis shows that some of the difficult problems that reach the worst-case bound of Thm. 1 are realized with constant functions, which is the standard stochastic bandits, for which MOSS-like [START_REF] Audibert | Minimax policies for adversarial and stochastic bandits[END_REF] algorithms achieve regret guarantees without the log T factor. Thus, the necessity of the extra log T factor for the worst-case regret of rotting bandits remains an open problem.

Sketch of the proof

We now give a sketch of the proof of our regret bound. We first introduce the expected value of the estimators used in FEWA. For any n and 1 ≤ h ≤ n, we define

µ h i (n) E µ h i (n) = 1 h h j=1 µ i (n -j).
Notice that at round t, if the number of pulls of arm i is

N i,t , then µ 1 i (N i,t ) = µ i (N i,t -1)
, which is the expected value of arm i the last time it was pulled. We introduce Hoeffding's concentration inequality and the favorable event that we leverage in the analysis. Proposition 4. For any fixed arm i, number of pulls n, and window h, we have that with probability 1 -δ,

µ h i (n) -µ h i (n) ≤ c(h, δ) 2σ 2 h log 1 δ • (4)
For any round t and confidence δ t δ 0 /t α , let

ξ t ∀i ∈ K, ∀n ≤ t, ∀h ≤ n, µ h i (n) -µ h i (n) ≤ c(h, δ t )
be the event under which the estimates constructed by FEWA at round t are all accurate up to c(h, δ t ). Taking a union bound gives

P(ξ t ) ≥ 1 -Kt 2 δ t /2.
Active set We derive an important lemma that provides support for the arm selection process obtained by a series of refinements through the Filter subroutine.

Recall that at any round t, after pulling arms {N πF i,t } i the greedy (oracle) policy would select an arm

i t N πF i,t i ∈ arg max i∈K µ i N πF i,t .
We denote by µ + t (π F ) max i∈K µ i (N πF i,t ), the reward obtained by pulling i t . The dependence on π F in the definition of µ + t (π F ) stresses the fact that we consider what the oracle policy would do at the state reached by π F . While FEWA cannot directly match the performance of the oracle arm, the following lemma shows that the reward averaged over the last h pulls of any arm in the active set is close to the performance of the oracle arm up to four times c(h, δ t ). Lemma 1. On the favorable event ξ t , if an arm i passes through a filter of window h at round t, i.e., i ∈ K h , then the average of its h last pulls satisfies

µ h i (N πF i,t ) ≥ µ + t (π F ) -4c(h, δ t ). ( 5
)
This result relies heavily on the non-increasing assumption of rotting bandits. In fact, for any arm i and any window h, we have

µ h i (N πF i,t ) ≥ µ 1 i (N πF i,t ) ≥ µ i (N πF i,t
). While the inequality above for i * t trivially satisfies Eq. 5, Lem. 1 is proved by integrating the possible errors introduced by the filter in selecting active arms due to the error of the empirical estimates.

Relating FEWA to the oracle policy While Lem. 1 provides a link between the value of the arms returned by the filter and the oracle arm, i t is defined according to the number of pulls obtained by FEWA up to t, which may significantly differ from the sequence of pulls of the oracle policy. In order to bound the regret, we need to relate the actual performance of the optimal policy to the value of the arms pulled by FEWA. Let h i,t

N πF i,t -N i,t be the absolute difference in the number of pulls between π F and the optimal policy up to t. Since i N πF i,t = i N i,t = t, we have that i∈op h i,t = i∈up h i,t which means that there are as many total overpulls as underpulls. Let j ∈ up be an underpulled arm 5 with N πF j,T < N j,T , then, for all s ∈ {0, . . . , h j,t }, we have the inequality

µ + T (π F ) = max i∈K µ i (N πF i,T ) ≥ µ j (N πF j,T + s). (6)
As a result, from Eq. 3 we have the regret upper bound

R T (π F ) ≤ i∈op h i,T -1 h=0 µ + (π F ) -µ i (N π i,T + h) , (7)
where we have obtained the inequality by bounding µ i (t ) ≤ µ + T (π F ) in the first summation and then using i∈op h i,T = i∈up h i,T . While the previous expression shows that we can just focus on over-pulled arms in op, it is still difficult to directly control the expected reward µ i (N π i,T + h), as it may change at each round (by at most L). Nonetheless, we notice that its cumulative sum can be directly linked to the average of the expected reward over a suitable window. In fact, for any i ∈ op and h i,T ≥ 2, we have

(h i,T -1)µ h i,T -1 i (N i,T -1) = h i,T -2 t =0 µ i (N π i,T + t ).
At this point we can control the regret for each i ∈ op in Eq. 7 by applying the following corollary of Lem. 1. Corollary 1. Let i ∈ op be an arm overpulled by FEWA at round t and h i,t N πF i,t -N π i,t ≥ 1 be the difference in the number of pulls w.r.t. the optimal policy π at round t. On the favorable event ξ t , we have

µ + t (π F ) -µ hi,t i (N i,t ) ≤ 4c(h i,t , δ t ). ( 8
)

Problem-dependent guarantees

Since our setting generalizes the standard stochastic bandit setting, a natural question is whether we pay any price for this generalization. While the result of [START_REF] Levine | Rotting bandits[END_REF] suggested that learning in rotting bandits could be more difficult, in Thm. 1 we actually proved that FEWA nearly matches the problem-independent regret O( √ KT ). We may wonder whether this is true for the problem-dependent regret as well.

5 If such arm does not exist, then πF suffers no regret.

Remark 1. Consider a stationary stochastic bandit setting with expected rewards {µ i } i and µ max i µ i . Corollary 1 guarantees that for δ

t ≥ 1/T α , µ -µ i ≤ 4c(h i,T -1, δ t ) = 4 2ασ 2 log(T ) h i,T -1 or equivalently, h i,T ≤ 1 + 32ασ 2 log(T ) (µ -µ i ) 2 • (9)
Therefore, our algorithm matches the lower bound of [START_REF] Tze | Asymptotically efficient adaptive allocation rules[END_REF] up to a constant, thus showing that learning in the rotting bandits are never harder than in the stationary case. Moreover, this upper bound is at most α larger than the one for UCB1 (Auer et al., 2002a). 6 The main source of suboptimality is the use of a confidence bound filtering instead of an upperconfidence index policy. Selecting the less pulled arm in the active set is conservative as it requires uniform exploration until elimination, resulting in a factor 4 in the confidence bound guarantee on the selected arm (vs. 2 for UCB), which implies 4 times more overpulls than UCB (see Eq. 9). We conjecture that this may not be necessarily needed and it is an open question whether it is possible to derive either an index policy or a better selection rule. The other source of suboptimality w.r.t. UCB is the use of larger confidence bands because of the higher number of estimators computed at each round (Kt 2 instead of Kt for UCB).

Remark 1 also reveals that Corollary 1 can be used to derive a general problem-dependent result in the rotting case. In particular, with Corollary 1 we upperbound the maximum number of overpulls by a problem dependent quantity

h + i,T max h ≤ 1 + 32ασ 2 log(T ) ∆ 2 i,h-1 , ( 10
)
where ∆ i,h min j∈K

µ j N j,T -1 -µ h i N i,t + h .
We then use Corollary 1 again to upper-bound the regret caused by h + i,T overpulls for each arm, leading to Theorem 2 (see the full proof in App. D).

Theorem 2. For α = 5 and C α 32ασ 2 , the regret of FEWA is bounded as

E[R T (π F )] ≤ i∈K C 5 log(T ) ∆ i,h + i,T -1 + C 5 log(T ) + 2L .

2-arms

We design numerical simulations to study the difference between wSWA and FEWA. We consider rotting bandits with two arms defined as

µ 1 (n) = 0, ∀n ≤ T and µ 2 (n) = L 2 if n < T 4 , -L 2 if n ≥ T 4 •
The rewards are then generated by applying a Gaussian i.i.d. noise N (0, σ = 1). The single point of nonstationarity in the second arm is designed to satisfy Asm. 1 with a bounded decay L. It is important to notice that in this specific case, L also plays the role of defining the gap ∆ between the arms, which is known to heavily impact the performance both in stochastic bandits and in the rotting bandits (Cor. 2). In particular, for any learning strategy, the gap between the two arms is always ∆ = |µ 1 (n 1 ) -µ 2 (n 2 )| = L/2. Recall that in stochastic bandits, the problem independent bound O( √ KT ) is obtained by the worst-case choice of ∆ K/T . In the two-arm setting defined above, the optimal allocation is N 1,T = 3T /4 and N 2,T = T /4.

Algorithms Both algorithms have a parameter α to tune. In wSWA, α is a multiplicative constant to tune the window. We test three different values of α, including the recommendation of [START_REF] Levine | Rotting bandits[END_REF], α = 0.2. In general, the smaller the α, the smaller the averaging window and the more reactive the algorithm is to large drops. Nonetheless, in stationary regimes, this may correspond to high variance and poor regret. On the other hand, a large value of α may reduce variance but increase the bias in case of rapidly rotting arms. Thm. 3.1 of [START_REF] Levine | Rotting bandits[END_REF] reveals this trade-off in the regret bound of wSWA, which has a factor (αµ max + α -1/2 ), where µ max is the largest arm value. The best choice of α is then µ -2/3 max , which reduces the previous constant to µ 1/3 max . In our experiment, µ max = L and we could expected that for any fixed α, wSWA may perform well in cases when α ≈ µ -2/3 max , while the performance may degrade for larger µ max .

In FEWA, α tunes the confidence δ t = 1/(t α ) used in c(h, δ t ). While our analysis suggests α = 5, the analysis of confidence intervals, union bounds, and filtering algorithms is too conservative. Therefore, we use more aggressive values, α ∈ {0.03, 0.06, 0.1}.

Experiments In Fig. 1, we compare the performance of the two algorithms and their dependence on L. The first plot shows the regret at T for various values of L. The second and the third plot show the regret as a function of T for L = 0.23 and L = 4.24, which corresponds to the worst empirical performance for FEWA and to the L σ regime respectively. All experiments have T = 10000 and are averaged over 500 runs.

Before discussing the results, we point out that in the rotting setting, the regret can increase and decrease over time. Consider two simple policies: π 1 , which first pulls arm 1 for N 1,T times and then arm 2 for N 2,T times, and π 2 in reversed order (first arm 2 and then arm 1). If we take π 2 as reference, π 1 has an increasing regret for the first T /4 rounds, which then would plateau from T /4 up to 3T /4 as both π 1 and π 2 are pulling arm 1. Then from 3T /4 to T , the regret of π 1 would reverse back to 0 since π 2 would keep selecting arm 1 and getting a reward of 0, while π 1 transitions to pulling arm 2 with a reward of L/2.

Results Fig. 1 shows that the performance of wSWA depends on the proper tuning of α w.r.t. µ max = L, as predicted by Thm. 3.1 of [START_REF] Levine | Rotting bandits[END_REF]. In fact, for small values of L, the best choice is α = 0.2, while for larger values of L a smaller α is preferable. In particular, when L grows very large, the regret tends to grow linearly with L. On the other hand, FEWA seems much more robust to different values of L. Whenever T and σ are large compared to L, Thm. 1 suggests that the regret of FEWA is dominated by O(σ √ KT ), while the term KL becomes more relevant for large values of the drop L. We also notice that since L defines the gap between the value of µ 1 and µ 2 , the problemindependent bound is achieved for the worst-case choice of L ∼ 2 K/T , when the regret of FEWA is indeed the largest. Fig. 1 middle and right confirm these findings for the extreme choice of the worst-case value of L and the regime where the drop is much larger than the noise level, i.e., where the term KL dominates the regret.

We conclude that FEWA is more robust than wSWA as it almost always achieves the best performance across different problems while being agnostic to the value of L. On the other hand, wSWA's performance is very sensitive to the choice of α and the same value of the parameter may correspond to significantly different performance depending on L. Finally, we notice that EFF-FEWA has a regret comparable to FEWA.

10-arms

We also tested a rotting setting with 10 arms. The mean of 1 arm is constant with value 0 while the means of 9 arms abruptly decrease after 1000 pulls from +∆ i to -∆ i . ∆ i is ranging from 0.001 to 10 in a geometric sequence. In this setting, the regret can be written as R T (π) = i=1 h i,T ∆ i . Hence, the regret per arm is R i T (π) ∆ i h i,T . In Fig. 2, we compare the performance of different algorithms for their best parameter. The left plot shows the average regret as a function of time. The right plot shows the regret per arm (indexed by ∆ i ) at the end of the experiment. Results On Fig. 2 (left), we see that FEWA outperforms wSWA. On the right, we remark that no tuning of wSWA is able to perfom well for all ∆ i s. It is comparable with the 2-arms experiments where no tuning is good on all the experiments. We also test SW-UCB and D-UCB [START_REF] Garivier | On upperconfidence-bound policies for switching bandit problems[END_REF] with parameters tuned for this experiment. While the two algorithms are known benchmarks for non-stationary restless bandits, they are penalized in our rested bandits problem. Indeed, they keep exploring arms that have not been pulled for many rounds which is detrimental in our case as the arms stay constant when they are not pulled. Hence, there is no good choice for their forgetting parameters: A fast forgetting rate makes the policies repeatedly pull bad arms (whose mean rewards do not change when they are not pulled in the rested setting) while a slow forgetting rate makes the policies not able to adapt to abrupt shifts.

Last, we remark that EFF-FEWA is penalized for arms with small ∆ i , for which the impact of the delay is more significant. At the end of the game, EFF-FEWA suffers 22% more regret but reduce the computational time by 99.7% (Table 1).

Conclusion

We introduced FEWA, a novel algorithm for the nonparametric rotting bandits. We proved that FEWA achieves an O( √ KT ) regret without any knowledge of the decays by using moving averages with a window that effectively adapts to the changes in the expected rewards. This result greatly improves over the wSWA algorithm by [START_REF] Levine | Rotting bandits[END_REF], that suffers a regret of order O(K 1/3 T 2/3 ). Thus our result shows that the rotting bandit scenario is not harder than the stochastic one. Our technical analysis of FEWA hinges on the adaptive nature of the window size. The most interesting aspect of the proof technique is that confidence bounds are used not only for the action selection but also for the data selection, i.e., to identify the best window to trade off the bias and the variance in estimating the current value of each arm.

A Proof of core FEWA guarantees Lemma 1. On the favorable event ξ t , if an arm i passes through a filter of window h at round t, i.e., i ∈ K h , then the average of its h last pulls satisfies

µ h i (N πF i,t ) ≥ µ + t (π F ) -4c(h, δ t ). ( 5
)
Proof. Let i be an arm that passed a filter of window h at round t. First, we use the confidence bound for the estimates and we pay the cost of keeping all the arms up to a distance 2c(h, δ t ) of µ h max,t ,

µ h i (N i,t ) ≥ µ h i (N i,t ) -c(h, δ t ) ≥ µ h max,t -3c(h, δ t ) ≥ max i∈K h µ h i (N i,t ) -4c(h, δ t ), ( 11
)
where in the last inequality, we used that for all i ∈ K h ,

µ h max,t ≥ µ h i (N i,t ) ≥ µ h i (N i,t ) -c(h, δ t ).
Second, since the means of arms are decaying, we know that

µ + t (π F ) µ i t (N i t ,t ) ≤ µ i t (N i t ,t -1) = µ 1 i t (N i t ,t ) ≤ max i∈K µ 1 i (N i,t ) = max i∈K1 µ 1 i (N i,t ). (12)
Third, we show that the largest average of the last h means of arms in K h is increasing with h ,

∀h ≤ N i,t -1, max i∈K h +1 µ h +1 i (N i,t ) ≥ max i∈K h µ h i (N i,t ).
To show the above property, we remark that thanks to our selection rule, the arm that has the largest average of means, always passes the filter. Formally, we show that arg max

i∈K h µ h i (N i,t ) ⊆ K h +1 . Let i h max ∈ arg max i∈K h µ h i (N i,t )
. Then for such i h max , we have

µ h i h max (N i h max ,t ) ≥ µ h i h max (N i h max ,t ) -c(h , δ t ) ≥ µ h max,t -c(h , δ t ) ≥ µ h max,t -2c(h , δ t ),
where the first and the third inequality are due to confidence bounds on estimates, while the second one is due to the definition of i h max . Since the arms are decaying, the average of the last h + 1 mean values for a given arm is always greater than the average of the last h mean values and therefore, max

i∈K h µ h i (N i,t ) = µ h i h max (N i h max ,t ) ≤ µ h +1 i h max (N i h max ,t ) ≤ max i∈K h +1 µ h +1 i (N i,t ), (13)
because i h max ∈ K h +1 . Gathering Equations 11, 12, and 13 leads to the claim of the lemma,

µ h i (N i,t ) (11)
≥ max

i∈K h µ h i (N i,t ) -4c(h, δ t ) (13) ≥ max i∈K1 µ 1 i (N i,t ) -4c(h, δ t ) (12) ≥ µ + t (π F ) -4c(h, δ t ).
Corollary 1. Let i ∈ op be an arm overpulled by FEWA at round t and h i,t N πF i,t -N π i,t ≥ 1 be the difference in the number of pulls w.r.t. the optimal policy π at round t. On the favorable event ξ t , we have

µ + t (π F ) -µ hi,t i (N i,t ) ≤ 4c(h i,t , δ t ). (8)
Proof. If i was pulled at round t, then by the condition at Line 10 of Algorithm 1, it means that i passes through all the filters from h = 1 up to N i,t . In particular, since 1 ≤ h i,t ≤ N i,t , i passed the filter for h i,t , and thus we can apply Lemma 1 and conclude

µ h i (N i,t ) ≥ µ + t (π F ) -4c(h i,t , δ t ). (14)

B Proofs of auxiliary results

Lemma 2. Let h π i,t

|N π i,T -N i,T |.
For any policy π, the regret at round T is no bigger than

R T (π) ≤ i∈op h π i,T -1 h=0 ξ t π i (N i,T +h) µ + T (π) -µ i (N i,T + h) + T t=1 ξ t Lt.
We refer to the the first sum above as to A π and to the second one as to B.

Proof. We consider the regret at round T . From Equation 3, the decomposition of regret in terms of overpulls and underpulls gives

R T (π) = i∈up N i,T -1 t =N π i,T µ i (t ) - i∈op N π i,T -1 t =N i,T µ i (t ).
In order to separate the analysis for each arm, we upper-bound all the rewards in the first sum by their maximum µ + T (π) max i∈K µ i (N π i,T ). This upper bound is tight for problem-independent bound because one cannot hope that the unexplored reward would decay to reduce its regret in the worst case. We also notice that there are as many terms in the first double sum (number of underpulls) than in the second one (number of overpulls). This number is equal to op h π i,T . Notice that this does not mean that for each arm i, the number of overpulls equals to the number of underpulls, which cannot happen anyway since an arm cannot be simultaneously underpulled and overpulled. Therefore, we keep only the second double sum,

R T (π) ≤ i∈op h π i,T -1 t =0 µ + T (π) -µ i (N i,T + t ) . (15)
Then, we need to separate overpulls that are done under ξ t and under ξ t . We introduce t π i (n), the round at which π pulls arm i for the n-th time. We now make the round at which each overpull occurs explicit,

R T (π) ≤ i∈op h π i,T -1 t =0 T t=1 t π i N i,T + t = t µ + T (π) -µ i (N i,T + t ) ≤ i∈op h π i,T -1 t =0 T t=1 t π i N i,T + t = t ∧ ξ t µ + T (π) -µ i (N i,T + t ) Aπ + i∈op h π i,T -1 t =0 T t=1 t π i N i,T + t = t ∧ ξ t µ + T (π) -µ i (N i,T + t ) B .
For the analysis of the pulls done under ξ t we do not need to know at which round it was done. Therefore,

A π ≤ i∈op h π i,T -1 t =0 ξ t(N i,t +t ) µ + T (π) -µ i (N i,T + t ) .
For FEWA, it is not easy to directly guarantee the low probability of overpulls (the second sum). Thus, we upper-bound the regret of each overpull at round t under ξ t by its maximum value Lt. While this is done to ease FEWA analysis, this is valid for any policy π. Then, noticing that we can have at most 1 overpull per round t, i.e., i∈op

h π i,T -1 t =0 t π i N i,T + t = t ≤ 1, we get B ≤ T t=1 ξ t Lt   i∈op h π i,T -1 t =0 t π i N i,T + t = t   ≤ T t=1 ξ t Lt.
Therefore, we conclude that

R T (π) ≤ i∈op h π i,T -1 t =0 ξ t π i (N i,t +t ) µ + T (π) -µ i (N i,T + t ) Aπ + T t=1 ξ t Lt B . Lemma 3. Let h i,t h πF i,t = |N πF i,T -N i,T |.
For policy π F with parameters (α, δ 0 ), A πF defined in Lemma 2 is upper-bounded by

A πF i∈op h i,T -1 t =0 ξ t π F i (N i,t +t ) µ + T (π F ) -µ i (N i,T + t ) ≤ i∈op ξ 4 2ασ 2 log + (T δ -1/α 0 ) + 4 2ασ 2 h ξ i,T -1 log + (T δ -1/α 0 ) + L .
Proof. First, we define

h ξ i,T max h ≤ h i,T | ξ t π F i (N i,t +h)
, the last overpull of arm i pulled at round t i t πF i (N i,t + h ξ i,T ) ≤ T under ξ t . Now, we upper-bound A πF by including all the overpulls of arm i until the h ξ i,T -th overpull, even the ones under ξ t ,

A πF i∈op h π F i,T -1 t =0 ξ t π F i (N i,t +t ) µ + T (π F ) -µ i (N i,T + t ) ≤ i∈op ξ h ξ i,T -1 t =0 µ + T (π F ) -µ i (N i,T + t ) ,
where op ξ i ∈ op| h ξ i,T ≥ 1 . We can therefore split the second sum of h ξ i,T term above into two parts. The first part corresponds to the first h ξ i,T -1 (possibly zero) terms (overpulling differences) and the second part to the last (h ξ i,T -1)-th one. Recalling that at round t i , arm i was selected under ξ ti , we apply Corollary 1 to bound the regret caused by previous overpulls of i (possibly none),

A πF ≤ i∈op ξ µ + T (π F ) -µ i N i,T + h ξ i,T -1 + 4 h ξ i,T -1 c h ξ i,T -1, δ ti (16)
≤ i∈op ξ µ + T (π F ) -µ i N i,T + h ξ i,T -1 + 4 h ξ i,T -1 c h ξ i,T -1, δ T (17)
≤ i∈op ξ µ + T (π F ) -µ i N i,T + h ξ i,T -1 + 4 2ασ 2 h ξ i,T -1 log + T δ -1/α 0 , (18)
with log + (x) max(log(x), 0). The second inequality is obtained because δ t is decreasing and c(., ., δ) is decreasing as well. The last inequality is the definition of confidence interval in Proposition 4 with log + (T α ) ≤ α log + (T ) for α > 1. If N i,T = 0 and h ξ i,T = 1 then

µ + T (π F ) -µ i (N i,T + h ξ i,T -1) = µ + (π F ) -µ i (0) ≤ L,
since and µ + (π F ) ≤ L and µ i (0) ≥ 0 by the assumptions of our setting. Otherwise, we can decompose

µ + T (π F )-µ i (N i,T + h ξ i,T -1) = µ + T (π F ) -µ i (N i,T + h ξ i,T -2) A1 + µ i (N i,T + h ξ i,T -2) -µ i (N i,T + h ξ i,T -1) A2 .
For term A 1 , since arm i was overpulled at least once by FEWA, it passed at least the first filter. Since this h ξ i,T -th overpull is done under ξ ti , by Lemma 1 we have that

A 1 ≤ 4c(1, δ ti ) ≤ 4c(1, T -α ) ≤ 4 2ασ 2 log + T δ -1/α 0 . The second difference, A 2 = µ i (N i,T + h ξ i,T -2) -µ i (N i,T + h ξ i,T -1) cannot exceed L,
since by the assumptions of our setting, the maximum decay in one round is bounded. Therefore, we further upper-bound Equation 18as

A πF ≤ i∈op ξ 4 2ασ 2 log + T δ -1/α 0 + 4 2ασ 2 h ξ i,T -1 log + T δ -1/α 0 + L . ( 19
)
Lemma 4. Let ζ(x) = n n -x . Thus, with δ t = δ 0 /(Kt α ) and α > 4, we can use Proposition 4 and get

E[B] T t=1 p ξ t Lt ≤ T t=1 Ltδ 0 2t α-2 ≤ Lδ 0 ζ(α -3) 2 •
C Minimax regret analysis of FEWA Theorem 1. For any rotting bandit scenario with means {µ i (n)} i,n satisfying Asm. 1 with bounded decay L and any horizon T , FEWA run with α = 5, i.e., δ t = 1/t 5 , suffers an expected regret 7 of

E[R T (π F )] ≤ 13σ( √ KT + K) log(T ) + 2KL.
Proof. To get the problem-independent upper bound for FEWA, we need to upper-bound the regret by quantities which do not depend on {µ i } i . The proof is based on Lemma 2, where we bound the expected values of terms A πF and B from the statement of the lemma. We start by noting that on high-probability event ξ T , we have by Lemma 3 and α = 5 that

A πF ≤ i∈op ξ
4 10σ 2 log(T ) + 4 10σ 2 (h i -1) log(T ) + L .

Since op ξ ⊆ op and there are at most K -1 overpulled arms, we can upper-bound the number of terms in the above sum by K -1. Next, the total number of overpulls i∈op h i,T cannot exceed T . As square-root function is concave we can use Jensen's inequality. Moreover, we can deduce that the worst allocation of overpulls is the uniform one, i.e., h i,T = T /(K -1),

A πF ≤ (K -1)(4 10σ 2 log(T ) + L) + 4 10σ 2 log(T ) i∈op (h i,T -1) ≤ (K -1)(4 10σ 2 log(T ) + L) + 4 10σ 2 (K -1)T log(T ). (20)
Now, we consider the expectation of term B from Lemma 2. According to Lemma 4, with α = 5 and δ 0 = 1,

E[B] ≤ Lζ(2) 2 = Lπ 2 12 • (21)
Therefore, using Lemma 2 together with Equations 20 and 21, we bound the total expected regret as

E[R T (π F )] ≤ 4 10σ 2 (K -1)T log(T ) + (K -1)(4 10σ 2 log(T ) + L) + Lπ 2 6 • (22)
Corollary 2. FEWA run with α > 3 and δ 0 2δ/ζ(α -2) achieves with probability 1 -δ,

R T (π F ) = A πF ≤ 4 2ασ 2 log + T δ 1/α 0 K -1 + (K -1)T + (K -1)L.
Proof. We consider the event t≤T ξ t which happens with probability

1 - t≤T Kt 2 δ t 2 ≤ 1 - t≤T Kt 2 δ t 2 ≤ 1 - ζ(α -2)δ 0 2 •
Therefore, by setting δ 0 2δ/ζ(α -2), we have that B = 0 with probability 1 -δ since ξ t = 0 for all t. We can then use the same analysis of A πF as in Theorem 1 to get

R T (π F ) = A πF ≤ 4 2ασ 2 log + T δ 1/α 0 K -1 + (K -1)T + (K -1)L.
D Problem-dependent regret analysis of FEWA Lemma 5. A πF defined in Lemma 2 is upper-bounded by a problem-dependent quantity,

A πF ≤ i∈K 32ασ 2 log + (T δ -1/α 0 ) ∆ i,h + i,T -1 + 32ασ 2 log + (T δ -1/α 0 ) + (K -1)L.
Proof. We start from the result of Lemma 3,

A πF ≤ i∈op ξ 4 2ασ 2 log + (T δ -1/α 0 ) 1 + h ξ i,T -1 + (K -1)L. (23)
We want to bound h ξ i,T with a problem dependent quantity h + i,T . We remind the reader that for arm i at round T , the h ξ i,T -th overpull has been on ξ ti pulled at round t i . Therefore, Corollary 1 applies and we have µ

h ξ i,T -1 i N i,T + h ξ i,T -1 ≥ µ + T (π F ) -4c h ξ i,T -1, δ ti ≥ µ + T (π F ) -4c h ξ i,T -1, δ T ≥ µ + T (π F ) -4 2ασ 2 log + T δ -1/α 0 h ξ i,T -1 ≥ µ - T (π ) -4 2ασ 2 log + T δ -1/α 0 h ξ i,T -1 , with µ - T (π ) min i∈K µ i N i,T
-1 being the lowest mean reward for which a noisy value was ever obtained by the optimal policy. µ - T (π ) < µ + T (π F ) implies that the regret is 0. Indeed, in that case the next possible pull with the largest mean for π F is strictly larger than the mean of the last pull for π . Thus, there is no underpull at this round for π F and R T (π F ) = 0 according to Equation 3. Therefore, we can assume µ - T (π ) ≥ µ + T (π F ) for the regret bound. Next, we define ∆ i,h µ - T (π ) -µ h i N i,t + h as the difference between the lowest mean value of the arm pulled by π and the average of the h first overpulls of arm i. Thus, we have the following bound for h ξ i,T ,

h ξ i,T ≤ 1 + 32ασ 2 log + T δ -1/α 0 ∆ i,h ξ i,T -1 •
Next, h ξ i,T has to be smaller than the maximum such h, for which the inequality just above is satisfied if we replace h ξ i,T by h. Therefore,

wh ξ i,T ≤ h + i,T max    h ≤ T h ≤ 1 + 32ασ 2 log + T δ -1/α 0 ∆ 2 i,h-1    • (24)
Since the square-root function is increasing, we can upper-bound Equation 18 by replacing h ξ i,T by its upper bound h + i,T to get

A πF ≤ i∈op ξ 4 2ασ 2 log + (T δ -1/α 0 ) 1 + h + i,T -1 + L ≤ i∈op ξ   32ασ 2 log + (T δ -1/α 0 )   1 + 32ασ 2 log + (T δ -1/α 0 ) ∆ i,h + i,T -1   + L   .
The quantity op ξ is depends on the execution. Notice that there are at most K -1 arms in op ξ and that op ⊂ K. Therefore, we have

A πF ≤ i∈K   32ασ 2 log + T δ -1/α 0 ∆ i,h + i,T -1 + 32ασ 2 log + T δ -1/α 0   + (K -1)L.
Theorem 2. For α = 5 and C α 32ασ 2 , the regret of FEWA is bounded as

E[R T (π F )] ≤ i∈K C 5 log(T ) ∆ i,h + i,T -1 + C 5 log(T ) + 2L .
Proof. Using Lemmas 2, 4, and 5 we get

E[R T (π F )] = E[A πF ] + E[B] ≤ i∈K 32ασ 2 log(T ) ∆ i,h + i,T -1 + 32ασ 2 log(T ) + (K -1)L + Lπ 2 6 ≤ i∈K 32ασ 2 log(T ) ∆ i,h + i,T -1 + 32ασ 2 log(T ) + L • Corollary 3. FEWA run with α > 3 and δ 0 2δ/ζ(α -2) achieves with probability 1 -δ, R T (π F ) ≤ i∈K   32ασ 2 log + T ζ(α-2) 1/α (2δ) 1/α ∆ i,h + i,T -1 + 32ασ 2 log + T ζ(α -2) 1/α (2δ) 1/α   + (K -1)L.
Proof. We consider the event ∪ t≤T ξ t which happens with probability

1 - t≤T Kt 2 δ t 2 ≤ 1 - t≤T Kt 2 δ t 2 ≤ 1 - ζ(α -2)δ 0 2 •
Therefore, by setting δ 0 2δ/ζ(α -2), we have that with probability 1 -δ, B = 0 since ξ t = 0 for all t. We use Lemma 5 to get the claim of the corollary.

E Efficient algorithm EFF-FEWA

In Algorithm 3, we present EFF-FEWA, an algorithm that stores at most 2K log 2 (t) statistics. More precisely, for j ≤ log 2 (N πEF i,t ), we let s p i,j and s c i,j be the current and pending j-th statistic for arm i. The red circles denotes the number of pulls of arm i at which a new estimate s c i,j is created corresponding to a window h = 2 j , while the green boxes indicate the number of pulls for which s c i,j is updated with the last 2 j samples. receive r i (N i,t+1 ) ← r i(t),t

15:

EFF Update(i(t), r i (N i,t+1 ), t + 1) 16: end for As N i,t increases, new statistics s c i,j for larger windows are created as illustrated in Figure 3. First, at any time t, s c i,j is the average of 2 j-1 consecutive reward samples for arm i within the last 2 j -1 sample. These statistics are used in the filtering process as they are representative of exactly 2 j-1 recent samples. Second, s p i,j stores the pending samples that are not yet taken into account by s c i,j . Therefore, each time we pull arm i, we update all the pending averages. When the pending statistic is the average of the 2 j-1 last samples then we set s c i,j ← s p i,j and reinitialize s p i,j ← 0.

Algorithm 4 EFF Filter

Input: K 2 j , j, δ t , σ 1: c(2 j , δ t ) ← 2σ 2 /2 j log δ if n i,j = 2 j then 12:

s c i,j ← s p i,j /2 j 13:

n i,j ← 0 14:

s p i,j ← 0 15:
end if 16: end for In analyzing the performance of EFF-FEWA, we have to account for two different effects: (1) the loss in resolution due to windows of size that increases exponentially instead of a fixed increment of 1, and (2) the delay in updating the statistics s c i,j , which do not include the most recent samples. We let µ h ,h i be the average of the samples between the h -th to last one and the h -th to last one (included) with h > h . FEWA was controlling µ 1,h i for each arm, EFF-FEWA controls µ h i ,h i +2 j-1 i with different h i ≤ 2 j-1 -1 for each arm depending on when s c i,j was refreshed last time. However, since the means of arms are non-increasing, we can consider the worst case when the arm with the highest mean available at that round is estimated with its last samples (the smaller ones) and the bad arms are estimated on their oldest possibles samples (the larger ones). Lemma 6. On favorable event ξ t , if an arm i passes through a filter of window h at round t, the average of its h last pulls cannot deviate significantly from the best available arm i t at that round, µ 2 j-1 ,2 j -1 i ≥ µ + t (π F ) -4c(h, δ t ).

We proceed with modifying Corollary 1 to have the following efficient version. Corollary 4. Let i ∈ op be an arm overpulled by EFF-FEWA at round t and h πEF i,t

N πEF i,t -N i,t ≥ 1 be the difference in the number of pulls w.r.t. the optimal policy π at round t. On favorable event ξ t , we have that

µ + t (π EF ) -µ h π EF i,t (N i,t ) ≤ 4 √ 2 √ 2 -1 c(h πEF i,t , δ t ).
Proof. If i was pulled at round t, then by the condition at Line 10 of Algorithm 3, it means that i passes through all the filters until at least window 2 f such that 2 f ≤ h πEF i,t < 2 f +1 . Note that for h πEF i,t = 1, then EFF-FEWA has the same guarantee as FEWA since the first filter is always up to date. Then for h πEF i,t ≥ 2,

µ 1,h π EF i,t i (N i,t ) ≥ µ 1,2 f -1 i (N i,t ) = f j=1 2 j-1 µ 2 j-1 ,2 j -1 i 2 f -1 (25) ≥ µ + t (π EF ) - 4 f j=1 2 j-1 c(2 j-1 , δ) 2 f -1 = µ + t (π EF ) -4c(1, δ t ) f j=1 √ 2 j-1 2 f -1 (26) = µ + t (π EF ) -4c(1, δ t ) √ 2 f -1 (2 f -1)( √ 2 -1) ≥ µ + t (π EF ) -4c(1, δ t ) 1 √ 2 f ( √ 2 -1) (27) = µ + t (π EF ) - 4 √ 2 √ 2 -1 c 2 f +1 , δ t ≥ µ + t (π EF ) - 4 √ 2 √ 2 -1 c h πEF i,t , δ t , (28)

Figure 1 :Figure 2 :

 12 Figure 1: Comparison between FEWA and wSWA in the 2-arm setting. Left: Regret at T = 10000 for different values of L. Middle-right: Regret over time for L = 0.23 (worst case for FEWA) and L = 4.24 (case of L σ). We highlight the [10%, 90%] confidence region.

Figure 3 :

 3 Figure3: Illustration of the functioning of EFF-FEWA. The red circles denotes the number of pulls of arm i at which a new estimate s c i,j is created corresponding to a window h = 2 j , while the green boxes indicate the number of pulls for which s c i,j is updated with the last 2 j samples.

  h then

	11:	i(t) ← arg min i∈K h N i,t
	12:	end if
	13:	end while
	14:	receive r i (N i,t+1 ) ← r i(t),t
	15:

Table 1 :

 1 Average running time for the 10-arms experiment in seconds.

	FEWA EFF-FEWA wSWA SW-UCB D-UCB
	2271	7	1	5	2

  Algorithm 3 EFF-FEWA Input: K, δ 0 , α 1: pull each arm once, collect reward, and initializeN i,K ← 1 2: for t ← K + 1, K + 2, . . . do K 2 j+1 ← EFF Filter(K 2 j , j, δ t ) if ∃i ∈ K 2 j such that N i,t ≤ 2 j then

	3:	δ t ← δ 0 /(t α )
	4:	j ← 0 {initialize bandwidth}
	5:	K 1 ← K {initialize with all the arms}
	6:
	9:	j ← j + 1
	10:
	11:	i(t) ← i
	12:	end if
	13:	end while
	14:

i(t) ← none

7:

while i(t) is none do 8:

  Algorithm 5 EFF UpdateInput: i, r, t 1: N i(t),t ← N i(t),t-1 + 1 2: R total i ← R total i+ r {keep track of total reward } 3: if ∃j such that N i,t = 2 j then for j ← 0 . . . log 2 (N i,t ) do

	4: 5:	s c i,j ← R total i s p i,j ← 0	/N i,t {initialize new statistics}
	6:	n i,j ← 0
	7: end if
	8: 9: 10:	n i,j ← n i + 1 s p i,j ← s p i,j + r
	11:
			-1
			t
	2: s c max,j ← max i∈K h s c i,j
	3: for i ∈ K h do
	6:	add i to K 2 j+1
	7:	end if
	8: end for
	Output: K 2 j+1

4:

∆ i ← s c max,j -s c i,j 5: if ∆ i ≤ 2c(2 j , δ t ) then

This analysis is worst-case. In many cases, the number of samples for the suboptimal arms may be much smaller than O(t). For instance, in stochastic bandits it is as little as O(log t), thus reducing the complexity to O(KT log T ).

As pointed by a reviewer, a similar yet different approach has appeared independently in the context of streaming mining[START_REF] Bifet | Learning from timechanging data with adaptive windowing[END_REF].

See Corollary 2 and 3 for the high-probability result.

To make the results comparable to the one ofAuer et al. (2002a), we need to replace 2σ 2 by 1 /2 for sub-Gaussian noise.

Acknowledgements We thank Nir Levine for his helpful remarks and Lilian Besson for his bandits package (Besson, 2018). The research presented was supported by European CHIST-ERA project DELTA, French Ministry of Higher Education and Research, Nord-Pas-de-Calais Regional Council, Inria and Ottovon-Guericke-Universität Magdeburg associated-team north-European project Allocate, and French National Research Agency projects ExTra-Learn (n.ANR-14-CE24-0010-01) and BoB (n.ANR-16-CE23-0003). The work of A. Carpentier is also partially supported by the Deutsche Forschungsgemeinschaft (DFG) Emmy Noether grant MuSyAD (CA 1488/1-1), by the DFG -314838170, GRK 2297 MathCoRe, by the DFG GRK 2433 DAEDALUS, by the DFG CRC 1294 Data Assimilation, Project A03, and by the UFA-DFH through the French-German Doktorandenkolleg CDFA 01-18. This research has also benefited from the support of the FMJH Program PGMO and from the support to this program from Criteo. Part of the computational experiments were conducted using the Grid'5000 experimental testbed (https://www.grid5000.fr).

where Equation 25 uses that the average of older means is larger than average of the more recent ones and then decomposes 2 f -1 means onto a geometric grid. Then, Equation 26 uses Lemma 6 and makes the dependence of c(2 j-1 , δ) on j explicit. Next, Equations 27 and 28 use standard algebra to get a lower bound and that c(h, δ) decreases with h.

Using the result above, we follow the same proof as the one for FEWA and derive minimax and problem-dependent upper bounds for EFF-FEWA using Corollary 4 instead of Corollary 1.

Corollary 5 (minimax guarantee for EFF-FEWA). For any rotting bandit scenario with means {µ i (n)} i,n satisfying Assumption 1 with bounded decay L and any time horizon T , EFF-FEWA with δ t = 1/(t 5 ), α = 5, and δ 0 = 1 has its expected regret upper-bounded as

Corollary 6 (problem-dependent guarantee for EFF-FEWA). For δ t = 1/(t 5 ), the regret of EFF-FEWA is upper-bounded as

with C α 32ασ 2 and h + i,T defined in Equation 10. In Figure 4 we compare the performance of FEWA against UCB [START_REF] Tze | Asymptotically efficient adaptive allocation rules[END_REF] on two-arm bandits with different gaps. These experiments confirm the theoretical findings of Theorem 1 and Corollary 2: FEWA has comparable performance with UCB. In particular, both algorithms have a logarithmic asymptotic behavior and for α = 0.06, the ratio between the regret of two algorithms is empirically lower than 2. Notice, the theoretical factor between the two upper bounds is 80 (for α = 5). This shows the ability of FEWA to be competitive for stochastic bandits.