Agathe Pirog
email: agathe.pirog@ofb.gouv.fr

Adrien Tableau
email: adrien.tableau@ofb.gouv.fr

Jean-Marc M Gillier

Jean-François F Maillard
email: francois.maillard@ofb.gouv.fr

Sébastien Reeber
email: reeber@snpn.fr

Justin Potier
email: justin.potier@ofb.gouv.fr

M Gillier

LIFE Oxyura Technique Assessment Report 2019 Techniques implemented and results

Preface

This national assessment is carried out in the framework of:

-Recommendation n°185 of the Berne Convention Permanent Committee, adopted 18 November 2016, on the eradication of the ruddy duck (Oxyura jamaicensis) in the Western Palearctic by 2020 (Council of Europe 2016).

-EU regulation n°1143/2014 concerning the prevention and management of alien invasive species (AIS) introduction and propagation (Council of Europe, 2014).

-The 2015-2025 national plan to eradicate the ruddy duck, validated by the Water and Biodiversity Directorate on 24 June 2016 (Ministère de l'Ecologie du Développement Durable et de l'Energie, 2016).

-The implementation of the EU AIS regulation listing the ruddy duck as a species of concern for the European Union, transcribed into national law by Decree n°2017-595 of 21 Avril 2017 regarding the control and management of the introduction and propagation of certain animal and plant species, and the Decree of 14 February 2018 regarding the prevention of the introduction and propagation of alien invasive animal species in Metropolitan France.

-The national strategy on AIS, Axis II "Operations for the management of gestion species and restoration of ecosystems", Objective 5 "Control widespread alien invasive species", Action 5.3 "Implement national combat plans".

-The EU project LIFE Oxyura, led by the OFB in partnership with the SNPN, which lasts from October 2018 to December 2023. This financial instrument strengthens past actions in order to achieve two objectives: eradication of the population in the wild by 2025 and control and eradication of the captive population by 2030 (European Commission, 2018). This assessment report covers the period from the 1th of January to 31 December 2019. The information it contains is derived from the results of censuses, field surveys and destruction operations carried out by the OFB and the SNPN -Grand-Lieu RNN. Concerning observations outside of Grand-Lieu Lake, this assessment report is also based, to a large extent, on information gathered from the naturalist community, in particular through regular consultation of the ornithological data of the Visionature network.

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier 6 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier 1 Background

A conservation issue for the white-headed duck

The LIFE Oxyura project responds to a conservation issue of a diving duck species, the white-headed duck -Oxyura leucocephala (European Commission, 2018). This species range is subdivided in two geographically isolated populations. One of them, located in the western Mediterranean, is threatened [START_REF] Green | Action plan for the white-headed duck Oxyura leucocephala[END_REF][START_REF] Hughes | International single species action plan for the conservation of the white-headed duck Oxyura leucocephala[END_REF]. Currently only present in Spain and to a lesser extent in Morocco and Tunisia, it used to nest in Italy and France, the last sedentary pair being observed in Corsica in 1966. In 1977, the population was close to extinction, with 22 individuals recorded in southern Spain. Conservation measures have enabled it to recover, with a fairly stable population of around 2,500 individuals. This increase is associated with occasional sightings of isolated birds in France since the 1980s. (Figure 1). The Iberian white-headed duck population faces a new threat since the introduction of a closely-related species, the ruddy duck -Oxyura jamaicensis [START_REF] Gutiérrez-Expósito | Ruddy duck (Oxyura jamaicensis Gmelin, 1789)[END_REF]. This species, native to North America, is a potential competitor for nesting sites and food resources. Although they probably diverged genetically 1-2 million years ago (Muñoz-Fuentes et al., 2007), these two species nevertheless produce fertile hybrids, of which 69 specimens have been observed in Spain since 1991 [START_REF] Gutiérrez-Expósito | Ruddy duck (Oxyura jamaicensis Gmelin, 1789)[END_REF]. This genetic introgression would likely lead to the extinction of the Iberian white-headed duck population in favour of a hybrid population or a new nucleus of ruddy duck. This transition could be rapid due to the aggressive behaviour of male ruddy ducks during the breeding season, favouring their access to female white-headed ducks (Figure 2) [START_REF] Gutiérrez-Expósito | Ruddy duck (Oxyura jamaicensis Gmelin, 1789)[END_REF].

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier In view of conserving the white-headed duck, the Spanish authorities carried out a systematic campaign to destroy ruddy ducks and hybrids in their territory until 2007. Since then, a few occasional sightings have been made1 . Initially introduced in the UK, the threat of the ruddy duck returning to Spain cannot be ruled out as long as the species is still present on the European continent. Under pressure from the white-headed duck conservation issue, the British authorities in association with local ornithological organisations implemented an eradication plan in 2005 2 , which reduced the population from 6000 individuals to 12 individuals in 2019 and 2020. Breeding in the wild was observed every year between 1953 and 2018, with 2019 marking the end of the population renewal (Henderson, pers. comm.). However, two small nuclei have been established on the continent, one in France remaining at around 200 individuals between 2004 and 2018 thanks to the efforts of the French Biodiversity Agency (OFB) and the French National Nature Protection Society (SNPN), and one in the Netherlands of around 80 individuals 3 . In order to prevent natural breeding in France, the LIFE Oxyura project, led by the OFB, was set up in 2018 to support the 2015-2025 national management plan. In particular, it enabled the hiring of four officers specifically to control the ruddy duck throughout France for five years (European Commission, 2018).

The ruddy duck in brief

The ruddy duck is a small duck with a long tail cocked at 45°, which is characteristic of the Oxyura genus (Figure 3). The male's plumage is predominantly red with a black head and white cheeks [START_REF] Johnsgard | Ruddy ducks & other stifftails: Their behavior and biology (animal natural history series[END_REF]. The bill is blue in the breeding season. The female has brown plumage with dark brown on the upper parts of the head and darker-striped light brown on the lower parts. The undertail feathers are white. Regardless of sex, the C-shaped bill of the ruddy duck is the main feature distinguishing it from the white-headed duck, whose bill is S-shaped. Juveniles have a similar profile to the female and can be mistaken for young common pochard. In its native range, the ruddy duck winters on large, shallow bodies of water (coastal bays or lowland lakes). Its preferred nesting sites are smaller bodies of water with extensive riparian vegetation consisting of sedges, reeds and willow saplings. The ruddy duck feeds on aquatic plants, molluscs, worms and insects [START_REF] Sanchez | The diets of the white-headed duck Oxyura leucocephala, ruddy duck O. Jamaicensis and their hybrids from spain[END_REF][START_REF] Woodin | Foods and dietary strategies of prairie-nesting ruddy ducks and redheads[END_REF]. Midge larvae and nymphs (Tendipedidae), specifically Chironomus, are the primary food sources during the breeding season [START_REF] Siegfried | Summer food and feeding of the ruddy duck in Manitoba[END_REF].

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier A ruddy duck population has been established in France since 1974. At the onset of the first cold weather, most of them gather as from late November on Grand-Lieu Lake, in the Loire-Atlantique department, to spend the winter. They form part of a group of about 20,000 wintering ducks of various species in the central area of the lake (Figure 4). This period is favourable for estimating the size of the French population as almost all the birds are visible at this site. The ducks move from the east to the west of the central area of the lake, depending on the wind and the disturbance caused by any predators present or the few professional fishermen (Figure 5). Until mid-February, the birds stay away from the edges of the lake and show fleeing behaviour as soon as they are approached within 500m.

10 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier From mid-February onwards, in good weather, the ruddy ducks show the first signs of breeding behaviour. They then move closer to the banks, particularly in the coves beside the riverine woodland to the north of the wintering area and around the islands to the west (Figure 5). The males acquire their breeding plumage in late winter without forming pairs. Only part of the population remains on Grand-Lieu to breed, the rest gradually moving to smaller ponds, mainly in the north-western quarter of France. The maximum dispersal of the birds is unknown5 . It is therefore not ruled out that there are also connections with the other European populations (notably the Netherlands one).

The first nuptial displays are observed in late winter, when the weather is mild. The male calls throughout the breeding season, which lasts until the end of summer, both day and at night (S. Reeber & A. Laroche, pers. comm.). This call is very characteristic and thus enables the species to be identified. Ruddy ducks make their nests in the vegetation, for example in sedge clumps. They often lay eggs in the nests of other duck species. The first chicks are seen in May at the earliest, but the breeding peak is usually around early July.

The location of the ducks during moulting after breeding has not been clearly identified. On the one hand, the sudden gathering of the population at the wintering site suggests the existence of a potential site where the ducks gather to moult (see analysis in Appendix A). On the other hand, a few individuals located during the same period at very distant sites suggest that the birds moult, camouflaged, close to their nesting site in the vegetation. The moulting behaviour of the native North American population is also poorly identified [START_REF] Baldassarre | Oxyura against Oxyura -eradicate the ruddy duck to save the endangered white-headed duck from extinction -LIFE17-NAT_FR_000542[END_REF]. Males appear to moult as early as August, after breeding. Some have been seen in large, open areas. The main hypothesis for females is that individuals moult on the nesting sites camouflaged in the vegetation.

The winter strategy

The main difficulty in winter on the Grand-Lieu lake is getting close enough to the birds to have them within shooting range. For this purpose, several complementary tools are used. Investigations have shown that chasing them in boats combined with the use of guns is not very effective and above all counterproductive in terms of disturbing other species that find refuge on the lake in winter. The LIFE programme has enabled the construction of 3 hides positioned on the banks as close as possible to the area occupied in winter (Figure 5 and Figure 6). In windless conditions, these shooting stands are ideal for firing high-precision rifles capable of hitting targets at long distances (300m maximum). The area occupied by the birds during winter shooting campaigns is often far from the stands, so the birds must be moved. An unobtrusive boat can be used to gently bring the birds closer to the shooting stands (Figure 7). From mid-February onwards, playbacks 6 combined with plastic decoys positioned near the shooting stands are used to attract the ducks. An alternative to shooting is capturing ducks with a cage trap. A bird can be lured either with food or by using fellow ruddy ducks. The first option is not selective and may disturb other species. It is also not very effective on ruddy ducks due to their diet, so the use of live decoys is favoured. In partnership with Branféré Zoo7 , the Grand-Lieu SNPN and OFB are developing this approach (Figure 8). The English experience shows that cage traps are time-consuming to use. In order to minimize handling time, the cage developed within the LIFE framework is fitted with an automatic trap door that opens at regular intervals. Photographic traps transmit images by telephone before the trap doors are opened. If a ruddy duck is spotted, the team arrives before it opens to capture the bird.

The summer strategy

Some of the birds nest on Grand-Lieu Lake from spring to the end of summer. The suitable areas vary according to water levels and are not very accessible. The most suitable approach is by boat from the centre of the lake. The males are often found near the nests on the edge of the vegetation and are an indicator of the presence of a female and her nest. Another method is to detect birds calling at night using a sound amplifier. Once the birds have been located, they are shot using a high-precision rifle from a boat stabilised by water lilies. Nests are also occasionally located during surveys in the riverine woodland. In this case, trap cages positioned on the nests are effective for capturing females (Figure 9). As in winter, an alternative at Grand-Lieu Lake is to attract the birds. The sexual activity of the ducks can be exploited to attract them to the hides which are still situated in the water in spring. The combination of playbacks and ruddy duck-shaped decoys is the most attractive solution. Attempts to shoot with rifles fitted with silencers and night vision equipment are possible under favourable weather conditions (little wind and clear nights). In addition, the use of a calling cage is a solution considered throughout the summer season given the continuous sexual activity during this period.

At the end of winter, some birds leave Grand-Lieu Lake to nest in smaller ponds with suitable riparian vegetation. For this part of the population, the main challenge is locating them. The main detection method is regular surveying of historical sites of ruddy duck presence and potential nesting sites (Figure 10). Bird watchers and professional ornithologists observations are also used and are particularly useful for detecting birds outside the main range of ruddy duck in France. Once located, officers secure the area and adapt their approach to the birds and the shooting solution according to the local configuration. 14 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

Management of the captive population

The last study focusing on the captive ruddy ducks in France dates from 2014 with a total of 203 birds counted in 26 departments. Among these individuals, 103 were held by recreational breeders in 21 departments, 80 were detained by 10 zoos in nine departments and 20 were detained by six professional breeders in five departments.

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier The current legislation, which is a translation of the European regulation of the 14 February 2018 on the prevention and management of the introduction and spread of invasive alien species on the metropolitan territory, forbid the detention, transportation, sale or acquisition of the ruddy duck. Except for some zoos authorized to display captive ruddy ducks, breeders who previously detained individuals at this date were authorized to keep their birds until their death, if they declare their birds to the prefecture of their residency place before the 31 December 2019 and if they don't do any breeding (at least eggs destroyed).

In order to monitor the captive population evolution, a survey on the breeders identified in 2014 will be conducted in 2022. It will allow to evaluate at which extent is the legislation respected, and thus conclude on the renewal of the captive population and its threat for the wild population.

On the other hand, the new legislation may lead recreational breeders of diving ducks to turn to another species of the Oxyura genus. The lake duck, also called Argentine ruddy duck (Oxyura vittata), native from South America, may also hybridize with other Oxyura species, and notably the white-headed duck. To date, it is difficult to estimate the number of captive lake ducks in France, but if some of them escape into the wild, they are likely to provoke, as the ruddy duck before, the same issues of genetic introgression in the white-headed duck. It is thus paramount to prevent this species introduction into the wild, notably by promoting new legislation or preventive actions.

16

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier At the beginning of 2019, the number of ruddy ducks reached 91 individuals on Lake Grand Lieu8 . The trend is a decrease of 46% compared to the previous winter. We have to go back to 2002 to find similar numbers (Figure 13).

Figure 13: Numbers of the French wintering population with respect to the numbers culled. Numbers before culling in mid-January 2019 in purple, and numbers of the resulting population in mid-January 2020 in pink.

After the 2018-2019 wintering period, ruddy ducks were observed at 37 sites, 5 less than the previous year (Figure 14). This result is coherent with the wintering numbers trend. These sites are located in the same areas as in previous years (See details per department in Appendix B).

Detection and culling in 2019

Continuous work is carried out by the LIFE project field officers who survey potential bird habitats throughout the year. Online ornithology platforms9 are also routinely consulted to detect part of the birds. This partnership is essential because the birdwatching community covers the whole of France and allows for the rapid recognition of new areas. Finally, the officers from the OFB's departmental units assist this detection work by transmitting information during their field missions.

Surveys by LIFE officers were divided as follows: 2/3 monitoring of historical sites of ruddy duck presence, 1/3 exploration of sites with high presence potential. This year, LIFE officers made 529 visits to 241 sites (Figure 15). From the observation data derived from the various sources available, the number of birds actually detected without the breeding period is looked for 10 . To make this estimation, we assume that two successive observations of a ruddy duck 11 on the same site correspond to two different individuals if over one month has elapsed between observations. Naturally, major sources of error 12 result in an uncertain estimation. However, this approach remains the best indicator of bird detectability outside the wintering period.

Using the method described above, we estimate that 83 different individuals were detected. Among these, 51 adults of which 26 males and 25 females. The remaining 32 individuals were juveniles or individuals of undetermined age. The wintering population consisted of 91 individuals. The detection rate for adults that wintered in France is thus 56% 13 .

In parallel with detection, the culls performed by SNPN and OFB officers eliminated 92 individuals 14 , of which 48 were adults: 26 males and 22 females. The remaining 44 individuals were juveniles 15 &16 . The culling rate for adults that wintered in France is 53%. The culling rate for detected adults is 94% 17 .

10 A ruddy duck may be observed repeatedly at a site by one or more bird watchers or professional ornithologists 11 Observations are considered to be potentially of one single individual only if the age and sex characteristics are identical 12 There are two sources of error in this estimation of the number of birds detected: i) the time interval between two observations for which individuals are considered different, and ii) individuals that have travelled between several sites and are therefore counted as many times as they are detected 13 A detection rate over 100% is not necessarily an error as it is likely that birds from a population not wintering in France are detected.

14 During the autumn-winter period, juvenile birds are of adult size but with female plumage. Birds culled during this period are systematically sexed and aged by dissection 15 Very few individuals are not retrieved and could be adults 16 A higher number of juveniles culled than detected is possible due to the uncertainty in the estimation of the number detected 17 We cannot estimate the effectiveness for juveniles because we do not have the number of young produced during the year 20 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier An interesting indicator is the proportion of adults culled before they have reproduced. Indeed, by definition, they have not had time to produce young and therefore contribute to population renewal, which has a significant impact on the decline in numbers the following winter. This rate of culling before reproduction amounts to 27%.

In order to make comparisons from year to year, the detailed culling time series is presented in Figure 13. Historical detection and culling efficiencies are presented in Figure 16. The decrease in population over the recent period (2016-2019) supports the effectiveness of the control at the current intensity level. This first indicator needs nevertheless to be put into perspective. Indeed, an other continuous population decrease over four consecutive years (2007-2010) was then followed by significant recruitment in 2011, allowing the population to recover its initial size. In contrast to this period, the current decline is much more pronounced, with a decrease by a factor of two over the last two seasons (summers 2018 and 2019). If we project this trend into the future, the French population will consist in fewer than 10 individuals by the end of the LIFE project. Considering that the current English population of 12 individuals is no longer reproductive, this population size would mean the success of the LIFE project, and therefore a clear step forward in the protection of the white-headed duck in Europe.

Naturally, this projection is highly uncertain for several reasons. Climatic factors are the primary source of uncertainty. A harsh winter could lead to the migration of part of the Netherlands wintering population and revive the French population. This underlines the importance of coordination not only at national but also at European level. In the past, favourable conditions in spring and summer for breeding and the survival of young have allowed the French population to renew itself. Human factors could also influence the future of the duck population. A decline in the participation of the various field officers collaborating with the LIFE officers could be observed due to the reduction in the frequency of operations carried out inherent to low numbers. A decrease in the involvement of the stakeholder network related to the fact that the objectives have virtually been achieved would also be detrimental. These aspects show the importance of communicating effectively on the contribution of each person and on the long-term objectives of such a project.

The contribution of the LIFE project already clear

LIFE resources were used for the first time over an entire season in 2019 18 . Despite surveying work, which were multiplied by the presence of specific officers in the field, the overall detection rate seems to have decreased. One likely reason for this is that the decrease in numbers results in lower bird densities, and therefore less detectable groups. The learning ability of the birds and the selection pressure in favour of warier and therefore more furtive individuals is also expected when population culling is high. This last point suggests that the opportunities to capture or cull the last remaining birds diminish enormously and it is therefore crucial to maximise capture or culling efficiency when numbers are low. This is in fact what the officers managed to do as nearly all the birds detected were culled in 2019, a record efficiency only achieved once in the past (2015).

As in the previous year, almost half of the wintering population was culled during the year despite the reduction in numbers and the resulting difficulty for locating them. Culling was carried out earlier than in previous years, which undoubtedly reduced breeding success and thus contributed to the drop in wintering numbers in January 2020.

Alternative methods in use

Despite increased surveying pressure, detection rates are levelling off or even decreasing. The responsiveness and efficiency of the field officers mean that the culling rate for detected birds is nearly 100%. The potential for further improvement in bird detection and sampling efficiency is therefore low.

18 The LIFE project's actions in the field only started in December 2018

22

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier Even a small improvement in pre-breeding culling rates could have significant impacts on the population's ability to renew itself. This is why a major effort is being made to test alternative capture methods, particularly in winter (Section 2.1.2). The use of hides has been the subject of initial tests, enabling shooting campaigns. A cage trap will be used from autumn 2020.

Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

A Evidence of moulting in coastal areas?

The moulting season of ducks is a vulnerable period that is propitious for capturing them. If, as with other species, ruddy ducks gather together in this period of the year, it would therefore be efficient to identify the moulting sites. This aspect of ruddy duck ecology is poorly known within its natural range. The low number of sightings during this period shows that the birds are more wary at this time.

Throughout the year, ruddy ducks in France are mainly observed in areas less than 100 km from the coast (Figure 17). This could reflect the potential species' dependence on coastal habitats, notably during the moulting season. Analysis of the positions of the sites where the ruddy ducks are observed shows that the birds sighted during the moulting season19 are more numerous in coastal areas than during the rest of the year. This could be evidence that some of the population moults in coastal areas. Unfortunately, this result is not significant enough to conclude that the moult occurs in coastal areas, and therefore to consider carrying out specific surveys to identify a potential moulting site. A plausible hypothesis concerning the presence of ruddy ducks in this area in autumn is the utilization of the wastewater treatment lagoons of seaside resorts as a food source and refuge area during the hunting season.

Figure 1 :

 1 Figure 1: Evolution of the white-headed duck presence in France

Figure 2 :

 2 Figure 2: Two males in breeding plumage fighting in Mayenne -France; a ruddy duck to the left, a white-headed duck to the right. © Eric Médard

•

  Figure 3: A pair of ruddy ducks in summer. © OFB

  into the biological life cycle of the ruddy duck in France 4

Figure 4 :

 4 Figure 4: Geography of Grand-Lieu, a lowland lake in western France (47°05' 45" N, 1°40' 3" W).The lake has two tributaries, the Boulogne and the Ognon, and flows into the Acheneau which in turn flows into the Loire. The light green areas are water lily beds which disappear in winter. © Geoportail

Figure 5 :

 5 Figure 5: Area where ruddy ducks spend the winter (blue) -Location of hides (orange) -Location of the calling cage (red). © Geoportail

Figure 6 :

 6 Figure 6: One of the three hides built on Grand-Lieu Lake with funds from the LIFE Oxyura project. © OFB 6 An audio system to transmit the sound of the males' nuptial calls

Figure 7 :

 7 Figure 7: Unobtrusive boat with electric motor for gently bringing birds towards the shooting location. © SNPN

Figure 8 :

 8 Figure 8: Calling cage under testing at Branféré Zoo. © Diagram: Jean-Marc Gillier, © Photo: Branféré Zoo

Figure 9 :

 9 Figure 9: Cage trap system on nests, here an example with a common pochard. © Alain Caizergues

Figure 10 :

 10 Figure 10: Boat survey during the breeding season. © Valentin Boniface

Figure 11 :

 11 Figure 11: Ruddy duck numbers detained in captivity per sector

Figure 12 :

 12 Figure 12: Argentine ruddy duck, Oxyura vittata, male

Figure 14 :

 14 Figure 14: Trends in spatial range of ruddy ducks -See the interactive map for the historical details of the sites.

Figure 15 :

 15 Figure 15: Cumulative number of surveys in 2019.

Figure 16 :

 16 Figure 16: Proportion of wintering numbers detected and culled. Please note that these figures are only estimates and therefore uncertain.

Figure 17 :

 17 Figure 17: Distribution of ruddy ducks with regard to the coast during and outside the moulting season.

The last three sightings date from

2011, 2014 and 2020, respectively 2 The plan was set up after a 5-years research phase 3 Some birds have been observed in Belgium, but are not corresponding to an entire population 8 Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

The information in Section 2.1.1 is based on observations made by staff from the Grand Lieu SNPN and French Biodiversity Agency, as well as by bird watchers and professional ornithologists.

In the British island context, maximum dispersal appears to be about 150km, but continental European birds are known to have originated from the founder population in England Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

Zoo located in Morbihan and member of the French Association of Zoological Parks (AFDPZ)Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

A few isolated individuals can be observed elsewhere in France, but they do not necessarily belong to the French population and are therefore not taken into account here

http://www.faune-france.org -http://www.observation.org -BDBiodiv Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

This period is defined in the analysis as between the 1th of August and the 1th of December, from the evidence provided by[START_REF] Baldassarre | Oxyura against Oxyura -eradicate the ruddy duck to save the endangered white-headed duck from extinction -LIFE17-NAT_FR_000542[END_REF] Technical Report -LIFE17 NAT/FR/000542 A. Pirog, A. Tableau, J.M. Gillier, J.F. Maillard, S. Reeber, and J. Potier

‡ National Society for the Protection of Nature (SNPN) -Director of the Grand-Lieu national natural Reserve -jean-marc.gillier@ snpn.fr § French Biodiversity Agency (OFB) -Project manager on Alien Invasive ¶ National Society for the Protection of Nature (SNPN) -Grand-Lieu national natural Reserve project manager-sebastien.

B Data by department

For the detail of the historical observation sites, see the interactive map online.


Yunmo Chen

Seth Ebner

Tongfei Chen

Patrick Xia

Elias Stengel-Eskin

Tzu-Ray Su

J Edward Hu

Nils Holzenberger

Ryan Culkin

Craig Harman

Max Thomas

Thomas Lippincott

Aaron Steven White

Kyle Rawlins

Benjamin Van Durme

NIST TAC SM-KBP 2019 System Description: JHU/UR Framework

). Some of our key contributions include: a new dataset supporting event argument linking across multiple sentences; a novel algorithm for predicting such linkages, even when arguments are not co-referent with mentions in the same sentence as the event trigger; and a novel algorithm for supporting hierarchical typing of events, relations and entities.

Introduction

The Streaming Multimedia Knowledge Base Population (SM-KBP) task aims at constructing knowledge graph that supports structured queries against unstructured multi-media sources. In the 2019 evaluation there were three main subtasks: information extraction and population of knowledge elements (KE) (TA1), knowledge graph construction (TA2), hypotheses generation from KB (TA3). Here we are focussed on the TA1 task.

In the evaluation, participants were required to process a set of independent documents under an existing ontology, first without and then with some notion of downstream information needs (referred to as a hypothesis). For a each document we sequentially performed a series of tasks: Entity Mention Detection (EMD), Coreference Resolution, Event Trigger Detection, Entity / Event / Relation Typing, and Argument Linking. In addition, as documents might be in Russian or Ukrainian, we employed techniques such as Language Identification, Machine Translation, and Word Alignment, which enables us to pivot on the models trained on English corpora and project the discovered text span offsets back to the original documents. For hypothesisbased population in TA1b, we followed a strategy similar to our AIDA M9 approach, incorporating a textual similarity matching model for Entity Linking. The final processed results were converted to AIDA Interchange Format (AIF). Our system is depicted in Figure 1.

We experimented with different encoders, including BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] and ELMo [START_REF] Peters | Deep contextualized word representations[END_REF], and decided to use BERT as the main source of extraction features.

To combat data sparsity in the released practice data from LDC ahead of the evaluation, we developed a new annotated resource based on the AIDA Annotation Ontology (as of Spring 2019).

Multi-Sentence Events and Relations Dataset

We constructed a crowd-sourced dataset with 9,100 AIDA event and relation annotations. Each data point consists of a typed trigger span and 0 or more typed argument spans in an English document.

A trigger span is a word or phrase that evokes a certain event or relation type in the context of a document (e.g. "pledge" may evoke the "Contact.CommitmentPromiseExpressIntent.Broadcast" event in certain contexts), while argument spans denote participant types in the event or relation (e.g. the "Communicator" or the "Recipient"). Both event and arguments spans are token-level (start, end) offsets into a tokenized text document.

Typically, event and relation datasets force argument spans to be in the same sentence as the trigger span, but we present annotators with a multi-sentence window: the sentence containing the trigger span, some number of sentences before, and some number sentences after. The annotator can then select argument spans anywhere inside of the context window. In practice, we showed annotators five sentences: the trigger sentence, two sentences before, and two after.

We used Reddit, a popular internet forum, to identify a suitable collection of texts that were likely to contain AIDA events and relation mentions. On Reddit, users post submissions containing links to news articles, images, videos, or other kinds of documents, and other users may then vote or comment on posted submissions. We considered news articles matching the following criteria:

1. Article was posted to the r/politics sub-forum

AIF

Figure 1: Data flow for populating knowledge graphs from a document. The processes within the dashed rectangles are optional and will be applied to certain inputs; Machine Translation and Alignment modules are activated only when the language is not English; Entity Linking module is only activated for TA1b evaluation.

2. Article contained at least one mention of the string "Russia"

3. Article had at least 25 comments 4. Article was posted between January and October 2016

By considering politically-themed articles containing the word "Russia", the texts tended to describe geopolitical events and relations like the ones in the AIDA ontology. In order to filter out low-quality, fake, or disreputable news articles, we only considered submissions that had generated at least 25 comments. After applying these criteria we identified approximately 12,000 news articles, each with an average of approximately 40 sentences.

Next, we manually constructed a mapping from each AIDA event and relation (sub-(sub-))type to a small list of lexical units (LUs) likely to evoke that type. This mapping was high-precision, low-recall, in the sense that for a given (Type, WordList) pair, the LUs in the WordList were all likely to evoke the Type, but the WordList could lack many LUs that also evoked the Type. Each AIDA type had 3.89 manually-curated LUs on average.

Using the AIDA-to-LU mapping we performed a soft match between every LU in the mapping and every word in our collection of texts in order to select candidate sentences with respect to each event and relation type. In the soft matching procedure, we lemmatized all words to their base form, removing any inflectional endings, and lower-cased the strings in order to get a high-recall set of candidate sentences. This matching procedure returned approximately 94,000 candidates, which we then balanced at the lexical unit level, i.e. we sampled the same number of candidate sentences for each lexical unit.

Candidate sentences were then manually vetted to ensure that they matched their associated proposed event or relation type. Each vetting task given to annotators contained an event or relation definition and several candidate sentences, each with a highlighted LU. Annotators were asked to judge how well each highlighted LU, in the context of its sentence, matched the provided event or relation definition. They were also asked to assess the factuality of the sentencewhether the event or relation actually happened. We collected judgments on approximately 17,500 candidate sentences.

Of the 17,500 candidate sentences, 52% were determined to match their provided definition and have positive factuality, giving us 9,100 sentences, each with a highlighted LU known to evoke a given event or relation type. Using these sentences we then collected multi-sentence argument annotations, presenting annotators with a 5-sentence window with two sentences of context before the sentence with the trigger, and two sentences after. On average, we have 66 full annotations (trigger and argument) per AIDA type.

Models and Experiments

Machine Translation

For our information extraction systems, it was necessary to translate Ukrainian (Uk) and Russian (Ru) into English (En). For the purposes of our alignment system (see Section 3.7), we also needed models that could translate Ukrainian and Russian to English. We built models for all 4 settings, and compared to Google translate's performance.

Ukrainian -English

Uk-En parallel corpora are quite scarce, in particular for the news domain, which was most relevant to the evaluation scenario. We relied on two datasets. The first one, OpenSubtitles 1 , is as far as we know the largest openly available Uk-En parallel corpus, with 878k sentences. The sentences were extracted from movie subtitles [START_REF] Lison | Opensubtitles2016: Extracting large parallel corpora from movie and TV subtitles[END_REF]. We used this corpus as an initial training set for our Uk→En and En→Uk models. The second corpus we used is the SUMMA corpus [START_REF] Germann | Scalable understanding of multilingual media[END_REF]. The data was scraped from the beginning of 2016 to November 2018, from two Ukrainian news sites, censor.net (75%) and UNIAN (25%), making the SUMMA corpus very relevant domain-wise. Out of the 88k sentences that the unprocessed SUMMA corpus contains, we randomly set aside 5% as a development set, which we report performance on. The remaining 95% were used to fine-tune models trained on OpenSubtitles.

We used the Sockeye toolkit [START_REF] Hieber | Sockeye: A toolkit for neural machine translation[END_REF]) to build RNN-based and Transformer-based sequenceto-sequence models, together with Byte-Pair Encoding (BPE) [START_REF] Sennrich | Neural machine translation of rare words with subword units[END_REF]. The best models for both Uk→En and En→Uk used 2 LSTM layers with 512 hidden units, 30k BPE operations, 128-dimensional embeddings for subword units, an initial learning rate of 0.001, a batch size of 64 and dot product attention.

The performance of our models on the SUMMA dev set are reported in the top of Table 1 under "NMT", together with the performance of Google translate. Given that we outperformed Google translate, we used our own models in the evaluation. The performance of our models on the WMT'17 test set are reported in the bottom of Table 1 under "NMT", together with the performance of Google translate. Based on Google translate's better performance for En→Ru, we used it for the evaluation.

Entity Mention Detection and Coreference Resolution

Entity Mention Detection and Coreference Resolution were performed sequentially, both with the goal of abstracting the entities present in the semantics of the document from the mentions, or referents present in the text. The underlying model is based on a coarseto-fine model [START_REF] Lee | Higher-order coreference resolution with coarse-tofine inference[END_REF], which takes a candidate set of spans from within the text and clusters them based on mentions. For each span, a span representation is formed by pooling over the word embeddings of each word within the span. A mention scorer subsequently prunes all spans, leaving only likely referents. A coarse, similarity-based scorer then determines for each possible referent its top 50 most likely candidates antecedents based. A fine, neural scorer then scores these 50. During decoding, chains of referents are formed, each chain representing one entity. The entire model is typically trained (and evaluated) endto-end on OntoNotes 5.0 [START_REF] Sameer Pradhan | Towards robust linguistic analysis using ontonotes[END_REF], and the average test F1 reported with ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] is 73.0. One drawback of the coarse-to-fine model is that it only returns chains of size two or more. In SM-KBP, we are also interested in entities that are mentioned once. We can additionally extract the high-scoring spans according to mention scorer that were not chained, as that should capture the remaining singleton chains. However, those predictions only have a 70.1 F1 with noun-like phrases (according to gold syntactic parses). We attempted multi-task training of the noun-like phrases (objective is to minimize the loss of the mention scorer) and of the coreference chains (with the coreference cluster objective) and arrived at a compromise at 88.3 F1 against noun-like phrases and 72.4 coreference F1. However, the superior option was to train two separate models (a mention scorer and a coarse-to-fine linker) and pipe the span predictions of the first into the second. This yields a 94.1 F1 against noun-like phrases and 72.8 coreference F1. Finally, we used static BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] embeddings as input (with mean-pooling across subtokens), which improved the coreference F1 to 73.8.

Event Trigger Detection

An event trigger is the word or phrase that expresses the occurrence of an event. In order to identify triggers, We employed a linear chain Conditional Random Field (CRF) as a sequence tagger for trigger identification. The potential function of the CRF is generated by passing BERT features through a feed-forward neural network.

The model is pre-trained on PredPatt output 3 fine-tuned on the LAO dataset, obtaining 83.9 F1.

Typing Entities, Events, and Relations

Given the BERT encodings of entities (the span), the event (the event trigger) and the relation (3 encodings, consisting of the relation trigger and the two argument spans), we classify them into a hierarchical ontology.

In the AIDA entity / event / relation hierarchy, there are 3 levels: types, subtypes, and sub-subtypes. Given the vector representation of an object (either the representation of an entity, an event, or a relation) x and its corresponding types y 1 , y 2 , y 3 (matching to the 3 levels in the ontology respectively), we employ a multi-level ranking loss:

3 l=1 y l ∈Y l [ξ l -F (x, y l ) + F (x, y l )] + (1)
where l is the level of the ontology ranging from 1 to 3; y l are negative samples on level l; F (x, y) is a scoring function between the object and the type; and ξ l is a margin hyperparameter for level l.

Intuitively, coarser types are easier and finer types are harder for classifiers -hence larger margin hyperparameter for coarser types should yield better performance. Our preliminary experiments indeed show that this is the case -a graded set of margins perform better than a uniform set of margins. In our experiments, (ξ 1 , ξ 2 , ξ 3 ) = (0.75, 0.5, 0.25).

3 https://github.com/hltcoe/PredPatt

The typing scorer function F (x, y) follows the matching function in [START_REF] Mou | Natural language inference by tree-based convolution and heuristic matching[END_REF]: we first concatenate x and a type embedding for type y, together with their elementwise product and elementwise distance, then pass through a multi-layer feed-forward neural network with ReLU activation functions in between.

We randomly sample 80% samples for training and 20% as development in the AIDA practice annotation dataset. For the event typer, on English, we get 0.741, 0.721 and 0.675 accuracy for the 3 levels. For the entity typer, 0.641, 0.438 and 0.419; the relation typer, 0.430, 0.418 and 0.413.

Entity Linking: Textual Similarity Matching

We follow the approach we used in the AIDA M9 evaluation. Namely, we determine a match between a document mention and an entity string from a hypothesis by a string similarity model [START_REF] Neculoiu | Learning text similarity with siamese recurrent networks[END_REF]. Given two strings, the model outputs a score indicating how similar these two input strings are. The model is a Siamese network where two identical, parametershared modules are stacked upon the two input strings, with each string considered as a character sequence. Each module comprises of 4 bidirectional LSTM layers (size 64) followed by a feedforward layer that results in a vector of size 128. The model was trained using a dataset based on WikiNames, arising from earlier research at JHU [START_REF] Andrews | Name phylogeny: A generative model of string variation[END_REF].

Argument Linking

Argument linking is the task of determining which mentions are arguments for the events evoked by the event triggers and which role each argument serves. We perform this task in two steps for a given event: 1) for each role, we predict mentions that best fill the role (with confidences); 2) we jointly resolve the trigger's type and each mention's type ( §3.4) by greedily selecting the types that yield the highest overall confidence (link confidence and type confidence) while also obeying the type constraints in the LAO. 4We adapt the semantic role labeling model proposed by [START_REF] He | Jointly predicting predicates and arguments in neural semantic role labeling[END_REF], modified for the AIDA task. The original neural model learns span representations for predicates (triggers) and arguments (mentions), then scores them jointly with role-specific parameters to choose the best role for a predicate-argument pair. In our model, we instead find the best mention that fills a given role for a given trigger. We additionally extend the model to operate across sentences using techniques from [START_REF] Lee | End-to-end neural coreference resolution[END_REF][START_REF] Lee | Higher-order coreference resolution with coarse-tofine inference[END_REF], as many of the arguments in the training annotations do not appear in the same sentence as their trigger (23.1% of event arguments are out-of-sentence, 5.5% for relation arguments, 16.1% overall). 5 We note that restricting the model to operate on only in-sentence arguments may have led in the evaluation to higher performance, trading off recall for precision, but we operated on the assumption that we should employ models that could concievably capture any of the arguments that were being annotated by the LDC.

We pretrained the argument linking model with our new crowd-sourced data, then finetuned on the practice annotations augmented with event and entity mentions predicted by upstream components (to match testtime conditions). All training data was in English. Because the argument linking model does not observe or predict type information (types are resolved in postprocessing), we report performance on untyped (trigger, role, mention) links. On development data, our model achieved precision of 43% and recall of 58%, yielding F 1 of 49.6% for untyped link prediction.

Alignment

The evaluation scenario requires extracting information in the source language. Since we first translated Ukrainian and Russian into English to run target-side information extraction tools, we need a way to map spans extracted in English back to spans in the source language. In addition, when receiving a query span for a Ukrainian Russian document, we need to map this span onto its English translation.

Both these problems can be solved with alignment models. We alignment models in all 4 possible directions (Uk ↔ En, Ru ↔ En) following [START_REF] Stengel-Eskin | A discriminative neural model for cross-lingual word alignment[END_REF]. Due to a lack of gold alignments for Russian and Ukrainian, we used fast-align [START_REF] Dyer | the 5 These numbers are only for arguments that are entity mentions. About 12.5[END_REF] on the datasets used for Machine Translation (see Section 3.1).

Table 1 :

 1 1 http://opus.nlpl.eu/download.php?f= OpenSubtitles2018/en-uk.txt.zip Machine translation performance (BLEU)

		NMT Google translate
	Uk → En	28.2	24.4
	En → Uk	22.5	19.5
	Ru → En	28.4	-
	En → Ru	27.0	32.4
	3.1.2 Russian -English
	We used the 25M sentences from the WMT'17 news
	translation challenge 2 to build En→Ru and Ru→En
	models. The corpus in question contains the Common
	Crawl corpus, News Commentary v12, Yandex Cor-
	pus, Wiki Headlines, and UN Parallel Corpus V1.0.
	The WMT'17 news corpus is thus appropriate domain-
	wise. We report performance on the WMT'17 test set.
	Models were built using the trained with the Sock-
	eye toolkit and BPE. Our model used a Transformer-
	based sequence-to-sequence model, with 30k BPE op-
	erations, 6 encoder and decoder layers, 8 attention
	heads, 512 hidden units and 2048-dimensional feed-
	forward layers. Other hyperparameters were set to
	Sockeye's default.

http://data.statmt.org/wmt17/ translation-task/preprocessed/ru-en

If a mention does not obey the LAO type constraints, we consider the mention that fills the role with the next most confidence.


Nils Holzenberger

Raman Arora

Multiview Representation Learning for a Union of Subspaces

Canonical correlation analysis (CCA) is a popular technique for learning representations that are maximally correlated across multiple views in data. In this paper, we extend the CCA based framework for learning a multiview mixture model. We show that the proposed model and a set of simple heuristics yield improvements over standard CCA, as measured in terms of performance on downstream tasks. Our experimental results show that our correlation-based objective meaningfully generalizes the CCA objective to a mixture of CCA models.

Introduction

Multiview, correlation-based representation learning has been shown to be useful on a variety of tasks [START_REF] Hardoon | Canonical correlation analysis: An overview with application to learning methods[END_REF]Wang et al., 2015b;Arora & Livescu, 2013;2014;[START_REF] Benton | Learning multiview embeddings of twitter users[END_REF][START_REF] Vásquez-Correa | Multi-view representation learning via GCCA for multimodal analysis of parkinson's disease[END_REF][START_REF] Holzenberger | Learning from multiview correlations in open-domain videos[END_REF]. Its main workhorse, Canonical Correlation Analysis (CCA) [START_REF] Hotelling | Relations between two sets of variates[END_REF], enjoys several non-linear extensions with theoretical guarantees (kernel CCA and deep CCA) (Wang et al., 2015a;[START_REF] Lai | Kernel and nonlinear canonical correlation analysis[END_REF], and can be extended to more than 2 views [START_REF] Horst | Generalized canonical correlations and their applications to experimental data[END_REF][START_REF] Rastogi | Multiview LSA: Representation learning via generalized CCA[END_REF][START_REF] Benton | Deep generalized canonical correlation analysis[END_REF].

A related approach that yields an arguably richer representation is to learn a mixture model -instead of learning a single, high-dimensional, complex subspace, we can seek to find a union of subspaces. Mixture models, especially Gaussian Mixture Models, have a wide range of applications, from topic modeling [START_REF] Blei | Latent dirichlet allocation[END_REF] to speech recognition [START_REF] Gales | The application of hidden markov models in speech recognition[END_REF]. In many cases, data can naturally be described as a union of distributions. For instance, phonemes in speech are a superposition of low-dimensional processes, either explained as templates of time-frequency vectors, or as typical articulatory motions [START_REF] Sugamura | Isolated word recognition using phoneme-like templates[END_REF]. In the domain of information extraction, documents fall into categories such as newswire, blog posts, agency reports or tweets. In machine translation, domain match or mismatch plays an important part in the performance of a translation system [START_REF] Koehn | Six challenges for neural machine translation[END_REF]. Identifying the underlying components of the mixture, either explicitly or implicitly, can help unsupervised modeling of the data, useful for a host of downstream tasks.

Learning a mixture of subspaces rather than a single subspace also makes personalization simpler. For an unseen example, being able to assign it quickly to a subpopulation can make classification tasks require less data to achieve good performance. For example, in the context of speaker adaptation, being able to assign a speaker to a specific group of speakers improves speech recognition performance [START_REF] Kuhn | Rapid speaker adaptation in eigenvoice space[END_REF].

In this paper, we propose to extend the framework of CCA to learn a union of subspaces. Specifically, we assume that each data point belongs to one of a finite number of sources, and we learn a pair of linear CCA transformations for each source. CCA with a single transformation can detect canonical directions that span the entire dataset. A mixture of CCA is also able to detect the main canonical directions, because they can be found in each subpopulation. In addition, a mixture of CCA can pick up correlations which are present at the subpopulation level but cancel each other out at the population level. This argument would suggest we can provide a much finer grained representation, possibly without increasing the dimensionality of the representation.

While the premise might seem straightforward, combining CCA and mixture models poses a number of challenges: what objective is to be maximized? How to simultaneously learn the cluster assignments and the transformations? Are there any guarantees regarding convergence, and recovery of cluster memberships? Once the parameters of the mixture model are learned, how to assign a new point (x, y) to a cluster? How to assign a single x without corresponding y to a cluster? This paper is meant primarily as a proof of concept, and leaves most of the above questions open. We propose a new objective function, as well as a heuristic way of maximizing it, and test its performance in two distinct settings.

Related work

We use Canonical Correlation Analysis (CCA) [START_REF] Hotelling | Relations between two sets of variates[END_REF] to learn representations for a primary view. We assume that at training time, we are given two views of the same data point. For instance, for a given speech utterance, the audio recording and the articulatory measurements. These two views are represented by random variables X and Y (d X -and d Y -dimensional respectively). Linear CCA seeks two linear transformations U ∈ R d X ×k and V ∈ R d Y ×k such that the components of U T X and V T Y are maximally correlated. Formally, we want to maximize E

X,Y [tr (U T XY T V)] subject to the constraints that E X [U T XX T U] = E Y [V T Y Y T V] = I k .
Given a dataset {x i , y i } N i=1 , we define C XY the empirical cross-covariance matrix between X and Y , and C XX and C Y Y the empirical auto-covariance matrices of X and Y , respectively. U and V are given by the k left and right singular vectors of

(C XX ) -1/2 C XY (C Y Y ) -1/2 with the largest singular values, multiplied by (C XX ) -1/2 and (C Y Y ) -1/2 .
At test time, we assume that only the primary view is available. Ideally, we would want to reconstruct the second view with the primary view [START_REF] Ngiam | Multimodal deep learning[END_REF]. However, in general, that is a difficult task -take for example generating speech from text. Instead, it is easier to predict the dependent variate which has the largest multiple correlation. In other words, when no single regression provides a fully adequate solution, CCA is a better objective than predicting one view with the other.

Deep CCA (DCCA) [START_REF] Andrew | Deep canonical correlation analysis[END_REF]) is a natural extension of linear CCA, where one seeks to maximally correlate U T f (X) and V T g(Y ). f and g are non-linear feature extractors, which can be learned via gradient descent on the CCA objective. It is also natural to extend CCA to multiple views [START_REF] Horst | Generalized canonical correlations and their applications to experimental data[END_REF].

CCA can also be reformulated as a probabilistic model. [START_REF] Bach | A probabilistic interpretation of canonical correlation analysis[END_REF] show that the solution to the linear CCA objective function is, up to arbitrary rotation and scaling, contained in the maximum-likelihood solution for the parameters of a Gaussian. [START_REF] Podosinnikova | Beyond CCA: moment matching for multiview models[END_REF] extend this probabilistic formulation of CCA to fit any type of statistical distribution, including when one or both views are discrete. [START_REF] Klami | Local dependent components[END_REF] place the probabilistic model of [START_REF] Bach | A probabilistic interpretation of canonical correlation analysis[END_REF] in a Bayesian setting, with a Dirichlet process, which naturally leads to a mixture of Gaussians. They then maximize log-likelihood under this GMM model using methods from variational inference. As a final step, they extract CCA projections from each Gaussian. This amounts to performing GMM-based clustering first, and then performing CCA on each of the learned clusters. In contrast, we define a correlation-based objective for multiple transformations, and propose to maximize it directly. It remains to be examined whether both objective functions are equivalent, or even whether stationary points of both objective functions are equivalent. In fact, the model of [START_REF] Klami | Local dependent components[END_REF] might be suboptimal in terms of our objective. Other works [START_REF] Wang | Variational bayesian approach to canonical correlation analysis[END_REF][START_REF] Viinikanoja | Variational bayesian mixture of robust CCA models[END_REF][START_REF] Hosino | High dimensional non-linear modeling with bayesian mixture of CCA[END_REF] have used very similar approaches.

Most related to this work is that of [START_REF] Fern | Correlation clustering for learning mixtures of canonical correlation models[END_REF], where the authors use of a mixture of CCA, and provide an optimization heuristic similar to ours. However, their work differs on a number of points. First, [START_REF] Fern | Correlation clustering for learning mixtures of canonical correlation models[END_REF] do not provide an objective function to optimize. Second, they do not test the CCA projections in a downstream task, and are thus arguably not concerned with questions pertaining to representation learning. Third, [START_REF] Fern | Correlation clustering for learning mixtures of canonical correlation models[END_REF] do not provide any way of assigning a point with a single view to a cluster (i.e., they specify how to assign (x, y) to a cluster, but not how to assign x to a cluster in the absence of y). The main focus of our work is the objective function, as a possible extension of CCA and representation learning method.

Mixture model for Canonical Correlation Analysis

We propose to learn a union of subspaces that best characterizes a set of points with two views, using canonical correlation analysis (CCA). It involves assigning cluster memberships to data points, and we limit ourselves to linear versions of CCA. Throughout this paper, we will refer to this method as mixture of CCA (MCCA).

Objective formulation

We assume that our data consists of two sets of points. Let

X ∈ R d X ×N , Y ∈ R d Y ×N be the matrices containing the paired views, with x i ∈ R d X (resp. y i ∈ R d Y ) being the i-th column of X (resp. Y ).
For example, in our application to speech recognition, X represents a set of MFCC frames and Y the corresponding articulatory measurements. We consider a mixture of R subspaces and consider the following CCA formulation. We define the MCCA objective as:

O(U, V, α) = R r=1 tr (U T r C XY r V r )
subject to the constraints that

∀r ∈ {1..R}, U T r C XX r U r = V T r C Y Y r V r = I k where U = {U r ∈ R d X ×k | r = 1, . . . , R}, and
V = {V r ∈ R d Y ×k | r = 1, .
. . , R} are the CCA transformation matrices associated with each mixture component.

The assignment of data points to mixture components is given by the following scalar weights:

α = {α i,r ∈ R ≥0 | i ∈ [N ], r ∈ [R], ∀i, R r=1 α i,r = 1},
where [p] denotes the set {1, . . . , p}.

The weighted empirical covariance and cross-covariance matrices for each mixture component are given as

C XY r = 1 N i=1 α i,r N i=1 α i,r (x i -µ X r )(y i -µ Y r ) T , C XX r = 1 N i=1 α i,r N i=1 α i,r (x i -µ X r )(x i -µ X r ) T ,
and

C Y Y r = 1 N i=1 α i,r N i=1 α i,r (y i -µ Y r )(y i -µ Y r ) T ,
and the weighted means of the mixture components are given as

µ X r = 1 N i=1 α i,r N i=1 α i,r x i , µ Y r = 1 N i=1 α i,r N i=1 α i,r y i .
At training time, the goal is to find

U * , V * , α * = argmax U,V,α O(U, V, α) (subject to constraints),
i.e., to find for each mixture component, both CCA transformation matrices and the assignments for each data point (x i , y i ).

At test time, given x and/or y, one needs to estimate the {α r } r before projecting x (respectively, y) to

R r=1 α r U T r x (respectively, to R r=1 α r V T r y).
Algorithm 1 Heuristic optimization for MCCA

Input: MCCA hyperparameters {R, k, w X , w Y } and dataset {X ∈ R d X ×N , Y ∈ R d Y ×N } Output:
Transformation matrices U and V , and cluster assignments α initialize α using a clustering algorithm over the native space find U, V = argmax

Ũ , Ṽ O( Ũ , Ṽ , α) return U, V, α
To avoid spurious correlations, we regularize the covariance matrices by adding scaled identity matrices, i.e., replacing

C XX (respectively, C Y Y ) with C XX + w X I (respectively, C Y Y + w Y I).

Learning and inference

The optimization problem in Section 3.1 is nonconvex, and jointly minimizing over all parameters seems daunting from a computational perspective due to intractability. Therefore, we consider the following optimization approach that first estimates α, then estimates U and V given α, as summarized in Algorithm 1. When α is fixed, it is well known that we can globally maximize the R CCA objectives over the choice of U and V , as described in Section 2. Similarly, fixing the CCA subspaces, maximizing over α is essentially learning a mixture model over the shared representation.

At training time, we first cluster the data points using a CCA projection and k-means clustering, yielding hard assignments for α, i.e. α ∈ {0, 1} N R . We then learn U and V .

At test time, we need a way of inferring the {α i,r } r for a given (x i , y i ) where x i or y i may be missing. In our experimental setting, only x is available at test time. From the properties of the CCA projections, we know that var(U T X) = I k . Heuristically, we expect U T (x -µ X ) to be drawn from a unit variance Gaussian distribution. Thus, we assign x to the cluster

r = argmin r∈{1,...,R} ||U T r (x -µ X r )|| 2 2 -log π r ,
where π r is the fraction of points belonging to cluster r.

In some of our speech recognition experiments, α will be given at training time.

Experiments

In this section, we illustrate the possible uses of MCCA, and test its performance against standard, vanilla linear CCA described in Section 2, and referred to as VCCA. All nonlinear extensions of CCA -deep CCA and kernel CCAhave significantly more representational power than linear

Phoneme classification

The University of Wisconsin X-ray microbeam database (XRMB) [START_REF] Westbury | X-ray microbeam speech production database[END_REF] [START_REF] Malkov | Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs[END_REF], then measure the score on the dev set. We use the average dev score over the 6 folds to compare hyperparameters of a given method. This procedure allows us to pick the best hyperparameters for a given CCA method (VCCA or MCCA). Methods are then compared on the 6 test sets, with hyperparameters Θ * and Φ * . We report average and standard deviation of the performance over the 6 dev and test sets. Knn classifier hyperparameters Φ contain the type of distance used (L2 or cosine), the number of neighbors (8, 16, 32, 64, 128 or 256), and whether to append the original MFCC features to the CCA features.

We sweep k over {10, 30, 50, 70, 90, 110}, and w X and w Y independently over {1, 0.1, 0.001}. For VCCA, we use the projection of the MFCC features, and experiment with appending the original MFCC features to perform knn classification. For MCCA, we also experiment with appending the original MFCC features. In addition, when projecting point

x, we have the possibility of mapping x to R r=1 α r U T r x ∈ R k or to the concatenation of U T 1 x, U T 2 x, ..., U T R x, which is an Rk dimensional vector. Note that the former requires inferring α at test time, while the latter doesn't. We experiment with both settings, reporting the former as "projection" and the latter as "concatenation".

Table 2 reports our experimental results. We experiment with the oracle of Table 1 being present or absent at training time (denoted by "oracle" and "no oracle" respectively). When the oracle is absent at training time, we sweep R over {2, 4, 8, 16}. In this setting, both instances of MCCA yield a significant improvement over VCCA. The best dev score for VCCA is achieved with k = 70, w X = 0.001, w Y = 1, while the best score for MCCA with oracle is achieved with k = 50 and the same values of w X and w Y ; the best score without oracle is achieved with k = 30, w X = w Y = 0.001 and R = 8. There is no reduction in the dimensionality of the representation because the best dev score is achieved by concatenating all R representations.

The results in Table 2 show that MCCA yields a significant improvement over VCCA, with and without oracle at training time. In practice, it is likely that there is no oracle at training time, either because of a lack of labels or because there is no clear partitioning of the data. The bottom two lines in Table 2 show that the absence of the oracle does not necessarily imply a loss in performance. In fact, learning each point's assignment to a mixture component yields better results than following the heuristic from Table 1.

If the oracle had been available at test time in addition to training time, the instance of MCCA projection reported in Table 2 would reach 77.5 ± 2.2% on the dev set and 77.5 ± 2.1% accuracy on the test set. This shows that while our test-time heuristic is able to outperform standard CCA, there is still room for improvement. It also shows that, although our MCCA projection results are below our MCCA concatenation results, given correct point assignments, projection is enough to guarantee a useful representation.

In Figure 1, we show perplexity matrices for some of our MCCA models on the test set. On that plot, phonemes are sorted according to their membership in the 4 groups of Table 1. The first column has access to the oracle at training time, and achieves the best score when projecting points with a single CCA transformation. In that case, ideally, we would want the assignments to be 4 disjoint bands following the 4 clusters. However, each phoneme is mostly assigned to groups 1 and 2, which are also the groups with the largest mixing weights. This mis-assignment can be somewhat alleviated by removing the log π r term (see Section 3.2), but worsens the dev score. Columns 2 shows that, given 4 clusters but no access to the oracle, the clustering heuristic used comes up with sharp clusters for consonants. Roughly, cluster 1 contains the labials, cluster 3 contains the alveolars, and the velars are spread out among clusters 1 and 3. Consonants are mostly spread out between clusters.

If we do not constrain the number of clusters to be 4, the best performing model without oracle at training time and using projection has 2 clusters, shown in the third column.

In that case, we see mostly sharp assignments, i.e. almost all phonemes are assigned to a single cluster, despite no access to any phonemic information. With some exceptions, consonants belong to cluster 1, and vowels fall into either cluster. The fourth column shows cluster assignments for the best performing MCCA model; note that this model concatenates representations, and thus cluster assignments do not matter at test time. Clusters 2 and 7 seem to be unused, and cluster 4 loosely corresponds to labials. Alveolars and velars are spread out among clusters 1 and 8. Rows are normalized to sum to 1. Each model was chosen as the best in its category, as described by the label. Columns 1, 3 and 4 correspond to the models reported in Table 2.

Twitter data

To provide an illustration of MCCA in a different domain, we use the corpus of Twitter data of [START_REF] Benton | Learning multiview embeddings of twitter users[END_REF]. In that paper, various methods are used to build representations of Twitter users, based on their tweets and friend networks. Following the nomenclature of [START_REF] Benton | Learning multiview embeddings of twitter users[END_REF], we use the ego view, i.e. a PCA-based representation of the user's tweets, and the friends view, i.e. a PCA-based representation of the user's friend network or graph. We use the ego view as the primary (available at training and test time) and the friends view as the secondary view (available at training time only). Using the notation of 3.1, X (resp. Y ) is the primary (resp. secondary) view. Both are 1000 dimensional. To estimate how much the use of CCA is improving the performance on a given task, we also report results using the primary view at training time, under "raw features".

USER ENGAGEMENT PREDICTION

To evaluate the learned representations, we perform a user engagement prediction task, using hashtag as a proxy (following [START_REF] Benton | Learning multiview embeddings of twitter users[END_REF]). We have 2 sets (dev and test) of 200 unseen hashtags each, and for each hashtag, a set of 16k unseen users who have used them. The task is to predict, for each hashtag, which users are likely to use it, based on the first 10 users who have used it. We follow the setup of [START_REF] Benton | Learning multiview embeddings of twitter users[END_REF], but report slightly different metrics. Note that because of missing views, we have had to restrict the number of users, including in the test set, and thus the results are not comparable. The train, dev and test set contain respectively 79900, 8220 and 8071 users. For a given method, we use the performance on the dev set to select the best hyperparameters, then compare methods on the test set. We decide whether or not to concatenate the original representation to the CCA projection based on the dev set performance. When concatenating, we scale each representation by its average norm taken over the dataset, in order to balance out the weight of each representation in the cosine similarity.

More specifically: we first project all the users u 1 , u 2 , ..., u N of the dev (resp. test) set using the learned CCA transformations. For each hashtag i in the dev (resp. test) set, we have a list of users who have used them. We pick the first 10, and compute the average of their embeddings; we take this as the representation h i of the hashtag. Hashtag representations {h i } i and user representation {u j } j are separately mean-centered. Then, for each hashtag h i and each user u j , we compute the scaled cosine similarity 1 2 (1+ hi•uj ||hi||||uj || ), yielding a confidence score between 0 and 1. We use that score as a measure of how likely the user j would use the hashtag i. We can compare these scores with the ground truth of which user actually used each hashtag.

We report Recall@1000, mean reciprocal rank, and the area under the ROC curve (ROC-AUC) [START_REF] Fawcett | ROC graphs: Notes and practical considerations for researchers[END_REF] of a multiclass classifier mapping users to hashtags. Recall@1000 and ROC-AUC are bounded between 0 and 100, and mean reciprocal rank between 0 and 1. For all of these metrics, higher is better. For each hashtag, we rank the users based on their confidence score, in decreasing order. Recall@1000 tells us how much the top 1000 users overlap with the ground truth, and so how many relevant answers are present in our top 1000 answers. Mean reciprocal rank tells us how close the first relevant result is to the top of our result list. If we were to build the simplest classifier for hashtag j, it would rule that user i is going to use hashtag j if 1 2 (1 + hi•uj ||hi||||uj || ) > θ j where θ j has to be set for each hashtag. The ROC-AUC tells us how well the confidence scores of correct users are separated from those of incorrect users. A ROC-AUC score of 100 means we could set the {θ j } j randomly and still classify each user and hashtag correctly, and 50 means we are doing as well as a random classifier. The higher the ROC-AUC score, the better the performance of the classifier will be, regardless of the values of the {θ j } j .

In contrast to the experiments in Section 4.1, there is no obvious partitioning of Twitter users on this dataset, and thus we have no oracle at all in this case. We sweep k over {200, 400, 600, 800, 1000}, and w X and w Y independently over {1, 0.1, 0.01}. In the case of MCCA, we sweep the number of clusters R over {2, 4, 8, 16} and follow Section 3.2 for learning and inference.

We summarize the results of the user engagement prediction task in Table 3.

On all three metrics considered, MCCA outperforms VCCA. For Recall@1000 and ROC-AUC, the hyperparameters achieving the best scores are identical. For VCCA, k = 600, w X = 0.1, w Y = 1; for MCCA projection, R = 2, k = 1000, w X = 0.1, w Y = 0.001; for MCCA concatenation, R = 8, k = 400, w X = 0.001, w Y = 1. For all three, the best scores are achieved with the CCA representation alone (without appending the raw features). In the case of mean reciprocal rank, the best scores are achieved by concatenating the CCA representation and the raw features. For VCCA, k = 400, w X = 0.001, w Y = 1; for MCCA projection, R = 8, k = 800, w X = 0.001, w Y = 1; for MCCA concatenation, R = 2, k = 1000, w X = 0.001, w Y = 1. The large difference in best performing hyperparameters shows how important task-based hyperparameter selection is. In that respect, MCCA has 2 more hyperparameters that can be tuned based on the task: the number of mixture components R, and whether to project x to R r=1 α r U T r x ("projection") or to the concatenation of U T 1 x, U T 2 x, ..., U T R x ("concatenation"). Except for mean reciprocal rank, MCCA projection performs consistently worse than VCCA. This could possibly be because the clusters picked by our heuristic have no intrinsic relationship with the hashtags. It is interesting to note that MCCA projection drastically improves the mean reciprocal rank to perfect or almost perfect score, meaning that the top result is a correct answer, almost always. Higher ROC-AUC scores for MCCA concatenation show that this method is better able to separate, for each hashtag, relevant from irrelevant users. This is consistent with higher Recall@1000 scores, which indicate more relevant users In a similar setting, we perform friend recommendation [START_REF] Benton | Learning multiview embeddings of twitter users[END_REF]. 500 user accounts were set aside (250 for each dev and test), and it was recorded which of the dev and test users followed those accounts. The friend recommendation task amounts to predicting which user accounts a given user is likely to follow. The train, dev and test set contain respectively 6522, 82608 and 81985 users. The task setup is identical to user engagement prediction. We sweep over the same values of hyperparameters as for user engagement prediction.

We report our results in Table 4. Based on the results using the raw features, this task is much harder than user engagement prediction, described in 4.2.1. In this setting, MCCA performs better or on par with VCCA. The smaller performance gap between VCCA and MCCA could be explained by the much smaller size of the train set, and possibly the difficulty of obtaining a coherent clustering. For each of the best performing MCCA instances, the clustering at training time shows that one of the clusters collapses to 1, 2 or 3 points. For each method, the best performing hyperparameters vary from one task to the other, without any pattern emerging. Again, this shows the impact of task-based hyperparameter selection.

While the task is quite different from user engagement prediction described in Section 4.2.1, the conclusions regarding VCCA and MCCA remain mostly identical.

Conclusion

In this paper, we proposed a novel objective for representation learning, combining CCA and mixture models, in conjunction with simple heuristics to maximize the objective at training time, and use the representations at test time.

Evaluating our representations in different settings, we have shown the usefulness of both our objective and our heuris- tics. Overall, across tasks, our proposed method performs on par or better than the standard version of CCA.

Our results suggest that the proposed method is a valid objective, potentially a good generalization of the standard CCA objective. With more hyperparameters than the standard CCA objective, and the possibility of informing the mixture components with hierarchical structure present in the data, it has the potential to better adapt to downstream tasks. It would further benefit from a more thorough optimization scheme with provable guarantees, and possibly extensions to non-linear methods.

Figure 1 .

 1 Figure1. Perplexity matrices for various MCCA models on the test set. Rows are labeled with phonemes and columns with mixture components. Rows are normalized to sum to 1. Each model was chosen as the best in its category, as described by the label. Columns 1, 3 and 4 correspond to the models reported in Table2.

Table 1 .

 1 4 groups used to partition the English phonemes. AH, IH, AO, ER, EH, IY, UW, AE, OW, UH, AY, EY, AW, OY, HH, Y ALVEOLARS CH, D, DH, JH, L, N, R, S, SH, T, TH,

		Z, ZH
	LABIALS	B, F, M, P, V, W
	VELARS	G, K, NG
	CCA. Most likely, deep CCA would outperform all other
	methods. Thus, for the comparison to non-linear extensions
	of CCA to be fair, one would have to extend MCCA to non-
	linear methods. This would pose many more challenges
	related to optimization. Instead, our intention is to carefully
	illustrate the linear version, leaving non-linear extensions to
	future work.

VOWELS

AA,

  is a set of sound recordings and articulatory measurements, acquired during production of English read speech. The acoustic recordings are processed into sequences of 13 dimensional MFCC frames. The articulatory measurements are the horizontal and vertical displacements of eight pellets affixed to critical articulators (tongue, lips, jaw) at a given point in time, resulting in a 16 dimensional observation vector. Both views are temporally aligned, and thus we have pairs of MFCC frames and articulatory features. This correspondence is the only supervisory signal when learning representations; while the manual annotations of the spoken text are available, we do not use it for feature learning to avoid any task dependence in the learned representations. The location of the pellets in the original dataset go missing for various reasons; we use the completed version of the XRMB dataset due to[START_REF] Wang | Reconstruction of articulatory measurements with smoothed low-rank matrix completion[END_REF]. The MFCC frames are augmented with deltas and double deltas, and are mean-centered and variancenormalized per speaker. The articulatory features are also mean-centered and variance-normalized per speaker.In this section, we illustrate the usefulness of learning a union of subspaces, by partitioning the 39 phonemes of the XRMB dataset into 4 groups, as detailed in Table1. For some experiments, we assume access at training time to an oracle providing us with the correct group for each training instance; in those cases, we are not concerned with inferring α. At test time, we also experiment with the presence and absence of said oracle. In the absence of the oracle, we use the heuristic described in Section 3.2 to infer the membership of each point.Using the notation of Section 3.1, X consists of 7 stacked MFCC frames with deltas and double deltas, centered around the frame of interest, and is thus 273 dimensional vectors. Y consists of 7 stacked articulatory feature vectors, corresponding to the 7 MFCC frames, and is thus 112 dimensional. The training set has 1.4M data points, and each of the 6 cross-validation folds mentioned below has between 79k and 86k data points.

	To evaluate the quality of the learned representations, we
	follow Wang et al. (2015b) and leave out 12 speakers. The
	other 35 speakers are used to learn representations. The 12
	left-out speakers are used to measure how well the learned
	representations for audio can be used to perform phoneme
	classification on unseen data. To estimate generalization
	error, the 12 speakers are partitioned into 6 sets of 2 speak-
	ers each, and used in a 6-fold cross-validation, each fold
	composed of 4 training speakers, 2 dev speakers, and 2 test
	speakers. We use the 4 train and 2 dev speakers to find the
	best parameters for a k-nearest neighbor classifier

Table 2 .

 2 Phoneme classification accuracy on XRMB dataset without oracle at test time. Oracle (resp. no oracle) indicates access (resp. no access) to the oracle at test time.

	FEATURES	DEV (%)	TEST (%)
	MFCC	60.6 ± 2.8	60.6 ± 2.5
	VCCA	65.3 ± 2.8	65.3 ± 2.9
	MCCA (ORACLE, PROJEC-	64.1 ± 2.7	64.2 ± 2.6
	TION)
	MCCA (ORACLE, CONCATE-	68.0 ± 2.7	68.0 ± 2.8
	NATION)
	MCCA (NO ORACLE, PROJEC-	64.7 ± 2.8	64.8 ± 2.7
	TION)
	MCCA (NO ORACLE, CON-	69.2 ± 2.8	69.3 ± 2.8
	CATENATION)

Table 3 .

 3 User engagement prediction results on Twitter dataset. REC: Recall@1000, ROC-AUC: area under the ROC curve, MRR: mean reciprocal rank.

			REC	ROC-AUC	MRR
	FEATURES		DEV TEST DEV TEST DEV TEST
	RAW	FEA-	53.0 44.0 78.4 72.0 .858 .876
	TURES
	VCCA		62.0 51.8 83.5 77.1 .911 .969
	MCCA (PRO-	60.1 50.6 82.7 76.4 1.0	1.0
	JECTION)
	MCCA (CON-	64.3 53.2 85.1 78.4 .920 .958
	CATENATION)
	within the top 1000 results.
	4.2.2. FRIEND RECOMMENDATION

Table 4 .

 4 Friend recommendation results on Twitter dataset. REC: Recall@1000, ROC-AUC: area under the ROC curve, MRR: mean reciprocal rank.

			REC	ROC-AUC	MRR
	FEATURES		DEV TEST DEV TEST DEV TEST
	RAW	FEA-	3.21 2.96 59.6 59.3 .821 .788
	TURES
	VCCA		4.02 4.04 62.3 62.1 .924 .909
	MCCA (PRO-	3.83 3.48 62.2 61.8 .994 .996
	JECTION)
	MCCA (CON-	4.03 4.05 62.6 62.0 .957 .942
	CATENATION)

Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA. Correspondence to: Nils Holzenberger <nholzen1@jhu.edu>.