Tommaso Venturini
email: tomm.venturini@gmail.com

Tobias Blanke

Kari De Pryck

Similarity Sampling by Machine Learning A Social Science Experiment with Artificial Intelligence and IPCC Leadership

Keywords: Network analysis, organisations studies, international organisations, IPCC, sampling, machine learning, artificial intelligence

In this paper, we devise a machine learning protocol to tackle a complex sociological task: to create a research sample from a few examples of interest, but in the absences of a clear definition of the target subset. As an example, we create a sample of organisational leaders starting from a list of nominees for the Bureau of the Intergovernmental Panel on Climate Change. The difficulty in this task lies in the impossibility to spell out the characteristics that define leadership in a complex and highly distributed organization like the IPCC. To bypass this lack of explicit definition, we use a series of techniques for anomaly detection to identify IPCC contributors with profiles similar to official Bureau nominees. We find that we can construct a precise (albeit implicit) model of IPCC leadership despite its social and political complexity, and that we can usefully use this model to expand our initial sample.

Introduction

This paper presents an experiment in the use of machine learning to tackle a complex social phenomenon: the identification of informal leaders in a highly distributed international organisation (the Intergovernmental Panel on Climate Change, IPCC). We found that we can build a model capable of an accurate reading of IPCC leadership despite the social and political intricacy of this organisation. Accuracy, however, is not our prime goal. Despite recurring claims that computational methods will revolutionize social sciences [START_REF] Carley | Artificial intelligence within sociology[END_REF][START_REF] Lazer | Computational social science[END_REF][START_REF] Conte | Manifesto of computational social science[END_REF][START_REF] Raghavan | It's time to scale the science in the social sciences[END_REF][START_REF] Alvarez | Computational Social Science[END_REF], the rolling out of such methods is still hindered by the mismatch between their functional requirements and the actual conditions of social research, e.g. the lack of large and well-structured datasets [START_REF] Kitchin | Small data in the era of big data[END_REF] and the focus on interpretation [START_REF] Woolgar | Why not a Sociology of Machine? The Case of Sociology and Artificial Intelligence[END_REF][START_REF] Grimmer | We Are All Social Scientists Now: How Big Data, Machine Learning, and Causal Inference Work Together[END_REF][START_REF] Venturini | Fill in the Gap: A New Alliance for Social and Natural Sciences[END_REF]. For computational methods to realize their full sociological potential, it is therefore crucial to investigate their usage in vivo.

This is precisely what we try to do in this paper, by putting ourselves in a situation that is common in social sciences but in many ways opposite to the one generally chosen for machine learning: we deal with a task that is highly disputed among human experts; we rely on a small dataset (the 5,676 individuals who have contributed to the IPCC main reports); we exploit artificial intelligence (AI) techniques available in free packages and compatible with a normal laptop; and, finally, we aim at sociological relevance rather than technical efficiency.

Contemporary AI techniques do not aim at in-depth understanding [START_REF] Andler | From paleo to neo connectionism[END_REF], [START_REF] Lecun | Deep learning[END_REF]. Quite the opposite, their success derives largely from their capacity to give up some of interpretability in exchange for better performances [START_REF] Jordan | Machine learning: Trends, perspectives, and prospects[END_REF], [START_REF] Cardon | Neurons spike back[END_REF]. Being deliberately devised a black boxes, it is not surprising these techniques are at odds with the ultimate goal of social research, that is to advance our capacity to make sense of collective life [START_REF] Burrell | How the machine 'thinks': Understanding opacity in machine learning algorithms[END_REF]- [START_REF] Rudin | Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead[END_REF]. This does not mean, however, that AI cannot be helpful. The black boxed nature of machine learning, in particular, can come in handy to perform research tasks whose contours are inherently vague or implicit. In this paper, we exemplify such use of AI for social research by proposing a technique of similarity sampling based on a protocol to teach the machine to recognise individuals with outlying leadership profiles.

Implicit similarity sampling

In order to understand our methodological proposition, it is important to notice that our technique departs deliberately from the conventional approach of random sampling, which aims at selecting items whose features can be generalized to the population it represents [START_REF] Teddlie | Mixed Methods Sampling[END_REF]. Instead, we propose here a form of purposive sampling [START_REF] Patton | Qualitative Research and Evaluation Methods[END_REF], also known as non-probabilistic or judgement sampling [START_REF] Etikan | Comparison of Convenience Sampling and Purposive Sampling[END_REF] because it "involves the pursuit of the kind of person in whom the researcher is interested" [20 p. 137]. This type of sampling is preferred when the objective is to yield "insights and in-depth understanding rather than empirical generalizations" [21 p.1] and when the target of the research is not the whole population but a particularly important subgroup.

In this paper, we are interested in identifying IPCC informal leaders. While leadership is a classic topic in International Relations (cf. among others, [START_REF] Cox | The Executive Head: An Essay on Leadership in International Organization[END_REF]- [START_REF] Reinalda | Leadership of International Organizations[END_REF]), most research tends to focus on formal leaders -i.e. individuals who hold specific chairs in international organisations [START_REF] Tallberg | The power of the chair: Formal leadership in international cooperation[END_REF] -and struggles to identify "informal leaders" [START_REF] Pielstick | Formal vs. Informal Leading: A Comparative Analysis[END_REF] whose influence is palpable but not explicitly defined. In the IPCC, for example, the formal leadership is clearly represented by the Bureau, an organ that is elected at the beginning of each assessment cycle and that coordinates the activities of the organisation. Yet, sitting in the Bureau is not the only way to be influential in the IPCC. In an organisation whose activities are based on voluntary work and whose mandate is to serve as an interface between different national and disciplinary communities, individuals who have been around for a longer time, or played key facilitating roles, or represented powerful countries or disciplines may have considerable sway, whether or not this is officially acknowledged.

The difficulty of identifying IPCC informal leaders exemplifies the difficulty of purposive sampling when the contours of the target subgroup cannot be explicitly defined. In the absence of a clear condition or rule, techniques such as criterion sampling, typical (or deviant) case sampling, intensity sampling are impracticable [START_REF] Patton | Qualitative Research and Evaluation Methods[END_REF], [START_REF] Suri | Available from Deakin Research Online : Purposeful Sampling in Qualitative Research Synthesis[END_REF]. Snowball sampling can help extend the sample to the individuals who are most connected to official leaders, but often (as in our case) no data is available on the connections between individuals.

The failure of classic sampling to cope with the elusiveness of organisational leadership suggests a possible sociological use of machine learning. While we cannot formulate an explicit definition of IPCC informal leadership, we know from fieldwork [START_REF] Pryck | Expertise under Controversy : The Case of the Intergovernmental Panel on Climate Change (IPCC)[END_REF] that the Bureau nominees are generally chosen among the informal leaders of the organisation. As a consequence, we can reasonably assume that informal leaders resemble Bureau candidates even though the specific dimensions of this similarity cannot be made explicit. This suggests an avenue for the sociological exploitation of AI techniques, as one of the tasks in which these techniques are most efficient is the extrapolation from a learning dataset. Several techniques originally conceived for prediction (e.g., to anticipate crime outbursts, [START_REF] Benbouzid | From situational crime prevention to predictive policing. Sociology of an ignored controversy[END_REF]) or categorization (e.g., to improve marketing campaigns, [START_REF] Syam | Waiting for a sales renaissance in the fourth industrial revolution: Machine learning and artificial intelligence in sales research and practice[END_REF]) can therefore be repurposed for social sciences [START_REF] Blanke | Predicting the Past[END_REF] as methods for similarity sampling.

The IPCC leadership

The case of IPCC leadership is interesting for the key role that the organisation plays in the climate regime [START_REF] Aykut | Gouverner le climat ?[END_REF]. Founded in 1988, the IPCC produces regular assessments of climate change research. Its reports are meant to offer a scientific foundation to the negotiation in the United Nations Framework Convention on Climate Change (UNFCCC). For its "efforts to build up and disseminate greater knowledge about man-made climate change", the IPCC was awarded the Nobel Peace Prize in 2007 and has since become a model for several other international expert organisations, notably the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES) and possibly the future International Panel on Artificial Intelligence (IPAI). The IPCC is a remarkable organisation for the cooperation it establishes between its authors who write the reports and its national delegates who supervise the assessments work and approve its outcomes. While this dual nature allows the IPCC to serve as an interface between science and politics, the cohabitation of scientists and diplomats is not always easy and tensions may arise in relation to the distribution of power in the organisation [START_REF] Hughes | Narrowing the Climate Field: The Symbolic Power of Authors in the IPCC's Assessment of Mitigation[END_REF], and particularly in its Bureau.

Elected by the member states at the beginning of each assessment cycle, the Bureau coordinates and oversees all IPCC activities. The composition of the Bureau (which comprises 34 members in AR6) is the result of complex voting procedures and of intense behind-the-scenes negotiations [START_REF] Pryck | Expertise under Controversy : The Case of the Intergovernmental Panel on Climate Change (IPCC)[END_REF]. The election of the IPCC Bureau is based on a complicated combination of scientific and diplomatic constraints meant to assure that its composition reflects a "balanced geographic representation with due consideration for scientific and technical requirements", Principles Governing the IPCC Work 1 , paragraph 5). To guarantee both the geographical equilibrium and the scientific value, the IPCC established complex set of rules detailed in seven-page document the "Procedures for the election of the IPCC Bureau" 2 . Rule 19, for instance, stipulates that governments "should refrain from nominating non-nationals without the consent of the nominee's national government". The rule was introduced after a major controversy broke out in 2002, when the sitting Chair Robert Watson was nominated by Portugal and New Zealand against the will of his own country (the United States), which ended up supporting the election of the Indian economist Rajendra Pachauri.

Being nominated for election in the IPCC Bureau is an extremely subtle social and political process which signals the influence achieved by someone within the organisation but in ways that cannot be straightforwardly measured or explicitly defined. And this is where machine learning can come in handy, offering a series of techniques to define a sample of informal leaders similar to the Bureau nominees without having to explicitly define the nature of this similarity.

Featurization

The first step of our protocol is the definition of the features that will be used to create a model of IPCC leadership. In this paper, we draw on a database of IPCC authors and delegates that we started to collect in two previous research projects (ANONYMISED) 3 and extended and updated since. The database contains the names of all the individuals who have contributed to the first five assessment reports (ARs) of the IPCC (and the members and candidates of the Bureau) 4 . Great effort was invested to disambiguate homonyms and merge different names of the same person, but errors may remain. In total, we have counted 5,676 individual contributors to the IPCC. Our database, furthermore, separates the different roles held by the same individual, thus containing about 18,000 rows, each corresponding to the contribution by a given individual in a given role (delegate, Bureau member, coordinating lead author, lead author, review editor, contributing author). For each of the contributions we also collected the national affiliation declared by the contributor. All the features employed in this machine learning model described in this paper are, more or less directly, extracted from this database.

Engagement features (ENG-)

A first set of features is meant to measure the level of engagement of IPCC contributors and can be extracted directly from the database. 1. The number of plenary sessions to which an individual has participated as a member of a national delegation (ENG-CountSessions) 2. The number of chapters authored by an individual (ENG-CountSignatures) 3. The responsibility roles hold by an individual (ENG-ClaSpmSyrBureau) -i.e. Contributing Leading Authors (CLA), author of Summary for Policymakers (SPM) or the Synthesis Report (SYR), or member of the Bureau in an earlier AR. 4. The last AR to which a participant has participated (ENG-LastActive), assuming that individuals active in closer ARs have more chances to be nominated for the Bureau.

Networks features (NET-)

While the features above capture the level of engagement in the organisation, they do not consider the nature of the IPCC as a 'network organisation'. Unlike other international organisations, the IPCC has no permanent organs and no stable employees (except for the dozen people of its Secretariat). The 5,676 individuals who contributed to its assessments did it voluntarily and on the payroll of other institutions. Rather than by hard institutionalization, the IPCC is kept together by the networking activities of its participants. To account for the relational features of IPCC participants, we turn our database into a bipartite network of individuals 5 and capacities, defined as the 24 different roles in which IPCC members may have contributed to the organisation6 (see the table below)

Table 1.

temporal, functional and thematic divisions of the IPCC and the roles they form

The edges of the graph in We also transform the bipartite network above in a monopartite network of individuals who have served in the same capacities 7 . From the two networks, we derive the following features (NET-) 1. The number of different capacities occupied by an individual (NET-Degree).

2. The betweenness centrality of an individual, as the number of shortest paths passing through his/her node in the monopartite network (NET-Betweenness) 3. The closeness centrality of an individual, as the sum of shortest-paths distance of his/her node to all the other nodes (NET-Closeness) 4. The eigen-centrality of an individual, as a recursive measure the connectivity of nodes, their neighbours, the neighbours of their neighbours, etc. (NET-Eigen).

Bridgeness features (BR-)

While the network features capture the relational connectedness and centrality of individuals, they do not consider the specificity of the IPCC network. In particular, they do not take into account that some roles are easier to cumulate, while others are more difficult. As Figure 1 suggests, some capacities are more "distant" than others and connecting them is then more valuable in the IPCC. The organisation itself acknowledges this point and identifies a group of 7 To generate a monopartite network from our bipartite graphs, we create a matrix in which each couple of individuals is connected by an edge if the Pearson correlations between the list of capacities they have occupied is positive. For all capacities occupied by two contributors x and y, the Pearson correlation divides the sum of their frequencies in x and y minus their respective mean frequencies m(x) and m(y) by the square root of the squares of (x -m(x)) and (y -m(y)). As illustrated by the figure below (relative to the first five ARs), the distribution of the Pearson correlation for each couple of individuals is highly uneven and this is why we chose zero as contributors of particular relational importance. The so-called "bridge authors" are authors who deal "with cross-cutting topics across WGs 8 ." In a previous paper (ANONYMIZED), we extend this notion, by calculating three distinct types of bridgeness 9 as well as their sum: 1. Thematic bridgeness, characterising the individuals, who have participated in different working groups during the same AR (BR-Thematic). 2. Functional bridgeness, characterising the individuals who have been both author and delegate during the same AR (BR-Functional). 3. Temporal bridgeness, characterising the individuals who have served in the same capacity across different ARs (BR-Temporal). 4. The sum of all the bridgeness above (BR-BridgenessSum).

National feature (NAT-)

Finally, as in all international organisations, the composition of the IPCC leadership is decided not only by the individual profiles, but also by the power equilibrium between countries. While IPCC Bureau Election Procedures demand a "balanced geographic representation", this provision only stipulates that the five regions of the world (according to the classification of the World Meteorological Organisation) should be represented in the Bureau but does not specify which countries should represent each region. As a consequence, the most influential and active countries in the organisation (e.g., the US, the Russian Federation, Japan, Brazil, China, the UK, Canada, Saudi Arabia, India, Germany or France) are also more present in its Bureau.

To account for the national affiliation of IPCC members, we add to our feature list the average percentage of gross domestic product of the country dedicated to research and development between 1996 and 2017 according to the World Bank (NAT-GDPRD) 10 . We choose this particular feature for its capacity to capture both the richness of countries and their technical and scientific power.

8 https://archive.ipcc.ch/scoping_meeting_ar5/doc10.pdf, p. 10. 9 To quantify the bridging function in the IPCC, we develop a metric called "bipartite-bridgeness" [START_REF] Venturini | Bridging in Network Organisations the Case of International Panel on Climate Change (IPCC)[END_REF], which we define as the summation of the number of indirect connections created by a node weighted by the importance of such connections and by their rarity. More precisely, the bipartite-bridgeness of a node 𝛼 of partition A is equal to the number of pair of nodes in partition B bridged by 𝛼, each weighted by:

• the total number of A-nodes to which the two B-nodes are connected (i.e. the union of their neighbours)

• and the inverse of the number of A-nodes bridging the same pair (i.e. the intersection of their neighbours)

BB(n) = ∑(i,j) [ |neighbours(i) ∪ neighbours(j)| / |neighbours(i) ∩ neighbours(j)| ] BB(n) = ∑(i,j) [ 1 / Jaccard (neighbors(i),neighbors(j)) ]
As the intersection of two sets divided by their union is commonly known as their Jaccard coefficient, the bipartitebridgeness can be defined as the summation of the inverse Jaccard coefficient of the neighbourhoods of all pairs of neighbours of 'n'. Because the Jaccard coefficient is a measure of similarity between sets, bipartite-bridgeness grows not only with the number of couples bridged but also with their relational diversity.

In the case of our networks of contributors and capacities, the bipartite-bridgeness of an individual is then defined as the summation of all the pair of capacities in which she has served weighted by their importance and the rarity of their connection.

Taking time into account

A final issue in data preparation is related with time. To train our model, we use the list of the Bureau nominees in AR5 and AR6, as well as its elected members in AR4 11 . Because these individuals have been nominated for the Bureau at a specific moment in time, we have to make sure that our model considered their features at that moment. While considering the AR4 Bureau, therefore, we should only use data from the three earlier reports (AR1, AR2 and AR3). For the AR5 Bureau, we can add the information about AR4 as well. And finally, for the AR6 Bureau, we can consider information about the five previous reports. In practice, this means that all the individual features above (ENG-, NET-and BR-) need to be calculated at three different moments in time and using only the data concerning IPCC participation that occurred before the Bureau election under consideration.

Model training

After having extracted and normalized our 13 features, we train our model to recognise good candidates for the IPCC Bureau. The problem is that out of the 11,742 rows in our dataset, only 148 rows (or 1.3% of the base), correspond to Bureau's members or candidates. As a consequence, any model that simply assumes that no individual is a Bureau candidate will achieve the stunning accuracy of 98.7%. This "accuracy paradox" prevents us from relying straightforwardly on accuracy as a metric to evaluate and improve our predictive model, thus excluding most basic machine learning approaches. However, as suggested by the figure below, the distribution of values for most of our features is highly uneven and with several outlying values, many of which correspond to Bureau candidates. Individuals nominated to the Bureau seem to have a special profile that sets them apart from all other IPCC contributors and this encourages us to rely on anomaly detection algorithms to identify them and their likes. 11 Unfortunately, you could not find the names of the candidates for AR4.

As our main anomaly detection algorithm, we used isolation forests as implemented in the opensource H2O platform [START_REF] Candel | Deep learning with H2O[END_REF]. This technique is a special case of random forests, in which the machine represents the training dataset through a series of tree-like structures with the objective of isolating the members of the target class with as few branches as possible. At each branching, the records of the dataset are divided in two classes according to a threshold on one of the features (e.g., separating the class of individuals who have occupied more than N capacities and then the subclass of those who had been active in the last AR, etc.). Averaging over many trials (hence the name of 'forest'), the machine learns which features are most important to identifying the target class. An isolation forest is an unsupervised version of the random forest, which tries to single out not a target class but each individual item. Being distinctively different from other nodes, outliers tend to be closer to the root and require fewer splits to be isolated. The final outlier score is the average number of splits needed to create a leaf for a given entry across all the trees in the isolation forest. In our case, the lower this number is for an individual, the more likely she or he is to be an anomaly and thus a Bureau candidate.

To improve our results while avoiding overfitting to the training datasets, we first run a grid search with cross-validation to find the optimal hyperparameter range for our isolation forest. Finding the ideal structure is called "hyperparameter tuning" and is a crucial albeit trial-and-error process. In our case, we looked for the overall number of trees in the forest, their maximum depth and the optimal encoding of the capacities. The result is that the best forest has trees of maximum depth 30 (mean depth of ~16 and a minimum depth of 9), with a densely populated forest of up to 10,000 trees. We split our corpus into a training set and a test set with a 75-25 split (taking care that the distribution of bureau candidates is similar in both sets and thus having 37 nominees in the test set). 2,935 entries are thus held back to evaluate how the model reacts to unseen data.

With a default scoring option, our anomaly detection algorithm cannot overcome the accuracy paradox. A straightforward application of the isolation forest method (see table 2) achieves an accuracy of ~0.95 but, while it performs very well in excluding non-Bureau individuals and avoiding false positives (specificity = 0.96), it is incapable to identify almost half of the Bureau candidates (accuracy = 0.49). Our task is therefore to find an approach that maximises the relation of specificity and sensitivity without sacrificing too much accuracy. To counter the accuracy paradox, we thus change our scoring function to over-reward the identification of bureau candidates and over-penalize their missing. The metric we are aiming to maximise is thus the 'harmonized mean' of sensitivity and specificity:

𝑆𝑐𝑜𝑟𝑒 = 𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 * 𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 + 𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦

As intended, this approach sacrifices some accuracy (which drops to ~92%), but our confusion matrix results looked more appropriate to the task of predicting bureau candidates. The sensitivity is now improved to 0.65 while specificity is still a strong 0.93. We can further improve our performance by combining our isolation forest (which identifies anomalous contributors but does not exploit the existing information on who is a bureau candidate and who is not) with a semi-supervised version of autoencoder neural network [START_REF] Baldi | Autoencoders, Unsupervised Learning, and Deep Architectures[END_REF]. Autoencoders aim at reducing the dimensionality of the data, discarding noise by determining the most important latent features of the dataset (or 'deep features') generated as black box combinations of the initial features. Our semi-supervised approach consists in generating our autoencoders using exclusively the data from the 'normal' non-bureau individuals, which will make it easier to detect the bureau candidates as outliers. By building an isolation forest on top of the second hidden layer of the autoencoder and using our custom scoring function, we are able to keep a high accuracy of 92% but at the same time maximise the relationship between sensitivity (0.92) and specificity (0.92). Only 3 bureau candidates are now missed from the test data.

Results and discussion

So far, following the conventions of predictive analytics, we presented the results of our model in terms of false positives and false negatives. Our objective, however, has never been to predict which individuals were going to be nominated as IPCC chairs and vice chairs, but to exploit our existing knowledge on the Bureau nominations to identify other individuals with similar 'leadership potential' or, to say it with other words, similar ways to be exceptional in the organisation. We can therefore abandon the binary classifications of the confusion matrices and consider our results according to a continuous measure. For each of the individuals (and for each AR), our algorithm computes the average number of splits needed to isolate the entry across the trees of the isolation forest. This metric, which we call "MeanLength" is our measure of Bureaulikeness.

To test MeanLength, we compare it to the thirteen individual features described above to show its better capacity to identify 'leader-like' individuals. To gauge such a discriminating capacity, we rank all the individuals in our dataset according to each of the features and according to the MeanLength and we calculate the average ranking of Bureau candidates and of all other IPCC participants and then simply subtract these two averages. The idea behind this test is that, as Bureau nominees are the people that the organisation itself recognises as its potential leaders, an effective measure of organisational leadership should rank them higher than other participants. In the best-case scenario, in which the 148 candidates are ranked in the first 148 positions, the difference in the average rank between candidates and non-candidates would be equal to 5871. We can use this maximum as a benchmark of the discriminating capability of different metrics. Figure 3 displays the ranking of Bureau nominees for each individual feature and for the MeanLength and table 6 compares their discriminating capacity (as the difference between the average ranking of candidates and the average ranking of non-candidates divided by 5871, the maximum possible difference).

Figure 3. rank of bureau nominees according to individual features and isolationMeanLength

Table 6. difference between the average rank of Bureau nominees and all other IPCC participants. The figure and table above demonstrate that the MeanLength is significantly more effective than all individual features in identifying IPCC Bureau nominees, being almost twice as good as the best features (BR-BridgenessSum and NET-Degree). This suggests that there is a real gain in combining features through machine learning instead of using them in isolation.

To improve our discrimination capacity even more, we can exploit the non-deterministic nature of our protocoli.e., the fact that, if executed several times, it will produce different models and different final classifications of the individuals in our corpus. Since each of these implementations of our protocol is differently affected by the noise in the data and the chances of the training process, their average results should be more robust than the results of any single instance. Indeed, running our protocol ten times and averaging the results we obtain an average MeanLength with an even greater discrimination capacity (0.972). But we can do even better. Figure 4 shows that Bureau candidates (the triangles) have not only a lower average MeanLength, but also that their evaluation is more stable across different runs of our protocol.

Application to sampling

Given the complexity of the process for the nomination of the IPCC Bureau, obtaining a metric with an almost perfect discriminating capacity is quite a computational feat. By itself, however, this achievement is not particularly useful, as being able to predict Bureau candidates has little use for IPCC actors and little interest for IPCC scholars. To achieve sociological relevance, we still have to demonstrate how our prediction machine can be repurposed into a similarity sampling protocol capable of identifying individuals with special leadership profiles.

Figure 4 shows a clear separation between candidates and non-candidates, but it also shows that some non-candidates have a position that is relatively close to that of Bureau nominees (we highlighted them as orange dots). These are the informal leaders that we should add to the formal ones to complete our sample of IPCC leaders. To identify them, we focus on the current report, AR6. Since our dataset contained 67 candidates for the Bureau of AR6, we decided to use our method to double this sample and identify the 67 other individuals who have not been candidates for AR6 but are rankest the highest for leadership score in that AR. 16 of these informal leaders have been members of the Bureau in one or more of the earlier ARs. We will, therefore, not consider them because they could have added easily to the sample because of their previous role. 51 individuals, however, have never been in the Bureau so their inclusion in the sample is not trivial. Their names are listed in the table below. Our analysis stops at this list. However, since the very objective of our protocol is to build an extended sample of organisational leaders, this result is where a sociological investigation of IPCC leadership could begin.

Conclusions

In this paper we presented an experiment with machine learning in which we purposely put ourselves in a situation that, while decidedly more difficult than most AI experiments, is closest the actual research conditions of the social sciences: we measured ourselves against an elusive phenomenon (IPCC informal leadership) and worked with small data and limited computational resources. The results show that, even with these constraints, it is possible to train an efficient model. By using a combination of autoencoders and isolation forests, we developed a surprisingly precise measure of leadership potential. If we had tried to guess the 67 nominees of AR6 Bureau by taking the 67 individuals ranking the highest according to our isolation MeanLength, we would have predicted 60 of them (quite a stunning result, considering that our model was choosing among 5.559 different individuals).

As social scientists, however, we could not content ourselves with such a result, for prediction is rarely a goal in our disciplines and certainly not as the IPCC leadership is concerned. To make our protocol sociologically relevant, we then repurpose our model shifting its use from prediction to sampling. In this new use, we are not trying to guess which individuals would be nominated for the IPCC Bureau, but we use our knowledge about these nominations to find other individuals with a similar profile and thus device a protocol for sampling people with a leadership profile. Yes, our "leadership-score" metric is a black box because it is based on an opaque machine learning algorithm that does not provide us with the exact recipe employed to compute it. Yet, opacity is not necessarily a drawback. Because there is no clear or unique definition of what makes an IPCC leader, there is no way to develop a transparent algorithm for leadership sampling. Still, as we tried to show, this does not mean that it is impossible to devise such a sampling if the transparency condition is relinquished.

The situation in which we carried out our experiment, we believe, is not exceptional. It is common for social scientists to be interested in the subset of a population (e.g., the elite of an international organisation) that they identify by some prototypical examples (e.g., the official leaders of the same organisation) rather than by an exact definition. Because we were particularly interested in leaders, who, by definition, occupy an outlying position in their organisation, we relied on techniques for anomaly detection, but other techniques of machine learning can be employed in cases in which the sample to be extended does not have the same character of exceptionality.

Beyond the details of our protocol, the crucial point of our demonstration was to show that the black box nature of AI does not disqualify its use in social sciences but can be exploited as a workaround in situations in which the research is hindered by a lack of explicit definition. In these circumstances, it may be desirable to extend one sample by similarity, even if we are unable to characterise precisely the nature of such similarity. This is a situation in which machine learning can come in handy for its capacity to find resemblance in a multi-dimensional space.

  Figure below represent the contribution of each individual. An individual, for instance, is connected to "AR4-author-WG1" if during AR4 he/she has authored one of the chapters of the report of Working Group I. Edges are weighted according to the number of times that each individual has served in the same capacity (for example by authoring two or more chapters in the same AR and WG).

Figure 1 .

 1 Figure 1. Bipartite network of the IPCC contributors (dark grey) and the capacities in which they have served (coloured according to their functional and thematic division) in the first five assessment reports.

Figure 2 .

 2 Figure 2. Boxplots of the distribution of the values of the 13 features described above.

Figure 4 .

 4 Figure 4. Scatterplot of the average and of standard deviation for the MeanLength of the individuals of our corpus across the 10 implementations of our protocol.

Table 2 .

 2 Confusion matrix (for the test set) showing the results of the isolation forest with default scoring.

		actual "not-bureau"	actual "bureau"	total
	predicted "not-bureau"	2775 (true neg)	19 (false neg)	2794
	predicted "bureau"	123 (false pos)	18 (true pos)	141
	total	2898	37	2935

Table 3 .

 3 Confusion matrix (for the test set) of the isolation forest with customized scoring function.

		actual "not-bureau"	actual "bureau"	total
	predicted "not-bureau"	2683 (true neg)	13 (false neg)	2794
	predicted "bureau"	215 (false pos)	24 (true pos)	141
	total	2898	37	2935

Table 5 .

 5 Confusion matrix (test set) of the isolation forest with customized scoring and autoencoded features.

		actual "not-bureau"	actual "bureau"	total
	predicted "not-bureau"	2683 (true neg)	3 (false neg)	2794
	predicted "bureau"	215 (false pos)	34 (true pos)	141
	total	2898	37	2935

Table 7 .

 7 Informal leaders identified through our similarity sampling protocol.

	Benndorf, Rosemarie	Germany	269	3	389	0	0	389	3	0	0
	Spradley, Julian R.	USA	275	3	84	0	310	394	4	1	0
	Beetham, R.	UK	276	2	0	0	310	310	1	1	1
	Barrell, Susan Lesley	Australia	278	4	1271	0	0	1271	8	0	0
	Ivanov, Teodor	Bulgaria	279	3	878	0	0	878	13	0	0
	Harasawa, Hideo	Japan	281	6	648	0	1051	1699	7	7	2
	Oppeneau, Jean Claude	France	282	2	0	0	310	310	3	1	1
	Tilley, John	Australia	285	2	0	0	310	310	1	1	1
	Sathaye, Jayant	USA	288	5	209	143	0	352	0	12	3
	Yokobori, Keiichi	Japan	292	3	84	0	310	394	5	1	1
	Luxmoore, R.J.	USA	303	1	0	0	0	0	0	4	1
	Hulme, Mike	UK	308	2	0	350	0	350	0	8	2

https://www.ipcc.ch/site/assets/uploads/2018/09/ipcc-principles.pdf

https://archive.ipcc.ch/pdf/ipcc-principles/ipcc-principles-elections-rules.pdf

The database includes authors that have contributed to the IPCC Assessment Reports (AR), and not to the Special Reports (SP).

It is important to remark that, as far as the authors are concerned, our network only includes contributing lead authors, lead authors or review editors (i.e. the authors with roles of coordination and responsibility and who are directed selected by the IPCC Bureau). We have not considered contributing authors who are not selected by the Bureau (but invited by the selected authors), who generally do not participate in the meetings and whose contribution is in most cases limited to specific paragraphs or topics.

The main role division in the IPCC corresponds to the functional separation between the scientists that review the scientific literature on climate change and write the assessment reports (i.e. "the authors") and the diplomats who serve in the national delegations overseeing the work of the organisation (i.e. "the delegates"). The authors are then subdivided in three Working Groups with different thematic specialisations: WGI focuses on the physical bases of climate change; WGII on impacts, adaptation and vulnerability and WGIII on mitigation. We also include participation to the writing of the Synthesis Report, which brings together the conclusions of the three WGs. Finally, the work of the IPCC is temporarily articulated in Assessment Cycles each lasting several years.

The last four features are extracted from the official data or the World Bank (https://data.worldbank.org).


A context in literature

In the last decade, growing interest in 'literary methods' for spatial practice has opened a field of inquiry to explore creative writing's capacity to influence architectural research and design 1 . While this literary tendency has led to vibrant reconsiderations of how imaginative language participates in design processes, it has largely privileged the study of narrative strategies, and particularly the role of storytelling in both the understanding and making of place. If narrative brings certain qualities to spatial design, including an evocative sequential language and the semantic apparatuses intertwined with accounts of causality, the logic which undergirds these affordances also carries limits, for example on its capacity for non-sequential representation 2 . At the same time, poetic texts, or more specifically the writing practices that produce them, have been relatively underexamined, even within the purview of literary design methods. If the metaphor as poetic figure has begun to be studied in its role in architectural and urban projects 3 , and if there has been a recent rise in interest in the use of 'poetic acts' in the pedagogy developed at the Valparaíso School of Architecture in the late 1960s and early 1970s 4 , we still know relatively little about how poetic writing itself can contribute to the design process. What else enters into the dynamics of the design that might be absent without the introduction of poetic language, when we undertake writing within the spatial project? This is a question of how poetic practices and the linguistic materials they produce can offer new fields of action and agency for spatial practitioners. Looking beyond isolated poetic figures, how can writing in the lyric mode, for example, support the reading of a territory in its material-sensorial extension, but also as it manifests in the imaginary? How can Oulipian constraints and procedural poetics reveal characteristics or dynamics of a site that are potentially hidden from traditional architectural tools and their habitual modes of intention/attention? At a time when the spatial project, faced with environments of ever-increasing complexity, seeks means to avoid fixedness and rigidity, how can a poetic text give name to the uncertain, the ambient, and the ephemeral in the flux of a spatial figure?

In this paper, we explore the hypothesis that unique forms of situated knowledge 5 , complex and evocative in their tenor, arise from poetic practices undertaken in the design process, affording forms of 1 Dutch architect and professor Klaske Havik's 2014 work Urban Literacy: Reading and Writing Architecture has been a critical reference for a range of research and design activities with a focus on literary methods, notably in Europe. In the book, a publication adapted from her doctoral thesis, Havik specifically names and calls for "scriptive" methods for design, and that appeal has found echo in the research activities of her Chair of Methods of Analysis & Imagination at TU Deflt, and the EU Cost Action "Writing Urban Places". 2 Philosopher Galen Strawson has famously argued "Against Narrativity" (2004), in particular for its difficulty in adapting to episodic understandings of reality. 3 Cf. Gerber and Patterson (2013). 4 Cf. Andrade and Reyes Gil (2018). 5 Cf. Donna Haraway, 1988. critical spatial practice and spatial agency6 that are, as yet, underdeveloped. As part of the experimental work contributing to our practice-based doctoral research at the University of Glasgow, we set out to test this hypothesis in the context of the development of exploratory scenarios within an international research consortium, the SuMo Rhine research programme (Promotion of Sustainable Mobility in the Upper Rhine Region, INTERREG, 2018-2021), as part of the ENSAS/AMUP/IMM research team, directed by Andreea Grigorovschi7 . Within this interdisciplinary and intercultural consortium, bringing together architects and urbanists, data scientists, sociologists, geographers, and other specialists, we engaged in poetic practices to explore the sites of future mobilities through written territorial engagement, particularly in the peri-urban spaces of the metropolis, and on the verges of major infrastructure. Through the creation of a corpus of texts produced through various poetic practices -sitewriting, collage, erasure, concrete poetry, and lyric writing -we were confronted with nonrational elements in both the territory and in the spatial imagination which exceeded traditional discursive logics and gave material to design ideas in the development of our exploratory scenarios, not replacing traditional project tools, but accompanying and nourishing their work.

Writing to read the territory

Within the SuMo Rhine research programme, the ENSAS/AMUP/IMM team was set with the task of developing exploratory scenarios to test a new indicator-based evaluation tool in its application as a support for municipal-level decision making in the Upper Rhine Region encompassing the trinational area between France, Germany and Switzerland. The team developed contrasting sustainable mobility scenarios for 2030 Strasbourg. In developing these, several challenges confronted the research team, including the need to spatialize and/or materialize both theories and concepts, the need to produce a qualitative environment in which to test a quantitative evaluation tool, as well as the expansive complexity of the scenarios themselves, across both spatial and temporal scales, in addition to the increasing urgency around the need to define our specific forms of agency in a world where automated mapping and data-driven modeling tools are beginning to usurp the human prerogative in design.

While it is beyond the scope of this paper to detail the full extent of SuMo Rhine or the ENSAS/AMUP/IMM team's contribution to the larger research programme, it is within this context, which may at first seem inhospitable to literary methods in general, that we undertook poetic practices as a means to launch and nourish reflection on what would become spatialized scenarios. If scenario design is often conflated, however, with certain modes of storytelling, from the earliest phases of the project a familiar need to grasp the territory, to understand the metropolitan landscape and to name its character, preceded any occasion to narrate. How would our team seek to understand the spaces and places of Strasbourg, in order to later project possible futures across its territory?

Early experiments included site-writing visits led by the author and in which the architecturbanist team members were given a series of writing constraints to provoke new, non-disciplinary language to emerge. In the texts produced, participants found themselves noting subtle Doppler shifts in the conversations of approaching and departing cyclists, or how the motorway resembles a ribbon of asphalt, or even how the flow of traffic forms an endless spectacle for neighboring residents who sit on their balconies watching the vehicles pass like an eternal, metallic, and monotonous passeggiata.

Later, sessions of association-driven collage writing were created to help produce texts assembling remembered space, desired mobilities, and imagined landscapes. Three distinct prompts guided a phased writing session where the architect-urbanist participants recorded feelings about the current state of the Strasbourg metropolis, characterized a sample site alongside the A35 motorway, and provided memories of mobility spaces that left strong impressions. A second operation led to the cut-up and collage of the writing into new, oneiric texts [Fig. 1]. The often-somber results revealed a peri-urban landscape ossified by lines of infrastructure, perceived as abandoned, deserted space, despite its fringes of vegetation that can be read in the texts almost like leftover strips of ornamental piping. Still later, a range of poetic landscape transcriptions sought to find new ways of rendering perceived relationships between territorial objects, infrastructures, and materialities in textual forms (and which revealed a strong imprint from police and military infrastructure throughout the urban fabric). Concrete poetry, as well, gave form to perceptions of deleterious atmospheres, including a series of works titled "motors" which presented what might be considered the only existing territorial constant in the western region of contemporary Strasbourg: the ubiquitous presence of traffic noise [Fig. 2].

The practices described until this point can be characterized as poetic on multiple counts. First, on the page they work as examples of the poetic function of language, as described by the linguist Roman Jakobson, in that they call attention to themselves as language itself, with a "focus on the message for its own sake" 8 . More transparent means to describe a territory and its elements are available when efficiency of communication is desired. Poetic language, however, differing from speech or writing oriented toward the transmission of information, or vocative interpellation, for example, presents itself as language (not as pure information, not as appeal), with all the material qualities that are specific to language. When a writer-researcher names a territorial object as "a slow + massive spill," the resistance to our understanding it as "a river" calls attention to language in its radical opacity. Jakobson refers to this poetic function as "promoting the palpability of signs" 9 , which can be understood as an indication of the materiality that undergirds the poetic as such. In her 2014 volume, Poetics and Place: The Architecture of Sign, Subject and Site, Kristen Kreider draws on Jakobson and others to posit a material poetics, in contrast to "formalist linguistic theory wherein language is conceived solely in terms of an immaterial order of signs." 10 In Kreider's reading, the speech event -including or especially the written -becomes an embodied situation, one in which the reality effects of poetic language have material and thus spatio-temporal manifestations that are the grounds for experience and knowledge.

8 Jakobson (1960), p. 356. 9 ibid. 10 Kreider (2014), p. 20.

Figure 2 -'Motors' series of concrete poetry, produced on-site, by Jeremy Allan Hawkins (2021)

A second principal means by which the practices described above can be characterized as poetic is in the sense that they stand as exercises in poiesis, the act of creation at the very root, etymologically speaking, of poetry and poetics. In "What is the act of creation?", Giorgio Agamben asks, "What is poetry if not an operation in language that deactivates and renders inoperative its communicative and informative functions in order to open them to a new possible use?" 11 This aspect can also be considered in terms of Victor Shklovsky's ostranenie (estrangement) 12 or Brechtian Verfremdungseffekt (alienation or distancing effect) 13 , but by whichever frame we adopt, the poetic act of rupture in the standard relations of signification and communication offer a possibility to reposition oneself and, through new perspectives, create new language, new meaning, new understanding. This is perhaps a hidden sense of what the poet Ezra Pound meant by his famous maxim, "Make it new," regardless of its dubious translation history 14 .

What materiality and creation afford through poetic practices in a spatial design context is the ability to produce specific types of situated knowledge. When Donna Haraway calls for situated knowledges as an answer to the instrumentalizing power of idealist and transcendental objectivity 15 , she makes no mention of modernist planning modes, but does set forward a very ambitious claim to the making of modest ones: claims of what can we know here, where we can be held accountable for our behaviors. In employing poetic practices in a design setting, we are able to connect this ambition to the language of the spatial project. The embodied act of writing connects the text to a site, its material history, the people who live with/within it, and so what we can make known conforms to a very localized objectivity, and in forms that poetic texts favor, such as ambiguity, contradiction, uncertainty, and doubt. In the experimental writing undertaken in the SuMo Rhine research programme, for example, we learn that these architect-urbanists, confronted in the fabric of Strasbourg with the reality of major infrastructural objects, will refer to the motorway as a "cement frontier" or a "hard border", a "gash" in the landscape, an impermeable boundary-regardless of the many well-practiced and safe crossings or passageways which would make this territory, in theory, one of relatively high porosity. The temptation, of course, would be to generalize the conclusion, to make claims about perceptions of heavy infrastructure, notions of safety, and boundaries in the imaginary that impact crossing behavior. At the same time, a brief trip to Shanghai, or any number of other places where infrastructure inserts differently into the perceived and lived fabric of the city, would invalidate the generalization. Instead, we have highly specific knowledge, difficult to exchange or redistribute across other territories.

Describing a way towards spatial agency

As a form of conclusion, we would like to look at the lyric, and at its ramifications for spatial agency. While it is necessary to defer entering the lengthy and ongoing debate around the lyric as a category of literary activity, it is important to indicate here that by lyric we mean not a poetic genre but a mode of linguistic production, one that is not mimetic, and which produces its effects via other, nonrational logics than those we associate with reason 16 . Often associated with the brief expression of subjective emotion, the lyric -through effects of voice, effects of presence, and poietic (rather than 11 Agamben (2017), p. 55. 12 Berlina A. (2017). 13 Brecht B. (2017). 14 An entertaining and informative account of Pound's questionable rigor in translation can be found in Michael North's "The Making of 'Make It New'," published at Guernica in 2013: https://www.guernicamag.com/themaking-of-making-it-new/ 15 Haraway (1988). 16 For important overviews of the lyric mode, cf. Jackson V. and Prins Y. eds (2014), The Lyric Theory Reader: A Critical Anthology, Johns Hopkins Press, Baltimore, and Culler J. (2015), Theory of the Lyric, Harvard University Press, Cambridge. mimetic) production -does tend toward what we often call the subjective position, precisely in the sense that lyric is localizable in a form of consciousness that is produced in the text itself. But as we've suggested earlier, the divide between subjective and objective is not especially pertinent here. When Walt Whitman wrote, "Stand up, tall masts of Mannahatta! stand up, beautiful hills of Brooklyn! / Throb, baffled and curious brain! throw out questions and answers!"17 , rather than ask if the poem carries truth in some universal sense, it would be better to question when in time and where in space this particular language means, and how that meaning changes with time and space. We know, for example, that the sphere of influence over which the lyric mode holds sway may collapse down to the scale of a single person, with highly localized knowledge, here as much in the vocalization of a specific consciousness as in the naming of the borough of Manhattan.

In the language of the spatial project, poetic practices that produce lyric writing stand outwith the logic of the automated, the seemingly transparent, the optimal, the universalizing and thus violent, the purely market-driven, and the data-obsessed. The material and creative forces at work in these poetic practices are rather oriented towards the embodied utterance, entangled with its environment, creating knowledge on a highly local scale, whatever openings for transposition there may be or not. In the case of the SuMo Rhine research programme for sustainable mobility, a brief lyric text, written within the project activities, contributed to reflections that ultimately guided one scenario design towards a specific materialization of 'soft mobility':

where walls and roads converge the water describes a way through from before This short meditation on how local waterways managed to cut through, under and around apparent impasses in the urban fabric of Strasbourg came to feed a scenario concept for regaining the hydrographic network as a platform for low environmental impact mobility. In its first line, it offers a descriptive name for Strasbourg, before the second line where it gives the water network the capacity to communicate movements, and then carrying onto the third line where the water passes through, as does the knowledge of prior generations. In the ENSAS/AMUP/IMM team's scenarios, this is materialized in a spatialized network of river-born passenger transport.

In a certain sense, this is an example of spatial agency simply by merit of how the lyric writing -performed in-situ, in this case -allows the author to contribute to a scenario concept, to impact the design itself. The notion of agency here, however, is larger, in that the lyric mode as a choice of writing strategy is to choose non-rational means of taking part in the design process. While it does not preclude other rational and often strongly practical design tools, it does break with notions of mastery and/or optimization, let alone information and/or communication. In so doing, especially when practiced as an embodied and active form of refusal of normalizing imperatives, including enshrined scientific methods, at stake or at risk is not the individual or artistic genius, but rather the ability to bring the design back down to the scale of human consciousness, especially in its relations to the human body. Which we might posit, tentatively, as a critical competency to maintain or develop if we should ever aspire to spatial practices that take into account our relation to the other beings and forces that build this world with us.

Lyric

  Spatial Agency: Poetic Practices and Situated Knowledge in the Design Process Jeremy Allan Hawkins *Version submitted on 3 August 2021 for publication in the proceedings of the 2 nd International ICONA Conference 2021

Figure 1 -

 1 Figure 1 -Poetic text produced by collage techniques, by Andreea Grigorovschi and Jeremy Allan Hawkins, map background by Marie Fruiquière (2020).

In terms of the radical extension of the field of architecture beyond the act of building, we position ourselves in alignment with Jane Rendell's notion of critical spatial practice, Paola[START_REF] Viganò | Territories of Urbanism: The Project as Knowledge Producer[END_REF] claim for the project as knowledge producer, as well as with the notion of spatial agency developed byAwan, Schneider and Till (2011).

The research team works under the auspices of the Ecole National Supérieure d'Architecture de Strasbourg, AMUP / UR 7309 Unité de Recherche Commune, and the Franco-Chinese Chair of Innovative Metropolitan Mobilities. Additional information on the INTERREG programme is available on the project website: https://sumo-rhine.com/en/

Whitman, W., "Crossing Brooklyn Ferry", first published in

1865, and more recently inscribed in a waterfront guardrail in Brooklyn Bridge Park.


Linda Cappellato
email: cappellato@dei.unipd.it

Nicola Ferro
email: ferro@dei.unipd.it

Gareth J F Jones
email: gareth.jones@computing.dcu.ie

Jaap Kamps
email: kamps@uva.nl

Josiane Mothe
email: josiane.mothe@irit.fr

Karen Pinel-Sauvagnat

Eric San
email: eric.sanjuan@univ-avignon.fr

Jacques Savoy
email: jacques.savoy@unine.ch

Report on CLEF 2015: Experimental IR Meets Multilinguality, Multimodality, and Interaction

This is a report on the sixth edition of the Conference and Labs of the Evaluation Forum (CLEF 2015), held in early September 2015, in Toulouse, France. CLEF was a four day event combining a Conference and an Evaluation Forum. The focus of the Conference is "Experimental IR" as carried out in the CLEF Labs and other evaluation forums, it featured keynotes by Greg Greffenstette, Mounia Lalmas, and Doug Oard, and 43 papers have been presented, covering a wide range of topics. There were a total of eight Labs: eHealth, ImageCLEF, LifeCLEF, Living Labs for IR, NEWSREEL, PAN, QA, and Social Book Search, addressing a wide range of tasks, media, languages, and ways to go beyond standard test collections.

Introduction

The 2015 edition of the Conference and Labs of the Evaluation Forum1 (CLEF) was hosted by the Institut de Recherche en Informatique de Toulouse (IRIT), University of Toulouse, France, from 8-11 September 2015. For its sixth edition, the conference,was entitled "Experimental IR meets Multilinguality, Multimodality, and Interaction" and addressed issues around multilingual and multimodal information access, information interaction, as well as the evaluation of search systems.

CLEF was established over 15 years ago with a specific focus on stimulating research and innovation in multimodal and multilingual information access and retrieval. Over the years CLEF has fostered the creation of language resources in many European and non-European languages, promoted the growth of a vibrant and multidisciplinary research community, provided sizeable improvements in the performance of monolingual, bilingual, and multilingual information access systems [START_REF] Ferro | CLEF 15th birthday: What can we learn from ad hoc retrieval? In Information Access Evaluation[END_REF], and achieved a substantial scholarly impact [START_REF] Tsikrika | Assessing the scholarly impact of imageclef[END_REF][START_REF] Tsikrika | The scholarly impact of CLEF (2000-2009)[END_REF].

In its first 10 years, CLEF hosted a series of experimental labs that reported their results at an annual workshop held in conjunction with the European Conference on Digital Libraries (ECDL). In 2010, now a mature and well-respected evaluation forum, CLEF expanded to include a complementary peer-reviewed conference for discussion of advancing evaluation methodologies and reporting the evaluation of information access and retrieval systems regardless of data type, format, language, etc. Moreover, the scope of the evaluation labs was broadened, to comprise not only multilinguality but also multimodality in information access. Multimodality here is intended not only as the ability to deal with information coming in multiple media but also in different modalities, e.g. the Web, social media, news streams, specific domains and so on. Since 2010 the CLEF conference has established a format with keynotes, contributed papers, lab sessions, and poster sessions, including reports from other benchmarking initiatives from around the world. Since 2013, CLEF has been supported by an association, a lightweight not-for-profit legal entity, which thanks to the financial support of the CLEF community, takes care of the small central coordination needed to operate CLEF on an ongoing basis and makes it a self-sustaining activity [START_REF] Ferro | CLEF 15th birthday: Past, present, and future[END_REF].

CLEF 2015 was attended by 181 participants, out of which 72 (40%) were students, denoting a young and vibrant community, from different academic institutions and industrial organisations. Although the majority (140) of the participants came from 24 different European countries, there was also considerable interest in CLEF worldwide, with 17 participants from the Americas, 19 from Asia, and 5 from Africa.

The CLEF Conference

CLEF 2015 redefined the focus of the CLEF conference to be on "experimental IR" as carried out at evaluation forums (CLEF Labs, TREC, NTCIR, FIRE, MediaEval, RomIP, TAC, . . . ) with special attention to the challenges of multimodality, multilinguality, and interactive search. We invited submissions on significant new insights demonstrated on the resulting IR test collections, on analysis of IR test collections and evaluation measures, as well as on concrete proposals to push the boundaries of the Cranfield/TREC/CLEF paradigm [START_REF] Mothe | Experimental IR Meets Multilinguality, Multimodality, and Interaction -6th International Conference of the CLEF Association, CLEF 2015[END_REF].

Keynotes Three eminent scholars in the field headlined each day of the conference, addressing different areas of evaluation and information retrieval. Gregory Grefenstette (INRIA Saclay, France) talked about personal information systems and personal semantics. Mounia Lalmas (Yahoo Labs, London, UK) addressed the topic of user engagement evaluation. Douglas W. Oard (University of Maryland, USA) examined issues in privacy and ethics when searching among secrets.

Technical program

The clear focus on experimental IR helped the CLEF conference grow in terms of the submissions, see the details in Table 1, making the labs and conference parts of CLEF equal in size. CLEF 2015 received a total of 68 submissions, a dramatic increase over previous years. Each submission was reviewed by at least three PC members, and the two program chairs oversaw the reviewing and often extensive follow-up discussion. A novel feature of the CLEF 2015 conference was to invite CLEF 2014 lab organizers to nominate a "best of the labs" paper that was reviewed as a full paper submission to the CLEF 2015 conference according to the same review criteria and PC. This resulted in 8 full papers accepted corresponding to each to the CLEF 2014 labs. We received 24 regular full paper submissions, of which 8 (33%) full papers were accepted for regular oral presentation, and additional 7 full paper submissions (29%, making a total of 63%) was accepted with short oral and a poster presentation. We received 36 short paper submissions, and accepted 20 (55%).

Awards CLEF 2015 introduced three kinds of awards for the papers presented at the conference: a Best Short Paper Award, a Best Paper Award, a Best Student Paper Award (given only in case the best paper award has no student main author).

The Best Short Paper Award went to Wakeling and Clough [START_REF] Wakeling | Integrating mixed-methods for evaluating information access systems[END_REF], "Integrating Mixed-Methods for Evaluating Information Access Systems." According to the jury, "I found the paper to be fascinating, extremely well written and possibly one of those papers that leads to much future work. I have no real suggestions, other than I hope that the results from this work will be presented in the future. It is a perfect paper for CLEF." There were honorable mentions for Abnar et al. [START_REF] Abnar | Meta text aligner: Text alignment based on predicted plagiarism relation[END_REF], "Meta Text Aligner: Text Alignment Based on Predicted Plagiarism Relation," and Shen and Nie [START_REF] Shen | Is concept mapping useful for biomedical information retrieval?[END_REF], "Is Concept Mapping useful for Biomedical Information Retrieval?''

The Best Paper Award went to Ermakova [START_REF] Ermakova | A method for short message contextualization: Experiments at CLEF/INEX[END_REF], "A Method for Short Message Contextualization: Experiments at CLEF/INEX." The jury commented: "This paper makes a substantial contribution to the research of complex evaluation procedures combining NLP and IR. I will read this very carefully and cite this in future work." There were honarable mentions for four more papers: Budíková et al. [START_REF] Budíková | Search-based image annotation: Extracting semantics from similar images[END_REF], "Search-based Image Annotation: Extracting Semantics from Similar Images;" Grotov et al. [START_REF] Grotov | A comparative study of click models for web search[END_REF], "A Comparative Study of Click Models for Web Search;" McMinn and Jose [START_REF] Mcminn | Real-time entity-based event detection for twitter[END_REF], "Real-Time Entity-Based Event Detection for Twitter;" and Xu et al. [START_REF] Xu | Question answering via phrasal semantic parsing[END_REF], "Answering Natural Language Questions via Phrasal Semantic Parsing." As the best paper award was won by a student author, no separate Best Student Paper Award was handed out.

Social Program A rich social program was organised in conjunction with the conference, starting with a welcome reception with local food and wine specialities, continuing with the city hall reception, local band "La mal Coiffée" (bad hair) just before the social dinner in a famous organic restaurant named "Saveur Bio", and master classes in (1) Traditional polyphonic singing with Bastien Zaoui from the famous Vox Bigerri group and (2) Wine and Food Pairing with Yves Cinotti.

The CLEF Lab Sessions

Eight laboratories were selected and run during CLEF 2015. To identify the best proposals, besides well-established criteria from previous years' editions of CLEF such as topical relevance, novelty, potential impact on future world affairs, likely number of participants, and the quality of the organizing consortium. This year we further stressed the connection to real-life usage scenarios and we tried to avoid as much as possible overlaps among labs in order to promote synergies and integration. This was possible also thanks to a new activity introduced two years ago and supported by the ELIAS network5 : a lab organizers and proposers meeting, co-located with the European Conference on Information Retrieval (ECIR) and held in Vienna on March 31st 2015, in order to improve the synergies and interactions among different labs and to set the stage for next year proposals.

The Labs at CLEF 2015 [START_REF] Cappellato | Working Notes of CLEF 2015 -Conference and Labs of the Evaluation forum[END_REF], building on previous experience, demonstrate the maturity of the CLEF evaluation environment via the incorporation of new tasks, new and larger data sets, new ways of evaluation or more languages. Details of the individual Labs are described by the Lab organizers in these proceedings, here we just provide brief comment on each one.

CLEF eHealth -ShARe/CLEF eHealth Evaluation Lab CLEFeHealth explores scenarios which aim to ease patients and nurses understanding and accessing of eHealth information. The goals of the lab are to develop processing methods and resources in a multilingual setting to enrich difficult-to-understand eHealth texts, and provide valuable documentation. The lab contains two tasks: Information Extraction from Clinical Data (with two subtasks: Clinical speech recognition and Named entity recognition from clinical narratives in European languages) and User-centered Health Information Retrieval (with two subtasks: Monolingual IR and Multilingual IR).

ImageCLEF ImageCLEF aims at providing benchmarks for the challenging task of image annotation for a wide range of source images and annotation objective. The tasks address different aspects of the annotation problem and are aimed at supporting and promoting the cutting-edge research addressing the key challenges in the field, such as multi-modal image annotation, domain adaptation and ontology driven image annotation. The Lab tasks are: Robot Vision, Scalable concept Image Annotation, Liver CT Annotation and Domain Adaptation.

LifeCLEF The LifeCLEF lab continues image-based plant identification task which has originally run within ImageCLEF since 2011, with the same tasks of last year (BirdCLEF, PlantCLEF and FishCLEF). However, the LifeCLEF tasks radically enlarges the evaluated challenge towards multimodal data by (i) considering birds and fish in addition to plants, (ii) considering audio and video content in addition to images, (iii) scaling-up the evaluation data to hundreds of thousands of life media records and thousands of living species.

LL4IR -Living Labs for IR CLEF 2015 sees the first edition of this new lab, which features one task (Product search and web search). The main goal of the Lab is to provide a benchmarking platform for researchers to evaluate their ranking systems in a live setting with real users in their natural task environments. The lab acts as a proxy between commercial organizations (live environments) and lab participants (experimental systems), facilitates data exchange, and makes comparison between the participating systems.

NEWSREEL -News Recommendation Evaluation

CLEF 2016 and Beyond

More information on the CLEF 2015 conference, the CLEF initiative and the CLEF Association is provided on the Web: The call for papers for the CLEF 2016 Conference will be released around November 2015, and the expected deadline for the submission of papers will be around April 2016.

•
As far as labs are concerned, CLEF 2016 will run seven evaluation activities:

CLEF eHealth Goals of the lab are to develop processing methods and resources (e.g., dictionaries, abbreviation mappings, and data with model solutions for method development and evaluation) in a multilingual setting to enrich difficult-to-understand eHealth texts and provide personalised reliable access to medical information. In addition, CLEF eHealth will provide valuable documentation; develop an evaluation setting and release evaluation results for these methods and resources. The lab will contribute to the participants and organisers' professional networks and interaction with all interdisciplinary actors of the ecosystem for producing, processing, and consuming eHealth information.

ImageCLEF Following the ImageCLEF 2003-2015 evaluation campaigns, the ImageCLEF 2016 lab aims to provide an evaluation forum for the language independent annotation and retrieval of images. Motivated by the need to support potentially multilingual users from a global community accessing the ever growing body of visual information, the main goal of ImageCLEF is to support the advancement of the field of visual media analysis, indexing, classification, and retrieval, by developing the necessary infrastructure for the evaluation of visual systems operating in monolingual, language-independent and multimodal contexts, providing reusable resources for benchmarking. To meet its objectives, ImageCLEF organises tasks that benchmark the annotation and retrieval of diverse images such as general photographic and medical images, a domain with a particularly active community.

LifeCLEF This lab aims at evaluating multimedia analysis and retrieval techniques on biodiversity data for species identification. It will continue the three tasks offered in the previous year, namely BirdCLEF, PlantCLEF and FishCLEF, but scaling-up the evaluation data to hundreds of thousands of life media records and thousands of living species.

Living Lab for IR (LL4IR) The main goal is to provide a benchmarking platform for researchers to evaluate their ranking systems in a live setting with real users in their natural task environment. This second edition of the lab focuses on three use-cases and one specific notion of what a living lab is, with a view to expanding to other use-cases and other interpretations of living labs in subsequent years. Use-cases for the second lab are: product search (through REGIO Jatek, an e-commerce site), web search (through Seznam, a large scale commercial web search engine), and academic search (through SSOAR a Social Science Open Access Repository). The first two use-cases are a continuation of last year (see Section 6 for details), the last use-case is new this year.

News Recommendation Evaluation Lab (NEWSREEL) In the third iteration of the lab, we provide two tasks that address the challenge of real-time news recommendation. Participants could: a) Benchmark News Recommendations in a Living Lab, i.e., researchers gain access to the resources of a company to evaluate different recommendation techniques using A/B testing; b) Benchmark News Recommendations in a simulated environment, where we will provide a large data set comprising interactions between users and various news portals in a two-month time span. Participants in this task have to predict users' clicks on recommended news articles in simulated realtime.

Uncovering Plagiarism, Authorship and Social Software Misuse (PAN) This lab will focus on authorship tasks from the fields of author identification, author profiling, and author obfuscation. More specifically, the tasks will include two variants per field, namely author clustering and linking, age and gender prediction, and author masking and imitation.

Social Book Search (SBS) Real-world information needs are generally complex, yet almost all research focuses instead on either relatively simple search based on queries or recommendation based on profiles. The goal of the Social Book Search Lab is to investigate techniques to support users in complex book search tasks that involve more than just a query and results list. Tasks: Suggestion Task, Interactive Task, and Mining Task.

and two workshops/lab incubators:

CLEF Cultural Microblog Contextualization (CMC) The CMC task aims at focusing on different ways to gather, organize, and deliver relevant social data related to events that generate large numbers of microblog messages. Such structuration for gathering broader views of the social impact of an event. The goal of this CLEF Cultural Microblog Lab is to develop processing methods and resources to mine the social media sphere surrounding cultural events such as festivals. Tweets linked to an event makes a dense, rich but very noisy corpus: informal language, out of the language phrases and symbols, hashtags, hyperlinks The information is also often imprecise, duplicate, or non-informative. The interest of mining such data is to extract relevant, and informative content, as well as to potentially discover new information.

Multi-Perspective Health Information Search (MPHIS) Term "Multi-Perspective Health Information Search" (MPHIS) denotes queries for which there is "No Single Best Correct Answer". Instead multiple and diverse perspectives/points of view (which very often are contradictory in nature) are available on the web regarding the queried information. The MPHIS workshop investigates complex health information search in scenarios where users search for health information with more than just a single correct answer, and look for multiple perspectives from diverse sources both from medical research and from real world patient narratives. The goal of MPHIS is to research and develop techniques to support users in complex multi-perspective health information queries.

More information about CLEF 2016 are available on its homepage at: The level of sponsorship allowed us to offer 20 grants for students in addition to a free registration for the 25 volunteers including 11 further students.

Last but not least without the important and tireless effort of the enthusiastic and creative authors, the organizers of the selected labs, the colleagues and friends involved in running them, and the participants who contribute their time to making the labs and the conference a success, as well as financially supporting them through the CLEF Association, CLEF would not be possible.

Thank you all very much!

  CLEF 2015: http://clef2015.clef-initiative.eu/ • CLEF initiative: http://www.clef-initiative.eu/ • CLEF Association: http://www.clef-initiative.eu/association CLEF 2016 6 will be hosted by the Computer Science Department of the School of Sciences and Technology of the University of Évora, Portugal, 5-8 September 2016.

  http://clef2016.clef-initiative.eu/ CLEF 2017 will be jointly hosted by Dublin City University and Trinity College Dublin, Ireland, in early September 2017. Finally, bids for hosting CLEF 2018 are now open and will close around April 2016. Proposals can be sent to the CLEF Steering Committee Chair at chair@clef-initiative.eu. Paul Sabatier, Institut de Recherche en Informatique de Toulouse UMR 5505 CNRS, INFORSID, Université Toulouse Capitole, EGC, and ARIA.

Table 1 :

 1 Statistics of the technical program

	Category	Submitted Accepted (%)
	Best of the Labs	8	8 100%
	Full papers (12p)	24	8 33%
	+ short presentation		7 29%
	Short papers (6p)	36	20 55%
	Total	68	43 63%
	Other Evaluation Initiatives The conference hosted also a panel session on other evaluation
	initiatives, which informed delegates of current and upcoming activities within evaluation initia-
	tives worldwide. The panel session included Ian Soboroff from the National Institute of Standards
	and Technology (USA) who presented TREC 2 (Text REtrieval Conference), the first large-scale
	evaluation activity organised in the field of IR, which began in 1992. Noriko Kando from the
	National Institute of Informatics (Japan) presented NTCIR 3 (NII Testsbeds and Community for
	Information access Research), which promotes research in information access technologies with a
	special focus on East Asian languages and English. Gareth Jones from Dublin City University
	(Ireland) presented MediaEval 4 , which is dedicated to evaluating new algorithms for multimedia
	access and retrieval.

  PAN Lab on Uncovering Plagiarism, Authorship, and Social Software Misuse This is the 12th edition of the PAN lab on evaluation of uncovering plagiarism, authorship, and social software misuse. PAN offers three tasks at CLEF 2015 with new evaluation resources consisting of large-scale corpora, performance measures, and web services that allow for meaningful evaluations. The main goal is to provide for sustainable and reproducible evaluations, to get a clear view of the capabilities of state-of-the-art algorithms. The tasks are: Plagiarism Detection, Author Identification, Author Profiling.QA Track -CLEF Question Answering Track In the current general scenario for the CLEF QA Track, the starting point is always a Natural Language question. However, answering some questions may need to query Linked Data (especially if aggregations or logical inferences are required); whereas some questions may need textual inferences and querying freetext. Answering some queries may need both. The tasks are: QALD (Question Answering over Linked Data), Entrance Exams (Questions from reading tests), BioASQ (Large-Scale Biomedical Semantic Indexing), and BioASQ (Biomedical Question answering).SBS -SocialBook Search The Social Book Search Lab was previously part of the INEX evaluation benchmark (since 2007). Real-world information needs are generally complex, yet almost all research focuses instead on either relatively simple search based on queries or recommendation based on profiles. The goal of the Social Book Search Lab is to investigate techniques to support users in complex book search tasks that involve more than just a query and results list. SBS runs two tasks: Suggestion Track and Interactive Track.

Lab CLEF 2015 is the second iteration of this lab. Participants can: a) develop news recommendation algorithms and b) have them tested by millions of users over the period of a few weeks in a living lab. NEWSREEL provides two tasks designed to address the challenge of real-time news recommendation: Benchmark News Recommendations in a Living Lab and Benchmarking News Recommendations in a Simulated Environment.

http://clef2015.clef-initiative.eu/

http://trec.nist.gov/

http://research.nii.ac.jp/ntcir/

http://www.multimediaeval.org/

http://www.elias-network.eu/

http://clef2016.clef-initiative.eu/

ACM SIGIR Forum

Vol. 49 No. 2 December 2015

Acknowledgments

The success of CLEF 2015 would not have been possible without the huge effort of several people and organizations, including the CLEF Association 7 , the Program Committee, the Lab Organizing Committee, the local organization committee in Toulouse, the reviewers, and the many students and volunteers who contributed along the way.

We would like to acknowledge the Institut de Recherche en Informatique de Toulouse UMR 5505 CNRS and its director, Prof. Michel Daydé for the support we got, first for bidding to host the conference, then to organize it. We also received the support from the following universities and schools: Ecole Supérieure du Professorat et de l'Education, Université Toulouse Jean Jaurés, Université Paul Sabatier, Université du Capitole.

We also gratefully acknowledge the support we received from our sponsors. The ESF Research Networking Programme ELIA 8 , the ACM SIGIR 9 , the Université Toulouse-Jean Jaurés 10 , and the Région Midi-Pyrénées 11 for their strong financial support; but also: Springer 12 , the Université