Y Kawahara

M De La Bastide

J P Hamilton

H Kanamori

W R Mccombie

S Ouyang

D C Schwartz

T Tanaka

J Wu

S Zhou

K L Childs

R M Davidson

H Lin

L Quesada- Ocampo

B Vaillancourt

K F X Mayer

R Waugh

P Langridge

T J Close

R P Wise

A Graner

T Matsumoto

K Sato

A Schulman

G J Muehlbauer

N Stein

R Ariyadasa

D Schulte

N Poursarebani

R Zhou

B Steuernagel

M Mascher

U Scholz

B Shi

K Madishetty

J T Svensson

P Bhat

M Moscou

J Resnik

P Hedley

H Liu

J Morris

Z Frenkel

A Korol

H Bergès

B Steuer- Nagel

S Taudien

M Felder

M Groth

M Platzer

A Himmelbach

S Lonardi

D Duma

M Alpert

F Cordero

M Beccuti

G Ciardo

Y Ma

S Wanamaker

F Cattonaro

V Vendramin

S Scalabrin

S Radovic

R Wing

M Morgante

T Nussbaumer

H Gundlach

M Martis

J Poland

M Spannagl

M Pfeifer

C Moisy

J Tanskanen

A Zuccolo

J Russell

A Druka

D Marshall

M Bayer

D Swarbreck

D Sampath

S Ayling

M Febrer

M Caccamo

S Wannamaker

T Schmutzer

J W S Brown

G B Fincher

P S Schnable

D Ware

R S Fulton

J C Stein

F Wei

S Pasternak

C Liang

J Zhang

L Fulton

T A Graves

P Minx

A D Reily

L Courtney

S S Kruchowski

C Tomlinson

C Strong

K Delehaunty

C Fronick

B Courtney

S M Rock

E Belter

F Du

K Kim

R M Abbott

M Cotton

A Levy

P Marchetto

K Ochoa

S M Jackson

B Gillam

W Chen

L Yan

J Higginbotham

M Cardenas

J Waligorski

E Applebaum

L Phelps

J Falcone

K Kanchi

T Thane

A Scimone

N Thane

J Henke

T Wang

J Ruppert

N Shah

K Rotter

J Hodges

E Ingenthron

M Cordes

S Kohlberg

J Sgro

B Delgado

K Mead

A Chinwalla

S Leonard

K Crouse

K Collura

D Kudrna

J Currie

R He

A Angelova

S Rajasekar

T Mueller

R Lomeli

G Scara

A Ko

K Delaney

M Wissotski

G Lopez

D Campos

M Braidotti

E Ashley

W Golser

H Kim

S Lee

J Lin

Z Dujmic

W Kim

J Talag

C Fan

A Sebastian

M Kramer

L Spiegel

L Nascimento

T Zutavern

B Miller

C Ambroise

S Muller

W Spooner

A Narechania

L Ren

S Wei

S Kumari

B Faga

M J Levy

L Mcmahan

P Van Buren

M W Vaughn

K Ying

Etienne David
email: etienne.david@inrae.fr

Simon Madec

Pouria Sadeghi-Tehran

Helge Aasen

Bangyou Zheng

Shouyang Liu

Norbert Kirchgessner

Goro Ishikawa

Koichi Nagasawa

Minhajul A Badhon

Curtis Pozniak

Benoit De Solan

Andreas Hund

Scott C Chapman

Frédéric Baret

Ian Stavness
email: ian.stavness@usask.ca

Wei Guo
email: guowei@isas.a.u-tokyo.ac.jp

Global Wheat Head Detection (GWHD) Dataset: A Large and Diverse Dataset of High-Resolution RGB-Labelled Images to Develop and Benchmark Wheat Head Detection Methods

Keywords: Plant counting, Deep-learning, Robustness, unmanned aerial vehicle, phenotyping

A PhD is a transformative journey that was completely different from what I expected. Far from being a solitary experience, it was, in my case, the result of a collaborative effort.

My first thanks go to Fred, my thesis director, and Benoit, my supervisor at Arvalis. Fred, working with you was sometimes far from easy, but fun. I learned from you to be a creative and curious researcher but also rigorous, demanding and hardworking. Always with a smile and good humour, of course! The freedom of research I have had with you is a great gift. Benoit, you have been much more than a mentor. Your sincere benevolence helped me to overcome the moments of doubt during the thesis, and, thanks to your teaching, wheat and maize no longer hold any secrets

Abstract Le défi des estimations robustes de trait par Deep Learning à partird'imagerie haute résolution RVB

Le phénotypage à haut débit des plantes, notamment dans le cadre d'acquisitions en plein champ, repose sur l'interprétation de données issues de différents capteurs mis en oeuvre sur des vecteurs variés tels que des tracteurs, des robots ou des drones. Initialement, ces données étaient interprétées à l'aide d'algorithmes de télédétection exploitant la résolution spectrale du signal. Mais depuis 2015, les progrès du "Deep Learning", basé sur l'entrainement à partir d'exemples, ont permis des résultats prometteurs pour mesurer des traits essentiels comme le taux de couverture ou le comptage de plantes ou d'organes. Ces algorithmes utilisent des couches de convolution apprises, permettant de tirer parti de l'organisation spatiale du signal. L'avantage de ces méthodes est qu'elles sont basées sur des capteurs Rouge-Vert-Bleu (RVB), qui sont beaucoup moins coûteux que les imageurs multi-ou hyperspectraux. Cependant, les algorithmes de Deep Learning sont sensibles aux changements de la distribution entre les données utilisées pour l'entrainement et les données prédites. En pratique, des erreurs de prédiction variables et non prédictibles d'un site à l'autre peuvent être observées. L'objectif de la thèse est de comprendre les causes de ces variations et de proposer des solutions pour des estimations de traits phénotypiques fiables en utilisant le Deep Learning. L'étude porte sur la détection de plantes et d'organes à partir d'images RVB haute résolution acquises sur le terrain. Nos travaux ont d'abord porté sur la constitution de bases de données d'images diversifiées provenant de différents lieux et stades de développement pour l'émergence de plantes (maïs, betterave, tournesol) et les épis de blé, ce qui a permis la publication de deux bases de données annotées, regroupant 27 sessions d'acquisition pour le drone et 47 pour la détection d'épis. Ces jeux de données démontrent la différence de performances entre les résultats publiés et les nôtres en raison du changement de distribution. Pour dépasser les limites des méthodes habituelles, nous avons organisé deux concours de données, les Global Wheat Challenges, en 2020 et 2021, qui nous ont permis d'obtenir des solutions entraînées pour la robustesse sur un jeu de données différent de celui de l'entraînement. L'analyse des solutions a montré l'importance des stratégies d'entraînement pour la robustesse audelà des architectures utilisées. Nous avons également montré que ces solutions peuvent être déployées efficacement en remplacement du comptage manuel. Enfin, nous avons démontré l'inefficacité des fonctions d'entraînement conçues pour l'entraînement robuste. Notre travail ouvre la perspective d'une meilleure évaluation du Deep Learning dans le contexte du phénotypage à haut débit et donc de la confiance dans son utilisation en conditions réelles.

The challenge of robust trait estimates with DeepLearning on high resolution RGB images

High throughput plant phenotyping, especially in the context of open field acquisitions, relies on the interpretation of data from different sensors implemented on various vectors such as tractors, robots or drones. Initially, these data were interpreted using remote sensing algorithms that exploit the spatial resolution of the signal. Since 2015, however, progresses of "Deep Learning", based on the training on examples, has already obtained promising results for measuring the rate of cover, counting plants or organs. It uses learned convolution layers, can take advantage of the spatial organization of the signal. The advantage of these methods is that they are based on Red-Green-Blue (RGB) sensors, which are much less expensive than multi-or hyperspectral imagers. However, these methods are sensitive to changes in the distribution between the data used in training and the predicted data. In practice, variable prediction errors from site to site can be observed using these methods.

The objective of the thesis is to understand the causes of these variations and propose solutions for reliable phenotypic trait estimates using Deep Learning. The study focuses on detecting plants and organs from high-resolution RGB images acquired in the field. Our work first focused on the constitution of diversified image databases from different locations and stages of development for plant emergence (maize, beet, sunflower) and wheat ears, which allowed the publication of two annotated databases, grouping 27 acquisition sessions for the drone and 47 for the ear detection. The datasets demonstrate the performances difference between the published results and ours due to the change in distribution. To go beyond the limits of the usual methods, we organized two data competitions, the Global Wheat Challenges, in 2020 and 2021, which allowed us to obtain solutions trained for robustness on a different data set than the training one. The analysis of the solutions showed the importance of the training strategies for robustness beyond the architectures used. We have also shown that these solutions can be effectively deployed as a replacement for manual counting. Finally, we have demonstrated the inefficiency of training functions designed for robust training. Our work opens the prospect of a better evaluation of Deep Learning in the context of high-throughput phenotyping and thus of confidence in its use in real-life conditions.

List of Tables

Introduction

1.1 Agriculture must adapt shortly to face the future challenges 1.1.1 Climatic, environmental, and anthropic constraints on the food system.

Our food system results from the interactions between food production and the consumers [1]. It is challenged by its strong impact on the environment, including global warming: some of the hottest years (2018,2019,2020) recorded in the world (6 th , 3 rd and 1 st ) [2] and in France (2 nd , 4 th and 1 st ) [3], happened during the writing of this thesis. Agriculture can also mitigate climate change by sequestering carbon in the soil [4]. Despite the numerous political summits, there is no indication that the trends will reverse in the future, and the average yield of major crops will reduce by 3% to 6.0% for a one degree Celsius increase [5], [6]. The expected increase of +4°C for 2100 is associated with a decrease of 22% for wheat, 12.1% for rice, 26.4% for maize and 1.8% for soybean. Further, extreme climatic events that can destroy all crops at a country scale are expected to become more frequent. Our food system faces additional challenges: the global population is expected to jump from 7,8 billion people in 2020 to 11 billion in 2100. It requires an increase of 40% of the land area to be cultivated at constant productivity, although cropland represents already 38% of the land surface according to FAO [7]. The diet quality needs to evolve: 39% of adults aged over 18 years old were considered as overweighed in 2016 according to WHO [8]. This phenomenon is described as a "nutrition transition" [9] resulting from the larger quantity of food accessible to the population with more carbohydrates, fats and oils, and combined with a lack of activity. The quality of the diet needs to be addressed for all in the future. At the same time, hunger is still a main issue [10] with still 690 million humans (8.9% of the global population) suffering from too limited access to food in 2016 [11].

At the same time, agriculture should reduce its environmental footprint with less use of pesticides, herbicides or artificial fertilizers, not only in developed countries but also developing ones [12]. Current industrial agriculture practices harm biodiversity while agriculture can adopt new practices emphasizing the role of biodiversity [13]. The extensive use of chemical inputs impacts also human health [14]. Further, the use of genetic modified crop (GMO) [15][16] [17] [18] is still questioned regarding possible negative impacts on human health and the environment. GMOs are largely used in North America, while most of them are regulated in the European Union.

To summarize, in the next decades, the global food system is required to produce more or to keep the current productivity, with better and more nutritional diets, sequestrate CO2 with a changing climate, less agricultural inputs.

Productivity, potential yield and genetic gain

Food production increases during the 20 th century were achieved mainly with a massive rise in crop productivity while the cropland area remained mostly stable [19]. Therefore, the production of food is primarily controlled by crop productivity, i.e. mass of harvested organs per unit area, most often called yield. However, the harvested organs must have sufficient quality to be transformed into food products: wheat requires a minimum protein content to make bread and pasta. Yield results from the accumulation of plant carbohydrates in the harvested organs generally corresponding to the reproductive organs or the root. Photosynthesis processes capture the incoming light to convert water and CO2 into O2 released in the atmosphere and carbohydrates accumulated in the plant (Figure 1). Evans defined the potential yield as "the yield of a cultivar when grown in environments to which it is adapted, with nutrients and water non�limiting and with pests, diseases, weeds, lodging, and other stresses effectively controlled" [20]. It corresponds to the maximum yield given a quantity of light, temperature for a shared genotype genetic. To keep the yield close to its potential requires avoiding any nutrients and water limitations. Nutrients such as nitrogen, phosphate, and potassium are essential to plant protein production, including those involved in photosynthesis. They can be found in the soil and supplemented by fertilization. Water is the primary source of protons for the Calvin cycle that converts photons into chemical energy (ADP). Water is also necessary to transport nutrients from the soil to the leaves. Diseases (bacterial, virus or fungi) may induce substantial yield loss when they develop in the field and often require using fungicides or bactericides when no natural alternatives can control diseases. Keeping crop yield high requires limiting the competition for light, water and nutrients from the weeds and limiting damages from insects and animals or extreme climatic events such as hail, heatwave or frost. Innovative farming strategies attempt to address all these issues to keep the yield close to its potential. However, the potential yield can also be increased by improving plant functions: a better efficiency of the photosynthesis, better root efficiency in capturing nutrients in the soil, a better leaf orientation to capture light or limit transpiration, high resistance to pests, an increased fraction (harvest index) of the plant dry mass into the harvested organs. For a given environment, most of these functions are governed by the plant genome. Gain of yield potentials with genetic improvement is called genetic gain.

Crop domestication and Scientific revolutions: a brief history of yield improvement science

Humans improved crop yield through several steps driven by key scientific breakthroughs. The genome of modern crops results from these processes, summarized in Figure 2. The first step was the crop domestication [21], which started at least 10.000 years ago for crops such as wheat [22] , maize [23] , rice [24], [25], or sorghum [26][27] [28]. This process happened independently at different places such as northeastern America, Mesoamerica, central mid-altitude Andes, West African sub-Sahara, east Sudanic Africa, Near East, northern China, Yangtze China. Domestication is a long process that consists of random crosses between races or close species. Domestication of wilds species is a spectacular example of plant-animal co-evolution: it is even considered that wheat has domesticated humans [29]. Surprisingly, this process usually requires few chromosomic changes despite spectacular changes in appearance and performance.

Figure 2. Milestones in crop breeding

The second stage started with the discovery and understanding of Mendel's Laws of genetic. The breakthrough has paved the way for more structured breeding that took into account the parents' characteristics and led to the development of a more formal science of plant breeding which started to be popular during the end of the 19 th century, using field trials to select the best performing crosses. The first company, Gartons Agricultural Plant Breeders, was the first to commercialize a new variety of plants thanks to cross-pollination.

From the start of scientific breeding to the end of the second world war, several innovations emerged: cross-breeding (crossing two different varieties), hybrid breeding, or mutation breeding. The development of statistics to handle the effect of the environment during trials was also a critical innovation, which gave birth to modern statistics with the work of Ronald Fisher, in charge of the analysis of the Rothamsted Station [30], near London. Varieties with better yield, disease resistance, and easier harvesting were created thanks to these scientific revolutions. Cross-breeding helped to introduce genes from one cultivar to another. Mutation breeding provokes mutation in plant cells to create new, possibly unusual characters. During this period, the harvest index was improved by reducing the height of stems thanks to cross-breeding. It allows more biomass to go in the grain, less lodging, and easier harvesting. The differences between wheat anterior to this period that is still taller and modern wheat cultivars are presented in Figure 3.

The "Green Revolution", which started in Mexico in the 60s and then spread to other places such as India and Pakistan, was based on creating new varieties of wheat, the most consumed crop, thanks to crop breeding. It introduces genetic gain with new cultivars based on a semi-dwarf gene and a rust-resistant gene. Rust was a significant disease that induced hunger on a large scale. The Green Revolution also introduced more dense sowing density combined with intensive use of fertilizer. Similar approaches were introduced for rice [31] and maize [32]. Despite being already a combination of several innovations, the Green Revolution faces limitations for the identified challenges that agriculture will face in the coming decades. The next revolution would require better exploitation of the ecological services to reduce the environmental impact and create resilient farming systems while producing healthier food.

A change in crop management and genetic improvement are required

With current technologies, reducing negative externalities, making our system more resilient and producing enough food is probably not possible. Some crops rely on the use of few key inputs that, if removed, call into question the system's economic sustainability. The recent sugar beet crisis in France, which took place in 2020, is a good example. Sugar beet is one of the most productive plants with a potential of 24T of sugar per hectare [33], to compare to bread wheat which has only a potential between 8-12T per hectare. It has been grown in France since the 19 th century as a replacement for sugar cane. This crop is sensitive to the Beet Chlorosis Virus and Beet Mild Yellowing Virus, that are transmitted by insect and cause a loss of chlorophyll, an essential pigment for the photosynthesis.

Neonicotinoids allowed to grow sugar beet without the viruses' presence during the last two decades, preventing spraying pesticides during the growing season. The pesticides coat the sugar beet seed and spread during the season. In comparison to former solutions, neonicotinoids seemed more reliable and less harmful to the environment. However, the molecule is highly remanent and accumulates in the soil, affecting other micro-organisms that the target pathogen insect. It has led to their ban in France in 2019. However, all alternatives were less effective and more harmful. No alternative practices such as organic farming provide a convicting alternative. Such limitations, called "technical dead end" since no proven solution providing similar productivity with less environmental impact is yet available.

Complexity will be fundamental to increase resilience, mitigate negative impacts while producing enough food. Such complexity will require many experiments to build sufficient knowledge. In the case of sugar beet, scientists envision that the alternative will be a combination of genetic and farm management to repel the aphid and attract him to other parts of the field, to have a different crop that limits the spread of the pathogen, have varieties which are resistant to the viruses. Strategies need to adapt since insects and diseases will probably find ways to avoid too simple solutions. For instance, the rust resistance that made the wheat cultivars successful for the Green Revolution was based on two genes. At the end of the '90s, new strains of rust have bypassed the defence mechanism, making the varieties less relevant. The pace of experiments for agriculture must increase to find more complex and dynamic strategies. Experiments have always been key in agriculture, but the dynamic nature of crops and the associated environment and the inherent difficulty in evaluating outputs make progress slow.

Towards genomic selection

Since the 2000s, a new era of crop breeding has emerged thanks to a better description and understanding of DNA. Instead of crossing two parents in the hope of adding few genes of interest into the genes pool of a high yield cultivar, the genome is precisely characterized with methods such as SNP (single-nucleotide polymorphism) or SSR (simple sequence repeat). A SNP is "a single base pair mutation at a specific locus" [34]. Genotyping is today cheap and accessible. Such innovations are now accessible thanks to the low cost of sequencing techniques that describe the genome : rice [35], barley [36], maize [37] , soybean ( [38] and more recently wheat [39]. Association between genome and phenotype can be measured with "Genomic Wide Association Study" (GWAS). The method finds "Quantitative Trait Locus", a specific region of a chromosome responsible for the variation of a phenotypic trait by observing the concurrent changes in SNP distribution and the phenome. Models that predict phenotypic traits forgiven a genome are developed. The more scientists can understand the genome's role, and the faster new varieties can be created. The study of transcriptomics is also of interest by understanding the expression of a gene through RNA sequencing. It has already led to applications such as development stage control by silencing or overexpressing some genes [40], [41]. As illustrated in Figure 4, the introduction of genomic variation can be done not only by selection (Conventional plant breeding) but also with induced mutation thanks to chemicals of radiation (plant mutation breeding). Foreign genes can directly be introduced into a host (Plant transgenic breeding), with bacteria such as Agrobacterium tumefaciens commonly used inbreeding. The introduction of faster, more precise techniques known as genome editing enable precise modifications of few bases. The most known strategy is the CRISPR/Cas9 [42] which allow determining specific targeting sequence for the new DNA addition. It is a challenging exercise as genes can have interactive effects: genes that induce a disease's resistance can trigger a physiological process that can lead to a loss of yield.

High-Throughput phenotyping is a key tool for this transition 1.2.1 The bottleneck of the digital and genomic revolution: the phenotyping problem for agricultural experiments.

Experimenting in agriculture is central to improving knowledge, evaluating management practices' efficiency by comparing modalities, evaluating new accessions and cultivars, and calibrating crop models. The field experiment is therefore usually positioned such as the soil is homogeneous, the area around the trial is protected with a large buffer zone, and all modalities or genotypes are tested on a small piece of land with a fixed number of rows and a fixed length. Each modality is often replicated on several elementary experimental units, which are called a "microplot". Conventional plant breeding takes 8 to 10 years. This process is done in several steps: first individual plants are grown in-door to get as many crosses as possible. Then, given a set of criteria, the best performing genotypes are kept and used for a new test with more space. Finally, candidate genotypes are tested in a set of different environments. This process is usually done by commercial companies, universities or non-profit organizations such as CIMMYT. In France, the quality of a genotype is also independently tested on a national network to provide independent feedback to farmers. At this end of this process, the selected genotype can be registered as a variety, or cultivar, if it demonstrates a significant improvement compared to a set of reference varieties. At each step of these experiments, many measurements are involved in assessing the performances. The exact process is applied to agronomical experiments to evaluate the efficiency of inputs, new cropping systems or other agricultural innovations.

Measuring the performances is still mostly done manually by field workers. Measurements can be either destructive or non-destructive. Destructive measurements are made by gathering the plants in a given sampling area and are used to quantify several traits, including the dimensions of the organs, their dry biomass, or the nitrogen content. Destructive measurements can be repeated only a few times during the season, considering the size of the microplot and sampling area required each time. Conversely, non-destructive measurements can be replicated multiple times during the season. It is used to measure the evolution of architectural traits such as canopy height or plant and organ counting. Visual notations are also used to evaluate some traits but are prone to variability among the different field workers. When relying on the field workers, another challenge arises around critical development stages: the number of activities is so high that not all needed measurements can be done, leading to a long day of work for field workers and incomplete data for agronomists. These methods were exploited during the Green Revolution for plant phenotyping. However, this currently constitutes a substantial limitation for the current genetic revolution based on highthroughput genotyping. Therefore, Plant phenotyping appears to be now the bottleneck of genetic improvement.

High throughput plant phenotyping (HTPP) is a recent field of research exploiting the most recent technological advances in vectors, sensors and interpretation algorithms to estimate traits characterizing crop architecture, physiology or disease symptoms. Additionally to its interest in plant breeding, the same technologies can be adapted to agronomical experiments and for farmers' decision-making. Before reviewing the current trends in HTPP, we first define few critical terms following the glossary proposed by [44].

• Genotype: The genotype of an organism is a set of inherited instructions carried within its genetic code. • Trait: A trait is a quantitative or qualitative characteristic of an individual resulting from the expression of its genome in a given environment. For plant phenotyping, it can be determined at the plant scale or at population scale. The term "phene" is also used.

• Variable: A variable is an estimate of the trait resulting from the application of an interpretation method on sensor measurements. • Phenotype: the phenotype is the set of observable traits of an organism. It covers morphology, development, cellular, biochemical, or physiological properties. • Phenome: the phenome is the set of all possible phenotypes for a plant or a population.

• Phenomics: The study of the phenome and its relationship with the genome, the transcripts, the proteins or metabolites. • Ideotype: A crop ideotype is an ideal collection of traits which optimize crop performance to a specific end-use in a particular environment and crop management. The role of HTTP in plant science is to develop automatic systems that measure traits. The phenotyping systems are made of three main components: a vector that can move the plants to the sensor or the sensor to the plants, a sensor that records a signal from the plant, and an interpretation algorithm that transforms the signal into a trait estimate. These systems are operated over experimental installations with a range of levels of control of the environmental conditions.

A phenotyping process can be defined as "high-throughput" if the acquisition for one microplot is a magnitude faster than a manual measurement [45]. However, the measurements' cost efficiency and accuracy need also to be accounted for when compared with traditional phenotyping methods. In the next section, different HTPP systems will be presented, the produced traits will be analyzed, and finally, the current interpretation algorithms will be discussed. While HTPP can be viewed as one engineering science, its multidisciplinary nature (robotics, mechatronics, statistics, computer vision, machine learning, artificial intelligence, modelling, agronomy, ecophysiology, botany) allows the emergence of "Plant Phenomics": a science that relates the phenome with the genome, the transcripts, the proteins, and the metabolites.

The several traits accessible from HTPP

The measurement of traits with HTPP produce variables: a discrete or continuous quantity that quantifies the traits that can then be used to compare different individuals. Traits availability and the precision of the associated variables depend both on sensors' technological development and interpretation methods maturity. Four categories of traits are proposed and summarized in Figure 5: • State traits which are measured at a specific time and are intrinsic properties of the crop/plant/organ • Dynamic traits which are based on repeated estimates of state traits over a specific period but do not need additional inputs except a time scale • Radiative traits which are measuring the flux reflected or emitted by the plant that will depend both on plant state and on the illumination and view conditions • Functional traits that describe some processes and that need additional information about the environmental conditions.

States traits

The states traits can be measured directly on the canopy and take exactly one value at a precise time. They can be categorized into three groups:

• Biophysical traits describe the structural and morphological characteristics of canopies, plants, or organs. The aerial parts of the plant that are more easily accessible are often described: at the canopy level focus is generally on Green Fraction, Green Area index, Leaf orientation, plant height, lodging, plant density, ear density. • Biochemical traits provide information on the plant biochemistry. For instance, the chlorophyll content of a leaf or a canopy is an important biochemical trait that determines the photosynthetic potential. • Sanitary traits provide information on the plant health. For example, the detection of disease symptoms, the determination of contaminated parts of the plant are sanitary traits that are often derived from visual scoring of the symptoms severity made by human experts.

Dynamic traits

Dynamic traits are based on repeated observations of state traits. Popular dynamic traits among breeders are the early vigour or the stay-green. The early vigour is the plant or canopy growth speed, while the stay green is the plant or senescent canopy rate. Specific characteristics of the plant architecture plasticity are evaluated dynamically, such as the leaf rolling. Phenological traits are also dynamic traits that are measured by detecting qualitative changes in plant morphology. In wheat, tillering, stem elongation and heading or flowering are evaluated by monitoring biophysical traits such as plant height or wheat head density.

Radiative traits

Radiative traits result from the interaction between light and canopies, plants or organs. They thus depend both on the view and illumination conditions and canopy, plants, or organ state. Canopy reflectance or organ radiance are low-level radiative traits commonly measured and then transformed into higher-level traits such as vegetation indices. The fraction of intercepted photosynthetically active radiation is one of the main radiative traits computed from canopy structure and the illumination conditions. Brightness temperature is also a low-level radiative trait that depends both on canopy state and illumination (and environmental) conditions. Chlorophyll fluorescence is related to photosynthesis activity and depends on illumination (and environmental) conditions and canopy state.

Functional traits Several definitions of functional traits exist in the literature due to the concept being explored in the context of plant phenotyping and ecology. Caruso [46] proposes that "functional traits are generally considered aspects of plant phenotypes that influence growth, survival, and reproduction by mediating interactions with the biotic and abiotic environment". We propose to define functional traits as traits describing canopy, plants, or organ reactions to the environment. Since they account explicitly for the environmental conditions on some processes, they are expected to be less sensitive to some environmental factors. They will therefore be more heritable than most of the other traits. Efficiency traits are commonly used functional traits that evaluates the efficiency with which elements are used by the plant to grow. They include the radiation (RUE), water (WUE) and nitrogen (NUE) use efficiencies.

A more exploratory approach for functional phenotyping is to adjust the parameters of crop growth models to match the dynamics of the state variables. Blancon [47] proposed to calibrate a simple maize growth model [48], [49] using Leaf Area Index calculated from UAV RGB imagery to retrieve parameters that were more heritable than the LAI itself. Recently, Shouyang Liu [50] proposed to use a 3D model to retrieve few parameters of the ADEL-wheat model. The advantage of HTPP phenotyping is also to propose traits that can be use more frequently by crop models such as CHN [START_REF] Soenen | chn", a crop model to jointly manage water and nitrogen on winter wheat[END_REF], Apsim [START_REF] Hammer | Adapting apsim to model the physiology and genetics of complex adaptive traits in field crops[END_REF][53] [START_REF] Keating | An overview of apsim, a model designed for farming systems simulation[END_REF] or STICS [START_REF] Brisson | An overview of the crop model stics[END_REF] [START_REF] Brisson | Stics: A generic model for the simulation of crops and their water and nitrogen balances. i. theory and parameterization applied to wheat and corn[END_REF] [START_REF] Brisson | Conceptual basis, formalisations and parameterization of the STICS crop model[END_REF], which represent our current knowledge on plant physiology. Advanced functional traits are meant to breed on specific processes that allow a better understanding of the reaction of the crop under a wide range of environmental conditions. As a result, it is possible to find an optimal set of functional traits that will define an ideotype for given environmental conditions [START_REF] Donald | The breeding of crop ideotypes[END_REF].

Phenotyping installations to control/describe the environment and measure traits

The several types of installations

Experimental installations, also called field stations or platforms, are the facilities where the experiments take place. One or more trials can be conducted in such installations. A platform is equipped to describe the environmental growing conditions and the several traits of interest. The environmental conditions can be fully controlled as in ecotrons [START_REF] Roy | Ecotrons: Powerful and versatile ecosystem analysers for ecology, agronomy and environmental science[END_REF], or only some environmental factors are controlled/manipulated as in field conditions. HTPP installations have a high degree of automatization both for measuring the environmental conditions and the plant traits. Installations can be very diverse, and some examples are shown in figure 6: • Low control / Low detail The most basic installations, often used for the study of agricultural practices (e.g. inputs, farm system etc…). It can be as simple as a farmer's field with few large microplots. • Low control / High detail A high-cost installations which does not control the environment is used to study the performances in real conditions with high temporal and/or spatial resolution and equipped with numerous instruments. The Arvalis station in Gréoux Les Bains, or INRAe in Auzeville, both in France belong to this category since they are equipped with a Phenomobile, an automatic rover system. Such installations are used in the final stage of the crop selection and cultivar evaluation. • High control / Low detail A low-cost installation that controls the environment is used to measure few key traits of plants to determine the impact to specific environmental conditions. It is still used to measure the plant transpiration for instance. • High control / High detail An installation with high control and high throughput systems to measure accurately many traits under a range of fully controlled environmental conditions.

The phenotyping systems used to measure traits Under field conditions, a vehicle, called the vector, moves the sensors that record the signals emitted or reflected by the plants. The combination of a vector and sensors determine a specific temporal and spatial resolution and coverage, as presented in figure 7. Satellites such as a Sentinel or Planet provide field data a few times per week everywhere globally, at the expense of a metric to decametric spatial resolution. Gantries and Robots are the most advanced vectors used for phenotyping purposes. These systems can be automatic (Phenomobile [START_REF] Deswarte | Platform development for drought tolerance evaluation of wheat in france[END_REF], FIP [START_REF] Kirchgessner | The eth field phenotyping platform fip: A cable-suspended multi-sensor system[END_REF], Rothamsted Field Scanalyzer [START_REF] Virlet | Field scanalyzer: An automated robotic field phenotyping platform for detailed crop monitoring[END_REF]) or modified agricultural machinery driven by a human such as a tractor [START_REF] Andrade-Sanchez | Development and evaluation of a field-based highthroughput phenotyping platform[END_REF]. An important feature is the presence of an active illumination or not. These vectors can "scan" the agricultural experiment every 3-7 days with a very high spatial resolution and a flexible viewing geometry. A cheaper alternative appeared: the UAV, which is an efficient solution for plant phenotyping, achieving very high throughput with a high spatial resolution and numerous flights covering the whole crop cycle and acquired traits such as the green fraction [START_REF] Duan | Comparison of ground cover estimates from experiment plots in cotton, sorghum and sugarcane based on images and ortho-mosaics captured by uav[END_REF], plant counting [START_REF] Gnädinger | Digital counts of maize plants by unmanned aerial vehicles (uavs)[END_REF], [START_REF] Liu | A method to estimate plant density and plant spacing heterogeneity: Application to wheat crops[END_REF] or height [START_REF] Madec | High-throughput phenotyping of plant height: Comparing unmanned aerial vehicles and ground lidar estimates[END_REF]. Hand-held devices or smartphones can also be used for phenotyping in the field, allowing some key traits to be measured. Such devices are attractive to conduct experimentation in farmers' fields and are already used for decision making for farmers: apps such as Xarvio scouting or PlantVillage Nuru [START_REF] Mrisho | Evaluating the accuracy of a smartphone-based artificial intelligence system, plantvillage nuru, in diagnosing of the viral diseases of cassava[END_REF] provides automatic advice based on a smartphone photo. Finally, field IoT (Internet of Things), which are connected cameras fixed on a boom, can monitor daily the growing stages [START_REF] Velumani | An automatic method based on daily in situ images and deep learning to date wheat heading stage[END_REF] and disease development [START_REF] Mishra | Modified ride-nn optimizer for the iot based plant disease detection[END_REF]. However, the coverage of a single system is minor, in the order of few square meters. The choice of the sensors is constrained by the vector and is generally a trade-off between cost, possible payload and energy consumption, spectral resolution, and spatial resolution. RGB imagery is the cheapest way to acquire data, with an unmatched resolution of millions of pixels and is widely used aboard all vectors. Recent drones can be equipped with a 100Mpx camera. Multispectral and hyperspectral imaging systems are popular for phenotyping applications, as these sensors can retrieve information on the biochemical composition of plants. However, the gain in spectral richness is achieved at the expense of spatial resolution. Vectors are also often equipped with a lidar system to reconstruct the 3D structure of the canopy finely.

While the HTPP systems are designed to increase the throughput with the possibility of monitoring crop growth with a high-revisit frequency, they also improve the accuracy with which the traits are estimated and allow access to new traits not measurable with the traditional techniques. However, the large volume of data generated and the complexity of interpreting the recorded information currently constitute the bottleneck of phenotyping. In the following, we will focus on this third component of HTPP systems: data interpretation. High throughput plant phenotyping is today an independent field with its journals and methods, although it was created by an association of remote sensing scientists and geneticists. It strongly influenced the first algorithms used to transform raw data acquired by HTPP into valuable traits. The focus was first put on exploiting the spectral information with the use of vegetation indices that are simple combinations of reflectance or radiance in few spectral bands. For instance, NDVI that combines the red and near-infrared (NIR) bands is a good proxy of green vegetation. Several vegetations indices were proposed in the literature for hyperspectral, multispectral and even RGB sensors [START_REF] Bannari | A review of vegetation indices[END_REF], [START_REF] Xue | Significant remote sensing vegetation indices: A review of developments and applications[END_REF]. Multispectral and hyperspectral cameras are popular to estimate the content of pigments used in photosynthesis, such as chlorophyll. However, most of these simple interpretation methods are based on empirical relationships with few traits. These relationships are generally established over a limited dataset, and great caution should be paid when applying them under conditions not represented in the training dataset. Further, the simple vegetation indices may also lack causal relationships with the targeted trait, resulting in poor accuracy and robustness. Alternatively, physically-based approaches have been developed to get more robust estimates of a few critical structural and biochemical traits. The PROSAIL model [START_REF] Jacquemoud | Prospect+sail models: A review of use for vegetation characterization[END_REF] is a popular radiative transfer model that combines PROSPECT [START_REF] Jacquemoud | Prospect: A model of leaf optical properties spectra[END_REF] that simulates leaf optical properties, and SAIL [START_REF] Verhoef | Light scattering by leaf layers with application to canopy reflectance modeling: The sail model[END_REF] that simulates bidirectional canopy reflectance. PROSPECT was used to estimate GF from the multispectral reflectance. When observing at 45°inclination, we can estimate the GAI (Green Area Index), the AIA (Average Inclination Angle) when combined with nadir observations, which can finally be used to compute FiPAR (Fraction Intercepted Photosynthetically Active Radiation), which summarizes the biophysical capacity of a plant to capture solar radiation. The FIPAR can then be used to calculate the light use efficiency if independent biomass measurements are available. However, PROSAIL is based on simple assumptions on canopy architecture that are well verified for crops such as wheat while being violated for other crops such as maize or sunflower.

From remote sensing interpretation methods to deep learning ones

Other instruments complement the RGB, multispectral or hyperspectral cameras: Infrared thermal cameras measure the surface temperature that indicates water stress under specific conditions. Active and passive fluorescence cameras have also been used to access the functioning of photosynthesis and other characteristics of the leaves. More recently, approaches based on exploiting the high-resolution imagery have been developed, especially using RGB cameras. Computer vision techniques [START_REF] Ubbens | Deep plant phenomics: A deep learning platform for complex plant phenotyping tasks[END_REF] based on geometrical rules have successfully estimated several structural and morphological traits. Further structure-frommotion (SfM) methods applied on UAV images allows generating the surface elevation model from which the plant height can be derived [START_REF] Madec | High-throughput phenotyping of plant height: Comparing unmanned aerial vehicles and ground lidar estimates[END_REF]. Alternatively, stereoscopic and multi-vision methods and LiDAR also provides a 3D description of the non-occluded parts of the plants [START_REF] Dandrifosse | Imaging wheat canopy through stereo vision: Overcoming the challenges of the laboratory to field transition for morphological features extraction[END_REF]. Such methods are widely used for in-door phenotyping to describe the architecture of the aerial parts [START_REF] Paulus | Measuring crops in 3d: Using geometry for plant phenotyping[END_REF], and roots of the plants [START_REF] Bucksch | Imagebased high-throughput field phenotyping of crop roots[END_REF]. Vegetation indices, physically-based methods, computer vision ones, or Deep Learning approaches are already able to produce an extensive range of traits for HTPP. Table 1 presents an example of the traits that are routinely produced at Arvalis.

High-resolution RGB imagery allows the exploitation of the texture and the shape of the objects in the image to extract a range of traits. Compared to the multispectral and hyperspectral imagery where the lower spatial resolution forces to interpret the signal concurrently for identifying the organs of interest and characterizing them to derive the targeted trait. It results in possible confounding effects, leading to generally to lower performances of the estimation. The interpretation of high-resolution RGB imagery has benefited for approximately five years from the advances in deep learning techniques. Several traits are now estimated using DL, including plant segmentation ( [START_REF] Kuznichov | Data augmentation for leaf segmentation and counting tasks in rosette plants[END_REF], [START_REF] Sadeghi-Tehran | Multi-feature machine learning model for automatic segmentation of green fractional vegetation cover for high-throughput field phenotyping[END_REF] and plant and organ detection [START_REF] Fan | Automatic tobacco plant detection in uav images via deep neural networks[END_REF], [START_REF] Madec | Ear density estimation from high resolution rgb imagery using deep learning technique[END_REF], disease classification and quantification ( [START_REF] Fuentes | A robust deep-learning-based detector for real-time tomato plant diseases and pests recognition[END_REF][85] [START_REF] Shakoor | High throughput phenotyping to accelerate crop breeding and monitoring of diseases in the field[END_REF], biomass and yield prediction ( [START_REF] Aich | Deepwheat: Estimating phenotypic traits from crop images with deep learning[END_REF], [START_REF] Dreccer | Yielding to the image: How phenotyping reproductive growth can assist crop improvement and production[END_REF] ). Deep learning is now considered as the state-of-the-art approach that handles a large diversity of traits difficult or impossible to estimate with pixel-based classification and computer vision techniques [START_REF] Pound | Deep machine learning provides state-of-the-art performance in image-based plant phenotyping[END_REF].

Robust estimation of traits with deep learning

The development of DL models requires much human effort to prepare large datasets of labelled images used both for training the model and its evaluation. It is due to the empirical nature of the DL approach, inferring the predictions rules from a training dataset. It poses a problem of reliability of the predictions when applied to an extensive range of HTPP data. This section first presents the principles of Machine Learning and Deep Learning, emphasising the robustness problem. Deep Learning is a subfield of Machine Learning (ML) and is based on multiple processing layers to extract features and solve a large range of tasks. It achieves state-of-the-art performances in a wide range of applications including speech recognition [START_REF] Deng | New types of deep neural network learning for speech recognition and related applications: An overview[END_REF] , visual object recognition [START_REF] Alom | The history began from alexnet: A comprehensive survey on deep learning approaches[END_REF][92] [93][94], localization [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF][96] [START_REF] He | Mask r-cnn[END_REF] , segmentation [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], language translation [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], and plant phenotyping as previously described. A neural network is still the state of the art of ML for relatively simple problems. It is based on the use of a network of neurons. A neuron is an elementary processing unit that receives several inputs, which are transformed into one output. The most simple form of a neural network is the perceptron [START_REF] Rosenblatt | The perceptron: A probabilistic model for information storage and organization in the brain[END_REF] which is made of one neuron. However, neural networks are generally organized in several layers of neurons and are called Multi-layer perceptron (MLP) [START_REF]Analytic techniques for the study of neural nets[END_REF]. The neuron's input can be the raw input data or the outputs of the previous layer.

Basic of Deep Learning in computer vision

A brief history of recent progress in computer vision

Although neural networks were already described in the '60s, DL was discovered supposedly in 1989 [START_REF]Backpropagation applied to handwritten zip code recognition[END_REF] or 1995 [START_REF] Lecun | Comparison of learning algorithms for handwritten digit recognition[END_REF], during LeCun stay at the ATI&T labs. The first DL model was able to recognize cursive letters on mails thanks to few layers of convolutions learned with backpropagation [START_REF]Learning representations by back-propagating errors[END_REF]. DL was, however, not very popular at this period as compared to other ML algorithms. The rise of DL started with the winning of AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] on the ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] large scale visual competition: it beats XRCE ( [START_REF] Mensink | Lear and xrce's participation to visual concept detection task -imageclef 2010[END_REF]) by almost 10 points (from an error of 26% to 16% as shown on figure 8). From this date, no alternative approaches to DL won any major challenges in computer vision, almost ten years later.

These successes have spread to other domains such as language processing which also use DL intensively since the introduction of the attention mechanism by Bengio et al. [START_REF] Chorowski | Attention-based models for speech recognition[END_REF]. Figure 8B presents two unbelievable breakthroughs made by the DeepMind company: Alphazero [START_REF] Silver | A general reinforcement learning algorithm that masters chess, shogi, and go through self-play[END_REF] which beats a human on the chess and go games without looking at historical games, and AlphaFold [109] which converts the 3D structure of proteins from their 2D sequence. This problem previously required a lot of human inputs with the program FoldIt [START_REF] Kleffner | Foldit standalone: A video game-derived protein structure manipulation interface using rosetta[END_REF]. From a theoretical point of view, the main breakthroughs were theorized well before the emergence of AlexNet: convolution, backpropagation, neural networks were already known but not widely used. Practical factors are contributed to the development of DL for many applications:

• The release of CUDA [START_REF] Sanders | CUDA by example: an introduction to general-purpose GPU programming[END_REF] in 2007 by NVIDIA allowed researchers to use GPU with languages such as C, C++ or Fortran. DL requires a lot of computer power, but most of the operations can be parallelized. A GPU can be viewed as a set of thousands of low-frequency CPUs. AlexNet was one of the first networks implemented on GPU, allowing for more layers and training on more data. • Large labelled datasets were available starting 2009, with ImageNet release [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF].

It was on the first time that 100 million images were used to train a network. Later, other datasets such as the MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] were crucial to train detectors. Such benchmarks helped researchers to focus on improving the architecture and increase the performances of the DL models. • The quality of DL libraries also has helped to democratize. Caffe, written in C++, was one of the first libraries that could be leveraged to adapt DL networks for new domains, alongside Torch, written in Lua. The emergence of Tensorflow, which had a python API, also helped a lot. Today, multiple frameworks exist. Pytorch is popular among researchers while more beginner-friendly yet powerful libraries exist, such as Keras. Figure 9 presents few popular libraries, sorted by their level of ease.

Deep Learning is about learning a better representation

In the traditional ML approach, an algorithm extracts features from an object of interest (image, text, audio file), and a classifier is trained to decide on these features. Choosing the right set of features is then critical to get high performances. In DL, the representation, i.e. features extraction, and the classification processes, are trained jointly, so no human interactions are needed to find the best features for a given problem. It is illustrated in figure 20 for image segmentation based on pixel colours. Instead of choosing a set of vegetation indices or changing the colour space to train a Support Vector Machine (SVM), DL can learn a directly used representation to segment an image. The change of representation helps to simplify the problem as we show it in example A (Figure 10): in this example, the better representation allowed by the simple rotation of the space reduces the complexity of the classification that can be done only on the x-axis. It can be viewed as a big and non-linear principal components analysis. DL methods are the most powerful way to learn a representation, but such an approach is not new in ML. Mairal and Ponce introduced the concept of "supervised dictionary learning " [START_REF] Mairal | Supervised dictionary learning[END_REF] where a set of wavelets are learned to compress the data.

Fundamental blocks of convolutional networks

As described in 1.3.1, the popularity of DL is due to the emergence of convolutional neural networks (CNN), which are only one subclass of possible DL algorithms. One CNN is made of few key blocks that help to learn a better representation. In Figure 11, we can differentiate the two components of a DL model: the encoder, or backbone, that extracts the features, and the task solver, which is usually a stack of fully connected neurons. The encoder is made of successive convolutional layers. Convolution is the multiplication of one part of the matrix (i.e. one part of the image) by a weights matrix. All elements are then summed and affected to a pixel of the output. Then the convolution works as a sliding window to get the value for the other pixels. We present an example 11 of one convolution output (ResNet-50, first block) on a wheat image. It shows that the role of the convolution is to activate or de-activate some specific parts of the image. In our example, the leaves are activated in some dimensions but not in others. This process is repeated several times: the number of dimensions increases at each iteration while the spatial resolution decreases. The deeper we go, the more complex the representation of the image is.

To reduce the spatial resolution without learning additional parameters, one can also use a pooling layer, which sums or averages the values of a group of adjacent pixels to a unique value. This operation does not change the dimension but reduces the spatial resolution. The output is a unique features vector, usually between 1024 and 2048 dimensions, which are then used by a classifier. The features vector is meant to discriminate specific parts of the image. A last important building block is the activation function, which is usually used on the convolution output. The activation layer, which needs to be differentiable, helps the gradient not to vanish and brings useful non-linearity to the network. Usual functions are presented in figure 12, but it is a research area by itself! All CNN models such as AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] , ResNet-50 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], and EfficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] propose variations of the described architecture. Some works are conducted with AutoML [START_REF] He | Automl: A survey of the state-of-the-art[END_REF] to learn the optimal architecture instead of handcrafting it.

Training a Deep Learning network right

The architecture of CNN models is not the only reason for their success, but the learning process, called training, is important to tune optimal weights for the convolution. It is done by using a "loss function". During the training process, presented in figure 13, each data point (for instance, an image) is presented to the network, which will provide an output (for instance, the class "wheat" or "maize"). This output is compared to the labelled target ("Wheat"), and the loss function computes a penalty. The goal of the training is to minimize this loss by changing weights values. The derivative of the loss function is used to update the weights and minimize the loss. The backpropagation gradient, or "backward propagation of errors", introduced by Rumelhart and Hinton [START_REF]Learning representations by back-propagating errors[END_REF], helps to efficiently evaluate the gradient of error at each layer of the network. The optimizer is the algorithm that updates the weight given the gradient value. A simple optimizer such as the stochastic gradient descent (SGD) updates the weights by subtracting the gradient multiplied by a learning rate value. The learning rate can be interpreted as the sensitivity to new data: a high learning rate will make the network prone to overfit, while a low learning rate prevents the network from learning. The loss function can be adapted to plant phenotyping scenarios to encode expert knowledge. For instance, in the case of multi-species vegetation classification, one can set the model to predict the class "vegetation" AND "wheat" or "maize" and set a penalty if the algorithm attributes the class "background' and "wheat" to the same pixel. It is an area of research that is more probably accessible to the plant phenotyping community. Training from scratch a DL model on a very large database is time consuming and costly. The completion of 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. [START_REF] You | Imagenet training in minutes[END_REF]) and costs 1.2 M USD to train it in 24 minutes. However, it can be optimized to 18 minutes with an associated cost of 40 USD [119]). It is then not always accessible for everybody to train a state-of-the-art detector. Also, not everybody has access to large datasets to train a network for their application. However, a popular technique called transfer learning helps to solve this issue. The learned representation of models trained on large datasets can be re-used for other problems, even with smaller datasets. The process is described in figure 14: the classifier can be removed while keeping the encoder (trained convolutional base), and one can append a new classifier on top of the representation. At the beginning of the training process, the error (loss) will usually be high. Therefore, we freeze the encoder so we do not lose the representation because of the first step, and we let the classifier train on top of good features.

Few model architecture have allowed the large varieties of algorithms in 2.4

DL models can be designed to solve more complex tasks than classification: they can use outputs larger than an n-dimension vector, and the loss function can be adapted. It is important to underline that most traits in plant phenotyping will be based on classic architecture and do not require designing a specific architecture. However, conferences such as the Computer Vision Problems in Plant Phenotyping (CVPPP, CVPPA) promotes plant phenotyping as an area of research for Computer Scientists. Despite the vast number of models published every week, few stand above the crowd with time. Classification networks (figure 15a) are widely explored with architecture such as AlexNet, VGG, GoogLeNet which is very deep, ResNet [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], which introduces the concept of skip connection which concatenates the output of the convolution with its input, so information is better kept within a very deep network and does not vanish. Segmentation(figure 15b usually relies on the encoderdecoder architecture, with model such as U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. After compression of the information to a very deep representation, up-convolution is used to restore the original resolution. Skip connection between the encoder and decoder is essential to share information of layers with the same resolution. The DeepLab [START_REF] Chen | Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[END_REF] architecture relies on the "atrous convolution" to capture multi-scale information for segmentation usage. Such convolution allows getting a large field of view without increasing the number of parameters. Other encoder-decoder architectures such as Feature Pyramidal Network have been proposed for the segmentation process. A pre-trained classifier can use to initialize the weight of the encoder.

Encoders are also used for object detection, which can be separated into two families (figure 15c. One-Stage encoder which regroups SSD [START_REF] Liu | Ssd: Single shot multibox detector[END_REF], YOLO [START_REF] Redmon | You only look once: Unified, realtime object detection[END_REF], Mobilenet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF] and directly regresses the coordinates of the bounding boxes with their corresponding classes based on the encoded representation. A two-stage object detector will use the encoder to train a region proposal network (RPN) to propose possible objects. Then each object is extracted with its features with the Region of Interest pooling layer (ROI) and will be finally assigned to a class. One stage detector is usually faster to compute but less robust. Such object detector can be applied to instance segmentation, which retrieves not only a box around an object but also its mask (figure 16). Finding the best architecture can then be a challenge when designing a method to evaluate a trait. Models such as ResNet, Faster-RCNN or U-Net were designed between 2015 and 2017 and are still intensively used because training tricks or new layers can improve their robustness. These models are readily available in many DL libraries. The theory of DL is still moving, and cascades of new models are proposed every week. Convolutional neural networks are starting to be less central in the performance of DL.

An explosion of alternatives to CNN

Convolutional Neural Network made it possible to learn DL features with neural networks by reducing the number of parameters compared to a multi-layer perceptron and is a significant block of the algorithms used in this PhD which focus on RGB imagery. However, as illustrated in figure 17 alternatives are available for another type of data. For instance, in Neural Language Processing, recurring neural networks (RNN) were used to process text data. It includes as a basic bloc a gated recurring unit (GRU) instead of a convolutional layer. GRU can process a data point with the information of previous data points: predicting the next word or sentence or the next value in the case of temporal series. The NLP community also developed Transformers, networks which are based on the attention mechanism. The attention mechanism can learn the correlation between two variables for all inputs. The attention mechanism can be applied in an unsupervised manner with encoder-decoder architecture. It is then called self-attention. It has been critical to the performances of the DeepMind solutions to solve the go game and protein folding. It is interesting to notice that such a "Vision Transformer" can reach state-of-the-art performances without any convolutional layer, despite being more sensitive to shapes than texture compared to CNN networks [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]. It has already been extended for detection, and segmentation [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. On top of the new neural network architecture, progress has been achieved for non-grid structures such as 3D meshes or protein structures usually represented as graphs. Graph Attention Networks [START_REF] Veličković | Graph attention networks[END_REF] extend the attention mechanism by taking as input the nodes features and the adjacent matrix as input.

Training robust DL algorithm for plant phenotyping can be tricky

The bias problem

When DL algorithms are trained, it is assumed that both the training and test dataset are independent and identically distributed (IID hypothesis). Creating such a dataset requires capturing most of the ordinary conditions of variations of an object of interest. A consistent distribution is called "domain". Defining the domain is entirely open depending on which statistics we want to use to determine the distribution: is it a set of expert metadata? Or statistics of the extracted features? Determining the exact distribution of a set of images, which are high dimensional objects, is not as easy as describing the distribution of 2-D or 3-D vectors. Using metadata or expert information is the first step to describe a domain. For instance, a set of images of cars in "Los Angeles streets in summer" can form a domain defined by specific weather or a specific architecture. Training an algorithm on such a dataset and testing it in a set of images of cars from "Minnesota in winter" could results in a dramatic drop in performances, as the weather, the illumination, the architecture was never shown to the model previously. We call the "domain shift": the drop of performances on a new domain. It is then critical to have a dataset that covers the whole distribution of the "application domain" as illustrated in figure 18: the expected distribution of the data to predict. Some critical ethical issues were raised when using DL algorithms: Google's algorithm deployed for its photography application was mistaken "black people" for "gorilla" ??. It was due to a wrong design of the training dataset that did not contain enough examples, and the evaluation was not robust enough.

Training domain Application domain

?

Figure 18. Illustration of the shift between application and train domain It is crucial in plant phenotyping to control the IID hypothesis, mainly because we expect the error computed on the evaluation dataset to be the lowest possible and randomly spread. It is hence not acceptable to have a DL error that will vary with the domain. Many sources of variation exist in plant phenotyping. The first strong one is the difference between indoor and field conditions. Indoor images often correspond to individual plants, with controlled illumination and a uniform background: the only difference between two images is then the expression of the genotype to the proposed environment and phenological stages. The difficulty is high in field conditions due to multiple, overlapping plants, variable lighting, and background. The environment influences more the characteristics of the acquired data. Genetics and environment for in-field phenotyping are then sources of variations. Acquisition conditions (flash use, RGB sensors quality and configuration) influence the image quality. Usually, data in phenotyping are acquired during an "acquisition session": a set of images that are acquired on the same experiment field, during a short time window with the same vector and sensors". Such acquisition sessions are made of highly similar images. Only the possible modalities within the experiment, and changes in illumination conditions, will bring additional variability to the genetic one. Acquisition sessions correspond, therefore, to very narrow distributions of images compared to the application domain, which generally corresponds to other acquisition sessions. The DL models are expected to be robust to the genetic, the environment and the sensor as described in figure 20. Training a DL model for plant phenotyping on only a few acquisition sessions does not ensure robustness. It does not prevent learning spurious correlation and uses features that are present only in the training domain. The goal of DL for plant phenotyping is to learn generic features that can extend to an "out-of-training domain" object.

Building a large and diverse dataset

A solution to represent a data distribution covering all applications cases is to have a vast training dataset. A training dataset comprises data points, i.e. images, an audio file, text, and a corresponding target. For classification, the target is a "label" (a category); in semantic segmentation, it is a mask containing one or several labels per pixel; in object detection, it is a set of bounding boxes that enclose the aimed objects. Targets are created through a process called "annotation" or "labelling", where a human generate the target interactively. This process can be burdensome, depending on the interface used. The computer vision community relies on few large key datasets to benchmark their approach. As discussed in 1.3.1, ImageNet or MS COCO were key for DL development. In contrast, the plant phenotyping community suffers from a lack of such tools. Initiatives like the Leaf segmentation and counting datasets [START_REF] Minervini | Finely-grained annotated datasets for image-based plant phenotyping[END_REF] exist but were limited to indoor phenotyping until today. Datasets can be found on platforms such as www.quantitative-plant.org or on more general platforms such as Kaggle or AIcrowd that host several plant phenotyping datasets linked to actual or past competitions. The PlantVillage dataset (disease classification) [START_REF] Hughes | An open access repository of images on plant health to enable the development of mobile disease diagnostics through machine learning and crowdsourcing[END_REF] or PlantCLEF [127][128][129] are large datasets available on internet. Due to the diversity of traits to evaluate, there is a strong need to label even more data, and the current shared data is still not sufficient.

This issue can first be viewed as an engineering problem to solve. Several companies built labelling tools that fulfil all DL needs, including plant phenotyping. Such interfaces are progressing fast and have started to propose "smart tools" to accelerate the creation of manual targets. Figure 20 presents an advanced labeling platform (Datatorch). DEXTER (Deep Extreme Cut) [START_REF] Maninis | Deep extreme cut: From extreme points to object segmentation[END_REF] is an example of innovation for labelling, by allowing to create masks of a single instance by indicating only a few points on the frontiers, by leveraging deep learning features to find the contour of the object. Laboratories and institutes can develop additional tools to enhance the labelling experience: the CAPTE team has developed a tool that labels superpixels instead of single-pixel to create segmentation masks. In contrast with DL models primarily shared in open-source, labelling innovations start to be proprietary and less described; for example, the "smart tool" proposed by V7-Darwin can infer an object mask from a loose box and interact with the user, but the technology behind is private. The tedious nature of the labelling process leads to creative developments to accelerate the process. A direct way is to leverage the previously accumulated datasets to train the DL model and propose possible targets that must be validated or corrected by a human. These approaches have been tested in plant phenotyping by computing the outputs of a sorghum head detector on new images and letting a human correcting the output [START_REF] Ghosal | A weakly supervised deep learning framework for sorghum head detection and counting[END_REF]. It is claimed to accelerate the process by a factor of 4. An alternative is to re-use "classic" computer vision algorithms to generate the possible targets. The SNORKEL [START_REF] Ratner | Snorkel: Rapid training data creation with weak supervision[END_REF] algorithm combines several heuristics to produce large training datasets for the DL algorithm. Recently, Liao et al. [START_REF] Liao | Towards good practices for efficiently annotating large-scale image classification datasets[END_REF] proposed a method to understand the strength and weaknesses of human labellers, which have all different priors and exploit them for a more efficient process. Generating large datasets to label is burdensome for small research teams, but more and more companies propose to label images at a competitive price.

Data augmentation

Long training using the same data points can reduce results due to the overfitting phenomena. More diversity can be achieved easily with the use of data augmentation when loading the training data. Data augmentation is a process to generate a valid variation of the training data. It can be easily implemented for straightforward strategies that change the image's geometry with rotation, padding or shearing, for instance or the photogrammetry with additional blur or change in the colour histogram. Simple perturbations such as Gaussian noise are also a form of Data Augmentation. The diversity introduced works better if the resulting images still belong to the application domain. For example, a random variation in the image's contrast should result in a new image where the objects of interest are still visible. Expert knowledge is then crucial to choose the proper data augmentation, though it can also be determined with automatic approaches such as AutoAugment [START_REF] Cubuk | Autoaugment: Learning augmentation policies from data[END_REF]. Most of the usual Data Augmentations are included in python libraries such as albumentations [START_REF] Buslaev | Albumentations: Fast and flexible image augmentations[END_REF] which can be integrated into DL frameworks such as Pytorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF], or Keras [START_REF] Chollet | [END_REF].

The challenge of overcoming bias without very large datasets

Even with improvements in labelling, and possible outsourcing, it is desirable to constrain the learning process to get more robust networks. We can leverage three possible ideas: transductive transfer learning, invariant features learning or modelling the diversity.

• Transductive transfer learning is close to the idea of active learning but without human supervision. Its most simple form is pseudo-labelling [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF], where we apply the model to the new domain. However, suppose the performance is still acceptable, the predictions can be used as labels, called "weak labels", and re-train the model jointly on the clean and weak labels, so the classifier is adapted to the aimed domain. It works when the source and the target domain are close and can help to boost the performance. However, while being an excellent solution to align the features from the source and target domain, it is not explicitly doing it. • Domain adversarial model [START_REF] Ayalew | Unsupervised domain adaptation for plant organ counting[END_REF], [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF] propose to append a second classifier that tries to predict the domain, but with a Gradient Reversal Layer (GRU) that invert the gradients to force the network not to discriminate the domain. The advantage of the method is that it does not require any additional labels on the task for the target domain and can be trained only on the source domain labels. Instead of using a classifier, DeepCORAL [START_REF] Sun | Deep coral: Correlation alignment for deep domain adaptation[END_REF] proposes to add a term on the loss function that constrains the cross-correlation of the features network between the train and the source domain. These algorithms can be extended to n domains. The domain adaptation can also be accomplished during the training. The loss function is usually calculated without prior on the domain and is an "Empirical Risk Minimization". In contrast, the "Invariant Risk Minimization" (IRM) [START_REF] Arjovsky | Invariant risk minimization[END_REF] has been proposed to force the classifier to generalize over the distributions of the train domains and is expected to work better on out-of-distribution objects. Other algorithms such as GroupDRO [START_REF] Sagawa | Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization[END_REF] propose to minimize the worst performance case to reach generalization. • Generative Adversarial network: The third idea is to learn the model the diversity of the dataset and generate more examples. A generative Adversarial Network learns to generate a large diversity of images. Models such as CycleGAN [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF] can model the transformation required between images from two domains without any labels or paired images. It is an appealing solution to model a domain. However, using the output of a CycleGAN has not yet been demonstrated to solve the domain shift problem.

How to evaluate the robustness of DL models ?

Pitfalls of the current Deep Learning evaluation benchmark DL community often works on one or a few datasets such as MNIST, CIFAR-10, CIFAR-100, CelebA, MS COCO, or ImageNet to benchmark the models. These datasets contain several limits: a study in 2021 [START_REF] Northcutt | Pervasive label errors in test sets destabilize machine learning benchmarks[END_REF] have identified an average of 3.4% of error in the labels. Some examples are shown in figure 21. After correcting the errors, the study demonstrates that shallower models such as ResNet 34 perform better than their deeper counterpart, ResNet-50. Recht et al. [START_REF] Recht | Do imagenet classifiers generalize to imagenet?[END_REF] extended the ImageNet test set by proposing a similar yet different test set and observe an averaged 11-14% increase of performances. It is difficult in this context to conclude that a slight improvement (1-3%) on a benchmark dataset translates to an actual improvement on any other independent dataset. It explains why some "old" models are still popular despite the many new propositions published every year. The small number of metrics used to evaluate the model performances is also problematic. Their usefulness for benchmarking models is generally not discussed, but standard metrics are still widely used. For instance, the popular intersection over union (IoU) measures how well two bounding boxes match. However, when they do not overlap, the IoU is always equal to 0, whatever the distance between the two boxes. Rezatofighi [START_REF] Rezatofighi | Generalized intersection over union: A metric and a loss for bounding box regression[END_REF] proposed a new metric to correct this weakness. Using a single metric is therefore not sufficient to thoroughly benchmark models. Further, most of the mentioned datasets propose a train, validation, and test dataset part of the same distribution and then respect the IID hypothesis. It is also valid for plant counting, as shown in our work [START_REF] David | Plant detection and counting from high-resolution rgb images acquired from uavs: Comparison between deep-learning and handcrafted methods with application to maize, sugar beet, and sunflower crops[END_REF]. Work on domain shift was conducted on shifts that are not likely to happen in real life: mixing drawings to images [START_REF] Li | Deeper, broader and artier domain generalization[END_REF]; fake snow or flare, change of colour of MNIST digits [START_REF] Arjovsky | Invariant risk minimization[END_REF].

Assessing the robustness of traits estimated with Deep Learning

Assessing the robustness of a trait estimated with DL is necessary to ensure its quality when deployed operationally. The dataset used for the evaluation needs to cover the range of environmental variability, the genotypic diversity, the development stages, the sensors, and the different acquisition protocols used. Further, a part of the test dataset needs to be labelled the same way as the training one. However, because of the effort required for labelling, it is not always possible to represent the whole expected variability. Alternatives exist, such as labelling the test dataset with easier labels that cannot be used for training but still correlates with the measurement of the trait. The masks used for the segmentation task can be replaced by a set of labelled pixels, or the set of boxes used for detection can be replaced by a set of points on the objects of interest. Visual inspection is also helpful to find meaningful types of errors. Tools can be used to interpret DL algorithm outputs to understand how they took a decision. Visualizing intermediate outputs can also help to understand possible mistakes. Algorithms such as GradCAM [START_REF] Selvaraju | Gradcam: Visual explanations from deep networks via gradient-based localization[END_REF] are used to interpret outputs of CNN, but new architectures such as transformer directly learn the correlation between pixels of an image. Most networks also output a confidence score with their predictions, which can indicate possible problems. However, high confidence scores are also observed on erroneous predictions when the model is poorly calibrated [START_REF] Guo | On calibration of modern neural networks[END_REF].

The robustness of the DL method can be evaluated by comparison with independent measurements of the trait. Traits measured destructively, such as the biomass or the chlorophyll content, can provide a solid reference. Manual measurements also produce useful references, although they can vary between field workers and may depend on their fatigue. A perfect match between the trait estimates and the independent measurement is therefore not always desired. However, the error between the DL estimates and the independent measurement needs to be as low as possible, especially the possible bias. For phenotyping applications, the error should not depend on the genotype, and this should be systematically checked, although it is rarely the case. Another difficulty arises from the difference in spatial support used for the DL input measurements, i.e. the image footprint and the independent measurement. For example, disease scoring is generally based on the inspection of the whole microplot, while the images used as input to the DL model cover only a sample of the microplot. Ideally, the spatial support should be the same for the DL trait estimates and their independent measurement.

Additional consistency tests can be conducted, even when no reference method is available. The broad-sense heritability [START_REF] Piepho | Computing heritability and selection response from unbalanced plant breeding trials[END_REF], or reliability [START_REF] Bernardo | Reinventing quantitative genetics for plant breeding: Something old, something new, something borrowed, something blue[END_REF], can be used as a quality indicator by describing how repeatable the measurement is when exploiting the repetitions within a specifically designed experimental plan. The consistency between the trait estimated and another one, such as the final yield or critical performance characteristics such as lodging, disease resistance, or a particular yield component, can be a helpful indicator [45]. Finally, the quality of the possible association between a trait and markers of the genome, as found in a genome-wide association study (GWAS), can indicate consistency. However, these consistency checks are not sufficient since the estimated traits can be confounded with other highly heritable traits and possibly associated with some genome markers.

Objectives and organization of the study

The study aims to investigate possible strategies to train deep learning models that can be used operationally for high-throughput plant phenotyping experiments under field conditions. Special attention will be brought to the robustness of the estimates. For this purpose, the study relies on diverse and extended datasets collected over several partners and covers several traits and several species. The study is organized into four main parts, the three first ones corresponding to articles published, submitted, or to be submitted to international journals:

1. The first chapter discusses the robustness of DL method to identify plants from highresolution RGB imagery against handcrafted methods that rely on expert knowledge. This corresponds to an article submitted to European Journal of Agronomy. 2. The second chapter focuses on the building of a very large datasets of wheat head gathered from different institutions, and how it has been updated. It corresponds to two articles that have been published in Plant Phenomics. 3. The third chapter explains how this dataset has been leveraged to train robust wheat head detector thanks to the collaboration of competitors across the world for two successive challenges. The corresponding article is to be submitted to Gigasciences 4. We finally discuss our results in a final chapter that determine the perspective of Deep

Learning for plants phenotyping. The chapter is based on an article published at ICML.

References 2 Evaluation of the robustness of handcrafted and deep learning methods for plant density estimation 2.1 Foreword

Before the emergence of DL for phenotyping applications, some traits were already operationally extracted from high-resolution RGB imagery using computer vision classical approaches. Such "handcrafted" methods are based on expert knowledge on the problem from which features are identified and used to estimate the trait. The explicit use of wellverified assumptions can help scale to unseen conditions and, therefore, obtain more robust performances than DL methods that can be sensitive to domain shift. These handcrafted methods can also be used to post-process DL results and correct possible inappropriate ones. In this first chapter, we propose to compare a handcrafted approach with a DL approach for a plant detection problem. Plant detection is the mandatory first step before accessing higher-order traits such as plant density and plant characteristics. As reviewed in the introduction, we will focus on UAV observations that are a very efficient way to phenotype extensive experiments. It is very popular among breeders. However, the spatial resolution can be limited in case of too high altitude flights. We will consider in this chapter three species with relatively big leaves and sown with relatively even distances between plants: maize, sugar beet and sunflower.

Assessing the robustness of an approach is different from demonstrating the performance of new detection methods. While demonstrating the effectiveness of a new approach on a small dataset, evaluating its robustness needs to explore various possible conditions of acquisitions. In the introduction of this chapter, we demonstrate that most studies on plant counting or detection operate on a limited set of conditions, often with the validation achieved over the dataset used for the training of the DL model. A diverse dataset composed of UAV images from 27 flights (18 locations) covering three crops at different development stages has been compiled to compare a handcrafted approach with a DL approach. We further propose to combine both approaches into a hybrid one to beneficiate from their respective advantages.

Plant detection and counting from high-resolution RGB images acquired from UAVs

The following manuscript has been submitted at European Journal of Agronomy and is currently under revision.

Plant detection and counting from high-resolution RGB images acquired from UAVs: comparison between deep-learning and handcrafted methods with application to maize, sugar beet, and sunflower

Etienne David Measurement of the plant density is required for a number of applications since it drives part of the crop fate. The standard manual measurements in the field could be efficiently replaced by highthroughput techniques based on high-spatial resolution images taken from UAVs. This study compares several automated detection of individual plants in the images from which the plant density can be estimated. It is based on a large dataset of high resolution Red/Green/Blue (RGB) images acquired from Unmanned Aerial Vehicules (UAVs) during several years and experiments over maize, sugar beet and sunflower crops at early stages. A total of 16247 plants have been labelled interactively on the images. Performances of handcrafted method (HC) were compared to those of deep learning (DL).

The HC method consists in segmenting the image into green and background pixels, identifying rows, then objects corresponding to plants thanks to knowledge of the sowing pattern as prior information.

The DL method is based on the Faster Region with Convolutional Neural Network (Faster RCNN) model trained over 2/3 of the images selected to represent a good balance between plant development stage and sessions. One model is trained for each crop.

Results show that DL generally outperforms HC, particularly for maize and sunflower crops. A significant level of variability of plant detection performances is observed between the several experiments. This was explained by the variability of image acquisition conditions including illumination, plant development stage, background complexity and weed infestation. The image quality determines part of the performances for HC methods which makes the segmentation step more difficult. Performances of DL methods are limited mainly by the presence of weeds. A hybrid method (HY) was proposed to eliminate weeds between the rows using the rules developed for the HC method. HY improves slightly DL performances in the case of high weed infestation. When few images corresponding to the conditions of the testing dataset were complementing the training dataset for DL, a drastic increase of performances for all the crops is observed, with relative RMSE below 5% for the estimation of the plant density. Introduction

Plant density at emergence is a main yield component particularly for plants with reduced tillering or branching capacities such as maize, sugar beet and sunflower. The plant density at emergence is controlled by the seeding density and the emergence rate. Further, the seeding pattern defined by the distance between row and between plants influences the competition between plants and possibly with weeds. In addition to the estimation of plant density, the position of each plant can be documented to describe the local competitive environment [1]. For agronomical or phenotyping experiments, the plant density is mainly used to evaluate the quality of each microplot with consequences on the whole trial.

It is also used by farmers to decide to stop spending resources to grow the crop in case of too low density or too much heterogeneity. While plant density is not directly governed by the genotype, but results from the seeding density, seed vigor and the emergence conditions, it is considered as an agronomical trait in some widely used ontology [2].

Plant density is currently mostly evaluated manually in breeding programs. Operators count plants in the field over a limited sampling area since this process is tedious, time-consuming, and therefore expensive. Consequently, this traditional method can lead to significant uncertainties due to the limited representativeness of the sampled area and possible human errors. Further, the position of plants is generally not documented because it would be even more tedious to measure each plant location. The recent technological advances of plant phenotyping solutions including Unmanned Aerial Vehicles (UAV), sensors, computers, and image processing algorithms, offer potentials to develop alternative methods to the manual counting. Several authors already reported accurate estimates of plant or organ counting and density from RGB images (Table 1). Plants or organ can be characterized either with 3 machine learning (ML) algorithms where standard local image features are extracted and a used in a supervised classification to identify the objects of interest (Guo et al., 2018;Fernandez-Gallego et al., 2019). Handcrafted (HC) methods rely on expert knowledge to compute the pertinent features and use them to identify the objects of interest. Most of them belong to the Object Based Image Analysis [9]- [13]. The identification process can be done based also on the expert knowledge [5]- [7] or by calibrating a statistical model over a training dataset [8]. More recently, approaches based on deeplearning (DL) have been proposed. The features are automatically extracted from the image and then used to identify and localize the individual objects of interest ( [14], [15], [19], [20]). However, these features can also be used to estimate directly the density of objects through a regression [16]- [18]. Localization, is more popular (78% of the studies in Table 1) in plant phenotyping as it documents the sowing heterogeneity including missing plants, allowing to explore the competition between plants as outlined earlier. DL based methods are being common now to detect plant and organ and represent almost 30% of the localization studies (Table 1). Madec et al. [14] demonstrated that the Faster RCNN DL model [21] provides accurate localization of wheat ears with higher robustness than previous methods, including direct regression method. A higher heritability than that of manual counting was also reported. More recently, [19], [20] applied similar strategies to locate plant and organ from UAV images. DL applications to plant phenotyping are supervised learning methods, requiring large and diverse labelled datasets to converge to a generic solution. The recent progress in DL applied to detection/localization tasks beneficiated from the availability of large image collections such as ImageNet [22] and COCO Dataset [23] that are used to pre-train the DL model.

However, Geiros et al. [24] raised the overfitting risk and the resulting lack of robustness associated with most DL algorithms. They can reach excellent performances for datasets like those used for their calibration, while often failing when applied to cases different from the training dataset. In comparison, HC methods are based on expert knowledge which select the main features to identify the target objects. This reduces the risk of overfitting but can hardly account for all the specific cases. On the 11 methods listed (Table 1) that require a training dataset, only 3 [13], [14], [17] proposed a proper evaluation framework where the training and the test datasets do not come from the same acquisition sessions. This questions the accuracy, scalability and robustness of HC and DL methods that was investigated in the case of liver disease [25], but not for the plant detection problem within phenotyping applications.

The objective of this study is to compare a HC approach based on the knowledge of the sowing and plant patterns and a DL approach based on object detection to localize plants and count them. This study includes three species (maize, sugar beet and sunflower) observed with a RGB (Red Green Blue) camera aboard a UAV during 27 acquisition sessions with plants at different development stages few weeks after emergence. This study appears therefore to be the most comprehensive one on the subject (Table 1), while keeping always the training and test datasets as independent as possible. Further, we will also propose to combine the DL approach with expert knowledge from the HC one.

Materials and methods

Dataset

Experiments

The dataset used was acquired over maize, sugar beet and sunflower experiments from 2016 to 2019 in several experimental sites in France (Table 2). The sites cover a large diversity of agronomic modalities while managed with conventional tillage practices. However, some crop residues from the previous season can be observed on few microplots. Generally, few weeds were present in the microplots, except for some of them (Table 3). The sites include clay, brunisolic and limestone soil types (Table 2) with a variety of surface roughness and moisture. The soil color varies from gray to brown due to soil type, surface aspect and illumination conditions. Each site included an ensemble of microplots corresponding to many genotypes from which 3 to 12 were selected to get approximately 600 plants (Table 3). Some sites were flown several times (Table 2), corresponding to several acquisition sessions. This allows to get a larger variation in the crop development stage during image acquisition (Table 3). For maize, a total of 51 microplots was available from 9 acquisition sessions (Table 3) with contrasted microplot size, row spacing (0.3-1.1m), and plant density (5.1-11.2 plt.m -2 ).

For sugar beet, a total number of 60 microplots was available from 9 acquisition sessions with microplot size, row spacing and plant density varying within a small range (Table 2). For sunflower, a total of 78 microplots was available from 9 acquisition sessions with a large variability of microplot size, row spacing, and plant density.

Acquisition and labelling details

Image acquisition was carried out by UAVs embarking three different RGB cameras including the Sony Alpha 5100, Sony Alpha 6000, both with a resolution of 6024x4024 pixel, and the Zenmuse X7 (DJI) in the case of Epoisses site in 2019 with a resolution of 6016 x 4008 pixels. The cameras were fixed on a two axes gimbal to maintain the nadir view direction during the flight. The camera was set to speed priority of 1/1250 s to limit motion blur. The aperture and ISO were automatically adjusted by the camera. The camera was triggered by an intervalometer set at 1Hz frequency corresponding to the maximum value allowed to record the RGB images in JPG format on the memory card of the camera. Flight altitude above ground varied between 20 to 50m to get a ground sampling distance (GSD) between 2 mm and 5 mm per pixel (Table 3). The flight trajectory was designed to ensure more than 70% overlap between images across and along tracks. Ground control points were placed in the field and their coordinates were measured with a real-time kinetic GPS device ensuring an absolute 129 centimetric accuracy of their position. Agisoft Photoscan Professional software was used to align the images. The high overlap between the 141 images and structure from motion algorithm permits to compute the position and orientation of the 142 cameras. The pipeline described in Jin et al. [26] was then run to extract from each image the portion 143 corresponding to the contained microplots. Using the original images avoids the possible distortions and artefacts observed in the orthomosaïque. Several extracts may represent the same microplot viewed from different positions of the UAV [27]. For each microplot, the sharpest extract that contained the whole microplot is selected. For each session, a few microplots were selected for labelling (Table 2). Approximately 600 plants per session were labelled to ensure consistency across sessions which resulted in a total of 16247 labelled plants. Images were rescaled to match the best available GSD (2.5 mm, Table 3). This was necessary to control the apparent size of object, which can make the Deep Learning methods fail. Then all images were labelled using the coco-annotator tool [28] The labelling consisted in drawing a bounding box (BB) around each plant. Six different operators contributed to the labelling. The labelling from one operator was always reviewed at least once by a different operator. The typical size of the BB for one session (Table 3) was computed as the square root of the mean area of all the BBs.

The plant development stage during the acquisition sessions was scored into three relative levels, where stages 1 ,2 and 3 correspond respectively to early (few days after emergence), intermediate, and late stages (leaves start to fill the gap between plants). The correspondence between the stages for each crop, and their BBCH scale is presented in Table S1. The level of weed infestation (Table 3) was also visually evaluated from 0 (no weeds), 1 (sparse presence of weeds), 2 (infestation). The level of blurriness for each session (Table 3) was evaluated by calculating the average variance of the Laplacian [29].

Plant detection methods

Handcrafted method

The method developed here is based on several assumptions: (1) the plants are green and can be accurately separated from the background; (2) plants are sown in rows relatively evenly spaced and parallel; (3) the weeds are mainly located in between the rows and are not too dominant; (4) plants are relatively evenly spaced on the row and are not too variable in shape and size. The method first extracts each single row and then identifies each individual plant on the row. All the parameters of our HC method are expressed in relative value to the row or plant spacing, to allow adaptation to a larger number of sowing patterns. This makes our method scalable to all our experimental conditions across the three species (Table 2 andtable 3). The values of the parameters were set based on reasonable assumptions and were not calibrated on a dataset.

Row extraction

The original RGB images are first transformed into a black and white one (BW) using the excess green index (ExG =(2G-B-R)/G) where R, G, B correspond respectively to the red, green and blue colours of the original image [30]. Pixels are then assigned to the green (1) or background (0) classes using the ExG threshold value defined with the Otsu algorithm for each session [31].

The Hough transform [32] is used to identify the main alignments corresponding to the rows and find their orientation. The image is then rotated to display the rows horizontally (Figure 2). The number of green pixels in each line is computed to obtain a profile of green pixels across the rows. The peaks of the green pixel profiles are localized using the prior knowledge on row spacing (Row_spacing_prior) to prevent finding unexpected peaks between rows. The prior knowledge of the number of rows per microplot (Row_number_prior) is also used when identifying the peaks. The prior values of row and plant spacing are not always known precisely. Therefore, the row extraction pipeline (Figure 2) provides also updated and more accurate values of Row_spacing_prior for each session. Finally, each row is extracted using the fine-tuned value of the row width.

Plant identification with an object-based method

After the row extraction, the algorithm individualizes the objects (groups of connected pixels) in the image and classifies them as plants or weeds. Weeds are eliminated based on the distance to the row center. If the centroid of an object is located at a distance larger than a threshold value (Minimum_distance_to_row), it is considered as a weed. The threshold value is expressed in relative value to the row spacing and set to 0.25 (Table S2). Objects with dimensions along the row direction larger than the Plant_spacing_prior value (Table 2) are expected to include several plants. The number of plants contained in these big objects is derived from the number of peaks observed when summing the green pixels along the row direction, where a peak may correspond to a plant position. Further, the number of plants found by the number of peaks is crosschecked with the expected number of plants computed by dividing the extension of the object by the Plant_spacing_prior value. Results are illustrated in Figure 3 for the two objects on the right of the bottom row.

Finally, some objects may be located too close together to be considered as separate plants because these objects correspond to several parts of the same plant. Figure 3 illustrates it with the second plant starting from the left on the top row, where a leaf and the main plant are separated. If the distance between the centroids of the closest object is smaller than the maximum acceptable distance, Big_plants_tolerance x Plant_spacing_prior, the two objects are merged as a single plant. Table S2 in the supplementary materials presents the value used for each parameter. The centroid (center of mass of the object), and the bounding box (smallest rectangle that contains all object's pixels) of the objects are finally computed.

Deep-learning method

Model architecture

An object detection method was selected to predict the bounding box around each plant. This information can then be used to derive more traits to characterize every individual plant. Object detection is a fast-growing area within DL techniques since the emergence of networks such as R-CNN (Regions with Convolutional Neural Network , [33] ) or SSD (Single Shot Detector, [34] ). Most DL object detection models fall into one-stage or two-stage models. In the one-stage model, the object is localized and categorized in a single step. In the two-stage model, a first stage detects possible objects, and a second stage categorizes them. The Faster-RCNN two-stage model [21] is used because it performs well in the context of plant phenotyping. Madec et al. [14] used it successfully for counting wheat heads. It allows also to analyze the nature of the possible errors by visualizing them.

Faster-RCNN can be implemented in many forms which can influence the final results. We use the implementation made by the mmdetection library [35] .It contains many detectors, and is written upon PyTorch [36]. The default implementation of the library is used and contains a Feature Pyramidal Network (FPN) [37], which differs from the original paper [21]. It is used to provide object proposition at different scales. A ResNet-34 model [38] was used as the backbone network because it offers a good compromise between accuracy and speed of training. The backbone extracts the deep features which are used by the Region Proposal Network (RPN) to detect potential objects which are then classified as crop or background. All other architectural details are given in the code (https://github.com/EtienneDavid/plants-counting-detection) . We also choose to train one model by crop as preliminary tests show lower performances when mixing the three crops.

Pre-processing and data augmentation

The input image size of the network is set to 512 x 512 pixels to match memory constraints during training. However, images from the microplots are larger. A preprocessing step first splits them randomly into patches of 512 x 512 pixels. For each session in the training dataset, 100 patches were randomly selected which results in a total of 900 patches to train the model for each crop over the nine available sessions. Randomly sampled patches provide more diversity than evenly sampled ones. During the training process, data augmentation is applied to extend the diversity of images. The complete data augmentation pipeline is a set of geometric distortions (Random rotation, Random Translation, Random Shear), blur (Gaussian Blur), noise (Gaussian noise) and colorimetric augmentation (Random hue value, Random contrast). All data augmentation details are given in the code. Once trained, the model is applied to all the patches. Predictions from the overlapping patches are finally merged together by using the Non-Max-Suppression algorithm [39] with an Intersection over Union (IoU) threshold of 0.70.

Hybrid method

DL methods detect individual plants based on many features automatically extracted while HC methods exploit expert prior knowledge on the sowing pattern to eliminate plants located at a nonexpected position between rows. We propose therefore a hybrid method that combines the benefits of both HC and DL ones. The DL method is first applied to detect plants. Then, the HC method presented earlier is used to identify the row position and eliminate all remaining weeds corresponding to plants with centroids located at a larger distance to the row than a threshold value distance_to_row (Table S2).

Evaluation strategy for plant detection

Strategies for training and evaluation

Detection models were developed and evaluated independently for each crop. DL method requires an extensive training dataset that should represent the expected diversity of situations. Due to the limited number of labelled images, two strategies are defined: "Out-Domain" and "In-Domain". "Out-Domain" is the more rigorous strategy where the performances of the DL method are evaluated over sessions not used during the training process. For each crop and each stage, two sessions were used for training and the remaining one for testing. This allows to balance the stages between the training and testing datasets. A three-fold cross-validation strategy that exploits all sessions while providing relatively independent test cases is used. Three different models were trained for each crop using six sessions, representing about 3800 plants, and tested on the remaining three sessions representing around 1900 plants. The "In-Domain" strategy is based on adding few images randomly selected in the testing datasets to the training dataset. It aims at reducing possible lack of representativeness in the training dataset. The same three-fold cross-validation process was used for each crop, except that 1/3 of the 600 plants used previously as testing datasets were added to the training dataset. The remaining 2/3 images (400 plants) are used to evaluate the performances of the models for each crop. The same test dataset (1200 plants corresponding to the 400 test plants for each of the three test sessions) is finally used to compare the Out-domain and In-domain approaches.

Evaluation metrics Detection

The "Centroid matching strategy" (C_MS ) is used to evaluate whether a plant was correctly detected. The C_MS is based on the distance between the centroids of the plants. If the distance between centroids of a detected plant and the closest labelled one is smaller than Plant_distance_prior / 2 it is considered as true positive (TP). Otherwise, it is a false positive (FP). If a labelled plant has no detected plant within a distance smaller than Plant_distance_prior / 2, it is a false negative (FN).

The plant detection performance was quantified per session with the terms of the confusion matrix normalized by the number of labelled plants (TP+FN) for easier comparison between crops and stages, which correspond to rates of TP (TPR), FP (FPR) and FN (FNR). The accuracy is also used, defined as TP/(TP+FN+FP). DL method produces a confidence score for each predicted BB. A box is considered as a prediction for the DL and HY methods if its score is above 0.5.

Plant density

Plant density (PD) was calculated by dividing the number of plants in the microplot by its area. The area is computed as the number of rows multiplied by the row spacing and the row length. The relative root mean square error (rRMSE) is used to compare the estimated and the reference PD values and assess the accuracy of the method. The accuracy levels were split into four classes to better assess the robustness of the method. A rRMSE<5% was considered as good, between 5%<rRMSE<10% as satisfactory, between 10%<rRMSE< 20% as poor, and rRMSE>20% as very poor. The percentile of microplots belonging to each class was therefore used to evaluate the robustness of the methods.

4

Results and discussion

DL and HY methods detect better plants than the HC one

Detection performances are very different depending on the crops (Table 4 and Figure 4). Detection of maize plants appears difficult for the three methods and particularly for HC with a low TPR and a high FNR (Table 4). This is mostly explained by the small size of the plants that overlap, resulting into groups of overlapping plants that are interpreted as a single plant (Figure 5b), or to poor threshold values determined by the Otsu method for the green segmentation used in the first step to identity objects (Figure 5a). However, a high FNR is also observed for the first development stage with the HC method, due to the poor quality of the green segmentation where background artifacts such as small rocks or crop residues were interpreted as plants (Figure 5g). A large variability between the three instances of the three-fold cross validation is observed for this early stage (Figure 4) due to the variability in image quality. Marginal differences are observed between DL and HY methods. They both show relatively balanced FPR and FNR. FPR is mostly explained by possible confusion between plants and their shadows or soil artifacts (Figure 5c) while FNR is explained by the small size of the plants that are difficult to detect (Figure 5d). This results into accuracy values between 0.77 to 0.80 with little variation between stages (Table 4). However, a larger variability across the three instances of the three-fold cross validation is observed for the late stage (Figure 4). Detection of sugar beet plants appears to be much easier, with performances similar between the three methods. The sugar beet crops better verify the assumptions described in 3.2.1. The plots were not infested by weeds (Table 4), which seems to be an important explanation for the success of all methods. A small FPR is observed for the three methods, particularly for the latest stage, which explains the decrease in accuracy (Table 4). This is due to difficulties when plants are overlapping (Figure 5e). Slightly higher FNR is observed for HC corresponding to non-detected plants in the case of small plants and image of poor quality. This is also observed with DL for the very early stages (Figure 5f). The variability across the three instances of the three-fold cross validation is also small (Figure 4). Marginal differences are observed between DL and HY methods mostly because of the good control of weeds. 4 and Figure 4). The HC shows lower TPR and higher FPR and FNR as compared to DL and HY. In the late stage, the HC shows very high FPR corresponding to problems of plant separation when they are overlapping. Further, the weeds close to the row line are not well eliminated and confounded with plants (Figure 5g). Similar problems are observed for the DL method, with weeds confounded 12 with the crop. However, the HY methods allows to eliminate part of the weeds that are located in between rows (Figure 5h). and HY shows high and similar TPR (Table 4). However, a high FPR is also observed for the first stage with the HC method, due to the poor quality of the green segmentation where background artifacts, such as small rocks or crop residues, were interpreted as plants (Figure 4g). Conversely, high FPR are observed for the late stage where DL shows difficulty to detect plants in a group of overlapping ones and confounds weeds with the crop. A large variability between the three instances of the three-fold cross validation is observed for sunflower (Figure 4). It is explained by a high degree of heterogenety in the microplots and between them, as well as between sessions. Image quality appears therefore mandatory for HC methods to get a good segmentation. The HC methods appears also limited to eliminate weeds on the rows and to separate efficiently the overlapping plants. DL methods are similarly limited in separating crops from weeds, with confusions made mostly on unseen type of weeds (Figure 5h). However, the HY methods allows to eliminate part of the weeds.

The DL methods also show some difficulties in detecting plants when they are small or when their shadows or other soil artifacts such as cracks are present. Nevertheless, the DL methods outperform the HC ones in most cases.

Tests were further conducted to evaluate the impact of the four qualitative factors (crop type, development stages, weeds, and soil type) using the p-value computed from a variance analysis. Results show (Table 5) that crop-type is an important factor (p_value smaller than 0.05) for HC and HY, while weeds are important for HC and DL, and soil-type for HC. However, the low number of examples (27 sessions in total), and the non-evenly distribution of the several factors (for instance most examples of 13 high levels of weed infestation are found in sunflower sessions only) prevents from drawing final conclusions. The impact of the four quantitative factors (sowing density, plant size, original resolution, and blurriness) were also evaluated using a Pearson test. It reveals (Table 5) that no factors appear significant (p-value smaller than 0.05), while the lowest p-values are observed for the sowing density and plant size that are closely related to the crop type.

Plant density is better estimated with DL and HY methods

The HC method provides the poorest performances for maize plant density estimation, with rRMSE generally higher than 0.2 (Figure 6), which is consistent with the poorer detection performances (Figure 4). Image acquisition during the early stages tends to degrade the performances conversely to what was observed for the detection (Figure 5). This may be explained by the unbalance between false positives and negatives observed for the early stages (Table 4). Marginal differences are observed between DL and HY methods for maize where weeds were not the main issue.

The rRMSE threshold values to

All the methods reach good performances (rRMSE<0.05) for sugar beet, with even better performances for the two first stages when plants are easily identified and weeds not too developed (Figure 6). The poorer detection performances noticed earlier for HC (Figure 4) do not impact the density estimation because the FPR is well compensated by the FNR.

Sunflower shows more variability between sessions and stages, with rRMSE around 0.1 for the intermediate development stage showing better performances than the early one and moreover than the late one (Figure 6). The models for sunflower are very poor for the session 3_auzeville_2019_5 (Figure 6), mainly because of weed infestation. DL performs better than HC while HY improves marginally the performances for the two early stages, but significantly for the late stage where significant weed infestation was observed.

Overall, our results show lower performances than those of the studies where the training and testing datasets were not independent. For maize detection accuracy between 0.93 and 0.96, and relative counting error around 1.5%, were reported [11], [15] while none of our methods achieve such performances. Similar range of results are obtained on rapeseed (counting error of 6.83%) [12], or safflower with rRMSE approximately under 5% [13]. However, our results with DL and HY are comparable to studies keeping the training and test datasets independent; on maize Gnädinger and Schmidthalter [6] reports a counting error of +/-15%. The HC approach applied when its main assumptions are verified performs well and comparably to DL.

Adding few images from the test domain improves drastically the DL performances

The performances of DL methods are closely related to the number of images used in the training dataset and their representativity of the possible situations [24]. DL method works very well for sugarbeet where all the images were relatively similar across sessions for each development stage. However, the acquisition conditions were quite different from the ones experienced in the other sessions for the sunflower on Epoisses_2019_1, explaining why the DL models had more difficulties to detect plants for this session. The "Out-domain" strategy used previously was compared here to the "In-domain" one where 1/3 of the images of the initial testing sessions were used to finetune the model. Performances are evaluated on the remaining 2/3 images of the initial testing sessions to keep some independence between the training and test datasets.

Note first that the plant density estimation performances (Figure 7) evaluated on a limited test data set (1200 images) are very consistent with the ones presented previously over the full test dataset including 1800 images (Figure 6). Results show that the additional images used in the training process and having similar characteristics as those in the testing dataset decreased significantly the rRMSE for all crops (Figure 7). This outperforms the marginal gain observed with the HY method on few sessions. Training with the In-domain strategy reduces the variability of performances across sessions. The 5% rRMSE value is reached for all crops except maize, where performances are anyway close to this target. Plant overlapping and the small leaf size makes the DL method for maize more challenging. However, there are still some outliers for Maize and Sunflower, corresponding to Pleinefougeres_2019_1 and Epoisses_2019_1 sessions. The images of these two sessions are highly blurred (Table 3) explaining most of their poor detection performances. A large part of this performance can be attributed to the elimination of almost all weeds by the DL methods, without the need of the HY correction, which have learned the pattern of the weeds, instead of relying on the location, and a better recognition of the plants. Our results demonstrate that active learning techniques [39] could greatly improve DL model performances for these new sessions. A small sample of images coming from the new sessions to be processed have to be labelled to complement the training dataset, but more than quantity, it is uniquely due to the diversity: only 40m² of maize or sugarbeet, and between 50 and 100m² of sunflower have been added to the training dataset, leading to a dramatic increase of the performances which cannot be attributed only to the dataset size increase.

Row detection

Prior value of the row spacing as defined in Table 1 Determined in Table 1 Peak prior

Row detection

The fraction of the maximum height of the peaks used to consider a peak as corresponding to a row 0.5

Plant spacing prior

Split object

Prior value of the plant spacing as defined in Table 1 Determined in Table 1 Find plant

Minimum distance to row

Weed elimination Minimum distance from the row centre (expressed relatively to Row spacing prior) to consider the objects as weeds 0.25

Remove false positives

Big Plants Tolerance

Leaves detection

All centroids under Big Plants Tolerance x Plant spacing prior are considered to belong to the same plant 0.9

Table S2. List of parameters used for row extraction and plant identification

Figure S3: Justification of a centroid matching strategy Centroid matching strategy (C_MS) is preferred to the IoU one (IoU_MS)

The C_MS was initially compared with an intersection over union matching strategy (IoU_MS), which is more commin The IoU_MS is based on the Intersection over Union between the detected and labelled BB with a standard threshold of 0.5. A detected plant is considered true positive (TP) if its IoU is larger than 0.5. Otherwise, it is a false positive (FP). If a labeled BB has no overlap with any detected BB, it is classified as false negative (FN).

The size of BB of plants detected by the HC method have different dimensions as compared to the labelled BB (Figure 4, left): The distribution of the size of BB for HC is gaussian, while that of labelled, DL and HY are very similar and skewed with significantly smaller BBs as well as larger ones. That means that the HC is missing small object with the IoU_MS. This resulted into lower values of accuracy computed with IoU_MS (Figure 4, right) because of a significant amount of mismatch between the predicted and reference BBs at IoU=0.5. Rather than adapting the IoU threshold level, the distance between centroids is preferred to evaluate the match between predicted and interactively labeled plants.

The accuracy computed with C_MS (Figure 4, right) is significantly larger than that computed with IoU_MS, particularly for the low accuracy values as well as for the HC method for the reasons exposed above. Therefore, in the following, the centroid distance is used to compute the terms of the confusion matrix and the accuracy. Detailed metrics can be found in Table S2. Table S4. Complete results for the three methods on all sessions. Accuracy, precision and recall are presented with the IoU matching strategy.

Conclusion

Chapter 1 demonstrated that with the correct training information, DL is generally more accurate than handcrafted methods. However, the possible domain shift prevents to use of DL for breeding applications without any human supervision. The hybrid method can slightly improve the robustness capacity of DL methods, but additional data is essential. While the diversity of the proposed dataset was high, the still limited size of the training and testing datasets prevents us from drawing firm conclusions on the factors that degrade performances, including image blurriness, soil type or the presence of artefacts such as rocks.

Our dataset is available on Zenodo https://zenodo.org/record/4890370#.YTm7sBnityw, and we hope it can help the community to benchmark their solutions.

3 Design of a large and diverse dataset for training and evaluating deep learning models: application to wheat head detection

Foreword

Chapter 1 demonstrated the need for a large and diverse dataset to analyze the robustness of the Deep Learning method. Building such datasets for plant phenotyping is rare and mainly dedicated to the controlled conditions installations. For field conditions, it is almost inexistent because up to very recently, acquiring data from HTPP platforms was not as mature as today. Further, the limited labelling capacity of research institutions prevented to build of large datasets. Chapter 2 explores how to re-use the datasets from several sources with different acquisition and labelling protocols. It aims at harmonizing the datasets by applying specific processing of the images and labels. The resulting dataset can then be used to develop robust models to detect wheat heads. Head detection in wheat is essential for estimating the head density and exploring their spatial distribution and the mandatory pre-processing step before characterizing wheat heads. Wheat is also a crop well studied globally, making it possible to cover use cases across contrasted situations globally. Labelled wheat head datasets were already existing, as the one compiled by Madec et al. [1]. However, they were not always readily available on the internet and were not always well documented. The challenge of chapter 2 is to build a large dataset used to train and evaluate models for head detection. For this purpose, we propose solutions for the labelling issue, harmonising the observations and documenting the diversity of the dataset and the possible confounding factors. This dataset will be later used for crowdsourcing models through open challenges. This chapter is made of two successive papers. The first one presents the dataset used for the first wheat head detection challenge. Based on the results of this first challenge, the original dataset was revised, extended and re-organized to serve a second wheat head detection challenge. The second paper describes this improved Global Whead Head Detection dataset.

Global Wheat Head Detection 2020

The detection of wheat heads in plant images is an important task for estimating pertinent wheat traits including head population density and head characteristics such as health, size, maturity stage, and the presence of awns. Several studies have developed methods for wheat head detection from high-resolution RGB imagery based on machine learning algorithms. However, these methods have generally been calibrated and validated on limited datasets. High variability in observational conditions, genotypic differences, development stages, and head orientation makes wheat head detection a challenge for computer vision. Further, possible blurring due to motion or wind and overlap between heads for dense populations make this task even more complex. Through a joint international collaborative effort, we have built a large, diverse, and well-labelled dataset of wheat images, called the Global Wheat Head Detection (GWHD) dataset. It contains 4700 high-resolution RGB images and 190000 labelled wheat heads collected from several countries around the world at different growth stages with a wide range of genotypes. Guidelines for image acquisition, associating minimum metadata to respect FAIR principles, and consistent head labelling methods are proposed when developing new head detection datasets. The GWHD dataset is publicly available at http://www.global-wheat .com/and aimed at developing and benchmarking methods for wheat head detection.

Introduction

Wheat is the most cultivated cereal crop in the world, along with rice and maize. Wheat breeding progress in the 1950s was vital for food security of emerging countries when Norman Borlaug developed semidwarf kinds of wheat and a complementary agronomy system (the Doubly Green Revolution), saving 300 million people from starvation [1]. However, after increasing rapidly for decades, the rate of increase in wheat yields has slowed down since the early 1990s [2,3]. Traditional breeding still relies to a large degree on manual observation. Innovations that increase genetic gain may come from genomic selection, new high-throughput phenotyping techniques, or a combination of both [4][5][6][7]. These techniques are essential to select important wheat traits linked to yield potential, disease resistance, or adaptation to abiotic stress. Even though high-throughput phenotypic data acquisition is already a reality, developing efficient and robust models to extract traits from raw data remains a significant challenge. Among all traits, wheat head density (the number of wheat heads per unit ground area) is a major yield component and is still manually evaluated in breeding trials, which is labour intensive and leads to measurement errors of around 10% [8,9]. Thus, developing image-based methods to increase the throughput and accuracy of counting wheat heads in the field is needed to help breeders manipulate the balance between yield components (plant number, head density, grains per head, grain weight) in their breeding selections.

Thanks to increases in GPU performance and the emergence of large-scale datasets [10,11], deep learning has become the state of the art approach for many computer vision tasks, including object detection [12], instance segmentation [13], semantic segmentation [14], and image regression [15,16]. Recently, several authors have proposed deep learning methods tailored to image-based plant phenotyping [17][18][19]. Several methods have been proposed for wheat head quantification from high-resolution RGB images. In [8,9], the authors demonstrated the potential to detect wheat heads with a Faster-RCNN object detection network. They estimated in [8] a relative counting error of around 10% for such methods when the image resolution is controlled. In [20], the authors developed an encoder-decoder CNN model for semantic segmentation of wheat heads which outperformed traditional handcrafted computer vision techniques. Gibbs et al. [21] developed a wheat head detection and probabilistic tracking model to characterize the motion of wheat plants grown in the field.

While previous studies have tested wheat head detection methods on individual datasets, in practice, these deep learning models are difficult to scale to real-life phenotyping platforms, since they are trained on limited datasets, with expected difficulties when extrapolating to new situations [8,22,23]. Most training datasets are limited in terms of genotypes, geographic areas, and observational conditions. Wheat head morphology may significantly differ between genotypes with notable variation in head morphology, including size, inclination, colour, and the presence of awns. The appearance of heads and the background canopy also change significantly from emergence to maturation due to ripening and senescence [24]. Further, planting densities and patterns vary globally across different cropping systems and environments, and wheat heads often overlap and occlude each other in fields with higher planting densities.

A common strategy for handling limited datasets is to train a CNN model on a portion of a phenotyping trial field and test it on the remaining portion of the field [25]. This is a fundamental flaw of empirical approaches against causal models: there is no theoretical guarantee that a CNN model is robust on new acquisitions. In addition, a comparison between methods from different authors requires large datasets. Unfortunately, such large and diverse phenotyping head counting datasets do not exist today because they are mainly acquired independently by single institutions, limiting the number of genotypes, the environmental and the observational conditions used to train and test the models. Further, because the labelling process is burdensome and tedious, only a small fraction of the acquired images are processed. Finally, labelling protocols may be different between institutions, which will limit model performance when trained over shared labelled datasets.

To fill the need for a large and diverse wheat head dataset with consistent labelling, we developed the Global Wheat Head Detection (GWHD) dataset that can be used to benchmark methods proposed in the computer vision community. The GWHD dataset results from the harmonization of several datasets coming from nine different institutions across seven countries and three continents. This paper details the data collection, the harmonization process across image characteristics and labelling, and the organization of a wheat head detection challenge. Finally, we discuss the issues raised while generating the dataset and propose guidelines for future contributors who wish to expand the GWHD dataset with their labelled images.

Dataset Composition

2.1. Experiments. The labelled images comprising the GWHD dataset come from datasets collected between 2016 and 2019 by nine institutions at ten different locations (Table 1) covering genotypes from Europe, North America, Australia, and Asia. These individual datasets are called "sub-datasets." They were acquired over experiments following different growing practices, with row spacing varying from 12.5 cm (ETHZ_1) to 30.5 cm (USask_1). The characteristics of the experiments are presented in Table 1. They include low sowing density (UQ_1, UTokyo_1, UTokyo_2), normal sowing density (Arvalis_1, Arvalis_2, Arvalis_3, INRAE_1, part of NAU_1), and high sowing density (RRes_1, ETHZ_1, part of NAU_1). The GWHD dataset covers a range of pedoclimatic conditions including very productive context such as the loamy soil of the Picardy area in France (Arvalis_3), silt-clay soil in mountainous conditions like the Swiss Plateau (ETHZ_1), or Alpes de Haute Provence (Arvalis_1, Arvalis_2). In the case of Arvalis_1, Arvalis_2, UQ_1, and NAU_1, the experiments were designed to compare irrigated and water-stressed environments.

2. Image Acquisition. The GWHD dataset contains RGB images captured with a wide range of ground-based phenotyping platforms and cameras (Table 2). The height of the image acquisition ranges between 1.8 m and 3 m above the ground. The camera focal length varies from 10 to 50 mm with a range of sensor sizes. The differences in camera setup lead to a range of Ground Sampling Distance (GSD) ranging from 0.10 to 0.62 mm with the half field of view along the image diagonal varying from 10 °to 46 °. Assuming that wheat heads are 1.5 cm in diameter, the acquired GSDs are high enough to detect heads and even awns visually. Although all images were acquired at the nadir-viewing direction, some geometric distortion may be observed for a few sub-datasets due to the different lens characteristics of the cameras used.

Data

Harmonization. An important aspect of assembling the GWHD dataset was harmonizing the various subdatasets (Figure 1). A manual inspection of images was first conducted to ensure that they could be well interpreted. Images acquired at too early of a growth stage were removed when heads were not clearly visible (Figure 2(d)). Most of the images were also acquired before the appearance of head senescence since heads tend to overlap when the stems start to bend at this stage. Object scale, i.e., the size of the object in pixels, is an important factor in the design of object detection methods [8]. Object scale depends on the size (mm) of the object and on the resolution of the image. Wheat head dimensions may vary across genotypes and growth conditions, but are generally around 1.5 cm in diameter and 10 cm in length. The actual image resolution, at the level of wheat heads, varied significantly between sub-datasets: the GSD varies by a factor of 5 (Table 1) while the actual resolution at the head level also depends on canopy height and the panoramic effect of the camera. The panoramic effect will be much larger when images were acquired too close to the canopy. Images were therefore rescaled to keep more similar resolution at the head level. Bilinear interpolation was used to up-or downsample the original images. The scaling factor applied to each sub-dataset is displayed in Table 3.

Most deep learning algorithms are trained with squaresized image patches. When the original images were cropped into square patches, the size of the patches was selected to reduce the chance that heads would cross the edges of the patches and be partly cut off. Images were therefore split into 1024 × 1024 squared patches containing roughly 20 to 60 heads each, with only a few heads crossing the patch edges. The number of patches per original image varied from 1 to 6 depending on the sub-dataset (Table 3). These squared patches will be termed "images" for the remainder of the paper.

Labelling.

A web-based labelling platform was developed to handle the evaluation and labelling of the shared subdatasets using the coco annotator (https://github.com/ jsbroks/coco-annotator; [26]). The platform hosts all the tools required to label objects. In addition, it also grants simultaneous access to different users, thus allowing contributions from all institutions. Wheat heads were interactively labelled by drawing bounding boxes that contained all the pixels of the head. Labelling is difficult if heads are not clearly visible, i.e., if they are masked by leaves or other heads. We did not label partly hidden heads unless at least one spikelet was visible. This was mostly the case for images acquired at an early stage when heads were not fully emerged. Overlap among heads was more frequently observed when the images were acquired using a camera with a wide field of view as in UTokyo_2 or ETHZ_1. These overlaps occurred mainly towards the borders of the images with a more oblique view angle. When the bounding box was too large to include the awns, it was restricted to the head only (Figure 2(a)). Further, heads cropped at the image edges were labelled only if more than 30% of their basal part was visible (Figure 2(e)).

Several institutions had already labelled their subdatasets. For the datasets not labelled, we used a "weakly supervised deep learning framework" [27] to label images efficiently for these sub-datasets. A YoloV3 model [28] was 5 Plant Phenomics trained over UTokyo_1 and Arvalis_1 sub-datasets and then applied to the unlabelled sub-datasets. Boxes with an associated confidence score greater than and equal to 0.5 were retained and proposed to the user for correction. This semiautomatic active learning increased the throughput of the labelling process by a factor of four as compared to a fully manual process. The process is detailed in Figure S1.

This first labelling result was then reviewed by two individuals independent from the sub-datasets institution. When large discrepancies between reviewers were observed, another labelling and reviewing round was initiated. Approximately 20 individuals contributed to this labelling effort. This collaborative process and repeated reviews ensure a high level of accuracy and consistency across the sub-datasets. 3). It represents 188445 labelled heads which average 40 heads per image in good agreement with the 20 to 60 targeted heads per image. However, the distribution among and within sub-datasets is relatively broad (Figure 3(a)). We included about 100 images that contain no heads to represent in-field capturing conditions and add difficulty for benchmarking. Few images contain more than 100 heads with a maximum of 120 heads. Multiple peaks corresponding to the several sub-datasets (Figure 3(b)) can be observed due mainly to variations in head density that depends on genotypes and environmental conditions. The size of the bounding boxes around the heads shows a slightly skewed Gaussian distribution with a median typical dimension of 77 pixels (Figure 3(b)). The typical dimension is computed as the square root of the area. It corresponds well to the targeted scale, i.e., 1:5 cm × 10 cm approximate head size with an average resolution close to 0.4 mm/pixel which represents a typical dimension of 97 pixels per head, although the simple horizontal area projected does not correspond exactly to the viewing geometry of the RGB cameras. The harmonization of object scale across sub-datasets can be further confirmed visually in Figure 4.

Description of the Dataset

Diversity of Sampled Genotypes, Environments, and

Developmental Stages. The diversity of acquisition conditions sampled by the GWHD dataset is well illustrated in Figure 4: illumination conditions are variable, with a wide range of heads and background appearance. Further, we observe variability in head orientation and view directions, from an almost nadir direction up to a mostly oblique direction as in the case of ETHZ_1 (Figure 4). A selection of bounding boxes extracted from the several sub-datasets (Figure 5) shows a variation of bounding-box area and aspect ratio, depending on the head orientation and viewing direction. A large diversity of head appearance is observed, with variation in the presence of awns and awn size, head colour, and blurriness. In addition, a few heads were cut off when the bounding box crossed the edge of the image.

Comparison to Other Datasets.

Several open-source datasets have already been proposed in the plant phenotyping community. The CVPPP datasets [29] have been widely used for rosette leaf counting and instance segmentation. The KOMATSUNA dataset also includes segmented rosette leaves, but in time-lapse videos [30]. The Nottingham ACID Wheat dataset includes wheat head images captured in a controlled environment with individual spikelets annotated [17]. However, comparatively few open-source datasets include images from outdoor field contexts, which are critical for the practical application of phenotyping in crop breeding and farming. A few datasets have been published for weed classification [31,32]. The GrassClover dataset includes images of forage fields and semantic segmentation labels for grass, clover, and weed vegetation types [33]. Datasets for counting sorghum [27,34] and wheat heads [35] have also been published with dot annotations.

Plant Phenomics

In terms of phenotyping datasets for object detection, our GWHD dataset is currently the largest open labelled dataset freely available for object detection for field plant phenotyping. MinneApple [36] is the only comparable dataset in terms of diversity in the field of phenotyping but proposes fewer images and less diversity in terms of location. Other datasets like MS COCO [37] or Open Images V4 [38] are much larger and sample many more object types for a wide range of other applications. The corresponding images usually contain fewer objects, typically less than ten per image (Figure 6). 7 Plant Phenomics However, some specific datasets like PUCPR [39], CARPK [40], and SKU-110K [41] are tailored to the problem of detecting objects (e.g., cars, products) in dense contexts. They have a much higher object density than the GWHD dataset, but with fewer images for PUCPR and CARPK, while SKU-110 contains more images than our GWHD dataset (Figure 6). The high occurrence of overlapping and occluded objects is unique to the GWHD dataset. This makes labelling and detection more challenging, especially compared to SKU-110K, which does not seem to present any occlusion. Finally, wheat heads are complex objects that have a wide variability of appearance as demonstrated previously, surrounded by a very diverse background which would constitute a more difficult problem than detecting cars or densely packed products on store shelves.

Target Use Case: Wheat Head Detection Challenge

The main goal of the dataset is to contribute to solving the challenging problem of wheat head detection from highresolution RGB images. An open machine learning competition will be held from May to August 2020 to benchmark wheat head detection methods using the GWHD dataset for training and testing (http://www.global-wheat.com/2020challenge/).

Split between Training and Testing Datasets.

In machine learning studies, it is common to randomly split a dataset into training and testing samples. However, for the GWHD competition, we specifically aim to test the performance of the method for unseen genotypes, environments, and observational conditions. Therefore, we grouped all images from Europe and North America as the training dataset, which covers enough diversity to train a generic wheat head detection model. This training dataset corresponds to 3422 images representing 73% of the whole GWHD dataset images. The test dataset includes all the images from Australia, Japan, and China, representing 1276 images to evaluate model performance, including robustness against unseen images.

Evaluation Metrics.

The choice of bounding boxes as labels in the GWHD dataset allows it to be used for object detection. The mean average precision computed from the true and false positives is usually used to quantify performance in object detection tasks. A true positive corresponds to a predicted bounding box with an intersection over union (IoU) greater than and equal to 0.5 with the closest labelled bounding box. A false positive corresponds to a predicted bounding box with an IoU strictly lower than 0.5 with the closest labelled bounding box. In the case of two predicted boxes with an IoU greater than or equal to 0.5 on the same bounding box, the most confident one is considered as a true positive and the other as a false positive. The mean Average Precision noted as mAP@0.5 is the considered metric for evaluating the localization performance. Detection of individual wheat heads is required for characterizing their size, inclination, colour, or health. However, the number of wheat heads per image is also a highly desired trait. Future competitions using the GWHD dataset could focus on wheat head counting with metrics such as the Root Mean Square Error (RMSE), relative RMSE (rRMSE), and Coefficient of Determination (R 2 ) to quantify the performance of object counting methods.

Baseline Method.

To set a baseline detection accuracy for the GWHD dataset, we provide results based on a standard object detection method. We trained a two-stage detector, Faster-RCNN [12], with a ResNet34 and ResNet50 as the backbone. Faster-RCNN is one of the most popular object detection models and used in Madec et al. [8]. ResNet34 is used along with ResNet50 because it is less prone to overfitting and faster to train. Due to memory constraints, the input size was set to 512 × 512 pixels. We randomly sampled ten patches of size 512 × 512 pixels for each image in the training dataset resulting in a training dataset composed of 34220 patches. We predicted on a set of overlapping patches of size 512 × 512 pixels regularly extracted from the test images of size 1024 × 1024 pixels and then merged the results. After 10 epochs, representing 342200 iterations in total, the best model is obtained at epoch 3 for both backbones. It yielded a mAP@0.5 of 0.77 and a mean RMSE of 12.63 wheat heads per image which corresponds to rRMSE = 39%. The coefficient of regression is 0.57. All results are provided in Figure S2. The relatively poor performance of a standard object detection network on the GWHD dataset provides an opportunity for substantial future improvement with novel methods. The GWHD competition is expected to instigate new wheat head detection approaches that will provide more accurate results. 8 Plant Phenomics

Discussion

Image Acquisition Recommendations.

To successfully detect wheat heads, they should be fully emerged and clearly visible within the images, with minimum overlap among heads and leaves. For some genotypes and environmental conditions, we observed that the wheat stems tend to bend for the latest grain filling stages, which increases the overlap between heads. Conversely, for the stages between heading and flowering, some heads are not yet fully emerged and are therefore difficult to see. Therefore, we recommend acquiring images immediately after flowering when the wheat heads have fully emerged and are still upright in the field.

For image acquisition, a near nadir viewing direction is recommended to limit the overlap between heads, especially in the case of populations with high head density. Likewise, a narrow field of view is preferred. However, a narrow field of view may result in a small image footprint when the camera is positioned at a height close to the top of the canopy. Therefore, we recommend increasing the camera height to get a larger sampled area and reduce the number of heads that will be cropped at the edge of the image. The size of the sampled area is important when head identification is used for estimating the head population density. The minimum sampled area should be that of our squared patch, i.e., 1024 × 1024 pixels of 0.4 mm/pixel which corresponds to an area of about 40 cm 2 . To achieve this sampled area, while maintaining a narrow field of view of ±15 °, the distance between the camera and the top of the canopy should be around 1.0 m. However, a larger sampling area is preferable for head population density estimation, where at least 100 cm 2 should be sampled to account for possible heterogeneity across rows. This would be achieved with a 2.5 m distance between the camera and the top of the canopy.

When estimating wheat head density, i.e., the number of heads per unit ground area, accurate knowledge of the sampled area is critical. The nonparallel geometry of image acquisition, with significative "fisheye" lens distortion effects, induces uncertainty about the sampled area. Even for our typical case with limited distortion effects (±15 °field of view), for an image acquired at 2.5 m from the top of the canopy, an error of 10 cm in canopy height estimation induces 8% error in the sampled area, which directly transfers to the head density measurement. Further, the definition of the reference height at which to compute the sampled area is still an open question, because within a population of wheat plants, the heights of the heads can vary by more than 25 cm, which induces a 21% difference in the sampled area between the lowest and highest head. Further work should investigate this important question.

Finally, our experience suggests that using a submillimetre resolution at the top of the canopy is required for efficient head detection. However, the optimal resolution is yet to be defined. Previous work [8] recommended 0.3 mm GSD, while the GWHD dataset includes GSD ranging from 0.28 to 0.55 mm. Further work should investigate this important aspect of wheat imaging, particularly regarding the possibility to use UAV observations for head density estimation in large wheat breeding experiments.

Camera characteristics:

Model Focal length of the lens (mm) Size of the pixel at the sensor matrix (μm)

Sensor dimensions (pixels × pixels)

5.2. Minimum Information Associated with the Sub-datasets and FAIR Principles. The FAIR principles (Findable, Accessible, Interoperable, and Reusable [42]) should be applied to the images that populate the GWHD dataset. A minimum set of metadata should be associated with each image as proposed in [43] to verify the FAIR principles. The lack of metadata was an issue for precise data harmonization and is limiting factor for further data interpretation [44] and possible meta-analysis. Therefore, we recommend attaching a minimum set of information to each image and sub-dataset. In our case, a sub-dataset generally corresponds to an image acquisition session, i.e., a series of images acquired over the same experiment on the same date and with the same camera.

The experiment metadata are all the metadata related to agronomic characteristics of the session; the acquisition metadata are all the metadata related to the camera and acquisition vehicle used. Both can be defined at the session level and the image level. Our recommendations are summarized in Table 4. We encourage attaching more metadata such as camera settings (model, white balance correction, et al.) when possible because it adds context for further data reuse.

Need for GWHD Expansion.

The innovative and unique aspect of the GWHD dataset is the significant number of contributors from around the world, resulting in a large diversity across images. However, the diversity within each continent and environmental conditions is not well covered by the current dataset: more than 68% of the images within the GWHD dataset come from Europe and 43% from France. Further, some regions are currently missing, including Africa, Latin America, and the Middle East. As future work, we hope to expand the GWHD dataset in order to get a more comprehensive dataset. Therefore, we invite potential contributors to complement the GWHD dataset with their subdatasets. The proposed guidelines for image acquisition and the associated metadata should be followed to keep a high level of consistency and respect the FAIR principles. We encourage potential contributors to contact the corresponding authors through http://www.global-wheat.com. We also plan to extend the GWHD dataset in the future for classification and segmentation tasks at the wheat head level, for instance, the size of the wheat head or flowering state. This expansion would require an update of the current labels.

Conclusion

Object detection methods for localizing and identifying wheat heads in images are useful for estimating head density in wheat populations. Head detection may also be considered as a first step in the search for additional wheat traits, including the spatial distribution between rows, the presence of awns, size, inclination, colour, grain filling stage, and health. These traits may prove useful for wheat breeders and some may help farmers to better manage their crops.

In order to improve the accuracy and reliability of wheat head detection and localization, we have assembled the Global Wheat Head Detection dataset-an extensive and diverse dataset of wheat head images. It is designed to develop and benchmark head detection methods proposed by the community. It represents a large collaborative international effort. An important contribution gained through the compilation of diverse sub-datasets was to propose guidelines for image acquisition, minimum metadata to respect the FAIR principles and guidelines, and tools for labelling wheat heads. We hope that these guidelines will enable practitioners to expand the GWHD dataset in the future with additional sub-datasets that represent even more genotypic and environmental diversity. The GWHD dataset has been proposed together with an open research competition to find more accurate and robust methods for wheat head detection across the wide range of wheat growing regions around the world. The solutions proposed in the competition will be made open-source and shared with the plant phenotyping community.

The Global Wheat Head Detection (GWHD) dataset was created in 2020 and has assembled 193,634 labelled wheat heads from 4700 RGB images acquired from various acquisition platforms and 7 countries/institutions. With an associated competition hosted in Kaggle, GWHD_2020 has successfully attracted attention from both the computer vision and agricultural science communities. From this first experience, a few avenues for improvements have been identified regarding data size, head diversity, and label reliability. To address these issues, the 2020 dataset has been reexamined, relabeled, and complemented by adding 1722 images from 5 additional countries, allowing for 81,553 additional wheat heads. We now release in 2021 a new version of the Global Wheat Head Detection dataset, which is bigger, more diverse, and less noisy than the GWHD_2020 version.

Introduction

Quality training data is essential for the deployment of deep learning (DL) techniques to get a general model that can scale on all the possible cases. Increasing dataset size, diversity, and quality is expected to be more efficient than increasing network complexity and depth [1]. Datasets like ImageNet [2] for classification or MS COCO [3] for instance detection are crucial for researchers to develop and rigorously benchmark new DL methods. Similarly, the importance of getting plant-or crop-specific datasets is recognized within the plant phenotyping community ([4-10], p. 2, [11][12][13]). These datasets allow benchmarking the algorithm performances used to estimate phenotyping traits while encouraging computer vision experts to further improvement ( [10], p. 2, [14][15][16][17]). The emergence of affordable RGB cameras and platforms, including UAVs and smartphones, makes in-field image acquisition easily accessible. These high-throughput methods are progressively replacing manual measurement of important traits such as wheat head density. Wheat is a crop grown worldwide, and the number of heads per unit area is one of the main components of yield potential. Creating a robust deep learning model performing over all the situations requires a dataset of images covering a wide range of genotypes, sowing density and pattern, plant state and stage, and acquisition conditions. To answer this need for a large and diverse wheat head dataset with consistent and quality labeling, we developed in 2020 the Global Wheat Head Detection (GWHD_2020) [18] that was used to benchmark methods proposed in the computer vision community and recommend best practices to acquire images and keep track of the metadata.

The GWHD_2020 dataset results from the harmonization of several datasets coming from nine different institutions across seven countries and three continents. There are already 27 publications (accessed July 2021) that have reported their wheat head detection model using the GWHD_2020 dataset as the standard for training/testing data. A "Global Wheat Detection" competition hosted by Kaggle was also organized, attracting 2245 teams across the world [14], leading to improvements in wheat head detection models [23,25,31,41]. However, issues with the GWHD_2020 dataset were detected during the competition, including labeling noise and an unbalanced test dataset.

To provide a better benchmark dataset for the community, the GWHD_2021 dataset was organized with the following improvements: (1) the GWHD_2020 dataset was checked again to eliminate few poor-quality images, (2) images were re-labeled to avoid consistency issues, (3) a wider range of developmental stages from the GWHD_ 2020 sites was included, and (4) datasets from 5 new countries (the USA, Mexico, Republic of Sudan, Norway, and Belgium) were added. The resulting GWHD_2021 dataset contains 275,187 wheat heads from 16 institutions distributed across 12 countries.

Materials and Methods

The first version of GWHD_2020, used for the Kaggle competition, was divided into several subdatasets. Each subdataset represented all images from one location, acquired with one sensor while mixing several stages. However, wheat head detection models may be sensitive to the developmental stage and acquisition conditions: at the beginning of head emergence, a part of the head is barely visible because it is still not fully out from the last leaf sheath and possibly masked by the awns. Further, during ripening, wheat heads tend to bend and overlap, leading to more erratic labeling. A redefinition of the subdataset was hence necessary to help investigate the effect of the developmental stage on model performances. The new definition of a subdataset was then formulated as "a consistent set of images acquired over the same experimental unit, during the same acquisition session with the same vector and sensor." A subdataset defines therefore a domain. This new definition forced to split the original GWHD_2020 subdatasets into several smaller ones. The UQ_1 was split into 6 much smaller subdatasets, Arvalis_1 was split into 3 subdatasets, Arvalis_3 into 2 subdatasets, and utokyo_1 into 2 subdatasets. However, in the case of utokyo_2 which was a collection of images taken by farmers at different stages and in different fields, the original subdataset was kept. Overall, the 11 original subdatasets in GWHD_2020 were distributed into 19 subdatasets for GWHD_2021.

Almost 2000 new images were added to GWHD_2020, constituting a major improvement. A part of the new images comes from the institutions already contributing to GWHD_2020 and was collected during a different year and/or at a different location. This was the case for Arvalis (Arvalis_7 to Arvalis_12), University of Queensland (UQ_7 to UQ_11), Nanjing Agricultural University (NAU_2 and NAU_3), and University of Tokyo (Utokyo_1). In addition, 14 new subdatasets were included, coming from 5 new countries: Norway (NMBU), Belgium (Université of Liège [46]), United States of America (Kansas State University [47], TERRA-REF [7]), Mexico (CIMMYT), and Republic of Sudan (Agricultural Research Council). All these images were acquired at a ground sampling distance between 0.2 2

Plant Phenomics and 0.4 mm, i.e., similar to that of the images in GWHD_ 2020. Because none of them was already labeled, a sample was selected by taking no more than one image per microplot, which was randomly cropped to 1024 × 1024 px patches that will be called images in the following for the sake of simplicity.

With the addition of 1722 images and 86,000 wheat heads, the GWHD_2021 dataset contains 6500 images and 275,000 wheat heads. The increase in the number of subdatasets from 18 to 47 leads to a larger diversity between them which can be observed on Figure 1. The subdatasets are described in Table 1. However, the new definition of a subdataset led also to more unbalanced subdatasets: the smallest (Arvalis_8) contains only 20 images, while the biggest (ETHZ_1) contains 747 images. This provides the opportunity to possibly take advantage of the data distribution to improve model training. Each subdataset has been visually assigned to several development stage classes depending on the respective color of leaves and heads (Figure 2): postflowering, filling, filling-ripening, and ripening. Examples of the different stages are presented in Figure 2. While being approximative, this metadata is expected to improve model training.

Dataset Diversity Analysis

In comparison to GWHD_2020, the GWHD_2021 dataset puts emphasis on metadata documentation of the different subdatasets, as described in the discussion section of David et al. [18]. Alongside the acquisition platform, each subdataset has been reviewed and a development stage was assigned to each, except for Utokyo_3 (formerly utokyo_2) as it is a collection of images from various farmer fields and development stages. Globally, the GWHD_2021 dataset covers well all development stages ranging from postanthesis to ripening (Figure 2).

The diversity between images within the GWHD_2021 dataset was documented using the method proposed by Tolias et al. [48]. The deep learning image features were first extracted from the VGG-16 deep network pretrained on the ImageNet dataset that is considered representing well the general features of RGB images. We then selected the last layer which has a size of 14 × 14 × 512 and summed it into a unique vector of 512 channels, which is then normalized. Then, the UMAP dimentionality reduction algorithm [49] was used to project representations into a 2D space. The UMAP algorithm is used to keep the existing clusters during the projection to a low-dimension space. This 2D space is expected to capture the main features of the images. Results (Figure 3) demonstrate that the test dataset used for GWHD_2020 was biased in comparison to the training dataset. The subdatasets added in 2021 populate more evenly the 2D space which is expected to improve the robustness of the models.

Presentation of Global Wheat Challenge 2021 (GWC 2021)

The results from the Kaggle challenge based on GWHD_ 2020 have been analyzed by the authors [14]. The findings emphasize that the design of a competition is critical to enable solutions that improve the robustness of the wheat head detection models. The Kaggle competition was based on a metric that was averaged across all test images, without 3 Plant Phenomics distinction for the subdatasets, and it was biased toward a strict match of the labelling. This artificially enhances the influence on the global score of the largest datasets such as utokyo_1 (now split into Utokyo_1 and Utokyo_2). Further, the metrics used to score the agreement with the labeled heads and largely used for big datasets, such as MS COCO, appear to be less efficient when some heads are labeled in a more uncertain way as it was the case in several situations depending on the development stage, illumination conditions, and head density. As a result, the weighted domain 5 Plant Phenomics accuracy is proposed as a new metric [14]. The accuracy computed over image i belonging to domain d, AI d ðiÞ, is classically defined as

AI d i ð Þ = TP TP + FN + FP , ð1Þ
where TP, FN, and FP are, respectively, the number of true positive, false negative, and false positive found in image i.

The weighted domain accuracy (WDA) is the weighted average of all domain accuracies:

WDA = 1 D 〠 D d=1 1 n d * 〠 n d i=1 AI di , ð2Þ
where D is the number of domains (subdatasets) and n d is the number of images in domain d. The training, validation, and test datasets used are presented in Section 5.

The results of the Global Wheat Challenge 2021 are summarized in Table 2. The reference method is a faster-RCN with the same parameters than in the research paper GWHD_2020 [18] and trained on the GWHD_2021 (Global Wheat Challenge 2021 split) training dataset. The full leaderboard can be found at https://www.aicrowd.com/ challenges/global-wheat-challenge-2021/leaderboards.

How to Use/FAQ

(i) How to download? The dataset can be download on Zenodo: https://zenodo.org/record/5092309

(ii) What is the license of the dataset? The dataset is under the MIT license, allowing for reuse without restriction (iii) How to cite the dataset? The present paper can be cited when using the GWHD_2021 dataset. However, cite preferentially [18] for wheat head detection challenges or when discussing the difficulty to constitute a large datasets (iv) How to benchmark? Depending on the objectives of the study, we recommend two sets of training, validation, and test (Table 3):

(a) The Global Wheat Challenge 2021 split when the dataset is used for phenotyping purpose, to allow direct comparison with the winning solutions (b) The "GlobalWheat-WILDS" split is the one used for the WILDS paper [50]. We recommand to use the GlobalWheat-WILDS split when working on outof-domain distribution shift problems It is further recommended to keep the weighted domain accuracy for comparison with previous works.

Conclusion

The second edition of the Global Wheat Head Detection, GWHD_2021, alongside the organization of a second Global Wheat Challenge is an important step for illustrating the usefulness of open and shared data across organizations to further improve high-throughput phenotyping methods.

In comparison to the GWHD_2020 dataset, it represents five new countries, 22 new subdatasets, 1200 new images, and 120,000 new-labeled wheat heads. Its revised organization and additional diversity are more representative of the type of images researchers and agronomists can acquire across the world. The revised metrics used to evaluate the models during the Global Wheat Challenge 2021 can help researchers to benchmark one-class localization models on a large range of acquisition conditions. GWHD_2021 is expected to accelerate the building of robust solutions. However, progress on the representation of developing countries is still expected and we are open to new contributions from South America, Africa, and South Asia. We started to include nadir view photos from smartphones, to get a more comprehensive dataset and train reliable models for such affordable devices. Additional works are required to adapt such an approach to other vectors such as a camera mounted on unmanned aerial vehicle, or other high-resolution cameras working in other spectral domains. Further, it is planned to release wheat head masks alongside the bounding box given the very large number of boxes that already exists and provides more associated metadata. Ethz_1, Rres_1, Inrae_1, Arvalis (all), NMBU (all), ULiège-GxABT (all) UQ_1 to UQ_6, Utokyo (all), NAU_1, Usask_1 UQ_7 to UQ_12, Ukyoto_1, NAU_2 and NAU_3, ARC_1, CIMMYT (all), KSU (all), Terraref (all) GlobalWheat-WILDS Ethz_1, Rres_1, Inrae_1, Arvalis (all), NMBU (all), ULiège-GxABT (all) UQ (all), Utokyo (all), Ukyoto_1, NAU (all) CIMMYT (all), KSU (all); Terraref (all), Usask_1, ARC_1 7 Plant Phenomics

Conclusion

Chapter 2 demonstrated how to build process and organize a valuable dataset for crowdsourcing wheat head detection models. The resulting dataset is one of the largest and diverse acquisition protocols, genotype, location for field phenotyping. Even for a structure such as CAPTE (INRAe / Arvalis / Hiphen), one of the main contributors of the Global Wheat Head Dataset, the gain in diversity is expected to increase the accuracy and robustness of the wheat head density models. The next step is to analyze the winning solutions provided by the two challenges, emphasising the domain shift problem.

Competition design to train robust Deep Learn model: the example of the Global Wheat Challenges

Foreword

As presented in the introduction, several techniques, including data augmentation, model architecture and choice of the hyper-parameters, can affect the final robustness of a DL model. Rather than relying on a local team to experiment with each modality to get the best model, we organized open challenges to crowdsource the best solutions. Two competitions based on the Global Wheat Head Challenge 2020 and 2021 have been organized to gather the best strategies to tackle the domain shift problem. Chapter 3 explores the output of the two Global Wheat Challenges.

Challenges are conducted on platforms that link Data scientists, people with a solid Deep Learning background, with significant problems. As the competitors are expected to focus on the DL training and optimize a single metric, the design of the competition should incentivize them to solve the domain shift problem with a meticulously designed metric. Moreover, the data split should reflect the expected domain shift. In chapter 3, the performance of the winning solutions of two challenges and the impact of the competition design are analyzed.

Global Wheat Challenge 2020 and 2021: Analysis of the competition design and winning models

The following manuscript will be submitted the "GigaSciences" journal.

Introduction

Crowdsourcing is an increasingly popular approach for scientists to make advances in their field by collecting diverse raw or labelled data [1]- [4], solving problems that are difficult for algorithms but easy for humans, such as protein folding [5], or accessing large-scale distributed computing power [6].

Crowdsourcing of data analysis has increased rapidly in recent years due to the popularity of Big Data challenges [7] on web platforms such as Kaggle or Codalab. In particular, problems that are amenable to machine learning approaches, such as computer vision problems, have been promoted and popularised through open competitions such as ImageNet or COCO [8], [9]. , 2016). In this context, crowdsourcing of data and analysis helps to connect domain experts, e.g. plant scientists, with computer and data scientists. Such collaboration with data scientists outside the traditional scope of plant phenotyping is essential to solve fundamental data science problems within the domain.

Early computer vision competitions for plant phenotyping problems [10], [11] such as leaf counting [12] and leaf segmentation [10] were highly successful. Plant phenotyping competitions have typically focused on indoor, controlled single plant images although a few recent competition have included outdoor field data such as CVPR Agriculture-Vision competition [13] or the GrassClover competition [14]. Expanding data competition to agricultural applications are crucial for facing the new challenges of global food production in the context of global change. The availability of sensors, vectors, and data for agriculture applications is rapidly expanding, while effective data interpretation pipelines are still limited. Therefore, crowdsourcing new approaches could help addressing the data interpretation challenge in plant phenotyping and agriculture.

Despite the use of deep learning in recent plant phenotyping studies, the robustness and generalizability of these methods remains an open question, particularly for small plant datasets. The issue of robustness in trained plant phenotyping models has been difficult to study due to a lack of real-world datasets [START_REF] Geirhos | Shortcut learning in deep neural networks[END_REF]David et al., 2021). A large and diverse dataset is required to study the robustness problem. Most of the Deep Learning algorithms require a large set of labelled images to be trained on, such as ImageNet [8] or MS COCO [9]. The process of labelling such images is long and tedious, limiting the availability of large training datasets. The presence of diversity in the datasets is also important to study the robustness of the models. While large datasets in plant phenotyping are already available for in-door conditions [12], [17], only few comprehensive ones exist for field conditions.

Detecting wheat head under field conditions is highly desired by breeders and agronomists: it allows estimating the head density, one of the main yield components for wheat; it allows also localising plants and describing the emergence pattern with consequences on plant competition and microplot heterogeneity; it is finally the first step before further characterisation of the heads. Several studies proposed successful methods from high resolution RGB images and deep learning methods [18], [19]. However, the training and testing datasets were limited, and it is unclear if their results apply to new datasets because of the possible variation of sensors, illumination conditions, development stages and genotypes could impact the performance as described in [16].

We compiled the Global Wheat Head Dataset (GWHD) [20], [21] to study the robustness problem in plant phenotyping. The GWHD is a large and comprehensive labelled dataset for wheat head localisation. Based on the GHWD, we organised two challenges to attract a large cohort of ML practitioners to solve the wheat head detection problem: the Global Wheat Challenge 2020 (GWC_2020) on Kaggle, which took place from 4th May to 4th August 2020 and attracted up to 2245 competitors and the Global Wheat Challenge 2021 (GWC_2021) on AIcrowd, which took place from 4 th May to 4 th July and attracted up to 432 competitors. This paper summarises the competitions, including a description of the most successful approaches employed. We further evaluate their robustness on two additional test datasets. Finally, we discuss lessons learned from this competition and provide recommendations for future ones.

3 Material and Methods

Datasets

Three datasets are used in the study: The Global Wheat Head Dataset, the Wheat Head Frame Dataset, and the Arvalis LITERAL dataset (table 1). Only the UQ frame dataset includes both annotated images and infield ground counting achieved over the same 0.25 x 0.25 m sampling area. The GWHD-2021 dataset have no in-field counting, while the Arvalis-LITERAL have no image annotation. More details on the three datasets are provided below.

Global Wheat Head Dataset

The Global Wheat Head Detection dataset used for the competition was extensively presented in [20], [21].

It contains 6515 high-resolution RGB images representing 275.187 wheat heads from 16 institutions across 5 continents and 12 countries (Table 2). It contains 47 sub-datasets corresponding to an image acquisition session where images are acquired at a single date over a single site with a single camera system. The images were taken from one to two meters from the soil, with different high-resolution RGB cameras providing a ground sampling distance between 0.1 to 0.43 mm. The images were carefully labelled by several operators to ensure consistency and reliability. The latest version is openly available on Zenodo (Global Wheat Head Dataset 2021 | Zenodo).

UQ Frame Dataset

The UQ frame dataset is a collection of images acquired on three locations for a total of five dates across Australia from august to October 2020 (Table 3). For each experiment, a frame was placed in a wheat microplot and a RGB image was taken from nadir view with a Sony RX0 camera. It provides a ground sampling distance around 0.3 mm/px. Each image is rotated and cropped to the limit of the frame. Different frames were used and presented in table 2. A human operator counted the number of wheat head in-field, and each processed images were also manually labelled with bounding boxes with the same methodology as described in DAVID et al., 2020. These images belong to the UQ_7 to UQ_11 sub-datasets corresponding to single acquisition sessions used in the test dataset during the 2021 competition. A sample of the dataset is shown in figure 1.

Arvalis LITERAL Dataset

The Arvalis LITERAL Dataset is a collection of RGB images collected on five locations for a total of 6 different sessions of measurement in France from May to June 2021 (Table 4). A set of 4 images per microplot was acquired with a Sony RX0 with a resolution of 1424px x 1424 px, and an estimated canopy sampling distance (CSD) of 0.3mm. The distance between the sensor and the canopy was estimated with a stereovision algorithm and was provided with the dataset. A sample of the dataset is shown in figure 2. The head density for each image was computed as the number of heads in the image divided by the corresponding footprint area derived from the distance to the canopy and the focal length. Finally, the head density per microplot was averaged over the four images. Reference in-field head density was measured by counting the heads over two adjacent rows of 1 m length segment corresponding to a 0.35 m² sampling area. Despite progress in wheat head detection thanks to deep learning algorithms [18], [19], no large and diverse dataset was available to study the robustness of their solutions. The Global Wheat Head Detection 2020 [20] and 2021 [21] were the first datasets designed to match the need for more variety. The two challenges organised in 2020 and 2021 were designed to crowdsource the best solutions for wheat head detection. The experience gained with the 2020 challenge was crucial in improving both the dataset and the metrics used to compare the proposed solutions. The first challenge took place on Kaggle from 4 th May to 4 th August 2020 and gathered 2235 competitors with a cash prize of 15.000 USD using the GWHD_2020. The second challenge was held on AIcrowd from 4th May to the 4th of July 2021 and drew 432 contestants with a cash prize of 4.000 USD using GWHD_2021. Different aspects of a challenge organisation impact the output solutions: the cash prize, the platform and the easiness to onboard the competition greatly influence the total number of participants.

Most of the rules applied for both challenges: the competitors could access a training dataset composed of images from Europe and submit their predictions on a separate dataset consisting of images from North America, Africa, Asia and Oceania. This set was blindly split into a public test and a private test. For each submission during the competitor, the competitor could obtain a score on the public test set, but the final ranking was made once on the private test set, later called test dataset. Competitors could get their score on the public test set up to 50 times to optimize few hyperparameters, later called for this reason validation dataset. The winning solutions were expected to be opensource, with a MIT licence for the GWC_2020 and any open source licence allowing unrestricted reuse for the GWC_2021. The labels were a set of boxes around all wheat heads that can be used to train Deep Learning detection algorithms such as Faster-RCNN [22].

Two main changes occurred between the GWC_2020 and the GWC_2021 regarding the measure of robustness: the data volume and split and the metric evolved to match the original purpose of the competition.

• Data volume and split. The dataset used increased from 4700 images to 6515 and from 18 to 47 sub-datasets. It represents an addition of 9 sub-datasets for training and 6 for the private test set. Further, the public and private test sets are entirely disjoint in GWC_2021, while public and private test sets were randomly drawn from the same sub-datasets. The European sub-datasets cover all necessary development stages. The diversity in terms of data owner on both test set in GWC_2021 help to avoid overfitting on specific conditions. The notion of sub-datasets, as defined in [21], was introduced to understand better the factors controlling the model robustness: a subdataset is composed of a unique acquisition session, i.e. images acquired on a single site, at a single date, with specific illumination conditions, and with the same system. However, for the Kaggle competition, the sub-datasets were not as clearly defined.

• The metrics used. The Intersection over Union (IoU) ratio is used to define the confusion matrix terms. A true positive (TP) is a labelled bounding box that matches a predicted one with an IoU ratio larger than the threshold value. A false positive (FP) is a predicted bounding box having an IoU ratio lower than the threshold with any labelled bounding box. A false negative (FN) is a labelled bounding box having an IoU ratio lower than the threshold with any predicted bounding box. In the case of GWC_2020; the accuracy at the image level, 𝐴 𝑖 , is the average of the accuracy across IoU values ranging from 0.5 to 0.75 with 0.05 steps.

𝐴 𝑖 = 1/6 ∑ 𝑇𝑃(𝐼𝑜𝑈) 𝑇𝑃(𝐼𝑜𝑈)+𝐹𝑃(𝐼𝑜𝑈)+𝐹𝑁(𝐼𝑜𝑈) 𝐼𝑜𝑈=0.75 𝐼𝑜𝑈=0.5
Note that the accuracy is also the true positive rate (TPR). Then the accuracy at the image level was averaged over all the images of the test dataset to get the metric used in the GWC_2020, 𝐴𝐴 2020 ;

𝐴𝐴 2020 = 1 𝑛 * ∑ 𝐴 𝑖 𝑛 𝑖=1
, For GWC_2021, the accuracy is calculated for a unique IoU threshold of 0.5 to favor detection over overfitting on a particular labelling style. Further, because the test dataset was composed of several subdatasets having very different sizes (Table 1 andTable 2), we proposed to use an average domain accuracy, 𝐴𝐴 2021 : during GWC_2021 competition, based on the analysis of the GWC_2020 metric (Supplementary material 1).

𝐴𝐴 2021 = 1 𝐷 ∑ 1 𝑛 𝑑 𝐷 𝑑=1 * ∑ 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 𝑑𝑖 𝑛 𝑑 𝑖=1
Images from the UQ frame dataset and Arvalis LITERAL dataset are larger than the 1024px X 1024 px image size used during the challenge. Two strategies are used to handle such use cases. The baseline and GWC_2020 required a specific adaptation where the large image is split into a set of 1024px patches with a stride of 800px, representing an overlap of 22%. Models are then applied on each patch and predictions are merged with a Non-Max Suppression algorithm, using the same Intersection over Union value than used within the model (0.92 for GWC_2020 and 0.7 for Faster-RCNN). The solution GWC_2021 can automatically adapt to images with higher resolution thanks to the use of adaptive pooling layer and adaptive convolution layer. Part of the improved performances comes from test-time augmentation combined with the weighted boxes fusion (Solovyev, Wang and Gabruseva, 2019) that was used on the six winning solutions. Winners used several data augmentation techniques such as Mixup [START_REF] Zhang | mixup: Beyond Empirical Risk Minimization[END_REF] and Mosaic augmentation, described in YoloV4 (Bochkovskiy, Wang and Liao, 2020). It is, however, difficult to assess the effect of the specific data augmentation strategies on robustness. We identified this particular topic as a question to investigate to find a better baseline strategy than the proposed Faster-RCNN. Winners all used pseudolabelling. Pseudo-labelling [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF] is the practice of converting predictions to labels over the test set called "pseudo-labelled" data and then fine-tuning the model with a mix of training data and pseudolabelled data. Further, an ensemble approach was proposed by five winners over six, where several models were trained on different subsampling of the training dataset, and their solutions were fused into a single solution. Despite being widely used for data competition, these techniques are not standard in plant phenotyping.

Additional strategies were developed during the competition. First, the participants optimized few hyperparameters including the score threshold, the IoU threshold, and the image size. None of the winning solutions used the same set of hyperparameters as the proposed baseline solution. For instance, the second winner of GWC_2021 (david_jeon) upsampled the images to 1600px before prediction, which could help detect small wheat heads. Another popular approach during GWC_2020 was to generate more diversity from the training dataset based on "jigsaw puzzle" techniques. Given that some images were cropped from an original larger one, competitors have recreated images by re-cropping new patches randomly instead of applying a regular grid as in the baseline solution preprocessing step (David et al., 2020). The performances measured with the average accuracy (AA2021) are very heterogeneous between domains while the ranking of AA2021 across domains is similar for all the three approaches considered here (Figure 3). It appears that the level of complexity of an acquisition domain depends on its inner characteristics and all the solutions considered experienced difficulties in such situations. However, the GWC_2021 always beats the other solutions except for Terraref_2 and NAU_3 domains. Conversely, the baseline solution has always lower AA2021 values than the two winning solutions. The GWC_2020 solution show generally intermediate AA2021 except for few domains with values close to that of GWC_2021, and even better for Terraref 2 and NAU 3 domains.

While the results just discussed correspond to those obtained over the test dataset, it is interesting to compare them to those of the training and validation datasets: it provides some insight on the possible domain shifts and the robustness. The AA2021 values of the training dataset always get the best accuracy as expected. This is also observed for the false positive and false negative rates that are lower than those of the training and test datasets (Figure 3). The validation dataset used for hyperparameter optimization and pseudo-labelling has AA2021 values in between the training and the test datasets. The domain shift evaluated by the difference in AA2021 between the training and test datasets is reduced for GWC_2021, but significant for the two other solutions. The GWC_2021 solution appears more robust than the baseline and GWC_2020 models. The better AA2021 obtained for GWC_2021 is mostly coming from a reduction of the false negatives (Figure 3 bottom): GWC_2021 gets always the lower FNR values across all the domains. The GWC_2020 and baseline solutions show similar FNRs. Conversely, the GWC_2020 gets always the smallest false positive rates, while the baseline solution shows the higher values of FPR (Figure 3, middle). The GWC_2021 solution shows intermediate, but significant values of FPRs. Combining the GWC_2021 and GWC_2020 solutions could be a possible pertinent solution when selecting the bounding boxes to be kept. This will be investigated in a future work.

Part of the low performance comes from the specificity of certain subdatasets, which are more complex than others. Some common patterns can be found: first, low resolution is a factor in the difficulty of Arvalis_4. Some subdatasets such as UQ_1 and UQ_2 contain many empty images, which lower or increase enormously the accuracy on the subdataset. The combination of wheat head bending and awns explain Arvalis_11, UQ_5 and UQ_7. Additionally to these difficulties, Terraref_1 and Terraref_2 suffer from intense illumination resulting in stark contrasts in the images. Finally, for some subdatasets (Terraref_1, Terraref_2, UQ_1, Ukyoto_1), a part of the wheat heads are not out of the stalk, despite the majority of the wheat heads being out.

The false negative rates indicate the number of heads that were not detected by the model. Even with the best model represented by GWC_2021, all the FNRs are larger than 0.10, except on four sessions. This is still a strong limitation for an accurate detection of the heads, explaining why AA2021 reaches values ranging from 0.2 to 0.8, with an average close to 0.7. The distribution of the size of the missing boxes (Figure 4) for all the solutions show that they generally correspond to smaller heads that are more difficult to detect. The missed heads are about 35% smaller than all the heads. If the GWC_2021 detects twice more small wheat heads, it is still missing some of the smaller heads. The missing heads are often underdeveloped wheat heads or hidden wheat heads (Figure 5 top line). The difficulty of this class of small objects is the uncertainty attach to the labeling itself. Some of the labeled heads could be considered as non-wheat heads for some observer. Playing with the IoU threshold may partly contribute to improve the solution for the missed heads. However, improving the detection of the smaller wheat heads may increase the false positive rates, i.e. detecting heads that do not exist. The false positive are generally small bounding boxes that contain features close to those of actual heads (Figure5, bottom line), for instance in FP-A, a curved wheat head is detected twice. In FP-B and FP-C, a leaf part is mistaken for a wheat head. The balance between false positive and false negative is difficult to get as illustrated by the GWC_2021 that reduces the number of FNR at the expense of an increase of FPR.

Conversely, GWC_2020 have low FPR at the expense of an increase of FNR. A solution that would reduce both FNR and FPR is still expected, while the balance between both terms is critical when counting heads for wheat head density estimation. False negatives and positives are indicated by red bounding-boxes GWC_2020 and GWC_2021 are very similar and use several common ideas: for the data augmentation, a mosaic augmentation, described in supplementary material S2, is applied; A pseudo-labelling procedure is applied during inference test time augmentation; Weighted boxes fusion is used. It also relies on more than one model to make the prediction. The differences are the architecture, with Yolov5 (GWC_21) instead of EfficientDet and the sampling to train the several sub-models. In GWC_2021, the selection is made at the domain level instead of the image level, explaining why the various networks used during inference have more contrasted performances. Interpreting carefully which part of the approach makes the GWC_2021 solution more robust is difficult as the hazard plays probably a significant role. However, regarding the much lower number of competitors for GWC_2021 compared to GWC_2020, the quality of the solution is much satisfactory.

Performances for head counting

The rRMSE for each domain were calculated on all splits, including the training and validation splits (Table 7). The rRMSE varies widely across domains, from 0.66 to 2.15. A more detailed inspection of the rRMSE as a function of the model and the dataset split (Figure 6) shows as expected that the training dataset gets always the lower rRMSE values with no outliers except for GWC_2020: all the models are capable to learn the specificities of the several training domains with only small differences across the three models. The performances on the validation dataset used for hyperparameter tuning and pseudo-labeling degrade significantly, with very large outliers for the baseline and GWC_2021 models. Some domains appear difficult since they are probably too different from those of the training dataset: UQ_2 in the validation dataset shows the largest outliers for the baseline and GWC_2021 models. For the test dataset, the variability between models and domains is still large (Figure 6). The GWC_2021 model presents the lower rRMSE values and a moderate dispersion between domains.

It seems the more robust solution in agreement with the detection performances presented earlier. The baseline model shows also relatively good performances except for three outliers including Terraref_2 that gets also large rRMSE for the two other models. The GWC_2020 solution presents the worst performances on the test dataset. The outliers are similar to the ones reported in 4.2. The accuracy for plant detection appears not to be a good indicator of the counting performances of a model. While very poor accuracy corresponds also to high rRMSE for head counting (Figure 7), when the accuracy is higher than 0.5, there is no relationship with the rRMSE (Figure 7). For few domains such as NAU_2 and NAU_3, the rRMSE is above 0.2 despite excellent accuracy. This is mostly due to the false negative and false positive that are strongly imbalanced (Figure 3). The counting performances achieved with GWC_2021 over the test dataset are not as good as reported in other studies such as Faster-RCNN (Madec et al. [18]: rRMSE=0.06) or DeepCount (Sadeghi-Tehran et al. [23]: rRMSE=0.11). However, similar performances are observed for some domains for CIMMYT, ARC, KSU and UQ test datasets. The complexity of generalization over a wide range of conditions corresponding to the several sessions considered in the test dataset makes the detection problem much harder to solve.

Comparison with head density measurements in the field

The standard low throughput method for head density measurements is based on head counting over relatively small sampling area as described earlier in the dataset section. Two datasets, UQ frame dataset that includes images from UQ_7 to UQ_11 (Table 3) used in the validation dataset, and Literal handheld system dataset, acquired concurrently to Arvalis_1 to Arvalis_12 (used for training) but with a different RGB camera. Only the GWC_2021 model is presented here since we already demonstrated that it outperformed the baseline and GWC_2020 solutions. The performances are evaluated at the level of the microplots sampled within UQ and Literal datasets.

2.

Uncertainties in the area used to compute the density. For the UQ dataset, this uncertainty does not exist since the same frame was used to count heads in the field and to crop the images for head labeling and head prediction. For the Literal dataset, the area sampled by the image was computed from the focal length and size of the CMOS matrix that are well known and the knowledge of the distance between the top of the canopy and the camera. This distance was estimated from the distance between the camera and the canopy measured with few cm accuracy. Therefore, the error induced on the area computation should be small, on the order of few percent. 3. Uncertainties in the model to detect heads as seen in the previous section. 4. Differences between heads visible in the RGB image and heads counted in the field. The occluded heads are expected to be more frequent in dense crops (and high head density) as well as when the heads are bending as observed for many genotypes and conditions for the later maturity stages. 5. Errors in head counting by operators in the field. This may increase with the head density with the fatigue of operators as well as possible occlusions of heads. When the head density measured by an operator in the field is compared to that derived from the RGB images labelled by another operator over the same area delimited by the frame used for the UQ dataset, the agreement is only approximative with rRMSE=21.5%. and a slight overestimation of the head density on the images (Figure 9). Although an underestimation was expected due to the occluded heads in the images as compared to what an operator can see in the field with the possibility to change the point of view and even to penetrate in the canopy, the overestimation is surprising. It may be explained by heads missed by the operator in a systematic way. When counting in the field, the operator must identify each head that are placed approximately randomly in the horizontal plane (see figures 1 and 2). It is therefore not an easy task to avoid double counting and not miss any head. Comparatively to counting on the images where the operator identifies heads incrementally with a bounding box or any other marker. This prevents double counting and limits the risk to miss heads visible on the image. A comparison between the rRMSE obtained over the UQ dataset for head counting in the frames (Figure 1) shows that the best match is observed between the head counting estimated by the GWC_2021 solution and the head counting by the operator on the images in term of correlation. Although some occluded heads may not be included in the head counting on the images, this way seems to be a better reference as compared to counting in the field where the errors made by the operator appears relatively high. Further, counting in the field is tedious and is generally done once by an operator on a limited sampling area. Conversely, counting on images can be repeated by several operators, which should improve the reliability of the result. Therefore, counting on images appears more reliable, and the confidence that we could have on counting in the field considered a reference must be altered. However, the bias between prediction and head counting in images is more significant than between the head counting in the field and the head counting on images. It means that detecting small wheat heads is a common problem for humans and the DL method. Further works are required to improve the relationship between counting in the field and counting on images by replicating on earlier stages: three dates on five were acquired during the ripping stage.

Conclusion

The Global Wheat Challenge 2020 and 2021 were important steps toward a robust solution to wheat head detection from high resolution RGB imagery. It complements similar initiatives focusing on plant seedling classification [11], plant pathology classification [24] , Agriculture-vision challenge [13], or plant species recognition [25], [26]. The Global Wheat Challenge 2020 and 2021 attracted a lot of attention to a central problem in Plant Phenotyping and contributed to expose the question to a broader community, including that specialized in image processing based on artificial intelligence algorithms. It is unique in terms of the diversity of in-field situations.

The design of the competition has evolved between the two editions. Additional datasets were used in the GWC_2021, creating more diversity and more images for training. The session, i.e., images acquired with the same system at the same date and location, was recognized as a key structuring factor that defines domains. Greater attention was therefore paid on the balance between sessions in terms of the number of available images. Conversely, in the GWC_20 edition, Utokyo_1 and Utokyo_2 that contained most of the images in 2020 created an "overfit" artifact. The metrics and the split were also improved to better tackle the robustness of the models.

The three winning models of both editions were made open-source and can therefore be used by the community.

The corresponding solution and weights are available on Github (https://github.com/ksnxr/GWC_solution). The proposed solutions are not very innovative in terms of model architecture, based on standard one step model such as Yolo and EfficientDet. They use specific data augmentation techniques to increase robustness, including mixup, cutmix and mosaic. Pseudo-labeling was also used to increase the diversity and size of the training dataset. Ensemble approaches were also part of the winning solutions, where several models are fused to get a more robust solution. These several elements of the winning solutions can be applied for other problems including additional traits and crops. It however requires to access to a minimal set of metadata used to define the domains [20], [21] Solutions based on new architectures focusing on small objects that are more difficult to detect, still need to be explored. Similarly, robust algorithms such as GDRO [27] , Deep CORAL [28] should also be investigated. Further, the information of the domains/sessions was not intensively exploited during the two competitions, although it is expected to increase the robustness of the algorithms. However, the format of a challenge is perhaps not optimal for such approaches where an ensemble approach could be developed based on a series of models trained, validated, and tested on several splits using a multi-fold cross validation approach. In this case, keeping a private split for the competition will reduce the size of the training and validation datasets. Nevertheless, the Global Wheat Head Detection 2021 used for the GWC_2021 is available on the toolbox WILDS [29] to explore more advanced algorithms. Some progress was observed on robustness in the GWC_2021 winning solution as compared to the baseline and GWC_2020 solutions both for head detection and counting. Further we demonstrated that these techniques based on high-resolution imagery were more reliable than the standard low-throughput head counting in the field by an operator: it prevents from systematic errors made by the operator and allows to increase the size of the sample used to compute the average head density at the microplot level. Further, the use of images makes the counting process more traceable as compared to counting directly in the field. Nonetheless, our study demonstrates that for well-defined conditions of acquisition, GWC_21 can be used as a replacement of manual measurement and is already used operationally both at INRAe and Arvalis. The GWC_21 solution is 14 times faster to compute compared to GWC_20 with less than one second per image on computers equipped with a Geforce 1080 GTX or a Geforce 3090 RTX Graphical processing unit board.

However, the performances on the training dataset with accuracy lower than 0.8 for detection and rRMSE larger than 0.10 for counting still needs significant improvement for the models to be used operationally and allow deciphering small differences between genotypes or modalities. Better understanding of the impact of the acquisition system, i.e., camera type, setting, resolution is required to develop both image normalization pre-processing and efficient data augmentation techniques. Further, using bounding boxes to identify the heads may be also replaced efficiently by point identification that presents the advantage to take less time for the labeling. Further, the problems related to the score threshold and IoU are simplified in this case. Finally, at least for head counting, regression models similar to Tasselnetv2 [19] and its extension Tassenetv3 [30] or the ones used for ACID [31] may be also an interesting approach, as previous studies [32] already demonstrate superiority of regression for GWHD 2020.

Acknowledgements

The work received support from ANRT for the CIFRE grant of Etienne David, co-funded by Arvalis. The study was partly supported by several projects, including: The design of a competition is critical to obtain solutions that satisfy its objective. The metric proposed during the Kaggle competition presents the drawback of not having an open implementation. Our open re-implementation from scratch reaches similar but unidentical scores. These small changes can lead to severe changes in ranking. The results presented in figure 3-A reflect how the corrections and the reimplementation could have drastically changed the ranking on the Kaggle annotations.

Domain generalisation is a core problem that the dataset aims to solve but the metric chosen for the Kaggle challenge was too heavily influenced by the performance on the largest domain -utokyo_1. The use of weighted accuracy promotes solutions that have more balanced performance. Figure 3-B displays how the weighted accuracy is less saturated than the original metric with performance varying between 0.3 and 0.6 while the AA varies between 0.65 and 0.7.

Applying these results would have strongly influenced the final ranking.

[Praxis](Overfeat), which has ranked 2nd and VinBigDataMedical (DungNB) which has ranked 1st would have ranked 1st and 3rd, respectively. Peculiarly, the solution that was ranked 9th would be 2 nd , and the 3 rd place solution would have dropped below a rank of 100. These results demonstrate the robustness of the solutions by Praxis and VinBigDataMedical despite the original metric. However, the former metric could have discouraged even more robust solutions from rising to the top. The solution by Praxis will be studied closely in the rest of the paper because it scored significantly higher (+0.03) than the second solution on the weighted accuracy. It will be called the "Kaggle solution" in the rest of the study. Henceforth, the score will be computed with the weighted accuracy on the whole public and private test set.

Impact of data augmentation: ablation study

The 20 solutions getting the highest AGWC_2020 were reviewed to evaluate the impact of the data augmentation strategies on the model detection performances. About 40 types of augmentation techniques were used, most of them being applied to single images, while three others require a several images to be exploited concurrently. The three multi-image augmentation techniques were selected for further analysis. About 15 single-image augmentation techniques were selected to represent the main types of techniques used, while eliminating those that were a priori not pertinent in our context such as RandomSnow or RandomSunFlare. The single-image data augmentation techniques were implemented using a popular image processing library known as Albumentations.

The multi-image augmentation techniques (Mixup, Cutmix) were used for Overfeat solution (Table 3, Figure 2). Mixup [START_REF] Zhang | mixup: Beyond Empirical Risk Minimization[END_REF] and Cutmix (Yun et al., 2019) both mix 2 images to form an augmented image. While Mixup uses a weighted sum of the pixels from the two images, Cutmix replaces a patch in one image with a randomly cut patch from another image. In this study the weights for the Mixup operation were set to a constant, 0.5, although it could also be drawn from a random distribution for each image. The Mosaic augmentation technique is an extension of Cutmix and takes a batch of 4 images, randomly scales each image and patches them together into a single image. Because of its close nature to cutmix, it has not been included in the ablation study. The annotations for each image are proportionately scaled and stacked together to form the augmented annotations.

The EfficientDet-D4 [35] object detection model, pretrained on the Microsoft Common Objects in Context (MS-COCO) dataset [9] was selected as the baseline model for evaluating the effects of these data augmentation techniques on the wheat detection task using the GWHD dataset. All data augmentation strategies used in this study were applied only to the training dataset, with a probability of 0.5. Only one augmentation per group of similar operations is selected randomly for each batch. The impact of Data augmentation is important to improve the quality of the robustness. The classic data augmentation increases the weighted accuracy by 0.045, almost 10% of the baseline score. Advanced data augmentation techniques such as Cutmix [34] or Mixup [33] do not seem beneficial for the trainingwhile Cutmix alone improves performance by 0.023, it does not add any performance compared to classic data augmentation techniques. An interesting finding is that some data augmentation can decrease the robustness the use of Mixup always decreases the performance compared to the baseline data. Our results suggest that while the use of Cutmix or Mixup theoretically increases robustness in the use case of classification, it does not seem to translate for detection; it yields marginal gains. Conclusions are similar when using the same accuracy as in the Kaggle challenge. The result is contradictory to the popularity of such approaches during the Kaggle competition.

In our experiment, Mixup could be drawn with a probability of 0.5, while it's reduced to 0.165 in the case of the winning solution. The quality of our classic data augmentation pipeline, which is inspired by the strongest solutions could also explain the results. Our results confirm the importance of Data Augmentation for robustness but call for more careful exploration when applying usual, typically multi-image, Data Augmentation techniques. It is particularly important to remember that results on classification tasks may not translate well on detection. The use of bounding boxes may also limit the use of strategies such as Cut, Paste and Mix to increase the diversity of the data. A potential axis of research would be to generate synthetic wheat heads to increase the diversity of the data. The use of GAN or Style Transfer may be a promising solution.

Conclusion

The challenges' approach is an empirical tentative at obtaining robust models. We demonstrated the importance of the competition design on the question. The results also show the need for more in-field validation data to measure the actual error between the DL prediction and the effective number of wheat heads. Our approach on the database constitution and challenges' organization can be scaled for more traits than wheat head counting. It is, however, not tackling more conservative strategies to train robust algorithms for plant phenotyping. Data challenges' are time-consuming but rewarding in terms of optimization with a high return on value. In the chapter, we will try to tackle the problem more directly by optimizing the training for robustness. • In domain generalization: the training and test distributions include data from related but distinct domains. This problem arises naturally in many applications, as it is generally not possible to collect a training set that covers equally all domains of interest. For example, in medical applications, it is common to train a model on patients from few hospitals, and then deploy it more broadly to hospitals outside the training set [2]; in wildlife monitoring, an animal recognition model is trained on images from one set of camera traps to be then applied to new camera traps [3]. The wheat head detection falls into the first category of subpopulation shift. • In subpopulation shift: we consider test distributions that are subpopulations of the training distribution, with the goal of keeping good performances for the worst-case subpopulation. For example, it is well-documented that standard models often perform poorly on under-represented demographics [4]; [5], and so we might seek models that can perform well on all demographic subpopulations

Existing ML benchmarks for domain shifts

Domain shifts have been a longstanding problem in the ML research community [6]; [7]. Earlier work studied shifts in datasets for tasks including part-of-speech tagging citemarcus93treebank, sentiment analysis [8], land cover classification [9], object recognition [10], and flow cytometry [11]. However, these datasets are not as widely used today, in part because they tend to be much smaller than modern datasets. The GlobalWheat-WILDS corresponds to the domain generalization type of domain shift: we aim to generalize to test domains that are disjoint from the training domains. However, to make this problem tractable, the training and test domains are expected to be similar. A robust model should minimize the average error on the test distribution.

The aim of the study is to construct train, validation and test splits allowing to demonstrate the existence of a significant performance drops in standard models trained via empirical risk minimization (ERM) loss function. In a second step, two alternatives loss functions are tested on GlobalWheat-WILDS, to improve the robustness of the models.

Materials and methods

In this section, we detailed the contribution of the GlobalWheat-WILDS to investigate the domain shift problem.

The GlobalWheat-WILDS dataset

The GlobalWheat-WILDS dataset comprises 6,515 images containing 275,187 wheat heads. Most of the dataset is coming from the GWHD_2021 dataset described in chapter 2. The main difference is the repartition of the different domains in the training, validation, and test datasets. In total, 47 acquisition sessions coming from 16 research institutes across 12 countries are available. We describe the metadata and statistics of each acquisition session in Table 11. Many factors contribute to the variation in wheat appearance across acquisition sessions with substantial variation due to differences in wheat genotypes, growing conditions (e.g., planting density), illumination conditions, sensors, and measurement protocols.

We consider the domain generalization setting, where the goal is to learn models that generalize to images taken from new acquisition sessions. The task is wheat head detection, which is a single-class object detection task. Concretely, the input is an overhead outdoor image of wheat plants, with labels corresponding to bounding box coordinates that enclose the wheat heads (the spike at the top of the wheat plant containing grain), excluding the hair-like awns that may extend from the head. The domain specifies an acquisition session, which corresponds to a specific location, time, and sensor for which a set of images were collected. Our goal is to generalize to new acquisition sessions that are unseen during training. In particular, the dataset split should capture a shift in location, with training domain made with images from countries different from those used to build the test domain.

The data splits used for GlobalWheat-WILDS

The goal of the GlobalWheat-WILDS, as part of the WILDS collection, is to provide a training, validation and test splits that present significant domain shifts to quantify the possible performance drop. Two main domains were defined:

• The in domain (ID) mages from 18 acquisition sessions in Europe (France ×13, Norway ×2, Switzerland, United Kingdom, Belgium), containing 147 957 wheat heads across 3 300 images. The ID was then split into two subdatasets by randomly drawing images from the 18 sessions: -ID1: used for training the models. This is the largest dataset.

-ID2: used as a test dataset for testing the intrinsic performances in the case of Train to train setting. • The out of domain (OOD). Two subdomains were identified:

-OOD1: used for the validation, i.e. optimization of the hyperparameters. It contains Images from 7 acquisition sessions in Asia (Japan × 4, China × 3) and 1 acquisition session in Africa (Sudan), containing 44,873 wheat heads across 1,424 images. -OOD2: used for the test, i.e. performance evaluation. It contains Images from 11 acquisition sessions in Australia and 10 acquisition sessions in America (USA × 6, Mexico × 3, Canada), containing 66,905 wheat heads across 1,434 images. The several splits were differently used to serve the two main objectives targeted: • Evaluating the performance drop. In this case, two models were compared:

-Train to test: the model is trained over ID1, validated on OOD1, and tested on OOD2. The separation between OD1 and OD2 was preferred to provide more robustness of the model by allowing optimizing the hyperparameters on an OOD dataset not used for testing the performances. -Mixed to test: the model is trained over a mix of ID1 and OOD2, validated on OOD1, and tested on OOD2. The hyperparameters used as the same than train to test. • Evaluating algorithms expected to improve robustness. The model is trained on ID1 with validation on OOD1 and tested on ID2 (Train to train) and OOD2 (Train to test). The following table 2 describes the details of the splits.

The detection model and hyperparameters

For the GlobalWheat-WILDS, we use the Faster-RCNN detection model [40], which has been successfully applied to the wheat head localization problem [30], [41] To train, we fine-tune a model pre-trained with ImageNet, using a batch size of 4, a learning rate of 10 -5 , and weight decay of 10 -3 for 10 epochs with early stopping. The hyperparameters were chosen from a grid search over learning rates 10 -6 , 10 -5 , 10 -4 and weight decays 0, 10 -4 , 10 -3 . We report results aggregated over 3 random seeds.

Training algorithms used

Additionally to the study of the performance drop, we compared several training algorithms used during the learning process, some being designed to get a more robust model.

• Empirical Risk Minimization (Equation 1) is the classic training algorithm. algorithm h on x i , y i is the corresponding label and L is the loss function.

R emp (h) = 1 n n i=1 L(h(x i ),y i ) (1)
• Group Distributional Robustness Optimization (Group DRO) algorithm (Equation 2) is proposed by Sagawa [42] as an alternative to ERM to minimize the loss on the worst group for each batch in contrast to the ERM algorithm. It is equivalent to calculate the empirical risk for each group (g 1 ,...,g n ) and minimize the maximum.

R GDRO (h) = max(R emp (h g 1 ), ..., R emp (h gn )) (2)
• Deep CORAL [43] proposes to train Deep Learning model by adding a second loss term which forces to harmonize the features statistics of different domains. In practice, it minimizes the difference between the covariance matrices of two different groups in each batch. It has not been used in a detection use case to our knowledge. To adapt the algorithm, we propose to use the features of the encoder part of Faster-RCNN, used by the RPN, to calculate the Deep CORAL objective. Our framework is summarized on figure 24. Note that the encoder used for Deep CORAL was applied here on dimensions superior to 10.000, while Deep CORAL was designed for classification use case with a limited dimensionality, between 512 and 4096.

Metrics used to evaluate the performances

We used the average accuracy similarly to what was proposed previously for the Global Wheat Challenges with the dataset [44] . The accuracy of a bounding box detection is measured at a fixed Intersection over Union (IoU ) threshold of 0.5. This permissive IoU threshold of 0.5 was selected because of the uncertainties regarding the precise outline of wheat head instances due to the stem and awns extending from the head. The accuracy of an image is computed as T P T P +F N +F P , where T P is the number of true positives, which are ground-truth bounding boxes that have and IoU > 0.5 with some predicted bounding box; F N is the number of false negatives, which are ground-truth bounding boxes that have an IoU < 0.5 with any predicted bounding box; F P is the number of false positives, which are predicted bounding boxes that have an IoU < 0.5 with any ground-truth bounding box. We first computed the accuracy within each image. We then computed the average accuracy for each acquisition session by averaging the per-image accuracy, and finally averaged the accuracies of each acquisition session. Such average accuracy presents the advantage to get a more even contribution from the several sessions that show a large variability in their number of images (from 17 to 200 images in the test set). The average accuracy was preferred to the worst-case accuracy because few acquisition sessions were more difficult and yielded a poor representation of the actual model performances.

Results and Discussion

The performance drop is large in plant phenotyping

The per domain average accuracy of the models is very variable, with larger dispersion for the Train to test setting (Figure 25, left). The average accuracy of the mixed-to-test setting is significantly higher (Ac=63.3) than that obtained with the train to test setting (Ac=49.6). The mixed-to-test setting allows the model to learn both on ID and OOD, even if the OOD images of the train dataset represent only 24% of the total training images (but 50% of the OOD2 dataset, the other 50% being used for test, Table 2). The hyperparameters were optimized on the OOD1 dataset for the Train to test setting. The same hyperparameters were used for the Mixed to train setting. Consequently, the performances of the Mixed to train setting could have been improved if the hyperparameters were retuned on OOD1.

The performances for the two settings are very correlated (Figure 25, right). There are easy domains where the accuracy is high for both settings, with only small performance drop. Conversely, there are difficult domains for the two settings such as Terraref, because of the strong contrast in its images and the presence of partly occluded heads. The KSU sessions have also relatively poor accuracy with a large performance drop, may be due to differences in the development stage when the images were acquired. However, it is difficult to assign the performance drop to a particular factor including genotype, sensor and acquisition protocol used. Some artifacts may also contribute to make the performance drop more variable: some domains in OOD2 have a relatively small number of images which may contribute to some unbalanced distribution of the image features between the 50% OOD2 used for training the Mixed to test setting and the 50% other images used though some of the domains to compute the average accuracy. Further, a high variance across images is observed for UQ_2 and UQ_3 belonging to OOD2. It is due to the high proportion of empty images (88% for UQ_2 and 57% for UQ_3) that are accuracy scored either 0% or 100% depending on the presence or not of false positives and negatives.

Alternative training algorithms do not improve robustness

The several training algorithms and loss functions tested provide relatively similar average accuracy over the three test domains considered. Further, the ERM shows always slightly higher performances as compared to Group DRO and CORAL. As expected, the average accuracy with ERM loss function evaluated on 100% of OOD2 (Ac=51.2) is consistent with the value reported for the same model over 50% of OOD2 (Ac=49.6, Figure 25). The performance drop is significant when comparing the average accuracy obtained over ID2, and in a lesser way on OOD1. However, both the group DRO and CORAL models do not reduce this performance drop despite their promises claimed by their authors. This results obtained over GlobalWheat-WILDS is consistent with those reported for most of the other WILDS datasets as demonstrated in table 3. Additionally, to the empirical results, theorical arguments are made against the IRM loss function not adapted for GlobalWheat-WILDS, but used for the other WILDS [1]. We hope that the WILDS benchmark can provide a clear framework to measure gain of robust training algorithm.

Conclusion

The inclusion of the wheat head detection problem within the WILDS initiative was an opportunity to investigate the robustness problem of deep learning approaches within a wider scientific community and to benefit from their theoretical advances. Our results demonstrate the existence of the domain shift with performance drop, which reinforces our previous conclusions on plant detection and counting from UAV observations [45]. More work is however needed to better understand the possible causes of the performance drop, either by the image features extracted by deep learning approaches or using the known metainformation that document factors such as the illumination conditions, image quality, development stage, genotype features. While this issue is not well covered in the other detection problems considered in WILDS, it appears to be also the case for the classification and segmentation problems investigated in WILDS. The domain shift and corresponding performance drop is therefore expected to impact any traits derived from a machine learning approach.

Our results also demonstrate the lack of effectiveness of some training algorithms claimed to improve the robustness of the model. The robustness seems to be better improved using ensemble approaches where several models trained differently are combined to provide a consensus solution as demonstrated in the Global Wheat Challenge [44]. Some questions are also still open: how to optimize the selection of the training, validation and test datasets? How to define in-domain (ID) and out of domain (OOD) datasets?

The use of Generative Adversarial Networks [46]- [48] could be also a solution to transfer the conditions of a domain to standard ones. Data pre-processing could also contribute to this standardization, particularly regarding the illumination conditions and color distributions. Finally, data augmentation that was not considered in WILDS may contribute to create some additional diversity and improve the robustness of the models as demonstrated in the Global Wheat Challenges. However, the growth in size and diversity of the available datasets with images labeled or not appears to be still a safe way for solving collectively a given problem and get robust models that can scale operationally over most of the situations encountered.

Conclusion and perspectives

This work focuses on the interpretation of phenotyping measurements to extract pertinent traits for breeders, ecophysiologists, and agronomists. While data acquisition currently reached a high level of maturity, the estimation of traits from the row measurements is still recognized as a bottleneck before within the high-throughput phenotyping systems. The recent emergence of deep learning (DL) methods was a breakthrough in data interpretation. It allows not only to improve the traitestimation performances, but also to extract traits which were impossible to access with standard computer vision and machine learning techniques . Our study focused on plants and organ identification and counting that are the main components of the yield and therefore highly desired within the set of high-throughput phenotyping traits. Identifying plants is not only a way to count them but also to localize them and describe their local environment while allowing further characterization of these individual objects. For plant counting, we demonstrated that DL techniques generally outperform more standard computer vision techniques at least over maize, sunflower, and sugar beet. However, as with any interpretation technique, DL should reach a given level of accuracy, precision, and robustness to satisfy the users' requirements fully.

Accuracy, precision, and robustness of the models are desired for plant phenotyping

Accuracy is the closeness of the estimated values with the true value. Accuracy is mandatory when estimates are being compared with reference values. It is also desired by breeders if the trait considered is combined with other information or models to get higher-order traits. For example, plant count needs to be accurate if the yield is computed from each plant's grain production. However, when ranking genotypes, precision is mandatory, while accuracy is not. Precision is the closeness between repeated measurements/estimates. Precision quantifies the repeatability, which is essential when identifying differences between genotypes/treatments. Robustness is defined by keeping performances under an extensive range of situations. If estimates of traits need to be precise, robust, and generally accurate, the actual values required by breeders, ecophysiologists and agronomists for these properties are rarely available. The most accepted requirement is to beat a reference method. However, this is difficult to demonstrate in most cases.

Accuracy If the reference method is associated with marginal uncertainties, the reference is good, and any method should be as close as possible to the reference one. It was the case for the plant counting since counting visually on the rows from the images is non-ambiguous and straightforward. However, this was not the case for head counting, where the considered reference is head counting by operators in the field. We demonstrated that the uncertainties associated with this reference method are significant and probably higher than counting on images. In this case, uncertainties of the reference method are due to counting errors in the field by the operator, as well as to the limited spatial representativeness if the whole microplot needs to be characterized. However, if another method is available in addition to the reference and the DL ones, the triple collocation error measurement (TCEM) technique [1] could be used to evaluate the intrinsic accuracy of each method regarding the actual value that is unknown. Further work is to be developed in this direction to quantify better the intrinsic performances of traits estimation techniques Precision Precision is much easier to quantify using repeatability or broad sense heritability metrics. However, this requires replicates, which was not available in a systematic way in the datasets that we compiled. More attention should be paid to this critical metric, with consequences on selecting the datasets used at least to test the proposed DL models.

Robustness Robustness was the main focus of this work since any machine learning solution can fail outside its definition domain, i.e. the training domain or "in the domain" (ID). Initial work not reported in this manuscript was trying to characterize the definition domain by describing the space of features extracted from the images of the training dataset. Some results are given in chapter 2 to characterize the datasets added to the Global Wheat Head dataset. However, the results were not as interesting as expected, and the several attempts to optimize the selection of examples to represent the diversity better were deceiving. These techniques were generally describing the distribution of trivial factors associated with the acquisition sessions. However, there were efficient to automatically identify outliers such as blurred images or images corresponding to other scenes than those targeted, such as images of the sky! One result of our study is that the acquisition session defines well a domain. This is a serious conclusion that implies that most of the literature reported that training and testing over the same acquisition sessions are likely to be not robust. Robustness for phenotyping problems could be defined as the capacity to generalize over all possible sessions. It is measured by the performance drop as proposed in the WILDS study in chapter 4: the change in performances (generally a decrease) when measured over the training domain (ID) and that measured on the test domain (OOD) when the test domain corresponds to acquisition sessions not used for the training. Our work highlights the difficulty to build robust models. This was the case for plant counting: when adding few examples of the test domains, plant identification and counting performances increase drastically. Similar results were reported over the head identification and counting when the "Mixed to Train" setting showed much improved performances compared to the "Train to test setting". It may advocates for active learning, i.e. systematic inclusion of examples from the inference dataset into the training one. However, it is not straightforward to implement for operational applications. Alternative techniques may be implemented to improve model robustness as well as accuracy and precision.

Possible improvements in the accuracy, precision and robustness of DL approaches

The constitution of large, diverse, and consistent datasets is the necessary first step. Then, data preparation and augmentation, learning strategy, and model design are critical to yield a solution that could be exploited operationally.

Large, diverse and consistent datasets preparation is a crucial first step.

In chapter 1, a dataset composed of 15.000 plants from 3 crops, 18 locations and 27 UAV flights has been gathered across different locations in France. The Global Wheat Head Dataset, described in chapter 2, comprises 6,515 images containing 275,187 wheat heads. These images were collected over 47 acquisition sessions in 16 research institutes across 12 countries. Chapter 2 presented the challenge of building such datasets and how to expose them to the community. The quality and the consistency of the dataset appear very important. The quality is mainly defined by the diversity of the examples, the balance between the several sessions, the quality of the labelling that was reviewed several times in our case, and the quality of the images, avoiding blurriness and under or overexposed images. The consistency across examples is also critical to avoid complexifying the problem for the DL model. The consistency includes a limited range of spatial resolution and view directions and labels clearly defined. The protocol used for image acquisition appears, therefore, mandatory to ensure both quality of images and consistency across sessions. Metainformation is also required to select the images/sessions better and try to understand problems when the model fails on some images/sessions. However, the constitution of large and diverse datasets for training and evaluating robust DL models requires a significant effort to get images and the associated labels and metainformation from many institutions, check and organize the database, and make it open to the scientific community. This is generally at the initiative of one or a small group of institutions. These databases need to be lively, i.e., to grow and eventually evolve to accept additional traits of interest. For example, after head identification, some characteristics of the heads could be of interest, such as the length, diameter, and a number of spikelets' ranks. This needs to be organized and requires dedicated resources.

Labelling is tedious work to accomplish, although critical and mandatory. It has evolved these last years/months rapidly, with the availability of dedicated ergonomic platforms. Further, the emergence of companies that can be contracted for labelling images for a reasonable price and good quality has changed the paradigm: the remaining tasks to accomplish locally is mostly to precisely define the objects to be labelled and to review the labels proposed by the labelling companies.

Data preparation and augmentation

Data preparation • Illumination is an essential source of variability for RGB imagery as it is often not controlled in a field experiment. Advanced preprocessing techniques can help to mitigate the environmental effect. Relighting of an image from multi-view [2], [3], videos [4] or even single view image [5] Data augmentation Before the training, data augmentation generates new examples that provide more diversity. Most studies used such techniques but were limited to geometric (rotation, shear, translation, zoom in and out) that have been exploited to generate new images for seed detections by randomly rearranged individuals who were already labelled [10], [11] with success. However, we demonstrated in chapter 3 that CutMix [12] has a limited impact on the performances, and MixUp [13] may even lower the results. Photometric transforms (random noise, random blur, change of histogram color) may widen the illumination conditions for more comprehensive training of the models. Generating more complex transformations require to use advanced techniques such as plant 3D model [14]- [16] or GAN generated images [2], [3]. As compared to a 2D approach, the 3D one provides more consistent scenes, with realistic interactions between the background and the canopy. Progress on GAN performances with StyleGAN-V2 [17] have been leveraging a new large and fully labelled dataset with approaches such as [18], allowing to label much more precisely the images -42 labels for one human face. The recent graphical processing unit also proposes a core dedicated to ray tracing that can generate photo-realistic illuminations, shadows and reflection on a dedicated 3D engine such as Unity. Combining these approaches seems to be interesting for high-resolution synthetic data generation. In the context of plant phenotyping, advanced data augmentation strategies should combine two characteristics:

• take full advantage of the handmade labels, as they are costly and tedious to obtain, and • include as much expert knowledge as possible to model the diversity of the data. Geometric transformation and 3D models generate realistic scenes in terms of structure, but the result generally is not photorealistic. In contrast, GAN model can achieve photorealistic texture at the expense of possible alteration of the scene composition structure, which prevents using them directly for a task other than classification. Geometric transformations are easy to implement with no background. Still, when there is a background, some mismatches, such as differences in the illumination of the plants and the soil, result in unrealistic scenes, which cannot be used for training. GAN's success has almost always been achieved for tasks that did not require conserving the structure or only one central object in the image, such as Arabidopsis thaliana in a single pot [16]. Controlling the scene's structure seems essential to include expert knowledge such as the sowing pattern, covering the whole range of the expected phenotype. Generating realistic sets for Deep Learning, as presented in figure 27, would require:

• A process to generate a new and realistic structure of the countable objects on an image. • A process to generate unique background.

• A process to generate new objects.

• A process to make the background and the objects coherent The objects in the scene do not need to be generated with a 3D model: actual 3D RGB plants captured in controlled conditions could be used to create a synthetic canopy. The labelling can then be done precisely at the object scale and not at the canopy scale. Synthetic scenes generation pipeline can be re-used for new traits on the same crop. It would access more detailed traits associated to smaller organs such as spikelets and flowers or allow to separate leaves from tillers. Such approaches is likely to increase the number of traits and the level of details that DL can access. The same database of 3D scenes could therefore be re-used to estimate new traits. Such synthetic scenes allow exploring the occlusion problem: the organ can be partially hidden under a leaf, making the measurement difficult. The synthetic scene is crucial for a full perception model that can pre-process RGB images and calculate many measures used to calculate structural traits. It is also a field that will not be explored by another community other than the plant community.

Training

Split of the dataset between training, validation, and test First, the question of "in domain" (ID) and "out of domain" (OOD) is complex and was discussed within the WILDS investigation over several problems, including wheat head detection. For phenotyping applications, a domain may be generally associated with an acquisition session where the sensors, acquisition protocol, illumination conditions, and growth stages are well defined, while genotypes and cultural practices vary. However, a possible bias of our work was the use of data generated from different acquisitions protocols. We expect to be able in the future to define domains beyond the acquisition session, at least when the protocol is controlled. The question of the validation dataset is exciting and was not explicitly discussed in this work. The validation dataset is part of the training process: it is used both for finetuning hyperparameters and Training algorithm Several algorithms were proposed to improve the robustness of models by playing on the way the coefficients of the network are optimized. In chapter 4, we compared the traditional ERM loss function with more sophisticated algorithms. However, results consistently indicated over the problems and datasets considered in WILDS, that the standard ERM was outperforming these training algorithms. However, their implementation is not trivial, and more work is needed to refine these first results.

Semi-supervision learning

In our studies, the dataset used for training always was labelled entirely. However, unlabelled data can help the model generalize over distributions without any labels. This strategy, when a mix of labelled and unlabelled is used to train a DL model, is called semi-supervised learning or sometimes Weakly supervised learning. A straightforward semi-supervision learning strategy is pseudo-labelling which was successfully used by most high-scoring submissions to the Global Wheat Challenges: a model is used to predict labels on a set of unlabelled images. Usually, an ensemble of models is used for more robustness [19], and during training, each batch contains more than 50% of real labelled data. An extension of the GlobalWheat-Wilds is expected to study how to improve robustness with unlabelled data.

Self-supervision learning Self-supervision learning is a branch of Deep Learning which allows training encoders on a large set of unlabeled data points thanks to a surrogate task. These encoders are can directly used for other tasks with transfer learning. The quantity of acquired data is vast within high-throughput phenotyping installations with terabytes of RGB imagery acquired during a growing season. Two families of self-supervised algorithms can be distinguished: (1) contrastive learning aimed at aligning the features of two similar images or two variations of the same images [20], [21]; (2) predictive approach, for instance, with solving a jigsaw [22] or by predicting a class determined with a simple clustering approach on the deep features (DeepCluster V2 [23], Swav [24]). Self-supervision learning is also becoming more accessible thanks to open-source libraries such as Facebook VISSL [25]. Another specific aspect of the phenotyping platform is to have a fixed set of sensors that make simultaneous observations of the same scene portion. Multimodal Deep Learning aims at learning a representation not only for RGB images but also for Lidar or multispectral images and is already used for autonomous driving [26]. Self-supervision learning is also applied to the multimodal model: OpenAI recently proposed CLIP [27], a model trained jointly on text and images. A consequence of Selfsupervision Learning is the emergence of huge models with billions of parameters such as GPT-3 [28], also called "foundation models" [29]. A characteristic of these very large models is the ability to execute new tasks without any training (zero-shot learning). However, they are challenging to train as it requires massive computing power and can be viewed as an infrastructure more than an algorithm.

Multitask learning

One DL algorithm is generally designed to access one trait or provide a pre-processing of the raw image. With a growing number of traits to measure from the same images, running multiples DL models in parallel can be burdensome in practice. Additionally, each new trait will generate a different annotated dataset that is not used for another task, which can be seen as underuse of the data. As described in the introduction, most DL models will learn to extract meaningful features from the data before solving the problem. The same features could be learned from more extensive datasets for multiple tasks. Multitask models can learn better representation from several data sources and have already been explored for wheat phenotyping [30] and Arabidopsis phenotyping [31]. It was shown that it requires fewer labels for underrepresented tasks and reduces overfitting [32], while improving adversarial robustness [33]. A comparison of both approaches is presented in figure 28. Learning an encoder for all tasks of one platform instead of having different models allows one to use all the available data and avoid overfitting if only small datasets are available. The learned representation can also be used for other purposes than estimating traits. It could be used to monitor outliers or discover subgroups.

Model selection

The baseline model was selected after the pioneering study of Madec et al. 2019 [34]. It is based on the Faster-RCNN [35] that first detect potential objects of interest and then find out which should be considered the actual ones. The challenges organized offered the opportunity to evaluate the diversity of models used and their associated performances. Results demonstrated that the standard one-step models such as YoloV3 [36] and V5 and [37] were part of the winning solutions proposed. However, it is not easy to evaluate the contribution of the architecture of the model used to the observed performances of the solution. More investigation is needed to quantify the contribution of each element of the solution.

Figure 1 .

 1 Figure 1. Schematic presentation of the photosynthesis. Chloroplast are cells contains in the leaves that are able to convert CO2 and water to 02 and carbohydrates, that are accumulated in reproductive organs

Figure 3 .

 3 Figure 3. Wheat varieties, 19th century to present, John Innes Centre

Figure 4 .

 4 Figure 4. The different plant breeding strategies. Reproduction from [43] with author's authorization.

Figure 5 .

 5 Figure 5. The different traits in phenotyping

Figure 6 .

 6 Figure 6. : Classification of the experimental installations defined by the targeted levels of trait details and control/description of the environmental conditions.

Figure 7 .

 7 Figure 7. : Presentation of the different vectors.

Figure 8 .

 8 Figure 8. Illustration of some of the most important breakthroughs made by Deep Learning

Figure 9 .

 9 Figure 9. History of the Deep Learning packages used to interact with the GPU

Figure 10 .

 10 Figure 10. Comparison of change of representation in machine learning and in Deep Learning (Credit: François Chollet, Deep Learning with Python)

Figure 11 .Figure 12 .

 1112 Figure 11. Illustration of the processing of a RGB image by a CNN

Figure 13 .

 13 Figure 13. Illustration of the optimization of a Deep Learning Network (credit: François Chollet, Deep Learning with Python)

Figure 14 .

 14 Figure 14. Presentation of a transfer learning process (Credit: François Chollet, Deep Learning with Python)

( a )

 a Presentation of classification architectures (b) Presentation of segmentation architectures (c) Presentation of detection architectures

Figure 15 .

 15 Figure 15. Example of common architures used for plant phenotyping traits

Figure 16 .

 16 Figure 16. Example of different detections modalities on wheat)

Figure 17 .

 17 Figure 17. Illustration of the diversity of Deep Learning methods)

Figure 19 .

 19 Figure 19. Illustration of the possible factor of variations

Figure 20 .

 20 Figure 20. Screenshot of a Deep Learning annotation platform

Figure 21 .

 21 Figure 21. Visualization of label error in well-know datasets. reproduced from [145] with authorization of the author

Figure 1 :

 1 Figure 1: Samples of images for the three-development stage. All images were resampled to 0.25mm.px -1 . The bounding boxes were drawn interactively around the plants.

Figure 2 .

 2 Figure 2. Flowchart of the rows extraction process from the original RGB image.

Figure 3 .

 3 Figure 3. Typical output of the HC algorithm illustrated for two sugar beet rows. The dashed white line indicates the row. The white curve represents the profile of number of green pixels perpendicular to the row, with peaks identified by a circle. The object-based method is illustrated by the colors assigned to each identified plant. Note that big objects have been split into individual plants (bottom row, the four last plants) and isolated plant parts have been reconnected to form a single plant (top row, fourth plant starting from the left). The white squares correspond to the position of missing plants.

Figure 4 :

 4 Figure 4: Accuracy for all methods and crops. For each crop and method, the stages are represented by a specific color. Each point corresponds to a test session used in the three-fold validation process. The squares represent the average of the three points. Detection of sunflower plants shows accuracy values intermediate between maize and sugar beet (Table4and Figure4). The HC shows lower TPR and higher FPR and FNR as compared to DL and HY. In the late stage, the HC shows very high FPR corresponding to problems of plant separation when they are overlapping. Further, the weeds close to the row line are not well eliminated and confounded with plants (Figure5g). Similar problems are observed for the DL method, with weeds confounded

Figure 5 :

 5 Figure 5: Possible detection errors for HC and DL methods. The green BBs correspond to the labelled plants. The red BBs correspond to the detected plants and yellow boxes correspond to weeds detected as crop. RGB images are displayed for the DL method. BW images are displayed for the HC method. a, b, c, d corresponds to maize, e, f, to sugar beet and g, h to sunflower.

Figure 6 :

 6 Figure 6: rRMSE for plant density estimation for all methods crops, and stages. Results obtained over the testing dataset. For each crop, method and stage, the three instances (corresponding to three testing sessions) of the three-fold cross validation process are displayed as colored dio sks,

Figure 7 :

 7 Figure 7: Distribution of relative absolute error for each microplots for the Out-Domain and In-Domain approaches for DL. Box-plot representation where the black horizontal bar represents the median, the box represents ±25%, the whiskers while the whiskers extend to the the lowest (highest) data point still within 1.5 interquantile range of the lower (upper) quartile. Diamonds are outliers. 1 outlier for Out-domain Maize and 3 outliers for Out-Domain Sunflower are above 0.5 and are not presented on the graph.

Figure S3 :

 S3 Figure S3: Left: distribution of the typical size of BB annotated and those defined around the plants identified by the HC method. Right: comparison of Accuracy computed either with IoU_MS, and with C_MS for HC (green discs), and DL methods (blue squares).

Figure 1 :Figure 2 :

 12 Figure 1: Overview of the harmonization process conducted. Images were first rescaled using bilinear interpolation up-or downsampling techniques. Then, the rescaled images were split into 1024 × 1024 squared patches.

3. 1 .

 1 General Statistics. The GWHD dataset represents 4698 squared patches extracted from the 2219 original highresolution RGB images acquired across the 11 sub-datasets (Table

Figure 3 :

 3 Figure 3: Distribution of the number of bounding boxes per image (a) and bounding boxes size * (b) in the GWHD dataset. * The bounding box size is defined as the square root of the bounding box area in pixel.

Figure 4 :Figure 5 :

 45 Figure 4: Example of images from different acquisition sites after cropping and rescaling.

Figure 6 :

 6 Figure 6: Comparison of GWHD dataset with other object detection datasets. Both axes are in log-scale.

Figure 1 :

 1 Figure 1: Sample images of the Global Wheat Head Detection 2021. The blue boxes correspond to the interactively labeled heads.

  6515 275187 * VLB: Villiers le Bâcle; VSC: Villers-Saint-Christophe. * * Utokyo_1 and Utokyo_2 were taken at the same location with different sensors. * * * Utokyo_3 is a special subdataset made from images coming from a large variety of farmers in Hokaido between 2016 and 2019. Italic: Europe: bold: North America; underline: Asia; bold italic: Oceania; bold underline: Africa.

Figure 2 :

 2 Figure 2: Distribution of the development stage. The x-axis presents the number of subdataset per development stage.

Figure 3 :

 3 Figure 3: Distribution of the images in the two first dimensions defined by the UMAP algorithm for the GWHD 2021 dataset. The additional subdatasets as well as the training and test datasets from GWHD_2020 are represented by colors.

  Crowdsourcing of data and analysis have also expanded to specific application areas, such as image-based plant phenotyping, where deep learning methods have been employed for plant disease classification (Albetis et al., 2017; Fuentes et al., 2017; Toda and Okura, 2019), plant and organ detection and counting (Madec et al., 2019; David et al., 2020; Ayalew, Ubbens and Stavness, no date), vegetation segmentation (Mortensen et al.

Figure 1 :

 1 Figure 1: Sample from UQ frame dataset

Figure 2 :

 2 Figure 2: Sample from the Arvalis LITERAL Dataset 3.2 The Global Wheat Challenge 2020 and 2021

Figure 3 :

 3 Figure 3: Detailed performance of the best solutions against the baseline. Top: Accuracy per domain; center: False positive rate (FPR); Bottom: False negative rate (FNR)

Figure 4 :

 4 Figure 4: cumulated frequency of the size of the missing bounding boxes compared to the distribution of the size of all bounding boxes from the test set (GT).

Figure 5 :

 5 Figure 5: False negatives (FN) and positives (FP) from GWC_2021 illustrated over a random sample images of the test dataset.

Figure 6 :

 6 Figure 6 : Boxplot representation of the rRMSE grouped per dataset split (Train, Validation, Test) and models (Baseline, GWC2020, GWC_2021). The red line indicates the median and the blue box the 25%-75% quantiles. The whiskers extend to the most extreme (1%-99% assuming a normal distribution), while the red cross correspond to outliers. The black cross at the top indicates outliers larger than rRMSE=1, with the value on top.

Figure 7

 7 Figure 7 Relationship between accuracy and rRMSE for the test dataset and GWC_2021 model. Each of the 18 points corresponds to a specific domain.

Figure 9

 9 Figure 9 Comparison between the head density measured in the field and that measured on the RGB images by human operators over the same sampling area delimited by a frame. Data coming from the UQ frame dataset (n=185).

Figure 10 :

 10 Figure 10 : Comparison of the rRMSE obtained on the UQ dataset from different ways of counting heads

-

  Canada: the Global Institute Food Security, University of Saskatchewan supported the organisation of the competition -France : PIA #Digitag Institut Convergences Agriculture Numérique , Hiphen supported the organization of the competition -Japan: Kubota supported the organisation of the competition -Australia: Grains Research and Development Corporation (UOQ2002-008RTX Machine learning applied to high-throughput feature extraction from imagery to map spatial variability and UOQ2003-011RTX INVITA -A technology and analytics platform for improving variety selection) supported competition 7 Bibliography with metric used during Kaggle against the ADA. Part C presents a simulation of the new ranking based on the new score and limited to the solutions sampled for the analysis.

Figure 2 :

 2 Figure 2: Presentation of Mixup, Cutmix and Mosaic strategy

5

  GlobalWheat-Wilds: Global Wheat Head Dataset as a benchmark of in-the-wild distribution shiftsThis chapter corresponds to my participation to the article "WILDS: A Benchmark of in-the-Wild Distribution Shifts", published in at ICML[1]. It focuses on the contribution of the Global Wheat Dataset to study the robustness of deep learning algorithms. Etienne David, Ian Stavness and Wei Guo are the main investigators for the contribution of Global Wheat Head Dataset 2021 (GWHD_2021) to WILDS. GWHD_2021 is the only agriculture dataset of WILDS and the only detection dataset. Therefore, it forces me to implement several changes in the WILDS codebase.

Figure 22 . 1 Introduction 5 . 1 . 1

 221511 Figure 22. Illustration of the WILDS paper

Figure 23 .

 23 Figure 23. The Wilds benchmark contains 10 datasets across a diverse set of application areas, data modalities, and dataset sizes. Each dataset comprises data from different domains, and the benchmark is set up to evaluate models on distribution shifts across these domains.

Figure 24 .

 24 Figure 24. Deep CORAL loss objective

Figure 25 .

 25 Figure 25. On the left, distribution of the average accuracy evaluated on the test (OOD) dataset when the model is trained over the mixed to train (ID+OOD) and train to test (ID) dataset. Box plot representation with the median indicated by the red line. The box contains the percentiles between 25% and 75%, while the whiskers extend to the 1%-99% percentiles. On the right, relationships between the average accuracy for each domain of the model trained on ID+OOD dataset and that trained on the ID dataset. The dashed line is the 1:1 line.

  has been made possible by recent development in Deep Learning, which could open an avenue to control the illumination. Shadow removal technics are also a possibility to remove illumination [6][7][8][9]. An example of how such technics can perform directly on plant phenotyping imagery is presented in figure 26. Normalization of the pixel values across different RGB cameras can also prevent unexpected behaviour of the Deep Learning algorithms on new cameras or particular illumination conditions.

Figure 26 .

 26 Figure 26. Presentation of a shadow removal GAN network [7] on wheat head canopy (left: original ; center: GAN ; right : CLAHE).

Figure 27 .

 27 Figure 27. Example of a simple pipeline to generate new UAV microplot from RGB images and previous labels. Expert knowledge can be used to conditioned the generation of new scenes. Objects and background can be retrieve from a bank of image, generate with a 3D model or a GAN

Figure 28 .

 28 Figure28. Comparison of a system based on multiple networks with separated encoder ("one task network") and tasks networks sharing the same encoder which is improved with all labels ("Multitask network").

1

  Available traits for wheat at Arvalis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 GlobalWheat-wilds splits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Average accuracy computed over several domains (ID1, OOD1, OOD2) for the model trained on ID1 and validated on OOD1 with different loss functions (ERM, Group DRO, CORAL). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 1 .

 1 Presentation

	Trait	Description	Category		Unit	Method	Phe-nomo-bile (RGB)	Hand-held (RGB)	UAV (RGB)	UAV (Multi-spec-tral)
	GreenFr	Green Cover Fraction	State Biophysical	-	Unitless [0:1]	Segmentation with DL or SVM
	GAI	Green Area Index	State Biophysical	-	Unitless (% of observed area)	radiative trans-fer model inver-sion
	ALA	Average Leaf Angle	State Biophysical	-	deg	radiative trans-fer model inver-sion
	FIPAR	Fraction of inter-cepted PAR	Radiative		Unitless [0:1]	radiative trans-fer model inver-sion
	CropFr	Crop Cover Fraction	State Biophysical	-	Unitless [0:1]	Segmentation with DL
	Senes-centFr	Senescent Fraction of the crop	State Biophysical	-	Unitless [0:1]	Segmentation with DL
	Height-Max	Maximum height	plant	State Biophysical	-	m	Photogramme-try
	Spikes-Density	Density of spikes	State Biophysical	-	number/m2	Detection with DL
	Plants-Density	Density of plants	State Biophysical	-	number/m2	Detection with DL
	NDVI	NDVI		State Biophysical	-	Unitless	Vegetation Index
	Clgr	CI green @ nadir	State Biochemical	-	Unitless	Vegetation Index
	MTCI	MTCI @nadir		State Biochemical	-	Unitless	Vegetation Index
	LCC	Leaf Chlorophyll Con-tent	State Biochemical	-	microgram of Chl/cm2 of leaf	radiative trans-fer model inver-sion
	CCC	Canopy Chlorophyll Content	State Biochemical	-	g of Chl/m2 at ground level	radiative trans-fer model inver-sion
	Lodg-ingScore	Plant Lodging		State Biophysical	-	Unitless [0:1]	Photogramme-try
	Disease-Fraction	Fraction of the crop (or organ) affected by a disease symptoms	State -Sanitary Unitless [0:1]	Segmentation with DL
	Phenol-ogy	Growth Stage		Dynamic		Zadock scale	Classification with DL

of the available traits at Arvalis for production ( ) or in development ( )

1,*2 , Gaëtan Daubige 2 , François Joudelat 3 , Philippe Burger 4 ,Alexis Comar 5 , Benoit de Solan 2 , Frédéric Baret 1 1 Abstract

Table 1 : Comparison of the different approaches used for plant and organ counting referenced in the literature. 1 random selection of samples for training and testing; 2 No proper calibration; 3 Calibrated with synthetic data; 4 Testing is made on two sessions, one session being already used for training

 1

	#	Study	UAV Crop	Object	Sessions	Localiz	Method	Test
							ation		independency
	1	[3]	Yes Sorghum	Head	1	Yes	ML	No 1
	2	[4]	yes	Wheat	Plant	5	yes	ML	No 1
	3	[5]	no	Wheat	Plant	several	yes	HC	Yes 2
	4	[6]	yes	Maize	Plant	1	yes	HC	yes 2
	5	[7]	yes Sunflower	Plant	1	yes	HC	Yes 3
	6	[8]	yes	Agave	Plant	3	yes	HC	No
				Maize
	7	[9]	yes	Sunflower	Plant	6	no	HC (OBIA)	Yes 2
				wheat
	8	[10]	yes	Thuja	Plant	3	yes	HC (OBIA)	Yes 2
	9	[11]	yes	Maize	Plant	2	yes	HC (OBIA)	No 1
	10	[12]	yes Rapeseed	Plant	2	yes	HC (OBIA)	No 1
	11	[13]	Yes Safflower	Plant	2	Yes	HC (OBIA)	No 4
	12	[14]	No	Wheat	Head	2	yes	DL	Yes
	13	[15]	No	Maize	Plant	10	yes	DL	No 1
	14	[16]	Yes Sorhgum	Plant	2	no	DL	No 1
	15	[17]	Yes	Wheat	Head	several	no	DL	Yes
	16	[18]	Yes	Spinach	Plant	1	no	DL	No 1
	17	[19]	Yes	Maize	Head	2	yes	DL	No 1
	18	[20]	Yes Sorghum	Head	2	yes	DL	No 1
		This study		Maize
			Yes	Sugar beet	Plant	27	yes	HC / DL	Yes
				Sunflower

Table 2 . Characteristics of the crops for the several sites considered.

 2

	Crop	Site Name	Lat (°)	Long	Year	Nb.	Nb.	Microplot	Microplot	Row	Plant density	Soil type
				(°)		sessions	microplots	width (m)	length (m)	spacing	(plt.m -2 )
										(m)
		Menainville	47.9	1.4	2016	1	6	2.2	7.0	1.10	5.1	Clay
		Nerac	44.1	0.3	2016	1	8	1.6	7.0	0.80	8.5	Clay
		Villedieu	47.8	1.5	2016	1	6	0.9	11.0	0.30	19.9	Clay
	Maize	Thenay Blois Castetis	47.3 47.7 43.4	1.2 1.2 -0.7	2017 2019 2019	1 1 1	6 7 5	4.4 1.7 2.8	6.0 7.0 4.0	0.63 0.83 0.70	7.3 9.5 11.2	Clay / Flint Brunisolic Brunisolic
		Ermine	46.5	-1.0	2019	1	4	3.2	5.5	0.80	8.6	Limestone
		Selommes	47.7	1.2	2019	1	7	1.8	5.3	0.88	9.5	Brunisolic
		Pleinefougeres	48.5	-1.5	2020	1	2	3.2	11.0	0.80	7.7	Brunisolic
		Bucy	49.6	3.9	2017	2	7	1.4	6.2	0.45	11.1	Loam
	Sugar beet	Charmont Etienne Memmie Charmont	48.3 49.2 48.9 48.3	4.1 4.3 4.3 4.1	2017 2017 2017 2018	1 1 2 2	7 6 6 8*	1.4 1.2 1.4 1.4	5.5 7.6 7.6 5.5	0.45 0.40 0.48 0.45	11.1 15.6 10.8 11.4	Limestone Limestone Limestone Limestone
		Memmie	48.9	4.3	2018	1	6	1.4	7.6	0.45	11.4	Limestone
	Sunflower	Rivière Auzeville Auzeville Epoisses	43.5 43.5 43.5 47.2	1.5 1.5 1.5 5.1	2017 2018 2019 2019	1 2 5 1	8 3 12 4	3.0 3.3 2.9 2.4	4.1 9.5 9.0 10.0	0.50 0.55 0.96 0.60	7.1 6.1 3.7 5.1	Clay Clay Clay Limestone

  [START_REF] Maninis | Deep extreme cut: From extreme points to object segmentation[END_REF]

		Session_name	plant number	plot number	Stage	GSD (mm)	typical BB size (cm)	typical BB size (pixel)	Weed infestation	Blur
		Selommes_2019_1	510	7	1	3.5	6.5	26		233
		Hermine_2019_1	542	4	1	3.5	7.8	31		79
		Thenay_2017_1	617	6	1	2.5	8.5	34		1149
		Castetis_2019_1	575	5	2	3.3	10.0	40		121
	MAIZE	Pleinefougeres_2019_1 Blois_2019_1	504 579	2 7	2 2	3.5 3.3	11.5 12.3	46 49		39 346
		Menainville_2016_1	620	6	3	3.4	12.3	49		78
		Villedieu_2016_1	629	6	3	2.7	13.3	53		261
		Nerac_2016_1	594	8	3	4.0	15.0	60		37
		Total	5170	51
		Memmie_2017_1	667	6	1	4.5	8.0	32		26
		Charmont_2018_1	556	7	1	4.2	11.5	46		93
		Memmie_2018_1	602	6	1	4.3	11.5	46		77
	BEET SUGAR	Bucy_2017_1 Memmie_2017_2 Etienne_2017_1 Charmont_2017_1	634 679 635 669	7 6 6 8	2 2 2 3	5.3 5.7 4.5 3.4	12.8 14.8 16.0 20.5	51 57 64 82		25 72 27 191
		Charmont_2018_2	647	8	3	4.1	20.5	82		102
		Bucy_2017_2	558	6	3	4.5	23.0	92		31
		Total	5647	60
		Auzeville_2019_1	579	12	1	5.0	8.5	34		28
		Auzeville_2019_2	640	12	1	5.0	13.5	54		510
		Epoisses_2019_1	596	4	1	2.5	14.3	57		10
	SUNFLOWER	Auzeville_2018_1 Auzeville_2019_3 Auzeville_2019_4 Rivière_2017_1	596 657 603 634	3 12 12 8	2 2 2 3	2.3 5.0 5.0 5.2	14.3 19.3 24.5 25.0	57 77 98 100		488 350 221 42
		Auzeville_2018_2	560	3	3	2.6	27.5	110		1286
		Auzeville_2019_5	565	12	3	5	27.5	110		176
		Total	5430	78

Table 3 . Characteristics of each measurement sessions. For sugar beet, microplots from one 131 session to another are the same. For sunflower the microplots considered change between 132 sessions. The typical size of the BB for one session is computed as the square root of the mean 133 area of all the BBs. The typical bounding box (BB) size in pixels is computed after up sampling 134 the images at 2.5 mm resolution. The plant stage at the time of the session is quantified as: 1: 135 early, 2: intermediate, 3: late. The correspondence with BBCH scale is provided as a table in the 136 supplementary material section. The weed infestation is scored from 0 (no weed), from 0 (no 137 weeds), 1 (less than 5% coverage), 2 (more than 5% coverage). The image blur is quantified by 138 the average variance of the Laplacian: high blur results in low value of the variance of the 139 Laplacian. 140

 3

Table 4 : Terms of the confusion matrix for the three methods the three crops, and the three stages. True Positive Rate (TPR), False Positive Rate (FPR), and False Negative Rate (FNR) are displayed. N is the true number of plants (N=TP+FN). Green color corresponds to good metrics values (high for TPR, low for FPR and FNR), and red for poor metrics values (low for TPR, high for FPR and FNR).

 4

Table 5 : p-values computed from an ANOVA for the qualitative factors and Pearson test for the quantitative factors.

 5

	Factors	Type	HC	DL	HY
	Crop type	qualitative	0.009130**	0.127550	0.032050**
	Development stage	qualitative	0.857810	0.479530	0.643620
	Weed infestation	qualitative	0.032610**	0.001600**	0.074540
	Soil type	qualitative	0.026430**	0.781090	0.830650
	Sowing density	quantitative	0.067379	0.076542	0.091679
	Original resolution	quantitative	0.905626	0.572383	0.616534
	Plant size	quantitative	0.791437	0.064765	0.211019
	Blurriness	quantitative	0.111743	0.562775	0.501980

  These results demonstrate the importance of having a proper design of DL training dataset when proposing a new trait to get robust estimates as required by agronomists, breeders, and farmers. Our results are consistent with those of previous studies: detection and density estimation performances are generally lower when the training and the test datasets are independent, i.e not coming from the same measurement sessions. Fernandez-Gallo [ref] report a rRMSE below 5%,Madec et al. [ref] report a rRMSE of 15% on an independent test set. Similar drop in performances seems to happen in maize when comparing the results of Varela et al. (counting error of 1.5%) to those of Gnädinger and Schmidhalter (counting error of +/-15%). The generalization potential of DL methods is high, requiring including more diverse situations in the training dataset at the expense of the tedious and expensive interactive labelling process. However, alternative techniques could be used to bypass this limitation, including data sharing between several organizations as this was done for the head counting problem(David et al., 2020). Data augmentation[40] could also improve greatly the generalization performances of DL methods. It would consist in manipulating the quality of the images, while creating synthetic images where a wide diversity of plants and weeds would be placed over different backgrounds with variation in the development stages and sowing pattern. //zenodo.org/record/4890370) to be possibly used as a benchmark for plant counting and detection from RGB images acquired from UAVs. Our results show that when the main assumptions on the sowing patterns are verified, simple HC methods can reach good enough performances to be used for applications as it was observed here for sugar beet. However, state-of-the art Deep Learning methods generally outperform the HC ones. Nevertheless, due to the large heterogeneity in terms of background, plant shape and phenological stages encountered across the wide collection of images considered, we demonstrated that the performances of the DL methods largely depend on the training and test datasets used. When the training domains used for the DL method are fully independent from the testing ones, the overall performances are reduced due to the failure of the model in a number of test cases poorly represented in the training dataset. Conversely, when adding few examples of images representative of the test domain, the performances increase drastically to reach those reported in most studies where training and test domains are not differentiated. Important gain in robustness could therefore be reached by including in the training dataset few images coming from the inference domains. Alternatively, a better understanding of the factors of variability between domains could constitute the basis to generate efficient data augmentation techniques that may even include synthetic images. An extended version of the dataset is needed to conclude on the main factors of error on plant counting with UAV. The hybrid method proposed to better eliminate weeds could be replaced efficiently by including images of the canopy where weeds were artificially incrusted.

	5 This study was based on a comprehensive dataset covering three main crops, several growth stages and Conclusion acquisition conditions. It will be open to the community on Zenodo Row spacing (https:6 20 prior

Table 1 :

 1 Characteristics of the experiments used to acquire images for GWHD dataset.

	Sub-dataset name	Institution	Country Lat (	°)	Long ( °)	Year	No. of dates	Targeted stages	Row spacing (cm)	Sowing density (seeds•m 2 )	No. of genotypes
	UTokyo_1	NARO & UTokyo	Japan	36.0 N 140.0 E 2018	3	Postflowering	15	186	66
	UTokyo_2	NARO & UTokyo	Japan	42.8 N 143.0 E 2016	6	Flowering *	12.5	200	1
	Arvalis_1	Arvalis	France	43.7 N 5.8 E 2017	3	Postflowering-ripening	17.5	300	20
	Arvalis_2	Arvalis	France	43.7 N 5.8 E 2019	1	Postflowering	17.5	300	20
	Arvalis_3	Arvalis	France	49.7 N 3.0 E 2019	3	Postflowering-ripening	17.5	300	4
	INRAE_1	INRAE	France	43.5 N 1.5 E 2019	1	Postflowering	16	300	7
	USask_1	University of Saskatchewan	Canada 52.1 N 106 W 2019	1	n.a	30.5	250	16
	RRes_1	Rothamsted research	UK	51.8 N 0.36 W 2016	1	n.a	n.a	350	6
	ETHZ_1	ETHZ	Switzerland 47.4 N 8.6 E 2018	1	n.a	12.5	400	354
	NAU_1	Nanjing Agric. University	China	31.6 N 119.4 E 2018	1	Flowering *	20	300 or 450	5
	UQ_1	UQueensland	Australia 27.5 S 152.3 E 2016	1	Flowering -ripening	22	150	8

* Images were checked carefully to ensure that heads have fully developed and flowered.

Table 2 :

 2 Image characteristics of the sub-datasets comprising the GWHD dataset. All cameras looked vertically downward.

	Sub-dataset name	Vector	Camera	Focal length (mm)	Field of view ( °) *	Shooting mode	Image size (pixels)	Distance to ground (m)	GSD (mm/px)
	UTokyo_1	Cart	Canon PowerShot G9 X mark II	10	38.15	Automatic	5472 × 3648	1.8	0.43
	UTokyo_2 Handheld	Olympus μ850 & Sony DSC-HX90V	7/4	45.5	Automatic	3264 × 2488 & 4608 × 3456	1.7	0.6
	Arvalis_1	Handheld Sony alpha ILCE-6000	50 & 60	7.1	Automatic	6000 × 4000	2.9	0.10-0.16
	Arvalis_2	Handheld	Sony RX0	7.7	9.99	Automatic	800 × 800 †	1.8	0.56
	Arvalis_3	Handheld	Sony RX0	7.7	9.99	Automatic	800 × 800 †	1.8	0.56
	INRAE_1	Handheld	Sony RX0	7.7	9.99	Automatic	800 × 800 †	1.8	0.56
	USask_1	Minivehicle FLIR Chameleon3 USB3	16	19.8	Fixed	2448 × 2048	2	0.45
	RRes_1	Gantry	Prosilica GT 3300 Allied Vision	50	12.8	Automatic	3296 × 2472	3-3.5 §	0.33-0.385
	ETHZ_1	Gantry	Canon EOS 5D mark II	35	32.2	Fixed	5616 × 3744	3	0.55
	NAU_1	Handheld	Sony RX0	24	16.9	Automatic	4800 × 3200	2	0.21
	UQ_1	Handheld	Canon 550D	55	17.3	Automatic	5184 × 3456	2	0.2

Datasets UTokyo_1 and ETHZ_1 are particularly affected by this issue. Each institution acquired images from different platforms, including handheld, cart, minivehicle, and gantry systems. The diversity of camera sensors and acquisition configurations resulted in a wide range of image properties, which will assist in training deep learning models to better generalize across different image acquisition conditions.

Table 3 :

 3 Statistics for each component of the Global Wheat Head Detection.

	Sub-dataset name	No. of acquired images	No. of patch per image	Original GSD (mm)	Sampling factor	Used GSD (mm)	No. of labelled images	No. of labelled heads	Average no. of heads/images
	UTokyo_1	994	1	0.43	1	0.43	994	29174	29
	UTokyo_2	30	4	0.6	2	0.3	120	3263	27
	Arvalis_1	239	6	0.23	0.5	0.46	1055 *	45716	43
	Arvalis_2	51	4	0.56	2	0.28	204	4179	20
	Arvalis_3	152	4	0.56	2	0.28	608	16665	27
	INRAE_1	44	4	0.56	2	0.28	176	3701	21
	USask_1	100	2	0.45	1	0.45	200	5737	29
	RRes_1	72	6	0.33	1	0.33	432	20236	47
	ETHZ_1	375	2	0.55	1	0.55	747 *	51489	69
	NAU_1	20	1	0.21	1	0.21	20	1250	63
	UQ_1	142	1	0.2	0.5	0.4	142	7035	50
	Total	2219	-	-	-	-	4698	188445	-

* Some labelled images have been removed during the labelling process.

Table 4 :

 4 The minimum metadata that should be associated with images of wheat heads.Position of the image in the microplot §

		Session level	Image level
		Name of the experiment (PUID) †	Microplot id
		Name of institution	Row spacing
	Experiment metadata	GPS coordinates ( °)	Sowing density
		Email address of the contact person	Name of the genotype (or any identifier) †
		Date of the session (yyyymmdd)
		Wheat species (durum, aestivum …) *	Presence or not of awns
		Development stage/ripening stage *
		Vector characteristics:	Camera aperture
		Name	Shutter speed
		Type (handheld, cart, phenomobile, gantry, UAV)	ISO
		Sampling procedure	Distance from camera to canopy (m) ‡
	Acquisition metadata	Distance to the ground (m)

*

Table 1 :

 1 Continued.

	GWHD_2021 subdataset name	GWHD_2020 subdataset name	Owner	Country	Location	Acquisition date	Platform	Development stage	Number of images	Number of wheat head
	NAU_3		NAU	China	Baima	09/05/2020	Cart	Filling	100	4596
	UQ_1	uq_1	UQueensland	Australia	Gatton	12/08/2015 Tractor Postflowering	22	640
	UQ_2	uq_1	UQueensland	Australia	Gatton	08/09/2015 Tractor Postflowering	16	39
	UQ_3	uq_1	UQueensland	Australia	Gatton	15/09/2015 Tractor	Filling	14	297
	UQ_4	uq_1	UQueensland	Australia	Gatton	01/10/2015 Tractor	Filling	30	1039
	UQ_5	uq_1	UQueensland	Australia	Gatton	09/10/2015 Tractor	Filling-ripening	30	3680
	UQ_6	uq_1	UQueensland	Australia	Gatton	14/10/2015 Tractor	Filling-ripening	30	1147
	UQ_7		UQueensland	Australia	Gatton	06/10/2020 Handheld	Ripening	17	1335
	UQ_8		UQueensland	Australia	McAllister	09/10/2020 Handheld	Ripening	41	4835
	UQ_9		UQueensland	Australia	Brookstead 16/10/2020 Handheld	Filling-ripening	33	2886
	UQ_10		UQueensland	Australia	Gatton	22/09/2020 Handheld	Filling-ripening	53	8629
	UQ_11		UQueensland	Australia	Gatton	31/08/2020 Handheld Postflowering	42	4345
	ARC_1		ARC	Sudan	Wad Medani	03/2021	Handheld	Filling	30	888
								Total

Table 2 :

 2 Presentation of the Global Wheat Challenge 2021 results.

	Solution name	WDA
	randomTeamName (1 st place)	0.700
	David_jeon (2 nd place)	0.695
	SMART (2 nd place)	0.695
	Reference (faster-RCNN)	0.492

Table 3 :

 3 Presentation of the different splits which can be used with the GWHD_2021.

	Splits	Training	Validation	Test
	Global Wheat
	Challenge 2021

Table 1 :

 1 Summary of the different datasets used in the study

	DATASETS	USE	IMAGE	IN-FIELD	COMMENTS
			ANNOTATION	COUNTING
	GLOBAL WHEAT HEAD Training / Validation / Test	Yes	No	Described in [21]
	UQ FRAME	Test	Yes	Yes	In-field count are
					made on the
					same sampling
					area than the
					digital images
	ARVALIS LITERAL	Test	No	Yes	In-field density is
					manually
					evaluated on
					other parts of the
					microplots

Table 2 :

 2

	GWHD_2021 sub-dataset name	GWHD_2020 sub-dataset name	owner	country	location	Acquisition date	Platform	Development stage	Number of images	Number of wheat head
	Ethz_1	ethz_1	ETHZ	Switzerland	Usask	06/06/2018 Spidercam	Filling	747	49603
								Filling -
	Rres_1	rres_1	Rothamsted	UK	Rothamsted 13/07/2015	Gantry	Ripening	432	19210
	ULiège-GxABT_1		Uliège/Gembloux	Belgium	Gembloux	28/07/2020	Cart	Ripening	30	1847
	NMBU_1		NMBU	Norway	NMBU	24/07/2020	Cart	Filling	82	7345
	NMBU_2		NMBU	Norway	NMBU	07/08/2020	Cart	Ripening	98	5211
	Arvalis_1	arvalis_1	Arvalis	France	Gréoux	02/06/2018 handheld Post-flowering	66	2935
	Arvalis_2	arvalis_1	Arvalis	France	Gréoux	16/06/2018 handheld	Filling	401	21003

The sub-datasets for GWHD_2020 and GWHD_2021. The column "2020 name" indicates the name given to the subdatasets for GWHD_ 2020, which were split into several new sub-datasets. Red = Europe, Blue = North America, Yellow = Asia, Green= Oceania, Orange. The table is a reproduction from the original paper.

Table 3 :

 3 Description of the different sub-datasets of UQ Frame dataset

	Sub-dataset	Date	of	Location	Lat (°) Long (°)	Sowing date	Frame	Frame
	name	acquisition						number	size
	Frame_UQ_7	31/08/2020	Gatton	-27.55	152.27	12/05/2020	42	500mm x
									1000mm
	Frame_UQ_8	22/09/2020	Gatton	-27.55	152.27	06/06/2020	53	500mm x
									1000mm
	Frame_UQ_9	06/10/2020	Gatton	-27.55	152.27	06/06/2020	17	500mm
									*500mm
	Frame_UQ_10	09/10/2020	McAlister	-33.98	116.40	10/06/2020	40	515mm x
									515mm
	Frame_UQ_11	16/10/2020 Brookstead -27.75	151.44	12/06/2020	33	515mm x
									515 mm

Table 4 :

 4 Presentation of the different sub-datasets of Arvalis LITERAL Dataset

	Sub-dataset	Date	of	Lat(°) Long	Sowing date	Number of	in field counting
	name	acquisition		(°)		microplots	date and stage

Table 5 :

 5 Summary of the different challenges * in 2020, the public and private test set were randomly sampled from the same subdatasets

	Platform Number of	Train	Validation	Test
	participants		(Public test)	(Private Test)

Table 6 :

 6 Summary of the winning solutions

		Rank	Solution name	Domain Data Augmentation	Architecture	Ensemble approach	Challenge score	Comments
	Baseline		Madec	No	Faster-RCNN	No	𝑨𝑨 𝟐𝟎𝟐𝟏 = 𝟎. 𝟒𝟕𝟒 *
		1	DungNB	Mixup ; Custom mosaic	EfficientDet; FasterRCNN	Random subsampling	𝐴𝐴 2020 =0.69 0
	GWC_2020	2	OverFeat Mixup, Cutmix	Efficentdet	Random subsampling	𝑨𝑨 𝟐𝟎𝟐𝟎 =0.68 8	Rank first with AA2021
		3	Javu	Mixup	YoloV3	No	𝐴𝐴 2020 =0.68 4
		1	Random TeamNa me	Mosaic	Yolov5	Domain subsampling	𝑨𝑨 𝟐𝟎𝟐𝟏 =0.70 0
	GWC_2021	2	David_jeo n	Mosaic; CutMix	Yolov5	No	𝐴𝐴 2021 =0.69 5	Model is applied on 1600 px images
		3	SMART	CutMix	Yolov4	Yes	𝐴𝐴 2021 =0.69 5	A network is jointly trained to improve image quality
	4.2 Challenges solved most of the false positives, but still miss small wheat heads

*The score was not obtained during a competition

Table 7 :

 7 rRMSE for the three algorithms on all domains. Colors indicate the rRMSE value: green is the lowest and red the largest

	Split	Domain	Baseline	rRMSE GWC_2020	GWC_2021
	Training	Arvalis_1	0.23	0.153	0.215
		Arvalis_10	0.371	0.164	0.268
		Arvalis_11	0.254	0.42	0.168
		Arvalis_12	0.395	0.181	0.247
		Arvalis_2	0.139	0.144	0.152
		Arvalis_3	0.123	0.114	0.125
		Arvalis_4	0.421	0.123	0.339
		Arvalis_5	0.217	0.097	0.251
		Arvalis_6	0.084	0.155	0.129
		Arvalis_7	0.204	0.206	0.12
		Arvalis_8	0.172	0.179	0.093
		Arvalis_9	0.154	0.189	0.062
		ETHZ_1	0.073	0.164	0.134
		Inrae_1	0.145	0.083	0.185
		NMBU_1	0.079	0.313	0.097
		NMBU_2	0.241	0.224	0.124
		Rres_1	0.091	0.067	0.146
		ULiège-GxABT_1	0.091	0.341	0.097
	Validation	NAU_1	0.239	0.291	0.472
		UQ_1	0.319	0.443	0.235
		UQ_2	2.015	0.271	1.128
		UQ_3	0.245	0.333	0.21
		UQ_4	0.264	0.275	0.122
		UQ_5	0.351	0.563	0.141
		UQ_6	0.18	0.288	0.224
		Usask_1	0.248	0.103	0.252
		Utokyo_1	0.193	0.068	0.146
		Utokyo_2	0.233	0.07	0.156
		Utokyo_3	0.511	0.246	0.308
	Test	ARC_1	0.485	0.253	0.111
		CIMMYT_1	0.176	0.372	0.123
		CIMMYT_2	0.141	0.24	0.101
		CIMMYT_3	0.224	0.27	0.094
		KSU_1	0.266	0.364	0.074
		KSU_2	0.185	0.34	0.101
		KSU_3	0.177	0.323	0.173
		KSU_4	0.2	0.346	0.103
		NAU_2	0.286	0.191	0.202
		NAU_3	0.257	0.187	0.237
		Terraref_1	0.895	0.56	0.281

Table 3 : Ablation study different Data Augmentation strategies

 3

	Modality	Aw	AGWC_2020
	Baseline	0.452787	0.654569
	Baseline + Cutmix	0.476214	0.659364
	Baseline + Mixup	0.367481	0.553144
	Data Augmentation (DA)	0.498065	0.704271
	DA + Cutmix	0.501241	0.685546
	DA + Mixup	0.437714	0.686517
	DA + Cutmix + Mixup	0.358226	0.508334

Table 2 .

 2 The splits used for the performance drop and robustness experiments. (*) the hyperparameters used for the Mixed to Test experiment were the same as those optimized for the Train to Test experiment.

Table 3 .

 3 Average accuracy computed over several domains (ID1, OOD1, OOD2) for the model trained on ID1 and validated on OOD1 with different loss functions (ERM, Group DRO, CORAL).

* The field of view is measured diagonally. The reported measure is the half-angle. † Original images were cropped, and a subimage of size 800 × 800 was extracted from the central area.§ The camera was positioned perpendicular to the ground and automatically adjusted to ensure a 2.2 m distance was maintained between the camera and canopy.3 Plant Phenomics

* This may be alternatively reported at the image level if it is variable within a session. † Persistent unique identifier (PUID). This may be a DOI as for genetic resources regulated under the on Plant Genetic Resources for Food and Agriculture (https://ssl.fao.org/glis) or any other identifier including the information of the maintainer of the genetic material, ripening stage. ‡ The distance between camera and canopy is an essential piece of information to harmonize dataset and calculate the density and should be carefully monitored during an acquisition. § In case of multiple images over the same microplot.

Plant Phenomics

Acknowledgements Acknowledgments

The work received support from ANRT for the CIFRE grant of Etienne David, co-funded by Arvalis. The study was partly supported by several projects, including ANR PHENOME. Many thanks to the dataset contributors:

-Arvalis (maize) : Menainville, Nerac, Villedieu, Thenay, Castetis -Hiphen Plant (maize) : Blois, Selommes, Ermine, Pleinefougère -Institut Technique de la Betterave (ITB) : All sugarbeet datasets -Terres Inovia (sunflower) : Epoisses -INRAe (sunflower): Auzeville, Rivière Site Name Acknowledgments The French team received support from ANRT for the CIFRE grant of Etienne David, cofunded by Arvalis. The study was partly supported by several projects including ANR PHE-NOME, ANR BREEDWHEAT, CASDAR LITERAL, and FSOV "Plastix". Many thanks are due to the people who annotated the French datasets, including Frederic Venault, Xiuliang Jin, Mario Serouard, Ilias Sarbout, Carole Gigot, Eloïse Issert, and Elise Lepage. The Japanese team received support from JST CREST (Grant Numbers JPMJCR16O3, JPMJCR16O2, and JPMJCR1512) and MAFF Smart-Breeding System for Innovative Agriculture (BAC1003), Japan. Many thanks are due to the people who annotated the Japanese dataset, including Kozue Wada, Masanori Ishii, Ryuuichi Kanzaki, Sayoko Ishibashi, and Sumiko Kaneko. The Canadian team received funding from the Plant Phenotyping and Imaging Research 10 Plant Phenomics Acknowledgments We would like to thank the company "Human in the loop", which corrected and labeled the new datasets. The help of Frederic Venault (INRAe Avignon) was also precious to check the labelled images. The work received support from ANRT for the CIFRE grant of Etienne David, cofunded by Arvalis for the project management. The labelling work was supported by several companies and projects, including Canada: The Global Institute Food Security, University of Saskatchewan which supported the organization of the competition. France: This work was supported by the French National Research Agency under the Investments for the Future Program, referred as ANR-16-CONV-0004 PIA #Digitag. Institut Convergences Agriculture Numérique, Hiphen supported the organization of the competition. Japan: Kubota supported the organization of the competition. Australia: Grains Research and Development Corporation (UOQ2002-008RTX machine learning applied to highthroughput feature extraction from imagery to map spatial variability and UOQ2003-011RTX INVITA-a technology and analytics platform for improving variety selection) supported competition.

The integration of Deep Learning to estimate plant traits at Arvalis

The doctoral thesis results were obtained in the context of a collaboration between the National Research Institute for Agriculture and Environment (Institut National de Recherche en Agronomie et Environnement -INRAE), and Arvalis a private research institute funded by the French farmers. Its mission is to disseminate innovations in agriculture to farmers by conducting research and evaluating novelties for more than 60 years. Arvalis is deploying its high-throughput phenotyping technologies for its use on various trial stations throughout France, ranging from affordable and low-tech machines like LITERAL to high-tech, automated systems like Phenomobile and UAVs. Robust estimates of plant traits are then a requirement to put in production technologies based on Deep Learning. More challenges arise with production problems: the code needs to be fast enough to process several sessions of acquisitions for one night, and the results need to be appropriately tested against manual baseline. Arvalis provided all the required manual and automatic data during three campaigns for the study. In return, the codes developed for the papers were rewritten adequately for production usage, with a specific focus on the clarity of code and the execution speed and tested within the phenotyping team. I also work a lot on the new architecture of the whole processing chain and the processing code of several other Deep Learning modules, such as semantic segmentation. For instance, the plant counting pipeline developed for Chapter 1 and the wheat head counting pipeline for Chapter 3 are today used. In 2021, Arvalis conducted several non-methodological trials without manual measurements of the wheat head density, relying on the Deep Learning wheat head density estimation module.

Data Availability

The dataset is available on Zenodo (https://zenodo.org/ record/5092309).

List of Figures

Conflicts of Interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Authors' Contributions

Conflicts of Interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Global Wheat Challenge 2020 and 2021: Analysis of the competition design and winning models Etienne David

The models to be compared

We considered the best solutions as ranked using the average domain accuracy defined previously, and not strictly the winning models: for GWC_2020, the "Praxis" solution was selected although it was ranked second according to the GWC_2020 metrics (see Supplementary material S1); for GWC_2021 the "RandomTeamName" solution was selected and was consistently ranked first in 2021. Further, we use Faster RCNN as described in [18], [20] with an input size of 512px as the baseline solution. All solutions were trained on the GWC_2021 splits which means that the GWC_2020 was retrained from scratch from the provided code. The Faster-RCNN was trained during 30 epochs on a Nvidia geforce 3090 RTX with 24Go of RAM.

Metrics used to evaluate the winning solutions

In addition to the metrics presented above for GWC_2021, other metrics were used to compare the solutions better: the rates of false positive (FPR=FP/(TP+FP+FN)) and false negative (FNR=FN/(TP+FP+FN) to better quantify possible detection problems. Further, the Root Mean Square Error (RMSE), the relative RMSE (rRMSE), the bias andthe determination coefficient (r2) were used to quantify the head count and head density performances.

Results and Discussion

The GWC_2020 and GWC_2021 winning solutions

The three best solutions of the GWCs are open-source models in the form of Jupyter notebooks (Kaggle for GWC_2020) or python code (AIcrowd for GWC_2021) to reproduce the inference process. During the competition design, we expected candidates to propose domain-adaptation approaches with advanced data augmentation strategies, domain-adversarial training, along with new model architectures that could solve the difficulty of overlapping heads and small heads. Surprisingly, all winners used standard existing opensource architectures such as EfficientDet, Faster-RCNN, Yolo-v5 and Yolo-v3, without any specific domain adaptation module. The use of different architectures indicates that more than one architecture can generalise to unseen datasets. However, Yolo-v5 seems was most frequently used during GWC_2021 while it was forbidden during GWC_2020. There is a good agreement with the head density measured in the field at the plot level (Figure 8 and Table 8). The discrepancies seem to increase with the head density. Detecting all the heads in the image and in the field is more difficult because of possible occlusions in dense crops. The discrepancies may come from five main factors:

1. The spatial sampling: for Literal dataset, images and ground samples are not located at the same place, with images covering a larger area (1.4 m²) as compared to that sampled for ground-level head counting (0.7 m²).

8 Supplementary material 8.1 Limits of GWC_2020 competition design Despite their ubiquity in real-world deployments, these types of distribution shifts are currently underrepresented in the datasets widely used in the ML community today [12]. Recent papers have focused on object recognition datasets with shifts induced by synthetic transformations, such as ImageNet-C [13], which corrupts images with noise; the Backgrounds Challenge [14] and Waterbirds ( [15], which alter image backgrounds; or Colored MNIST [16], which changes the 6 colors of MNIST digits. It is also common to use data splits or combinations of disparate datasets to induce shifts, such as generalizing to photos solely from cartoons and other stylized images in PACS [17]; generalizing to objects at different scales solely from a single scale in DeepFashion Remixed [18]; or using training and test sets with disjoint subclasses in BREEDS [19] and similar datasets [13] These existing benchmarks are useful and important testbeds for method development. As they typically target welldefined and isolated shifts, they facilitate clean analysis and controlled experimentation, e.g., studying the effect of backgrounds on image classification [14], or showing that training with added Gaussian blur improves performance on real-world blurry images [18]. Moreover, by studying how off-the-shelf models trained on standard datasets like ImageNet perform on different test datasets, we can better understand the robustness of these widely-used models [13], [18], [20]- [23]. However, as we discussed in the introduction, robustness to these synthetic shifts need not transfer to the kinds of shifts that arise in real-world deployments [22]- [24], and it is thus challenging to develop and evaluate methods for training models that are robust to real-world shifts on these datasets alone.

Study objectives of WILDS

In the proposed WILDS initiative, a curated benchmark of 10 datasets is compiled with evaluation metrics and train/test splits representing a broad array of distribution shifts that ML models face in the wild (Figure 23). With WILDS, we seek to complement existing benchmarks by focusing on datasets with realistic shifts across a diverse set of data modalities and applications: animal species categorization [25], tumor identification [26], bioassay prediction [27], [28], genetic perturbation classification [29], wheat head detection [30], text toxicity classification [31], land use classification [32], poverty mapping [33], sentiment analysis [34], and code completion [35], [36].

These datasets reflect natural distribution shifts arising from different cameras, hospitals, molecular scaffolds, experiments, demographics, countries, time periods, users, and codebases.

WILDS builds on extensive data-collection efforts by domain experts, who are often forced to grapple with distribution shifts to make progress in their applications. To design WILDS, we worked with the WILDS team to identify, select, and adapt datasets that fulfilled the following criteria:

• Distribution shifts with performance drops. The train/test splits reflect shifts that substantially degrade model performance, i.e., with a large gap between in-distribution (ID) and out-of-distribution (OOD) performances. • Real-world relevance. The training/test splits and evaluation metrics are designed in conjunction with domain experts to reflect real-world scenarios. In Appendix A, we further discuss the framework we use to assess the realism of a dataset. • Potential leverage. Domain shift benchmarks must be non-trivial but also possible to solve, as models cannot be expected to generalize to arbitrary distribution shifts. We constructed each WILDS dataset to have training data from multiple domains, with domain annotations and other metadata available at training time. We hope that these can be used to learn robust models: e.g., for domain generalization, one could use these annotations to learn models that are invariant to domain-specific features [37], [38], while for subpopulation shift, one could learn models that perform uniformly well across each subpopulation [15], [39].