Alexis Mortelier
email: alexis.mortelier@unicaen.frfrancois.rioult@unicaen.fr

François Rioult

Addressing the Complexity of MOBA Games through Simple Geometric Clues (extended version)

Keywords: trajectory analysis, MOBA games, important moments highlighting, RNN

Recently, the analysis of e-sport data has particularly attracted the attention of machine learning researchers. We contribute here by proposing, on the MOBA (Multiplayer Online Battle Arena) game DotA2 (Defense of the Ancients 2), an analysis of the relevance of geometric clues relating to the behavior of the players. The geometric clues we have chosen concern the trajectories of the players and the characteristics of the polygon described by the players of a team. More precisely, for the polygon clues, we computed, for each second of the game: the convex area, the diameter, the grouping capacity (average of the distances to the barycenter), the inertia (second order moment of the distances to the barycenter) and finally the distance to the opponent's fortress. During a regression on the number of alive heroes, the learning performance of an RNN (precisely a LSTM) fed by our simple clues allowed us to evaluate their relevance in describing this complex game.

Introduction

Context Electronic sport or e-sport consists of the competitive playing of video games. This activity provides numerous traces that data science researchers like to study. Indeed, these researchers are often gamer themselves so they are familiar with the subject of their study. Moreover, e-sport has emerged as a serious discipline, operating in a similar way to sport, with the arrival of classic clubs such as Paris SG, AS Monaco, AS Roma and FC Grenoble. This craze is reflected in the particularly attractive financial amounts (up to millions of dollars) offered to the teams participating in the tournaments. For example, in 2018, The International 2018 tournament allowed the first 18 teams to share more than 25 millions Dollars (40 millions this year). E-sport is a boon for data science research because activity traces are easily and massively available. Unlike traditional sports where it is difficult to obtain accurate data, each e-sport game is archived on servers in a very precisely exploitable format. The interest of e-sport also lies in the expression of human and collective organizations, for example the tactics and strategy of a team of players, which are comparable to those implemented in real sport.

Motivations Our main motivation is to analyze MOBA games (Multiplayer Online Battle Arena games) in order to better understand the performance of the best teams. This understanding would allow us to explain the reasons for the failure or success of a team and would provide means of analysis for the games. These means could then be used to automatically produce game summaries by characterizing highlights.

However, MOBA games are particularly complex to analyze: each player can choose his avatar from among a hundred, each avatar can cast four exclusive spells and improve his skills (movement speed, attack frequency, armor, mana regeneration, etc.) thanks to six items of equipment to be chosen from among a hundred. It is therefore desirable to reduce this complexity.

We believe that the spatial organization of the players is crucial for analyzing performance in these games and that it is not necessary to examine all the minute details of a game to characterize its important moments. Moreover, MOBA games are team games and the study of spatial organization allows us to focus attention on the collective aspects of this practice rather than on individual details.

Position Compared to other works in the field, our proposal presents two originalities. Firstly, we focused our study on the collective character of the practice of a MOBA, because it is fundamentally a team game emphasizing group actions to the detriment of individual ones [START_REF] Katona | Time to die: Death prediction in dota 2 using deep learning[END_REF]. To do so, we compared the relevance of clues related to the individual position of the players on the field and the geometry of the collective organization. We are looking for simple clues to calculate and exploit and we do not wish to focus our attention on individual behaviors and characteristics (equipment, skill tree).

We therefore compared the contribution to a learning method of the individual trajectories of the heroes versus the geometric measurements on the polygon described by the players (area, diameter, distance to the objective, etc.). This approach is qualified as wrapper in [START_REF] Kohavi | The wrapper approach[END_REF]: the relevance of a descriptor is measured by its contribution to the classification or regression model, it is a method of feature selection. The target of our model was the number of living heroes in each team.

Secondly, we have decided to work only on data from games played by esport professionals, grouped in renowned teams. We wanted to study games performed by players who are used to play together and producing a high level of performance. This position is original in the literature [START_REF] Mora-Cantallops | MOBA games: A literature review[END_REF][START_REF] Katona | Time to die: Death prediction in dota 2 using deep learning[END_REF]. In order to privilege quality over quantity, we did not wish to study millions or even billions of games [START_REF] Deja | Topological clues for predicting outcomes of multiplayer online battle arena games[END_REF]. Indeed, the matchmaking of occasional players is based on their expectation of winning, which leads to the constitution of balanced teams. However, it is very frequent that these matches oppose two extremely efficient players, accompanied by extras. In this case, the collective performance is lousy.

This article is organized as follows: Section 2 presents the material and the method of our study, Section 3 delivers our results and their comments, we end with a quick review of the literature in Section 4 and conclude in Section 5.

Related work

The popularity of video games in general and e-sports in particular has led to a number of articles studying the mechanics of these games, using a data science approach. MOBA games are very relevant for comparative studies because they offer a complex but relatively formatted game, both in terms of the playground and the way the games are played. Themes range from studies of individual behaviour to game summaries, outcome prediction, the use of geometric clues and RNN [START_REF] Mora-Cantallops | MOBA games: A literature review[END_REF].

In order to make this review understandable, we start by explaining the main characteristics of a MOBA game.

MOBA games

We analyzed data from a video game, DotA2 (Defense of the Ancients 2), which is emblematic of the MOBA type: Multiplayer Online Battle Arena. These games feature two teams of five players embodying avatars called heroes. Each player develops his hero during the game by collecting gold and experience. Thanks to these resources, the hero can get new equipment items and new abilities to help him in the battle against the heroes of the opposing team.

The goal in DotA2 is to gain ground on the opponent and ultimately destroy the opposing team's fortress as a condition of victory. Each of the two fortresses (the yellow stars at the bottom-left and top-right corners in Figure 1, a.k.a Ancient) is accessible by three lanes (top, middle, bottom). These routes are defended by three towers, symbolized by a yellow circle on Figure 1. In order to destroy the towers and the opponent's fortress, each team escorts the progression of waves of minions. These minions are creatures generated in groups of four every 30 seconds, and they travel across the map from the allied fortress towards the enemy fortress. When two waves of opposite minions are facing, they fight for about twenty seconds and neutralize each other, which defines a front line. The objective of the game is thus, with the help of the heroes embodied by the players, to favor the progression of its minions in the direction of the opponent. A MOBA could be seen as a simulation of rugby with five players teams on a very large field (60s are needed to go from one fortress to the opposite) with three balls in three different lanes. A game of DotA2 lasts between 35min and 50min.

Individual actions and regulation

In spite of the strong collective character of the practice of a MOBA, individualities are studied a lot. It is indeed important to categorize players according to their behavior during the game, which provides useful information on the game's progress but also on their behavior outside the game [START_REF] Sifa | Profiling in games: Understanding behavior from telemetry[END_REF]. Behavioral telemetry can be used for this task, producing usable patterns of behavior. Analysis of in-game behavior then leads to elements of data-driven co-design: exceptional behavior is detected relative to established standards in professional games [START_REF] Block | Narrative bytes: Data-driven content production in esports[END_REF]. A visual analysis system to help game designers find key events and game parameters that lead to a snowball effect (one team overpowering the other) is proposed in [START_REF] Li | A visual analytics approach for understanding reasons behind snowballing and comeback in MOBA games[END_REF].

As gaming activity sometimes involves millions of players and competitive events with large prizes, its regulation is an essential task. For example, the phase of choosing avatars can be studied, one can formulate recommendations for winning and set out rules for choosing one character over another [START_REF] Wang | Predicting Multiplayer Online Battle Arena (MOBA) Game Outcome Based on Hero Draft Data[END_REF][START_REF] Summerville | Draft-analysis of the ancients: predicting draft picks in dota 2 using machine learning[END_REF]. The audience's adherence to the show generated also arouses interest. In particular, an analysis of the data shows a correlation between the snowball effect and the loss of audience [START_REF] Li | A multi-phased co-design of an interactive analytics system for MOBA game occurrences[END_REF].

Game summary

Performed manually, automatic generation of game summaries and their indexing are tedious tasks, as shown in [START_REF] Ahmad | Modeling individual and team behavior through spatio-temporal analysis[END_REF], which presents action labelling for the supervision of a graph system, all over only two matches. The ultimate goal is to understand the underlying tactics, complex dynamics and strategic decisions made in specific contexts. Automatic summary generation is also desirable: during a MOBA game, there are many events and fights, which ones are particularly interesting? These questions are also of interest to researchers in traditional sports (see for example [START_REF] Zawbaa | Event detection based approach for soccer video summarization using machine learning[END_REF][START_REF] Ekin | Automatic soccer video analysis and summarization[END_REF][START_REF] Stensland | Bagadus: An integrated real-time system for soccer analytics[END_REF] for football) and more recently for e-sports [START_REF] Ringer | Deep unsupervised multi-view detection of video game stream highlights[END_REF].

As there are many games, their exhaustive analysis is tedious and one wishes to obtain a summary of the important moments or, during a live transmission, to anticipate the essential actions, for example to determine the axis of broadcast. Companions applications are available, fed with elements of analysis [START_REF] Kokkinakis | Dax: Data-driven audience experiences in esports[END_REF]. In [START_REF] Chu | Event detection and highlight detection of broadcasted game videos[END_REF], [START_REF] Chu | On broadcasted game video analysis: event detection, highlight detection, and highlight forecast[END_REF] the authors design and extract game-specific features taking into account visual content, event semantics and viewer reaction. Then, they integrate clues from these three domains based on a psychological and data-driven approach to characterize the game's highlights.

Geometric clues

The analysis of a collective practice on a given field naturally lends itself to studies of the trajectories and geometry of player organizations. Thus, for football, a work similar to the one presented in this article is proposed in [START_REF] Frencken | Oscillations of centroid position and surface area of soccer teams in small-sided games[END_REF], with the aim of predicting the outcome of the match. For DotA2, the geometric clues allow to correctly predict the outcome of a match and this relevance is confirmed on 6 billion matches of League Of Legends (MOBA competitor of DotA2) via a RandomForest system with an F-measure of 92%. Regularities in the trajectories, discriminating for the winning of the match, could also be extracted [START_REF] Cavadenti | What did I do Wrong in my MOBA Game?: Mining Patterns Discriminating Deviant Behaviours[END_REF]. Behavior on a MOBA court is indeed crucial and differences between levels of play are revealed by positional clues: change of zone, distance between players, which allows a grouping of matches according to the similarity of movements [START_REF] Drachen | Skill-based differences in spatio-temporal team behaviour in defence of the ancients 2 (dota 2)[END_REF].

Use of an RNN

The chronological nature of a video game match and the popularity of neural network methods are leading to a significant use of recurrent neural network techniques, for example to predict the outcome of the match [START_REF] Qi | Dotanet: Two-stream match-recurrent neural networks for predicting social game result[END_REF][START_REF] Silva | Continuous outcome prediction of league of legends competitive matches using recurrent neural networks[END_REF][START_REF] Lan | A player behavior model for predicting win-loss outcome in MOBA games[END_REF]. This type of network can help to understand the strategic preparation phases of the game (pick/ban of heroes) [START_REF] Yu | E-sports ban/pick prediction based on bi-lstm meta learning network[END_REF]. As the game progresses, an important task is the recommendation of equipment items, supported by an LSTM in [START_REF] Yao | Hierarchical attention based recurrent neural network framework for mobile MOBA game recommender systems[END_REF] that models player interest from historical match sequences. An RNN can also predict a player's performance based on previous performances [START_REF] White | Scalable psychological momentum estimation in esports[END_REF].

Outcome

This quick overview of the contributions around DotA2 shows two things. On the one hand, the tools used in this article (geometric clues, trajectories, RNN) are part of the classic toolbox for this domain. On the other hand, the relevance of the clues is rarely discussed as we did here, and the predictions mostly address the outcome of the match or the recommendation of behavior, whereas our predictions were linked to a measure of performance.

Addressing the complexity through geometric clues

DotA2 is a team game where coordinated and synchronized movements are paramount, independently of individual behavior. We believe (as [START_REF] Deja | Topological clues for predicting outcomes of multiplayer online battle arena games[END_REF]) and show here that geometric clues from player trajectories are sufficient to reveal the complexity of this game. Considering these geometric clues alone rather than examining individual player behaviors in terms of inventory and skill tree allowed us to propose a generic approach for the MOBA genre and to work on the basis of simple hypotheses.

We therefore compared the relevance of three types of geometric or positional clues: (i) player trajectories, (ii) trajectories enriched by the coordinates of the hero's orientation vector (this data is simply provided by the game engine) and (iii) the characteristics of the polygon described by all the players of a team. We normalized the data provided by the game engine so as to obtain one measurement per second of game time.

More precisely, for the polygon representing each team, we calculated:

the area of the convex envelope; the diameter (the maximum distance between two points on the polygon); the clustering capacity (average of distances to the barycentre); the inertia (second-order moment of the distances to the barycentre); the distance from the barycentre to the opposing fortress.

We considered that a game is divided into three major phases of play:

the early-game: the first 15 minutes during which the heroes are weak and seek to obtain gold and experience to gain power. It is during this phase that ambushes are attempted to gain an advantage and ruin the opponent's progress; the mid-game: (from 16 to 35mn) some heroes are powerful enough to make team fights possible; the late-game: (36mn and beyond), the heroes have their maximum power and no longer seek to collect gold or experience: towers and the enemy fortress becomes the only objective.

DotA2 provides traces of game activities in the form of a replay file that can be retrieved after each game. For each hero played by a human, the following information is available at each second: its coordinates, its orientation, its life points, its progress in the skill tree, the items in its inventory, etc.

The 258 games we analyzed were those of a major tournament of the game, The International 20181 . Our goal was to determine clues, characteristic of the spatial organization of the players. Highlights occur approximately every three minutes and correspond to the confrontations of the players giving rise to the disappearance of some (virtual death followed by a resurrection after a variable time). In order to obtain results that are sufficiently general and do not depend on the intrinsic complexity of the game, we focused our attention on geometric clues linked to the positions of the heroes.

Wrapper Approach with RNN

We used a wrapper approach [START_REF] Kohavi | The wrapper approach[END_REF] which assumes that the relevance of a feature is related to its capacity to improve the quality of learning. Unlike the filter approach, more common in statistics, which attempts to evaluate the credit of features from the data alone, the wrapper approach attempts to identify the best subset of features to use with a learning algorithm. We used this approach to compare the suitability of trajectories with that of polygon features by examining the performance obtained by models computed on these clues.

The learning algorithm we used was a Long Short Term Memory (LSTM), a particular recurrent neural network (RNN) [START_REF] Sherstinsky | Fundamentals of recurrent neural network (rnn) and long shortterm memory (lstm) network[END_REF] architecture. RNNs are designed to handle temporally evolving data: if a cell computes an output signal from the inputs, it additionally transmits a hidden state to the next cell, that influences it. The cells of an LSTM additionally include a memory/forget mechanism that solves the problem of vanishing gradient.

The LSTM was here well adapted to our goal which was the study of a long game (35 to 50mn x 60 measures/mn). The target was the number of players alive 5 seconds into the future (as recommended by [START_REF] Chu | Event detection and highlight detection of broadcasted game videos[END_REF]), because the position of a dead hero was conventionally frozen at the place of its death, and then it was reborn at the fortress position. If this time lag was not introduced, the LSTM could use this information and introduce a bias in the results. The performance of the LSTM was evaluated by the MSE (Mean Square Error ), whether it is fed by the trajectories, the enriched trajectories or the polygon features.

The results we report in the next section were obtained on a single-layer LSTM with 128 cells, which allowed us to best assess the difference in relevance between our choices of clues. However, adding additional layers allowed the network to acquire a universal approximator behavior and thus, fed by the trajectories, to potentially capture the expressive power of the geometric clues of the polygon. The experiments we have carried out in this sense show that the more layers we added, the less noticeable the differences between the types of clues (see Figure 4 .

Finally, we performed a Student's t test to check whether there was a significant difference in relevance between the use of trajectories, enriched trajectories or polygon clues. The Student test calculates a value between two distributions called p-value. If the score is less than 0.05, then the hypothesis that the means are equal can be rejected and therefore one descriptor can be valued over the other.

Experimental conditions

In the experiments whose results are presented in the next section, we fed an LSTM with either player coordinates or polygon geometric clues and evaluated the relative relevance of these attributes by examining the MSE. The objective function was the number of living heroes in each team (from 0 to 5). We considered that the LSTM worked in regression because it had to predict two During our experiments, we also considered two types of chronology for the LSTM inputs: learning since the beginning of the game phase: the LSTM was fed continuously by a single sequence, with data between instants 0 and t and predicted a value at t + 5; sliding window learning: the LSTM was fed with multiple sequences from t -w to t and predicted at t + 5, for w a window width that we let vary between 5 and 120s.

The following section presents the results obtained in regression, in continuous and sliding window mode.

Results

Learning since the beginning

Figure 3 represents the MSE boxplots, for each phase of the game. It can be observed that, independently of the phase, the polygon clues allowed a better prediction: the median of the MSE is the lowest and the dispersion is less important. Table 1 provides the values of the corresponding Student's tests: when the cell has a green background, the means are not equal. During the early and middle of the game, the trajectories induced a slightly lower median error than the enriched trajectories, but the dispersion for the enriched trajectories is lower. For the end of the game, the results were more marked, the dispersion increased for the three types of clues and there were many more outliers. This could be explained by the fact that at the end of the game the virtual deaths are much more punitive (their duration depends linearly on the level of the player, maximum at this period). When a collective behavior led to an important moment, the model detected it, however the players were more cautious because of the stake and canceled their action at the last moment.

Figure 4 shows the evolution of the relevance when increasing the number of layers of the LSTM (each one has 128 cells). Although the polygon clues were still the most relevant, there was a convergence of the three types of clues when the number of layers increased. However, the addition of layers had a significant algorithmic cost, particularly in terms of the number of epochs. Finally, the polygons were the most relevant geometric clues for the regression problem, regardless of the game phase, both from an evaluation and computational time perspective. We also found that the LSTM converged faster when fed by the polygon clues rather than the trajectories.

Sliding window learning

Regression learning When using sliding windows, Figure 5 shows the performance of the different types of clues, depending on the width of the window. Switching to sliding windows greatly improved the results obtained with the clues on the trajectories, equally for the trajectories and the enriched trajectories. The performance of the clues on the polygon was much poorer, and using sliding windows did not allow them to be sufficiently highlighted. These clues need to be taken into account for the entire period to reveal their potential relevance.

However, in the middle and at the end of the game, the best results obtained on sliding windows by trajectories did not improve the results obtained by polygons on the continuity of the game. This is potentially due to the low frequency of events at the beginning of the game.

Conclusion

Using an LSTM that predicted the number of living heroes, we have shown that it was possible, by valuing geometric clues about the players' positions, to draw attention to important moments in a DotA2 game. Although DotA2 is a complex game, there is a strong relationship between simple positional clues and the occurrence of important moments. It is therefore not necessary to scrutinize the precise behavior of heroes in terms of attack or skill development, nor their equipment.

Among the examined clues (geometry of the polygon described by the heroes, trajectories and trajectories enriched by orientations), those on the polygon proved to be more relevant for all phases, but it was necessary to learn on the continuity of the game. When using windows of variable size, the trajectories relatively gave better results but did not reach the performance of the polygon on the continuity of the game.

We have also found that increasing the number of layers in the LSTM allowed it to behave as a universal approximator: the more layers, the less sensitive the differences in performance between the feature types, although trajectories still required more learning epochs. The polygon clues, with the development of a small amount of preliminary computation, thus allowed for a reduction in the computation required for learning.

Fig. 1 .

 1 Fig. 1. DotA2 map from http://dota2.fandom.com/wiki/Map

Fig. 2 .

 2 Fig. 2. LSTM training scheme on the continuity of the game.

Figure 2

 2 shows a schematic of our training process.

Fig. 3 .

 3 Fig. 3. Distribution of the MSE over the continuity of the game in regression.

Fig. 4 .

 4 Fig. 4. Evolution of the MSE according to the number of layers for the regression.

Fig. 5 .

 5 Fig. 5. Distribution of MSE by sliding window in regression.

Table 1 .

 1 P-values of Student test for regression.

	Phase Poly vs traj Poly vs enriched traj Enriched traj vs traj
	Early 2.989.10 -6	1.104.10 -10	0.113
	Mid 1.121.10 -11	4.633.10 -13	0.782
	Late 4.366.10 -9	1.761.10 -16	4.043.10 -5

see https://www.dota2.com/esports/ti10 for this year's edition


Dmitry Ershov

Minh-Son Phan

Joanna W Pylvänäinen

Stéphane U Rigaud

Laure Le Blanc

Arthur Charles-Orszag

James R W Conway

Romain F Laine

Nathan H Roy

Daria Bonazzi

Guillaume Duménil

Guillaume Jacquemet
email: guillaume.jacquemet@abo.fi

Jean-Yves Tinevez
email: jean-yves.tinevez@pasteur.fr

TrackMate 7: Integrating state-of-the-art segmentation algorithms and tracking pipelines

Keywords: Single-Particle Tracking, Deep Learning, Machine Learning, Cell Tracking, Lineage Tracing

TrackMate is an automated tracking software used to analyze bioimages and is distributed as a Fiji plugin. Here we introduce a new version of TrackMate. TrackMate 7 is built to address the broad spectrum of modern challenges researchers face by integrating state-of-the-art segmentation algorithms into tracking pipelines. We illustrate qualitatively and quantitatively that these new capabilities function effectively across a wide range of bio-imaging experiments.

Main text

In biosciences, object tracking is an essential image analysis technique used to quantify dynamic processes. In Life Sciences, tracking is used, for instance, to follow single particles, subcellular organelles, bacteria, cells, and whole animals. While tech developments have drastically improved image acquisition capabilities and allowed increasingly sophisticated experimental setups, they have also led to bottlenecks in downstream image analyses. Due to the sheer diversity of images, no single software can address every Life Science research tracking challenge. This has prompted the development of flexible and extensible software tracking platforms 1-5 , including TrackMate, that enable biologists to build automated tracking pipelines tailored to a specific problem.

TrackMate can now create, use, analyze and store object contours segmented from 2D images. These contours enable TrackMate to extract morphological features of the tracked objects over time. We also wrote a new application programming interface (API) to allow the integration of external components in TrackMate. We use this API to incorporate popular segmentation tools including ilastik, the Weka Trainable-Segmentation Fiji plugin, cellpose, StarDist, and the morphological segmentation tool MorphoLibJ within TrackMate. TrackMate can also import segmentation results as masks or label images and use them for tracking, making it tracking compatible with any segmentation algorithm.

Most tracking algorithms proceed in two steps. First, a detection algorithm detects or segments individual objects at each time point. Second, a linking algorithm links the detections to build tracks that follow each object over time. Importantly, the accurate detection of objects is crucial for the tracking process 6 . However, the low signal-to-noise ratio (SNR) typical of live-cell fluorescence microscopy often makes segmentation challenging. Aberrant object detection then leads to missing links and the generation of tracks that end prematurely, with multiple short tracks representing an individual object over time. Objects at high density can also be challenging to segment due to overlap or close contact. Most detection algorithms will treat tightly packed objects as a single entity, resulting in breaks in tracks or single tracks linking groups of objects. Several linking strategies can partly rescue these issues, but overall all tracking algorithms tested in 6 displayed a decreasing performance with increasing object density and decreasing SNR. Modern segmentation algorithms, in particular, those based on machine learning (ML) and deep learning (DL) approaches, can address these challenges as they excel at image segmentation tasks in low SNR and high-density images 7 .

TrackMate 4 , developed by us, is a user-friendly Fiji 8 plugin for tracking objects in fluorescence microscopy images. TrackMate offers automated and semi-automated tracking algorithms, together with advanced visualization and analysis tools. TrackMate is interactive and enables users to filter and curate tracking results based on defined parameters. As such, it can accommodate a wide range of tracking challenges. However, until now, TrackMate detectors were solely based on the Laplacian of Gaussian (LoG) filter. The LoG filter is efficient against sub-resolved particles 9 or other blob-like objects but performs poorly for textured objects, objects with complex shapes, and other imaging modalities than fluorescence. These detectors are also limited to measuring the object's position and not their shape.

Here we introduce a new version of TrackMate (TrackMate 7) rewritten to improve performance, usability, and versatility, all of which present several advantages over other available tracking tools (Table S1). In particular, we developed a new API that enables developers to integrate segmentation tools as TrackMate detectors. As examples, we provide detectors based on ilastik 10 , Weka 11 , cellpose 12 , MorphoLibJ 13 , and StarDist 14 . While the training of custom ML and DL models must be performed with external tools (using e.g. ilastik or the ZeroCostDL4Mic platform for StarDist and cellpose), popular built-in models are now fully integrated into TrackMate with a user-friendly interface and scripting capabilities. TrackMate can also import segmentation results as mask or label images for tracking, making it possible to perform tracking with any segmentation algorithm. Importantly, as TrackMate now detects object contours in every frame, we reconfigured the TrackMate data model to store, display and analyze 2D morphological features of the tracked objects over time (Figure 1). The new detectors work for 2D and 3D images when possible, but the analysis of object contours is currently limited to 2D images. These new features widely increase the breadth of TrackMate applications and capabilities (Figure 2, Movie 1-11, Supplementary Figure 1234, Supplementary manual, and tutorials) and its tracking performance (Table S2). For instance, the StarDist integration offers efficient and versatile nuclei detection in fluorescence images via the built-in model (from image set BBBC038v1 in 15 ). Our integration also provides an interface to use custom StarDist models. To illustrate this, we used custom StarDist models trained with the ZeroCostDL4Mic platform 16 to track fluorescently labeled nuclei of collectively migrating breast cancer cells, or rapidly migrating T cells from brightfield images (Figure 2a-b and Movie 1-2). Before this integration, fully automated tracking of label-free cells was not possible in TrackMate.

As TrackMate supports multi-dimensional images, users can now track objects using one channel, while measuring the changing intensities of the tracked objects in separate channels over time. As an example, we tracked the nuclei of breast cancer cells expressing a kinase translocation reporter, following changes in ERK activity in single cells as they migrated (Figure 2c, Supplementary Figure 1, and Movie 3).

To further showcase the versatility of TrackMate, we used a Weka model (trained using the Weka Fiji plugin) together with the new overlap tracker (linking algorithm based on object overlap between consecutive frames) to follow focal adhesions in endothelial cells (Supplementary Figure 2 and Movie 4). We also used an ilastik pixel classifier (trained using ilastik) to follow Neisseria meningitidis growth and correlate lineage information to single bacteria morphological measurements (Figure 2d and Movie 5). To showcase that TrackMate can now import segmentation results directly, then follow the imported objects, we tracked migrating cancer cells (fluorescent images and brightfield images), and hematopoietic stem cells ( 17 , brightfield images) previously segmented using cellpose 12 (Figure 2e, Supplementary Figure 3 and Movie 6-8).

TrackMate's new detectors can also be used to perform 3D segmentation. Indeed, by swapping the Z dimension of the source image with time, TrackMate can link 2D segmentation results across Z planes and generate a 3D segmentation result. This new feature makes the segmentation of 3D objects accessible, flexible, and possible without programming knowledge (Figure 2f, Supplementary Figure 4, and Movie 9-11).

TrackMate v7 currently offers a choice of 10 segmentation detectors (plus the integration of custom models for some of them) and five particle-linking algorithms for tracking the detected objects. To facilitate choosing an optimal combination for a specific dataset, we developed an additional module, the TrackMate helper (Supplementary manual and Supplementary Figure 5). This module is a user-friendly application that performs parameter sweeps over any combination of detectors and particle-linking algorithms. Using the ground-truth provided by the user, TrackMate helper computes the Cell-Tracking-Challenge (CTC) metrics 19 for each parameter combination and reports the optimal one for each of the CTC metrics (Table S2). In a nutshell, TrackMate helper allows the optimization of the tracking parameters for a whole dataset systematically.

Altogether, TrackMate now enables powerful segmentation approaches for tracking purposes directly in Fiji within a user interface already familiar to many. We envision that by enabling scientists to resolve complex tracking problems more efficiently, this new version of TrackMate will accelerate discoveries in Life Sciences. We expect that TrackMate will continue to evolve in the years to come. In particular, as core libraries handling 3D objects are further developed in Fiji, the analysis of 3D object contours and shapes during tracking could become an invaluable addition. TrackMate was also built as a software platform to be extended by others independently and is documented as such. As DL-based segmentation and tracking algorithms are being developed, we hope contributors will consider TrackMate as a platform to accelerate the dissemination of their work towards researchers in the Life Sciences and beyond 18,19 . c. MDA-MB-231 cells stably expressing an ERK activity reporter (ERK-KTR-Clover) and labeled using SiR-DNA were recorded live using a wide-field fluorescence microscope over 17 hours. Cell nuclei were automatically tracked over time using a StarDist model available in TrackMate (see Movie 3). For each tracked cell, the average intensity of the ERK reporter was measured in their nucleus over time (directly in TrackMate). Changes in ERK activity and in instant velocity are displayed as heatmaps (blue high, yellow low). d. The growth of Neisseria meningitidis expressing PilQ-mCherry was recorded using a spinning-disk confocal microscope. An ilastik pixel classifier, trained to segment individual bacteria, was loaded into TrackMate to follow bacteria growth. Representative fields of view and the lineage tree of the bacteria highlighted in green are displayed (see Movie 5). Changes in area and circularity of a bacterium over the tracking period are also highlighted (green track). Cell division events translate in sharp decreases in area, followed by a quasi-linear increase. The circularity roughly plateaus during cell growth then decreases before cell division. Scale bar = 25 µm. e. Glioblastoma cells migrating on a polyacrylamide gel were automatically segmented using a custom cellpose model trained in the ZeroCostDL4Mic platform. The resulting label images were automatically tracked using TrackMate (see Supplementary Figure 1 and Movie 6). Example raw and label images, as well as cell tracks, are displayed. f. MCF10DCIS.com 3D spheroids were stained for Dapi and imaged using a spinning disk confocal microscope. Across the Z volume, nuclei were detected at each Z plane using StarDist and tracked (all performed in TrackMate). Tracked nuclei were then exported as a label image to create 3D labels (see Movie 9).

Online Methods

Cells and reagents

MDA-MB-231 and U2OS cells were engineered to express the Erk KTR by first producing lentiviral particles in HEK 293FT packaging cells (ThermoFisher, R70007). Cells were co-transfected with the third generation lentiviral packaging system composed of pMDLg / pRRE (Addgene plasmid 12251), pRSV-Rev (Addgene plasmid 12253), pMD2.G (Addgene plasmid 12259), along with the pLentiPGK Puro DEST ERK KTRClover (a kind gift from Markus Covert; Addgene plasmid 90227) transfer plasmid, using Lipofectamine 3000 (ThermoFisher) in OptiMEM (Gibco, 31985070), as per the manufacturer's protocol 1,2 . After 24 hours the media was changed for complete growth medium and incubated for a further 24 hours, at which point the media was collected and filtered through a 0.45~µm syringe filter. MDA-MB-231 and U2OS cells were transduced with lentivirus for 48 hours in the presence of polybrene (8 µg/ml; Sigma, TR-1003-G), before washing and selection of stable positive cells using puromycin (2 µg/ml). Cells were then sorted by fluorescence-activated cell sorting (FACS) to isolate a population within a similar fluorescence range. MCF10 DCIS.COM cells were cultured in a 1:1 mix of DMEM (Sigma-Aldrich) and F12 (Sigma-Aldrich) supplemented with 5 % horse serum (16050-122; Gibco BRL), 20 ng/ml human EGF (E9644; Sigma-Aldrich), 0.5 mg/ml hydrocortisone (H0888-1G; Sigma Aldrich), 100~ng/ml cholera toxin (C8052-1MG; Sigma Aldrich), 10 µg/ml insulin (I9278-5ML; Sigma-Aldrich), and 1\% (vol / vol) penicillin / streptomycin (P0781-100ML; Sigma-Aldrich).

Tracking migrating breast cancer cells

Migrating MCF10DCIS.com cells were tracked using either StarDist directly implemented within TrackMate or using Cellpose and then TrackMate. To track MCF10DCIS.com cells labeled with sir-DNA using StarDist and TrackMate, a custom StarDist model was generated using the ZeroCostDL4Mic platform 3,4 . This custom StarDist model was trained for 100 epochs on 72 paired image patches (image dimensions: 1024x1024, patch size: 1024x1024) with a batch size of 2 and a mae loss function, using the StarDist 2D ZeroCostDL4Mic notebook (v1.12.2). The StarDist "Versatile fluorescent nuclei" model was used as a training starting point. Key python packages used include TensorFlow (v1.15), Keras (v2.3.1), CSBdeep (v0.6.1), NumPy (v1. 19.5) and Cuda (v10.1.243). The training was accelerated using a Tesla P100 GPU. This model generated excellent segmentation results on our test dataset (average F1 score > 0.96). This model, the training dataset as well as the code used for training is available on Zenodo 5 . In TrackMate, the StarDist detector custom model (score threshold = 0.41 and overlap threshold =0.5) and the LAP tracker (linking max distance = 30 µm; track segment splitting = 15 µm) were used. Tracks were filtered in the function of their total distance traveled and tracks shorter than 80 µm were excluded.

To track MCF10DCIS.com cells expressing lifeact-RFP (cell line described here 6 ) and labeled with sir-DNA, cells were first segmented using the ZeroCostDL4Mic cellpose 2D notebook (v1.12, 3,7 ). The cellpose model Cyto was used for the segmentation and the lifeact staining was used as the main segmentation channel. The Sir-DNA channel was used as the secondary segmentation channel. The following cellpose parameters were used Flow threshold = 0.4 and Cell probability threshold = 0, Object diameter: 50. This approach generated excellent segmentation results on our test dataset (average F1 score > 0.93). In TrackMate, the label image detector and the LAP tracker linking max distance = 30 µm; track segment gap closing = 15 µm and 2 frames; track segment splitting = 15 µm) were used. Tracks were filtered in the function of the total number of spots detected and tracks with less than 40 spots were excluded. This dataset is available on Zenodo 8,9 .

Tracking migrating T cells.

T cells migrating on ICAM-1 were automatically tracked using StarDist directly implemented within TrackMate. The StarDist model used was described previously 10 and is publicly available on Zenodo 11,12 . This model generated excellent segmentation results on our test dataset (F1 score > 0.99). In TrackMate, the StarDist detector custom model (Score threshold = 0.41 and Overlap threshold = 0.5) and the Simple LAP tracker (linking max distance = 30 µm; gap closing max distance = 15 µm, gap closing max frame gap = 2 frames) were used. This dataset is available on Zenodo 11 .

Following ERK activity in migrating cells.

MBA-MD-231 or U2OS cells stably expressing clover-ERK-KTR were seeded on fibronectin-coated (1 µg /ml) Ibidi 8 well slides (Ibidi) one day before imaging. 4h before imaging, the media was supplemented with 250 nM sirDNA (Cytoskeleton Inc) and 25 mM HEPES (Sigma). Cells were then imaged live (37°C, 5% CO 2 ) using a Nikon Eclipse Ti2-E microscope (Nikon) equipped with an sCMOS Orca Flash4.0 camera (Hamamatsu) and controlled by the NIS-Elements software (Nikon, v 5.11.01). MBA-MD-231 cells were imaged using a 20x Nikon CFI Plan Apo Lambda objective (NA 0.75), either one frame per minute for 2 h or one frame every 5 minutes for 17 h. In these experiments, a camera binning of 2x2 was used. U2OS cells were imaged using a 10x Nikon CFI Plan-Fluor objective (NA 0.3) every 5 minutes for 3 hours. Cell nuclei were automatically tracked over time by using StarDist in TrackMate.

To track the nuclei of U2OS cells, a custom StarDist model was trained using the ZeroCostDL4Mic platform 3 . The training source for the model was generated from 25 manually annotated images (Dimensions: 2048x2048) using the LOCI plugin in Fiji. The generated training source and target were randomly cropped into size 1024x1024, rotated, flipped, and multiplied by five using the Augmentor ZeroCostDL4Mic notebook 3,13 to generate a dataset of 120 paired images. The custom StarDist model was trained for 200 epochs on the 120 paired image patches (image dimensions: 1024x1024, patch size: 1024x1024) with a batch size of 2 and a mae loss function, using the StarDist 2D ZeroCostDL4Mic notebook (v1.12.2) 3 ). Key python packages used include TensorFlow (v1.15), Keras (v2.3.1), CSBdeep (v0.6.1), NumPy (v1.19.5) and Cuda (v11). The training was accelerated using a Tesla P100GPU. This model generated excellent segmentation results on our test dataset (average F1 score > 0.918). In TrackMate, the StarDist detector custom model (Score threshold = 0.41 and Overlap threshold = 0.5) and the LAP tracker (linking max distance = 20 µm; Track segment gap closing = 25 µm, Gap closing max frame gap = 10 frames) were used. Tracks were filtered in function of their track duration and tracks shorter than 34 frames (2h 40min) were excluded. This dataset is available on Zenodo.

To track the nuclei of MDA-MB-231 cells, the "Versatile fluorescent nuclei" StarDist model was used. In TrackMate, the StarDist detector (Score threshold = 0.41 and Overlap threshold = 0.5) and the LAP tracker (linking max distance = 40 µm; Track segment splitting = 30 µm) were used. Tracks were filtered in function of their duration and only the tracks spanning the whole movie were considered for further analysis (directly in TrackMate). For each tracked cell, the average intensity of the ERK reporter was measured in their nucleus over time (directly in TrackMate). To visualize the changes in ERK activity over time, results were uploaded to PlotTwist 14 , data were rescaled between 0 and 1, and visualized as heatmaps. This dataset is available on Zenodo 15 .

Tracking Mouse hematopoietic stem cells migrating in hydrogel microwells.

Mouse hematopoietic stem cells migrating in a hydrogel microwell 16 were automatically segmented using cellpose (Cyto model) implemented in the ZeroCostDL4Mic platform 3,7 . The following Cellpose parameters were used: flow threshold = 0.4 and cell probability threshold = 0, object diameter = 17. This approach generated excellent segmentation results on our test dataset (average F1 score > 0.93). The resulting label images were automatically tracked using TrackMate. In TrackMate, the label image detector and the LAP tracker (linking max distance = 30 µm; track segment gap closing = 15 µm and 2 frames; track segment splitting = 15 µm) were used. Spots were filtered in function of their circularity and area. Tracks were filtered in function of the total distance traveled; tracks shorter than 80 µm were excluded. This dataset is available from the Cell Tracking Challenge website (http://celltrackingchallenge.net/).

Tracking glioblastoma-astrocytoma cells migrating on a polyacrylamide gel

Glioblastoma-astrocytoma U373 cells migrating on a polyacrylamide gel were automatically segmented using a custom cellpose trained using the ZeroCostDL4Mic platform 3,7 . This cellpose model was trained for 500 epochs on 214 paired image patches (image dimensions: 520x 696), with a batch size of 8, using the cellpose ZeroCostDL4Mic notebook (v 1.13). The cellpose Cyto2 model was used as a training starting point. The training was accelerated using a Tesla K80 GPU. The following cellpose parameters were used for inference: flow threshold = 0.4 and cell probability threshold = 0, object diameter = 100. This approach generated excellent segmentation results on our test dataset (average F1 score > 0.97). The resulting label images were automatically tracked using TrackMate. In TrackMate, the label image detector and the simple LAP tracker (linking max distance = 100 µm; track segment gap closing = 100 µm and 10 frames) were used. This dataset is available from the Cell Tracking Challenge website (http://celltrackingchallenge.net/) and on Zenodo 17 .

Neisseria meningitidis sample preparation and imaging

The Neisseria meningitidis strain 2C43 18 pilQ/pilQ-mCherry ind was grown on GCB agar plates (Difco) containing Kellog's supplements, 3µg/ml vancomycin and 5~µg/ml chloramphenicol at 37°C in a moist atmosphere containing 5% CO 2 . The pMGC17 plasmid was designed in order to generate the 2C43 pilQ/pilQ-mCherry ind strain allowing IPTG-inducible expression of the type IV pilus secretin protein PilQ with a carboxy-terminal fusion to mCherry expressed from the Neisseria meningitidis chromosome.

First, pilQ was PCR-amplified from Neisseria meningitidis chromosomal DNA] with primers pilQ-F: TTAATTAAAGGAGTAATTTTATGAATACCAAACTGACAAAAATCGTCGACTCAATAGCGCAGGCTGTTG C

This PCR fragment was cloned in a pCRII-Blunt-TOPO vector (Invitrogen). Then, the mCherry ORF was PCR-amplified with a forward primer containing a region homologous to the 3' of pilQ (minus the stop codon) as well as a Gly-Ser-Gly linker, and a reverse primer containing a SalI restriction site and a region homologous to the TOPO vector (MUTmChCT-F: AGCCTGCGCTATGGTTCCGGTGTGAGCAAGGGC, MUTmChCT-R: CTGCAGAATTCGCCCTTGTCGACTCACTTGTACAG).

This PCR fragment was used as a mutagenesis megaprimer to amplify pilQ from the TOPO vector 19 . Finally, this vector was digested with PacI and SalI restriction enzymes and the resulting insert ligated into pMGC10. The pMGC10 plasmid was generated by inserting the lacI gene and the lac promoter in the pMGC3 plasmid [START_REF] Soyer | Early sequence of events triggered by the interaction of Neisseria meningitidis with endothelial cells[END_REF] . The fragment of interest was PCR amplified from the pMMB207 plasmid [START_REF] Morales | A series of wide-host-range low-copy-number vectors that allow direct screening for recombinants[END_REF] using primers (LacIF2: GAA TTC GCT AAC TTA CAT TAA TTG CGT TGC, LacIPR: GTC GAC GAT CTT AAT TAT TTC CTG TGT GAA ATT GTT ATC CG) and cloned in pMGC3 using EcoRI and SalI restriction. The pMGC17 plasmid was used to transform Neisseria meningitidis, generating an intermediate strain that carries both a native pilQ and pilQ-mCherry. This strain was then transformed with chromosomal DNA from a pilQ mutant strain [START_REF] Geoffroy | Large-Scale Analysis of the Meningococcus Genome by Gene Disruption: Resistance to Complement-Mediated Lysis[END_REF][START_REF] Georgiadou | Large-scale study of the interactions between proteins involved in type IV pilus biology in Neisseria meningitidis: characterization of a subcomplex involved in pilus assembly[END_REF] .

Bacteria in exponential phase from a 2 hours pre-culture in RPMI+10% FBS supplemented with 100 µM IPTG at 37°C and 5% CO 2 were diluted to an optical density of 0.015 (~1.5x10 7 bacteria/ml) and dropped onto a 2% agarose gel supplemented with 100~µM IPTG. Once the bacteria-containing droplet had dried up, the agar pad was flipped down onto a Fluorodish (Ibidi, 60~µm-Dish, 35~mm high Glass bottom). Fluorescently labeled proliferating bacteria were acquired using an inverted spinning-disk confocal microscope (Nikon, TI Eclipse) equipped with a 100X immersion objective (PlanFluor, NA = 0.5 -1.3) at 37°C in the presence of 5\% CO 2 . Bacterial fluorescence was imaged in time-lapse at a 5 min frame rate with an exposure time of 300 ms for 5.5 hours and recorded with a CMOS Camera (Photometrics, 95BPrime) using Metamorph Imaging Software (Molecular Devices). The focus was maintained with the Perfect Focus System (PFS, Nikon).

We trained an ilastik model using the Pixel Classification workflow using images from these experiments. This model was then used in the TrackMate-Ilastik detector, with a threshold on the probability map of 0.5. Spurious detections larger than 4000 µm 2 were removed prior to linking. We used the LAP tracker for linking, using a max linking distance of 1 µm, with a max gap-closing distance of 2 µm over a maximum of 2 frames, and detecting cell divisions over a maximal distance of 2 µm. We then filtered out tracks that start after 43 minutes. This dataset is available on Zenodo [START_REF] Blanc | Neisseria meningitidis bacterial growth[END_REF] .

Tracking focal adhesions in endothelial cells.

Live imaging of the endothelial cells expressing Paxillin-eGFP was described previously [START_REF] Hakanpaa | Targeting β1-integrin inhibits vascular leakage in endotoxemia[END_REF] . Briefly, human dermal microvascular blood endothelial cells expressing Paxillin were imaged using a Marianas spinning disk confocal microscope. This microscope was controlled by SlideBook 6 (Intelligent Imaging Innovations, Inc.), equipped with a Yokogawa CSU-W1 scanning unit, an inverted Zeiss Axio Observer Z1 body, and a 100x, NA 1.4 oil (Plan-Apochromat, M27) objective. Images were acquired every two minutes using an Orca Flash4 sCMOS camera (chip size 2048x2048; 2x2 camera binning enabled; Hamamatsu Photonics), at 37°C and in the presence of 5% CO2. Acquired images were then processed using Fiji to remove background (rolling ball radius: 10 pixels), compensate for bleaching (exponential fit method), and correct drifting (StackReg, Rigid body). A custom Weka pixel classifier was then trained in Fiji to segment the focal adhesions. In TrackMate, the Weka detector (Threshold on probability = 0.5) and the overlap tracker (min IoU = 0.3, scale factor = 1) were used. This dataset is available on Zenodo [START_REF] Jacquemet | Tracking focal adhesions with TrackMate and Weka -tutorial dataset[END_REF] .

3D segmentation based on the association of 2D segmentation results.

To form spheroids, MCF10 DCIS.com cells were seeded as single cells, in standard growth media, at low density (3,000 cells per well) on growth factor reduced (GFR) Matrigel-coated glass-bottom dishes (coverslip No. 0; MatTek). After 12 h, the medium was replaced by a normal growth medium supplemented with 2\% (vol/vol) GFR Matrigel. After six days, spheroids were fixed with 4\% PFA for 10 min at room temperature and labeled using Dapi. Spheroids were then imaged using a spinning-disk confocal microscope (Z step = 0.5 µm). The spinning-disk confocal microscope used was a Marianas spinning disk imaging system with a Yokogawa CSU-W1 scanning unit on an inverted Zeiss Axio Observer Z1 microscope (Intelligent Imaging Innovations, Inc.) equip\-ped with a 100x (NA 1.4) oil, Plan-Apochromat, M27 (Zeiss). To generate 3D labels, nuclei were detected and tracked across the Z volume using StarDist implemented in TrackMate.

In TrackMate, the StarDist detector (score threshold = 0.41 and overlap threshold = 0.5) and the LAP tracker (linking max distance = 1 µm, track merging and splitting enabled) were used. Detected spots were filtered in function of their mean intensity to exclude spots with weak intensities. Tracks were filtered in function of the number of spots per track and only the tracks with more than 3 spots were considered for further analysis (directly in TrackMate). In TrackMate, tracked nuclei were then exported as a label image to create 3D labels. 3D labels were then visualized using the FPBioimage software [START_REF] Fantham | A new online tool for visualization of volumetric data[END_REF] . The video was also generated using the FPBioimage software. This dataset is available on Zenodo [START_REF] Tinevez | Segmenting cells in a spheroid in 3D using 2D StarDist within TrackMate[END_REF] .

Confocal images of Arabidopsis Thaliana floral meristem [START_REF]Original stacks and segmented data[END_REF] and light-sheet images of a developing Drosophila Melanogaster embryo [START_REF] Ulman | An objective comparison of cell-tracking algorithms[END_REF][START_REF] Maška | A benchmark for comparison of cell tracking algorithms[END_REF] were automatically segmented using Cellpose (Cyto2 model) implemented in the ZeroCostDL4Mic platform 3,7 . The following Cellpose parameters were used: flow threshold = 0.4 and cell probability threshold = 0, object diameter = 0. This approach generated excellent segmentation results on our test datasets (Arabidopsis Thaliana floral meristem, average F1 score > 0.97; Drosophila Melanogaster embryo, average F1 score > 0.89). To generate 3D labels, the 2D label images were tracked using TrackMate. In TrackMate, the label image detector and the simple LAP tracker were used. The videos were generated using the FPBioimage software.

• SEG -Segmentation accuracy. This metric measures how close the segmentation results are to the ground-truth object contours. It varies from 0 to 1, 1 indicating a perfect match with the ground truth. It is independent of the linking step. This is the metric to optimize when getting an accurate contour for all objects is essential.

• TRA -Tracking accuracy. This metric measures the normalized distance between the tracking results and the ground truth. TRA varies from 0 to 1, with 1 indicating a perfect match between the result and the ground truth.

• DET -Detection quality. DET reflects how much effort is required to edit the tracking results to match the ground truth. This metric extends the overall performance (OP) described in the original CTC paper [START_REF] Ulman | An objective comparison of cell-tracking algorithms[END_REF] and is subsequently introduced as a broad tracking performance measure. It combines the SEG and the TRA metrics and also varies from 0 to 1, 1 being the best.

• CT -Complete tracks. CT measures the fraction of ground truth tracks that a given method is able to reconstruct from the frame they appear to the frame they disappear. It varies from 0 to 1, with 1 indicating that all the tracks have been reconstructed fully. This is the metric to optimize when obtaining perfect cell lineages is required.

• TF -Track fractions. TF is the average ratio of the longest matching track with respect to the reference track. It also varies from 0 to 1, with 1 being the best case. It is the metric to optimize when following single objects over their whole lifetime, and measuring their dynamic is important.

• CCA -Cell cycle accuracy. This metric measures how accurately an algorithm reconstructs the length of the cell cycle. It varies from 0 to 1, with 1 indicating that all cell cycles measured on the source movie are correct. This is the metric to optimize when the experiment aims to measure cell cycle length accurately.

• BC -Branching correctness. BC measures how efficient a selected tracking method is at detecting division events. It varies from 0 to 1, with 1 indicating that all cell division events have been properly detected at the right time.

• TIM -Execution time. TIM measures the execution time in seconds. The lower, the better.

Migrating cancer cells imaged using fluorescence microscopy to visualize their nuclei

In this dataset, cells were labeled to visualize their nuclei using fluorescence microscopy. Nuclei can be segmented using several Deep-Learning (DL) based algorithms such as StarDist and cellpose that both ship a built-in model capable of segmenting nuclei. We also trained a custom StarDist model using other movies of the same dataset to test whether it would improve our tracking results. Since nuclei are more-or-less round and convex objects, they can be detected with the Laplacian-of-Gaussian (LoG) detector or the Difference-of-Gaussian (DoG) detector. Therefore we tested the performance of these five detectors to track this dataset. For the linking step, we used the Linear-Assignment-Problem (LAP) framework (first introduced in [START_REF] Jaqaman | Robust single-particle tracking in live-cell time-lapse sequences[END_REF] ) with or without cell-division detection, the Kalman filter tracker, and the nearest-neighbor tracker. Altogether we tested 5578 valid tracking parameters combinations.

We find that the StarDist model we trained only ranks 3rd for the SEG metric, after the cellpose builtin model and the StarDist builtin model, despite only marginally. We observe that the custom StarDist model tends to overestimate the nuclei size but detects fewer false positives than the StarDist and cellpose builtin models. This illustrates the sensitivity of the CTC metrics and reflects that different persons generated the annotation used to train the model and the ground-truth data. The LoG and DoG detectors use a circle of a specified diameter instead of a cell contour. They should not be used to yield accurate segmentation results and therefore lead to a poor SEG metric.

Regardless of the setting used, all TRA scores are high. We note that our custom StarDist model scores the best for this metric, reflecting its better robustness against false positive and false negative detections. The DET and the TF metrics also privilege this detector. We find that the tracker choice has only a small impact on these three metrics. Even the nearest neighbor tracker performs relatively well here. This result is likely due to the relatively simple motility displayed by these cells. Indeed, the nuclei move in a coordinated manner, and they do not cross and seldom touch another. These results illustrate that the detection step is crucial for such tracking problems.

The CT scores are comparatively lower, which was also noted in [START_REF] Ulman | An objective comparison of cell-tracking algorithms[END_REF] for other trackers and datasets. The CT metric is very sensitive to the slightest inaccuracy in object linking and in detecting cells and their division. The best CT score is obtained when our StarDist custom model is combined with the LAP linker that enables cell division. Using our StarDist custom model with the LAP linker without cell division detection lowers the CT score by 20%. The BC score is non-zero only when using the LAP tracker with cell division. The CT score is also strongly influenced by the choice of the detector. This dataset illustrates the importance of choosing an adequate detector to obtain high tracking accuracy, even when analyzing high-quality images acquired at a high frame rate. Modern segmentation algorithms based on DL perform better than classical detectors, even with objects as simple as nuclei. However, their use comes at the cost of execution time. Indeed, the TIM metric goes from about 2 seconds when using the LoG detector to more than 2 minutes with our custom StarDist detector. However, 2 minutes of computation time remain acceptable and could be diminished further by using GPU accelerated computation, which was not the case here.

Migrating T-cells imaged using phase-contrast microscopy

This movie is a difficult use case for the previous version of TrackMate and for many other tracking tools. Indeed, tracking cells imaged in phase-contrast remains challenging as many classical segmentation techniques are optimized for fluorescence microscopy images. However, DL algorithms excel at segmenting these types of images. We trained a custom StarDist model using similar images and used this model for tracking. We also tested the LoG and DoG detectors to serve as a comparison. Most cells in the movie display a dark ring on their contour, giving a solid response for the LoG and DoG filters. Therefore, they could serve as a crude cell detector that does not require a special algorithm, at the price of many false negatives and positives, particularly at the cell periphery.

Without surprise, we find that our StarDist custom detector has the best score for all metrics except TIM. This dataset shows how valuable modern segmentation techniques are to address complex tracking problems and how important it is for them to be customizable.

Neisseria meningitidis bacteria growth.

With this dataset, we want to follow bacteria cell division dynamics and how the morphology of single bacteria evolves during the cell cycle. For the quantitative assessment, we took a subset of frames of the data shown in Figure 2b and Movie 2. The objects to track are small and round but elongate before cell division. Due to many division cycles, the number of objects to follow goes from 4 to 400. Therefore we need to use a detector that can accommodate shape changes and a tracker that can detect cell divisions. We tested the ilastik detector with a model trained to segment the cell space inside the membrane. Because the membranes are fluorescently labeled, the cells form an edge object suitable for segmentation with the MorphoLibJ detector. We also included the LoG and DoG detector used on the inverted image. Indeed, on the inverted image, the cells resemble bright objects surrounded by a dark ring and are likely to elicit a strong response from the LoG and DoG filters.

TrackMate ships two linkers that can accommodate cell divisions, the LAP tracker and the overlap tracker. The overlap tracker computes the linking costs by measuring the Intersection-over-Union (IoU) metric and is suitable when the objects to track do not move too much from one frame to another one, which is the case here.

Combining the ilastik detector with the Overlap tracker consistently gives good results on most CTC metrics. It yields the best segmentation accuracy with a score of 0.726. Since the cells are small and sampled over a few pixels, this metric is very sensitive to slight differences between the results and the ground truth, which explains why it does not reach a higher value. Combining the ilastik detector with the Overlap tracker also provides the best TRA, DET, CT, TF, and BC metrics. The same detector used with the LAP tracker yields close but slightly lower values than the Overlap tracker. Despite being the intuitive choice of a segmentation algorithm for this kind of image, the morphological segmentation algorithm of the MorphoLibJ detector gives results that are bested by the ilastik detector. Visual inspection indicates that lower robustness in the segmentation phase mainly causes these lower values. Indeed, the MorphoLibJ detector often incorrectly segments cells close to a division event, negatively affecting all CTC metric values. Surprisingly, the DoG detector gives the best results for measuring the cell-cycle length. However, the results are only marginally better than the ilastik detector results and come at the price of making it impossible to measure the dynamics of morphological features such as the area and circularity over time.

Glioblastoma-astrocytoma cells.

Here we want to follow cell shape changes over time (see Supplementary Manual). This dataset was imaged using brightfield microscopy at high resolution. There are only a few cells to track in the movie. In addition, the cells do not cross or overlap, simplifying the linking step. However, the cells have a complex shape.

To strive for the best possible segmentation, we trained a custom cellpose model. This model was trained outside of TrackMate using the cellpose ZeroCostDL4Mic notebook and the Cyto2 pre-trained model as a starting point (see Methods).

As TrackMate is integrated in Fiji and given the excellent image quality, we also tested a more basic segmentation strategy. We first preprocessed the source image by applying a variance filter (with a radius of 9). Then, we used the Thresholding detector in TrackMate to track the cells. In this case, we filtered out spurious spots with an area lower than 1000 μm2. Finally, as the cells display some texture that can elicit a weak response from the LoG and DoG filters, we also used the detectors based on these two filters as a basis for comparison.

As expected, the cellpose detector used with a custom model yields a satisfactory SEG score on this dataset. The cell borders have such a low contrast that the other detectors fail to achieve better segmentation accuracy. However, the cellpose detector sometimes misses cells that touch the image's border. In turn, the missing detections negatively affect the metrics that require complete tracks, such as TRA, TF, and CT. The Thresholding detector on the variance image offers better results for these metrics. In practice, reconciling an accurate estimation of the cell's shape over an extended observation period would require acquiring a larger field of view to avoid measurements and tracking using the cellpose detector being perturbed by the image border.

Movie 2 -Using StarDist within TrackMate to track migrating T cells

Activated T cell plated ICAM-1 were recorded using a brightfield microscope and automatically tracked using StarDist integrated within TrackMate. The Color indicates mean speed.

Movie 5 -Using ilastik within TrackMate to follow bacteria growth

The growth of Neisseria meningitidis expressing PilQ-mCherry was recorded using a spinning-disk confocal microscope. An ilastik pixel classifier trained to segment individual bacterium was loaded directly into TrackMate to follow bacteria growth.

Movie 6 -Using cellpose and TrackMate to track Glioblastoma-astrocytoma cells migrating on a polyacrylamide gel

Glioblastoma-astrocytoma cells migrating on a polyacrylamide gel were automatically segmented using a custom cellpose model. The resulting label images were tracked using TrackMate.

Movie 7 -Using cellpose and TrackMate to track stem cells

Mouse hematopoietic stem cells migrating in a hydrogel microwell were automatically segmented using cellpose. The resulting label images were tracked using TrackMate. In the bottom left panel, the color of the object indicates the distance traveled (red longest distance, blue shortest distance). In the bottom right panel, track colors indicate ID.

Movie 8 -Using cellpose and TrackMate to track migrating cancer cells

MCF10DCIS.com cells expressing lifeact-RFP, labeled with Sir-DNA, were recorded using a spinning-disk confocal microscope. Cells were segmented using cellpose, and label images were tracked using TrackMate. The color indicates ID. Movie 9 -Using StarDist 2D within TrackMate to generate 3D labels MCF10 DCIS.com spheroids were imaged using a spinning-disk confocal microscope. To generate 3D labels, nuclei were detected and tracked across the Z volume using StarDist implemented in TrackMate.

The 3D rendering was performed using the FPBioimage software.

Figure 1 :

 1 Figure 1: The new capabilities of TrackMate. TrackMate can now create, use, analyze and store object contours segmented from 2D images. These contours enable TrackMate to extract morphological features of the tracked objects over time. We also wrote a new application programming interface (API) to allow the integration of external components in TrackMate. We use this API to incorporate popular segmentation tools including ilastik, the Weka Trainable-Segmentation Fiji plugin, cellpose, StarDist, and the morphological segmentation tool MorphoLibJ within TrackMate. TrackMate can also import segmentation results as masks or label images and use them for tracking, making it tracking compatible with any segmentation algorithm.

Figure 2 :

 2 Figure 2: TrackMate can be used to track objects from a wide variety of bio-imaging experiments. a. Migration of MCF10DCIS.com cells, labeled with SiR-DNA, recorded using a spinning disk confocal microscope and automatically tracked using a custom StarDist model loaded in TrackMate (see also Movie 1). Detected cells and their local tracks (colors indicate track ID) are displayed. Scale bar = 250 µm. b. The migration of activated T cells plated on ICAM-1 was recorded using a brightfield microscope and automatically tracked using a custom StarDist model loaded in TrackMate (see also Movie 2). Detected cells (colors indicate the mean track speed; blue slow-moving cells, red fast-moving cells) and their local tracks (colors indicate track ID) are displayed. Scale bar = 250 µm. c. MDA-MB-231 cells stably expressing an ERK activity reporter (ERK-KTR-Clover) and labeled using SiR-DNA were recorded live using a wide-field fluorescence microscope over 17 hours. Cell nuclei were automatically tracked over time using a StarDist model available in TrackMate (see Movie 3). For each tracked cell, the average intensity of the ERK reporter was measured in their nucleus over time (directly in TrackMate). Changes in ERK activity and in instant velocity are displayed as heatmaps (blue high, yellow low). d. The growth of Neisseria meningitidis expressing PilQ-mCherry was recorded using a spinning-disk confocal microscope. An ilastik pixel classifier, trained to segment individual bacteria, was loaded into TrackMate to follow bacteria growth. Representative fields of view and the lineage tree of the bacteria highlighted in green are displayed (see Movie 5). Changes in area and circularity of a bacterium over the tracking period are also highlighted (green

Movie 3 -Movie 4 -

 34 Measuring ERK activity in migrating cancer cells MDA-MB-231 cells expressing ERK-KTR-GFP and labeled with Sir-DNA, were recorded using a widefield microscope and automatically tracked using StarDist integrated within TrackMate. Only tracks remaining in the field of view over the whole duration of the movie are displayed. The color indicates ID. Using Weka within TrackMate to track focal adhesions Endothelial cells expressing Paxillin-GFP were recorded live using a spinning-disk confocal microscope. A custom Weka pixel classifier trained to segment focal adhesion was then loaded directly into TrackMate to track focal adhesions. In the middle panel, focal adhesions are color-coded to indicate their lifetime (red, long-lived, blue short-lived). In the right panel, track colors indicate ID.

Movie 10 -

 10 Using cellpose 2D and TrackMate to segment 3D images of Arabidopsis Thalian} floral meristem

Acknowledgments

The integration of existing algorithms as new detectors in TrackMate has been made possible thanks to the high quality of the code, documentation, and support provided by their respective authors. In particular, we would like to thank Anna Kreshuk, David Legland, Dominik Kutra, Ignacio Arganda-Carreras, Carssen Stringer, Marius Pachitariu, Martin Weigert, Siân Culley, and Uwe Schmidt. We can only hope for TrackMate to reach such a standard of quality to become a better tool of Science. We are also grateful for the support and help of the bioimage analysis community, in particular Curtis Rueden, Jan Eglinger, Nicolas Chiaruttini, Romain Guiet, Olivier Burri, Valdimír Ulman, Tobias Pietzsch, and Pavel Tomancak. We thank Helen Blau for giving us the permission to use the "mouse hematopoietic stem cells in hydrogel microwells" dataset made available on the Cell Tracking Challenge website. The authors thank Dr. Hellyeh Hamidi for her critical reading of the manuscript.

This study was supported by France BioImaging (Investissement d'Avenir; ANR-10-INBS-04, JYT), the Academy of Finland (GJ), the Sigrid Juselius Foundation (GJ), the Cancer Society of Finland (GJ), the Åbo Akademi University Research Foundation (GJ, CoE CellMech), the Drug Discovery and Diagnostics strategic funding to Åbo Akademi University (GJ) and the European Union's Horizon 2020 research and innovation program under Marie Sklodowska-Curie grant agreement 841973 (JRWC). JWP was supported by Health Campus Turku 2.0 funded by the Academy of Finland. RFL was supported by an MRC Skills development fellowship (MR/T027924/1). The Cell Imaging and Cytometry Core facility (Turku Bioscience, University of Turku, Åbo Akademi University, and Biocenter Finland) and Turku Bioimaging are acknowledged for services, instrumentation, and expertise.

Data availability statement

The version of TrackMate described here is available in the Fiji software 8 by simply updating it. TrackMate is documented on the ImageJ wiki: https://imagej.net/plugins/trackmate/ and the documentation for the new features can be accessed from https://imagej.net/plugins/trackmate/trackmate-v7-detectors. We also provide 14 test datasets that are made available via a dedicated Zenodo collection (https://zenodo.org/communities/trackmate/).

Author contributions

GJ and JYT conceived the project; JYT wrote source code; GJ, JWP, NHR, and LLB performed the image acquisition of the test and example data; GJ, JWP, RFL, JYT, MSP, DE, and SUR tested the code; JRWC, DB, GD and ACO provided critical reagents; GJ, JWP, JYT, MSP, DE, SUR, and JYT wrote the documentation and tutorials.; GJ and JYT wrote the manuscript with input from all co-authors.

Declaration of Interests

The authors declare no competing interests. Supplementary material for: TrackMate 7: Integrating state-of-the-art segmentation algorithms and tracking pipelines.

Supplementary notes.

Supplementary note 1: Quantitative comparison of tracking performance using TrackMate-Helper.

Introduction.

TrackMate now offers multiple detections and linking algorithms. Each of these algorithms needs to be configured with a parameter set that can take a wide range of values. While adequate values for these parameters can often be estimated intuitively, a systematic approach is often desirable. Indeed optimizing tracking parameters through visual assessment can be difficult when following many objects. To this end, we developed a new tool called TrackMate-Helper, which performs automatic parameter sweeps over any combination of detector and tracker available in TrackMate and measures the tracking results accuracy using the Cell-Tracking-Challenge (CTC) metrics [START_REF] Ulman | An objective comparison of cell-tracking algorithms[END_REF] . TrackMate-Helper requires an input image and the corresponding tracking ground-truth. Importantly, TrackMate-Helper is built as an end-user tool with a user-friendly interface that conveniently allows configuring parameter sweeps over many combinations of tracking parameters. We envision that this systematic approach will benefit medium and high-throughput automatic tracking studies. By optimizing the tracking parameters on one movie, users will find optimal tracking parameters for the rest of their dataset.

Here, we used TrackMate-Helper to assess the performance of TrackMate on four datasets that cover a wide range of biological and imaging situations:

1. Migrating cancer cells imaged using fluorescence microscopy to visualize their nuclei. In this dataset, the cells are densely packed and divide during the experiment (Figure 2a, Movie 1).

2.

Migrating T-cells imaged using phase-contrast microscopy (Figure 2b, Movie 2).

3. Neisseria meningitidis bacterial growth on agar pads. The bacteria are fluorescently labeled to visualize their membrane. Starting from a few single bacteria, the cells quickly divide several times and swarm the field of view (Figure 2d, Movie 5).

4. Glioblastoma-astrocytoma cells imaged using bright-field microscopy at high resolution (Figure 2e, Supplementary Figure 3, Movie 6).

Table S2 reports the best results for each detector and tracker combination. We tested from 3000 to 20000 different parameter settings for each combination, but Table S2 only lists the result with the optimal settings for each CTC metric. We discuss these results below. You can find the source images, the ground-truth files, the Trackmate-Helper configuration files, and results on Zenodo:

The CTC metrics.

The CTC project defines eight metrics that correspond to different possible features of interest depending on the final goal of the analysis. They are defined in [START_REF] Ulman | An objective comparison of cell-tracking algorithms[END_REF] , and we describe them here briefly: Supplementary Tables Supplementary Table 1: Feature comparison of recent open-source tracking tools.

This table compares the features of multiple open-source tracking tools designed with a user interface that are popular among biologists. 1. The table does not include DeepTrack2.0 [START_REF] Midtvedt | Quantitative digital microscopy with deep learning[END_REF] , btrack [START_REF] Ulicna | Automated Deep Lineage Tree Analysis Using a Bayesian Single Cell Tracking Approach[END_REF] , or 3DeeCell-Tracker [START_REF] Wen | 3DeeCellTracker, a deep learning-based pipeline for segmenting and tracking cells in 3D time lapse images[END_REF] , as these tools were developed as a library without a user interface. A user interface for DeepTrack2.0 is currently being designed but has not yet been released. btrack can be used with Napari, but tracking still requires Python knowledge. 2. The Spot tracker in Icy can use the output of the Spot detector and HK-Means plugins for tracking. 3. 3D visualization: '2D slice': View of the data in an XY plane; 'Orthogonal sections': View of the data in XY, XZ, YZ planes; 'Arbitrary slices': View of a 2D slice through the 3D data which orientation can be set arbitrary; 'Full' 3D visualization of the 3D data. LoG: Laplacian of Gaussian.

The table with the qualitative comparison

Supplementary Table 2: TrackMate tracking performance on test datasets

Cell-Tracking-Challenge metrics were computed using TrackMate Helper for four of our test datasets (See supplementary note 1 for more information). Only the best results for each TrackMate detector and tracker combination are displayed. Confocal images of Arabidopsis thaliana floral meristem were segmented using cellpose 2D implemented in ZeroCostDL4Mic. TrackMate was used to track the 2D labels across the Z volume and generate 3D labels. The FPBioimage software was used to perform the 3D rendering.

Supplementary Figures

Movie 11 -Using cellpose 2D and TrackMate to segment 3D images of a developing

Drosophila melanogaster embryo

Light-sheet microscopy images of a developing Drosophila melanogaster embryo were segmented using cellpose (2D). TrackMate was then used to track the 2D labels across the Z volume and generate 3D labels. The FPBioimage software was used to perform the 3D rendering.