Andres M Gonzalez

Jannis Uhlendorf

Joé Schaul

Eugenio Cinquemani

Gregory Batt

Giancarlo Ferrari-Trecate

Identification of biological models from single-cell data: a comparison between mixed-effects and moment-based inference

Keywords: Parameter identification, single-cell modeling, mixed-effects model, chemical master equation, moment-based approximation, HOG pathway in yeast Identification de paramètres, modèles de cellules uniques, modèles à effets mixtes, équation maitresse chimique, approximation basée

Experimental techniques in biology such as microfluidic devices and time-lapse microscopy allow tracking of the gene expression in single cells over time. So far, few attempts have been made to fully exploit these data for modeling the dynamics of biological networks in cell populations. In this paper we compare two modeling approaches capable to describe cell-to-cell variability: Mixed-Effects (ME) models and the Chemical Master Equation (CME). We discuss how network parameters can be identified from experimental data and use real data of the HOG pathway in yeast to assess model quality. For CME we rely on the identification approach proposed by Zechner et al. (PNAS, 2012), based on moments of the probability distribution involved in the CME. ME and moment-based (MB) inference will be also contrasted in terms of general features and possible uses in biology.

Identification de modèles biologiques à partir de données en cellules uniques

Résumé : Des techniques expérimentales en biologie comme la combinaison de microfluidique et de vidéo-microscopie permettent de suivre l'expression des gènes au cours du temps dans des cellules individuelles. L'utilisation de ces données pour modéliser la dynamique des réseaux biologiques dans des populations de cellules reste encore rare. Dans ce travail, nous comparons deux approches de modélisation capables de décrire la variabilité entre cellules: modèles à effets mixtes (ME) et modèles d'équations maitresse chimique (CME). Nous discutons de l'identification des paramètres des modèles à partir de données expérimentales et utilisons des données réelles de la voie de transduction du signal HOG chez la levure pour tester la qualité des modèles. Pour la CME, nous utilisons l'approche d'identification proposée par Zechner et al. (PNAS, 2012), basée sur les moments des distributions de probabilités de la CME. Plus généralement, nous discutons également des spécificités de ces deux méthodes d'inférence et de leurs utilisations possibles en biologie.

Introduction

It is now well recognized that the functioning of biological systems at the molecular and cellular levels is noisy [START_REF] Raj | Nature, nurture, or chance: stochastic gene expression and its consequences[END_REF][START_REF] Snijder | Origins of regulated cell-to-cell variability[END_REF]. Within a genetically-identical population, cells behave in a heterogeneous and stochastic manner. The biological role of noise, whether beneficial or detrimental, is still a major open question in systems biology [START_REF] Balázsi | Cellular Decision Making and Biological Noise: From Microbes to Mammals[END_REF], therefore, models that are able to capture various aspects of cell population's heterogeneous behaviors would shed light on the matter. In short, one distinguishes two major origins of noise: intrinsic noise, coming from stochasticity of the observed process itself, and extrinsic noise, coming from differences between cells or cell's environment [START_REF] Raj | Nature, nurture, or chance: stochastic gene expression and its consequences[END_REF][START_REF] Snijder | Origins of regulated cell-to-cell variability[END_REF]. Arguably, the modeling framework of choice for intrinsic variability is the use of CME. Unfortunately, the identification of such models is quite difficult in practice [START_REF] Suter | Mammalian genes are transcribed with widely different bursting kinetics[END_REF], although approximations of the CME have been recently proposed [START_REF] Munsky | Listening to the noise: random fluctuations reveal gene network parameters[END_REF] and used to fit probability distributions predicted by the model to empirical distributions from cell populations. Alternatives for the identification of CME model parameters, using moment-closure methods [START_REF] João | Moment closure for biochemical networks[END_REF], have also proved to be biologically relevant and computationally much more tractable [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF]; however, in this framework, only some statistics of the population's distribution are used. These modeling approaches are, therefore, well adapted to flow cytometry data, which provides information on population distributions, but are unable to exploit the rich information on single-cell temporal evolution that time-lapse movie data provide. For this kind of data, we propose to use ME models. In this framework, a population of individuals is described by an ODE model supplemented by parameter distributions. Importantly, in contrast to CME and MB approaches, ME models naturally capture extrinsic variability.

As a statistical analysis framework, ME has been previously used in fields such as medicine, ecology, manufacturing, psychology and, especially, it has been for decades the modeling paradigm in the field of pharmacokinetics [START_REF] Friston | Mixed-effects and fMRI studies[END_REF][START_REF] Olofsen | Nonlinear Mixed-Effects Modeling: Individualization and Prediction[END_REF][START_REF] Lu | High-dimensional ODEs coupled with mixed-effects modeling techniques for dynamic gene regulatory network identification[END_REF][START_REF] José | Mixed-Effects Models in S and S-PLUS[END_REF][START_REF] Davidian | Nonlinear models for repeated measurement data[END_REF]. However, to the best of our knowledge, the identification of ME models of cellular processes based on single cell fluorescence microscopy data has not yet been studied.

In this paper, we compare ME models with CME models identified through an MB approach very similar to the one presented in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF], using fluorescence microscopy data that shows the response of yeast cells to repeated osmotic shocks in control experiments [START_REF] Uhlendorf | Long-term model predictive control of gene expression at the population and single-cell levels[END_REF]. Our results show that methods for identification of ME models based on real video-microscopy data are computationally tractable, and that in comparison to state-of-the-art MB identification methods perform in a comparable way. We therefore conclude that based on our results, one cannot unambiguously decide whether the major source of variability in the observed cellular process is intrinsic or extrinsic noise. Lastly, we note that the performance of ME inference methods can be further improved by taking cell lineage, cell physiology, or other covariate information into account.

The paper is structured as follows: In section 2 we review mixed-effect models, CME models, and moment-closure methods. The biological system under study and the corresponding ME and MB models used in the inference process are described in section 3. The quality of the identified models is analized in section 4. Finally, section 5 provides a critical comparison of ME and MB inference procedures. Through the paper we will use bold letters to denote random variables.

Model inference approaches

This section is a short review of the approaches that we will compare. In particular, we will detail the underlying assumptions, data required and output provided.

Mixed-Effects model inference

ME modeling is an approach to the analysis of population data where samples are clustered into subgroups of the population, called "individuals", in which the observations are mutually dependent (e.g. time-series of single cell's fluorescence) and the behavior of the total population is inferred from these individuals. The general form of an ME model is [START_REF] Davidian | Nonlinear models for repeated measurement data[END_REF][START_REF] José | Mixed-Effects Models in S and S-PLUS[END_REF][START_REF] Demidenko | Mixed Models: Theory and Applications[END_REF]:

β i = d(α i , β, b i ), i = 1, . . . , N (1)
y ij = f (Z ij , β i ) + e ij , j = 1, . . . , J (2)
Equation ( 2) represents the model of the ith individual, where y ij ∈ R denotes the jth measure of individual i, defined by f , a function that depends on a set of regressors Z ij ∈ R N , and a vector of individual-specific parameters, β i ∈ R p . Measurement noise is represented by the independent random variable e ij . Further considerations can be made at this level, for example, when dealing with time series of data. In this case, the vector of regressors can be expressed as Z ij =(t ij , u ij ), where t ij ∈ R N and u ij ∈ R N are the time and input of the ith cell at jth instant. The parameters β i are extracted from ( 1), the population model, where d is a function of α i , a vector of "covariates" (known individual factors), β ∈ R p , a vector of "fixed effects", and b i , a vector of "random effects", which is characterized by a covariance matrix C ∈ R p×p . The population function d in (1) and statistical assumptions on b i induce a probability distribution on the parameters. The classical methods to estimate d, as well as some statistics of b i , are based on likelihood maximization [START_REF] José | Mixed-Effects Models in S and S-PLUS[END_REF].

Moment-Based inference

A standard approach for a detailed description of (bio)chemical reaction kinetics relies on probabilistic modelling of the reactions among discrete pools of molecules. Consider a reaction network involving n species taking part in R reactions. Denote with x s (t), where s = 1, . . . , n, the number of molecules of the sth species at time t. For r = 1, . . . , R, let ν r,s ∈ Z be the change in the number of molecules of the sth species upon occurrence of the rth reaction (i.e., signed stoichiometric coefficients), and let a r (x) • δt, with x = (x 1 , . . . , x n ) ∈ N n and a r (x) ≥ 0, be the probability that the reaction occurs in the infinitesimal time δt given x molecules of the various species. Then x = (x 1 , . . . , x n ) can be seen as the state of a continuous-time, discrete-jump Markov process taking values in N n [START_REF] Samad | Stochastic modelling of gene regulatory networks[END_REF]. The probability p(x, t) that x is equal to x at time t evolves according to the so-called Chemical Master Equation (CME)

d dt p(x, t) = R r=1 a r (x -ν r )p(x -ν r , t) -a r (x)p(x, t), (3)
where ν r = (ν r,1 , . . . , ν r,n ). The reaction propensities a r (x) are typically fixed by the the laws of mass-action [START_REF] Wilkinson | Stochastic Modelling for Systems Biology[END_REF] up to a vector of kinetic parameters. As an alternative characterization of the probability law p(x, t), one may consider the collection of all moments µ ℓ (t) = E[x ℓ ], where ℓ = (ℓ 1 , . . . , ℓ n ) ∈ N n is a multi-index, x ℓ = x ℓ1 1 • . . . • x ℓn n , and |ℓ| = ℓ 1 + . . . + ℓ n is the order of µ ℓ . Excluding certain special cases, neither the solution of (3) nor the related moments of x can be computed in practice, therefore several approximation techniques have been developed; one of such techniques is Moment Closure (MC). For propensities a r (x) polynomial in x (as in massaction kinetics) and some L ∈ N >0 , let µ denote a vector containing all moments up to order Inria L. For matrices A and B fixed by (3) and a suitable so-called moment closure function φ(•), an approximation μ(t) of µ(t) can be obtained through the system [START_REF] João | Moment closure for biochemical networks[END_REF]:

d dt μ(t) = Aμ(t) + Bφ μ(t) . (4)
Models of this form have been used in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF][START_REF] Samad | Stochastic modelling of gene regulatory networks[END_REF] to describe the dynamics of regulatory networks within single cells of a cell population. Let us denote with θ the vector containing the kinetic parameters of the propensity functions and the statistics of the noise (see ( 6)). Note that θ (as well as the system statistics at an initial time) may vary across cells. Assume first that θ is unknown but fixed across the population. Moment-Based Inference (see e.g. [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF]) assumes that parameters θ are estimated by fitting the solutions of equations like (4) to the corresponding empirical moments of experimental data. In particular, let c T x, with c ∈ R n , be an observed scalar output of a network (e.g. the expression of one gene), and assume that, for every cell in a population of N cells, measurements y follow the model

y = c T x + e, (5)
e = e a + e b c T x η, (6)
with η ∼ N (0, 1). Mean m y and second-order uncentered moment M y of y relate to the moments m x ≃ µ 1 and M x ≃ µ 2 of x by way of the equations

m y = c T m x , (7)
M y = (1 + e 2 b )var(c T x)+(e a +e b c T m x ) 2 +(c T m x ) 2 (8)
At any given measurement time, empirical versions my and My of m y and M y can be determined from cell population histograms. Then, an estimate of the model parameters θ may be defined by the solution of the optimization problem

min θ j D my,j , My,j ||m θ y,j , M θ y,j , (9)
where a superscript symbol θ denotes dependence on the parameters, my,j , My,j , m y,j , M y,j denote the corresponding moments at time t j , and D denotes a suitable distance. In practice, m θ y and M θ y are computed based on the solutions of a moment-closure equation ( 4) with A, B (and possibly φ) depending on θ. Similar to what is done for reaction rate (ODE) models (see e.g. [START_REF] Hasenauer | Identification of models of heterogeneous cell populations from population snapshot data[END_REF], and Section 3.2.2), the case where θ takes different values depending on the subject (extrinsic noise) could be included in this framework by assigning priors to θ with hyperparameters common to the whole population [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF]. In this case, for model inference, the process statistics µ ℓ must also account for this additional source of variability, and the hyperparameters of the prior, rather than the individual random values of θ, have to be estimated [START_REF] Hasenauer | Identification of models of heterogeneous cell populations from population snapshot data[END_REF]. In [START_REF] Koeppl | Accounting for extrinsic variability in the estimation of stochastic rate constants[END_REF] and [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF], an alternative approach was proposed in particular for the case study of this paper, where certain extrinsic noise quantities where treated as state invariants of a CME-like model. This allowed to confine the uncertainty of these quantities into an appropriately chosen initial distribution, and to compute the evolution of the augmented system moments in a way at all similar to (4). 3 Case study: Gene expression in yeast cell populations

Description of the system

In the budding yeast S. cerevisiae, an increase of the environmental osmolarity activates the high osmolarity glycerol (HOG) signal transduction pathway, a stress response pathway that coordinates the adaptation response to an osmotic shock. Adaptation is achieved by increasing the cellular glycerol level via various mechanisms, one of which is the upregulation of genes involved in glycerol production [START_REF] Hohmann | Osmotic Stress Signaling and Osmoadaptation in Yeasts[END_REF]. Here, we used the promoter of an osmoreponsive gene (STL1) to drive the expression of a yECitrine fluorescent protein (see Figure 1A) to monitor the gene expression response of the cells to repeated osmotic stresses. Recently, some of the authors presented a feedback control platform, utilizing the HOG pathway to control gene expression [START_REF] Uhlendorf | Long-term model predictive control of gene expression at the population and single-cell levels[END_REF] and the data used here for model parametrization and evaluation stems from this work. Figure 1B illustrates the visual information gathered with this method.

Inria

Models under comparison

Underlying reaction network

The system is described by the reaction network of HOG1-induced gene expression [START_REF] Herskowitz | Life cycle of the budding yeast Saccharomyces cerevisiae[END_REF] as proposed in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF] where the (delayed) gene activation rate, u(tτ ), caused by an osmotic shock, represents the system's input, and the amount of fluorescent protein yECitrine is the output.

pSTL1 off c 1 u(t-τ ) GGGGGGGGGGGGGGGB F GGGGGGGGGGGGGGG c 2 pSTL1 on pSTL1 on +CR c 3 GGGGGGB F GGGGGG c 4 CR •pSTL1 on CR •pSTL1 on c 5 GGGGGGA CR •pSTL1 on +mRNA mRNA c 6 GGGGGGA mRNA+yECitrine yECitrine c 7 GGGGGGA ∅ mRNA c 8 GGGGGGA ∅ (10)
In [START_REF] Herskowitz | Life cycle of the budding yeast Saccharomyces cerevisiae[END_REF], pSTL1 on and pSTL1 off represent the active and inactive state of pSTL1 promoter (with rates of activation/deactivation c 1 and c 2 ). CR is the concentration of chromatin remodeling complex, which binds/unbinds to pSTL1 with rates c 3 and c 4 to produce the CR • pSTL1 on complex. The complex starts the production of mRNA at a rate c 5 , which in turn produces yECitrine, with a synthesis rate c 6 that depends on the number of ribosomes and a kinetic parameter. Finally, yECitrine and mRNA are degraded with rates c 7 and c 8 .

Model for ME inference

Next, we will characterize the functions f in (2) and d in (1). In our case, individuals correspond to single cells whose fluorescence is tracked over time. By the laws of mass-action, the (average) system can be described (as illustrated in [START_REF] De | Modeling and simulation of genetic regulatory systems: a literature review[END_REF]) by the set of reaction rate equations [START_REF] João | Moment closure for biochemical networks[END_REF]:

ẋ1 = c 2 x 2 -c 1 u(t -τ )x 1 ẋ2 = c 1 u(t -τ )x 1 -c 2 x 2 + c 4 x 4 -c 3 x 2 x 3 ẋ3 = c 4 x 4 -c 3 x 2 x 3 ẋ4 = c 3 x 2 x 3 -c 4 x 4 ẋ5 = c 5 x 4 -c 8 x 5 ẋ6 = c 6 x 5 -c 7 x 6 (11)
where x 1 , x 2 , x 3 , x 4 , x 5 , and x 6 represent, respectively, the proportion of the promoters that are in the off state (pSTL1 off ), in the on state (pSTL1 on ), or bound to chromatin remodeling factors (CR • pSTL1 on ), and the concentrations of chromatin remodeling factors (CR), mRNA and fluorescent proteins yECitrine. The parameters to be identified are: reaction rates c = {c 1 ,c 2 ,c 3 ,c 4 ,c 5 ,c 6 ,c 7 ,c 8 }, the initial concentration of CR (x 3(0) ) and the time delay τ>0. The procedure for calculating the input u is developed in Appendix A. For these ten parameters we want to estimate a mean value β and a covariance matrix C. In order to restrict β i to positive values, we set b i ∼N (0, C) and d(a i , β, b i ) = exp(β + b i ) (exponentiation here is applied componentwise). We won't consider α i because our data doesn't provide measures of other cell-specific features. Let us define the vector of parameters β i = (c 1,i , c 2,i , c 3,i , c 4,i , c 5,i , c 6,i , c 7,i , c 8,i , x 3(0),i , τ i ) and let x 6,i (t|β i ) denote the state of an individual following the dynamics in [START_REF] João | Moment closure for biochemical networks[END_REF] with parameters β i , and initial conditions x 1(0),i = 1, x 2(0),i = x 4(0),i = x 5(0),i = x 6(0),i = 0. We can equip the system (11) with the output equation

y i (t) = x 6,i (t|β i ) + e i (t) (12)
where e i (t) = (e a + e b x 6,i (t|β i ))η i (t).

Note that ( 12) and ( 13) are equivalent to ( 5) and ( 6). In particular, η i (t), i = 1, .., n are independent white gaussian noises with unit intensity, and e 2 a ,e 2 b , define the intensity of the additive and multiplicative parts of the measurement noise e i,j . Then, in [START_REF] Charvin | A microfluidic device for temporally controlled gene expression and long-term fluorescent imaging in unperturbed dividing yeast cells[END_REF]

we have f (Z ij , β i ) = x 6,i (t|β i ) and e ij = e i (t ij ). Regressors are Z ij = (t ij , u(t)| tij t=0
) and the full set of parameters to be inferred is θ = {β, C, e a , e b }. Model ( 13) is in agreement with the types of noise present in fluorescence microscopy [START_REF] Jezierska | Poisson-Gaussian noise parameter estimation in fluorescence microscopy imaging[END_REF][START_REF] Pawley | Sources of noise in three-dimensional microscopical data sets[END_REF]. Finally, to find the parameters in function (1) that maximize the marginal likelihood of the simulated distribution, we use the SAEM algorithm [START_REF] Delyon | Convergence of a Stochastic Approximation Version of the EM Algorithm[END_REF]. Starting values for β that provide good convergence properties are estimated as proposed in section 8.1 of [START_REF] José | Mixed-Effects Models in S and S-PLUS[END_REF], by calculating individual estimates first and then performing analysis of variability.

Model for MB inference

We chose the model proposed in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF] given the similarity of the systems and the promising results presented. We will use a zero-cumulant moment closure of the 35 equations reported in section S.4.1 of [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF], which describe the time evolution of the population statistics. In this system of equations, the first 7 state variables represent the first-order moment (the mean) of each chemical species (including ribosomes) and the remaining equations represent the secondorder uncentered moments (from which covariance between pairs of species can be inferred). We are interested in the first-and second-order moments of the fluorescent protein, which is denoted in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF] as the G species. The c k , k = 1, ..., 8 parameters appearing in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF] will have the same meaning as the c k parameters introduced in section 3.2.2, with the exception of parameter c 1 , which in our case acts as a scaling factor for the input u, and parameter c 6 , which for simplicity of notation we have lumped together with the concentration of ribosomes (α 1 2 in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF]), given that they are mutually unidentifiable. We have also assumed the same sources of extrinsic and intrinsic variability as in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF]. The parameters to identify are a set of fixed parameters β f = (c 1 , c 2 , c 3 , c 4 , c 5 , c 7 , c 8 , τ ) that are common to all cells, and statistics of a distribution on parameters β v = (x 3 (0), c 6 ). In particular, we will identify average values E[x 3 (0)], E[c 6 ] and the entries of the matrix C v = cov(x 3 (0), c 6 ). Extrinsic variability comes from the variable parameters and intrinsic variability results from the stochasticity of the CME. For the function D(•||•) in (9) we will use the Kullback-Leibler divergence between Gaussian distributions [START_REF] Kullback | On Information and Sufficiency[END_REF]. This is a simpler approximation than the one used in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF], but still provides a convenient (pseudo)metric for matching first-and second-order moments even if the distributions are not Gaussian. At every time instant j we will compute D my,j , My,j ||m θ y,j , M θ y,j = log( σ θ y,j σy,j ) + σ2 y,j + ( my,jm y,j ) 2 2σ 2 y,j

+ 1 2 (14)
my,j , m y,j represent the mean of the observed and simulated distributions respectively, and σy,j ,σ y,j the corresponding standard deviations. Note that these values include the contribution of the measurement noise, as specified in [START_REF] Daniel | Exact stochastic simulation of coupled chemical reactions[END_REF]. Starting values for β f and β v , are estimated in the same way as for ME inference.

Inria

Experimental data and data pre-processing

The data used consisted of a set of two experiments, in which cells had been controlled to follow a constant yECitrine concentration, set at 1500 fluorescence units. In each experiment, y ij corresponds to the value of fluorescence for yECitrine in cell i at time t j . The sampling time is 6 minutes and the analysis covers 10 hours of experiments. The empirical first and second moments of the data,

my,j = 1 N N i=1 y ij , My,j = 1 N N i=1 y 2 ij ,
correspond to the first-order and second-order uncentered moments of the yECitrine species in the MB model. Raw data was processed as follows: first, a manual review of the snapshots taken in the two experiments was carried out to spot possible errors in tracking. This is because increasingly growing population and artifacts in the image capturing process led to misidentification of some cells during the real-time automatic analysis. At each time instant, cells whose fluorescence level was considered as outlier where marked, and cells marked as outliers more than 30% of their lifetime were removed from the dataset. Finally, we set a time threshold based on the approximate lifespan of yeast cells [START_REF] Herskowitz | Life cycle of the budding yeast Saccharomyces cerevisiae[END_REF], and only cells that where alive and successfully tracked for more than 90 minutes were included in the identification dataset.In Fig. 2, the identification and validation sets are compared. It can be seen that the mean values are very close to each other, as they represent the controlled outputs. The dispersion of the populations (represented in Fig. 2 by +/-two standard deviations) differs significantly, due to the difference in the initial number of cells. Note also that the input patterns are equal during the first 3 hours until the cells reach the desired output, but afterwards they differ.

4 Inference results

Evaluation criteria

Time series of yECitrine concentrations for 10000 cells were computed using the ME and CME models. To discuss the results of the identification process, we assessed how well the identified models can reproduce identification and validation data. In the first case, we generated two datasets (ME1, MB1) using the input profile of the identification dataset (Figure 2, blue input).

In the second case, we generated datasets ME2 and MB2 using the red input profile in Figure 2.

Results are described in detail in sections 4.2 and 4.3.

As explained in section 3.2.2, in the ME case a cell is instantiated by extracting a random vector b i from the distribution N (0, C), computing β i = exp(β + b i ), and generating yEcitrine samples according to [START_REF] Hohmann | Osmotic Stress Signaling and Osmoadaptation in Yeasts[END_REF]. For the CME, yECitrine samples for a cell are obtained by extracting a random vector β v,i from a lognormal distribution whose statistics are inferred from β v , C v (see [START_REF] Thomopoulos | Some measures on the standard bivariate lognormal distribution[END_REF]), and then running Gillespie's algorithm [START_REF] Daniel | Exact stochastic simulation of coupled chemical reactions[END_REF] to sample from the probability p(x, t) of the CME, instantiated with vectors β v,i and β f . Finally, measurement noise is added according to [START_REF] Delyon | Convergence of a Stochastic Approximation Version of the EM Algorithm[END_REF]. Identified models are then scored according to two criteria: the first one is the ability of the system to follow the time evolution of the mean and the standard deviation of the population. This was quantified in terms of the normalized RMSE

NRMSE = 1 T T j=1 (λ j -λj ) 2 λmax -λmin (15)
where, for an experiment spanning T time samples, λj is the jth sample of the analyzed feature (i.e. mean or standard deviation of the cell population) calculated from the observations, λ j is the same quantity calculated from the simulated population, and λmax , λmin are the maximum and minimum in the set { λ} T j=1 . In the sequel, NRMSE M corresponds to the NRMSE of the mean trajectory and NRMSE S is the NRMSE of the standard deviation of trajectories. The second criterion used is the Kolmogorov-Smirnov Two-Sample Test (KS), described in [START_REF] Hollander | Nonparametric Statistical Methods[END_REF] and used to assess in a non-parametric way (i.e. without assumptions of the underlying continuous distribution) the similarity between two distributions. At each time instant, we compare yECitrine distribution over the simulated and real populations and a p-value (pK) is computed. The KS test is based on the null hypothesis that the samples are drawn from the same distribution and the null hypothesis is rejected with 95% confidence when pK<0.05, hence, a higher pK indicates higher similarity between the distributions. The percentage of time samples with pK>0.05 (i.e. the success rate), will be denoted with hK.

Performance on identification data

The time necessary to estimate ME and MB parameters, using an Intel(R) Core(TM) i7 CPU @2.67 GHz with 4 GB RAM, was in the order of 22 hours for ME (using nlmefitsa, from statistics toolbox [START_REF]Matlab statistics toolbox[END_REF]) and 1.5 hours (using fminsearch, from optimization toolbox [START_REF]Matlab optimization toolbox[END_REF]). In both cases only one processor was used. The huge difference in identification times is due to the fact that solving the optimization problem (9) requires to solve several times only the ODE system that characterizes the first and second moments of the population distribution; on the other hand, the minimization of the marginal likelihood in ME using the SAEM method, requires to simulate several times a whole population of cells using the model [START_REF] João | Moment closure for biochemical networks[END_REF]. Table 1

summarizes the results

Inria

obtained when evaluating the model against the identification dataset. The NRMSE M shows that MB inference has a slightly better performance following the mean. The standard deviation is also better followed by MB inference with a NRMSE S significantly lower value for MB than for ME. The same general conclusion can be drawn from the KS test: the average of the pK value at all time instants is higher for MB as well as the hK success rate. Inferred values are listed in Appendix sections B and C. 2 show that quantitative differences between the two methods are overall small, although indicators NRMSE M and NRMSE S show a better performance using MB. The KS-test results don't favor MB in the validation stage as they did on the identification stage. Overall, ME presents an increase in performance between the identification and the validation stages, while MB's performance decreases for most indicators. This could be a sign that MB overfits the system, losing generalization capabilities. This aspect should be analyzed in detail in a future study in order to find criteria to prevent overfitting when using this technique. 5 Discussion and concluding remarks

In our experiments, both MB and ME have led to satisfactory results despite the differences in the frameworks and the assumptions on extrinsic or intrinsic variability. Therefore, the choice of the modeling framework will depend on the particular problem addressed. Two factors that would favor the choice of ME would be the default ability to account for expression profiles in individual cells, and the possibility of using characteristics such as cell size, cell-cycle state and others, adding them as covariates (vector α i ), to better quantify the deterministic factors that contribute to cell-to-cell variability [START_REF] Snijder | Origins of regulated cell-to-cell variability[END_REF]. The inclusion of these features could be, in principle, developed in a CME-MB framework, but the large complexity of such models suggests to look for simpler modeling approaches.

Inria

One important aspect that only can be handled through single-cell data and therefore is much better suited for a ME approach, is the analysis of correlations in individual cell behaviors as a function of their parental relationships via cell lineage reconstruction.This can provide a better understanding of various cellular processes, such as aging [START_REF] Lindner | Asymmetric segregation of protein aggregates is associated with cellular aging and rejuvenation[END_REF] or cell cycle duration [START_REF] Charvin | A microfluidic device for temporally controlled gene expression and long-term fluorescent imaging in unperturbed dividing yeast cells[END_REF]. Because it allows to associate individual behaviors with individual parameter values, the ME framework seems ideally suited to model the effect of parameter correlations across many cell generations.

Appendix A Computation of the input function

To obtain a complete framework one has to relate the promoter activation rate, u(t), appearing in reaction 1 of Fig. 10, with the actuation signal (ie, the valve status). Three phenomena need to be taken into account. Firstly, there is a known lag between valve actuation and actual change of the cells environment in the imaging chamber. This effect has been characterized in [START_REF] Uhlendorf | Long-term model predictive control of gene expression at the population and single-cell levels[END_REF] and has been modeled here by the piecewise affine function represented in Fig. 4 by a green line. Secondly, an increase of the cells environment osmolarity leads to the activation of the HOG pathway and results in an increased Hog1 nuclear localization. Denoting h(t) cell chamber osmolarity and s(t) the nuclear Hog1 enrichment (red line in Fig. 4), the Hog1 activation dynamics can be described by [START_REF] Muzzey | A systems-level analysis of perfect adaptation in yeast osmoregulation[END_REF][START_REF] Uhlendorf | Towards real-time control of gene expression: controlling the hog signaling cascade[END_REF] ṡ = κh(t)γs,

where parameter values were estimated based on experimental data in [START_REF] Uhlendorf | Towards real-time control of gene expression: controlling the hog signaling cascade[END_REF]. Lastly, the promoter activation rate u(t) is assumed to follow a Hill-type kinetics as a function of the nuclear Hog1 enrichment s(t) [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF].

u(t) = (s(t) + s 0 ) nH K nH d + (s(t) + s 0 ) nH , (17)
in which the values of the parameters K d , n H , and s 0 have been estimated in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF]. One should note that here, like in [START_REF] Zechner | Moment-based inference predicts bimodality in transient gene expression[END_REF], we assume that signal transduction does not show significant stochastic fluctuations nor cell to cell variability. Indeed, all these steps are assumed to be purely deterministic. The effective activation rate of the gene (pSTL1 off → pSTL1 on ) (light-blue line in Fig. 4) is then assumed to be proportional to u with a delay τ . Here the delay τ was set to 10 minutes. However, note that in generality, its value is estimated with the rest of parameters in section 3.2).

B Inferred parameters for ME model

Figure 1 :

 1 Figure 1: Hyperosmotic gene expression in yeast. (A) Hyperosmotic stress triggers phosphorylation and nuclear import of the protein Hog1,which thereupon activates osmo-stress responsive genes. In addition Hog1 stimulates enzymes involved in the glycerol production pathway, while closure of the membrane glycerol transporter Fps1 prevents glycerol from leaking out. Increasing the intracellular glycerol concentration is the main adaptation mechanism to hyperosmotic stress. Adaptation is prevented by our experimental setup, thus Fps1 and GPD1 mechanisms (depicted in light gray) are not considered in our model (B). Information gathered by fluorescence microscopy. Cells are grown in a microfluidic device which can select between normal and high osmolarity media. A microscope takes fluorescent images of the cells, which are segmented and tracked in real-time.

Figure 2 :

 2 Figure 2: Experimental data. The solid blue line and shaded blue area denote the mean +/-two standard deviations of the samples in the identification set. The dashed red line and shaded red area denote the same quantities of the validation sets. The pulses in the bottom of the figure represent the position of the valve that regulates the osmotic shocks applied to the system.

Figure 3 :

 3 Figure 3: Validation Experiments. Simulations are compared to data from a validation set.Figure (A) and (B) show the time evolution of ME and MB's identified models compared to the time evolution of the observations. (C) shows the time evolution of the pK value for both models, indicating how close the simulated distributions are to the real observed distributions at every time instant. The black dashed line represents the 95% confidence interval, below which, there is significant statistical evidence that the distributions are not equal.

Figure 4 :

 4 Figure 4: Input function.The plot represents the temporal evolution of the position of the microfluidic valve (0: normal media, 1: hyperosmotic media; dark blue), the osmolarity of the cells environment (h(t); green), the nuclear Hog1 enrichment (s(t); red), and the delayed gene activation rate (u(tτ ); light blue) during an 8-minute osmotic shock. Here the delay τ was set to 10 minutes. However, note that in generality, its value is estimated with the rest of parameters in section 3.2).

) 4 . 2 4 . 9 •10 2 C

 42492 •10 2 cov(c 4 ,c 5 ) 1.3 •10 -3 cov(c 1 ,c 3 ) -7.3 •10 -7 cov(c 4 ,c 6 ) 4.1 cov(c 1 ,c 4 ) -3.7 •10 -2 cov(c 4 ,c 7 ) -2.5 •10 -5cov(c 1 ,c 5 ) -1.9 •10 -1 cov(c 4 ,c 8 ) 6.2 •10 -4 cov(c 1 ,c 6 ) -5.5 •10 2 cov(c 4 ,τ ) 4.2 •10 -3 cov(c 1 ,c 7 ) 3.0 •10 -4 cov(c 4 ,x 3(0) ) 2.7 •10 -1 cov(c 1 ,c 8 ) -5.4 •10 -2 var(c 5 ) 1.4 •10 -2 cov(c 1 ,τ ) -3.1 •10 1 cov(c 5 ,c 6 ) 1.3 •10 1 cov(c 1 ,x 3(0) ) -3.5 •10 1 cov(c 5 ,c 7 ) -2.4 •10 -5 var(c 2 ) 2.7 •10 4 cov(c 5 ,c 8 ) 1.6 •10 -3 cov(c 2 ,c 3 ) -4.6 •10 -5 cov(c 5 ,τ ) -3.4 •10 -1 cov(c 2 ,c 4 ) -2.2 cov(c 5 ,x 3(0) ) 1.3 cov(c 2 ,c 5 ) -1.5 •10 1 var(c 6 ) 6.2 •10 4 cov(c 2 ,c 6 ) -2.8 •10 4 cov(c 6 ,c 7 ) -4.9 •10 -1 cov(c 2 ,c 7 ) 7.7 •10 -2 cov(c 6 ,c 8 ) 1.1 •10 1 cov(c 2 ,c 8 ) -3.6 cov(c 6 ,τ ) 3.4 •10 2 cov(c 2 ,τ ) -1.9 •10 2 cov(c 6 ,x 3(0) ) 3.2 •10 3 cov(c 2 ,x 3(0) ) -2.5 •10 3 var(c 7 ) 2.2 •10 -5 var(c 3 ) 1.5 •10 -12 cov(c 7 ,c 8 ) -1.7 •10 -4 cov(c 3 ,c 4 ) 9.8 •10 -9 cov(c 7 ,τ ) -2.1 •10 -3 cov(c 3 ,c 5 ) 2.0 •10 -8 cov(c 7 ,x 3(0) ) 2.4 •10 -2 cov(c 3 ,c 6 )1.9 •10 -4 var(c 8 ) 3.6 •10 -3 cov(c 3 ,c 7 ) -4.4 •10 -9 cov(c 8 ,τ ) 5.9 •10 -2 cov(c 3 ,c 8 ) 6.7 •10 -8 cov(c 8 ,x 3(0) ) -4.9 •10 -2 cov(c 3 ,τ ) 1.1 •10 -6 var(τ ) 1.1 •10 2 cov(c 3 ,x 3(0) ) -3.8 •10 -6 cov(τ ,x 3(0) ) -2.5 var(x 3(0) ) Inferred parameters for CME(MB) model

Table 1

 1 Figure3Areveals that the tracking of the mean is very good in both cases, but MB is better than ME in following the standard deviation, as the values in Table2confirm. Results in Table

	. Performance indices on identification data
		ME1	MB1
	NRMSE M	0.06	0.04
	NRMSE S	0.25	0.11
	Avg. pK	0.25	0.49
	hK	79%	87%
	4.3 Model validation

Table 2

 2

	. Performance indices on validation data
		ME2	MB2
	NRMSE M	0.08	0.06
	NRMSE S	0.20	0.13
	Avg. pK	0.34	0.32
	hK	87%	74%

Table 3

 3

	. Mean Parameters
	Parameter	Value	Unit
	c 1 c 2	7.3 •10 1 2.5 •10 3	min -1 min -1
	c 3	1.9 •10 -5 min -1
	c 4	3.7 •10 -2 min -1
	c 5 c 6	1.3 8.6 •10 2	min -1 min -1
	c 7	4.8 •10 -3 min -1
	c 8 τ	1.0 •10 -1 min -1 1.4 •10 1 min
	x 3(0)	1.4 •10 2	a.U.
	a	5.3	a.U.
	b	7.3 •10 -2 a.U.

Inria

Table 4

 4

		. Parameter Variance-Covariance
	Parameter	Value	Parameter	Value
	var(c 1 )	2.5 •10 1	var(c 4 )	3.2 •10 -4
	cov(c 1 ,c 2

Table 5

 5

	. Parameters and Variances
	Parameter	Value	Units
	c 1	2.1 •10 1	min -1
	c 2	5.1 •10 -1 min -1
	c 3	2.3 •10 -2 min -1
	c 4	3.7 •10 -1 min -1
	c 5	9.7 •10 -1 min -1
	c 6	3.9 •10 -1 min -1
	c 7	7.4 •10 -3 min -1
	c 8 x 3(0)	1.8 •10 -2 min -1 1.2 •10 2 a.U.
	τ	9.5	min
	var(x 3(0) )	1.5	a.U.
	var(c 6 )	9.7 •10 -4 min -2
	cov(c 6 ,x 3(0) )	1.2 •10 -6 min -1
	a	1.0	a.U.

b 3.7 •10 -3 a.U.

RR n°8288

InriaRESEARCH CENTRE PARIS -ROCQUENCOURTDomaine de Voluceau, -Rocquencourt B.P. 105 -78153 Le Chesnay Cedex Publisher Inria Domaine de Voluceau -Rocquencourt BP 105 -78153 Le Chesnay Cedex inria.fr ISSN 0249-6399

The research leading to these results has received funding from the European Union Seventh Framework Programme [FP7/2007-2013] under grant agreement n°257462 HYCON2 Network of Excellence, the INRIA/INSERM project Colage, and the French National Research Agency research grant Iceberg ANR-IABI-3096


Samuel Huron
email: samuel.huron@inria.fr

Petra Isenberg
email: petra.isenberg@inria.fr

Jean Daniel Fekete
email: jean-daniel.fekete@inria.fr

PolemicTweet: Video Annotation and Analysis through Tagged Tweets

Keywords: Backchannel, Video annotation, Crowdsourcing, Video analysis, Live tagging

We present PolemicTweet a system with an encompassing, economic, and engaging approach to video tagging and analysis. Annotating and tagging videos manually is a boring and time-consuming process. Yet, in the last couple of years the audiences of events-such as academic conferences-have begun to produce unexploited metadata in the form of micropost activities. With PolemicTweet we explore the use of tagged microposts for both video annotation and browsing aid. PolemicTweet is a system 1) to crowd source conference video tagging with structured sentiment metadata, 2) to engage audiences in a tagging process, and 3) to visualize these annotations for browsing and analyzing a video. We describe the the system and its components as well as the results from a one-year live deployment in 27 different events.

Introduction

We present a web-based structured video tagging, browsing, and analysis system called PolemicTweet as well as the results from a long-term evaluation of the system under live deployment. The system is unique in that it supports a wide variety of related activities ranging from video tagging to analysis and provides its different components to a variety of benefits for different kinds of audiences. PolemicTweet (PT) was developed in response to the need for effortless but rich video tagging for event recordings and their subsequent browsing and analysis. A number of organizations in the world, including ours, regularly organize mid-sized conference-style events, with audiences of 25-300 attendees. Many of these events are recorded using videos for archival purposes or for sharing on the web. For example, a rapidly growing global audience regularly follows the TED-events series [START_REF]TED: Ideas worth spreading[END_REF]. However, with current online video players it is difficult to get additional information on a recorded video. Video meta-data does not generally give good information scent about the events unfolding during the course of a video, such as the topic(s) covered, controversies that may have arisen, or reactions of the audience to a speaker shown in the video. This is because videos are complex to summarize, analyze, search and browse. Yet, we argue that providing additional information scent is highly valuable to external viewers. We thus, designed PT to capture structured annotations in a costeffective manner and provide this information for perusal. The captured meta-data is meant to help both casual browsing and analysis of video content. PT also provides visual summarization of annotations captured during live events and includes additional features for search and browsing.

We specifically contribute a web-based solution to video annotation and analysis that is easy to deploy and use. It is engaging in that the annotation phase offers benefits to participants beyond their involvement in a tagging activity. PT is a wellrounded system of integrated components and has been evaluated in a long-term study involving the capture and annotation of video from 27 different events over one year.

Motivation

Our design of PT was motivated by the proliferation of microblogging activities at live events associated with our organization: the Institut de Recherche et d'Innovation (IRI) du Centre Pompidou. IRI is a research association focusing on the study of cultural practices enabled by digital technologies. A part of IRI's activities is to organize seminars on subjects such as philosophy, design, and digital cultures. A typical seminar is video recorded. A formal procedure of human annotation and tagging on the video footage was used to follow the event in order to make the videos more searchable and comprehensible. This video annotation task was generally assigned to interns (often archivist students) using a custom software called "Ligne de temps". This process had two major disadvantages: without having attended the event the job was tedious and time-consuming for the student and costly for the organization.

After this internal annotation process, the video recordings were published online on a web platform displaying the same annotation interface as "Ligne de temps". In addition to the first set of annotations, the attendees of the event were allowed to add further annotations. However, the participation remained low: the interface for annotation was complex and did not entice the public participation we had wished for. From these past experiences, we wanted to redesign this process with new tools that took advantage of the micropost data produced by the audiences of our events. The resulting system, PolemicTweet, was, thus, motivated by three goals:

1. Engage the audience to tweet for crowdsourced video annotation and tagging, 2. Provide a visual backchannel to incite people to tag their tweets with predefined tags and thus provide structured annotations, 3. Provide easily accessible tools in an encompassing system that span the whole process of annotation to video analysis.

In order to reduce the complexity of the annotation process we decided to make it more engaging and to open the process to a wider audience. As indicated in Goal 1 above, we opted for a crowdsourced solution. Crowdsourcing was a promising solution since the behavior of participants in live events has evolved due to two digital enablers: microblogging and portable devices. The speaker nowadays is no longer the only information channel during an event, a second information channel (backchannel) is now regularly used: the audience's tweets. This information channel is the data source we wanted to leverage for video annotation and analysis in our tool PT.

Related Work

To remember, analyze or study an event, video recordings and/or audio tapes are frequently used [START_REF] Allen | The use of video in organizational studies[END_REF]. For this reason the study and design of video analysis software is a popular topic in information retrieval [START_REF] Hauptmann | Lessons for the future from a decade of informedia video analysis research[END_REF], computer human interaction [START_REF] Roschelle | VideoNoter: A productivity tool for video data analysis[END_REF][START_REF] Mackay | EVA: An experimental video annotator for symbolic analysis of video data[END_REF][START_REF] Abowd | The Family Video Archive: An annotation and browsing environment for home movies[END_REF][START_REF] Diakopoulos | Videolyzer: Quality analysis of online informational video for bloggers and journalists[END_REF], computer supported collaborative work [START_REF] Cockburn | CEVA: A tool for collaborative video analysis[END_REF][START_REF] Schroeter | Vannotea: A collaborative video indexing, annotation and discussion system for broadband networks[END_REF][START_REF] Engström | Amateur vision and recreational orientation: Creating live video together[END_REF], and visual analytics [START_REF] Daniel | Video visualization[END_REF][START_REF] Ruppert | A radial visualization tool for depicting hierarchically structured video content[END_REF][START_REF] Tonomura | VideoMAP and VideoSpaceIcon: tools for anatomizing video content[END_REF][START_REF] Parry | Hierarchical event selection for video storyboards with a case study on snooker video visualization[END_REF]. In this section, we discuss efforts most closely related to our goals and solutions in regards to our video analysis interface, backchannels for live-events, crowdsourced media annotation and tweet sentiment analysis.

3.1

Crowdsourcing media annotation and motivation

Social network activities such as tweeting can be used for crowdsourcing activities. Shamma and Diakopoulos [START_REF] Diakopoulos | Characterizing debate performance via aggregated twitter sentiment[END_REF] showed that the social structure and the conversational content of tweets can provide insight into a media event's structure and semantic content: quantitatively through activity peaks and qualitatively through keyword mining. Diakoloulos et al. [START_REF] Diakopoulos | Diamonds in the rough: Social media visual analytics for journalistic inquiry[END_REF], after having collected tweets during the U.S. State of the Union presidential address in 2010, used them to annotate a video of the event. Based on this assessment and our own experience observing twitter use during conferences, we decided for PT to use Twitter for crowdsourcing video tagging.

Yet, despite the benefits of crowdsourcing for annotation, there are disadvantages to consider. Bernstein et al. [START_REF] Bernstein | Crowds in two seconds: Enabling realtime crowd-powered interfaces[END_REF] named the time-consuming nature of crowdsourcing (due to the recruitment of the crowd and the time to achieve a task). To avoid this problem, the authors designed a recruitment strategy and a retainer model (paying workers to wait and respond quickly when asked). Another identified problem is the motivation of workers. Wu et al. [START_REF] Wu | Video summarization via crowdsourcing[END_REF] encountered this problem in their video summarization technique via paid crowd workers and expressed the need to further investigate incentive mechanisms. This is one of the issues we addressed in PT.

A large number of pitfalls have also been identified on crowdsourcing task quality [START_REF] Snow | Cheap and fast-but is it good? Evaluating non-expert annotations for natural language tasks[END_REF][START_REF] Ipeirotis | Quality management on Amazon Mechanical Turk[END_REF]. Workers are often untrained and lack task context, particularly in specific domains such as science or the humanities. In PT, however, we do not use a crowdsourcing platform but a casual social network (Twitter) to achieve the task through a community of interest. This way we can expect annotators who are engaged and aware of the context and do not require specific training.

Backchannels for live-events

A digital backchannel can be understood as a thread of information that accompanies a live event. For more than ten years now, digital backchannels, have become a growing research area, in human computer interaction [START_REF] Mccarthy | Digital backchannels in shared physical spaces: attention, intention and contention[END_REF][START_REF] Rekimoto | Adding another communication channel to reality: An experience with a chat-augmented conference[END_REF][START_REF] Harry | Backchan.nl: integrating backchannels in physical space[END_REF], computer supported collaborative work [START_REF] Mccarthy | Digital backchannels in shared physical spaces: attention, intention and contention[END_REF][START_REF] Mcnely | Backchannel persistence and collaborative meaning-making[END_REF][START_REF] Cogdill | Backchannel: Whispering in digital conversation[END_REF] and visual analytics [START_REF] Dörk | A Visual Backchannel for Large-Scale Events[END_REF][START_REF] Marcus | Twitinfo: Aggregating and visualizing microblogs for event exploration[END_REF]. Backchannels have been studied and used in several contexts like conferences [START_REF] Mccarthy | Digital backchannels in shared physical spaces: attention, intention and contention[END_REF][START_REF] Harry | Backchan.nl: integrating backchannels in physical space[END_REF], classrooms [START_REF] Bry | Promoting awareness and participation in large class lectures: The digital backchannel backstage[END_REF], and meetings [START_REF] Rekimoto | Adding another communication channel to reality: An experience with a chat-augmented conference[END_REF]. A taxonomy of different backchannels was presented by Cogdill et al. [START_REF] Cogdill | Backchannel: Whispering in digital conversation[END_REF]. The researchers present five types of backchannels: process-oriented, contentoriented, participation-enabling, tangential and independent backchannels. In PT we use a process-oriented public backchannel, to achieve a live tagging task. A digital backchannel such as ours is useful to provide awareness [START_REF] Bry | Promoting awareness and participation in large class lectures: The digital backchannel backstage[END_REF] to local and remote participants [START_REF] Rekimoto | Adding another communication channel to reality: An experience with a chat-augmented conference[END_REF] of questions, comments [START_REF] Harry | Backchan.nl: integrating backchannels in physical space[END_REF], shared work, and references [START_REF] Mccarthy | Digital backchannels in shared physical spaces: experiences at an academic conference[END_REF], and can encourage real-world discussions [START_REF] Rekimoto | Adding another communication channel to reality: An experience with a chat-augmented conference[END_REF]. McCarthy et al. [START_REF] Mccarthy | Digital backchannels in shared physical spaces: attention, intention and contention[END_REF] studied backchannels in a use case scenario similar to ours. They studied digital backchannels in academic conferences using IRC and relate different types of use and problems, like the cost to spread the information for a backchannel connection. McNely [START_REF] Mcnely | Backchannel persistence and collaborative meaning-making[END_REF] suggests that the problem has been partially solved by the increased availability of micropost web services like Twitter, their simplicity of use, and the large adoption rate. Sopan et al. [START_REF] Sopan | Monitoring Academic Conferences: Real-time Visualization and Retrospective Analysis of Backchannel Conversations[END_REF] showed that a micropost backchannel during a conference, like in our case, permits to connect local and distant participants. In another academic context, Harry et al. [START_REF] Harry | Backchan.nl: integrating backchannels in physical space[END_REF] implemented and described the use of a backchannel to leverage participation in an auditorium to allow for the audience to vote on questions and give feedback. In PT we also use an explicit backchannel to crowdsource the process of a real-time distributed user task but-in contrast to the previous work-we use it to produce a document that summarizes this activity as an annotated videotape.

With the prevalence of social networks, portable devices, and wireless network connections, backchannels are no more an emerging social phenomenon but a real trend. When this increased use of social network and portable device meets largescale events like the Olympics, the Arab spring, or national elections, visual analytic techniques can help to summarize and understand what has happened during the event. Dörk et al. [START_REF] Dörk | A Visual Backchannel for Large-Scale Events[END_REF], for example, introduced the visual backchannel, a timelinebased visualization that presents an overview of an event through social stream aggregation. Marcus et al. [START_REF] Marcus | Twitinfo: Aggregating and visualizing microblogs for event exploration[END_REF] presented an algorithm for event detection in a streams to produce annotation on a timeline-based visualization of the social activity surrounding an event. Diakopoulos et al. [START_REF] Diakopoulos | Diamonds in the rough: Social media visual analytics for journalistic inquiry[END_REF], synchronized a recorded TV show with related tweets and provided two timelines, one for sentiment analysis over time, the other for volume of tweets.

Similar to these three previous systems ( [START_REF] Diakopoulos | Videolyzer: Quality analysis of online informational video for bloggers and journalists[END_REF][START_REF] Dörk | A Visual Backchannel for Large-Scale Events[END_REF][START_REF] Marcus | Twitinfo: Aggregating and visualizing microblogs for event exploration[END_REF]) PT provides a summary visualization for video-recorded events but builds on this idea by using its backchannel as a specific crowdsourcing tool for tagged annotation.

Tweet content analysis

Transcripts, tweets, and other temporal data streams are often used to annotate video.

In the case of temporal data composed of text, it is common to use named-entity recognition, sentiment analysis, and natural language processing to recognize and tag events of interest. When we started developing PT, sentiment classification methods on tweets were not as effective as on classical natural language corpora due to the limited length of tweets and the common use of shortened and non-standard English words [START_REF] Pang | Opinion mining and sentiment analysis[END_REF]. Now, the combination of different techniques from machine learning [START_REF] Mccarthy | Digital backchannels in shared physical spaces: experiences at an academic conference[END_REF], semantic rule based approaches [START_REF] Jiang | Target-dependent twitter sentiment classification[END_REF], and graph-based optimization [START_REF] Speriosu | Twitter polarity classification with label propagation over lexical links and the follower graph[END_REF], have improved sentiment classification of tweets significantly, and can achieve 85.6% accuracy [START_REF] Jiang | Target-dependent twitter sentiment classification[END_REF]. In this last work, the authors considered three types of sentiments: negative, positive, and neutral. The reported accuracy rate was achieved on an English tweet corpus of the most popular queries on Twitter (Obama, Google, iPad, Lakers, Lady Gaga), and on a clearly targeted subject. We did not take advantage of this approach as it only works for English tweets, it is complex to set up, and has only been tested on a topic-limited tweet corpus. Instead, for PT [START_REF] Huron | User-Supplied Sentiments in Tweets[END_REF], we needed to find a solution that would work in multiple and mixed languages, was simple to deploy, and would work on a variety of specialized corpora like philosophy, aesthetics, or design. Diakopoulos and Shamma [START_REF] Diakopoulos | Characterizing debate performance via aggregated twitter sentiment[END_REF] used another approach for sentiment classification: they used Amazon Mechanical Turk to perform hand-annotated sentiment classification on tweets. Turkers were compensated $0.05 per ten tweets analyzed. Turkers were asked to tag four types of sentiments: negative, positive, mixed, and "other."

The corpus of tweets was in English and about politics. Others [START_REF] Diakopoulos | Diamonds in the rough: Social media visual analytics for journalistic inquiry[END_REF] described the use of machine-learning algorithms to perform the same analysis with lower accuracy and cost but higher speed. Crowdsourcing sentiment classification on tweets with its higher success rate and relatively low overall cost is more and more common. For instance, it is one of the products of CrowdFlower [35] a well-known crowdsourcing platform. However, this approach cannot be used to tag tweets in real-time.

Whether or not to use crowdsourcing as a tool depends on one's annotation goal [START_REF]Médiamétrie-eStat: La fréquentation des sites Internet français Février[END_REF] since there is a speed vs. quality tradeoff to consider. For PT, we wanted to have the best of both worlds: classify tweets with a low cost (both in computer processing and in money) and a high precision rate, even if the recall rate depended on the adoption of our tool. We also wanted to have this classification in real time.

Video Analysis Support

Mackay's EVA [START_REF] Mackay | EVA: An experimental video annotator for symbolic analysis of video data[END_REF], was one of the earliest systems on video annotation, tagging, and analysis. Mackay synchronized records of metadata with her video records, such as the movements of a mouse on a screen. Nowadays recorded metadata is often taken from real-time web social services like chat or microblogging. PT relates to this trend but asks Twitter users to tag their own tweets for structured annotation.

Another solution to aid in the process of video analysis has been to support multiple analysts in parallel. Cockburn and Dale [START_REF] Cockburn | CEVA: A tool for collaborative video analysis[END_REF], for example, designed and developed CEVA a synchronous collaborative video annotation system that focuses on supporting parallel analysis. Parallel analysis is based on the idea that multiple video analysts (five in their prototype) share their analyses in real-time for distributing the workload. The authors argue that this synchronicity property offers two potential benefits: 1) synergy of group participation and 2) distributing the analysis workload. We were particularly interested in supporting the synchronicity property for the PT tagging phases to similarly take advantage of the synergy of the audience during event recording. The Videolyzer tool [START_REF] Diakopoulos | Videolyzer: Quality analysis of online informational video for bloggers and journalists[END_REF] also includes an asynchronous formal semantic tagging tool to increase information quality by allowing users to collaboratively organize their comments on the video. Their tool differs from ours in that we did not want to rely on a complex tagging interface to more easily spur user tagging in real-time during the event. Nevertheless, Videolyzer inspired the social use case scenario of PT.

System Design

PT is made of four interconnected components: 1) the definition of four tags to annotate tweets and the backchannel interface to read and write tagged tweets during the conference event, 2) a social protocol to set up and run an event live, and 3) a web video player synchronized to a tweet visualization to navigate and replay conference video recordings, 4) a website to provide a fluid browsing between all PT components. We describe the components in that order.

4.1

Backchannel to crowd source tag annotation

To provide an enriched visualization of video annotations we wanted to collect tweets structured by specific annotation tags. We chose to provide a limited number of tags with a simple syntax and a specific color code (in brackets): agreement with the speaker (++,), disagreement with the speaker (--,), questions raised by or proposed to the speaker (??,), and references (==,), e.g. quotes or URLs to related content. Our choice of tags was informed by observation of the types of tweets typically used in the conference-style events that we target with PT. For choosing our set of tags, we had to balance expressive power and simplicity. To provide a memorable set of tags, we decided to use the simplest possible tagging format. At the same time, our interface needed to provide enough incentive for using the tags. The design rationales for choosing the tag syntax were:

• Simplicity of use and memorability: to facilitate its use and adoption;

• Brevity: to cope with the 140 character limitation of Twitter and for fast typing;

• Client and language neutrality: to be usable from any twitter client with any kind of text input method, usable in any language; • Ease of parsing: for automatic processing tools;

• Expressiveness and univocality: to allow clear statement of intent and for machines to interpret it unambiguously. The four tags were introduced to the audience before the start of an event. Fig. 3 shows the real-time backchannel interface we designed to allow an audience to monitor the tagged tweet activity in real time. It consists of six different components (Fig. 3): A) an enhanced Twitter client with a text field, four buttons to easily add tags to tweets, and a send button, B) a list of tweets relative to the scrollbar's focus, B') a vertical bar chart showing tweet activity over time, colored blocks represent tweets containing a tag using the tag's representative color, C) a video player showing the live video stream, D) abstract and program of the event, E) dynamic tag-cloud extract from the tweets in focus, color-coded according to the polemic tag colors. For organizers, setting-up this interface is easily done in two steps on the PT web server: 1) cloning the folder containing the default interface, and 2) changing the settings file to specify event title, hash tag, timestamp of the beginning, expected duration, and abstract.

4.2

Engaging the crowd: social protocol for synchronous annotation

One very unique feature of PT is a social protocol we developed in order to help event organizers make best use of PT. The main goal of our protocol is to inform the audience about the capture and future use of their tweets and to introduce the four specific tags. The protocol suggested event organizers to follow three phases: Before the talk: Instructing the crowd. We designed a "connection package" to inform the audience. It consists of a flyer (Fig. 1) given to attendees during the registration process, containing information about the network access policy, the PT tags, the website URL, and the Twitter hashtag for the event. Additionally, we asked organizers to make an announcement prior to the conference.

During the talk: Crowd source sentiment analysis and video annotation. Attendees send tweets with the PT tags, a program records all the tweets relative to the twitter event's hashtag. The visual backchannel website is set up to provide real-time visual feedback of tweets and to give an incentive for participation and easy access to people who do not use Twitter (Fig. 5). Organizers can also make use of this interface to get informed about questions to ask at the end of a talk.

After the talk: Publishing. Organizers synchronize the recorded data with the timestamp of the video streaming server and the tweets' timestamps. Then they publish the video on the web (copying and modifying a configuration file) with the MetadataPlayer, that we discuss in the next section.

Synchronization of the video and tweets could be complex due to the lack of reliable video timestamp. In the case of Diakopoulos and Shamma [START_REF] Diakopoulos | Characterizing debate performance via aggregated twitter sentiment[END_REF], synchronization was simple because the video was provided by an official TV channel and had a wellknown timestamp. In the academic context, an event can be recorded by different means and not be broadcasted live. Most of the time-if a video is not streamed-the video timestamp depends on settings on the recording hardware and is, thus, not reliable. We, thus, deploy several strategies for synchronizing video recording and tweet activity: we video-record a clock before the event, have a special tweet at the beginning of the event, and re-synchronizes it with a custom script. Of course for resynchronization of tweets to a videotape it is important to use the originally captured video footage and not an already cut and edited version.

4.3

Video analysis web video player

The PT video player (Fig. 4) is designed to play a conference video while showing the tagged and untagged tweets sent with the conference hashtag during the time of video recording. Our design goal was: 1) to provide a visualization that gives an overview of the tags and activity spikes, 2) to design a compact player that can be easily embedded, just like the YouTube player, and 3) to augment the player's video navigation techniques to get more benefit from the annotations.

We designed all the time-related components as graphical horizontal projections on the time axis. All components have the same width and horizontal scale to allow for a vertical alignment between them. Navigation on the video is performed using the standard video player components and controls: a controller line with the buttons to play, pause, change the audio level, and a seek bar. Two novel components show context and allow for contextualized interactions. The segment line allows quick access to chapters of the video-such as topics of a presentation or name of presenters for a panel. The data for this line can be manually or automatically extracted from the video. The polemic line visualizes the tweet activity during the event, positioned on the time axis; it provides contextual feedback on the tweet activity, a direct access to video segments highlighted by tweets, and direct access to the tweet contents through hovering. The polemic line has the following characteristics:

• Polemic line visualization: The visualization is a bar chart composed of colored tweet squares. Each square represents a tweet colored according to the tweet's tag or gray if the tweet does not contain any tag. The square has a fixed size of 5 pixels to facilitate its selection without taking too much screen real-estate. Depending on the length of the video, each 5px slot represents a certain time range. At each time-slot the recorded tweet squares are sorted top-to-bottom by type (green, red, blue, yellow, and gray) so that the most expressive tweets are at the top of the bars. • Interaction: Moving the mouse over a square shows a tooltip with the tweet's text, author name, and tag color. Clicking on it seeks the video to the time of the tweet, and makes meta-information appear in the details information component (Fig. 5).

The position of each square provides information about when the tweet was emitted but also about the contextual activity level at the moment. • Search: A search text-field (Fig. 5.) in the controller bar allows finding text in the tweets. When a search string is entered, the visualization is updated to highlight each represented tweet containing the string. As shown in Fig. 5 this feedback shows the distribution of tweets containing the specified string (in purple).

Website

To provide a fluid browsing experience between all the PT components we embed them inside a public website composed of two main components: a home page presenting the past, present and future events and an archive page to find older events. On the pages containing past events, the MetadataPlayer shows the annotated videos of these events. The backchannel interface is only available during a live event on the "present events" part of the page.

Assessing the Impact of Polemic Tweet

In order to better understand how PT supports our goals of cheap, useful, and engaging video annotation, we studied the tool using mixed-methods -both qualitative and quantitative approaches under long-term deployment. We successfully deployed PT in 27 real conferences, seminars, and events of different locations, topics, and sizes of audience over the last year. For these 27 events we recorded 46 tracks (video and tweets), each track being one speaker or one session. 20 events took place at the Pompidou Center in our conference room, 7 events took place at various places in the city of Paris. Nine events had between 25-50 attendees, 13 events 51-100 and five events 101-290. The topics of the events varied greatly, including academic topics such as science, technology, philosophy, and design. The duration of 24 events was limited to one day while three events took place over several days. The recorded tracks from each event ranged from 30 to 395 minutes. Overall, we received and collected 9,088 tweets from 1,012 unique accounts. Over the last year, the web site (player, backchannel, but also other additional pages like home and archives) received about 157,000 page views from over 15,000 unique visitors with an average visit duration of 3 minutes which is comparable to National video websites with substantially more content and public cover like Francetvod.fr [37] with a 3.2 minute average visit duration during February 2012.

Except for the video recording, sound capture, and the communication support for the tags (flyers and poster boards), the components of PT were deployed on the web, which greatly simplified the process for the organizers. The recorded videos are all published on the open site http://www.polemictweet.com.

Data Collection

To study the use of PT we gathered three types of data: all tweets using the various conference's hashtags, logs from the usage of the PT MetadaPlayer and the backchannel interface, and two questionnaires sent to various users of PT, either as designers, organizers, speakers, or audience participants. We used Google Analytics to log the website pages and a custom logging system to collect low-level traces of user interactions on the video player that Google Analytics could not provide. On the backchannel interface, we recorded usage logs only via Google Analytics to give us information about where users came from (direct access, twitter, devices, others) and where the users were physically located (inside or outside the conference room). This last information does not take into account attendees not using the provided free Wifi connection (e.g. using their mobile's own 3G network) and possible errors of domain detection. Despite a few tablet users most connected with personal computers. We also recorded the web client's signature for each tweet to understand where tweets were sent from.

We sent two web surveys, the first consisted mainly of close ended questions sent by tweet to 140 randomly chosen attendees of one of the recorded events; the second questionnaire was sent to understand how PT faired in the organizer's point of view. This questionnaire consisted of open-ended question sent by mail to five members of the event organization team in our institute.

For the first questionnaire we received 47 responses, 27 of which completed all parts of the survey. 70 % (19) of the respondents attended at least one of the conferences. 96 % (26) had already used twitter at conferences, 3 % just for reading. This shows a clear positive feedback towards the goals of PT. For the second questionnaire we received four completed responses.

Analysis

In this section we report on the analysis of our data according to three main questions: 1) Is the system sufficiently engaging to collect data for crowd sourced video tagging? 2) Does the visual backchannel incite people to tag their tweets and thus provide annotation structure? 3) Is the video web player with annotations useful?

Is the system sufficiently engaging? Through our live deployment of PT, we found that engagement in the crowdsourced video tagging was highly diverse. We recorded from 0.20 to 7.94 tweets per minute (mean at 1.47, median at 0.89), with 0.12 to 2.13 tags per minute (mean 0.47, median 0.32). Averaged across all events, tweets were tagged using our syntax 40% of the time. Considering tagging a tweet as an additional cost, we conclude that the system provided enough tagging incentive on average.

Otherwise, we found that engagement in the annotation process was highly varied and depended on several factors including the type of audience, the content of the event, the number of twitter users, and whether the PT protocol had been respected or not. We discuss these in more detail now: Type of Audience. Our audiences were varied in terms of size, culture, equipment, incentive to achieve a task, and distribution of local and distant attendees. All of these factors played a role in the participation and engagement of attendees in the tweeting and tagging task. In our collected data, we had a group of events in which the number of recorded twitter accounts in comparison to the number of attendees was very high (64%-187% on 7 events with 825 tweets on average, (e.g. Fig. 6. A). Groups with values higher than 100% represent those in which more twitter accounts were recorded than participants in the events. Here remote attendees also used the conference hashtag to participate in the backchannel. The recorded events can be categorized according to two participation types: medium participation (20%-51% on twelve events with 209 tweets on average) and low participation (1%-18%, on seven events with 61 tweets on average, Fig. 6. C). Type of Speaker and Twitter Reaction. Some attendees reported through our informal interview that writing a tweet and tagging it took more attention in-situ than when following the event online. We had originally hypothesized that a captivating speaker would probably lead to less Twitter activity, but from our observations organizing and attending our events we could not confirm this hypothesis. On some events we observed a high level of tweet activity during a captivating talk and others with a capti-vating speaker had hardly any tweet activity. On the other hand we also noticed that in cases of less captivating talks, a peak of tweet activity arose because the audience was no longer focused on the speaker. Overall, regardless of the quality of the speaker the audiences of our events were polite and only used negative tags 14% of the time. Type of Content. We found that the type of content of an event and its audience could be highly related factors when it comes to annotation activity. For Instance, one of the events' topic was the websites of the Pompidou Center and the Tate Gallery. In accordance to the topic, the audience consisted of people active on the web and social networks. During this event we saw a very high participation (Fig. 6. A) in terms of tweets and tags: 6.06 tweets per minute and 2.13 tags per minute. On the other hand, we were also surprised to see low participation for certain events where the content would have suggested high Twitter usage. For example the "Muséologie 2.0" had the topic: museum digitization, and preservation policies. Here, the level of participation was extremely low: 0.20 tweets per minute and 0.12 tags per minute. Observance of the PT Protocol. We found that the protocol was generally well applied by the event organizers. Yet, in some cases we observed that deviance from the protocol could impact user activity. In particular we found problems related to hashtag selection, instruction diffusion, and connectivity. For the event "L'Open Data, et nous, et nous, et nous ?", organizers chose to use an existing hashtag to boost interaction between audience and remote Twitter users. They used #opendata, a common hashtag, which had been in use daily before the event by this community and others. The effect was that the tweet number per minute exploded to 7.94 but the number of tags per minute stayed really low at 0.88 as the tags were not known outside of the event. The backchannel was crowded by other tweets relative to the subject but not to the conference. This was a source of frustration for some attendees and made our system less useful in real-time and as a video annotation tool. Choosing a specific hashthag is important if video annotation is a goal, even if the event should stay connected to existing Twitter communities. Another issue regarding the observance of the PT protocol was that the instruction distribution to the audience was always different. Sometimes flyers were put on a seat, sometimes handed out with oral instructions, and sometimes placed somewhere on a table at the entrance. Despite these differences, most of the time attendees found the information and tweeted. Yet on some events we observed that flyers were not provided and just an announcement of instructions was made. This dramatically impacted the audience participation. For example this resulted in only 0.27 tweets per minute for the event "Du vinyl au numérique" (Fig. 6. C). The last and most obvious factor that highly impacted the audience participation was network connectivity. For instance, in the event "Mashup and Remix" (Fig. 6. B) mobile networks (3G and others) were hidden because the conference room was usually a cinema. A wifi network was provided but connectivity information was just given orally and not on the flyer. Thus, despite an audience of 70 people, only five sent any tweets (0.85 per minute). Of the 27 responses to our first questionnaire, 89 % (24) reported to remember PT tags from attending the event. While our general goal in making the tags known was reached, we were interested to learn more about people's tagging practices and in particular in regards to our backchannel interface.

Does the visual backchannel incite people to add tags?

We observed that tweets sent by the PT backchannel interface had a higher rate of tag adoption (55% of tweets were tagged) than for the top ten other clients used (20% of tweet were tagged) (Fig. 7.). A Welch's t-test was conducted to compare percentage of tagged tweets emitted through the backchannel and by all others clients. There was a significant difference in the score for the backchannel (M=70%, SD=26%) compared to all other clients (M=24%, SD=11): t(23)=6.74, p<.001. Attendees sent significantly more tagged tweets through the backchannel than through all other clients, suggesting that the backchannel interface did engage people to participate in tagged tweeting. Remote attendees spent more time on the backchannel interface than local attendees. Of those attendees who spent more than 5 minutes on the interface, 62% were remotely connected through outside the conference wifi. From informal feedback from attendees we hypothesize that local attendees did not need some backchannel interface features like video streaming and preferred their own Twitter client but that these features were useful for remote attendees.

We observed that, on average, attendees who used our tags on more than one tweet per event tweeted significantly more than those who did not tweet using our tags (average of 5.3 tweets per person without tags, 15.5 with tags, Welch's t-test, p < .01). This shows a strong correlation between tweeting activity and their use of the tags. Although we cannot conclude on causality at this point, the two alternative explanations are either that 1) attendees using the syntax were tweeting more -meaning they were more engaged, or that 2) attendees who tweeted frequently had no difficulty adopting our tags. Informal feedback leads us to think that both situations happened.

What is the impact of presenting tweets and tags on a video web player?

To measure if the tweet visualization component was useful, we logged user interaction on the video player described in Section 4. To identify impact on the user activity on six events, we conducted a Welch's t-test to compare if there is a correlation between where the video was played and the density of tweets present at these positions. According to these measures, viewers seemed to specifically seek out spikes of Twitter activity on five of the six videos: three trails followed the twitter activity significantly (p < .01), two showed trends (.01 < p < .05) and one did not follow (p > .1). All trails that were correlated to the tweets had on average more than one tweet by bar in the chart. According to the attendees survey sent after the end of the event 40 % [START_REF] Daniel | Video visualization[END_REF] reported to have used the Polemic Line visualization. 90 % (10) of these respondents found that the interface provided the following useful information:

• Item summary of twitter activity: 90 % [START_REF] Engström | Amateur vision and recreational orientation: Creating live video together[END_REF] 81 % (9) respondents agreed or strongly agreed that the visualization helped them to browse the video and 100 % [START_REF] Daniel | Video visualization[END_REF] would recommend PT to a friend. We received additional feedback from open questions in our surveys and report the most interesting here. One particularly prominent comment was the notion of hotspot and points of interest, also resonating in the answers above. Several comments were similar to this one: "PT is useful to browse videos faster and find a hotspot (polemic, debate, synthesis, and minutes)". Another frequent comment related to video overview. Five users wrote PT was useful for "viewing the mood (of the audience)", and that this overview helped to "construct an opinion", and to put the talk in perspective to the audience's opinions and references". Others commented on the usefulness of the player after an event to "re-find some information and share it with people who could not attend." We also received a comment from one of the organizers that the player was useful to "prepare the chaptering, for taking notes of a speaker change, and references (to others resources)." On the same point, probably one of the most enthusiastic organizers reported that during the event he used the system for note taking: "it replaced my pencil and paper for note taking."

Discussion

Evaluating PT as a deployed system over a long period of time was challenging because we had little control over our audience and settings. Yet, our analysis and feedback of PT has been predominantly positive on both usability and usefulness as shown in the results from questionnaires and informal feedback. According to our analysis, the three main goals of PT were achieved: 1) the system engaged the audience to provide data for crowdsourced video tagging, 2) the visual backchannel incited people to tag their tweets and thus provided annotation structure, and 3) the media player augmented with the annotated tweet visualization was considered a significant improvement over traditional video players.

We found that PT was successful beyond its originally intended purpose of video tagging. In particular remote participants were able to get real-time visual feedback about audience sentiment and could be involved in the event as a commentator and annotator and not just a passive listener. The PT environment and backchannel interface provide a friendly tool for augmenting the public debate around conferences; this is something we consider valuable for improving the communication between important societal issues such as citizens and science or politics in the digital era.

Beyond that, the PT outcomes indicate that we can crowdsource an annotation task using social networks (and not only with dedicated crowdsourcing platforms) depending on the content and community. Last, the use of the backchannel interface can aid by providing an incentive to achieve a task, like tagging during a collaborative synchronous activity. In summary we found a considerable amount of excitement of participants and event organizers. PolemicTweet can be considered as a first step into exploring real-time information visualization of crowdsourced tasks. Our design is applicable to a variety of different scenarios and events such as classroom presentations, synchronous web seminars, public debates, and social TV or even to popular video websites to leverage the crowd for annotating the videos to provide a richer user experience. Despite this, some of our choices should be re-assessed:

• Possible optimization. We collected a large number of annotations through tweets but not all were tagged. With our pre-defined tags we could achieve a better precision than sentiment analysis algorithms, but for some events had only low recall. More work is needed to research incentives for the audience to tag tweets. • Shared attention. Asking the audience to tweet during an event and moreover to tag their tweets comes with challenges. First, it requires additional work to remember the tags and add them. The danger is that attendees may be losing focus of their main activity (listening to the conference) and even of their secondary activity such as writing a tweet. Can we further simplify the interface to spare some attention? • Tags Property. We received a lot of feedback from Twitter users on our tags. Some of them criticized their restrictiveness; others felt that tags should be based on existing usage for expressing sentiment. We have several reasons not to rely on existing tags or practices: tags like #fail, #happy, as well as Smileys, are not universal, are ambiguous and are longer than our syntax. • Scalability, User interface design and system. PT comes with some scalability problems. Due to the homothetic representation of tweets, the interface is limited by temporal density. Our design works well for events in which Twitter activity is between 0.5-5 tweets per minute. A graphical scalability problem existed with two of our recorded events: the standard representation took too much screen real estate. To solve this problem, we used a classical aggregated stacked area chart.

Conclusion

In this paper we present PolemicTweet a system and protocol that allows reducing video annotation complexity and cost for events of approx. 25-300 attendees, such as academic conferences. We crowdsourced the annotation process through Twitter and made the annotation process more engaging and open for a wider audience. The system's initial goal was to provide an open and engaging method for event organizers to tag and annotate the videos they recorded during their events, but also to supply an accessible tool to play the video over the web and show the annotations.

Fig. 1 .

 1 Fig. 1. The three stages of PolemicTweet: 1) printed information on connection & tags, 2) the live Twitter client & structured annotations, 3) video player showing tweet annotations.

Fig. 2 .

 2 Fig. 2. The Microsoft conference room in Paris where PolemicTweet was deployed; the use of laptops or other wifi-capable devices was prevalent and ensured active participation.

Fig. 3 .

 3 Fig. 3. The PolemicTweet backchannel interface.

Fig. 4 .

 4 Fig. 4. Each component of the MetadataPlayer used in PT.

Fig. 5 .

 5 Fig. 5. Search for "ushahidi" in the polemic line". The tweets referring to the section of the talk related to this query are highlighted in pink.

Fig. 6 .

 6 Fig. 6. Polemic lines of A) "Les Stratégies Virtuelles des Musé: l'Heure de Vérité", B) "Mashup Remix", C) "Du vinyl au numérique," a 2h event with different density of annotation, number of peaks, and tagging activity.

Fig. 7 .

 7 Fig. 7. Tweet with and without PT tags split by backchannel vs. other clients.

PolemicTweet is a deployed system, used regularly now at least once a month. This system was deployed for more than one year and co-evolved during several cycles with its users (audience and organizers). This is the reason for its success as well as for the difficulty of evaluating it according to its design goals: they have evolved.

We assessed the success of PolemicTweet and the factors influencing adoption, participation, and use in a long-term deployment. We recorded a wide range of different events and despite the differences in audience, topic content, and event location, the system showed its robustness and effectiveness: 1) to produce useful structured annotation and tagging in most cases, 2) to provide a method and tool to engage the audience in a live tagging activity, 3) to support web-based video browsing activity while providing useful landmarks with a simple yet powerful navigation tool.