Claire-Hélène Demarty

Cédric Penet

Guillaume Gravier

Mohammad Soleymani

A benchmarking campaign for the multimodal detection of violent scenes in movies

We present an international benchmark on the detection of violent scenes in movies, implemented as a part of the multimedia benchmarking initiative MediaEval 2011. The task consists in detecting portions of movies where physical violence is present from the automatic analysis of the video, sound and subtitle tracks. A dataset of 15 Hollywood movies was carefully annotated and divided into a development set and a test set containing 3 movies. Annotation strategies and resolution of borderline cases are discussed at length in the paper. Results from 29 runs submitted by the 6 participating sites are analyzed. The first year's results are promising, but considering the use case, there is still a large room for improvement. The detailed analysis of the 2011 benchmark brings valuable insight for the implementation of future evaluation on violent scenes detection in movies.

Introduction

MediaEval1 is a benchmarking initiative dedicated to evaluating new algorithms for multimedia access and retrieval. MediaEval emphasizes the multimodal character of the data (speech, audio, visual content, tags, users, context, etc). As a track of MediaEval, the Affect Task -Violent Scenes Detection -involves automatic detection of violent segments in movies. This challenge derives from a use case at Technicolor2 . Technicolor is a provider of services in multimedia entertainment, and solutions, in particular, in the field of helping users select the most appropriate content, according to, for example, their profile. Given this, a particular use case arises which involves helping users choose movies that are suitable for children in their family, by previewing the parts of the movies (i.e., scenes or segments) that include the most violent moments.

In the literature, violent scenes detection in movies has received very little attention so far. Monomodal static approaches were initially proposed [START_REF] Pikrakis | Gunshot detection in audio streams from movies by means of dynamic programming and Bayesian networks[END_REF][START_REF] Chen | Action Scene Detection With Support Vector Machines[END_REF].

Multimodality has been recently considered in [START_REF] Giannakopoulos | Audio-visual fusion for detecting violent scenes in videos[END_REF][START_REF] Gong | Detecting violent scenes in movies by auditory and visual cues[END_REF]. However, the main drawbacks of all these methods lie in the lack of a standard definition of violence and of standard databases. For example, a dataset of 20 minutes is used in [START_REF] Giannakopoulos | Violence content classification using audio features[END_REF], a 200-clip collection of scenes from action movies is considered in [START_REF] Bermejo Nievas | Violence detection in video using computer vision techniques[END_REF]. In [START_REF] Chen | Action Scene Detection With Support Vector Machines[END_REF][START_REF] Chen | Violence detection in movies[END_REF], four movies are considered for training and testing. Hence the need for a dedicated benchmark for violent scenes detection is beneficial to provide a consistent and substantial dataset, together with a common definition of violence and with evaluation protocols and metrics.

The choice of the targeted content, i.e., Hollywood movies, raises additional challenges which are not addressed in similar evaluation tasks, for example in the TRECVid Surveillance Event Detection or Multimedia Event Detection Evaluation Tracks 3 . Indeed, systems will have to cope with content of very different genres and special montage effects, which may alter the events to detect. The affect task of MediaEval 2011 therefore constitutes a first attempt to address all these needs.

The paper provides an overview of the 2011 task. Its main contributions are: first, the provision of a definition of violence in movies, second, the description of a comprehensive dataset of 15 Hollywood movies together with their annotations, and valuable insights on the elaboration of the dataset and annotation strategies. Last, this paper reports on the collective effort of the organizers and participants to detect the violent segments in movies. Section 2 details the chosen definition of violence, the task definition and the dataset and evaluation protocols and metrics. In Section 3, results of the benchmark are reported with a short comparative description of the systems. The paper concludes in section 4, with a summary of the lessons learned and directions for a future benchmark.

Task description

The 2011 Affect Task required participants to deploy multimodal approaches to automatically detect portions of movies depicting violence. This calls for a clear definition of violence that serves as a basis for annotating data for the benchmark.

Towards a definition of violence

The notion of violence remains highly subjective as it depends on viewers. The World Health Organization (WHO) defines violence as [START_REF]Violence: a public health priority[END_REF]: "The intentional use of physical force or power, threatened or actual, against oneself, another person, or against a group or community, that either results in or has a high likelihood of resulting in injury, death, psychological harm, maldevelopment, or deprivation". According to the WHO, three types of violence can be distinguished, namely, self-inflicted, interpersonal, and collective [START_REF] Krug | The world report on violence and health[END_REF]. Each category is divided according to characteristics related to the setting and nature of violence, e.g., physical, sexual, psychological, and deprivation or neglect. In the context of movies and television, Kriegel [START_REF] Kriegel | La violence à la télévision. rapport de la mission d'évaluation, d'analyse et de propositions relative aux représentations violentes à la télévision[END_REF] defines violence on TV as an "unregulated force that affects the physical or psychological integrity to challenge the humanity of an individual with the purpose of domination or destruction". These definitions only focus on intentional actions and, as such, do not include accidents, which are of interest in the use case considered, as they also result in potentially shocking gory and graphic scenes, e.g., a bloody crash. We therefore adopted an extended definition of violence that includes accidents while being as objective as possible and reducing the complexity of the annotation task. In MediaEval, violence is defined as "physical violence or accident resulting in human injury or pain". Violent events are therefore limited to physical violence, verbal or psychological violence being intentionally excluded.

Even though we attempted to narrow the field of violent events down to a set of events as objectively violent as possible, there are still some borderline cases. First of all, sticking to this definition leads to the rejection of some shots in which the results of some physical violence are shown but not the violent act itself. For example, shots in which one can see a dead body with a lot of injuries and blood were not annotated as violent. On the contrary, a character simply slapping another one in the face is considered as a violent action according to the task definition. Other events defined as 'intent to kill', in which one sees somebody shooting somebody else for example with the clear intent to kill, but the targeted person escapes with no injury, were also discussed and finally not kept in the violent set. On the contrary, scenes where the shooter is not visible but where shooting at someone is obvious from the audio, e.g., one can hear the gunshot possibly with screams afterward, were annotated as violent. Interestingly, such scenes emphasize the multimodal characteristic of the task. Shots showing actions resulting in pain but with no intent to be violent or, on the contrary, with the aim of helping rather than harming, e.g., segments showing surgery without anesthesia, fit into the definition and were therefore deemed violent. Another borderline case keenly discussed was shots showing the destruction of a whole city or the explosion of a moving tank. Technically speaking, these shots do not show any proof of people death or injury, though one can reasonably assume that the city or the tank were not empty at the time of destruction. Consequently, such cases, where pain or injury is implicit, were annotated as violent. Finally, shots showing the violent action and the result of the action itself happen to be separated by several non violent shots. In this case, the entire segment was annotated as violent if the duration between the two violent shots (action and result) was short enough (less than two seconds).

Data description

In line with the considered use case, the dataset consisted of 15 Hollywood movies from a comprehensive range of genres, from extremely violent to movies without violence. From these 15 movies, the 12 following ones were designated as development data: Armageddon, Billy Elliot, Eragon, Harry Potter and the Order of the Phoenix4 , I am Legend, Leon, Midnight Express, Pirates of the Caribbean and the Curse of the Black Pearl5 , Reservoir Dogs, Saving Private Ryan, The Sixth Sense, the Wicker Man. The three following movies were used as test set: Kill Bill 1, The Bourne Identity and the Wizard of Oz . Statistics on violent scenes in each movie are provided in Table 1. The development dataset represents a total of 21,608 shots-as given by automatic shot segmentation developed internally at Technicolor-for a total duration of 83,800 seconds. Violent content corresponds to 9.5% of the total duration and 12.7% of the shots, pointing out the fact that violent segments are not scarce in the database. We tried to respect the genre repartition (from extremely violent to non violent) both in the development and evaluation sets. This appears in the provided statistics, as some movies such as Billy Elliot or The Wizard of Oz contain a smaller proportion of violent shots (around 5%). The choice we made for the definition of violence impacts the proportion of annotated violence in some movies such as The Sixth Sense where violent shots amount to only 2.8% of the duration. However, the movie contains several shocking scenes of dead people which do not fit the definition of violence that we adopted. In a similar manner, psychological violence, such as what may be found in Billy Elliot, was also not annotated, which also explains the small number of violent shots in this particular movie.

The violent scenes dataset6 was created by seven human assessors. In addition to segments containing physical violence (with the above definition), annotations also include high-level concepts for the visual modality. For violent segments, the annotation was conducted using a 3-step process, with the same so-called 'master annotators' for all movies. A first master annotator extracted all violent segments. A second master annotator reviewed the annotated segments and possibly missed segments according to his/her own judgment. Disagreements were discussed on a case by case basis, the third master annotator making the final decision in case of an unresolved disagreement. Each annotated violent segment contained a single action, whenever possible. In the case of overlapping actions, the corresponding global segment was proposed as a whole. This was indicated in the annotation files by adding the tag "multiple action scene". The boundaries of each violent segment were defined at the frame level, i.e., indicating the start and end frame numbers.

The high-level video concepts were annotated through a simpler process, involving only two annotators. Each movie was first processed by an annotator and then reviewed by one of the master annotator. Seven visual concepts are provided: presence of blood, fights, presence of fire, presence of guns, presence of cold weapons, car chases and gory scenes. For the benchmark, participants had the option to carry out detection of the high-level concepts. However, concept detection is not among the task's goals and these high-level concept annotations were only provided on the development set. Each of these high-level concepts followed the same annotation format as for violent segments, i.e., starting and ending frame numbers and possibly some additional tags which provide further details. For blood annotations, a tag in each segment specifies the proportion of the screen covered in blood. Four tags were considered for fights: only two people fighting, a small group of people (roughly less than 10), large group of people (more than 10), distant attack (i.e., no real fight but somebody is shot or attacked at distance). As for the presence of fire, anything from big fires and explosions to fire coming out of a gun while shooting, a candle, a cigarette lighter, a cigarette, or sparks was annotated, e.g., a space shuttle taking off also generates fire and receives a fire label. An additional tag may indicate special colors of the fire (i.e., not yellow or orange). If a segment of video showed the presence of firearms (respectively cold weapons) it was annotated by any type of (parts of) guns (respectively cold weapons) or assimilated arms. Annotations of gory scenes are more difficult. In the present task, they are indicating graphic images of bloodletting and/or tissue damage. It includes horror or war representations. As this is also a subjective and difficult notion to define, some additional segments showing disgusting mutants or creatures are annotated as gore. In this case, additional tags describing the event/scene are added.

In addition to the video data, automatically generated shot boundaries with their corresponding key frames, as detected by Technicolor's software, were also provided with each movie.

Evaluation rules

Due to copyright issues, the video content was not distributed and participants were required to buy the DVDs. We provided the online store's URLs for the DVDs which were used for annotations. This was done to ensure that every participant can access the exact, same version of the movies. Participants were allowed to use all information automatically extracted from the DVDs, including visual and auditory material as well as subtitles. English was the chosen language for both the audio and subtitles channels. The use of any other data, not included in the DVD (web sites, synopsis, etc.) was not allowed.

Two types of runs were initially considered in MediaEval 2011, a mandatory shot classification run and an optional segment detection one. The shot classification run consisted in classifying each shot provided by Technicolor's shot segmentation software as violent or not, optionally with a confidence score-the higher the score, the more likely the violence. The segment detection run involved detection of the violent segment boundaries, regardless of the shot segmentation provided.

For official ranking of the systems, system comparison was based on a detection cost function weighting false alarms (FA) and missed detections (MI), according to

C = C f a P f a + C miss P miss (1)
where the costs C f a = 1 and C miss = 10 are arbitrarily defined to reflect (a) the prior probability of the situation and (b) the cost of making an error. P f a and P miss are respectively the FA (false positive) and MI (false negative) rates given the system's output and the reference annotation. In the shot classification, the FA and MI rates were calculated on a per shot basis while, in the segment level run, they were computed on a per unit of time basis, i.e., durations of both references and detected segments are compared. This cost function is called 'MediaEval cost' in the following. To avoid only evaluating systems at given operating points and enable full comparison of systems, we also used detection error trade-off (DET) curves whenever possible, plotting P f a as a function of P miss given a segmentation and a confidence score for each segment. Note that in the segment detection run, DET curves are possible only for systems returning a dense segmentation (a list of segments that spans the entire video): segments not present in the output list are considered as non violent for all thresholds.

Results

The Affect Task on Violent Scenes Detection was proposed in MediaEval as a pilot for the first year. Thirteen teams, corresponding to 16 research groups considering joint submission proposals, declared interest in the task. Finally, six teams registered and completed the task, representing four different countries, for a grand total of 29 runs submitted (see Table 2). From the number of groups interested and the number of participants, the Affect task has proved to be of interest for the research community. This was confirmed by the active mailing list, which also denoted more a collaborative spirit between the teams than a competitive one, as promoted by MediaEval campaigns. All participants submitted runs for the required shot classification task and none to the optional segment detection. Results are summarized in Table 2. The 29 submissions mostly correspond to 6 different systems which can be grouped in three main categories. Two participants (NII [START_REF] Lam | Nii, japan at mediaeval 2011 violent scenes detection task[END_REF] and LIG [START_REF] Safadi | Lig at mediaeval 2011 affect task: use of a generic method[END_REF]) treated the problem of violent scenes detection as a concept detection problem, applying generic systems developed for TRECVid evaluations to violent scenes detection, potentially with specific tuning. Both sites used classic video only features, computed on the keyframes provided, based on color, textures, edges, either local (interest points) or global, and classic classifiers. One participant (DYNI [START_REF] Glotin | Real-time entropic unsupervised violent scenes detection in hollywood movies -dyni @ mediaeval affect task 2011[END_REF]) proposed a classifier-free technique exploiting only two low-level audio and video features, computed on each successive frame, both measuring the activity within a shot. After a late fusion process, decision was taken by comparison with a threshold. The last group of participants (TUB [START_REF] Acar | Mediaeval 2011 affect task: Violent scene detection combining audio and visual features with svm[END_REF] , UGE [START_REF] Gninkoun | Automatic violence scenes detection: A multi-modal approach[END_REF] and TI [START_REF] Penet | Technicolor and inria/irisa at mediaeval 2011: learning temporal modality integration with bayesian networks[END_REF]) built dedicated supervised classification systems for the task of violent scenes detection. Different classifiers were used from SVM, Bayesian networks to linear or quadratic discriminant analysis. All used multimodal features, either audio-video or audio-video-textual features (UGE). Features were computed globally for each shot (UGE, TI) or on the provided keyframes (TUB). Both early (TUB, UGE, TI) or late (TI) fusions were used, together with a temporal integration of the decisions at the output of the classifiers (UGE, TI).

Based on the results achieved by the different systems, one may draw some tentative conclusions about the global characteristics that were more likely to be useful for violence detection. Local video features (SIFT-like), as used by LIG, NII and TUB, did not add a lot of information to the systems. On the contrary, taking advantage of different modalities seems to improve performance. This is confirmed by comparing the two runs from DYNI, and the TI runs among which monomodal or multimodal configurations were submitted. Although results do not prove their action in one way or another, it also seems of interest to use temporal integration. This was carried out in different manners in the systems, either by using contextual features, i.e., features at different times, or by temporal smoothing or aggregation of the decisions at the output of the chain.

Lessons learned

One goal of this task proposal was to provide a shared framework for violence detection systems for videos. Having the same database and annotations, and the same definition of the violent events is already a significant step towards building of such a framework. However, many lessons were learned from the implementation of the task as a pilot in MediaEval 2011.

Globally, it should first be noted from Table 2 that the overall performances of the proposed systems are not good enough to satisfy the requirements of a real-life commercial product. This means that the problem of automatically detecting violent scenes is still far from being solved and needs further attention.

Figure 1 shows the evolution of the detection error curves. These curves were build using the scores provided by all but one participant. On each of them, only the best run per participant, according to the MediaEval cost, was kept. Instead of giving an evaluation of the systems at different operating points, Figure 1 proposes a more complete comparison. Additionally, it should be noted that the ordering of the systems differs according to the chosen metric.

From Table 2, the metric also appears to be excessively biased towards MI errors or, equivalently, towards high recall at the expense of precision. A ratio of 10 between the FA and MI rates turned out to be so high that it leads to classifying all the scenes into the violent class for some systems (NII). Indeed, classifying all shots as violent results in a MediaEval cost value of 1 which is in most cases lower than what automatic systems obtained. This conclusion calls for a review of the metric in the future, towards a less biased criterion but still reflecting the Technicolor use case.

The definition of violence in the last campaign was chosen to satisfy a need of objectivity in the events, but does not cover all the violent scenes in the context of the Technicolor use case. With such a definition, some actions, e.g., one hurting himself accidentally against a chair, belong to the class of events to detect, despite their minor violent content. Conversely shocking scenes of dead or severely injured people will not be counted as violent. This emphasizes the need for further improvement of the current definition in the future campaigns. Another drawback of this definition is that it complicates the choice of relevant features for the task. For example, the presence of blood which could be a decisive feature is no longer enough to recognize a scene as violent.

A relatively large and standard dataset of movies for violence detection has been developed. The developed dataset is roughly four times larger than the largest reported dataset in the literature. But even 15 movies is still not large enough to cover the variety of violent scenes in movies. First, violent events remain relatively rare (10% of the dataset). Second, because of the large variation between existing violent actions or events, in a dataset with 15 movies, there are only few similar violent excerpts. Therefore, the development of a larger dataset is certainly beneficial.

Conclusions

The Affect Task on Violent Scenes Detection in the context of the MediaEval 2011 benchmarking initiative has been presented. As a pilot task, this first year reached its objective: a common definition of the events to detect, together with a standard dataset and its associated ground truth were proposed, leading to a first solid basis for further research on this topic. Detailing this set-up to the research community was the main contribution of this paper. The task also successfully attracted participants, showing that this task and its open issues are interesting for multimedia indexing and discovery research community. For MediaEval 2012, the task will be further developed, with improvements in the task definition, dataset and the chosen evaluation metrics.

Authors' note

We would like to thank all the task participants and other contributors whose combined efforts contributed to make this first year run successful, especially our annotators who did a tedious but nevertheless necessary and valuable job. We also greatly appreciate our participants for giving us consent to describe their systems and results in this paper.

Fig. 1 .

 1 Fig. 1. False alarms vs. missed detections for each best run per participant. The best runs were selected according to the MediaEval cost values.

Table 1 .

 1 Movie dataset (Dev. set: first 12 movies; test set: last 3 movies).

	Movie	Duration	Shot length	Violence	Violent Shots
				Duration	(%)
				(%)
	Armageddon	8680.16	3562	14.03	14.6
	Billy Elliot	6349.44	1236	5.14	4.21
	Eragon	5985.44	1663	11.02	16.6
	Harry Potter 5	7953.52	1891	10.46	13.43
	I am Legend	5779.92	1547	12.75	20.43
	Leon	6344.56	1547	4.3	7.24
	Midnight Express	6961.04	1677	7.28	11.15
	Pirates Carib. 1	8239.4	2534	11.3	12.47
	Reservoir Dogs	5712.96	856	11.55	12.38
	Saving Private Ryan 9751.0	2494	12.92	18.81
	The Sixth Sense	6178.04	963	1.34	2.80
	The Wicker Man	5870.44	1638	8.36	6.72
	Total	83805.9	21608	9.52	12.7
	Kill Bill	5626.6	1597	17.4	24.8
	The Bourne Identity 5877.6	1995	7.5	9.3
	The Wizard of Oz	5415.7	908	5.5	5.0
	Total	16919.9	4500	10.2	14.0

Table 2 .

 2 Participation and results (DYN: University of Toulon; NII: National Institute of Informatics; TUB: Technical University of Berlin; UGE: University of Geneva; LIG: Laboratoire d'Informatique de Grenoble; TI: joint participation Technicolor-INRIA. The MediaEval cost value corresponds to the best run per participant, according to this metric. MAP: mean average precision. (*) task organizers.

	Part. Country # Runs Med. Cost MAP
	DYN France	2	6.46	0.08
	NII	Japan	6	1.00	0.18
	TUB Germany 3	1.26	0.12
	UGE* Switzerland 5	2.00	0.17
	LIG France	1	7.93	0.04
	TI* France	12	0.76	0.25

http://www.multimediaeval.org/

http://www.technicolor.com

http://www.nist.gov/itl/iad/mig/sed.cfm

Harry Potter

[START_REF] Giannakopoulos | Violence content classification using audio features[END_REF] Pirates Carib. 1

The annotations, shot detections and key frames for this task were made available by Technicolor. The dataset can be obtained after signing the User Agreement form, available on the website (http://www.multimediaeval.org).


Mathieu Coureuil
email: mathieu.coureuil@inserm.fr

Guillain Mikaty

Florence Miller

Hervé Lécuyer

Christine Bernard

Sandrine Bourdoulous

Guillaume Duménil

René-Marc Mège

Babette B Weksler

Ignacio A Romero

Pierre Olivier Couraud

Xavier Nassif

Meningococcal Type IV Pili Recruit the Polarity Complex to Open the Cell-Cell Junctions of Brain Endothelium

Type IV pili mediate the initial interaction of many bacterial pathogens with their host cells. In Neisseria meningitidis, the causative agent of cerebrospinal meningitis, type IV pili-mediated adhesion to brain endothelial cells is required for bacteria to cross the blood-brain barrier. Here, Type IV pili-mediated adhesion of N. meningitidis to human brain endothelial cells was found to recruit the Par3/Par6/PKCζ polarity complex that plays a pivotal role in the establishment of eukaryotic cell polarity and the formation of intercellular junctions. This recruitment leads to the formation of ectopic intercellular junctional domains at the site of bacterial-cell interaction and a subsequent depletion of junctional proteins at the cell-cell interface with opening of the intercellular junctions of the brainendothelial interface.

Neisseria meningitidis is a commensal bacterium of the human nasopharynx that, after bloodstream invasion, crosses the blood-brain barrier (BBB) (1). Few pathogens have a tropism for the brain, indicating that N. meningitidis possess specific components to interact with the BBB. Meningeal colonization by invasive capsulated N. meningitidis is the consequence of the bacterial adhesion onto brain endothelial cells (2, 3) which is followed by bacterial division onto the apical surface of the cells (Movie S1). This process is mediated by Type IV pili (Tfp) (4)(5)(6)(7)(8)(9). In addition, by powering a form of cell locomotion, reported as twitching motility (10), Tfp lead to the spread of the bacteria on the surface of the cells and the formation of microcolonies. Subsequent to the formation of these microcolonies, Tfp trigger the recruitment of cortical actin and signal transducing proteins leading to the formation of filopodia-like structures (2,(11)(12)(13). The crossing of the BBB by N. meningitidis implies that following Tfp mediated adhesion, the bacteria transcytose through the brain capillaries and/or open the brain endothelium.

To investigate whether adhesion of N. meningitidis affects the integrity of the adherens (AJ) and/or tight (TJ) junctions of human brain endothelial cells, the consequences of infection by N. meningitidis on the distribution of junctional proteins were analyzed using the human brain microvascular endothelial cell line hCMEC/D3 (14). After infection, components of the AJ (VE-cadherin, p120-catenin, β-catenin) and TJ (ZO1, ZO2, and claudin-5) were targeted underneath N. meningitidis colonies (Fig. 1A). At the site of N. meningitidis adhesion, these junctional proteins co-distributed with each other and with the actin honeycomb-like network. In non infected cells, the recruitment of junctional proteins usually occurs at the cell-cell interface and is controlled by several polarity proteins (Par3/Par6/PKCζ) (15)(16)(17). In infected monolayers, Par3 and Par6 were observed underneath N. meningitidis colonies (Fig. 1B). Thus, N. meningitidis triggers a signal leading to the formation of an ectopic domain containing filopodia-like structures and enriched in junctional proteins, thus resembling spot-like adherens junctions observed during early steps of junctional biogenesis. We refer to this domain as an "ectopic early junction-like domain" (18). Using isogenic derivatives, Tfp-induced signaling was shown to be responsible for the formation of these ectopic early junction-like domains (Fig. S1A andB). However, Tfp retraction through the PilT motor was not required for formation of the ectopic domains (Fig. S1D andE).

The small GTPase Cdc-42 is required for polarization of mammalian cells (19,20). The role of this component in the recruitment of the polarity complex by N.meningitidis was investigated. Transfection of a dominant negative mutant of Cdc42 or knockdown of Cdc42 by RNAi inhibited the recruitment of Par6, Par3 (Fig. 2A, S2A), VE-cadherin, p120-catenin and actin (Fig. 2B,S2B,S3). These results link the Cdc42/polarity complex pathway with the formation of the ectopic early junction-like domains.

The role of the polarity complex in the recruitment of junctional proteins was further explored by studying the inhibition of Par3 and Par6 using either dominant negative mutants or knockdown by RNAi. PKCζ inhibition was assessed using a PKCζ pseudosubstrate inhibitor (PKCζ-PS) (21). Inhibition of Par6 and PKCζ reduced the recruitment of p120-catenin, VE-cadherin and actin (Fig. 2B,2C,S2C,S3) and that of Par3 (Fig. 2D,S2E), consistent with the finding that the Par6/PKCζ complex recruits Par3 at intercellular junction domains (22). On the other hand, inhibition of Par3 reduced only the recruitment of VE-cadherin (Fig. 2B, S2D, S3), consistent with Par3 being specifically needed for junctional proteins targeting at early cell-cell junctions (23).

These observations confirmed the role of the polarity complex in the recruitment of the junctional proteins by N. meningitidis.

The sequence of events leading to the targeting of AJ proteins at the cell-cell junctions during cellular polarization remains unknown. To get insight into this process, we engineered a VE-cadherin knockdown of hCMEC/D3 cells by stable expression of a VEcadherin shRNA (VEC shRNA) (Fig. 3A, 3B, S4A). In this cell line, p120-catenin and actin were still recruited beneath N. meningitidis colonies, whereas recruitment of βcatenin was dramatically reduced. On the other hand, down-regulation of p120-catenin using RNAi (Fig. 3C,S4B) resulted in inhibition of VE-cadherin and of actin recruitment. Consistent with a previous report, cortactin and Arp2/3 were not recruited by the bacterial colonies in p120-catenin knockdown cells (24) (Fig. S4C). Furthermore, inhibition of Src kinase, which phosphorylate cortactin and is activated following the formation of the cortical plaque (25) did not modify p120-catenin recruitment but inhibited VE-cadherin and actin recruitment (Fig. S4D,S4E). Taken together, these results strongly suggest that p120-catenin-mediated recruitment of actin and VE-cadherin requires the recruitment and phosphorylation of cortactin by the Src kinase. In summary, Cdc42, via the polarity complex, organizes this ectopic early junction-like domain, mainly by the initial recruitment of p120-catenin.

We asked whether the signal triggered by Tfp and leading to the formation of these ectopic early junction-like domains destabilized intercellular junctions, especially by redirecting a recycling pool of junctional proteins to the N. meningitidis adhesion site.

First, inhibition of protein synthesis did not prevent recruitment of VE-cadherin (Fig. S5A). Second, inhibition of clathrin coated pit formation blocked VE-cadherin recruitment (Fig. S5B andS5C) suggesting that VE-cadherin internalization is required for its targeting underneath N. meningitidis colonies. Third, when monolayers were tagged before infection with a VE-cadherin monoclonal antibody, antibodies are relocalized beneath colonies in infected monolayers (Fig. S6). Thus the VE-cadherin delocalized by the bacteria was coming from the intercellular junctions. This redistribution of the AJ proteins was associated with a reduction of the amount of tagged VE-cadherin at the intercellular junction (Fig. S6, Movie S2). Thus the junctional VEcadherin is internalized and then mistargeted at the site of bacterial cell interactions.

Depletion of intercellular junction proteins from the cell-cell interface could open a

paracellular route for bacterial spread. Indeed, N. meningitidis was shown to increase permeability to Lucifer Yellow (LY) a compound which mark passive paracellular diffusion (Fig. 4A) (26). Moreover, this increase relied on PKCζ activity and bacterial piliation (Fig. 4A). This modification of permeability was associated with the formation of gaps between infected cells (Fig. 4B). The number of gaps increased over time and was reduced by the PKCζ pseudosubstrate inhibitor (Fig. 4B and4C). Gaps did not form when cells were infected with a non piliated strain, showing that these gaps are due to Tfp-mediated signaling (Fig. 4C). Indeed, piliated strain cross the monolayer at a higher rate than non-piliated isogenic derivatives or a piliated strain in the presence of PKCζ PS (Fig. 4D). Thus the signaling induced by N. meningitidis Tfp leading to the recruitment of the polarity complex is associated with large alterations of the intercellular junctions sufficient for the bacteria to cross the brain endothelial cell monolayer.

In summary, N. meningitidis microcolonies trigger via type IV pili a signal resembling the one responsible for the formation of AJ at cell-cell junctions. This leads to the formation of ectopic early junction-like domains (Fig. S7), thus disorganizing the cell-cell junctions and opening the paracellular route allowing N. meningitidis to cross the BBB and to invade the meninges. meningitidis (Nm) or its non piliated isogenic strain (N.meningitidis ΔpilE). Size and quantity of gaps observed 4h after infection are calculated as described [START_REF]Described in the Supporting Online Material: Materials and methods section[END_REF]. (D) Diffusion of N. meningitidis (Nm) through a hCMEC/D3 monolayer, 4h post-infection, is

Figure 1

 1 Figure 1

Figure 2 The

 2 Figure 2

Figure 3 P120

 3 Figure 3

Figure 4 N

 4 Figure 4


Alexis Benichoux
email: alexis.benichoux@irisa.fr

Prasad Sudhakar
email: prasad.sudhakar@gmail.com

Frédéric Bimbot
email: frederic.bimbot@irisa.fr

Rémi Gribonval
email: remi.gribonval@inria.fr

Well-posedness of the permutation problem in sparse filter estimation with lp minimization

Keywords: sparse filters, convolutive blind source separation, permutation ambiguity, ℓ p minimization, Hall's Marriage Theorem, bi-stochastic matrices

Convolutive source separation is often done in two stages: 1) estimation of the mixing filters and 2) estimation of the sources. Traditional approaches suffer from the ambiguities of arbitrary permutations and scaling in each frequency bin of the estimated filters and/or the sources, and they are usually corrected by taking into account some special properties of the filters/sources. This paper focusses on the filter permutation problem in the absence of scaling, investigating the possible use of the temporal sparsity of the filters as a property enabling permutation correction. Theoretical and experimental results highlight the potential as well as the limits of sparsity as an hypothesis to obtain a well-posed permutation problem.

Introduction

Blind source separation and blind source localization are ubiquitous problems in signal processing, with applications ranging from wireless telecommunications to underwater acoustics and sound enhancement.

These problems can be considered as reasonably well understood and solved in simple linear instantaneous settings, where tools such as Independent Component Analysis, as well as techniques exploiting source sparsity, are now mature. However, the convolutive source localization / separation problem remains much more challenging. In particular, without further assumption than statistical independence between sources, the problem is known to be ill-posed because of the so-called frequency permutation (and scaling) problem: at best, one can hope to estimate for each frequency (up to a source and frequency dependent scaling factor) the collection of frequency components of all sources (and of the associated mixing filters); but one cannot match the estimated frequency components from different subbands to globally identify the sources (and mixing filters).

Several practical approaches have been proposed to solve the permutation and scaling problems in practice, by exploiting various properties of either the mixing filters or the sources to match different frequency subbands. While some of these methods may succeed in practice for certain types of sources and filters, there is no known theory guaranteeing the well-posedness of the permutation and scaling problem under appropriate assumptions.

This paper contributes to fill this gap, by exploring well-posedness guarantees for the permutation problem under sparsity assumptions on the mixing filters, as well as their limitations. Sparse filters, associated to impulse responses corresponding to a limited set of echoes, are commonly used to model a number of communication channels in various scenarios such as underwater acoustics [START_REF] Berger | Sparse channel estimation for multicarrier underwater acoustic communication: From subspace methods to compressed sensing[END_REF] and wireless telecommunications [START_REF] Sharp | Application of sparse signal recovery to pilotassisted channel estimation[END_REF][START_REF] Bajwa | Compressed channel sensing: A new approach to estimating sparse multipath channels[END_REF]. Can this sparsity be exploited for blind source localization and separation? A first set of results of this paper (Propositions 1,2,3) shows that, even if the scaling problem is ideally solved, the frequency permutation problem can remain ill-posed for filters that perfectly fit the sparse model with extreme levels of sparsity. In contrast, the main theoretical results achieved in this paper (Theorem 2) show that, when the scaling problem is solved, the frequency permutation problem is well-posed provided that the filters are of prime length and "sufficiently sparse", where the required level of sparsity depends on the number of sources. Whether the scaling problem itself can be expressed in a well-posed manner remains an open question, as well as the related practical question of designing computationally efficient algorithms to solve it.

Context and notations

Let x i [t], 1 ≤ i ≤ M be M mixtures of N source signals s j [t], resulting from the convolution with filters a ij [t] of length L such that:

x i [t] = N j=1 (a ij ⋆ s j )[t], 1 ≤ i ≤ M, (1)
where ⋆ denotes convolution. The filter a ij [t] typically models the impulse response between the j th source and the i th sensor. Denoting the Discrete Fourier Transform matrix by

F = 1 √ L e 2iklπ L 1≤k,l≤L
, we will use the common abuse of notation, Fa ij = {a ij [ω]} 0≤ω<L to distinguish the frequency coefficients of the filter from the time domain vector

a ij = {a ij [t]} 0≤t<L ∈ C L .
Also, the mixing equation ( 1) can be rewritten as X = A ⋆ S, with A the matrix of filters

A := ({a ij [t]} 0≤t<L ) 1≤i≤M, 1≤j≤N , (2)
X the observation matrix and S the source matrix.

In this context, blind filter estimation refers to the problem of obtaining estimates of the filters A from the mixtures X, without any explicit knowledge about the sources S. Mixing filters estimation is relevant for several purposes such as deconvolution, source localization, etc. [START_REF] Benesty | Springer Handbook of Speech Processing[END_REF]. It also has a relationship with the problem of Multiple-Input-Multiple-Output (MIMO) system identification in communications engineering [START_REF] Chen | Frequency domain blind mimo system identification based on second and higher order statistics[END_REF].

Frequency domain filter estimation

Estimating the mixing parameters is made easier when all filters are instantaneous, that is to say of length L = 1, as the convolution product in (1) coincides with the usual product. However, things get complicated in the general setting of convolutive mixtures.

A common approach for filter estimation then relies on the narrowband approximation [START_REF] Pedersen | A survey of convolutive blind source separation methods[END_REF][START_REF]Handbook of Blind Source Separation, Independent Component Analysis and Applications[END_REF][START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF] to transform the mixing model in Eq. ( 1) into the timefrequency domain, converting a single convolutive filter estimation problem into several complex instantaneous filter estimation problems. Using standard techniques for instantaneous mixing parameter estimation [START_REF]Handbook of Blind Source Separation, Independent Component Analysis and Applications[END_REF], complex mixing filter coefficients

Ã[ω] = {ã ij [ω]} 1≤i≤M, 1≤j≤N (3)
are then estimated for each frequency bin 0 ≤ ω < L.

Permutation and scaling ambiguities

Without further assumption on either the filters a ij [t] or the sources s j [t], one can at best hope to find an estimation à = (ã ij ) where for every frequency ω we have ãij [ω] = λ j [ω]a iσω(j) [ω],with λ j [ω] a scaling ambiguity and σ ω ∈ S N a permutation ambiguity, where S N is the set of permutations of the integers between one and N. Several methods [START_REF] Pedersen | A survey of convolutive blind source separation methods[END_REF] attempt to solve for these ambiguities by exploiting properties of either the sources S or the filters A [START_REF] Serviere | A novel method for permutation correction in frequency-domain in blind separation of speech mixtures[END_REF][START_REF] Sanei | A coupled hmm for solving the permutation problem in frequency domain bss[END_REF][START_REF] Wang | A novel hybrid approach to the permutation problem of frequency domain blind source separation[END_REF].

Exploiting sparsity to solve the permutation ambiguity

The focus of this article is the use of the sparsity of A in the time domain to find σ 0 , . . . σ L-1 ∈ S N , assuming the scaling is solved, i.e., λ j [ω] = 1.

Assuming that A is sparse means that each filter a ij has few nonzero coefficients, as measured by the ℓ 0 "norm"

a ij 0 := ♯{0 ≤ t < L, a ij [t] = 0}. (4)
We use the quoting marks following the early notations of D. Donoho [START_REF] Donoho | Sparse components of images and optimal atomic decompositions[END_REF], as a reminder that this is neither a norm nor a quasi norm. The approach considered in this article is to to seek permutations σ 0 , . . . σ L-1 yielding the sparsest estimated time-domain matrix of filters A = ( a ij ) where a ij [ω] := ãi σω(j) [ω]. Besides the ℓ 0 "norm" A 0 := ij a ij 0 , the following ℓ p quasinorms will be used to quantify the sparsity of A:

A p p := ij a ij p p = ijt | a ij [t]| p , 0 < p ≤ 1.
(5)

Main results

The main result of this paper (Theorem 2) is a theoretical guarantee that when the filter length L is prime, k-sparse filters (i.e., such that a ij 0 ≤ k) uniquely minimize the ℓ 0 "norm" of A (up to a global permutation) if k L ≤ α(N), where α(N) is a threshold that depends on the number of sources N.

To reach this bound we exploit uncertainty principles (Lemma 1, based on Chebotarev's theorem on roots of unity) as well as the bistochastic structure of the problem through an apparently new quantitative result on bistochastic matrices (Lemma 2, exploiting Hall's Marriage Theorem).

In the practical setting of orthogonal frequency division multiplexed transmission scheme with pilot symbol assisted modulation, relative sparsity levels k/L between 0.1 and 0.2 have been reported [START_REF] Sharp | Application of sparse signal recovery to pilotassisted channel estimation[END_REF]. These can be compared with the well-posedness thresholds α(2) = 0.25 and α(3) = 0.125, showing that (from the worst-case perspective considered in this paper), sparsity can be exploited for N = 2 sources and possibly for N = 3 sources.

Structure of the paper

The main theorems are stated and proved in Sections 2-3. In Section 4 we discuss the strength of the assumptions used in the theorems, and how much these could be relaxed. In Section 5, a naive combinatorial ℓ p minimization algorithm is proposed to resolve filter permutations and used for Monte-Carlo simulations. We conclude with a discussion of the potential, as well as the limits, of sparsity as a hypothesis to solve permutation problems, in connection with the theoretical and empirical results. The proofs of all technical lemmata are gathered in the appendix.

Theoretical guarantees

Given an M × N filter matrix A, made of filters of length L, and L permutations σ 0 , . . . σ L-1 ∈ S N , we let à be the matrix obtained from A by applying the permutations in the frequency domain, without scaling

ãij [ω] = a iσω(j) [ω], (6)
The effect of the permutations is said to coincide with that of a global permutation π ∈ S N of the columns of A if ãij = a iπ(j) , ∀i, j, or equivalently in the frequency domain:

ãij [ω] := a iσω (j) [ω] = a iπ(j) [ω], 0 ≤ ω < L, ∀i, j. (7)
This is denoted A ≡ Ã. To begin with, we show that for filters with disjoint time-domain supports, permutations cannot decrease the ℓ p quasi-norm, 0 ≤ p ≤ 1:

Theorem 1. Let Γ ij ⊂ {0, . . . , L -1} be the time-domain support of a ij .
Suppose that for all i and j 1 = j 2 we have

Γ i,j 1 ∩ Γ i,j 2 = ∅. (8)
Then, for 0 ≤ p ≤ 1, we have à p ≥ A p .

Proof First, notice that for each frequency 0 ≤ ω < L and channel i ≤ M, permutations preserve the equality

j a ij [ω] = j ãij [ω]
. Thus, the same holds in the time-domain : j a ij = j ãij . By the disjoint supports hypothesis and the triangle inequality satisfied by ℓ p quasi-norm raised to the p-th power, which defines a proper metric d(x, y) := xy p p , we have

j a ij p p = j a ij p p = j ãij p p ≤ j ãij p p . (9)
We conclude by summing over all channels i.

Note that filters with disjoint supports need not be very sparse: M filters of length L can have disjoint supports provided that max j a ij 0 ≤ L/M. Yet, disjointness of filter supports is a strong assumption, and Theorem 1 only indicates that frequency permutations cannot decrease the ℓ p quasinorm. Thus, the minimum value of the ℓ p quasi-norm might not be uniquely achieved (up to a global permutation). In the main result, we consider ksparse filters of prime length, and p = 0:

Theorem 2. Let A be an M ×N matrix of filters of prime length L. Assume that max ij a ij 0 ≤ k, ( 10
)
where k L ≤ α(N) := 2 N (N +2) if N is even, 2 (N +1) 2 if N is odd. ( 11
)
Then, up to a global permutation, A uniquely minimizes the ℓ 0 "norm" among all possible frequency permutations.

Proof of Theorem 2

The proof of Theorem 2 relies on a measure of the "amount" of incurred permutation, on uncertainty principles, and on combinatorial arguments related to bi-stochastic matrices, involving Hall's Marriage Theorem.

Measures of the amount of incurred permutations

To measure the "amount" of incurred permutations, one could count the number of frequency bands where a non-trivial permutation is incurred, with respect to the best matching global permutation π, i.e., min π ♯{ω, σ ω = π}. However, this would generally yield the maximum count L -1.

An alternative is to count the "size" of the incurred permutations, given a reference global permutation π, as the maximum number of frequencies where each estimated filter à actually differs from the (globally permuted) original filters A, yielding:

∆( Ã, A|π) := max i,j F(ã ij -a iπ(j) ) 0 (12) ∆( Ã, A) := min π∈S N ∆( Ã, A|π). (13)
Note that ∆( Ã, A) = 0 iff à ≡ A.

Exploitation of an uncertainty principle

With this notation, we have the following Lemma:

Lemma 1. Assume that à ≡ A, that L is a prime integer, and that

2k + ∆ ≤ L ( 14
)
with k := max ij a ij 0 . Then à 0 > A 0 and ãij 0 ≥ a ij 0 , ∀i, j. The latter inequality is strict when ãij = a ij . For a general L (not necessarily prime), the same conclusions hold when the assumption ( 14) is replaced with

2k • ∆ < L. (15)
The skilled reader will rightly sense the role of uncertainty principles [START_REF] Donoho | Uncertainty principles and signal recovery[END_REF][START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases, Information Theory[END_REF][START_REF] Tao | An uncertainty principle for cyclic groups of prime order[END_REF] in the above lemma. The case L prime allows to use a consequence of Chebotarev's theorem on roots of unity, which already lead to theoretical guaranties for exact sparse recovery in the context of linear inverse problems with partial Fourier measurements [START_REF] Candès | Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, Information Theory[END_REF].

Combinatorial arguments

Using Lemma 1 with prime L, a simple combinatorial argument could be used to obtain a weakened version of Theorem 2, with the more conservative constant α ′ (N) := 1/2N! [START_REF] Benichoux | Well-posedness of the permutation problem in sparse filter estimation with lp minimization[END_REF]. The proof of Theorem 2 with the constant α(N) exploits a stronger universal upper bound ∆( Ã, A) ≤ L(1 -2α(N)), obtained through an apparently new quantitative application of Hall's Marriage Theorem [START_REF] Hall | On representatives of subsets[END_REF] to bi-stochastic matrices.

Definition 1 (Bi-stochastic matrix). An N × N matrix B is called bistochastic if all its entries are non-negative, and the sum of the entries over each row as well as the sum of the entries over each column is one. Lemma 2. Let B be an N × N bi-stochastic matrix: there exists a permutation matrix P such that all the entries of B on the support of P exceed the threshold

2α(N) = 4 N (N +2) if N is even, 4 (N +1) 2 if N is odd. ( 16
)
Corollary 1. Let σ 0 , . . . , σ L-1 ∈ S N be L permutations. There exists a global permutation π such that

C jπ(j) = ♯{ℓ : σ ℓ (j) = π(j)} ≥ 2Lα(N), ∀1 ≤ j ≤ N. ( 17
)

Proof of Theorem 2

Using the above tools we can now prove Theorem 2. By Corollary 1, there is a permutation π such that for each j, we have

F(ã ij -a iπ(j) ) 0 ≤ ♯{ℓ : σ ℓ (j) = π(j)} ≤ L(1 -2α(N)), (18)
hence ∆( Ã, A|π) ≤ L(1 -2α(N)) and finally ∆( Ã, A) ≤ L(1 -2α(N)).

Combined with the assumption k ≤ Lα(N), we obtain 2k + ∆ ≤ L, and we conclude thanks to Lemma 1.

Discussion

The reader may have noticed that Theorem 2, while dropping the disjoint support assumption from Theorem 1, introduces new restrictions: the assumption that L is prime, and the restriction to p = 0 compared to 0 ≤ p ≤ 1 in Theorem 1. How important are these restrictions ? Could they be relaxed while exploiting sparsity together with the disjoint support assumption ? This is discussed in this section.

Extending Theorem 2 to non-prime filter length L?

As indicated by Proposition 1 below, for even filter length L ≥ 4, there exists sparse matrices of filters that are the sparsest but not unique (even up to a global permutation) solution of the considered problem: certain frequency permutations provide an equally sparse but not equivalent solution.

Proposition 1. For any integer k such that 2k divides L, there exists a matrix of k-sparse filters A and a set of L/2k frequency permutations resulting in à ≡ A such that for all 0 ≤ p ≤ ∞:

ãij p = a ij p , ∀i, j. ( 19
)
We have 2k

• ∆( Ã, A) = L.
The fact that the filter matrices A and à satisfy 2k • ∆( Ã, A) = L shows the sharpness of Lemma 1 for the case when L is even: the strict inequality in ( 15) cannot be improved.

Specializing Proposition 1 to k = 1 for even filter length L ≥ 4 yields ideally 1-sparse filters a ij and a set of L/2 frequency permutations such that: ãij are 1-sparse; the estimated filter à is not equivalent to the original A and cannot be discriminated from it by any ℓ p quasi-norm.

Stronger guarantees with disjoint supports and sparsity ?

Could one get improved results by combining the disjoint support assumptions from Theorem 1 and the sparsity assumption from Theorem 2 ? For even filter length L ≥ 4, Proposition 2 below indicates the existence of sparse matrices of filters with disjoint supports that are the sparsest but not unique (even up to a global permutation) solution of the considered problem: certain frequency permutations of "size" ∆ = L/2k provide an equally good but not equivalent solution.

Proposition 2. For any integers k ′ < k ≤ L/2 such that 2k ′ divides L, there exists a matrix of k-sparse filters A with disjoint supports [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF], and a set of L/2k ′ frequency permutations resulting in à ≡ A, such that for all 0 ≤ p ≤ ∞: à p = A p and ãij p = a ij p , ∀i, j.

(

) 20
We have

2k ′ • ∆( Ã, A) = L.
Specializing Proposition 2 to k ′ = 1 and k = 2 for even L ≥ 4 yields 2-sparse filters a ij and a set of L/2 frequency permutations such that: ãij are 2-sparse; Ã is not equivalent to A and cannot be discriminated from it by any ℓ p quasi-norm.

This shows that even by adding the disjoint support assumption, for even L ≥ 4, there is little margin to improve Lemma 1: at best, one can hope to replace the strict inequality in [START_REF] Tao | An uncertainty principle for cyclic groups of prime order[END_REF] with a large one. Can this actually be done? This is partially answered by the following results:

Proposition 3. Assume that à ≡ A, that 2k • ∆( Ã, A) = L ( 21
)
with k := max ij a ij 0 and that the filters in A have disjoint supports [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF].

Then, either à 0 > A 0 , or each row of à is obtained by permuting pairs of distinct filters a ij , a ij ′ from the corresponding row of A such that a ija ij ′ is proportional to a modulated and translated Dirac comb with 2k spikes.

For filter matrices with a single row, since à ≡ A means that the filters ã1j are permuted versions of a 1j , we obtain Corollary 2. Consider A with a single row (M = 1). Assume that

à ≡ A, that 2k • ∆( Ã, A) = L ( 22
)
with k := max ij a ij 0 and that the filters in A have disjoint supports [START_REF] Kowalski | Beyond the narrowband approximation: Wideband convex methods for under-determined reverberant audio source separation[END_REF].

Then à 0 > A 0 .

Excessive pessimism?

The counter-examples built in Propositions 1-2, which are associated to Dirac combs, are highly structured. They provide worst case well-posedness bounds, but existing probabilistic versions of uncertainty principles (see, e.g., the nice survey [START_REF] Tropp | On the linear independence of spikes and sines[END_REF]) lead us to conjecture that if the sparse filters in A are drawn at random (e.g. from Bernoulli-Gaussian distribution), the uniqueness guarantee of Theorem 2 will hold except with small probability O(L -β ), provided that k < c(β)L/ log L, for large L. This is left to further theoretical investigation.

Numerical experiments

The results achieved so far are theoretical well-posedness guarantee, but do not quite provide algorithms to compute the potentially unique (up to global permutation) solution of the frequency permutation problem. We conclude this paper with the description of a relatively naive optimization algorithm, an empirical assessment of its performance with Monte-Carlo simulations, and a discussion of how this compares with the theoretical uniqueness guarantees achieved above.

Proposed combinatorial algorithm

Given a "permuted" matrix Ã, one wishes to find a set of frequency permutations yielding a new matrix A with minimum ℓ p norm.

The proposed algorithm starts from A 0 = Ã. Given A n , a candidate matrix A n+1,π can be obtained by applying a permutation π at frequency ω n ≡ n [mod L]. Testing each possible permutation π and retaining the one π n which minimizes A n+1,π p yields the next iterate A n+1 := A n+1,πn . The procedure is repeated until the ℓ p quasi-norm A n ceases to change. Since there is a finite number of permutations to try, the stopping criterion is met after sufficiently many iterations.

Choice of the ℓ p criterion

In theory, it could happen that the stopping criterion is only met after a combinatorially large number of iterations. However, the algorithm stops much sooner in practice. In fact, if we were to use the ℓ 0 "norm" , the algorithm would typically stop after just one iteration, because the ℓ 0 "norm" attains its maximum value M × N × L for most frequency permutations except a few very special ones. For this reason, we chose to test the algorithm using ℓ p quasi-norms p > 0, which are not as "locally constant" as the ℓ 0 "norm" . To our surprise, the experiments below will show that the best performance is not achieved for small p, but rather for p = 2ǫ with small ǫ > 0. For p = 0 and for p ≥ 2, the algorithm indeed completely fails.

Monte-Carlo simulations

For various filter lengths L, sparsity levels k and dimensions M, N, random sparse filter matrices A made of independent random k-sparse filters were generated. Each filter was drawn by choosing: a) a support of size k uniformly at random; b) i.i.d. Gaussian coefficients on this support.

For each configuration (L, k, M, N), 200 such random matrices A were drawn. For each A, independent random frequency permutations were applied to obtain Ã. The algorithm was then applied to obtain A. The performance was measured using the SNR between A and the best permutation of A.

Figure 1 shows the histogram of SNR values achieved for L = 31, 1 ≤ k ≤ L, M ∈ {1, 2}, N ∈ {2, 3, 4}, p = 1. It shows that the algorithm either completely succeeds up to machine precision (SNR above 300 dB) or completely fails (SNR of the order of 0 dB). For this reason, in the rest of the 2) associated with the well-posedness guarantee (using an ℓ 0 criterion) of Theorem 2. Surprisingly, one can observe that the success rate increases when 0 < p < 2 is increased. The maximum success rate is achieved when p = 2ǫ with small ǫ > 0.

Beyond the well-posedness regime suggested by the theory (i.e., to the right of the vertical dashed line) the algorithm can succeed, but at a rate that rapidly decreases when the relative sparsity k/L increases. In the regime where the problem is proved to be well-posed, the proposed algorithm is often successful but can still fail to perfectly recover the filters, especially -and surprisingly-for small values of k. This phenomenon is strongly marked for p < 1 and essentially disappears for p > 1.

It remains an open question to determine the respective roles of the ℓ p criterion and of the naive greedy optimization algorithm in this limited performance when the problem is wellposed with respect to the ℓ 0 "norm" .

Role of the filter length L

Figure 3 shows the results for different L values with p = 1.9, M = N = 2. One can see that the average performance does not seem to depend on whether L is prime or not. As L increases, the performance for "small" k/L slightly increases, but the success rate degrades for "large" k/L close to α(2).

Role of the number of channels M

Figure 4 shows the results for increasing numbers of channels M, with a filter length L = 512, N = 2 sources, p = 1.9. One can observe that the success rate substantially increases when M is increased from M = 1 to M = 2, and slightly increases as M further increases. Although the worstcase well-posedness guarantees are the same, the algorithm seems to benefit from added filter diversity across channels.

Role of the number of sources N

Figure 5 shows the success rate as a function of the relative sparsity k/L, for N ∈ {2, 3, 4}, with L = 31, M = 5 with p = 1.9. The wellposedness limits k/L ≤ α(N) associated to Theorem 2 are indicated with vertical dashed lines. The empirical curves confirm that the algorithm can still succeeed beyond the worst-case well-posedness guarantees, but with a rapidly decreasing rate of success. When the well-posedness guarantees hold, the algorithm can fail, but its rate of success is high when the relative sparsity is sufficiently small compared to the bound provided by Theorem 2.

Computation time

The algorithm evaluates the ℓ p quasi-norm of the N! permutations of the sources for each of the L frequencies.

To evaluate the ℓ p quasi-norm of the filters, the permuted frequency coefficients have to be transformed back into the time domain by inverse Discrete Fourier transform. For each filter, the cost of the Discrete Fourier Transform through a Fast Fourier Transform is O(L log 2 L). There are MN filters and hence the cost of ℓ p quasi-norm evaluation for a given configuration of subbands is O(MNL log 2 L).

Hence, the complexity of each sweep through the set of all frequencies is O(N!MNL 2 log 2 L). This is rather expensive because the computational cost grows in factorial with the number of sources and in square with the filter length, but it is tractable for small problem sizes and very efficient compared to the brute force approach that would require O((N!) L-1 MNL log 2 L) operations to test all (N!) L-1 possible permutations up to a global permutation.

Figure 6 shows the average computation time over 200 trials for various filter length. The red dashed line corresponds to its prediction using the theoretical cost estimation as C × L 2 log 2 L with C ≈ 40 nanoseconds.

Conclusions

It is now well known that a sufficient sparsity assumption can be used to guarantee the well-posedness of under-determined linear inverse problems: without the sparsity assumption, the problem admits an affine set of solutions, which intersects at only one point with the set of sparse vectors. Besides this well-posedness property, a key factor that has lead to the large deployment of sparse models and methods in various fields of science is the fact that a convex relaxation of the NP-hard ℓ 0 minimization problem can be guaranteed to find this unique solution under certain sparsity assumptions. The availability of efficient convex solvers then really makes the problem tractable.

The problem considered in this paper is not a linear inverse problem. Even though it is a simplification of the original permutation and scaling problem arising from signal processing, it remains a priori a much harder problem than linear inverse problems in terms of the structure of the solution set: each solution comes with a herd of solutions that are equivalent up to a global permutation.

The fact that we managed to obtain well-posedness results in this context is encouraging, but this is at best the beginning of the story: without a solution to the scaling problem, this result remains theoretical. Can one hope to extend these results to the original permutation and scaling problem? Answers to these questions are likely to have an impact in fields such as blind source separation with sparse multipath channels.

A. Proof of Lemma 2 and Corollary 1

A weakened version of Lemma 2, with 2α ′′ (N) = 1 1+(N -1) 2 , can be obtained by combining the Birckhoff -Von Neumann theorem and Carathéodory theorem [START_REF] Benichoux | Well-posedness of the permutation problem in sparse filter estimation with lp minimization[END_REF]. Yet, this bound is suboptimal. The optimal bound in Lemma 2 follows from Hall's Marriage Theorem, which by the way is also a key ingredient in the proof of the Birkhoff-Von Neumann theorem.

Theorem 3 (Hall's Marriage Theorem [START_REF] Hall | On representatives of subsets[END_REF][START_REF] Oxley | Matroid theory[END_REF] ). Let (A j ) j∈J be a family of subsets of a set finite S. There exists a bijection π : J → S such that π(j) ∈ A j for all J if, and only if, for all

E ⊂ J ♯ ∪ j∈E A j ≥ ♯E. (23)
The bijection π is often referred to as a transversal for S.

Proof (Lemma 2) For brevity of notation we write α for α(N). B being the considered bi-stochastic matrix, we wish to exhibit a permutation matrix P such that B jn ≥ 2α for jn on the support of P. Define the sets J = S = 1, N , and A j := {n : B jn ≥ 2α}, j ∈ J, and consider the property

P k : ∀E ⊂ J, ♯E ≤ k ⇒ ♯ ∪ j∈E A j ≥ ♯E. (24)
We wish to prove that P k holds true for all 1 ≤ k ≤ N: then, by Hall's Marriage Theorem, there exists a bijection π : j → π(j) such that π(j) ∈ A j for all j, yielding in turn the permutation matrix P with ones at the entries (j, π(j)). We proceed by contradiction: assume that P N does not hold true. Since P 1 holds true, without loss of generality, for some 1 ≤ k 0 < N:

♯ ∪ 1≤j≤k 0 A j ≥ k 0 , and ♯ ∪ 1≤k≤k 0 +1 A j ≤ k 0 . (25)
Hence, without loss of generality:

∪ 1≤k≤k 0 A j = 1 k 0 ⊃ A k 0 +1 . (26)
It follows that for n > k 0 and j ≤ k 0 + 1, we have n / ∈ A j , hence B jn < 2α. Now we use the bi-stochasticity of B, j B jn = n B jn = 1, B jn ≥ 0, to obtain

k 0 ≥ n≤k 0 j≤k 0 +1 B jn = j≤k 0 +1 n≤k 0 B jn = j≤k 0 +1 1 - n>k 0 B jn > j≤k 0 +1 (1 -(N -k 0 )2α) = (k 0 + 1)(1 -(N -k 0 )2α) = k 0 + 1 -(k 0 + 1)(N -k 0 )2α . (27)
This implies 2α > 1/(k 0 + 1)(Nk 0 ). However, this yields a contradiction, since a simple functional study shows that

max 1≤k 0 <N 1 (k 0 + 1)(N -k 0 ) = 2α. (28)
Equipped with Lemma 2, we can now prove Corollary 1.

Proof (Corollary 1)

We consider the matching count matrix C with entries

C jn := ♯{0 ≤ ℓ < L : σ ℓ (j) = n}, 1 ≤ j, n ≤ N (29)
where σ ℓ , 0 ≤ ℓ < L are the considered permutations. Since j C jn = n C jn = L we have C = L • B where B is bi-stochastic, hence there is a permutation π such that C jπ(j) ≥ 2Lα(N).

We conclude this section by showing the sharpness of Corollary 1 through the construction of permutations that reach the bound. Consider N an integer, and k 0 := N/2 (N even) or k 0 := (N -1)/2 (N odd). Let L be a multiple of (k 0 + 1)(Nk 0 ). Consider the L × N matrix:

Σ :=         1 U k 0 . . . 2 × . . . × 2
. . . 2 1 × . . . ×         (30)
where: a) the left part, of size L × (k 0 + 1), is filled with the column vectors i ∈ R L/(k 0 +1) made of constant entries equal to the integer 1 ≤ i ≤ k 0 and the vector U ∈ R L/(k 0 +1) made of the vertical concatenation of the N -k 0 column vectors j ∈ R L/(k 0 +1)(N -k 0 ) with constant entries k 0 + 1 ≤ j ≤ N; b) the rows of the the right part, of size L×(N -k 0 -1), include exactly once each integer 1 ≤ ℓ ≤ N which does not already appear in the corresponding row of the left part. By construction, the L rows of the matrix Σ are associated to L permutations σ ℓ . We now show that, for any global permutation π, there is at least one column 1 ≤ j ≤ k 0 + 1 such that

♯{ℓ : σ ℓ (j) = π(j)} ≤ L/(k 0 + 1)(N -k 0 ) = L2α(N). (31)
Applying again the pigeonhole principle yields: among the k 0 + 1 indices j to consider, at least one, j ⋆ , must be mapped to an integer π(j ⋆ ) ≥ k 0 + 1. By construction, the columns of Σ are such that column j ⋆ contains at most (in fact: exactly) L/(k 0 + 1)(Nk 0 ) instances of the value π(j ⋆ ).

B. Proof of Lemma 1 and Propositions 1 2 3

We prove Lemma 1 first, then the statements of Proposition 1 in the following order: 1), 3), 2). We begin by some notations and fact regarding Dirac combs.

B.1. Dirac combs

Let p, q ≥ 1 be two integers and L = pq their product. The unit Dirac comb with q spikes and of step p, denoted x p , is the vector of C L defined by

x q [t] = 1/ √ q if t ≡ 0[p],
x q [t] = 0 otherwise. Its Fourier transform is the unit Dirac comb with p spikes and of step q: Fx q = x p . For 0 ≤ n < p an integer translation index and 0 ≤ m < q an integer modulation index, one can define the translated and modulated Dirac comb x q,n,m = T n M m x q where T n is the circular shift by n samples, and M m is the frequency modulation (M m u)[t] := u[t] • e 2iπmt/L . One can check that the collection {x q,n,m } 0≤n<p,0≤m<q is an orthonormal basis of C L .

B.2. Proof of Lemma 1

Let π 0 be the permutation such that ∆ 0 (A, Ã) = min π∈S N ∆ 0 (A, Ã|π). By abuse of notation we still denote A the matrix obtained by applying π 0 to permute the columns of the original filter matrix. For each channel i and instead of 2k < L/∆ 0 we repeat the steps (32)-( 35) to obtain ãij 0 ≥ a ij ′ 0 for any j ∈ E i := {j, a ij = ãij } and any j ′ . As a result ãij 0 ≥ a ij 0 for all i, j. The assumption that à 0 = A 0 implies that ãij 0 = a ij 0 for all i, j.

By assumption, Ã ≡ A hence there are indices i, j such that a ij = ãij . For such i, j, since a ij 0 = ãij 0 , each inequality in (32)-(35) (the inequality L/∆ 0 > 2k being replaced with L/∆ 1 ≥ 2k) must be indeed an equality. This implies that: a ij 0 = ãij 0 = k; 2k divides L and ∆ 1 = L/2k; the nonzero vector b ij := ãija ij must be an equality case of the ℓ 0 uncertainty principle with b ij 0 = 2k and Fb ij 0 = L/2k. As a result [START_REF] Tao | An uncertainty principle for cyclic groups of prime order[END_REF] b ij is a scaled, modulated and translated version of the Dirac comb x 2k made of 2k Diracs spaced every L/2k samples: there exists a scalar γ ij = 0, and two integers 0 ≤ n ij < L/2k, 0 ≤ m ij < 2k such that

b ij = γ ij • x 2k,n ij ,m ij . (41
γ ij = 0, (43)
Since γ ij = 0, whenever E i,n,m is not empty it contains at least two distinct indices.

By the disjoint support assumption: for j, j ′ ∈ E i,n,m , j = j ′ , the original filters a ij and a ij ′ have disjoint supports. Moreover, we know that these supports are subsets of the support of x 2k,n,m which is of size 2k, hence

♯E i,n,m • k = j∈E i,n,m a ij 0 ≤ 2k. (44)
Hence, whenever E i,n,m is not empty, it contains exactly two distinct elements: E i,n,m = {j, j ′ } where j = j ′ .

Further, observe that: a) a ij and a ij ′ have disjoint supports of size k which are subsets of the support of size 2k of x 2k,n,m ; b) a ij and ãij have the same property. As a result, ãij and a ij ′ have the same support, which is disjoint from that of a ij . Similarly, a ij has the same support as ãij ′ . Finally, Eq. ( 43) can be rewritten γ ij + γ ij ′ = 0, and implies b ij + b ij ′ = 0, that is to say ãij + ãij ′ = a ij ′ + a ij . We conclude that ãij = a ij ′ and ãij ′ = a ij .

Figure 1 :

 1 Figure 1: Histogram of SNR between best permutation of A and original A

Figure 2 :

 2 Figure 2: Filter recovery success as a function of p, 0 ≤ p ≤ 1.9

Figure 3 :

 3 Figure 3: Filter recovery success as a function of L, for p = 1.9

Figure 4 :

 4 Figure 4: Filter recovery success as a function of M , for p = 1.9

Figure 5 :

 5 Figure 5: Filter recovery success as a function of N , for p = 1.9

Figure 6 :

 6 Figure 6: Computation time of the permutation solving algorithm depending on the length L of the filter

)

  Moreover since a ij 0 = ãij 0 = k and ãija ij 0 = 2k, the filters ãij and a ij have disjoint supports of size k. Hence, they are the restriction of b ij (resp. of -b ij ) to their respective supports. Now, defineE i,n,m := {j ∈ E i , n ij = n, m ij = m}. (42)As observed in the proof of Theorem 1, the equality j a ij = j ãij holds, implying j∈E i b ij = j b ij = 0. Taking inner products with the Dirac comb orthonormal basis x 2k,n,m , 0 ≤ n < L/2k, 0 ≤ m < 2k, yields j∈E i,n,m

Acknowledgment

This work was supported by the EU Framework 7 FET-Open project FP7-ICT-225913-SMALL: Sparse Models, Algorithms and Learning for Large-Scale data, and by Agence Nationale de la Recherche (ANR), project ECHANGE (ANR-08-EMER-006).

a source index j such that a ij = ãij we obviously have a ij 0 ≤ ãij 0 . Now, since ∆ 0 ≥ 1 we have à ≡ A hence there exists a pair i, j such that ãij = a ij . By the ℓ 0 Dirac-Fourier uncertainty principle [14, Theorem 1], for any vector u ∈ C L we have u 0 Fu 0 ≥ L. Hence, by the hypothesis k < L/(2∆ 0 ) we have

where j ′ is an arbitrary source index. Hence for every i, j such that ãij = a ij and any j ′ , ãij 0 > a ij ′ 0 , and we obtain

Overall, we have shown that à 0 > A 0 . When L is prime, a stronger uncertainty principle u 0 + Fu 0 ≥ L + 1 holds [START_REF] Tao | An uncertainty principle for cyclic groups of prime order[END_REF] as an immediate consequence of Chebotarev's theorem on roots of unity.

Theorem 4 (Chebotarev). Let L be a prime and 1 ≤ n ≤ L. Let x 1 , . . . , x n be distinct elements of Z/LZ, and let ω 1 , . . . , ω n also be distinct elements of Z/LZ. Then the matrix (exp(2iπx j ω k /L)) 1≤j,k≤n has non-zero determinant.

Hence, under the assumption 2k + ∆ 0 ≤ L we can replace (33)-(34) with

to reach the same conclusion.

B.3. Proof of Proposition 1

We shall simply build an example where A = [α, β] is a 1 × 2 matrix of filters. Extensions to A an M × N matrix are trivial by adding mutually distinct sparse columns that are distinct from α and β, and duplicating the first row.

We exploit Dirac combs as described in Appendix B.

we obtain that, after permuting the Fourier transforms of α and β at the frequencies ω = 2kr, 0 ≤ r < L/2k,

We choose the vector u to be zero everywhere with two exceptions u

Lastly, all considered vectors have k entries of equal magnitude, hence α 0 = β 0 = α 0 = β 0 = k, and for any 0 < p ≤ ∞ α p = β p = α p = β p . In particular, Ã p = A p , 0 ≤ p ≤ ∞.

B.4. Proof of Proposition 3

We repeat the construction of the proof of Proposition 1 starting from the Dirac combs a = x k ′ ,0,0 , b = -x k ′ ,L/2k ′ ,0 . Since k ′ < k ≤ L/2, we have ℓ := kk ′ ≤ L/2k ′ hence we can choose an ℓ-sparse vector u which support is outside the support of x 2k ′ and such that T L/2 u and u have disjoint supports. The four vectors {a, b, u, T L/2 u} have mutually disjoint supports, hence α and β have disjoint supports, {α, β} = { α, β} and à ≡ A. Moreover, ∆ 0 ( Ã, A) = ∆ 1 ( Ã, A) = L/2k. Lastly, we have α 0 = β 0 = α 0 = β 0 = k ′ + ℓ = k, and the ℓ p norms of these vectors are also equal, hence à p = A p , 0 ≤ p ≤ ∞.

B.5. Proof of Proposition 2

As in the proof of Lemma 1 we consider A the permuted matrix associated to the optimal permutation π 0 . Using the inequality 2k ≤ L/∆ 1 ≤ L/2∆ 0


Morgan Brehinier
email: morgan.brehinier@inria.fr

Guillaume Gravier
email: guillaume.gravier@irisa.fr

Texmix: An automatically generated news navigation portal

Keywords: HTML5, hypervideo, topic segmentation, speech to text, named entities, similar images, dynamic resume, geotagging

The Texmix demonstration presents an original interface to navigate a collection of broadcast news shows, exploiting speech transcription, natural language processing and image retrieval techniques. Navigation is performed through keywords search or through time or through maps, with links automatically created either between reports to follow the story or to the Web to know more about a story, a person or a fact. Image search technology is also integrated to find portions of the collection with similar images. We also present two original features to dynamically access videos, namely dynamic summary and geotagging . . .

Introduction

The gradual migration of television from broadcast diffusion to Internet diffusion offers tremendous possibilities for the generation of rich navigable contents. However, it also raises numerous scientific issues regarding delinearization of TV streams and content enrichment. In this demonstration, we illustrate how speech in TV news shows can be exploited for delinearization of the TV stream. In this context, delinearization consists in automatically converting a collection of video files extracted from the TV stream into a navigable portal on the Internet where users can directly access specific stories or follow their evolution in an intuitive manner. The TEXMIX demonstration illustratess the result of the delinearization processand enables user to experience various navigation modes within a collection of news shows. An illustration of the interface is provided Figure 1. The demonstration can be accessed online at http://texmix.irisa.fr.

Structuring a collection of news shows requires some level of semantic understanding of the content in order to segment shows into their successive stories and to create links between stories in the collection, or between stories and related resources on the Web. Spoken material embedded in videos, accessible by means of automatic speech recognition, is a key feature to semantic description of video contents. At IRISA/INRIA Rennes, we have developped multimedia content analysis technology combining automatic speech recognition, natural language processing and information retrieval to automatically create a fully navigable news portal from a collection of video files [START_REF] Gravier | Exploiting speech for automatic TV delinearization: From streams to cross-media semantic navigation[END_REF].

The purpose of this demonstration, aside from exploiting technologies developed in our research group, is to provide a new experience from watching videos on the Internet. Exploiting HMTL5 features, dynamic content is provided as the video plays, and allow the watcher to have additional information and to interact during his watching, being no longer passive. We also provide new search services based on the technologies developed.

In this summary, we describe the various features of the demonstration, providing brief insight on the underlying technology.

Timeline navigation

An important feature to access and navigate a collection of news bulletin is time. A timeline is used to locate in time all information that you can access. It is also used to locate in time information that are displayed by positioning the cursor accordingly.

News shows are made of successive items which are usually introduced by the anchor speaker and developed in some story. The first step to structuring a collection therefore consists in segmenting each show into its constituting items. This segmentation step is done using statistical topic segmentation method which detect topic changes in the transcription, inspecting the distribution of words. Topic segmentation enables to automatically generate a table of contents of each show. Futhermore, keywords are automatically extracted from the transcript to characterize each story. Exploiting the keywords and the transcripts, links to the Web and to related stories in the collection are created and displayed, thus offering navigation capabilities to follow the evolution of an event or to know more about it.

Named entities

Named entities, that is proper names and location names, are also automatically detected using machine learning algorithms and used to locate news items and to dynamically provide additionnal information while playing the video. For instance, a Wikipedia pop-up is displayed whenever a person's name is uttered, with a link to the corresponding wikipedia page for more information. Similarly, locations are dynamically displayed on a Google Map as they occur, thus facilitating the localization of news stories.

Similar images

Spoken language provides crucial information but so do images! Indeed, the same image is often used to illustrate a story throughout time. Based on our image retrieval technology [START_REF] Jegou | Product quantization for nearest neighbor search[END_REF], we included a 'similar image' search feature in the news portal, as illustrated Figure 2. In this second illustration, we search for images similar to a graphic depicting the voting intentions for the 2007 French presidential election. This results in similar graphics at different dates, making it possible to follow the evolution of the poll in time.

User services

Based on the previous technologies, we developed two services for the user.

Dynamic summary

Based on the keywords extraction technology, the dynamic summary offers the possibility to catch-up news on a specific topic in a very quick way. The user starts by selecting the period of time he is interested in, and then selects a keyword related to this period. Those two actions result in a dynamic resume of the topic, by sequencing the thirty first seconds of each story related to the keyword.

Geotagging

In the same way 'dynamic resume' offered a navigation based on time and keywords, geotagging is a way to look for news stories related to their location. For this feature, the user starts by selecting the time period he is interested in, and markers are then displayed on a Google Map, showing where the news stories of this time period occured. The user can then zoom in or click on a marker in order to view the news bulletin related to the place he chose.

Technology

This interface has been developed in HTML5, CSS and Javascript. Those technologies allowed us to develop this multimedia portal without relying on proprietary software solutions. The synchronization of the contents and the videos is based on the popcorn.js library (www.popcornjs.org). Therefore, this demonstration is accessible on the Internet with any recent web-browser supporting HTML5 technologies.

Conclusion

This unique combination of speech and language technology and mulimedia retrieval offers countless opportunities to automatically generate Web content from TV streams, opening the door to a fully connected multimedia Web.

Fig. 1 .

 1 Fig. 1. Overview of the navigation interface

Fig. 2 .

 2 Fig. 2. Results of a search for similar images

Fig. 3 .

 3 Fig. 3. Geotagging interface

Acknowledgements

This work was partially funded by the Quaero project and by EIT ICT Labs via the OpenSEM project.