What happens when Phil throws Archie in to a pit? Yes No What happens after Danielle reveals to Archie that Ronnie is her mother?

Zero-Shot Classification of Events for Character-Centric Video Summarization ALISON REBOUD, ISMAIL HARRANDO, PASQUALE LISENA, and RAPHAËL TRONCY, EURECOM, France This paper describes an event classification and character-centered approach proposed by the D2KLab team at EURECOM for the 2021 TRECVID Video Summarization Task [START_REF] Awad | TRECVID 2020: comprehensive campaign for evaluating video retrieval tasks across multiple application domains[END_REF]]. Our approach relies on defining a list of typical important events in a soap opera and using this list of named events as candidate labels for a zero-shot text classification method. This additional data source is used together with the provided videos, scripts and master shot boundaries. We also use BBC EastEnders characters' images crawled from the Google search engine in order to train a face recognition system. All our runs use the same general method, but with varying constraints regarding the number of shots and the maximum duration of the summary. The runs submitted are as follows:

• EURECOM1: 5 shots with highest similarity scores and the total duration of the summary is < 150 sec; • EURECOM2: 10 shots with highest similarity scores and the total duration of the summary is < 300 sec; • EURECOM3: 15 shots with highest similarity scores and the total duration of the summary is < 450 sec; • EURECOM4: 20 shots with highest similarity scores and the total duration of the summary is < 600 sec.

INTRODUCTION

Considering video summarization as an important task for digital content retrieval and reuse, the TRECVID Video Summarization Task (VSUM) 2021 aims at fostering the research in this field by asking its participants to automatically summarize "the major life events of specific characters over a number of weeks of programming on the BBC EastEnders TV series" 1 [START_REF] Awad | TRECVID 2020: comprehensive campaign for evaluating video retrieval tasks across multiple application domains[END_REF]]. More precisely, for three different characters of the series, the participants have to submit 4 summaries with respectively 5, 10, 15 and 20 automatically selected shots. These generated summaries are evaluated by the assessors according to their tempo, contextuality and redundancy as well as with regards to how well they contain answers to a set of questions unknown to the participants before submission. In addition to the videos, the episodes transcripts are provided by the organizers. In 2020, we have addressed the VSUM task by matching 1 https://www-nlpir.nist.gov/projects/tv2021/vsum.html Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. fan-written synopsis to transcripts using as hypothesis that each paragraph mentioned in these synopsis correspond to important moments to include in a summary [START_REF] Harrando | Using Fan-Made Content, Subtitles and Face Recognition for Character-Centric Video Summarization[END_REF]. However, such synopsis are not always available. This year, we propose a new approach based on zero-shot classification of named events.

APPROACH FOR THE MAIN TASK

Face Recognition

The dataset considered for the task consists of 10 video episodes which amount to approximately 19 000 shots. The summarization task aims to produce shorter videos of 5 to 20 shots (which is respectively 0.02% and 0.10% of the original episode duration).

The compression rate being high, we discard all the scenes where the character of interest in not present in the scene. In order to do so, we extract and recognize faces using the Face Celebrity Recognition library [START_REF] Lisena | FaceRec: An interactive framework for face recognition in video archives[END_REF]], a method which uses images gathered from crawling the web with the character's name as the keyword query. We also added the phrase "EastEnders" to the names to avoid including images of people with the same name. The faces are first detected with an MTCNN. Each detected face then gets associated with a FaceNet embedding. We empirically define a threshold of standard deviation 0.24 for cosine similarity under which we consider that the faces are outliers and we eliminate them. Finally, a multi-class SVM classifier outputs the final prediction.

We also align the provided XML transcripts with the given shot segmentation. If a sentence encompasses multiple shots, we select all the shots as we expect a good summary to avoid including scenes with cut utterances. However, this increases the noise of our summaries and diminishes the number of distinct moments. We believe this constraint is a limitation of the shot segmentation and that a scene segmentation would be more relevant to the task.

Shot Transcript Classification

The instructions for VSUM state that the method developed for the task should be able to differentiate between meaningful and trivial events, choosing for example 'the birth of a child rather than a short illness'. Therefore, we tackle this task by trying to define what could be such events, hypothesising that soap opera episodes are repetitive enough that the type of major events of an episode can be defined in advance, without having watched the series. We use the results of a research work which investigates if soap opera viewers' perceptions of the likeliness of some life events differ from the nonviewers [START_REF] Seese | Soap opera viewers' perceptions of the real world[END_REF]]. In this work, the authors defined events which they thought often happen in soap operas (Table 1). We construct our model with the hypothesis that the least likely events are also the most interesting ones and should probably be included in the summary. For instance, if the scene contains a 'suicide attempt', it should be more interesting than a 'happily married' scene. For that reason, we take the inverse of the perceived likelihood (on a scale from 1 to 5) of an event as its weight (Table 1). We do not assume the evaluation team to be specifically composed of soap opera viewers and hence select the likelihood scores reported for the non-viewers group. The weight of the event gets further multiplied by the confidence score obtained from the zero-shot classifier (which was normalized for each class with RobustScaler2 ). Finally, because we wish to extract informative scenes which should therefore be long enough, the score per scene gets further multiplied by the log of the length of the shot dialogue (Equation 1).

( 1)

𝑠𝑐𝑜𝑟𝑒(𝑠ℎ𝑜𝑡_𝑖) = 𝑚𝑎𝑥_𝑙 ∈ 𝑙𝑎𝑏𝑒𝑙𝑠(𝑧𝑠𝑐(𝑡𝑟𝑎𝑛𝑠_𝑖, 𝑙) * 𝑤𝑒𝑖𝑔ℎ𝑡(𝑙) * 𝑙𝑜𝑔(𝑙𝑒𝑛(𝑡𝑟𝑎𝑛𝑠_𝑖))
where 𝑠ℎ𝑜𝑡 𝑖 is the unique id of the shot, 𝑡𝑟𝑎𝑛𝑠 𝑖 is its corresponding transcript, 𝑙𝑎𝑏𝑒𝑙𝑠 is the list of events, with their importance expressed with 𝑤𝑒𝑖𝑔ℎ𝑡().

Finally, to get one score per shot (and not per candidate event label), we select the max score on all event labels. To generate the submissions, we keep the N shots with the highest score. To respect the summary length requirement, in case the generated summary is too long, we un-select the longest scene from the top N and replace it with the N+1th one, recursively until the summary length constraint is met.

Table 1. Life events labels and their perceived likelihood (scale from 1 to 5) according to [START_REF] Seese | Soap opera viewers' perceptions of the real world[END_REF]

APPROACH FOR THE QUERIES SUBTASK

The goal of the substask is similar to the main one, except that the queries used for the evaluation by the task organisers are revealed for the subtask (after submission to the main task). Our approach considers this task to be similar to a Question-Answering task where the goal is to predict where the answer to the question lies in the text.

We use HuggingFace's Transformer QA pipeline (using longformer as a base model, pretrained on Squad-v2 QA task, or longformer-squadv2) to score each line in the script as a potential answer to the question for each character. We then rerank the top 10 answers using Sentence-BERT (paraphrase-mpnet-base-v2), scoring each by cosine similarity to the question. This tends to push answers that are more similar to the question to the top run. To avoid having long runs, we drop scenes that are too long. These scenes get picked consistently because they contain a lot of words and thus are likely to match with the questions somehow. In this submission, we limit shot length to 20s.

RESULTS

Table 2 shows our and the other teams results. We ranked second for the main task and first for the substask. For both tasks, our results are close to 30% which was also the type score we obtained in 2020 [START_REF] Harrando | Using Fan-Made Content, Subtitles and Face Recognition for Character-Centric Video Summarization[END_REF]] with an approach which was relying on the provision of fan made synopsis, contrary to this year. For the substask (where queries are known), it is somehow surprising to see that non of the teams achieved results better than the score of the best team for the main task. Tables 3 and4 display respectively the characters for which we obtained the best and worst results in the main task. We obtained the best score across characters with the run 4 (37.60%). Interestingly, for this run, our event classification method allowed to answer 9 of the 16 'What' questions and zero of the remaining 9 'Who', 'Why', etc. questions. These results could indicate that events/actions are the first important facts of a summary but also suggest that our model could gain from covering other aspects such as persons and locations.

Figure 1

 1 Figure 1 illustrates our general approach for the main task composed of three main steps: transcript classification, face recognition and shot selection.

Fig. 1 .

 1 Fig. 1. Fan-driven and character centered approach

  ]

	Label	Likelihood
	extramarital affair	1.98
	get divorced	1.96
	illegitimate child	1.45
	institutionalized for emotional problem	1.43
	happily married	4.05
	serious accident	2.96
	murdered	1.81
	suicide attempt	1.26
	blackmailed	1.86
	unfaithful spouse	2.23
	sexually assaulted	2.60
	abortion	1.41

Table 2 .

 2 Overall results

	Team	Main task Subtask
	ADAPT	30.15%	17.25%
	EURECOM	29.55%	30.10%
	NII_UIT	18%	29.85%

Table 3 .

 3 Detailed results for the queries about Archie with 20 shots included in the summary

https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing. RobustScaler.html


Fabien Delaleux
email: fabien.delaleux@u-pec.fr

Olivier Riou

Vincent Guiheneuf

Pierre-Olivier Logerais

Jean-Félix Durastanti

Outil de contrôle non destructif pour la quantification du vieillissement des concentrateurs solaires

Dans cet article, nous présentons une méthode de quantification du vieillissement de miroirs verre des héliostats par leur émissivité apparente. La méthode est raccordée aux propriétés spectrales du verre. Elle présente l'avantage d'être sans contact et permet de distinguer des altérations équivalentes à 3 ans de durée de vie standard. Cet outil permet de cartographier la surface étudiée en émissivité apparente puis de convertir ces valeurs en variation de propriétés optiques dans le visible (absorbance, réflectance ou transmittance) suivant la fonction attendue.

Nomenclature

t température d'émission, °C L thermosignal, OS LWIR long wavelength IR MWIR mid wavelength IR SMIR small wavelength IR Symboles grecs ελ émissivité spectrale ε Δλ émissivité apparente Indices et exposants app apparent env environnement

Introduction

L'objectif de ce travail est de concevoir un outil de contrôle non destructif et non intrusif pour l'évaluation du vieillissement des systèmes solaires à concentration. Celui-ci impose la définition préalable des indicateurs associés puis la constitution de moyens de mesurage permettant leur quantification in situ. Cette approche permet de décider du maintien ou du remplacement des systèmes selon des critères technico-économiques. La diminution du rendement de concentration au cours du temps est étroitement liée au vieillissement des matériaux constitutifs du concentrateur solaire (verre, couches minces etc..). Ce dernier est généralement évalué en laboratoire en appliquant des protocoles de vieillissement accéléré normalisés simulant des périodes d'exploitation de l'ordre de 20 à 30 ans. Cette approche ne permet pas une évaluation du vieillissement en exploitation par des moyens non destructifs puisque nécessitant un échantillonnage. Le vieillissement naturel des matériaux entraînent des altérations notamment en surface, modifiant ainsi leur émissivité spectrale. Etant donné que l'émissivité apparente est reliée à l'émissivité spectrale, cet indicateur doit être capable de détecter les altérations de surface. La quantification de l'émissivité apparente met en oeuvre une caméra thermique commerciale. D'un usage simple et rapide, cet outil permet de cartographier la surface étudiée en émissivité apparente puis de convertir ces valeurs en variation de propriétés optiques dans le visible (absorbance, réflectance ou transmittance) suivant la fonction attendue.

Emissivité apparente et émissivité spectrale

L'émissivité apparente intervient dans l'équation radiométrique qu'utilisent implicitement tous les imageurs thermiques : ce paramètre mesure le thermosignal d'émission de l'objet rapporté à celui d'un corps noir à la même température. Nous rappelons l'expression de l'équation radiométrique, mise en oeuvre dans les systèmes infrarouges :

) ( ) ) ( 1 ( ) ( ) ( ) ( . 0 0 . env app t L t t L t t L                (1)
et sont respectivement les thermosignaux de corps noirs associés aux températures d'émission et d'environnement

. Le thermosignal fourni par la caméra est associé à la température apparente Ces thermosignaux sont rapportés à la bande spectrale de détection de la caméra. Une fois la validité du modèle de l'émetteur admise, l'expression de l'émissivité apparente devient :

) ( ) ( ) ( ) ( ) ( . 0 0 . 0 . env env app t L t L t L t L t               (2)
Son mesurage est alors possible à partir des trois températures (apparente, d'environnement, émission) et de la courbe d'étalonnage de la caméra IR. La caractérisation des systèmes est détaillée dans la référence [START_REF] Riou | A self-method for resolving the problem of apparent LWIR emissivity for quantitative thermography up to 130[END_REF]. Cet indicateur est spécifique à la technique de détection mise en oeuvre dans la caméra, de la bande spectrale de détection et de l'émissivité spectrale de la cible.

L'émissivité apparente est liée aux propriétés spectrales de l'émetteur. Chrzanowski [START_REF] Chrzanowski | Problem of determination of effective emissivity of some materials in MIR range[END_REF] propose une formulation générique permettant de raccorder les deux émissivités :

    2 1 2 1 ) ( ) , ( ) ( ) , ( ) , ( ) ( 0 0               d R T L d R T L T T (3)
ε(λ,T) désigne l'émissivité spectrale de la cible, L 0 (λ,T) est la luminance du corps noir et R(λ) est la réponse spectrale normalisée du système. Le produit de ces quantités est enfin intégré sur la bande spectrale [λ 1 -λ 2 ] du système.

Une quantification indirecte de l'émissivité apparente est donc possible moyennant la connaissance de chaque terme de la relation [START_REF] Riou | Accurate methods for single-band apparent emissivity measurement of opaque materials[END_REF]. La connexion de l'émissivité apparente et de l'émissivité spectrale obtenue par réflectométrie IR a été vérifiée expérimentalement pour différents matériaux (acier, verre, polycarbonates) dans la référence [START_REF] Riou | Accurate methods for single-band apparent emissivity measurement of opaque materials[END_REF]. De nombreux mécanismes de vieillissement naturel induisent une modification de l'émissivité spectrale suite à des mécanismes de corrosion, de recristallisation ou d'altération chimique. Les effets du vieillissement naturel peuvent être quantifiés par émissivité apparente si les dégradations produisent des variations d'émissivité spectrale dans la bande spectrale du système infrarouge utilisé. Pour illustrer les capacités de détection de telles variations, nous traitons un corps gris (émissivité spectrale de 0,5) présentant des variations d'émissivité spectrale supposées résulter de modifications structurelles de la surface produites pendant la durée de vie du système. Une forme gaussienne est utilisée pour reproduire de telles variations. Elle est infiniment différentiable et elle permet de simuler le passage d'une variation de type bande Le barycentre de l'anomalie est calculé par: [START_REF] Corwin | Temperature error in radiation thermometry caused by emissivity and reflectance measurement error[END_REF] La position de par rapport au milieu de la fenêtre de détection ( ) détermine la dépendance en température de l'émissivité apparente [START_REF] Ibos | Infrared emissivity measurement device : principle and applications[END_REF]: a. aussi longtemps que λ c < λ m , l'émissivité apparente croît avec la température; b. il est stable avec la température pour λ c =λ m et c. elle diminue avec la température lorsque λ c > λ m . Comme le barycentre de l'anomalie change au cours du processus de vieillissement accéléré, la dépendance en température de l'émissivité apparente peut varier d'un échantillon à l'autre.

               2 1 d ) ( c . rect

Caractérisation des miroirs en émissivité apparente

Nous utilisons une caméra ThermaCAM E300 pour quantifier l'émissivité apparente des miroirs. Elle met en oeuvre une matrice VOx comportant 320 × 240 détecteurs. Sa bande spectrale est [7,9 -12,7μm] [START_REF] Riou | A self-method for resolving the problem of apparent LWIR emissivity for quantitative thermography up to 130[END_REF]. L'incertitude de mesure de température apparente est de 0,1°C. L'écart d'émissivité apparente rend compte du vieillissement accéléré. Les trois miroirs présentent des valeurs quasi-constantes pour des températures d'émission comprise entre 40°C et 110°C, ce qui signifie que la dépendance en température de l'émissivité apparente est conservée pendant le processus de vieillissement accéléré. D'autre part, le processus de vieillissement accéléré est de nature différente selon l'âge initial des miroirs. Il est d'autant plus prononcé que le miroir est initialement jeune. La résolution globale δε Δλ de mesure de l'émissivité apparente est la somme quadratique des résolutions associées à chaque incertitude de température. La répétabilité dépend à la fois de la qualité de la correction de non uniformité (typiquement 0,1°C à 35°C) et de la répétabilité de la température d'émission. L'incertitude combinée dépend de la qualité de la compensation de la dérive thermique. Sans précaution particulière, l'incertitude de la compensation chez la caméra ThermaCAM E300 est typiquement de 0,3°C.

Discussion

Est-il possible d'obtenir les mêmes performances en conditions extérieures? Dans une approche non-intrusive, nous ne pouvons pas envisager une régulation de la température d'émission. Sa valeur est imposée par les conditions ambiantes. Les trois paramètres clés pour atteindre des performances similaires sont a. le contraste t app -t env ; b. l'incertitude de mesure de la température apparente δt app et c. l'uniformité de la température d'émission δt. L'utilisation en extérieur d'un système infrarouge standard avec δt app = 0,1°C et une nonuniformité δt = 0,5°C de la température d'émission (20°C) permettrait d'atteindre une résolution de 1,7% en exposant la cible à un environnement type ciel clair (t env comprise entre -30 et -20°C). La recherche de meilleures conditions de contrôle (δt = 0,25°C) améliorerait la résolution (0,9%). On aurait également avantage à thermographier des miroirs « chaud » i.e. t = 40°C ce qui permettrait d'atteindre une résolution de 0,6% soit 6 ans d'exposition standard. Une étude poussée des performances théoriques de la méthode a été réalisée [START_REF] Riou | Accurate methods for single-band apparent emissivity measurement of opaque materials[END_REF].

Conclusion et perspectives

L'émissivité apparente LWIR est un indicateur pertinent pour quantifier le vieillissement du verre. Il présente des variations de l'ordre de 1,8% au cours du vieillissement accéléré. En comparant l'écart d'émissivité apparente avec l'incertitude combinée de mesure, une approche de laboratoire est capable de discriminer sans contact un vieillissement absolu avec une résolution de 8 années standard d'exploitation. Cette approche peut également distinguer des modifications équivalentes à 3 ans standards sur le même échantillon. L'approche peut être utilisée en situation extérieure. L'utilisation d'un système infrarouge standard (δt app = 0,1°C) associée à non-uniformité δt = 0,25°C de la température d'émission (20°C), permettrait d'atteindre une résolution de 0,9% (10 ans d'exploitation standard) en exposant la cible à un environnement type ciel clair (t env comprise entre -30 et -20°C).

La mesure d'émissivité apparente est sans contact et bénéficie des techniques d'imagerie développées pour la thermographie infrarouge. La cartographie de l'émissivité apparente est possible par l'élaboration d'un code de post-traitement spécifique traitant les thermosignaux. Enfin, il est inutile de développer un système IR spécifique dans la mesure où la caméra commerciale répond aux exigences de répétabilité (qualité de de correction de nonuniformité) et d'incertitude de combinée (qualité de la compensation de dérive thermique).

Un point important est la compréhension du vieillissement accéléré. Il implique des modifications chimiques dues à l'hydratation forcée du verre. Les effets sur la réflectivité des miroirs sont faibles. Ils sont significatifs sur de grandes périodes de temps généralement supérieures à 20 ans d'exploitation. D'autres processus de vieillissement peuvent être quantifiés : c'est le cas de l'érosion. L'érosion produit une fissuration et un écaillage de la surface, rendant les surfaces poreuses au voisinage de l'impact des aérosols solides. Les dommages vus sur le verre après un test de sandstorm [START_REF] Kolli | HF etching effect on sandblasted soda-lime glass properties[END_REF] sont similaires à ceux observés sur la céramique d'alumine de porosité croissante [START_REF] Rozenbaum | Texture and Porosity Effects on the Thermal Radiative Behavior of Alumina Ceramics[END_REF]. La porosité est connue pour modifier les propriétés émissives de la matière, en particulier dans la bande LWIR [START_REF] Rozenbaum | Texture and Porosity Effects on the Thermal Radiative Behavior of Alumina Ceramics[END_REF]. Par exemple, l'émissivité spectrale d'une céramique d'alumine de porosité croissante montre des variations de l'ordre de 80 µm% dans la bande LWIR et devrait produire une variation d'émissivité apparente de l'ordre de 18%. Si un seuil de détection de 1% est considéré, elle permettrait de quantifier les effets précurseurs dus à l'érosion équivalent à 5% de la surface du réflecteur en verre correspondant à 3 années d'érosion naturelle dans les sites océaniques / désert au Maroc [START_REF] Corwin | Temperature error in radiation thermometry caused by emissivity and reflectance measurement error[END_REF].

  étroite à une anomalie de type large bande en faisant varier son écart-type . La largeur de l'anomalie est définie par tandis que son amplitude est quantifiée par (figures 1a, b et c). Le produit définit la surface typique de l'anomalie qui diffère de la surface gaussienne dans un rapport . L'émissivité apparente est calculée en utilisant la relation (3), dans laquelle la réponse spectrale est connue sur la bande normalisée LWIR [8 µm -13μm]. Le résultat est montré sur la figure 1 pour des températures d'émetteur comprises entre -20°C et 180°C. Les effets sont significatifs si nous sommes capables de quantifier des variations d'émissivité apparente de l'ordre du pourcent.

Figure 1 :

 1 Figure 1 : Connexion de l'émissivité apparente avec l'émissivité spectrale des surfaces

Figure 2 :

 2 Figure 2 : Emissivité spectrale hémisphérique initiale des miroirs verre Odeillo

Figure 3 :

 3 Figure 3 : Emissivité spectrale hémisphérique d'Odeillo #1 1 à l'état initial et après 1000h de vieillissement en chaleur humide

Figure 4 :

 4 Figure 4 : Ecart d'émissivité spectrale hémisphérique des miroirs verre Odeillo. a. Odeillo #1; b. Odeillo #2 et c. Odeillo #3. En trait plein: écart [%]; ligne horizontale pointillée: valeur moyenne % calculée sur la bande spectrale [7,9 -12,7 µm] de la caméra ThermaCAM E300; ligne pointillée verticale: barycentre λ C de l'anomalie.

Figure 5 :

 5 Figure 5 : Emissivité apparente LWIR des miroirs Odeillo dans leur état initial.

Figure 6 :

 6 Figure 6 : Emissivité apparente d'Odeillo #1 avant et après 1000h de vieillissement accéléré La caractérisation de l'émissivité apparente est effectuée dans la plage [20°C -120°C] au moyen d'un banc de caractérisation détaillé dans la référence [3]. La figure 5 montre l'émissivité apparente des trois échantillons de verre dans leur état initial. A partir d'une température d'émetteur supérieure à 70°C, la discrimination fondée sur l'âge des échantillons est envisageable. Les valeurs d'émissivité apparentes augmentent de façon monotone avec l'état de vieillissement initial des miroirs. Odeillo #3 est clairement discriminé: l'écart

Figure 7 :

 7 Figure 7 : Ecart relatif % en émissivité apparente LWIR des miroirs verre Odeillo: a. Odeillo #1; b. Odeillo #2 et c. Odeillo #3

LTableau 1 :

 1 'émissivité apparente LWIR est un indicateur pertinent pour quantifier le vieillissement des miroirs verre des héliostats. Les limitations de l'approche résident dans ses capacités de mesurage ainsi que la connaissance des températures d'émission et d'environnement. Les performances typiques de mesures obtenues en situation intérieure sont détaillées dans le tableau 1. Incertitude de type A et B des mesures de températures pour la quantification de l'émissivité apparente LWIR en situation intérieure.


Ismail Harrando

Alison Reboud

Pasquale Lisena

Raphaël Troncy

Jorma Laaksonen

Anja Virkkunen

Using Fan-Made Content, Subtitles and Face Recognition for Character-Centric Video Summarization

This paper describes a fan-driven and character-centered approach proposed by the MeMAD team for the 2020 TRECVID [START_REF] Awad | TRECVID 2020: comprehensive campaign for evaluating video retrieval tasks across multiple application domains[END_REF]] Video Summarization Task. In terms of data, besides the provided videos, scripts and master shot boundaries, our approach relies on fan-made content, more precisely on the BBC EastEnders episode synopses from its Fandom Wiki 1 . We also use BBC EastEnders characters' images crawled from a search engine to train a face recognition system. All our runs use the same method, but with varying constraints of the number of shots and the maximum duration. The shots included in the summaries are the ones whose transcripts and visual content have the highest similarity with sentences from the synopsis. The runs submitted are as follows:

• MeMAD1:5 shots with highest similarity scores and the total duration is < 150 sec • MeMAD2:10 shots with highest similarity scores and the total duration is < 300 sec • MeMAD3: 15 shots with highest similarity scores and the total duration is < 450 sec • MeMAD4: 20 shots with highest similarity scores and the total duration is < 600 sec Surprisingly, the scores obtained for each run were very similar for the questions answering part. Only for the character Ryan, one question more was answered by choosing 15 shots rather than less. For all our runs, the redundancy score improved with the number of shots included in the summary while the relation with the scores for tempo and contextuality seem more varying. The scores were lower for the question answering evaluation part. This is rather unsurprising to us as we realized while deciding on a similarity measure score that it was rather challenging for humans too to choose between two potentially interesting moments without knowing beforehand the questions included in the evaluation set. Overall, we consider that the results obtained speak in favour of using fan-made content as a starting point for such a task. As we did not try to optimize for tempo and contextuality, we believe there is some margin for improvement here, however the task of answering unknown questions remains challenging.

ACM Reference Format:

Ismail Harrando, Alison Reboud, Pasquale Lisena, Raphaël Troncy, Jorma Laaksonen, Anja Virkkunen, and Mikko Kurimo. 2020. Using Fan-Made Authors' addresses: Ismail Harrando, ismail.harrando@eurecom.fr; Alison Reboud, alison.reboud@eurecom.fr; Pasquale Lisena, pasquale.lisena@eurecom.fr; Raphaël Troncy, raphael.troncy@eurecom.fr, EURECOM, Sophia Antipolis, France; Jorma Laaksonen, jorma.laaksonen@aalto.; Anja Virkkunen, anja.virkkunen@aalto.; Mikko Kurimo, mikko.kurimo@aalto., Aalto University, Espoo, Finland.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specic permission and/or a fee. Request permissions from permissions@acm.org. © 2020 Association for Computing Machinery. XXXX-XXXX/2020/11-ART $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn Content, Subtitles and Face Recognition for Character-Centric Video Summarization. 1, 1 (November 2020), 3 pages. https://doi.org/10.1145/nnnnnnn. nnnnnnn

INTRODUCTION

Considering video summarization as an important task for digital content retrieval and reuse, the TRECVID [START_REF] Awad | TRECVID 2020: comprehensive campaign for evaluating video retrieval tasks across multiple application domains[END_REF]] Video Summarization Task (VSUM) 2020 aims at fostering the research in the eld by asking its participants to automatically summarize "the major life events of specic characters over a number of weeks of programming on the BBC EastEnders TV series"2 . More precisely, for three dierent characters of the series, the participants had to submit 4 summaries with 5, 10, 15 and 20 automatically selected shots. These generated summaries are evaluated by the assessors according to their tempo, contextuality and redundancy as well as with regards to how well they answered a set of questions unknown to the participants before submission. In addition to the videos, the episodes transcripts are provided by the organizers. We propose a character centered content summary approach based on fan-written synopses. The approach relies on scraping the Fandom EastEnders Wiki content for the episode synopsis and casting, in order to align them with the corresponding episodes. We include the shots that obtain the best similarity score with a sentence from the synopsis in our runs.

APPROACH

Our fan-driven and character centered approach is presented in Figure 1. The rst step of our approach consists in scraping synopses found on the Fandom EastEnders Wiki3 . Our main hypothesis is that every sentence (ending with a period) represents an important event to be added to the nal video summary. We scrape the Synopsis and the Cast sections for each episode broadcasted between the dates of the provided episodes. The mapping between the episodes and their dates is in eastenders.collection.xml provided by the challenge organizers.

In parallel, we extract the shots in which the three characters of interest appear from the video. To do that, we run Face Celebrity Recognition4 . This system relies on pictures crawled from search engines using the actor's name as search keyword (we add "East-Enders" to their names to avoid picking pictures of dierent people with the same name). For each picture, faces are detected using the MTCNN algorithm and the FaceNet model is applied to obtain face embeddings. Following the assumption that the majority of faces are actually representing the searched actor, other faces -e.g. person portrayed together with the actor -are automatically ltered out by removing outliers until the cosine similarity of face embeddings has a standard deviation below a threshold of 0.24 (empirically dened). The remaining faces are used to train a multi-class SVM classier, which is used to label the faces detected on frames. For more consistent results between frames, the Simple Online and Realtime Tracking algorithm (SORT) has been included, returning groups of detections of the same person in consecutive frames. We select the shots displaying any of the the three characters of interests, keeping only those detections having a condence score greater than 0.5. We also tried to use speaker diaritization to corroborate the visual information about the characters. However,given the limitations of the current technologies in terms of number of characters and the diculty of identifying the character corresponding to each voice, we could not pursue the idea further.

Synopses and transcript alignment and preprocessing

Using eastenders.collection.xml and eastenders.episodeDescriptions.xml, a synopsis for each episode was created. Since these were "Eas-tEnders Omnibus" episodes, they correspond to multiple actual weekday episodes. We use the dates and the continuation to generate one synopsis for each "long" episode (typically made of 4 episodes). We then split the synopses into sentences and performed coreference resolution on the synopses to explicit character mentions5 . In parallel,the provided XML transcripts were also converted into timestamped text and aligned with the given shot segmentation. Finally, both the synopses sentences and shot transcripts were lower cased, stop words removed and lemmatized. We also produced automatically-generated visual captions following the method presented by the PicSOM Group of Aalto University's submissions for the TRECVID2018 VTT task [START_REF] Sjöberg | PicSOM Experiments in TRECVID 2018[END_REF]]. The hypothesis was that by describing the visual information of a shot, visual captions could complement well the dialog transcript and therefore allow for a better matching between the shots and synopses sentences.

Matching and runs generation

We perform a synopsis sentence / shot transcript pairwise comparison by generating a similarity score. We dene similarity between two sentences as the sum of TF-iDF weights (computed on the transcript) for each word appearing in both of them, divided by the log length of the concatenation of both sentences (thus penalizing long sentences that match with many transcript lines). We order the shot by similarity score, picking only the best match for each shot (but not the other way around). This gives us scenes we are sure to appear in the summary, but not necessarily any guarantee about how important these scenes are. We also performed the pairwise comparison adding the automatically generated captions. A qualitative assessment revealed, however, that the captions were too noisy to complement well the transcript. We also make sure that if a line of dialog runs through the next shot, we include the next shot as well (to improve the smoothness of the viewing). This however was only relevant for the longest run (20 shots). Each run is made by picking up the N most matching shots out of the top, in chronological order.

RESULTS AND ANALYSIS

The average results for every team and all runs are are presented on Table 1 and the MeMAD's detailed scores are presented in Table 2. MeMAD obtained the best scores for every run. The mean scores (range 1 -7. High is best) for tempo, contextuality and redundancy are all above average (respectively 4.75,4.75,4.1). However in terms of question answering, the results show that the shots selected did not allow to answer more than two (at best) of the ve questions.

DISCUSSION AND OUTLOOK

This paper describes a character centered video summarization method based on fan-made content, subtitles and face recognition. One of the key contribution of this paper is to have demonstrated that despite some noise from Face Detection and matching, this method captured multiple important plot points for all three query characters. We also conclude that adding more shots to the summaries did, quite surprisingly, not always allow to answer more key moments related questions.

Fig. 1 .

 1 Fig. 1. Fan-driven and character centered approach

Table 1 .

 1 Average score for each run and team

	TeamRun Percentage
	MeMAD1 31%
	MeMAD2 31%
	MeMAD3 35%
	MeMAD4 32%
	NIIUIT1	9%
	NIIUIT2	8%
	NIIUIT3	8%
	NIIUIT4	6%

https://www-nlpir.nist.gov/projects/tv2020/vsum.html , Vol. 1, No. 1, Article . Publication date: November 2020.

https://eastenders.fandom.com/wiki/EastEndersWiki

https://github.com/D2KLab/Face-Celebrity-Recognition

https://github.com/huggingface/neuralcoref

, Vol. 1, No. 1, Article . Publication date: November 2020.

Using Fan-Made Content, Subtitles and Face Recognition for Character-Centric Video Summarization


Héctor Laria

Jorma Laaksonen

Danny Francis

Benoit Huet

PicSOM and EURECOM Experiments in TRECVID 2019 Pre-workshop draft -Revision: 0.9

This year, the PicSOM and EURECOM teams participated only in the Video to Text Description (VTT), Description Generation subtask. Both groups submitted one or two runs labeled as a "MeMAD" submission, stemming from a joint EU H2020 research project with that name. In total, the PicSOM team submitted four runs and EURECOM one run. The goal of the PicSOM submissions was to study the effect of using either image or video features or both. The goal of the EURECOM submission was to experiment with the use of Curriculum Learning in video captioning. The submitted five runs are as follows:

• PICSOM.1-MEMAD.PRIMARY: uses ResNet and I3D features for initialising the LSTM generator, and is trained on MS COCO + TGIF using self-critical loss, • PICSOM.2-MEMAD: uses I3D features as initialisation, and is trained on TGIF using self-critical loss, • PICSOM.3: uses ResNet features as initialisation, and is trained on MS COCO + TGIF using self-critical loss, • PICSOM.4: is the same as PICSOM.1-MEMAD.PRIMARY except that the loss function used is cross-entropy, • EURECOM.MEMAD.PRIMARY: uses I3D features to initialize a GRU generator, and is trained on TGIF + MSR-VTT + MSVD with cross-entropy and curriculum learning. The runs aim at comparing the use of cross-entropy and self-critical training loss functions and to showing whether one can successfully use both still image and video features even when the COCO dataset does not allow the extractions of I3D video features. Based on the results of the runs, it seems that using both video and still image features, one can obtain better captioning results than with either one of the single modalities alone. The Curriculum Learning process proposed does not seem to be beneficial.

I. INTRODUCTION

In this notebook paper, we describe the PicSOM and EU-RECOM teams' experiments for the TRECVID 2019 evaluation [START_REF] Awad | Trecvid 2019: An evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval[END_REF]. We participated only in the Video to Text Description (VTT) subtask Description Generation. Our approaches are variations of the "Show and tell" model [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF], augmented with a richer set of contextual features [START_REF] Shetty | Image and video captioning with augmented neural architectures[END_REF], self-critical training [START_REF] Rennie | Self-critical sequence training for image captioning[END_REF] and Curriculum Learning [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF]. Both teams' systems have been used to produce the runs presented in this paper. The captioning models are described in more detail in Section II and their used training loss functions in Section III. Then, we describe the features in Section IV and the datasets used for training in Section V. Our experiments, submitted runs and results are discussed in Section VI and conclusions are drawn in Section VII.

II. NEURAL CAPTIONING MODELS

In our experiments we have used two different Python-based software projects for caption generation. The PicSOM team's DeepCaption, uses the PyTorch library, whereas EURECOM's CLCaption approach is based on using the TensorFlow library.

A. DeepCaption

The PicSOM team's LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] model has been implemented in PyTorch and is available as open source. 1 The features are translated to the hidden size of the LSTM by using a fully connected layer. We apply dropout and batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] at this layer. As the loss function, we similarly use cross entropy, in addition to Reinforcement Learning with self-critical loss function [START_REF] Rennie | Self-critical sequence training for image captioning[END_REF] in order to finetune a well-performing model. The fine-tuning is implemented either by switching to the self-critical loss in training time or by specifying a pre-trained model to load and fine-tune.

B. CLCaption

For EURECOM's first participation in the TRECVID VTT captioning task, we submitted a run based on a model trained by Curriculum Learning [START_REF] Bengio | Curriculum learning[END_REF]. We implemented our model using the TensorFlow framework for Python [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF].

The idea behind Curriculum Learning is to present data during training in an ascending order of difficulty: first epochs are based on easy samples, and after each epoch, more difficult samples are added to training data. We computed a difficulty score for a given sample composed of a video and a corresponding caption as follows: the caption is translated into a list of indices (the bigger the indices the less frequent the corresponding word), the score of the sample is then the maximum index of its caption. Once samples have been scored, we trained the model starting with an easy subset of the training set, and adding after each epoch more complex samples.

Video features have been extracted with an I3D neural network [START_REF] Carreira | Quo vadis, action recognition? A new model and the kinetics dataset[END_REF], input to a fully connected layer and then processed by a GRU [START_REF] Cho | Learning phrase representations using RNN encoder-decoder for statistical machine translation[END_REF] to generate captions. Cross-entropy loss has been used for training the model.

III. TRAINING LOSS FUNCTIONS

In order to train the architecture so that its output distribution approximates the target distribution at each decoding step t, several optimisation objectives are used. Recent progress on sequence training enables new optimisation paradigms, which are applied and compared in this work.

A. Cross-entropy

Traditionally, the teacher forcing algorithm [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF] is the most common method to maximise the log-likelihood of a model output X to match the ground truth y = {y 1 , y 2 , • • • , y T }. It minimises the cross-entropy objective

L CE = T X t=1 log p ✓ (y t | y t 1 , h t 1 , X) , (1)
where h t 1 is the hidden state of the RNN from the previous step and p ✓ the probability of an output parametrized by ✓.

In the inference time, the output can be produced simply by greedy sampling of the sequence being generated.

B. Self-critical

Lately, Reinforcement Learning ideas have been used to optimise a captioning system based on recurrent neural network language models. Such a system can be seen as an agent taking actions according to a policy ⇡ ✓ and outputting a word ŷt as an action.

One proposed approach is the self-critical algorithm [START_REF] Rennie | Self-critical sequence training for image captioning[END_REF], where the output at inference time of the model ŷg i,t is used, normally applying greedy search. The sequences are scored using a reward function r. Thanks to the properties of this optimisation, NLP metrics can be used as reward to affect the actual loss. In our case, CIDErD [START_REF] Ramakrishna Vedantam | Cider: Consensus-based image description evaluation[END_REF] is used. The final objective reads

L ✓ = 1 N P N i=1 P t log ⇡ ✓ (ŷ i,t | ŷi,t 1 , s i,t , h i,t 1 ) • ⇣ r(ŷ i,1 , • • • , ŷi,T ) r(ŷ g i,1 , • • • , ŷg i,T ) ⌘ . (2)
IV. FEATURES Table I summarizes the features used in our experiments and their dimensionalities.

A. CNN

We are using pre-trained CNN features from ResNet 101 and 152. The 2048-dimensional features from the pool5 layer average to five crops from the original and horizontally flipped images. These features have then been concatenated together and are referred to as "rn" in Table I and later in this paper. When applied to a video object, we have used the middlemost frame of the video.

B. FasterRCNN

The existence of certain objects in the visual scene has an effect on sentence formation and influences the adjectives used in human sentences. To extract this information, we use an object detector, specifically the Faster Region-based Convolutional Neural Network (R-CNN) [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF]. This network predicts the object locations as bounding boxes and object detection scores of the 80 object categories of Microsoft Common Objects in Common Context (MS-COCO) database. 2In our current approach we, however, ignore the location information and encode the object detection scores on the image level. We obtain thus an 80-dimensional feature vector using the detection score for each category, and refer to it as "fr". When applied to a video object, we have used the middlemost frame of the video.

C. I3D

To encode video features, the PicSOM team adopted Inflated 3D Convolutional Network (I3D) [START_REF] Carreira | Quo vadis, action recognition? A new model and the kinetics dataset[END_REF]. It builds upon already competent image recognition models (2D) and inflates the filters and kernels to 3D, thus creating an additional temporal dimension. Concretely, the base network used is ImageNetpretrained Inception-V1 [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] using two streams [START_REF] Feichtenhofer | Convolutional two-stream network fusion for video action recognition[END_REF]. The videos were first resampled to 25 frames per second as in the original I3D paper and 128 frames were taken from the center. For DeepCaption, the extractor is applied convolutionally over the whole video and the output is average-pooled in order to produce a 2048-dimensional feature vector.

Regarding CLCaption, features have been extracted before the softmax layer, thus obtaining a 600-dimensional features vector. These features have then been input to the CLCaption model.

V. TRAINING DATA

Table II gives a summary of the databases and the features we have extracted for them. In Tables II and III, we have shortened the dataset names with one letter abbreviations.

A. COCO

The Microsoft Common Objects in COntext (MS COCO) dataset [START_REF] Lin | Microsoft COCO: Common objects in context[END_REF] has 2,500,000 labeled instances in 328,000 images, consisting on 80 object categories. COCO is focused on non-iconic views (or non-canonical perspectives) of objects, contextual reasoning between objects, and precise 2D localization of objects.

B. MSR-VTT

The MSR-Video to Text (MSR-VTT) dataset [START_REF] Xu | MSR-VTT: A large video description dataset for bridging video and language[END_REF] provides 10,000 web video clips with 41.2 hours and 200,000 clipsentence pairs in total, covering a comprehensive list of 20 categories and a wide variety of video content. Each clip was annotated with about 20 natural sentences. Additionally, the audio channel is provided too.

C. TGIF

The Tumblr GIF (TGIF) dataset [START_REF] Li | TGIF: A new dataset and benchmark on animated gif description[END_REF] contains 100,000 animated GIFs and 120,000 natural language sentences. This dataset aims to provide motion information involved between image sequences (or frames).

D. MSVD

The Microsoft Research Video Description Corpus (MSVD) [START_REF] Chen | Collecting highly parallel data for paraphrase evaluation[END_REF] consists of 85,000 English video description sentences and more than 1,000 for a dozen more languages. It contains a set of 2,089 videos, showing a single, unambiguous action or event.

VI. EXPERIMENTS AND RESULTS

During the development stage, the PicSOM team ran a number of experiments to select the best combinations of features and training datasets. We evaluated our results using the previously released ground truth of TRECVID VTT 2018 test set. The four runs submitted are identified as "p-19-s1" to "p-19-s4" in Table III. The runs "p-18-s2" and "p-18-a3" we created using our best model in the last year's submissions and the best model we experimented with after the last year's workshop, respectively.

Runs identified as "p-19-s1" and "p-19-s4" use I3D video features extracted from the TGIF dataset. We used also the COCO dataset for training the models for those runs, but because we could not extract I3D video features from the still images of that dataset, we used the average value of the I3D feature vectors of the TGIF dataset for each COCO image.

Based on evaluation on the TRECVID VTT 2018 test set, we ended up using a 2-layer LSTM for DeepCaption with an embedding vector size of 512, and 1024 for the hidden state dimensionality in all PicSOM team's runs. Both in the input translation layer and in the LSTM we applied a dropout of 0.5. We used Adam optimiser [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] for the self-critical stage with a learning rate of 5 ⇥ 10 5 and no weight decay. Additionally, gradient clipping is performed when a range [ 0.1, 0.1] is exceeded. The models were pretrained using centered RMSprop [START_REF] Graves | Generating sequences with recurrent neural networks[END_REF] with a learning rate of 0.001 and weight decay (L2 penalty) of 10 6 .

EURECOM's CLCaption is based on a GRU with 1024dimensional hidden states. The size of the input I3D vectors is 600. The fully-connected layer output is of dimension 1024. No dropout nor batch normalization were used. The training algorithm we used was RMSProp with a learning rate of 0.0001 and mini-batches of size 64. The CLCaption run is identified as "e-19-e1" in Table III where all experiments are briefly summarized and their results presented.

The four "setup" columns in Table III specify the submission type (I=image, V=video, B=both), the loss function (ce=cross-entropy, sc=self-critical), the features, and the datasets used in the RNN model training.

The features are concatenations of the following:

rn = CNN ResNet, see IV-A fr = Faster R-CNN, see IV-B i3d = I3D, see IV-C
The used datasets are combinations of the following:

C = COCO, see V-A M = MSR-VTT, see V-B T = TGIF, see V-C V = MSVD, see V-D
Our results compared to those of the other submitted runs are visualized with bar charts for each automatic performance measure in Figures 12345.

VII. CONCLUSIONS

There were two main research question in the PicSOM team's set of four submissions. First, we wanted to compare the implementations of cross-entropy and self-critical training loss functions in our DeepCaption code. The results with selfcritical training were better in all measures, but this could of course be expected based on our and other teams' earlier experiments. Based on our observations, however, the use of this loss alone does not imply a straightforward jump in caption quality as much as the score increment suggests.

Second, we aimed to know whether we could successfully use both still image and video features even when the COCO dataset does not allow the extractions of I3D video features. The trick we applied was to use the average of the I3D video features extracted from the TGIF dataset for all images in the COCO dataset. For the COCO images the video features were thus non-informative, but still allowed us to use two datasets and two different feature extraction schemes together. The results of this approach were encouraging as they were better than those with either dataset or either feature used alone.

Additionally, we could now compared the current performance of the PicSOM team's DeepCaption model to its performance in the last year's evaluation. We have clearly made substantial progress compared to both the last year's submission and to the post-workshop experiments reported in our previous notebook paper. However, compared to the level of performance reached by some of the other research groups, we are still clearly behind as all the groups seem to have improved from the previous year.

The results obtained by CLCaption are far from standing comparison with the best runs of TRECVID VTT 2019. However, multiple ways to improve them can be explored, such as different scoring methods or finer curriculum learning algorithms. We will explore these directions to boost the results of CLCaption.

Fig. 1 .

 1 Fig. 1. METEOR results of our teams and others.

Fig. 3 .

 3 Fig. 3. CIDErD results of our teams and others.

Fig. 4 .

 4 Fig. 4. BLEU results of our teams and others.

Fig. 5 .

 5 Fig. 5. STS results of our teams and others.

TABLE I SUMMARY

 I OF THE FEATURES USED IN OUR EXPERIMENTS.

	abbr. feature	dim. modality
	rn	CNN ResNet	4096 image
	fr	Faster R-CNN	80 image
	i3d	I3D	2048 video

TABLE II SUMMARY

 II OF THE TRAINING DATASETS USED IN OUR EXPERIMENTS.

	dataset	items captions features
	C COCO	82,783 img 414,113 rn fr
	M MSR-VTT	6,513 vid 130,260 rn i3d
	T TGIF	125,713 vid 125,713 rn fr i3d
	V MSVD	1,969 vid	80,800 rn i3d

TABLE III RESULTS

 III OF OUR SUBMISSIONS (P-19-S1,. . . ,4, E-19-E1) AND SOME NOTEWORTHY EARLIER MODELS (P-18-S2, P-18-A3). THE P-* RUNS ARE BY THE PICSOM TEAM AND THE E-* RUN BY THE EURECOM TEAM.

				setup			2018					2019
	id	t loss	feat	data	METEOR CIDEr CIDErD BLEU METEOR CIDEr CIDErD BLEU	STS
	p-18-s2 I	ce	rn+fr	C+M	0.1541	0.1657	0.0476	0.0091	0.1773	0.1858	0.0722	0.0207	-
	p-18-a3 I	ce	rn	C+T	0.1776	0.1948	0.0700	0.0197	0.1993	0.2174	0.1004	0.0288	-
	p-19-s1 B sc rn+i3d	C+T	0.2055	0.3025	0.1157	0.0294	0.2285	0.3277	0.1615	0.0385 0.4168
	p-19-s2 V sc	i3d	T	0.1958	0.2718	0.0949	0.0348	0.2139	0.2773	0.1245	0.0379 0.4169
	p-19-s3 I	sc	rn	C+T	0.2007	0.2777	0.1074	0.0301	0.2254	0.3130	0.1569	0.0345 0.4282
	p-19-s4 B ce rn+i3d	C+T	0.1850	0.2190	0.0822	0.0213	0.2049	0.2348	0.1147	0.0319 0.4057
	e-19-e1 V ce	i3d	M+T+V	-	-	-	-	0.1743	0.2340	0.0710	0.0068 0.4214

Fig. 2. CIDEr results of our teams and others.

https://github.com/aalto-cbir/DeepCaption

http://cocodataset.org/

ACKNOWLEDGMENTS

This work has been funded by the grant 313988 Deep neural networks in scene graph generation for perception of visual multimedia semantics (DeepGraph) of the Academy of Finland, the ANR (the French National Research Agency) via the ANTRACT project, and the European Union's Horizon 2020 research and innovation programme under grant agreement No 780069 Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy (MeMAD). This work was supported by the Academy of Finland Flagship programme: Finnish Center for Artificial Intelligence, FCAI. The calculations were performed using computer resources provided by the Aalto University's Aalto Science IT project and CSC -IT Center for Science Ltd. We also acknowledge the support of NVIDIA Corporation with the donation of TITAN X and Quadro P6000 GPUs used for parts of this research.


est destinée au dépôt

  Fig.1 Données concernant le sentiment de compétence de suivi du neurodéveloppement des médecins inclus


Christo Kurisummoottil

Bruno Clerckx
email: b.clerckx@imperial.ac.uk

Luca Sanguinetti
email: luca.sanguinetti@unipi.it

Dirk Slock
email: slock@eurecom.fr

A Rate Splitting Strategy for Mitigating Intra-Cell Pilot Contamination in Massive MIMO

The spectral efficiency (SE) of Massive MIMO (MaMIMO) systems is affected by low quality channel estimates. Rate-Splitting (RS) has recently gained some interest in multiuser multiple antenna systems as an effective means to mitigate the multi-user interference due to imperfect channel state information. This paper investigates the benefits of RS in the downlink of a single-cell MaMIMO system when all the users use the same pilot sequence for channel estimation. Novel expressions for the SE achieved in the downlink by a single-layer RS strategy (that relies on a single successive interference cancellation at each user side) are derived and used to design precoding schemes and power allocation strategies for common and private messages. Numerical results are used to show that the proposed RS solution achieves higher spectral efficiency that conventional MaMIMO with maximum ratio precoding.

I. INTRODUCTION

Massive MIMO (MaMIMO) is a wireless technology where the base stations (BSs) are equipped with a large number M of antennas to serve a multitude of single-antenna K user equipments (UEs) by spatial multiplexing [START_REF] Marzetta | Noncooperative cellular wireless with unlimited numbers of base station antennas[END_REF]. The acquisition of channel state information (CSI) is the limiting factor in MaMIMO [START_REF] Marzetta | Noncooperative cellular wireless with unlimited numbers of base station antennas[END_REF]. In a time-division duplex (TDD) mode, channel reciprocity allows to acquire all the necessary CSI for uplink (UL) and downlink (DL) transmissions from a finite number of UL pilot signals [START_REF] Marzetta | Noncooperative cellular wireless with unlimited numbers of base station antennas[END_REF]. Thanks to the intense research performed over the last decade, MaMIMO is today a mature technology [START_REF] Marzetta | Fundamentals of Massive MIMO[END_REF], [START_REF] Björnson | Massive MIMO networks: Spectral, energy, and hardware efficiency[END_REF], which has been adopted into the 5G NR standard [START_REF] Björnson | Massive MIMO is a Reality What is Next? Five Promising Research Directions for Antenna Arrays[END_REF].

One phenomenon that is tightly connected with MaMIMO is pilot contamination, which can be briefly explained as follows [START_REF] Marzetta | Noncooperative cellular wireless with unlimited numbers of base station antennas[END_REF]. UEs that transmit the same pilot signal contaminate each others' channel estimates. This "pilot interference" not only reduces the CSI quality but also creates the so-called "coherent interference", which has been believed to fundamentally limit the spectral efficiency (SE) of MaMIMO, even when M → ∞ [START_REF] Marzetta | Noncooperative cellular wireless with unlimited numbers of base station antennas[END_REF], [START_REF] Marzetta | Fundamentals of Massive MIMO[END_REF]. Recently, [START_REF] Björnson | Massive MIMO Has Unlimited Capacity[END_REF] showed that with optimal signal processing and spatially correlated channels, the SE increases without bound as M → ∞ while K is fixed. The fact that there is no fundamental SE limit does not imply that the pilot contamination effect disappears; there is still an SE loss caused by estimation errors and interference rejection [START_REF] Sanguinetti | Toward massive MIMO 2.0: Understanding spatial correlation, interference suppression, and pilot contamination[END_REF]. The aim of this paper is to deal with this effect for a finite M .

Observe that, when the estimation error variance decays with the signal-to-noise-ratio (SNR) as O(SNR -δ ) for some 0 ≤ δ < 1, conventional precoding techniques result in a sum degrees of freedom (DoF) of Kδ. This in turn reveals that as δ → 0 (implies constant channel estimation error), the system becomes interference limited. A possible solution to this issue is to take a rate splitting (RS) approach [START_REF] Clerckx | Rate Splitting for MIMO Wireless Networks: a Promising PHY-Layer Strategy for LTE Evolution[END_REF] that splits the UEs' messages into common and private parts, encode the commmon parts into a common stream, and private parts into private streams and superpose in a non-orthogonal manner the common stream on top of all private streams. The common stream is drawn from a codebook shared by all UEs and is intended to one only, but is decodable by all UEs. On the other hand, the private streams are to be decoded by the corresponding UEs only. The sum DoF achieved by RS in the DL is 1 + (K -1)δ [START_REF] Joudeh | Sum-Rate Maximization for Linearly Precoded Downlink Multiuser MISO Systems With Partial CSIT: A Rate-Splitting Approach[END_REF], which is higher than Kδ and matches the upper bound obtained from the Aligned Image Sets in [START_REF] Davoodi | Aligned Image Sets under Channel Uncertainty: Settling Conjectures on the Collapse of Degrees of Freedom under Finite Precision CSIT[END_REF]. Interestingly, RS not only achieves the optimal sum-DoF but the entire DoF region of the K-UE channel with imperfect CSI [START_REF] Piovano | Optimal DoF region of the K-user MISO BC with partial CSIT[END_REF].

Motivated by the above results, the design and optimization of RS at finite values of SNR has been investigated and was found to provide significant benefits in the DL with imperfect CSI, compared to multi-user MIMO and NOMA [START_REF] Joudeh | Sum-Rate Maximization for Linearly Precoded Downlink Multiuser MISO Systems With Partial CSIT: A Rate-Splitting Approach[END_REF], [START_REF] Joudeh | Robust Transmission in Downlink Multiuser MISO Systems: A Rate-Splitting Approach[END_REF], [START_REF] Mao | Rate-Splitting Multiple Access for Downlink Communication Systems: Bridging, Generalizing, and Outperforming SDMA and NOMA[END_REF], but also to Dirty Paper Coding [START_REF] Mao | Beyond Dirty Paper Coding for Multi-Antenna Broadcast Channel with Partial CSIT: A Rate-Splitting Approach[END_REF]. The application of RS to an FDD MaMIMO system has been investigated in [START_REF] Dai | A Rate Splitting Strategy for Massive MIMO with Imperfect CSIT[END_REF], [START_REF] Papazafeiropoulos | Rate-Splitting to Mitigate Residual Transceiver Hardware Impairments in Massive MIMO Systems[END_REF]. Particularly, [START_REF] Dai | A Rate Splitting Strategy for Massive MIMO with Imperfect CSIT[END_REF] shows that a two-layer RS architecture, so-called hierarchical RS (HRS), can bring significant benefits in MaMIMO.

In this paper, we focus on a TDD single-cell MaMIMO network and assume that all the UEs use the same pilot signal for channel estimation. Novel expressions for the SE achieved in the DL by a single-layer RS strategy are derived by applying the hardening bound to both common and private messages [START_REF] Marzetta | Fundamentals of Massive MIMO[END_REF]. A maximum ratio (MR) precoding scheme is used for private streams while a precoder based on a weighted combination of the channel estimates of all UEs is adopted for the common stream. A novel algorithm is proposed to allocate the power among the common and private streams.

II. SYSTEM MODEL

We consider a single-cell MaMIMO network where the BS is equipped with M antennas and serves K UEs. We denote h i ∈ C M the channel from UE i to the BS, and consider a correlated Rayleigh block fading model h i ∼ CN (0, R i ) where R i ∈ C M ×M is the covariance matrix [START_REF] Björnson | Massive MIMO networks: Spectral, energy, and hardware efficiency[END_REF]Sec. 2.2]. The Gaussian distribution is used to model the small-scale fading variations, while R i describes the macroscopic propagation characteristics. The normalized trace β i = 1

M tr(R i ) is the average channel gain from the BS to UE i.

The UEs are perfectly synchronized and operate according to a TDD protocol with a data transmission phase and a pilot phase for channel estimation [START_REF] Björnson | Massive MIMO networks: Spectral, energy, and hardware efficiency[END_REF]. We consider the standard block fading TDD protocol in which each coherence block consists of τ channel uses, whereof τ p are used for UL pilots, τ u for UL data, and τ d for DL data, with τ = τ p + τ u + τ d . Only the DL is considered in this paper, i.e., τ u = 0.

A. Channel estimation

We assume that a single pilot sequence of length τ p is used. For a total uplink pilot power of ρ tr per UE, the BS obtains the MMSE estimate of h i as

h i = R i Q -1 K k=1 h k + 1 √ ρ tr n i ∼ CN (0, Φ i ) (1)
where

n i ∼ CN (0, I M ) is noise, Φ i = R i Q -1 R i and Q = K k=1 R k + 1 ρtr I M . The estimation error h i = h i - h i ∼ CN (0, R i -Φ i ) is independent of h i .
The mutual interference generated by the pilot-sharing UEs is known as pilot contamination and has two main consequences in the channel estimation process. The first is the reduced estimation quality, whereas the second is that the estimates

{ h i } become correlated. If R k is invertible, we have that [3, Sec. 3.2] h i = R i R -1 k h k (2)
from which it follows that

E{ h i h H k } = R i Q -1 R k . B.

Rate Splitting in Downlink transmissions

The RS scheme is used in the DL for transmission. The message intended to UE k is split into two parts, W k = (W k0 , W k1 ). We assume that W k0 ∈ W k0 represents the common part and W k1 ∈ W k1 is the private part. All the common parts are packed into one common message, W c = (W k0 , ..., W K0 ) ∈ W c , which is encoded into a common stream ς c using a common codebook. The private message W k1 is encoded in the conventional manner into the private stream ς k . The resulting transmitted DL signal is:

x = w c ς c Common message + K i=1 w i ς i Private messages (3)
where ς i ∼ CN (0, ρ i ) is assigned to a precoding vector w i ∈ C M that determines the spatial directivity of the transmission and satisfies E{ w i 2 } = 1 so that ρ i represents the average transmit power of UE ∀i. Similarly, ς c ∼ CN (0, ρ c ) denotes the common message, which is assigned to a precoding vector w c ∈ C M with E{ w c 2 } = 1 so that ρ c represents its average transmit power. We assume that

ρ c + K i=1 ρ i ≤ ρ T (4)
where ρ T is the total transmit power in the DL. The received signal y k ∈ C at UE k is given by

y k = h H k w c ς c + h H k w k ς k + K i=1,i =k h H k w i ς i + n k (5)
where n k ∼ CN (0, σ 2 ) is the receiver noise. At each UE k, the common stream is first decoded into W c , by treating the interference from the private streams as noise. Then, successive interference cancellation (SIC) is performed, which removes the common message part from the received signal. Further, the private stream ς k is decoded into W k1 by treating the intracell interference as noise. UE k reconstructs the transmitted message by extracting W k0 from W c . Further, combining with the decoded private stream to form W k = ( W k0 , W k1 ).

C. Spectral efficiency

Characterizing the SE in the DL is hard since it is unclear how UE k should best estimate the effective precoded channels h H k w c and h H k w k that are needed for decoding the common signal ς c and the private signal ς k . A common approach in classical MaMIMO is to resort to the hardening bound [START_REF] Björnson | Massive MIMO networks: Spectral, energy, and hardware efficiency[END_REF]Sec. 4.3]. This bound relies on the assumption that the deterministic average precoded channels E{h H k w c } and E{h H k w k } are known at UE k. The received signals for the common and private messages can then be expressed as

y k,c = E{h H k w c }ς c + (h H k w c -E{h H k w c }) ς c + K i=1 h H k w i ς i + n k (6)
and (after SIC)

y k,p = E{h H k w k }ς k + (h H k w k -E{h H k w k }) ς k + (h H k w c -E{h H k w c }) ς c + K i=1,i =k h H k w i ς i + n k . ( 7
)
The following bounds can be computed.

Lemma 1. Achievable rates for the common and private messages of UE k can be computed as

SE k,c = τ d τ log 2 (1 + γ k,c ) (8)
and

SE k = τ d τ log 2 (1 + γ k ) (9)
with γ k,c and γ k given by [START_REF] Joudeh | Robust Transmission in Downlink Multiuser MISO Systems: A Rate-Splitting Approach[END_REF] and [START_REF] Mao | Rate-Splitting Multiple Access for Downlink Communication Systems: Bridging, Generalizing, and Outperforming SDMA and NOMA[END_REF]. The expectations are computed over channel realizations.

Proof: It can be proved from ( 6) and ( 7) by using standard results in MaMIMO (e.g., [3, App. C.3.6]), which are omitted for space limitations.

The achievable rate of the common message is defined as

SE c = τ d τ log(1 + γ lmin,c ) (13)
where

l min = arg min k γ k,c (14)
γ k,c = ρ c |E{h H k w c }| 2 K i=1 ρ i E{|h H k w i | 2 } + ρ c E{|h H k w c | 2 } -|E{h H k w c }| 2 + σ 2 (11)
γ k = ρ k |E{h H k w k }| 2 K i=1 ρ i E{|h H k w i | 2 } -ρ k |E{h H k w k }| 2 + ρ c E{|h H k w c | 2 } -|E{h H k w c }| 2 + σ 2 (12)
Observe that the above achievable rates can be utilized along with any precoding scheme. Moreover, each of the expectations in γ k,c and γ k can be computed separately by means of Monte Carlo simulations. Closed forms will be provided next for the proposed precoding schemes.

III. POWER OPTIMIZATION AND PRECODING DESIGN

A common and popular choice for w k is MR precoding, defined as

w MR k = h k E{| h k | 2 } = h k tr{Φ k } (15)
which has low computational complexity and allows to compute some of the expectations in closed form. Particularly, we have that (e.g., [START_REF] Björnson | Massive MIMO networks: Spectral, energy, and hardware efficiency[END_REF]App. C.3.7])

|E{h H k w MR k }| 2 = tr{Φ k } (16)
E{|h H k w MR i | 2 } = tr{R k Φ i } + tr{R k Q -1 R i } 2 tr{Φ i } . ( 17
)
In the remainder, we assume that MR precoding is used for private messages. Next, we look for the transmit powers that maximize the sum SE of the network and design the precoding vector for the common message.

A. Power optimization

From the above section, the sum SE, for any given precoding scheme, can be computed as:

SE = SE c + K k=1 SE k (18)
where SE k and SE c are given in ( 9) and ( 13), respectively. The power allocation problem can thus be formulated as:

max {ρc≥0,ρ≥0} SE c (ρ c , ρ) + K k=1 SE k (ρ c , ρ) (19)
s.t. ρ c + K i=1 ρ i ≤ ρ T (20)
with ρ = [ρ 1 , . . . , ρ K ] T . Finding the solution to the above problem is a challenge since it is not in a convex form.

A possible way out consists in using the method in [START_REF] Kim | Optimal Resource Allocation for MIMO Ad-hoc Cognitive Radio Networks[END_REF], and linearize the sum SE in (18) using a first order Taylor series approximation. The optizimation is then carried out by adopting an iterative approach in which the variables ρ c and Algorithm 1 ILA-WF power allocation 1: initialize t = 0 and ρ (0) c = 0 (no RS) and ρ if ρ

(0) k = ρT /K. Also, µ (0) = 1 2 (µ (0) u + µ (0) l ) with µ (0) u = 10
(t) c + k ρ (t) k > ρT then 10: µ (t+1) l = µ (t) , µ (t+1) u = µ (t) u 11: else 12: µ (t+1) u = µ t , µ (t+1) l = µ (t) l 13:
end if

14: update µ (t+1) = µ (t+1) u +µ (t+1) l 2 15:
update t = t + 1 16: until convergence {ρ i : i = 1, . . . , K} are alternatively optimized. In Appendix A, it is shown that at iteration t the powers must be updated as follows

ρ (t) k = 1 µ (t) + σ (2,t) k - 1 σ (1,t) k + (21)
and

ρ (t) c = 1
µ (t) + σ (2,t) c - 1 σ (1,t) c + (22)
where (x) + = max(x, 0) and the quantities {σ

(1,t) k , σ (1,t) c } and {σ (2,t) k , σ (2,t) c
} are defined in Appendix A. The former represent the signal powers of private and common messages at iteration t, respectively, while the latter can be interpreted as the corresponding leakage powers. This is why (21) and ( 22) are called interference leakage-aware water-filling (ILA-WF) power allocations [START_REF] Thomas | Deterministic Annealing for Hybrid Beamforming Design in Multi-Cell MU-MIMO Systems[END_REF]. Note that the Lagrange multiplier µ (t) needs to satisfy the power constraint in (20) and can be computed by a bisection method [START_REF] Boyd | Convex Optimization[END_REF]. The entire procedure is summarized through Algorithm 1.

As done for γ k,c and γ k , we observe that all the quantities involved in the computation of {σ

c } are deterministic and can be computed by means of Monte Carlo simulations for any choice of the precoding scheme for the common message. Closed form expressions are provided below for a MR-inspired precoding scheme.

B. Precoding design for common message

The optimal design of the precoding vector w c for the common message requires to solve a multi-objective problem involving γ lmin,c and {γ i : ∀i}. To overcome this issue, we assume that the difference E{|h 11) is small so that it can be neglected. The precoding vector is then suboptimally selected as the solution to the following problem:

H k w c | 2 } -|E{h H k w c }| 2 in (
max wc min k π k |E{h H k w c }| 2 s.t. E{ w c 2 } = 1 (23)
where

π k = 1 K i=1 ρ i E{|h H k w i | 2 } + σ 2 . ( 24
)
Following [START_REF] Dai | A Rate Splitting Strategy for Massive MIMO with Imperfect CSIT[END_REF], we heuristically select w c as a linear combination of the estimated channel vectors { h i : ∀i}:

w c = α K i=1 a i h i ( 25
)
where α is needed to satisfy the constraint E{ w c 2 } = 1. Plugging (25) into E{h H k w c }, we may rewrite (23) as:

max {ai} min k π k K i=1 a i tr{R i Q -1 R k } 2 ( 26
)
where we have neglected the scaling factor α 2 . We now observe that (26) can be reformulated as a geometric programming problem [START_REF] Boyd | Convex Optimization[END_REF]:

max t>0 t, s.t. a T u i ≤ t, ∀ i = 1, .., K (27)
where we have defined a = [a 1 , ..., a K ] T and u i = [u i (1), . . . , u i (K)] T with entries

u i (k) = tr{R i Q -1 R k }.
Once the solution a to (27) is computed, the optimal w c is obtained as:

w c = K i=1 a i h i K i=1 K j=1 a i a j tr{R i Q -1 R j } . ( 28
)
The expectations that depend on w c can be computed in closed form as follows. By using ( 2) into (28) yields

E{h H k w c } = K i=1 a i tr{R i Q -1 R k } K i=1 K j=1 a i a j tr{R i Q -1 R j } . ( 29
)
To compute E{|h H k w c | 2 }, observe that it can be rewritten as

E{|h H k w c | 2 } = 1 K i=1 K j=1 a i a j tr{R i Q -1 R j } ×   K i=1 (a i ) 2 E{| h H i h k | 2 } + K i=1 K j=1,j =i a i a j E{h H k h i h j h k }   . ( 30
)
The first term in (30) becomes (e.g., [3, Eq. (C.65)])

E{| h H i h k | 2 } = tr{R k Φ i } + tr{R k Q -1 R i } 2 (31)
while the second one in (30) reduces to

E{h H k h i h H j h k } (a) = E{h H k h i h H i R -1 i R j h k } (32) (b) = tr R -1 i R j E{ h k h H k h i h H i } + tr R -1 i R j E{ h k h H k }E{ h i h H i } ( 33
) (c) = tr R -1 i R j E{ h k h H k h i h H i } + tr R -1 i R j (R k -Φ k )Φ i } (34)
where (a) uses h j = R j R -1 i h i (as it follows from (2)), (b) follows from h k = h k + h k and the independence between the estimate h k and estimation error h k , whereas (c) uses

E{ h k h H k }E{ h i h H i } = (R k -Φ k )Φ i . In Appendix B, it is shown that E{ h k h H k h i h H i } = tr{B ik }Φ k + Φ 1/2 k diag (B ik ) + B ik (Φ 1/2 k ) H (35)
where

B ik = (Φ 1/2 k ) H R i R -1 k Φ 1/2 k
and diag (•) indicates the main diagonal of the enclosed matrix.

Note that, by using the above expressions and those in ( 16) and ( 17), we can eventually compute in closed form all the expectations involved in [START_REF] Joudeh | Robust Transmission in Downlink Multiuser MISO Systems: A Rate-Splitting Approach[END_REF] and [START_REF] Mao | Rate-Splitting Multiple Access for Downlink Communication Systems: Bridging, Generalizing, and Outperforming SDMA and NOMA[END_REF].

IV. SIMULATION RESULTS

To quantify the SE that can be achieved in MaMIMO with RS, we consider a cell of size 250 m × 250 m. The UL pilot power is ρ tr = 20 dBm, whereas the noise power in UL and DL is σ 2 = -94 dBm. The samples per coherence block are τ = 200 with τ p = 10. Each BS is equipped with a uniform linear array with half-wavelength antenna spacing. Each channel consists of S = 6 scattering clusters, which are modeled by the Gaussian local scattering model [3, Sec.

2.6]. Hence, the (m 1 , m 2 )th element of R i is [R i ] m1,m2 = β i × 1 S S s=1 e jπ(m1-m2) sin(ϕi,s) e - σ 2 ϕ 2 (π(m1-m2) cos(ϕi,s)) 2 (36)
where β i is the large-scale fading coefficient given by (in dB)

β i | dB = -34.53 -38 log 10 d i 1 km + F i (37)
with UEs being placed uniformly at random and d i (>= 35 m) represents the distance of UE i from the BS. F i ∼ N (0, 10) is the logarithm of the shadow fading between UE i and BS. Also, let ϕ i be the geographical angle to UE i as seen from the BS. Cluster s is characterized by the randomly generated nominal angle-of-arrival ϕ i,s ∼ U[ϕ i -40 • , ϕ i + 40 • ] and the angles of the multipath components are Gaussian distributed around the nominal angle with standard deviation σ 2 ϕ = 10 • . Fig. 1 plots the sum SE as a function of the total transmit power defined as ρ T (in dBm) with M = 100 and K = 10. Comparisons are made with a classical MaMIMO system with MR precoding and power allocated through Algorithm 1 with ρ c fixed to 0. As seen, RS improves the sum SE significantly for values of ρ T higher than 5 dBm. Moreover, the sum SE with RS does not saturate at high ρ T values. This in contrast to what happens without RS, due to pilot contamination. Fig. 2 illustrates the sum SE as a function of number of antennas, M , with K = 10 and transmit power ρ T = 20 dBm. We observe that the RS scheme does help to mitigate the pilot contamination effect for a finite number of antennas.

Finally, in Fig. 3 we report the sum SE as a function of K with M = 100 and ρ T = 20 dBm. As K increases, the gain provided by RS decreases. The larger K, the lower the common rate since the common message has to be decoded by all UEs. This issue can be solved by using HRS approach as in [START_REF] Dai | A Rate Splitting Strategy for Massive MIMO with Imperfect CSIT[END_REF]; this is an interesting topic left for future work.

V. CONCLUSIONS

This paper focused on a single-cell MaMIMO system in which all the UEs use the same pilot signal in the training phase. To deal with the reduced channel estimation quality, caused by pilot contamination, a single layer RS approach was proposed and shown to improve the SE at high SNR values. However, we remark that much remains to be done, for e.g. c . For simplicity, we drop the iteration index t. We begin by rewriting the SE of UE k as (by explicating its dependence from ρ k )

SE k (ρ k ) = τ d τ log 2 NUM k (ρ k ) DEN k (ρ k ) = τ d τ (log 2 (NUM k (ρ k )) -log 2 (DEN k (ρ k ))) (38)
where

DEN k (ρ k ) represents the denominator of γ k in (12) while NUM k (ρ k ) = DEN k (ρ k ) + ρ k |E{h H k w k }| 2 . Observe that -log 2 (DEN k (ρ k )) is a non-concave function of ρ k .
By linearizing it around a tentative value ρ k , the following approximation is obtained:

log 2 (DEN k (ρ k )) ≈ E{|h H k w k | 2 } -|E{h H k w k }| 2 DEN k ( ρ k ) α k (ρ k -ρ k )
where the terms independent of ρ k have been neglected for simplicity. Similarly, we can rewrite SE i as

SE i (ρ k ) = τ d τ (log 2 (NUM i (ρ k )) -log 2 (DEN i (ρ k ))) . ( 39
)
By linearizing both terms around ρ k

log 2 (NUM i (ρ k )) ≈ E{|h H i w k | 2 } NUM i ( ρ k ) (ρ k -ρ k ) (40) log 2 (DEN i (ρ k )) ≈ E{|h H i w k | 2 } DEN i ( ρ k ) (ρ k -ρ k ) (41)
we obtain the following approximation for SE i (ρ k )

SE i (ρ k ) ≈ τ d τ E{|h H i w k | 2 } NUM i ( ρ k ) - E{|h H i w k | 2 } DEN i ( ρ k ) ζi (ρ k -ρ k ). σ (1) k = E{|h H k w k | 2 } σ 2 + ρ c E{|h H k w c | 2 } -|E{h H k w c }| 2 + K i=1,i =k ρ i E{|h H k w i | 2 } (45)
Following the same approach for the SE of the common message yields

SE c (ρ k ) ≈ τ d τ E{|h H lmin w k | 2 } NUM c,min ( ρ k ) - E{|h H lmin w k | 2 } DEN c,min ( ρ k ) ζc (ρ k -ρ k )
where NUM c,min (ρ k ) = DEN c,min (ρ k ) + ρ c |E{h H lmin w c }| 2 and DEN c,min (ρ k ) represents the denominator of γ lmin,c in [START_REF] Dai | A Rate Splitting Strategy for Massive MIMO with Imperfect CSIT[END_REF]. Putting all the above together, an approximation of the sum SE in [START_REF] Boyd | Convex Optimization[END_REF] is

SE(ρ k ) = τ d τ log 2 (NUM k (ρ k )) -σ (2)
k (ρ k -ρ k ) (42)
where σ

(2)

k = ζ c + α k + K i=1,i =k ζ i . ( 43
)
Taking the derivative of its Lagrangian (obtained after adding the power constraint in ( 20)) and equating it to zero yields

E{|h H k w k | 2 } NUM k (ρ k ) -σ (2)
k -µ = 0 (44) from which one obtain (21) in the text, with σ

1/2 k ) H R i R -1 k Φ 1/2 k . By recalling that h i = R i R -1 k h k yields E{ h k h H k h i h H i } = Φ 1/2 k E{cc H B ik cc H }(Φ 1/2 k ) H . ( 48
Putting the above results together yields E{ h k h H k h i h H i } = tr{B ik }I + diag (B ik ) + B ik .

Fig. 1 .Fig. 2 .

 12 Fig. 1. Sum SE versus transmit power, with M = 100 and K = 10.

Fig. 3 .

 3 Fig. 3. Sum SE versus number of UEs with M = 100 and ρ T = 20 dBm.extension of the current work to a multi-cell setting and the design of an efficient RS message scheme to mitigate the intercell and intra-cell interference.

k

  . A similar approach for ρ c yieldsE{|h H lmin w c | 2 } NUM c,min (ρ c ) -σ (2)care omitted for space limitation. Solving (46) yields (22) in the text, where σ c, with c ∼ CN (0, I) and define the deterministic matrix B ik = (Φ

)

  It then follows thatE {[cc H B ik cc H ] mn } = ] lj E {c m c * l c j c * n } . (49)If m = n, then (49) is always zero except for l = j:M l=1 [B ik ] ll E |c m | 2 |c l | 2 = 3[B ik ] mm + M l=1,l =m [B ik ] ll(50) where we have taken into that E |c m | 4 = 3. If m = n, then (49) is always zero except for l = m and j = n [B ik ] mn E {c m c * m c m c * n } = [B ik ] mn .

  5 (or some very large value) and µ

		(0) l	= 0.
	2: repeat
	3: 4: 5:	for k = 1 to K do compute σ (1,t) k and σ use µ t to update p t k in (21) (2,t) k
	6: 7: 8:	end for compute σ c (1,t) use µ t to update p t and σ c (2,t) c in (22)

9:

EURECOM's research is partially supported by its industrial members: ORANGE, BMW, Symantec, SAP, Monaco Telecom, iABG, and by the projects DUPLEX (French ANR), MASS-START (French FUI) and EU ITN project SPOTLIGHT. L. Sanguinetti was supported by the University of Pisa under the PRA 2018-2019 Research Project CONCEPT and by the Italian Ministry of Education and Research (MIUR) in the framework of the CrossLab project (Departments of Excellence).