Cortial Lucas

Nguyen Catherine
email: catherine.nguyen@inserm.fr

Montserrat Moliner
email: antonimontserrat@outlook.com

Blin Olivier
email: olivier.blin@ap-hm.fr

Managing rare diseases: Examples of national approaches in Europe, North America and East Asia

Keywords: Rare Diseases, National Plans, Europe, Expertise, Reference, Networks

Around 4% of the global population suffer from a rare disease. Apart the medical aspect, economical, organisational, and political approaches remain key aspects. We review here the principal specific national initiatives and organisations in Europe, North America and Asia. Thereafter, we propose the outlines of possible optimal approach, inspired by the successes of the individual national organisations. This multi-scale analysis, based on key performance indexes and strengths, should be taken into account in the definition of large scale multi-national rare diseases programs, such as European Joint Program on Rare Diseases.

INTRODUCTION

Three hundred million people worldwide -4% of the global population -suffer from a rare disease. This number includes 30 million patients in Europe. The first countries to implement a healthcare policy on orphan diseases were Japan with the Nan-Byo Consortium 1 in 1972, and the United States with the Orphan Drug Act in 1983. France followed in 2004 with its National Plan for Rare Diseases, setting an example for other European countries and Canada (Figure 1). Today, among European Union Member States, only Malta and Sweden have no plan in place for rare diseases. This study describes the national State-coordinated transdisciplinary approaches in place to support patients with rare diseases and oversee their access to healthcare (Table 1) in Europe, North America and Asia. It then seeks to identify efficient initiatives that could be replicated in order to bring together all the stakeholders in the rare disease health system. An approach of this type would benefit patients and accelerate the development of breakthrough innovations. Given that the purpose of this paper is to study the initiatives and organisations specific to each country, it will not look at the organisation of this sector at European Commission level. For the same reason, it does not cover the structures and organisations set up at association level. c) The se-Atlas platform for healthcare providers and patients. This platform stores data that will help to improve diagnosis as well as to facilitate innovations (3). In February 2020, the BMBF launched CORD-MI, a two-year project with funding of €6 million. Led by the Institute of Health in Berlin, the purpose of the project is to collect and structure data relating to rare diseases as part of a digital network. This will then be incorporated into the BMBF Medical Informatics Initiative (MII). The MII was set up to develop an infrastructure for the integration of all patient clinical data, regardless of the field. By late 2022, the BMBF will have invested around €180 million in the MII [START_REF]About the initiative[END_REF]. d) Alongside these commitments, the BMBF has announced plans from 2021 to invest €3 million every year in a new centre for neurodegenerative diseases (DZNE) in Ulm. This takes the number of DZNEs to ten2 . e) Further, in January 2021, the BMBF announced the launch of an independent Fraunhofer research centre dedicated to Translational Medicine and Pharmacology (Fraunhofer ITMP) (5) [START_REF][END_REF]. With three research sites in Frankfurt, Hamburg and Göttingen, the Fraunhofer ITMP will develop innovative solutions in collaboration with industry and with the support of government funding. Its objectives are to transfer innovations to medical applications and to train future leaders in the field of biomedical research. Preclinical and clinical research will be carried out on the basis of interdisciplinary collaboration3 .

Between 2003 and 2018, the BMBF invested €107 million in the field of rare diseases 3 .

SPAIN: SpainUDP is Spain's main programme for undiagnosed rare diseases.

In Spain, the rare diseases programme is led by the Carlos III Health Institute (ISCIII).

The ISCIII is a public health research institute founded in 1986 to promote research into biomedicine and health sciences and to develop and steer the scientific and technical strategies of the Spanish national health system. Located on two campuses in Madrid, it helps to promote research and innovation in the health sciences sector. It also provides scientific and technical services and educational programmes (7) (8).

Within the ISCIII, the Institute for Research into Rare Diseases (IIER) is studying the mechanisms underpinning the origin and progression of rare diseases and congenital malformations. The goal of the research projects is to develop innovative treatments.

The IIER is developing several translational programmes:

a) The Undiagnosed Rare Diseases Programme -SpainUDP -was set up to address the high number of consultations for undiagnosed rare diseases. SpainUDP offers patients a multidisciplinary approach. In 2014, the SpainUDP programme became part of the Undiagnosed Diseases Network International (UDNI), a global consortium of research centres and institutes.

The partners and tools of SpainUDP are listed below.

b) The Rare Disease Patient Registry (SpainRDR) provides healthcare professionals, researchers, patient groups and their families with extensive information on the number and geographical distribution of people affected by rare diseases in Spain.

SpainRDR is co-funded by the ISCIIII, the Regional Health Departments and the Ministry of Health and Social Services. Its objective is to increase the visibility of rare diseases and support decision-making in order to ensure adequate health planning and resource allocation. SpainRDR works with hospitals, a network of scientists, patient organisations, and the pharmaceutical industry.

c) The National Biobank for Rare Diseases (BioNER) provides the scientific community with a catalogue of biological samples associated with clinical and epidemiological information on these diseases.

d) The Human Genetic Area (AGH) includes a Genetic Diagnosis Service offering services for the diagnosis of the various genetic diseases identified. This centre is equipped with state-of-the-art technologies such as highthroughput sequencing and large-scale data analysis. A CRMR brings together a highly specialised hospital-based team with proven expertise in a particular rare disease or group of rare diseases. Each CRMR develops activities in the fields of prevention, healthcare, teaching and research. The medical team includes paramedical, psychological, medico-social, educational and social skills experts. It also forms partnerships with patient organisations. A CRMR is a multi-site entity. It is an expert referral centre of a regional, interregional, national and even international scope.

FRANCE: a pioneer in

b) 23 National Rare Disease Health Networks (FSMRs)

The FSMRs were set up in 2014-2015, as part of PNMR2. They each cover a broad but coherent field of rare diseases. Each FSMR groups the stakeholders involved in a given area, including caregivers, researchers, patient representatives and manufacturers. Each d) The BaMaRa national register, developed by the BNDMR (National Data Bank for Rare Diseases), has now reached the deployment stage. This process involves staff training, data entry checks, data exhaustiveness checks, administrative follow-up and harmonisation of coding.

Connected to the National Health Data System (SNDS), the BaMaRa register aims to develop medical-economic knowledge on rare diseases and to conduct epidemiological studies in order to better assess the prevalence and incidence of rare diseases.

e) Orphanet

Created in France in 1997, Orphanet was initially developed to gather the scarce information concerning rare diseases, to improve the diagnosis and the treatment of the patients. Orphanet has now become a network of 41 countries throughout Europe and the world and represents the reference source of information on rare diseases.

The objectives of Orphanet are to provide a common language in rare diseases with the Codes ORPHA to have a better comprehension between the actors; provide high quality data and expertise on rare diseases to allow an equal access to knowledge, and guide users in the field of rare diseases; contribute to a better understanding of rare diseases.

f) The French national programme on Rare Disease Cohorts, RaDiCo. This programme aims to establish national and international cohorts of patients with rare diseases. The objective is to better describe the diseases, establish phenotype/genotype correlations, understand the physiopathology, identify new therapeutic targets and to evaluate the medico-economic and societal impact of the diseases.

PNMR3 (2018-2022) has a budget of €778 million, most of which (€597 million

) is dedicated to the CRMRs [START_REF]Un 3e plan maladies rares doté de 780 millions pour améliorer le diagnostic et la prise en charge | Le Quotidien du Médecin[END_REF]. The initiatives in place include collaborative work with the France Genomic Medicine 2025 plan (11) to facilitate access to very-high-throughput sequencing platforms, in accordance with a scale based on the clinical situation.

Finally, France has internationally renowned research institutes involving public and private partners (12):  the Imagine Foundation was set up to conduct a project of medical and scientific excellence by organising, structuring and developing research, healthcare and teaching activities in the field of genetic diseases affecting both children and adults;  I-Motion is a clinical trial centre dedicated to children with neuromuscular diseases;  the Institute of Myology promotes the existence, recognition and growth of myology as a discipline in its own right. The muscular system is a true model of innovation for scientific and medical research;  The NeuroMyoGene Institute is dedicated to studying the physiopathology of the muscular and nervous system, with an opening on clinical and translational research;  The French Foundation for rare diseases is dedicated to the coordination and acceleration of research on all rare diseases. The Cell and Gene Therapy Catapult (CGT Catapult) [START_REF][END_REF] was set up to drive progress in gene and cell therapy at UK and international level. Its mission is to: In 2019, the CIHR announced an investment of €680 million in rare diseases [START_REF] Canada | National Strategy for Drugs for Rare Diseases Online Engagement -Closed consultation[END_REF]. Primarily dedicated to ensuring patient access to healthcare, thus sum will be divided into two equal payments in 2022 and 2023.

 conduct

UNITED STATES:

In 2012, the USA put in place an accelerated registration process for ultra-rare diseases [START_REF]Developing Products for Rare Diseases & Conditions[END_REF].

As is often the case in the US, resources are provided by the national i. One of the activities of TRND is to seek new applications for compounds that have already passed clinical trial phases [START_REF]National Center for Advancing Translational Sciences[END_REF].

8 Omics technologies: a transdisciplinary approach (chemistry, physics and information technology) based on a systematic analytical approach to the content of living cells at the molecular level e) The Genetic and Rare Diseases information Center (GARD) is a website set up to provide information that can be easily understood by all audiences [START_REF]Genetic and Rare Diseases Information Center (GARD) -an NCATS Program | Providing information about rare or genetic diseases[END_REF]. GARD is funded by the NCATS and the National Human Genome Research Institute (NHGRI), another department of the NIH. It is connected to Orphanet.

f) The Rare Diseases Registry Program (RaDaR) is a website dedicated to patients, their families, researchers and the public [START_REF]Rare Diseases Registry Program (RaDaR)[END_REF]. It is currently in Phase 1. The plan for rare diseases has two main strands: Decision-making policy for rare diseases and practical applications of the measures adopted [START_REF]Practical Research Project for Rare / Intractable Diseases | Japan Agency for Medical Research and Development[END_REF]. The investment priorities are as follows:  pathology and mechanism of rare diseases;

9 The exome is the part of the genome most directly linked to the phenotype of the organism, to its structural and functional qualities.

10 European journal of human genetics 5/7/2017

 repositioning of existing drugs and medical devices for clinical use in patients suffering from rare diseases;  improved clinical practices (training) to optimise the effectiveness of healthcare;  use of innovative omics technologies;  implementation of the Initiative on Rare and Undiagnosed Diseases (IRUD;  creation of a data registry compatible with international standards;  development of innovative drugs using gene therapy or nucleic acids;  production of vectors for the development of new therapies;  research into the links between medicine, nutrition and the physiological aspects of rare diseases.

In line with these priorities, AMED has put in place the following approach:

a) The Initiative on Rare and Undiagnosed Diseases (IRUD) implemented by AMED in 2015, is part Japan's Rare Disease Plan [START_REF] Adachi | Japan's initiative on rare and undiagnosed diseases (IRUD): towards an end to the diagnostic odyssey[END_REF]. IRUD is a national consortium that brings together patients, physicians and researchers, as part of the process below:

b) Launched by AMED in February 2017 and funded by AMED and the Ministry of Health, Labour and Welfare (MHLW), the National Platform for the Rare Diseases Data Registry of Japan (RADDAR-J) collects and structures data from research projects into rare diseases [START_REF]Rare Disease Data Registry of Japan[END_REF]. The experts involved in the IRUD consortium are connected to this registry, which includes data relating to over 2,000 undiagnosed patients11 . The database is designed to be linked to international platforms in compliance with the standards set out by the IRDIRC consortium as part of the Automatable Discovery and Access-Matrix (39). In the longer term, AMED is seeking a connection with the international platform of the IRDIRC consortium, scheduled for launch in 2027.

c) The Office of Pharmaceutical Industry Research (OPIR) was set up in January 1999 by the Japan Pharmaceutical Manufacturers Association (JPMA) 12 . OPIR analyses the internal and external environment of the Japanese pharmaceutical industry (in all fields), identifying the mid-and long-term issues, with a view to submitting recommendations to policy makers [START_REF]Office of Pharmaceutical Industry Research[END_REF]. 2. The description of the seven approaches coordinated by governments to support and treat patients with rare diseases shows a number of strengths (Table 2) but a number of divergences too. The approach adopted by each country reflects its history, the commitment of the State, the organisation of the healthcare and biomedical research systems, national legislation and the culture in terms of innovation and public-private partnerships.

Based on a few key criteria necessary for the development of a national system, we have compared the organisation of each country and highlighted a number of specific characteristics.

a) The government steers and funds all the programmes, sometimes in association with private partners, in Germany with the ASCHE patient organisation and in Canada with the Genome Canada structure.

b) Significant disparities can be seen in the establishment of centres of expertise:  in Canada and the United States, multidisciplinary research consortia are funded by the State, whereas centres of expertise are housed in hospitals that are mainly private;  in Spain, the SpainUDP translational programme for undiagnosed diseases is coordinated with the 19 centres of expertise;  in France, 400 reference centres look after patients and organise their healthcare, while 23 Health Networks for Rare Diseases bring together the players involved in one or more diseases;  in Japan, the centres of expertise are associated with clinical investigation centres;  in the United Kingdom, the government prioritises investments in patient care and empowerment.

Finally, the number of centres is not proportional to population density. c) In terms of genomics, Canada and Japan are pioneers, Japan as early as 1972 with the Nan-Byo programme, and Canada from 2000 with Genome Canada.

International collaboration is a key benefit that is very much a part of the system in North America, Spain and Japan.

d) The state of development of national registry programmes varies considerably. Canada, Spain, the United States, France and Japan have operational tools Canada, Spain and the United States are connected to the Undiagnosed Diseases Network International (UNDI)13 consortium, sharing their data for research purposes.

e) Depending on the country, industrial partners may be involved at different stages of the plan:  In Germany and the United Kingdom, the public-private centres specialised in a particular field, such as the Fraunhofer ITMP and the CGT Catapult, seek to promote industrial partnerships;  In Canada, private partners are associated with the State from the outset;  In Spain and France, private partnerships are mainly based on public university hospital campuses. In France, the launch in 2010 of University Hospital Institutes (IHU), jointly funded by government and industry, encourages public/private partnerships.  In the United States, the TRND co-funds the development of drugs with manufacturers until an IND is filed.

Moreover, 80% of university research (all fields) is funded by private partnerships;  In Japan, OPIR makes policy recommendations, supports research projects and sets up joint projects with third parties.

II. OUTLINING AN OPTIMAL APPROACH

Inspired by the successes of the organisations above, an optimal approach could be suggested (Table 3), avoiding pitfalls. In this approach:  The government and private partners (on a pro rata basis according to their participation) would steer and fund the system, in the same way as with the Genome Canada model.  The organisation of the centres of expertise would be based on the French model with increased financial support for patient empowerment (UK).  In terms of genomics, the emphasis would be placed on international collaboration and information sharing, with involvement in international consortia such as Care4Rare.  The ongoing development of national registries would be based on international standards, for connection to the European and international platforms currently under development.  The involvement of manufacturers from the outset in order to favour partnerships (Canada).

Furthermore, the results of the CGT Catapult in the United Kingdom underline the relevance of a specialised system dedicated to rare diseases. Finally, the business model of the French IHU has produced real economic benefits and deserves further development.

Figure 1 :

 1 Figure 1: Implementation of national plans and strategies dedicated to rare diseases and their management

  FSMR pursues three goals: better healthcare; research and education; training and information. c) At the same time, the French National Institute for Health and Medical Research (INSERM) is coordinating the European Joint Programme on Rare Diseases (EJPRD), a flagship programme co-funded by the EU, with the purpose of structuring the Rare Disease ecosystem at European and international levels.

4 .

 4 UNITED KINGDOM: an ambitious programme of government investment launched in 2012 has created a highly structured and efficient system that also includes private-sector partners. a) Within the National Health Service (NHS), the Highly Specialised Services (HSSs) are a network of 76 centres specialising in rare diseases (13). The HSSs place the emphasis on patient care and empowerment, with each centre treating no more than 500 patients per year. For the 2018/2019 financial year, the NHS provided €19.8 billion for the running of the 76 centres 4 . The governance of the Rare Diseases Advisory Group is described below. Source: Rare Diseases Advisory Group (RDAG) website b) At the same time, 13 genomic medicine centres were set up in 2019 with the participation of 85 hospitals. Their aim is to sequence 50,000 genomes by 2023. These initiatives are coordinated with the Cell and Gene Therapy Catapult launched in April 2012.c) The Catapult Centre[START_REF]The Catapult Network[END_REF], The nine Catapult Centres are private, independent organisations. Like Germany's Fraunhofer research centres, they have a mixed-funding model based on an equal split between:  grants awarded to each centre by Innovate UK 5 , primarily for purposes of infrastructure and the development of expertise. This grant can total to up to €12 million per centre;  R&D contracts and services, financed by the private sector;  grants obtained through calls for proposals from funding agencies.



  e. Federal government. The National Institutes of Health (NIH) play a key role in the field of rare diseases. They are structured as follows: a) The National Center for Advancing Translational Sciences (NCATS) is one of the 27 institutes making up the NIHs. It comprises seven divisions, one of which is dedicated to rare diseases. The grant received by the NIH in 2020 totalled €722 million, or 0.2% of its budget (€35 billion) (26). b) The Office of Rare Diseases Research (ORDR) is the steering organisation of the NCATS Rare Disease Division (27). Founded in 2003 by the NIH, the Rare Diseases Clinical Research Network (RDCRN) is made up of 20 consortia specialising in clinical research, bringing together scientists, clinicians, patients and associations, i.e. 2,600 experts at present (28) (29). Since its founding, the RDCRN has studied over 280 diseases and recruited around 40,000 patients for clinical trials. c) The Data Management and Coordinating Centre (DMCC) is based at the Cincinnati Children's Hospital. It collects, homogenises, manages and shares RDCRN data in order to increase dissemination for research purposes. d) The Therapeutics for Rare and Neglected Diseases (TRND) programme is involved from pre-clinical development through to submission of an IND to the FDA. It brings together NIH academic research teams and private partners to accelerate the development of new molecules.

: a forerunner in the field of detection, Japan uses innovative technologies for whole genome and exome sequencing 9 .

 9 g) The Undiagnosed Diseases Network (UDN) is funded by the National Human Genome Research Institute (NHGRI), the NCATS and the National Institute of Neurological Disorders (NIND) (33). Its 12 sites bring together clinicians and scientific experts pooling their efforts to accelerate gene identification. Since its launch in 2015, 200 cases have been diagnosed. The diagram below shows how the NIH is organised for the study of rare diseases. The university is the basic entity in research and development, whether clinical (hospitals and medical schools) or fundamental (research laboratories). Most funding is private, with the contribution from the federal government making up around 20% of the total. Since 2012, the Food and Drug Administration (FDA) has simplified the regulations governing clinical research in the field of rare diseases. Special terms are made available to manufacturers: financial and tax support and accelerated procedures for bringing new products to market. Implemented in 2014, the Japanese plan for Rare and Undiagnosed Diseases succeeded to the rare disease research programme set up in 1972 by the Japan Agency for Medical Research and Development (AMED) under the name "Nan-Byo Research" (34) (35). The Ministry of Health is a key player in the new plan, led by AMED. It involves 34 clinical investigation centres and four analysis centres, and funds around 500 doctors and 50 coordinators belonging to the National Coordination Committee 10 .

1 .

 1 OPIR has developed an economic and social structure that contributes to the rapid development of medical innovations. OPIR is an open laboratory for research, policy recommendations and the development of human resources. The structure supports its own research projects as well as collaborating with third parties, as shown in the diagram below. This study describes the plans put in place by the governments of the seven countries studied. For this reason, it does not mention the European Union or patient organisations, despite their essential role.  The European Union has been active in this field since 2008, making rare diseases a priority, with the creation of the European Committee of Experts on Rare Diseases (EUCERD), and the funding of shared tools for Member States, such as the European Reference Networks (ERNs), the European Joint Programme on Rare Diseases (EJPRD), the European Platform on Rare Diseases Registration (EU RD Platform) and the European Rare Disease Research Coordination and Support Action Consortium (ERICA) in 2021. A supplementary paper could be written on the commitment of the European Union, its work in bringing players together, and the diversity of the programmes funded in the field of rare diseases.  Patient organisations provide moral, practical, financial, social and legal support to patients and their families. Looking beyond this input, they contribute to research funding and are important partners for health professionals, decision-makers and pharmaceutical laboratories.

GERMANY: Since 2020, Germany has stepped up its commitment to research in the field of rare diseases. a) The German National Action League for People with Rare Diseases (NAMSE) was

  set up by the Federal Ministry of Health (BMG) in 2010. It is jointly coordinated by the BMG, the Federal Ministry of Education and Research (BMBF) and the National Alliance for Chronic Rare Diseases (ACHSE)[START_REF]'Europe et des affaires étrangères[END_REF]. NAMSE is responsible for the implementation, coordination and follow-up of the rare disease plan. ACHSE brings together 28 partners, including the Federal Ministry of Labour and Social Affairs, the German Medical Association and the National Health Insurance Association (2). As part of this organisation, the German plan for rare diseases came into force in August 2013.

	APPROACHES TO RARE DISEASES
	A. EUROPEAN COUNTRIES
	1.
				Poland
			Luxembourg
			Austria
			Canada
			Croatia
	2008 : Creation of the European Union Committee of	Denmark
	Rare Diseases	Estonia
	2000: European regulations on drugs for orphan	Finland
	diseases		Ireland
	1983: Orphan Drug Act in the United States	Italy
			Romania
			Germany
			Belgium
			Hungary
			Latvia
			Netherlands
			United Kingdom
		Cyprus
		Lithuania
		Slovakia
		Slovenia
		Bulgaria
		Spain
		Czech Republic
		Greece
		Portugal
		France
		USA
	Japan
	1972	1983 2004 2008 2009/2010 2011	2013 2014/2015 2018	2021

1 Nan-Byo means "difficult+illness" in Japanese b) Thirty-six centres of expertise collaborate with a network of 11 research centres, led by a Board of Directors.

Europe with the first Rare Disease Plan in 2004. Since this date, France has maintained its leadership.

	Rare diseases are a priority for France.
	The Public Health Act of 2004 led to the creation of a national plan for rare diseases (PNMR), which has been
	renewed twice (PNMR2 and PNMR3). As part of PNMR3, the State oversees healthcare and research activities. For
	these purposes, it is represented by the Ministry of Solidarity and Health and the Ministry of Higher Education,
	Research and Innovation (9).
	These initiatives have made it possible to accelerate patient care, boost teaching and training and drive progress in
	research and innovation. They are organised as follows (9):

a) nearly 400 Reference Centres for Rare Diseases (CRMRs) with a coordinating site and one or more constituent sites.

Rare Diseases Models and Mechanisms (RDMMs) is

  CGT Catapult has 64 public and private partners including UK universities and manufacturers(40 in all), as well as a number of British and international organisations. The main achievements of the CGT Catapult since 2013 are as follows:  Nearly €3 billion in industrial investment  26 production units meeting GMP standards  47 companies on campus  Over 90 therapies at the development stage  60% of therapies developed in the UK in this field are based on a collaboration with the CGT Catapult  A 45% increase in clinical trials (20) brings together Canadian clinicians, bioinformaticians, scientists and researchers working on the care of patients with rare diseases in Canada and around the world. Led by the Children's Hospital of Eastern Ontario (CHEO) and the Ottawa Research Institute in Ottawa, it has 21 academic centres across Canada with 200 physicists and 100 scientists. d) C4R-SOLVE (20) uses innovative new sequencing technologies, based on omic approaches 8 , to build an infrastructure with the tools necessary to improve the diagnosis of rare diseases worldwide. e) In 2014, the PhenomeCentral system brought together research and diagnosis in a single portal. Developed by the Centre

	The Achievements in 2019/2020
	 108 projects
	 €7.8 million investment in gene and cell therapy
	 28 phase III clinical trials
	 10 spin-outs
	 7 drugs reimbursed for patients
	 €349 million invested by industry in rare diseases
	 Over 3,000 jobs
	clinical trials;
	 deliver an infrastructure equipped to the standards required by Good Manufacturing Practices (GMPs);
	 provide expertise on regulatory issues (shorter start-up times for clinical trials);
	 set up national and international collaborative projects;
	 gain access to grants and funding.

Located in London, the CGT Catapult has a process development laboratory with dedicated facilities for clinical trials and reproducibility studies. Further, the CGT Catapult site in Stevenage, which opened in 2018 for a cost of €92 million (of which €4 million funded by the European Union), is an industrial-scale manufacturing centre. This centre is dedicated to manufacturing gene and cell therapy products in collaboration with industry (16).

B. NORTH AMERICA

5. CANADA: globally renowned expertise in gene identification

Although Canada only developed a National Plan for Rare Diseases in 2015, it has developed internationally recognised expertise in identifying the genes involved in rare diseases. a) Set up in 2000, Genome Canada is a not-for-profit organisation supported by the Canadian Institute of Health Research, Institute of Genetics (CIHR IG) (17). It coordinates federal and provincial studies through six genome centres with the following missions:  develop ties between public and private players.  invest in major scientific and technological projects. In 2017, for example, Genome Canada invested €7 million in the Silent Genome Project led by researchers from the University of British Columbia, whose purpose is to create a database of genetic variations in indigenous populations around the world 6 (18). b) The a national network coordinating research projects to gain a clearer understanding of the biology underlying rare diseases in children. It is funded by Genome Canada (19). Present across Canada, the RDMMs are managed by the Maternal Infant Child & Youth Research Network (MICYRN), which brings together 23 networks of experts and 20 hospitals and academic research centres specialising in maternal and childhood diseases. The diagram below shows the timeline in research studies conducted by the RDMM as part of the FORGE programme, 7 which became the international research consortium Care4Rare in 2013. c) Care4Rare (C4R)

for Computational Medicine at the Hospital for Sick Children in Toronto, this

  connected system is able to secure data[START_REF] Buske | PhenomeCentral: A Portal for Phenotypic and Genotypic Matchmaking of Patients with Rare Genetic Diseases[END_REF]. It provides a global link between patients with identical genotypes and phenotypes, as well as clinicians and scientists working on similar cases. PhenomeCentral is funded by the CIHR IG, the Ontario Genomics Institute, and the Natural Sciences and Engineering Research Council (NSERC). It is a partner of the NIH, RD-Connect (an EU-funded integrated platform) and the International Rare Diseases Research Consortium, funded by the US and the EU. f) Ontario Genomics is a not-for-profit organisation supported by the CIHR IG and the government of Ontario. It brings together companies, researchers, policy makers and funders to generate funding and accelerate the development of genomics[START_REF] We | Ontario Genomics[END_REF].

g) The Canadian Organization for Rare Diseases (CORD) was set up by the Canadian Parliament in 2015 (23).

Its main priorities are to:  implement a regulatory framework;  put in place appropriate funding for equitable patient access to treatment;  identify centres of excellence bringing together research, patients and care;  set up a programme for the screening of newborn babies.

Table 1 : A comparison of approaches based on specific key criteria

 1

		STEERING	SOURCE OF	CENTRES OF EXPERTISE	GENE IDENTIFICATION	NATIONAL REGISTER
			FUNDING
	GERMANY	State + patient	-State	36 centres working with 8	RD-Connect programme	. Se-Atlas: 8,000 genes
	2013	organisation (ASCHE).	-Patient organisation (ASCHE).	research centres + 10 centres for NDD.	(EU funding 2018-2022): -> Register relating to research Biobanks and the creation of an EU platform.	. CORD-MI project €160 million invested between 2016 et 2021.
	SPAIN	State	State	19 centres of expertise	. The Human Genetic Area	SpainRDR
	2009			Spain UDP consortium	(AGH), for diagnosed	Funding from the State
					diseases.	/ Regions and
					. SpainUDP for undiagnosed	collaboration with
					diseases.	private partners.
	FRANCE	State	State	. 400 reference centres for	France Genome 2025	Deployment of the
	2004			rare diseases (CRMRs);	programme	BaMaRa register.
				. 27 National Rare Disease	High-throughput platform
				Health Networks (FSMRs).
	UNITED	State	State	76 centres	13 centres, involving 85	-
	KINGDOM			Emphasis on patient care	hospitals.
	2012			and empowerment.	Target 2023: sequencing of
					50,000 genes.
	CANADA	State:	Genome Canada,	20 mother and child	. 6 genomic centres	PhenomeCentral:
	2015	Canadian	funded by federal	research centres.	. Care4Rare, international,	national register with
		Organization	government, the		transdisciplinary	international scope
		for Rare	provinces and		consortium,with 21 sites.
		Diseases	manufacturers.
		(CORD).
	USA	State: NIH	State	Network of 20 consortia,	Undiagnosed Diseases	Data Management and
	1983			with 350 sites in the US and	Network (2015), with 12 sites.	Coordinating Center
				22 worldwide.		(DMCC)
	JAPAN	State	State	34 clinical investigation	Consortium for the Initiative	RADDAR platform
	1972			centres	on Rare and Undiagnosed	connected to
				4 analysis centres.	Diseases (IRUD).	international platforms.

Source: French Embassy in Germany, Scientific Department, 4/5/2018.

The Fraunhofer Society is a German organisation dedicated to applied research. Funded by the government and private partners, it has 74 institutes across Germany. It is second only to the United States in its field of activity.

State funding and innovation body

Northern health indigenous health 28/2/2018

FORGE: Programme set up to identify the genes causing rare diseases in Canada

Source: European Journal of Human Genetics 5/7/2017

The Japan Pharmaceutical Manufacturers Association (JPMA) represents the research-based pharmaceutical industry in Japan. The JPMA has 74 members including 20 overseas affiliates.

UNDI partner countries: Australia, Austria, Belgium, Bulgaria, Canada, Hungary, India, Italy, Korea, Spain, Sri Lanka, Sweden, Thailand, Turkey, USA.

Acknowledgements

OrphanDev (https://www.orphan-dev.org/) is a French national network funded by F-CRIN aiming to bring solutions for the patients suffering with rare diseases. The authors would like to thank F-CRIN for their support.

JAPAN

The Office of Pharmaceutical Industry Research (OPIR) makes recommendations to political decision-makers. OPIR collaborates with academic research.

-Strong State commitment.

-Use of innovative technologies for sequencing.

-Register/Platform compatible with international standards.

FUNDING & COORDINATION

State, industry, patient organisations.

Germany: Patient organisation (ASCHE).

Canada: Genome Canada. Japan: Office of Pharmaceutical Industry Research (OPIR).

CENTRES OF EXPERTISE

REGISTER

Use of international standards such as the Automatable Discovery and Access Matrix defined by the IRDRIC.

Spain: SpainRDR. Canada: PhenomeCentral. USA: Genetic and Rare Diseases information system (GARD). Japan: IRUD data centre.

PUBLIC/PRIVATE PARTNERSHIP

(1) Agency for innovation and technology transfer, specialising in rare diseases, funded by the State and manufacturers.

(2) Centre for the production of innovative products accessible to manufacturers.

Conflict of interest

All authors declared that there are no conflicts of interest

Ethical approval and consent to participate Not applicable

Consent for publication Not applicable

Bibliography :


Sylvain Castagnos
email: sylvain.castagnos@loria.fr

Being Diverse is Not Enough: Rethinking Diversity Evaluation to Meet Challenges of News Recommender Systems

Keywords: CCS CONCEPTS, Information systems → Recommender systems, Information retrieval diversity, • Human-centered computing → Empirical studies in collaborative and social computing News recommender systems, User modeling, Diversity impact, Dataset analysis, Polarization, Filter bubbles

Modern societies face many challenges, one of them is the rise of affective polarization over the last 4 decades. In an attempt to understand its reasons, many researchers have questioned the role of Social Media in general, and Recommender Systems (RS) in particular, on the emergence of these extreme behaviors. Diversity in News Recommender Systems (NRS) was quickly perceived as a major issue for the preservation of a healthy democratic debate. However, after more than 15 years of research in Artificial Intelligence on the subject, the understanding of the real impact of diversity in recommendations remains limited. Through a case analysis on the well-known MIND dataset, we propose a critique of the diversityaware recommendation and evaluation approaches, and provide some take-home messages related to the need of adapted datasets, diversity metrics and analytical methodologies.

INTRODUCTION

Recommender Systems (RS) were first intended to direct users' attention towards resources that are more suitable to their needs or preferences [START_REF] Resnick | Recommender systems[END_REF]. This kind of approach was successful in many domains, such as e-commerce, VOD or search engines. However, various studies showed that user satisfaction is not restricted to the traditionally used precision measure, and McNee et al. have paved the way for a new generation of diversity-oriented RS [START_REF] Sean M Mcnee | Being accurate is not enough: how accuracy metrics have hurt recommender systems[END_REF]. Since then, the integration of diversity in machine learning models has shown many simultaneous benefits on user confidence, acceptance rate and user perceived qualities [START_REF] Hu | Helping Users Perceive Recommendation Diversity[END_REF], or on the reduction of overspecialization [START_REF] Adamopoulos | On Over-Specialization and Concentration Bias of Recommendations: Probabilistic Neighborhood Selection in Collaborative Filtering Systems[END_REF]. Diversity has also been integrated into almost all RS evaluation frameworks [START_REF] Eftimov | A Framework for Evaluating Personalized Ranking Systems by Fusing Different Evaluation Measures[END_REF][START_REF] Parapar | Towards Unified Metrics for Accuracy and Diversity for Recommender Systems[END_REF]. Nevertheless, diversity in its current form is beginning to show its limits in certain contexts.

Taking social media as an example, the literature still highlights plenty of scientific issues around diversity [START_REF] Raza | News recommender system: a review of recent progress, challenges, and opportunities[END_REF]. The shift toward online news sharing and consumption gave the opportunity to news readers to access a huge amount of news everyday. In that context, News Recommender Systems (NRS) have been developed to help them identify content they are interested in. Traditional RS transposed in news domain tend to provide recommendations that do not challenge users' prior beliefs, offering news similar to those previously consulted. This often results in the creation of filter bubbles [START_REF] Pariser | The filter bubble: what the Internet is hiding from you[END_REF]. An intuitive approach to this problem has been to increase as much as possible the diversity of the set of recommended news without altering too much the accuracy of the recommendations. The goals of such an approach are to provide a larger spectrum of opinions to all users, ensure ethical and fair recommendations [START_REF] Giunchiglia | Towards Algorithmic Transparency: A Diversity Perspective[END_REF][START_REF] Schelenz | Diversity-Aware Recommendations for Social Justice? Exploring User Diversity and Fairness in Recommender Systems[END_REF], and foster a healthy democratic debate [START_REF] Bernstein | Diversity in news recommendations. In erspectives Workshop: Diversity, Fairness, and Data-Driven Personalization in (News) Recommender[END_REF][START_REF] Machado Lunardi | A metric for Filter Bubble measurement in recommender algorithms considering the news domain[END_REF]. However, the role of NRS on user polarization is still disputed, and the benefits of diversity in this context may be overestimated. On the one hand, it is commonly accepted that the repeated consumption of news covering only a narrow range of opinions about a specific topic can admittedly lead to the adoption of extreme positions [START_REF] Sunstein | Going to extremes: how like minds unite and divide[END_REF]. Based on this, recent studies highlight that social media companies like Facebook1 and Twitter2 intensify political sectarianism [START_REF] Finkel | Political sectarianism in America[END_REF][START_REF] Van Bavel | How social media shapes polarization[END_REF]. On the other hand, social media is probably a key facilitator and an amplifier -rather than a cause -of polarization. A recent survey by Boxell et al. showed that political polarization is a phenomenon that has been growing for 40 years, before the advent of the Internet and social networks [START_REF] Boxell | Cross-Country Trends in Affective Polarization[END_REF]. They even point out that affective polarization -i.e. the tendency for partisans to dislike and distrust those from the other party [START_REF] Druckman | Affective polarization, local contexts and public opinion in America[END_REF] has risen in USA, Canada, New Zealand and Switzerland, while it decreased in UK, Australia, Germany, Norway and Sweden where social media is also widely used.

As stated by Kubin et al. [START_REF] Kubin | The role of (social) media in political polarization: a systematic review[END_REF], one can regret the lack of research exploring ways social media can depolarize. In March 2020, an experiment in which subjects stopped using Facebook for a month highlighted a significant reduction of polarization on policy issues [START_REF] Allcott | The Welfare Effects of Social Media[END_REF]. More surprisingly, it did not diminish divisiveness based on party identity. That can be explained by the fact that people tend to be more upset and angry at the other side when seeing political content on social media. That is consistent with the conclusions of Bail et al. according to which exposure to opposite views can increase political polarization [START_REF] Bail | Exposure to opposing views on social media can increase political polarization[END_REF].

In this context, it is questionable whether ensuring high diversity for all users is appropriate. Of course, this is a sensitive issue because it is vital to make certain that RS, by adapting their strategies to each individual, do not influence users and standardize opinions. Though, we support the idea that current NRS have a lack of expert knowledge on user personality traits, leading to a poor understanding and attention on users' behavior classes. Through this position paper, we would like to investigate this idea and raise the following research questions:

• RQ1. Does diversity of recommendations and content bring a systematic gain? In particular, we seek to determine the extent to which the diversity impacts the news consumption behaviors of users and contributes to broadening their spectrum of news access. • RQ2. Is it sufficient to measure the influence of an NRS afterwards with single-number metrics, or does this influence occur with some variations over time?

Enlarging our vision, this problem can be extended this way:

• RQ3. Do NRS really influence all users equally? If not, we can imagine that all users are not receptive to bias and are not all exposed to filter bubbles. • RQ4. Are there different classes of user behavior with respect to viewing diversity, independently of the recommender system? If so, do viewing habits prior to interactions with an NRS impact user trajectory during the recommendation phase?

We do not pretend to answer all these questions, but rather to open the debate on current practices in the field of social media and NRS. To contribute to this reflection, we study the well-known open MIcrosoft News Dataset (MIND) 3 , and apply both commonly evaluation methods of recommendations diversity and a novel indepth analysis taking into account the temporal aspect of that diversity. We aim to draw preliminary conclusions from this analysis and provide some take-home messages (TH) and perspectives for researchers in User Modeling, Adaptation and Personalization. In particular, we emphasize the need for (1) enriched datasets (including more information about users, opinions, context and triggering factors of news consumption, and topics of controversy) for a better understanding of users' behavior and personality traits, (2) new analytical methodologies to put the same effort into modeling the diversity trajectory of users with and without RS as was put into assessing the diversity provided by RS, (3) new diversity metrics suitable for NRS by taking the time dimension and opinions into account in addition to topics, and (4) new humancomputer interaction and recommendation models to bring the appropriate and personalized amount of diversity, and guarantee a fair and depolarizing NRS.

The remainder of this paper is structured as follows: In Section 2, we first provide a literature review on how RS and NRS are commonly evaluated, with a particular emphasis on the notion of diversity and time dimension. We then compare, in Section 3, the existing open datasets and choose one to analyze. Section 4 is dedicated to the analysis of diversity within the MIND dataset as an attempt to measure its influence on users' news consumption. Finally, we provide some take-home messages for the community.

RECOMMENDER SYSTEM EVALUATION FRAMEWORKS 2.1 Single-Number Metrics

The main purpose of recommender systems is to provide users with content adapted to their preferences and/or needs, which they would not have consulted spontaneously [START_REF] Resnick | Recommender systems[END_REF]. To achieve this goal, many accuracy metrics have been designed and are used both to compute recommendations and to evaluate the quality of the models through offline datasets [START_REF] Charu | Recommender Systems -The Textbook[END_REF]. These metrics can either be error-based or ranking-based [START_REF] Ricci | Recommender systems handbook[END_REF].

Error metrics are often preferred when evaluating single recommendations and/or predictions. They include for instance the Mean Absolute Error (MAE), the Mean Squared Error (MSE), and the Root Mean Squared Error (RMSE) [START_REF] Chicco | The coefficient of determination R-squared is more informative than SMAP, MAE, MAPE, MSE and RMSE in regression analysis evaluation[END_REF]. Rank metrics are adapted to listwise or sequence-based recommenders (precision, recall, nDCG, MAP, MRR. . . ) [START_REF] Charu | Recommender Systems -The Textbook[END_REF][START_REF] Chicco | The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation[END_REF][START_REF] Isinkaye | Recommendation systems: Principles, methods and evaluation[END_REF][START_REF] Parra | Recommender Systems: Sources of Knowledge and Evaluation Metrics[END_REF][START_REF] Silveira | How good your recommender system is? A survey on evaluations in recommendation[END_REF]. Their aim is to determine whether the ranking of the recommendations is appropriate, i.e. the items in the test set that correspond to the strongest user preferences are present in the top-N recommendations, and conversely the recommendations with the highest scores are highly valued items in the test set.

Error-based or rank-based metrics have been proven to assess the relevance of recommendations, but they suffer from two limitations. First, as stated by Scheidt and Beel, they are based on the entire dataset and hold no information if models perform the same over the whole time period or if they improved or worsened over time [START_REF] Scheidt | Time-dependent Evaluation of Recommender Systems[END_REF]. Second, accuracy has been proven to be insufficient to evaluate and explain user satisfaction [START_REF] Kaminskas | Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems[END_REF][START_REF] Maksai | Predicting online performance of news recommender systems through richer evaluation metrics[END_REF][START_REF] Sean M Mcnee | Being accurate is not enough: how accuracy metrics have hurt recommender systems[END_REF][START_REF] Zhou | Solving the apparent diversity-accuracy dilemma of recommender systems[END_REF]. It is to address this second limitation and go beyond-accuracy metrics that evaluation frameworks have been enriched with many other complementary dimensions, such as: Coverage [START_REF] Ge | Beyond accuracy: Evaluating recommender systems by coverage and serendipity[END_REF], Confidence [START_REF] Maciej | Estimating confidence of individual rating predictions in collaborative filtering recommender systems[END_REF], Trust [START_REF] John | Trust in recommender systems[END_REF], Acceptance and Adoption Rates [START_REF] Armentano | Applying the Technology Acceptance Model to Evaluation of Recommender Systems[END_REF], Novelty [START_REF] Zhang | The Definition of Novelty in Recommendation System[END_REF], Serendipity and Unexpectedness [START_REF] Adamopoulos | On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected[END_REF], and Diversity [START_REF] Kunaver | Diversity in recommender systems-A survey[END_REF]. The latter dimension has allowed recommender systems to progress considerably over the last 15 years, as we will see.

Diversity in RS was first introduced by Smyth and McClave [START_REF] Smyth | Similarity vs. Diversity. In Case-Based Reasoning Research and Development[END_REF]. At that time, it was considered as the opposite of similarity (1 -similarity), the similarity being another metric to measure the proximity of the recommendations with the known preferences of users. Smyth and McClave foregrounded the lack of diversity in recommendations produced by similarity-based approaches and the problems that it entails. For instance in e-commerce, the recommendations should all be similar to the product currently consulted by the active user. However, if they are also too similar to each other, it will be difficult for the user to make a decision among this set of alternatives. Diversity was therefore a means of modifying the list of recommendations without compromising accuracy, before becoming a full-fledged evaluation objective for RS. Subsequently, several diversity metrics have been proposed to take into account different needs and application contexts. The Intra-List Similarity (ILS) [START_REF] Ziegler | Improving recommendation lists through topic diversification[END_REF] and, in contrast, the Intra-List Distance (ILD) [START_REF] Zhang | Avoiding monotony: Improving the diversity of recommendation lists[END_REF] aim to measure the average level of diversity within a set of recommendations in a permutation-insensitive way: rearranging positions of recommendations does not affect the measure. With this new definition, it becomes possible to measure diversity within a group, and not simply between pairs of items. A very similar and still widely used formula for Diversity within a set of n items was also proposed by Smyth and McClave [START_REF] Bradley | Improving recommendation diversity[END_REF]. The diversity of a set of items (i 1 , i 2 , ...i n ) is seen as the average dissimilarity between all pairs of items, and computed as follows:

Diversity(i 1 , i 2 , ..., i n ) = n k=1 n j=1 (1 -Similarity(i k , i j )) n 2 * (n -1) (1)
Other metrics complete the picture such as the Relative diversity (RD) which measures the diversity brought by an item relatively to a set of items [START_REF] Bradley | Improving recommendation diversity[END_REF], the Expected Intra-List Diversity (EILD) which is rank-sensitive and rank-aware [START_REF] Vargas | Rank and relevance in novelty and diversity metrics for recommender systems[END_REF], the aggregate diversity used to measure the diversity of items across the recommendation lists of all users and prevent the long-tail problem [START_REF] Adomavicius | Maximizing Aggregate Recommendation Diversity: A Graph-Theoretic Approach[END_REF], and the inverse of Gini index to measure of distributional inequality [START_REF] Fleder | Blockbuster culture's next rise or fall: The effect of recommender systems on sales diversity[END_REF].

Many machine learning models have been developed to increase diversity in RS [START_REF] Karakaya | Effective methods for increasing aggregate diversity in recommender systems[END_REF][START_REF] Kunaver | Diversity in recommender systems-A survey[END_REF][START_REF] Wasilewski | Incorporating Diversity in a Learning to Rank Recommender System[END_REF][START_REF] Wu | Recent Advances in Diversified Recommendation[END_REF][START_REF] Rui Ye | Dynamic Graph Construction for Improving Diversity of Recommendation[END_REF] and have been designed for different application domains such as e-commerce [START_REF] Hu | Helping Users Perceive Recommendation Diversity[END_REF], online music services [START_REF] Baracskay | The Diversity of Music Recommender Systems[END_REF][START_REF] Amaury | Understanding Usages by Modeling Diversity over Time[END_REF], and social networks [START_REF] Sanz | Enhancing Structural Diversity in Social Networks by Recommending Weak Ties[END_REF]. In parallel, many user studies have been conducted to prove the added value of diversity in RS. Zhang and Hurley [START_REF] Zhang | Avoiding monotony: Improving the diversity of recommendation lists[END_REF] and Lathia [START_REF] Lathia | Temporal diversity in recommender systems[END_REF] showed that users may suffer frustration when facing a lack of diversity. Research has even revealed that, in the case of equivalent accuracy between different models, diversity is not only perceived by users but also plays a prominent role in the acceptance of recommendations [START_REF] Michael | User perception of differences in recommender algorithms[END_REF][START_REF] Jones | User Perceived Qualities and Acceptance of Recommender Systems[END_REF]. Castagnos et al. highlighted the need for diversity so as to increase user confidence during the purchase decision process [START_REF] Sylvain Castagnos | Eye-Tracking Product Recommenders' Usage[END_REF]. At last, Parapar et al. demonstrated that diversity can help to elicitate users' preferences, thus reducing the popularity bias in RS [START_REF] Parapar | Diverse User Preference Elicitation with Multi-Armed Bandits[END_REF].

Despite these scientific advances, we note that all these works use single-number metrics and therefore only partially address the challenges of beyond-accuracy RS. This type of evaluation can potentially show significant improvement, especially in terms of diversity and impact, on average on the whole sets of users, items and sessions. Nevertheless, it does not guarantee that the improvement is consistent over time, nor does it guarantee that it applies to all users. In the case of social media and news recommendation, interactions are built over a much longer duration than most application domains, ranging from several weeks to several years, and this problem becomes even more prevalent. The next subsection will illustrate the specificities of NRS.

Evaluation of News Recommender Systems

NRS are intended to select information of interest for news readers [START_REF] Raza | News recommender system: a review of recent progress, challenges, and opportunities[END_REF]. They face a major news-related characteristic that makes the recommendation process more complex [START_REF] Machado Lunardi | A metric for Filter Bubble measurement in recommender algorithms considering the news domain[END_REF]: news items have a very short lifespan and are constantly replaced by fresh releases. Considering this, it is obvious that traditional RS should be adapted to the news recommendation domain, as news quickly becomes obsolete. This adaptation of models is accompanied by the need to revise NRS evaluation. The widespread a posteriori evaluation based on a complete dataset could face major issues by recommending a given user and at a specific time news that no longer exist or that have not yet been written. Besides, considering online shopping, music listening, or movie viewing, users have short-term needs (at the session level). In the field of news, diversity is of course still desirable, but preference elicitation and recommendations spread over a longer period (several days, weeks, months...). Moreover, user satisfaction is not quantifiable on the basis of a single consumed item but rather on the numerous news accessed on their topics of interest. In this regard, the inclusion of diversity and the evaluation frameworks must take into account the time dimension.

Let us first have a look to diversity, which is equally important, if not more, than accuracy when recommending news because of the potential role of NRS in polarisation. Even though diversity is a key principle in news recommendation, there is no consensus on its definition, nor on the associated metrics. For example, Bauer and Werthner distinguish two types of diversity: content diversity and provider diversity, based on different concepts and objectives [START_REF] Michael | Diversity in News Recommender Systems[END_REF]. These diversities do not influence the recommendations in the same way: by diversifying the content, the recommended news will cover different topics, but will potentially come from the same source, whereas by diversifying the providers, the news will represent diverse providers, but may address similar topics. According to Joris et al., diversity represents a broad concept, relative to several fields (computer science, communication science, law and computational linguistics), and covers a large specter of aspects (news, media, democracy) [START_REF] Joris | News Diversity and Recommendation Systems: Setting the Interdisciplinary Scene[END_REF]. Vrinjenhoek et al. affirm that diversity can not be defined with a unique and absolute value, but rather with an aggregate value of many aspects, especially in the news domain [START_REF] Vrijenhoek | Recommenders with a mission: assessing diversity in news recommendations[END_REF]. They explain that what defines a "good" diversity in a NRS depends mainly of the goal of the system, and the normative framework it aims to follow. Bringing diversity in NRS is a complex task, as well as evaluating its potential impact on users' news consumption. The democratic role of NRS [START_REF] Helberger | On the democratic role of news recommenders[END_REF] strengthens even more the importance of controlled evaluation of diversity [START_REF] Bernstein | Diversity in news recommendations. In erspectives Workshop: Diversity, Fairness, and Data-Driven Personalization in (News) Recommender[END_REF]. NRS impact goes beyond acceptance or satisfaction issues, as it may play a role in the enclosure of users in filter bubbles [START_REF] Chitra | Analyzing the Impact of Filter Bubbles on Social Network Polarization[END_REF][START_REF] Pariser | The filter bubble: what the Internet is hiding from you[END_REF]. There is some evidence that providing diversity can have a depolarizing capacity [START_REF] Heitz | Benefits of Diverse News Recommendations for Democracy: A User Study[END_REF]: it may encourage users to consume news related to opposing viewpoints, and even encourage a higher tolerance toward them. However, diversity can sometimes have deleterious impacts: Bail et al. show that diversity can potentially lead to more extreme viewpoints [START_REF] Bail | Exposure to opposing views on social media can increase political polarization[END_REF]. At last, some researchers show that the potential impact of NRS, especially on social media, are different from one system to another, but also from one user to another [START_REF] Cinelli | The echo chamber effect on social media[END_REF][START_REF] Kitchens | Understanding Echo Chambers and Filter Bubbles: The Impact of Social Media on Diversification and Partisan Shifts in News Consumption[END_REF]. Non-personalized diversity -i.e. the diversity produced globally by the NRS with the same process for all users without taking into account their specificities -thus does not seem to have universal consequences on news consumption behavior. This encourages the modeling of the individual users' diversity needs, which may potentially reduce detrimental impacts [START_REF] Du | Is diversity optimization always suitable? Toward a better understanding of diversity within recommendation approaches[END_REF][START_REF] Wu | Personalizing recommendation diversity based on user personality[END_REF].

Let us now turn our attention to the temporal dynamics challenge. In the field of recommendation, it refers to the evolution of users' preferences over time and the need to adapt recommendations to this evolution [START_REF] Xiang | Temporal recommendation on graphs via long-and short-term preference fusion[END_REF][START_REF] Xiong | Temporal collaborative filtering with bayesian probabilistic tensor factorization[END_REF]. Joeran Beel fosters the need of a temporal evaluation, which can allow drawing more precise conclusions about RS and their actual performances over time [START_REF] Beel | It's Time to Consider 'Time' when Evaluating Recommender-System Algorithms[END_REF]. Raza and Ding highlight that the dynamic updating of news is also accompanied by temporal dynamics of users' preferences, which may change in the short or long term, even seasonally [START_REF] Raza | News recommender system considering temporal dynamics and news taxonomy[END_REF]. Hence, authors propose a model that exploits temporal dynamics for news recommendations. Very few other studies address temporal dynamics in NRS: Li et al. integrated long-term preferences of users by building a time sensitive weighting scheme, as well as short-term preferences by analyzing recent users' reading history [START_REF] Li | Modeling and broadening temporal user interest in personalized news recommendation[END_REF]. Other researchers propose to mine patterns of temporal changes in data streams, and then incorporate trends and temporal user habits in their NRS [START_REF] Lommatzsch | Incorporating context and trends in news recommender systems[END_REF]. To the best of our knowledge, Lathia et al. is the first and unique work that explores the importance of temporal diversity and explain that RS must necessarily adapt to temporal changes, mainly because of data renewal, and that changes in temporal diversity affect users' preferences [START_REF] Lathia | Temporal diversity in recommender systems[END_REF]. This work does not specifically focus on news and as far as we know, no work deal with temporal diversity in NRS.

From this literature review, we can conclude that the diversity approaches in NRS are based on different definitions, relate to many concepts, and there is no common framework, both for diversification processes and evaluation. Furthermore, standard evaluations of the impact of diversity on news consumption come to scattered conclusions: diversity can reduce user polarization, have no effect at all, or even exacerbate the adoption of extreme viewpoints. Thus, modeling users' need for diversity in a personalized way appears to be required to control more finely the impact of diversity according to different user profiles and needs. Moreover, the characteristics of the news induce strong temporal variations, and the evaluation of the individual dynamics, i.e. the trajectories specific to each user, are not currently applied. Together, these elements refer to the intuitions presented in the introduction, and we propose to confirm them on an existing dataset and highlight overarching perspectives responding to the challenges of NRS.

NEWS RECOMMENDER SYSTEMS DATASETS

The design of NRS strongly depends on the data that can be used: number of users, number of news, user and news attributes, user activities, recency, time span, etc. Due to copyright and privacy issues, few online media provide free access to news content and user news consumption. The number of publicly available datasets, even for research purpose, is thus notably reduced. As a consequence, many studies explore proprietary and non-public datasets that can not be shared nor reused [START_REF] Karimi | News recommender systems-Survey and roads ahead[END_REF]. In most cases, these datasets are small and gather information about a limited number of users during a short period of time. To cope with this limit, some researchers perform evaluations on datasets that are not related to news, such as the well-known MovieLens dataset [START_REF] Ji | Regularized singular value decomposition in news recommendation system[END_REF]. However, such datasets cannot be fully used to draw strong conclusions about news recommendation.

Yet, some datasets dedicated to news recommendation are freely available. A brief overview of these datasets is given in Table 1 [START_REF] Karimi | News recommender systems-Survey and roads ahead[END_REF][START_REF] Raza | News recommender system: a review of recent progress, challenges, and opportunities[END_REF]. We would like to note that these datasets contain information about news and user news consumption. Other domains, such as natural language processing, are also interested in news datasets, but these datasets only contain news related data.

Among those datasets, the one that seems to be the most adequate to address the questions raised in introduction is the MIND dataset. Indeed, it covers the largest time frame (6 weeks) of Englishwritten news consumption, and provides information about both news and recommendations (impressions) provided to users. Besides, MIND is widely used in the news recommender systems area [START_REF] Liu | KRED: Knowledge-aware document representation for news recommendations[END_REF][START_REF] Qi | Personalized news recommendation with knowledge-aware interactive matching[END_REF][START_REF] Wu | NewsBERT: Distilling pre-trained language model for intelligent news application[END_REF].

In detail, MIND gathers 15M impression logs about 1M users, interacting with 160k English news articles. Data were collected by randomly selecting users who have interacted with the Microsoft News website during six weeks, between October 12 and November 22, 2019. Data are split into training, validation and test sets: the clicks made during the first four weeks (from 12 October to 8 November, 2019) were used to construct the news click history, while those made on the fifth (from 9 November to 15 November, 2019) and sixth weeks (from 16 November to 22 November, 2019) were used for training, validation and test sets. To ensure user privacy, data are anonymized and users are identified using unique identifiers. In line with the choice made in some studies [START_REF] Wu | Feedrec: News feed recommendation with various user feedbacks[END_REF], we work on the small version of the dataset, provided by MIND developers. This subset contains information about 50k users randomly extracted from the large dataset. The associated history concerns the first four weeks (from 12 October to 8 November, 2019), and the impression (recommendation) logs were recorded during the fifth week only (from 9 November to 15), along with the clicks performed by users during that fifth week, within the set of impressions. Data about the sixth week is not part of this subset. We assume that the news accessed during the first four weeks are also chosen in a set of recommended news, upon which no information is given. A graphical temporal representation of this data is presented in Fig. 1.

ANALYSIS OF THE IMPACT OF DIVERSITY ON MIND USERS' NEWS CONSUMPTION

To study the impact of diversification of the recommendations on users' news consumption and to evaluate the extent to which diversification is a solution to affective polarization, we conduct experiments on the MIND dataset. (body) of the news is not provided. The same for the publication date, which is crucial to study user behavior. Fortunately, by processing the URL, the body of a news and its publication date can be retrieved. On the later, the lifespan of the majority of news is less than two days in the Microsoft News Service, as mentioned in [START_REF] Wu | Mind: A large-scale dataset for news recommendation[END_REF]. The delay between publication and user consultation is thus reduced. We propose to estimate users' consumption date by the news publication date. Filter bubbles are not related to the interest in a unique topic, but in the interest in a unique opinion. So, to conduct an analysis that specifically focus on this latter aspect, we select a subset of news that deal with one category, specifically the "news" category. Not only this category is predominant in the dataset, but it is also the closest category to the political domain, which is of particular interest as we are interested in political polarization.

Experimental settings

Finally, RQ2 being related to the temporal analysis of users news consumption, observations need to be made on users who consumed news throughout the complete dataset time span. We thus keep only users who accessed news each of the five weeks, and at least 3 news each week to have a valid interpretation of their consumption.

News Representation.

In line with the literature [START_REF] Garcin | Personalized news recommendation based on collaborative filtering[END_REF][START_REF] Luostarinen | Using Topic Models in Content-Based News Recommender Systems[END_REF][START_REF] Tintarev | Same, Same, but Different: Algorithmic Diversification of Viewpoints in News[END_REF], we used an unsupervised approach to represent the topic distributions of news items, namely LDA models 4 [START_REF] David M Blei | Latent Dirichlet allocation[END_REF]. After concatenating the original MIND train/validation/test news items, we performed near-deduplication using MinHashLsh, resulting in 126,649 4 https://bab2min.github.io/tomotopy/v0.12.2/en/ articles. Using the Spacy library 5 , we removed stop-words, digits, and we lemmatized words, forming bag-of-words representations. News containing less than 20 words were filtered out. Thus, postprocessed news items have between 21 and 4,351 words, with an average of 261 words. The model hyperparameters were tuned to optimize performance.

Diversity Metric.

The news representation serves as a basis to measure their diversity. We select a widely used measure, i.e. the average dissimilarity between each pair of news (see equation [START_REF] Adamopoulos | On Over-Specialization and Concentration Bias of Recommendations: Probabilistic Neighborhood Selection in Collaborative Filtering Systems[END_REF]), that ranges between 0 (no diversity) and 1 (maximal diversity). Similarity was computed using cosine similarity [START_REF] Rahutomo | Semantic cosine similarity[END_REF], as commonly done in NRS field [START_REF] Kompan | Content-based news recommendation[END_REF][START_REF] Möller | Do not blame it on the algorithm: an empirical assessment of multiple recommender systems and their impact on content diversity[END_REF].

The data structure used in the experiments are summarized in Table 2. The resulting dataset, which we call MIND5w, is made up of 1,475 users and 20,541 news.

Experimental Analysis

The experiments conducted below are designed to contribute to answer both research questions raised in Introduction. For RQ1, we adopt a holistic perspective of diversity. For RQ2, we adopt a temporal perspective to provide additional information about the temporal dynamics of users' news consumption.

As a prerequisite of the experiments, we ascertain that the news from the dataset actually offer users the possibility to get engaged with content more or less diverse. To this aim, we evaluate the average diversity of each pair of news in MIND5w. Herewith

Single-Number Analysis.

In this section we adopt a holistic perspective of diversity. The first four weeks of MIND5w are associated with the users' news consumption. The last week (5 th week) is also associated with users' news consumption, supplemented by the news recommended to users (see Figure 1). To make a thorough analysis of these sets of news over all users, we propose to evaluate four diversity measures.

• Diversity of history is the users' average diversity of the news they accessed during the first four weeks. Users accessed on average 39 news.

• Diversity of recommended news corresponds to the average diversity of news recommended during the 5 t h week. The number of recommended news that week is quite high, 140 news on average, which is significantly higher than the average number of news accessed in the history. For that 5 t h week, the dataset also provides information about the news the users chose to access among these recommendations and those they did not access. This allows to measure:

• Diversity of accessed news corresponding to the average user diversity of the news accessed during the 5 t h week (among the recommendations). It has a similar meaning than the diversity of history, but over a shorter period. Users access on average 9 news during the 5 t h week, which is similar to the weekly number of news accessed in the history.

• Diversity of unaccessed news corresponding to the average diversity of the recommended but not accessed news. The number of accessed news being relatively limited compared to the number of recommendations, these unaccessed news represent a large part of the recommended news, 131 news on average. This indicates that diversity of unaccessed and recommended news will be highly similar.

The distribution of the number of users over the diversity values for each of the four previously introduced diversity measures, is displayed in Figure 2.

Considering the distribution of the recommendations, we can see that the average diversity is 0.75, which is quite high and the standard deviation is 0.04, which is rather small. We would like to point out that the average diversity of recommendations and the average diversity of news are not directly comparable, although both diversities have been evaluated similarly. Indeed, the average diversity of recommendations is calculated over all pairs of news items in the recommendation sets, that we suppose to be made up pieces of news that are rather similar to each other (at least in terms of topics), even though the sets are diversified. The average diversity of the news in MIND5w is calculated over all pairs in the set of 20k news. As previously mentioned, the majority of the pairs have a really high diversity (>0.9) as most of them are totally unrelated, which naturally increases the average diversity.

Besides, the high diversity and small standard deviation in recommended news leads us to think that a diversification process actually exists in the recommender system of the Microsoft News Website, that guarantees that recommendations provided to users are diverse and meet a predefined diversity level.

As expected, the distribution of the diversity of the set of unaccessed news is similar to the one of the recommended news (Kolmogorov-Smirnov test, pvalue = 0.148 > α = 5%). Recall that this similar distribution is more than probably due to the fact that the unaccessed news is a subset (93%) of the recommended ones.

If we focus on the news accessed during the 5 th week, the distribution significantly differs from the distribution of the recommended news. On the one hand, the average diversity of accessed news is significantly lower than for recommended news. So, users access news items that are on average less diversified than those recommended. To be precise, 67.1% of the users have a diversity of accessed news lower than the average diversity of recommendations. On the other hand, the standard deviation (0.17) is significantly higher, and reflects a greater variability in the average diversity of news accessed by users during the 5 th week.

The distribution of the accessed news in history (first four weeks) seems to be as spread as the one of the accessed news during the 5 th week. Yet, the average diversity in history is significantly smaller (p -value ≈ 0). This leads us to question about the impact of a recommender system.

To refine this analysis, we study these diversities (recommended, accessed and history) for each user, displayed in Figure 3. If we focus on the diversity of recommended news and the diversity of history (Figure 3a), we note that the average diversity of the recommendations is consistent across all users, regardless of the number of news in the history. Beyond the previous finding about the high diversity of the recommended news in MIND5w, we confirm here that this diversity is not personalized, especially in terms of diversity in the history. Comparing the average diversity of accessed news and the diversity of history (Figure 3b), we see that the vast majority of the users are close to the diagonal, i.e. most of them have a diversity of history close to the diversity of accessed news. The prominence of users upper (but close to) the diagonal confirms the previously identified higher diversity during the 5 t h week.

With the goal to understand why some users do not have similar diversity in history and in 5 t h week, we conducted additional experiments, not presented here, that provide additional findings. First, about the impact of the number of news accessed in the history on the diversity of the accessed news, we confirm that this number does not explain the difference in diversity between history and accessed news. Second, during the 5 t h week, 67.7% of users access news less diversified than the recommendations they get.

From these experiments, we can conclude that a high diversity of recommendations is far from leading to a systematic diverse news consumption. Altogether, these results motivate us to further analyze the extent to which RS impact users' news consumption over time.

Temporal Analysis.

Let us now focus on the evolution of the diversity of users through time, namely on the complete time span of MIND5w. To this aim, we adopt a weekly perspective.

The distribution of users' average diversity for each week is presented in Figure 4. A first look at these distributions shows that they are close to each other. However, a Kolmogorov-Smirnov test attests that the distributions are not similar between weeks (p -values ≈ 0), except for the week 1 that is similar to weeks 3 and 4 (p -value = 0.124 and pvalue = 1.160, respectively). These results were confirmed with the Anderson-Darling test for k-samples. However, considering the evolution of the average diversity, it does not systematically increase or decrease.

To analyze more precisely the evolution of user diversity through time, we choose to form four equally sized groups of users. These groups are defined from the quartile values of their weekly average diversity, and the first quarter contains users with lower diversity values. A study of the values of the three quartiles highlight that they remain stable across the five weeks, which confirms that even if distribution of diversity between weeks are different, they are close.

Figure 5 represents the flow between quarters from one week to the next one in a Sankey Diagram. The thickness of each flow is proportional to the number of users in that flow. Let us start by highlighting that the transition patterns remain stable over the weeks: it confirms a similar global impact of the recommendations on users diversity of accessed news through weeks. Each possible transition (from any quarter to any quarter) actually occurs, although almost 80% of users remain in the same or adjacent quarter between two consecutive weeks. So, few users change significantly their consumption habits between two weeks.

This study of users' behavior between adjacent weeks remains limited and we would like to highlight user specific temporal behavior along the five weeks according to these quarter shifts. A variation ranges in [-3, +3], where a variation is equal to 3 when a user moves from the 1 st to the 4 th quarter, i.e. her diversity greatly increases. A variation equal to -3 represents a user who moves from the 4 th quarter to the 1 st one, i.e. her diversity dramatically decreases. If the variation equals to 0, the user remains in the same quarter.

These variations are visually represented with a heatmap in Figure 6. A red variation represents an increase in the diversity, the darker the larger. A blue variation represents a decrease in the diversity. The Figure represents the diversity variation of each week relatively to week 1 to study the impact of the recommender system through time. This provides an overall picture of the trajectories taken by users in relation to their average diversity during the first week. We define three types of users, depending on their receptiveness to recommendations' diversity.

• Positively receptive users: users accessing more diverse news after 5 weeks of use of the system. • Negatively receptive users: users accessing less diverse news at the end of the 5 weeks. • Resistant users: users accessing equally diverse news after the 5 weeks. On Figure 6, users are sorted according to their global evolution between weeks 1 and 5 (last column), then according to their evolution between weeks 1 and 2 (first column). Looking at the right-most column, we observe that previously presented types of users are divided into thirds: 32.1% of the users increase their diversity between week 1 and week 5, 32.5% of the users consume less diverse news in week 5 than in week 1. For both cases, the extrema represent users with the biggest change in their diversity. The white band in the center of the last column represents users whose average diversity did not change between weeks 1 and 5 (35.4% of the users).

If we now take a closer look at both positively and negatively receptive users, we observe that almost 90% of them observe a similar or null variation between week 1 and 2. Besides, more than 60% of the users have a variation direction in the first column similar to the one in the last column (red/red or blue/blue). For each of the three types of users, 25% of users have a consistent variation direction over the entire time span, equally distributed among the user types. In addition, only 15% of the users have temporary opposite variations (the others going through transient states with no variation). We can conclude that the variation between weeks 1 and 2 is an accurate predictor of the one at week 5. Finally, the most striking finding is that a constant high diversity of recommendations, as used in MIND5w, has a negative impact on as many users as there are users with a positive impact. This tends to confirm findings of Bail et al. according to which exposing potentially polarized users to diversity can be counterproductive [START_REF] Bail | Exposure to opposing views on social media can increase political polarization[END_REF].

With regard to resistant users, 50% of them do not have any variation in the first week either. The other users evenly split between positive and negative variation. For about 30% of resistant users, the diversity of news accessed is not impacted along the five weeks. Looking more closely, the vast majority of users whose average diversity remains stable are those who belong to extreme quarters in the first week.

We can conclude that users who have either a really high diversity or those who have a very low diversity during the 1 st week are the most resistant users to the recommendations in terms of further news consumption.

To go further, we wonder whether these variations are influenced by the number of news accessed by the users over the five weeks, or by their initial diversity (1 st week). Additional experiments provide us with arguments to state that all variations exist, whatever is the initial quarter of the users. In addition, the number of accessed news throughout the five weeks only impacts this variation if it is quite high: an extreme variation is never observed for such users.

Discussion

The Single-Number and temporal analyses allowed us to put forward important elements. First, new information has been highlighted about the dataset. The recommender system used in the Microsoft News Website is active all along the five weeks. The system ensures that the recommended news items are diverse, and fit a predefined high level of similarity close to 0.75. We are now able to answer RQ1: diversification does not lead to a systematic increase in the diversity of the news accessed, users significantly vary in terms of diversity of the news they access, ranging in the complete span of possible diversity values. More important, users' average diversity is lower (12%) than the average diversity of the recommendations, and 66% of users access less diverse news than recommendations.

From a temporal perspective, this varying impact is confirmed, and seems to operate every week, including at the user level. Each week, a significant proportion of users observe a variation in their average diversity and we highlighted three types of users: positively receptive, negatively receptive and resistant users. This temporal variation contributes to answer RQ2: single-number evaluations are insufficient, and there is a critical need for an NRS evaluation framework that takes into account both the differences between users and the temporal aspect.

CONCLUSION AND TAKE-HOME MESSAGES (TH)

The preliminary conclusions drawn above on the impact of the recommendations on user diversity, especially over time, deserve to be further discussed to highlight some needs and future research directions for the UMAP community. First, we have seen that providing users with equally diverse sets of recommendations neither impacts similarly users, nor increases systematically the diversity of the news they consume. This leads us to focus on two elements.

• About the time span covered by the dataset and the span of use of the system. Recall that users that have been selected from the original MIND dataset are those who used the system during the five weeks. We saw that the diversity of the news consumed by users after five weeks remains as spread as the one in the first week and that the average diversity slightly evolves. It is more than probable that these users were already using the system before the data were captured. We can ask whether the impact of the diversity of the news recommended on the diversity of the news consumed occurred ahead the first week, which could explain the small average impact during the 5 weeks span. However, no information about the prior use of the system before these five weeks, which could have helped us to understand this limited impact and model more precisely the actual impact. • About the individual impact of the recommendations. Although the average diversity of the news consumed by users are close between weeks 1 and 5, a specific focus shows that the impact differs fundamentally between users. Some users totally polarize, other users totally open their interests and some do not modify their consumption habits. However, no information in the dataset could contribute to explain the individual impact.

TH1: There is a cruel lack of open datasets that cover a significantly longer period to study the impact of the recommender systems in the long run, as well as richer datasets that can include additional information about the users (including a distinction between recommended and nonrecommended news in user history), the contexts, and even the news to provide a more accurate model of the impact and adoption of the recommendations.

Second, considering again the individual impact of diversified recommendations on users, that is not dependent on the users' prior diversity, we wonder who are the users who partially, even totally, reject the diversity of the recommendations and polarize progressively (RQ3). Why do they reject this diversity? We put forward the idea of the need of personalized diversification strategies, that could:

• consider user personality traits and user behavior classes to adapt the diversity level of the recommendations to these personal features to increase the adoption of diversified sets of recommendations. • redefine the traditional diversity metrics, even propose and use different diversity metrics. Such new metrics could consider specific elements that users may be sensitive to, or specific elements of the news: recency, title, abstract, author, source, etc.

Besides, beyond the need to model opinions conveyed in the news in order to ensure an opinion diversification, we wonder to what extent the recommendation strategy should consider or not the topic of interest. Given that the topics discussed in the news do not all have the same polarizing effect -they present higher or lower levels of controversy [START_REF] Garimella | Quantifying controversy on social media[END_REF], which impacts differently on polarizing behavior [START_REF] Baumann | Emergence of polarized ideological opinions in multidimensional topic spaces[END_REF] -, should this controversy be considered to define the diversification strategy? TH2: There is a crucial need to define new diversity measures and personalized recommendation strategies that consider the users and the topics.

Third, going into details of the news consumption along the weeks, even though the initial change in news consumption (first weeks) is an accurate indicator of the global change, the news consumption of some users significantly fluctuates, despite the constant diversity of the recommendations. Few elements have been highlighted in the experiments conducted, but some questions are still pending. What factors impact the actual news consumption from one week to another? What does a change in the diversity of news consumption between two or several weeks mean? Are there time-aware user behavior classes? Although these questions are naturally raised, methodologies for a temporal analysis of the news consumption are cruelly lacking in the literature to thoroughly understand the reasons for the evolution of users (RQ4). This is even more important as these answers should be used to provide personalized diverse recommendations, with the goal to control the impact on users news consumption.

TH3: The literature is lacking of well-established methodologies to model the diversity trajectory of users.

Figure 1 :

 1 Figure 1: Graphical representation of MIND5w

Figure 2 :

 2 Figure 2: Distribution of diversity among users.

  (a) Diversity of recommended news vs history. Density of the diversity of accessed news vs history.

Figure 3 :

 3 Figure 3: Comparison of diversity of history, recommendations and accessed news. Each dot in (a) and (b) represents a user.

Figure 4 :

 4 Figure 4: User diversity distribution for the 5 weeks

Q2Figure 5 :

 5 Figure 5: Variation flow over weeks (Q1 = quarter 1, ...)

Figure 6 :

 6 Figure 6: Heatmap of quarter changes between week 1 and every other week. Each pixel line represents a user.

Table 1 :

 1 Summary of free datasets for the news recommendation task.

	Dataset	Description	Data types	Size	Period
	Yahoo	Several datasets avalaible	News articles information	Depends on the	Depends on the
	Webscope	https://webscope.sandbox.yahoo.com/	Click data	dataset	dataset
		Plista & TU Berlin	Editor information	17M sessions
	Plista [43]	13 German news portals Accessible upon request for research	Readers information (clicks and impressions)	70k news 80M impressions	30 days (June 2013)
		https://www.plista.com/	Time-related information	1M clicks
	Adressa [32]	Adressavisen and the Norwegian U. of Sciences and Technology Large and small versions available https://reclab.idi.ntnu.no/dataset/	News articles information Click data	Large version: 3M users, 48k news 27M clicks Small version: 15k users, 1k news 2.7M clicks	Large version: 10 weeks (2017) Small version: 1 week (2017)
				Large version:	Large version:
	MIND [83]	Microsoft News Website Large and small versions available https://msnews.github.io/	News articles information Impressions logs	1M users, 200k news 15M impression logs Small version: 50k users, 60k news	6 weeks (Oct.-Nov. 2019) Small version: 5 weeks
				1M impression logs	(Oct.-Nov. 2019)
	Globo.com [21]	Brazilian news portal https://www.globo.com/	News articles information Click data	3M clicks 1.2M sessions 330k users, 50k news	2 weeks (October 2017)
	Outbrain	Outbrain Click Prediction challenge https://www.kaggle.com/c/outbrain-click-prediction	News articles information Click data	2 billion page visits 700M users, 17M clicks	2 weeks (June 2016)

4.1.1 Data Selection and pre-processing. From Section 3, we can notice that MIND provides limited information about the news (Category, SubCategory, Title, Abstract, URL). The actual content

Table 2 :

 2 Data in MIND5waverage diversity is 0.79, which shows that news items are globally diversified. The standard deviation (0.27) affirms that really high, as well as lower diversity values actually exist in MIND5w. This scattered diversity distribution affirms that users have the opportunity to access both highly and little diversified news. Refining this analysis by considering all news pairs individually, a large proportion (53.5%) have a diversity greater than 0.9.

	, the

https://www.facebook.com/

https://www.twitter.com/

https://msnews.github.io/

https://spacy.io/

ACKNOWLEDGMENTS

This work has been funded by the BOOM ANR Project -ANR-20-CE23-0024.