Clément Oury

Since its birth in the early seventeenth century, the press has played a prominent role in the political and social life of France. Over the two last decades, the economic and even cultural pillars on which the press ecosystem is built has been challenged by the growing use of digital technologies, and by the increasing role of the Internet as a way to distribute and access information.

Heritage libraries are affected by these major changes. They need to address the accelerating shift from analogue to digital in order to maintain the continuity of their objectives and of their missions: being able to collect and preserve cultural items, and being able to document the way these items were produced, distributed and used. Many aspects need to be taken into account: legal, scientific, technical, economic and organizational issues have to be identified and addressed.

This paper looks at the example of the National Library of France (Bibliothèque nationale de France or BnF), and at the way it has dealt with collecting newspapers in digital form.

During the ten last years, the BnF has launched several experiments, testing different approaches, with varying degrees of success:

-Direct deposit of electronic publications on physical media (CDs and DVDs) or through FTP. This way of collecting has been experimented with by BnF for some regional newspapers whose local versions were not kept in their paper form; and for which a digital substitute was searched out. This paper explains why the experiments were not conclusive.

-Fully automated web harvesting. Since December 2010, almost 100 news websites (national and daily newspapers, pure players, news portals…) are collected on a daily basis. This harvest gives a very good overview of the kind of information available to French Internet users, but does not allow the collection of publications for which payment is required.

-Web harvesting through agreements with producers. It will be showed how this third approach may act as an improvement of both previous solutions.

Since its birth in the early seventeenth century, the press has played a major role in the political and social life of France. In the seventeenth century, struggles between the domestic and foreign press created an open field for a first kind of public debate. During the Enlightenment, diffusion of a free press was considered a key condition to achieve the philosophers' demands for freedom, political equality and justice. Newspaper and journals were indeed widely used by leaders, parties, and activists during the French Revolution, and up to the present day. In the nineteenth and twentieth centuries, newspaper owners and journalists considered that no subject was beyond their reach, from political, diplomatic or military issues to economic, cultural or sporting topics.

This prominent role explains why the press is widely used by researchers working on the history of France and other countries, even for the most recent periods. And this is also the reason why acquiring, promoting and giving access to press collections is a major objective for heritage institutions.

At the Bibliothèque nationale de France, this mission is performed for printed journals thanks to the framework of legal deposit. Legal deposit is the obligation for every producer of cultural content to send exemplars of its works to the national library. It was introduced by King François I, at a time where the invention of the printing press radically enhanced the possibility of producing and distributing books. When the first periodicals were published, in the 1630s, they were automatically, as printed elements, submitted to legal deposit, thus theoretically allowing the library to gather all French press titleseven though this objective of comprehensiveness has never been perfectly reached. Legal deposit has been progressively extended to all kind of cultural items, from engravings (1672) to radio, television and software (1992), including also music (1745), sounds (1938) and videos (1975).

The digital shift: a threat for heritage institutions' missions?

However, over the two last decades, the economic and even cultural pillars on which the press ecosystem was built has been challenged by the growing use of digital technologies, and by the increasing role of the Internet as a way to distribute and access information. Some major press titles are encountering financial difficulties. New stakeholders are emerging: some of them, the "pure players" (or newspapers that only exist online) still come from the realm of the printed press, whereas others are companies related to computing or information technologies. Finally, all press titles are thinking about their business models, ranging from fully free content to subscription-based access, with many possible variations.

Heritage libraries are necessarily affected by the major changes that affect the journals, as well as all other actors in the cultural domain. Their mission remains the same: being able to gather and preserve all cultural items, and being able to document the way these items are produced, distributed and used. These institutions need to address this accelerating shift from analogue to digital in order to maintain the continuity of their objectives and of their missions. However, they are faced in this regard with two apparently contradictory problems:

-on one hand, radically new kinds of documents are appearing, that need to be gathered and preserved. The web allows a far larger number of people to publish journals online, hence multiplying the number of titles whose memory should be kept.

-on the other hand, digital technologies also simplify and make easier the ways people produce printed documents. The number of printed titles is therefore growing (even though at a slower pace than their online equivalents), challenging the libraries ability to acquire, index and store them. For example, 40 000 different titles are currently received by the periodicals legal deposit service at BnF, representing a total number of 330 000 issues.

To tackle these issues, heritage institutionsand especially national librarieshave to find groundbreaking solutions that should at the same time be consistent with the way they deal with analogue collections. Many aspects need to be taken into account: legal, scientific, technical, economic and organizational issues have to be identified and addressed.

We propose in this paper to look at the example of the National Library of France (Bibliothèque nationale de France or BnF), and at the way it deals with collecting newspapers in digital form. During the ten last years, the BnF has launched several experiments, testing different approaches, with varying degrees of success.

Deposit of digital substitutes for printed versions

Issues and methods

In the first approach, the publishers perform a deposit of the digital version of the journal they distribute on paper form. BnF librarians started considering this solution in the early 2000s. At this time, digitization of older press collections was considered a priority for the BnF online digital library, Gallica1 and it is still a priority. It therefore appeared logical and necessary to also be able to keep the memory of the newer ones.

Regional daily press titles were considered the best candidates in order to test this kind of deposit. Indeed, each of these regional titles generally proposes many local versions according to the district where they are distributed. Only a few pages vary between the different local versions, but strictly following the principles of legal deposit, all of them must be kept. This represents a huge storage cost compared to a rather small benefit in terms of new content. This is the reason why, at this date, BnF was microfilming the local editions of around 20 regional titles. As there was a threat to the maintenance of microfilm companies and microfilm reading devices, digital technologies were seen as a good replacement solution.

However the goal was not to digitize the paper version (as it was done in the microfilming process), but to get directly the digital version used by the publisher and the printer.

There was no legal basis to ask publishers for their digital masters. This is the reason why BnF started working with volunteers. Two regional newspapers answered positively: the Populaire du Centre (located in the centre of France) and the Union de Reims (located in eastern France). First discussions occurred in 2002, and an agreement was signed with the Populaire du Centre in December 2003. A few months later, discussions started with a third title, Ouest France (located in western France); the agreement was signed in 2005. These agreements allowed the retrieval of files from the publishers but also authorized the consultation of these files in BnF reading rooms. As they were experiments, they had a suspension date (even though it was possible to renew them).

At the same time, technical teams from BnF and from the publishing companies examined the processes to be put in place in order to get the data. Very thorough analyses were performed. It was first decided to have one PDF per page; and to use a FTP platform to exchange data between publishers and BnF. However, getting the PDF is just the easiest part. Collecting data is useless if the preservation and access issues are not taken into account.

-From a preservation point of view, it is necessary to validate the format of the files that are retrieved. This supposes automated identification and characterization of PDFs, and a way to send back the files that aren't considered satisfactory.

-from an access point of view, each delivery has to be accompanied by the metadata that will help in recreating the structure of the newspaper, and will allow the end-user to navigate within the document. This set of metadata is called the flatplan (in French chemin de fer, or "railway").

Many other questions were discussed: naming conventions for files, delivery schedules… Some hardware (servers) and software (access interfaces) were to be bought or developed on the BnF side.

However, it was decided to follow as strictly as possible the procedures used for mass digitization, in order to manage both kinds of documents in the same workflows. For example, the needs of this "ingest track" were taken into account in the specifications of our digital repository, SPAR (Scalable Preservation and Archiving Repository), whose development was already scheduled (it started actually in 2006) [START_REF] Bermès | A data first approach to digital preservation: the SPAR project[END_REF].

Finally, the legal framework for this kind of deposit was strengthened in June 2006. A decree was published that allowed the library to collect a digital equivalent instead of the paper version for all "printed, graphical or photographical documents" [START_REF] Illien | The state of e-legal deposit in France: looking back at five years of putting new legislation into practice and envisioning the future[END_REF]. This decree establishing legal deposit of a digital "substitute" was an important step: it meant that the digital version had the same legal value as the original, notably regarding the rights related to access and preservation (whereas the rights related to the documents received previously depended on the texts of the agreements). On the other hand, the decree stated that the digital version had to be strictly identical to the one distributed to the public: indeed, as previously said, the objective of legal deposit is not only to retrieve content, but also to document the way publications were received and used by the public.

An instructive experiment… but not a conclusive one

Everything was apparently in place for the deposit, but nevertheless the experience was not conclusive. Files were in effect delivered by publishers, but BnF teams were not able to manage them. Several problems occurred:

-First, the files received by BnF were frequently different than those that were actually printed. In fact, due to the necessity for daily newspapers to take into account important news that has just arrived (for example for sport events), changes occur between the moment the publisher validates the version and the moment the newspaper is printed; it may be challenging to get the very final version.

-Second, all modifications in the fabrication processes of the publisher resulted in a modification of the metadata delivered to BnF. It was very resource-consuming for BnF to quickly adapt to these changes, especially for modifications related to the structure of the documents (the flatplan). As each publisher had its own fabrication processes, this problem was multiplied by the number of publishers in the experiment… -Third, developments for access interfaces were more difficult than expected on BnF side.

As a consequence, BnF teams were not able to integrate the retrieved files neither in the mass digitization workflow, nor in a specific workflow. The agreement signed with the Populaire du Centre ended in 2007, the one with Ouest France in 2008.

Even though these experiments were not successful, they were not useless. First, they were a chance for BnF teams to establish strong relationships with publishers. Second, some important lessons were learnt. It is critical to be able to have a digital version as similar as possible as the distributed one, so the file sent to BnF should be chosen at the very last step of the publisher production workflow. Moreover, the workflow designed by BnF has to be as generic as possible, in order not to depend on the evolution of the publisher's internal processes. And the process should be as automated as possible; otherwise it is too time consuming to individually check each newspaper issue. In this regard, this experience was very helpful when working on another way of collecting newspaper: the web archiving approach.

The continuation of collections by other means: the web archiving approach

From project to mission: the building of a whole document processing workflow BnF reflections and experiments regarding web archiving started in the same years as those concerning direct deposit, in the early 2000s. As explained above, legal deposit has always been extended over time to new kinds of cultural items. At this period, it was becoming obvious that the Internet was becoming the main publishing platform, and that it was necessary to organize the safeguarding of documents distributed on the web. In order to do so, BnF (as well as many other heritage institutions that were working together in the framework of the International Internet Preservation Consortium 2 ) decided to rely on the technologies already developed by search engine companies: harvesting robots. This software acts as an automated web user: starting from a list of URLs given by the human administrator, the robots follow hyperlinks and copy all pages, files, PDFs, videos, etc. that they may discover. However, the robots also follow strict rules that allow them, or not, to collect certain contents; it is thus possible to restrict the crawl to a specific domain name or set of domain names (e.g. to crawl only files hosted on the bnf.fr domain name), or to a specific Top Level Domain (e.g. to crawl only files hosted on the .fr TLD).

2

The mission of the International Internet Preservation Consortium (IIPC) is to acquire, preserve and make accessible knowledge and information from the Internet for future generations everywhere, promoting global exchange and international relations. It currently groups more than 40 heritage institutions. See http://netpreserve.org/about/index.php.

The first "production" crawls (i.e. the first crawls that produced content that was kept) were performed on political websites, during the presidential and parliamentary elections of 2002. The volume of archived data was modest (a few hundred gigabytes), especially compared to 2012 standards, but this project was viewed as a success because the websites were properly archived and accessible (if only for librarians at this time). Moreover, the importance of this documentation for the history of French political life was obviousin fact, it is nowadays considered one of the most valuable web archive collections, because the data captured by BnF do not exist anymore elsewhere.

A similar project was launched in 2004, during the European and regional elections. This time BnF experimented also with cooperation with regional libraries, where the latter selected local websites and the former was in charge of selecting national websites, and of harvesting all the sites selected. The same year, thanks to an agreement with Internet Archive, BnF launched its first annual .fr domain crawl.

2006 was a decisive year from the legal point of view. In June, as written above, a decree was published allowing deposit of a digital substitute instead of the paper version. But an event of greater importance occurred a few months later. On August 1 st , a law was passed that established a legal deposit of "signs, signals, writings, images, sounds or messages of any kind communicated to the public by electronic means"i.e. a legal deposit of the Internet [START_REF] Illien | The state of e-legal deposit in France: looking back at five years of putting new legislation into practice and envisioning the future[END_REF].

The experiments made by BnF (and INA for websites related to television and radio) towards the safeguarding of the memory of the Internet were thus recognized and encouraged by the legislators, who entrusted them with a dedicated mission. However, the law also states clearly that access to the collection is restricted to readers who have a demonstrable need to use them.

In that sense, access is not only open to scholars but also to any person having a need to use the archive for personal or professional reasons.

The same year, another step was reached. To harvest websites related to the presidential and parliamentary elections of 2007, BnF team built a permanent web archiving workflow (whereas 2002 and 2004 campaigns were run in project mode). In this framework, BnF teams launched frequent crawls of major French newspapers that were of course commenting on the electoral campaign.

In 2008, access to web archive collections was opened in the reading rooms of the "research library" 3 . In 2010, for the first time, the .fr annual domain crawl was not performed by Internet Archive, but by BnF own machine and human resources. This in-house domain crawl was for the BnF digital legal deposit team the most important event in 2010, as it showed that the library was able to tackle the challenge of harvesting millions of websites and hundreds of 3

The BnF is divided in two parts: a "public library", proposing books specifically acquired for this purpose, and a "research library", proposing all legal deposit collections.

millions of files in a few weeks. But the mass isn't the only difficulty that faces web archiving activities; the frequency of harvest is also another issue.

Harvesting news on the web

As a matter of fact, some websites propose content whose lifecycle is very short. Harvesting this content at a rapid pace is critical in order to be able to catch the essence of the Web: its rapidity in publishing and removing information. In order to demonstrate that its workflow was also able to harvest short-lived content, BnF decided to consider harvesting news websites as a priority4 .

In the list of these news websites, it has been chosen to include the major national and local newspaper titles, but also sites that documented the way people were accessing news on the web. To this end, the Press Service (the team in charge of managing and giving access to press collections at BnF, and which belongs to the Law, Economics and Politics Department) prepared a list of 80 titles, including:

-Press agencies (Reuters, Agence France Presse…);

-National daily newspapers (Le Monde, Le Figaro…); After a few weeks of tests, the first harvest of this list of 80 websites was launched in December 2010and those sites have been harvested each day up to now. A few months later, 20 new titles were added in order to get a comprehensive list of national and regional newspapers online. In 2011, more than 110 millions of files have been harvested through these daily crawls, for a total sum of 0.8 TB.

Successes and limits

The web archiving workflow offers several advantages:

-Similarity between the published content and the collected content (as required by legal deposit principles). The robot acts as an automated web user, so what is captured by the robot is what is seen by the human user. Note however that the robot may encounter technical difficulties when harvesting files online; in that case this similarity may not be ensured.

-Continuity of collections. Most French newspapers have now an equivalent on the web. In recent years, some of them have ceased their printed edition and decided to keep only their online version, generally because web publication is less expensive. This has been the case for example for French daily newspapers France Soir (a journal intended for a broad audience that has existed since 1944) or La Tribune (an economic newspaper launched in 1985). Without web archiving, BnF collections for those titles would have ended in 2012.

-Integration in a wider context. After its publication, a printed newspaper is a stable and discrete element. On the other hand, newspapers on the web are only a part (even though the most visible part) of a whole publication space. Almost all newspapers offer a way for their readers to react to the articles thanks to online comments. Archiving these comments will help future researchers understand how news was received by the public, and compare reactions to the same news in different newspapers. Moreover, newspaper companies frequently propose blog platforms for their journalists or even for their readers. They now have Facebook or Twitter accounts that change the way they communicate. All these kinds of documents should be archived in order to let future researchers write the history of the press in the early 2010s.

-Integration in an existing workflow. Archiving web versions of newspapers does not imply running specific processes; news websites are crawled the same way as online scientific publications, literary blogs or the whole list of websites in .fr. The current web archiving workflow at BnF covers harvesting of content, quality assurance, indexing, and access in reading rooms. The last step, ingest in our digital repository for long-term preservation, will be effective in the second half of 2012.

-From this point of view, this solution offers obvious economic advantages, as all technical processes are fully mutualisedonly the selection process remains specific.

This rather positive overview should not hide some considerable shortcomings.

-The most obvious is the fact that robots are not able to capture automatically content that is password protected. The parts of newspaper websites that are only accessible to people that pay a subscription or a specific fee are the crawlers' terra incognitaor, in other words, are part of the "deep web".

-Consequently, the PDF versions of most of the newspapers (that are generally proposed in the payment-based part of the websites) are not archived. The problem mentioned in the first section of this paperthe difficulty of managing the local paper editions of regional newspapersis therefore not solved by the web archiving solution.

-Finally, the indexation of the captured documents may be considered insufficient.

Websites archived by BnF are not catalogued (unlike what is done in some other heritage institutions). Web archive access interfaces are separate from the BnF general catalogue, so newspapers captured online are not referenced in the latter. This is a problem in order to show the collection continuity: a reader accessing the bibliographical record of France Soir or La Tribune does not necessarily know that its online equivalent is still accessible in the Library.

Building an up and running web harvesting workflow, able to face both the challenges of mass and frequency, was a major objective for BnF and a considerable outcome. Harvesting newspaper websites was only a part of the wider goal of archiving the whole French web, but it appeared that processes designed for large-scale operations were also relevant for smaller targets.

At least, up to a certain point. The goals followed when designing direct deposit processes are not fully reached by the integration of online press collections in a generic workflow; and work is still to be done in this regard. However, as direct deposit did not show the expected results, BnF has changed its approach: instead of designing specific processes, it may be more beneficial to implement specificities in an existing generic workflow. This is the essence of the new way BnF is keen to work. It will combine some characteristics of both previous solutions: close relationships with the publisher in order to get protected content but integration in the existing web harvesting workflow.

Implementing specificities in a generic workflow: the "press project"

A pilot project with Ouest France…

Ouest France was still one of the best candidates to test this new approach. It is a regional daily newspaper, but its circulation is bigger than the sum of the circulations of the three best- Finally, these tests and the good relationship with Ouest France team helped in choosing crawler settings that allow the harvest of the whole content in 4 hours, every day. Each daily crawl harvests around 2 000 files, on which 1 600 PDFs, for a total sum of 850 MB (i.e. around 0.3 TB per year against around 1 TB for the other newspaper websites). This solution will enter in production mode in the middle of 2012. This way, it is possible to tackle the two main problems that led to the abandon of the direct deposit solution:

-as the PDFs are the ones that are published online, and as this harvest is performed under the mandate of the web legal deposit (not under the mandate of the deposit of a substitute of the printed form), it is possible to be sure that the crawled version is identical to the version distributed online;

-as the PDFs are crawled by a robot, they are inserted in a whole workflow that comprises automated indexing, preservation and access.

…that has to move forward

This first experiment is only a beginning. It is now necessary to test the harvesting of protected content by robot on other examples, with other publishers. BnF team is currently defining a list of potential candidates, trying to design a representative sample: for example a national daily title, another regional title, a "pure player"… When more newspapers are archived, it will be time to think about better ways to index them. For now, there is absolutely no relationship between the general catalogue, where the printed versions are described, and the access interfaces of the web archives. This represents an important shortcoming: if, for example, there is no way for a reader to know that the local editions of Ouest France are available on the web archives after a certain date, collection continuity will apparently (even if not in reality) not be ensured. Therefore a new set of technical developments will be necessary: on the web archive side, offering a way to propose permalinks to harvested content; on the catalogue side, activating hyperlinks from the catalogue interface to web archive interfaces.

In the early 2000s, BnF teams started to think about ways to perform a legal deposit of digital publications. They proceeded in two parallel and complementary directions. On one hand, the possibility of letting the publishers directly deposit their content was explored, while on the other hand the archiving of online content, offered by the technology of harvesting robots, was tested. The goals of these two experiments were different. On one hand, the goal was to help BnF and publishers reduce the number of paper versions sent to the library; on the other hand it was to gather a fully new kind of heritage. Even the legal frameworks were different, because the former deposit was supposed to be performed under the decree regarding the deposit of a digital substitute, and the latter under the law on Internet legal depositsurprisingly, both texts were passed the same year.

Comparing both experiments, one may wonder why the first went wrong and the second succeeded. A possible explanation would be that the first solution was too dependent on the specificities of the production workflow of each publisher. When formats, metadata schemes and technologies differ from one publisher to another, each new publisher adds a new level of complexity. With web archiving, on the contrary, the collection is made on the publishing side: as all publishers are supposed to follow the standards and the rules of the web, there is theoretically nothing different between harvesting news websites, scientific publications or literary blogs. Practically, however, some kinds of documents are not well archived by robots: dynamic files, streaming sounds and videos, password-protected content. This is the reason why it is probably time to try to find a way to combine the advantages of both approaches, using on one hand the relationships built with publishers, and on the other hand the technologies and workflows of web harvesting: a third way for legal deposit.

-

  Regional daily newspapers (Ouest France, La Montagne…); -Magazines (Le Nouvel Observateur, Le Point); -Portals (Google news, Yahoo news); -Pure Players (Rue89, Médiapart); -Internet information (data journalism such as Owni.fr or sites documenting the Internet as PCInpact).

  known French national daily newspaper5 . It is indeed the biggest circulation in France: more than 800 000 copies per day. Ouest France also holds a European record, with 50 local editions printed each day. These few figures explain why, on one hand, it is important to ensure Ouest France's legal deposit; and on the other hand why it is very resource-consuming to get all the paper editions. Finally, BnF and Ouest France have an agreement in order to digitize and publish online the BnF collection of the Ouest Éclair newspaper, which is the ancestor of Ouest France. Therefore, even though BnF decided in 2008 not to get DVDs from Ouest France anymore, the goal of testing legal deposit of online newspapers starting from its example was maintained.The project was actually launched again late 2011 / early 2012. BnF and Ouest France teams held discussions, by e-mail and face-to-face, in order to design a new workflow. Different issues were raised: what kind of content to collect, how, at what frequency. It was decided to have in parallel two kinds of harvests. First, each day, BnF harvesting robot should crawl the entirety of the web version of the newspaper (http://www.ouest-france.fr/). In fact, all content proposed in the paper edition is also freely available online. This kind of harvest is not different from that of the "press collection" described above.However, it doesn't allow keeping the memory of the editorial layout of the print edition, so another crawl is launched at the same time. Each day, BnF harvesting robot should also crawl the PDFs editions available online but password-protected (http://www.ouestfranceenligne.com). To this end, a specific feature of the Heritrix robot 6 has been used. It is indeed possible to indicate in the robot settings some passwords in order to access protected pages and content. of request sent by the robot in a minute. If there is a high politeness, the load on server is low but the harvest may take longer. So it has been decided to have for Ouest France a lesser politeness than the generic one, because Ouest France website was able to stand a stronger load.

Several months of tests have been necessary in order to define the best settings:

-frequency of harvest: it has been decided to get the content every day;

-

time of harvest: in the middle of the morning because Ouest France is a morning newspaper, and because BnF teams are available if anything goes wrong; -depth of harvest; -"politeness": number 5 Namely Le Figaro, Le Monde, and Libération. These figures were given by Ouest France team. 6 Heritrix has been developed by Internet Archive in the framework of the International Internet Preservation Consortium. See https://webarchive.jira.com/wiki/display/Heritrix/Heritrix (consulted on June 14th, 2012).

See http://gallica.bnf.fr (consulted on June 14th,

2012).

The French approach towards newspaper websites harvesting has been presented at the General Assembly of the International Internet Preservation Consortium in 2010. See[START_REF] Illien | Archiving news on the Web: current projects at BnF[END_REF] for BnF presentation, others are available here: http://netpreserve.org/events/singapore.php (consulted on June 14th, 2012).


Clément Oury

LARGE-SCALE COLLECTIONS UNDER THE MAGNIFYING GLASS: FORMAT IDENTIFICATION FOR WEB ARCHIVES

Institutions that perform web crawls in order to gather heritage collections have millions -or even billions -of files encoded in thousands of different formats about which they barely know anything. Many of these heritage institutions are members of the International Internet Preservation Consortium, whose Preservation Working Group decided to address the web archive format identification issue by dedicating it as a work package.

The first outcome of this WP is an overview of the formats to be found in different types of collections (large-, small-scale…) over time. It shows that the web seems to be becoming a more standardized space. A small number of formats -frequently open -cover from 90 to 95% of web archive collections, and we can reasonably hope to find preservation strategies for them.

However, this survey is mainly built on a source -the MIME type of the file sent in the server response -that gives good statistical trends but is not fully reliable for each and every file. This is the reason why the next step of the work package is to test in a collaborative way how to use, for web archives, identification tools developed for other kinds of digital assets.

BACKGROUND

The string of bytes that compose a digital document cannot be interpreted, rendered or transformed if we have lost the knowledge of the format in which it is encoded. This is the reason why the choice of a format is a key question when one wants to produce digital documents that are supposed to last for the long term. If the institution who has custody of these documents is not the same as the institution in charge of the production, the former should try to give recommendations and advice to the later -this is possible when institutional relationships exist between the two sides, such as a public institution and public archives. We find the worst case when heritage organizations are entrusted with the preservation of documents over whose format they haven't any control… which is the case for most web archiving heritage institutions.

These institutions use crawling software, or robots, to explore the web and retrieve content that they will hold for the long term. Web harvesting is intended to keep the memory of the material that public institutions, businesses and individuals produce and distribute thanks to the Internet. Many of these heritage institutions are members of the International Internet Preservation Consortium, whose goals are to collaboratively build collections of Internet content, to promote web archiving and "to foster the development and use of common tools, techniques and standards for the creation of international archives" [START_REF]IIPC. International internet preservation consortium[END_REF].

At first sight, from the point of view of formats, web archive collections appear to be a preservation nightmare. There is no need to recall here the huge number of files harvested by crawl engines. Even the most focused archiving project has to tackle millions of files -see the Harvard University Library, whose Web Archive Collection Service dates back only from 2009 and that already has to preserve 14 million files. These figures rise to hundreds of millions of files per year for those performing crawls of entire top level domains (.au, .fr), not to mention the huge collections of Internet Archive, which in less than 15 years of existence has gathered more than 150 billion files.

The second main issue is that virtually all kind of formats are likely to be available on the Internet -as the web has become a platform where all types of documents are converging. The number of video files is steadily increasing; and the web is more and more used to host and share a huge number of executable files.

At the same time, when a crawler harvests files online, it gets very little information about the formats of the documents it is capturing. The only indication generally available is the MIME type of the file that the server sends to the harvesting robot, in the http response header. Unfortunately, this information is often badly specified (such as image/JPG instead of image/jpeg), peculiar (we once found at the BnF a curious "application/x-something" MIME type), or even totally wrong (for example, a gif image may be indicated as text/html -webmasters do not see it as a problem for rendering, because a browser is able to read gif files directly).

In short, web archiving institutions generally have millions -or even billions -of files encoded in thousands of different formats about which they barely know anything. Heritage institutions tend therefore to turn to identification tools developed in order to ensure the preservation of other kind of digital material -or developed for other purposes than preservation. This is the reason why the Preservation Working Group of the IIPC (or PWG) acknowledged the need to specifically address this critical issue through a dedicated work package. In this paper, we will present the goals of this WP and its working methodology. We will then look at the first outcomes, and finally present future work.

RELATED WORKS

Several studies have been done in order to characterize parts of the web, particularly national web domains. Their goal is to analyze the main features of the websites and web files related to a single country: notably the number of domains, the number of files per domain, the number of hyperlinks betweens websites… In these studies, we generally find a section dedicated to formats: see for example [2] for the Danish web, [START_REF] Koerbin | Report on the Crawl and Harvest of Whole Austalian Web Domain Undertaken during June and July 2005[END_REF] and [START_REF] Koerbin | The Australian web domain harvests: a preliminary quantitative analysis of the archive data[END_REF] for the Australian web, [START_REF] Miranda | Trends in Web Characteristics[END_REF] for the Portuguese web. R. Baeza-Yates et al. proposed in 2006 a comparative study of the national web characterization of several countries across the world, at various dates between 1998 and 2005 [START_REF] Baeza-Yates | Characterization of national web domains[END_REF] 1 . Note that this list of web national domain studies is not comprehensive. However, we have not identified any works specifically dedicated to file format analysis. On the other hand, there are some -even though rare -studies that examine the ability of identification tools to deal with web archives. In 2007, Bart Kiers from the Dutch National Library tested the behaviour of Jhove and Droid on web archives [START_REF] Kiers | Web Archiving within the KB and some preliminary results with JHOVE and DROID[END_REF]. The test sample was limited to ten small and medium size websites, grouping 40 000 unique objects for a total uncompressed size of 2.2 Gb. Two years later, Andrew Long from the National Library of Australia tested five format identification tools (Droid, File identifier, Jhove, TrID and the inhouse developed tool Lister) on two web archive samples (from 115 000 to 18 million files) [START_REF] Long | Long-term preservation of web archives -experimenting with emulation and migration methodologies[END_REF]. Finally, the Danish National Library and the Aarhus University Library are currently testing the use of Droid and Jhove on a 100 Tb archive [START_REF] Jensen | Evaluation report of additional tools and strategies[END_REF].

OBJECTIVES AND ORGANIZATION

A work package dedicated to format identification

The "format identification tools gap analysis" work package has two main objectives: -first, to produce an overview of the main formats generally available in web archives (using the data obtained from a large number of institutions); -second, to analyze the ability of existing tools to process web archives. This analysis will be based on tests performed by institutions on their own collections.

The final step is to propose a set of recommendations for enhancements of these tools 1) based on the test report 2) and focused on the most important file formats found in web archives.

Designing a survey

The general overview we have produced as the first part of the work package is intended to give a brief insight into the formats that were to be found on the web at different times. This is a way to participate in the general PWG goal of describing the "web technical environment" (that is what formats, software, browsers… were used on the web) over time. On the other hand, this overview should help us in comparing different collections, to identify their characteristics and their specificities.

A first difficulty lies in the fact that institutions do not share the same ways of counting the file formats they have in their archives. Some institutions use automatic compilations of the MIME reports produced by the harvesting robots (a chart indicating, for each MIME type, the number of URLs and the number of bytes); others prefer looking at the MIME information attached to each harvested file as an ARC metadata [START_REF]ARC file format[END_REF]; one library uses the indexes (called CDX files) intended to give access to the collection. Finally one institution (Library and Archives Canada, or LAC) directly ran Droid on their web archives2 .

The study is thus built on information -MIME types sent in the server response -that is commonly considered unreliable. First, this has been done for practical reasons: this kind of information was the easiest to get from member institutions. Secondly, we made the assumption that even though the information was not reliable for each individual object, it was sufficient, at a larger scale, to reflect the big picture of format distribution. This assumption has been confirmed by the results of the survey: the proportions found for the institution that used an identification tool (Library and Archives Canada) were globally similar, from 2005 to 2009, to those we found for institutions having only sent MIME information 3 .

The diversity of information sources also explains why we decided not to set up a database that institutions could have filled directly. People were asked to answer on an excel spreadsheet; compilation of this data was easy and we ensured thus better consistency.

In the survey, a first distinction is made between domain and selective crawls. Domain crawls are launched on a very large number of websites (e.g. 1,7 millions for both the .fr and .au domains in 2010), but the crawling depth is limited. Moreover, domain crawls are only performed once or twice a year. They are generally launched by national libraries in the framework of a law on digital legal deposit. On the other hand, selective crawls are performed on a more limited number of websites (from hundreds to thousands) generally chosen by librarians or archivists. Those websites may be harvested many times a year, and crawling depth is generally better. The two approaches do not serve the same purpose: -Domain crawls are the best way to obtain a representative sample -a snapshot -of the web. According to R. Baeza-Yates et al. [START_REF] Baeza-Yates | Characterization of national web domains[END_REF], crawls of national domains provide a good balance between diversity and completeness by including pages that share a common geographical, historical and cultural context but written by diverse authors in different organizations. -Even though data from selective crawls may be considered less representative of all documents available on the web (since human, subjective selection replaces automatic selection by a robot), taking them into account is also important: 1) because the addition of many selective crawls performed in the same year can eventually produce an amount of data superior to that of a single domain crawl; 2) because data from selective crawls may be considered as more "valuable" and may thus deserve more costly preservation actions.

It would not have been feasible to gather information for every year; so the survey focuses on arbitrarily chosen years. It was decided to start from 1997 (date of the first Swedish domain crawl) and to take the years 2000, 2005, 2007 and 2009. We choose to add 2007 because more information was likely to be available for recent years (many institutions didn't start their web archiving program before 2007).

Finally, we asked people to give only the list of the 50 most ranked formats. The ranking was calculated according to the number of objects in this format. All institutions were indeed not able to compute the number of bytes per format.

A representative sample of respondents

So far, we have received answers from ten institutions 1 . We can consider that this sample is representative of the diversity of the different collections IIPC members may hold: three institutions sent data for domain crawls; eight institutions sent data for selective crawls (some institutions sent data for both domain and selective crawls). Moreover, within selective crawls, there were "topical" crawls (focused on a specific theme such as elections, activist websites, women blogs), or crawls 1 Namely the national libraries of Australia (NLA), France (BnF), Netherlands (KB-NL), Sweden, the Library of Congress (LC), the British Library (BL), Harvard University Library, Library and Archives Canada (LAC), The National Archives of United Kingdom (TNA), and the Internet Archive (IA). dedicated to the harvesting of governmental publications (notably from National Archives). This may have consequences on formats to be found in both types of collections. Finally, Internet Archive sent information on their crawls of the entire web.

In this sample, we found also institutions that were the first to harvest websites at a large scale (notably Internet Archive and the National Library of Sweden), as well as institutions that have recently entered this field. This explains the differences between collection sizes: from 15 million to 96 billion files. Note also that the collections analyzed come from American (four institutions) and European (five institutions) countries, and from Australia (National Library of Australia). To compile this information, we calculated the average percentage of formats in different web archives instead of using the total number of files of each collection (e.g. if a format represents 30% of collection A, 20% of B and 10% of C, the average percentage is 20%, even though institution A holds three times more data than the two others). This principle has been applied to all computations. We did so to avoid an over-representation of big collections against smaller ones, which would have prevented all comparisons.

It is not surprising to see, all over the period, a strong domination of html, jpeg and gif. It is even more impressive if we look at the percentage of files: for the year 2009, 70% of files are encoded in html, 18% in jpeg, 6% in gif. However, this chart allows us to identify the rise and fall of some formats. We may notice the destiny of png (0.006% of web collections in 1997), which now ranks in fifth place (that is… not even 1.2% of available files). Observe also the increasing rank of css and xml files (while its ancestor sgml quickly disappeared).

On the other hand, some formats that were very popular twelve years ago now rank at a very low place. This is the case of postscript (from the 7 th to the 45 th place), wav audio files, and even quicktime video files. This is another surprising lesson of this overview: even though we know that the web is increasingly becoming a huge video platform, large-scale crawls don't seem able to tackle the video harvesting issue. The number of captured audiovisual files is increasing (as an example, Sweden crawled 1 300 quicktime videos in 1997, 11 000 in 2005 and 25 000 in 2009), but not as fast as the overall growth of our collections -and definitively not as fast as the percentage of audiovisual content on the web.

From a preservation point of view, however, these figures are good news. Standardized formats are gaining ground against proprietary ones (for example jpeg against gif; xml and png are open formats).

Comparisons between domain crawls

Statistically, significant differences between collections should not appear in such a mass of data. We can expect web technologies -and formats -to be equally distributed within the various countries. In fact, if we look at the collections issued from the 2009 domain crawls (France, Sweden, Australia), we find exactly the same formats in the list of the ten most ranked 1 . And we find only 36 different formats in the list of the 30 most ranked.

However, older collections do not show such strong similarities. There is a greater variety of MIME types in the list of high ranked formats for the domain crawls of previous years. We can thus conclude that as the web becomes more commonly used, national dissimilarities in the use of web technologies tend to fade away.

Comparing selective and domain crawls

A similar compilation has been made for collections issued from selective crawls. The goal was also to examine if there were significant discrepancies for collections coming from large-and small-scale harvests, and between small-scale collections from different institutions.

Again, there are no obvious differences between collections. If we compare the average distribution of formats in domain crawls with the average distribution in selective crawls, from 2005 to 2009, we notice few variations. For example, we find exactly the same formats in the list of the 10 highest ranked, in 2007 and 2009.

However, a more careful analysis shows some interesting features of specific collections. At the end of the list of the 30 most ranked formats for selective crawls, we find many video formats (such as asf, windows media video or flash videos) that do not appear in domain crawls. Focusing only on formats available in large-scale collections would lead us to leave out these files.

It is also possible to identify characteristics that are related to the nature of the collection. As an example, The National Archives of the United Kingdom are entrusted with the harvesting of governmental publications and websites. This is probably the reason 1 Excluding the "no-type" format.

why we notice a larger proportion of pdf and desktop application formats2 .

Moreover, this survey allows us to discover formats that are specific to a collection, over time. Web archives from the Library of Congress show a rather strong proportion of real audio files (MIME type: audio/x-pnrealaudio). The proportion of flash video files which the French National Library (BnF) holds in its 2007 and 2009 selective collections is seven times higher than the average. This last case is explained by the fact that BnF launched in 2007 specific crawls of a video broadcasting platform called Dailymotion, the French equivalent of YouTube.

If we only look at major web archive trends, we will not consider Excel spreadsheets, real audio files or flash video files as being formats that deserve a specific preservation strategy. This is why institutions should also look at their own data in order to assess specific preservation needs. We should not forget the preservation operations won't apply to the web itselfthey will be designed for the heritage collections derived from the web.

FIT FOR PRESERVATION?

Following on from this, are heritage institutions familiar with such file formats? To answer this question, we can look at a report produced by the National Library of Netherlands (KB-NL). The library conducted a survey on the digital documents held by libraries, archives and museums [START_REF] Van Bussel | Gap analysis: a survey of PA tool provision[END_REF]. From the replies of 76 institutions, they drew up a list of 137 different formats, of which 19 were quoted by seven or more respondents.

5 of these 19 formats only do not figure in our top 20 formats of 2009 domain or selective crawls 3 . On the other hand, the distribution is very different. For example, the most cited format in the KB-NL study is tiff (50 occurrences), while it does not even appear in the top 20 lists for web archives. Similarly, gif and html appear only at the 8 th and 10 th rank (against 1 st and 3 rd in web archives). We found similar percentages only for jpeg (2 nd rank in both studies), pdf (respectively 3 rd and 4 th rank) and xml (4 th and 8 th rank).

The case of tiff files -frequently used for digitization -shows that heritage institutions producing digital documents rarely use the same formats as people that commonly publish online. Yet, can we conclude from this that web formats aren't fit for preservation? To have a first answer, let us refer to the list of "Recommended Data Formats for Preservation Purposes" established by the Florida Digital Archive [START_REF]Recommended Data Formats for Preservation Purposes in the Florida Digital Archive[END_REF].

Formats are classified in three categories: high, medium and low confidence level. We applied these criteria to the average distribution of 2009 Overall, the formats available on the web are not the worst we can imagine from a preservation point of view. Note that for some formats (such as html or pdf), there is a different level of confidence depending on the format version. Since this kind of information is not available in MIME type reports, we need to look at the response from Library and Archives Canada -and to assume that its sample is representative. Again, using the 2009 figures: -xhmtl files (high confidence) represent 11% of the html files (other versions have a medium confidence grade)1 ; -on the other hand, 98% of PDF files only have a "low confidence" grade. As a matter of fact, PDF-A (high confidence) and PDF-X2 and 3 (medium confidence) respectively represent 0.5 and 1.5% of the total.

USING FORMAT IDENTIFICATION TOOLS WITH WEB ARCHIVES

Although this survey provides a first insight into the formats of the collections we hold, this is not enough to guarantee their preservation in the long term. First, it only gives statistical trends: at the level of each individual file, the information is not reliable. No migration operation is possible without such knowledge. Secondly, nothing is said about the format versionwhich stands as an obstacle for emulation strategies, because we won't emulate the same browser, say, for html 2.0 and 4.0. Therefore, whatever preservation strategy is chosen, relevancy of format information remains a critical issue. This is the reason why the use of identification tools appears as a necessary step towards a better understanding of our collections. By identification tools, we mean all software that "describes" the format of a specific file. It can range from simple format identification to validation, feature extraction or assessment2 . This definition may include tools such as Droid [START_REF]DROID (Digital Record Object Identification[END_REF], Jhove (v1 [START_REF] Jhove | [END_REF] & 2 [START_REF][END_REF]) or the National Library of New Zealand metadata extraction tool [START_REF]National Library of New Zealand Metadata Extraction Tool[END_REF].

Previous reports have already outlined several issues that arise when using identification tools for web archives: -Some major formats are not supported by characterization tools. For example, neither the NLNZ Metadata extraction tool nor Jhove 1&2 are currently able to characterize PNG files. There is no Jhove module for MP3, even though it is the most frequent audio format within web archives… -Files may not be well formed, which is a problem for identification. This is mainly the case for html files that are frequently hand written or modified. KB-NL reported in 2007 that none of the 20 000 processed html files were considered valid or even well-formed [START_REF] Kiers | Web Archiving within the KB and some preliminary results with JHOVE and DROID[END_REF]. Let us hope that the growing use of xhtml will reduce this risk. -Scalability and performance probably remain the major issue for web archives. Tools need to be able to process hundreds of millions of files. NLA report evaluates that it would take 42 days for Droid to identify 18 millions files (0.8 Tb) on a single-core machine, whereas up to a billion files can be harvested in few weeks during a domain crawl [START_REF] Long | Long-term preservation of web archives -experimenting with emulation and migration methodologies[END_REF]. -Finally, in most cases, files are encoded in ARC files [START_REF]ARC file format[END_REF]. ARCs are container files where the objects harvested on the web are recorded -along with metadata sent by the server or computed by the robot such as harvesting date, IP of the server, MIME response… So far, analyzing a file requires unpacking all harvested files, which adds another step to the process. This is the reason why tests of identification tools on web archives were mostly run on non-ARC collections 1 .

FUTURE WORK

The objectives of the PWG are now to organize a collaborative review of the main identification tools. We will build upon the format overview to organize the test protocol and to define the test samples. These tests are intended to assess the efficiency of the tools (notably by providing metrics), and report on any difficulties encountered (e.g. with specific file formats, with the management of container formats, or due to the number of files). Recommendations and best practices for using these tools will be proposed. Finally, we hope to present a set of enhancements for these tools to address specific web archive issues and requirements. Fortunately, the institutions that are leading the development of the major tools generally hold web archives along with other digital collections, and are also IIPC members.

In addition, test outcomes will also help us to enrich the general overview of the formats in web archives. It will also be necessary to find a durable way to store, update and make available this format overview. An Excel spreadsheet was a convenient way to compile information coming from disparate sources. The work done so far can now be used as a test bench to design a real database, where each IIPC member institution could add its own data.

CONCLUSION

The first outcomes of this study allow us to avoid an overly pessimistic point of view: even though web archives consist of files over which we have no control, it is not impossible to ensure their preservation.

There is indeed much good news: considering the major trends, it looks like the web is becoming a more and more standardized space. Standard and open formats are gaining ground. Moreover, existing differences between "national" webs are tending to disappear.

The second reassuring piece of news is that most files are encoded in a very limited number of formats. Having a preservation strategy for the ten highest ranked 1 This is the case in [START_REF] Long | Long-term preservation of web archives -experimenting with emulation and migration methodologies[END_REF] and for data coming from Library and Archives Canada.

formats would be sufficient to render from 95 to 98% of the collection 2 . Yet, this shouldn't lead to an overly optimistic vision. The importance or the "value" of a format does not only depend on the number of files in which they are encoded. This is evident if we choose as the unit of reference not the number of object, but the size. In fact, the ten higher ranked formats (in terms of number of files) generally cover only 50 to 80% of the bytes of the collection 3 . Even the 30 most ranked formats cover only from 70 to 95% of the total collection size. This is mainly due to the size of audiovisual objects, which are commonly 1 000 to 10 000 times bigger than html pages. And precisely, video files may be considered by curators or researchers as more valuable -not only because they hold rich content, but also because without these files, heritage web archives collections would not be representative of the "living" web.

Moreover preservation actions need to be focused as a priority on file formats that risk becoming obsoleteand this is unlikely to be the case for the major web formats, at least in the short term. This is the reason why institutions may choose to focus on formats that they alone hold: and in this case, having an overview of what is available in other archives will be very useful. This is a way for collaborative work -at national or international level -to provide the tools, knowledge and advice to help institutions to define their own preservation objectives.

ACKNOWLEDGEMENTS

The author gratefully acknowledges all those who contributed to the survey. He wishes also to thank Johan van der Knijff, Sara van Bussel and René Voorburg from the National Library of the Netherlands for the compilation of the existing literature on web archives formats identification they performed as part of the work package.

Figure 1 .

 1 Figure 1. Number of collections for which responses were received

  Number of different in the list of high ranked formats of the three domain crawls collections, 2005 to 2009.

Table 1 .

 1 High ranked formats in large-scale collections, from 1997 to 2009 (increasing formats are in bold, decreasing in italic)

	1997	2000	2005	2007	2009
	1 text/html	text/html	text/html	text/html	text/html
	2 image/gif	image/gif	image/jpeg	image/jpeg	image/jpeg
	3 image/jpeg	image/jpeg	image/gif	image/gif	image/gif
	4 text/plain	text/plain	text/plain	application/pdf	application/pdf
	5 application/octet-stream unknown	application/pdf	image/png	image/png
	6 application/zip	application/pdf	no-type/unknown	text/plain	text/plain
	7 application/postscript	application/octet-stream	image/png	text/css	text/css
	8 application/pdf	application/zip	text/css	app./x-javascript	app./x-javascript
	9 audio/x-wav	audio/x-pn-realaudio	application/x-javascript	app./x-shockwave-flash app./x-shockwave-flash
	10 unknown	application/msword	app./x-shockwave-flash	no-type/unknown	text/xml
	11 application/msword	application/postscript	application/octet-stream	text/xml	no-type/unknown
	12 image/tiff	image/png	application/msword	application/xml	application/xml
	13 application/x-tar	text/css	text/xml	application/msword	application/octet-stream
	14 video/quicktime	audio/midi	application/zip	app./octet-stream	application/msword
	15 audio/x-aiff	audio/x-wav	application/x-tar	image/pjpeg	application/rss+xml
	16 application/rtf	application/x-tar	image/pjpeg	audio/mpeg	text/javascript
	17 video/mpeg	application/x-tex	application/postscript	application/zip	image/pjpeg
	18 app./vnd.ms-powerpoint audio/x-pn-realaudio-plugin audio/x-pn-realaudio	text/javascript	audio/mpeg
	19 audio/x-mpeg	audio/x-midi	audio/mpeg	application/rss	application/javascript
	20 Javascript	audio/x-sidtune	application/x-gzip	image/bmp	application/atom+xml
	21 app./x-shockwave-flash	application/mac-binhex40	application/xml	image/x-icon	application/zip
	22 image/png	image/tiff	application/vnd	app./x-zip-compressed	image/bmp
	23 application/sgml	video/quicktime	app./x-zip-compressed	application/atom	app./force-download
	24 text/css	application/x-gzip	image/bmp	application/vnd	image/x-icon
	25 video/x-ms-asf	chemical/x-pdb	text/javascript	video/quicktime	app./vnd.ms-excel
	26 x-world/x-vrml	audio/basic	image/jpg	audio/x-pn-realaudio	app./x-zip-compressed
	27 application/vnd	application/vnd.ms-excel	video/quicktime	video/x-ms-wmv	video/x-ms-wmv
	28 image/pjpeg	audio/mpeg	audio/prs.sid	audio/x-wav	video/quicktime
	29 application/x-gzip	application/rtf	video/mpeg	application/postscript	app./vnd.ms-powerpoint
	30 audio/x-midi	video/mpeg	image/tiff	app./force-download	audio/x-wav
	4. FIRST OUTCOMES: GENERAL
	CHARACTERIZATION
	4.1. Web (archive) trends, 1997 to 2009
	As a first outcome of this study, we can draw a general
	overview of the main format trends in web archives.
	This kind of study is not really new (see above in
	"related works"), but it is the first time that it is
	performed on collections at such a scale, from 1997 to
	2009. We have compiled information from Internet
	Archive (available from 1997 to 2005) and from domain
	crawls of Sweden (1997 to 2009), Australia and France 1
	(both 2005 to 2009 2 ). Note however that there is an
	unavoidable gap between the web trends and the web
	archive trends, because file formats that are hardly
	harvested by crawlers -flash files, rich media... -are
	under-represented in archives (and heritage institutions'
	objective is to reduce this gap by improving their
	harvesting tools).
	1 Information used from the BnF for the year 2009 actually dates from
	November/December 2008.
	2

Table 3 .

 3 domain and selective crawls (only top 20 highest ranked formats): Average

		Average	Confidence
	MIME Type	proportion	level
			High or
	text/html	69,319%	Medium
	image/jpeg	18,447%	Medium
	image/gif	5,722%	Medium
	application/pdf	1,204% High to Low
	image/png	1,144%	High
	text/plain	0,629%	High
	text/css	0,516%	Medium
	text/xml	0,311%	High
	no-type	0,275%	n/a
	application/x-javascript	0,336%	Medium
	application/x-shockwave-flash	0,316%	Low
	application/msword	0,087%	Low
	application/xml	0,121%	High
	image/pjpeg	0,066%	Medium
	text/javascript	0,067%	Medium
			Medium or
	application/octet-stream	0,089%	Low
	application/javascript	0,040%	Medium
	audio/mpeg	0,053%	Medium
	application/rss+xml	0,067%	High
	image/bmp	0,023%	Medium
	proportion of MIME types in 2009
	domain crawls
		Average	Confidence
	MIME Type	proportion	level
			High or
	text/html	67,979%	Medium
	image/jpeg	11,885%	Medium
	image/gif	6,613%	Medium
	unknown/no-type	3,440%	n/a
	application/pdf	3,256% High to Low
	text/plain	1,286%	High
	image/png	1,182%	High
	text/css	0,847%	Medium
	application/x-javascript	0,551%	Medium
	text/xml	0,444%	High
	application/x-shockwave-flash	0,326%	Low
	application/atom+xml	0,187%	High
	application/xml	0,180%	High
	application/msword	0,167%	Low
			Medium or
	application/octet-stream	0,114%	Low
	text/javascript	0,104%	Medium
	application/rss+xml	0,097%	High
	audio/mpeg	0,077%	Medium
	application/vnd.ms-powerpoint	0,069%	Low
	application/vnd.ms-excel	0,061%	Low

Table 4 .

 4 Average proportion of MIME types in 2009 selective crawls

  Proceedings of the Fifth International Conference on Preservation of Digital Objects, London, Great Britain, 2008. [2] Andersen, B. "The DK-domain: in words and figures". Netarkivet.dk, Aarhus, Denmark, 2005. Online: http://netarchive.dk/publikationer/DFrevy_ english.pdf. 2 For domain crawls, from 2005 to 2009, these 10 formats are: html, jpeg, gif, pdf, plain text, png, css, javascript and shockwave-flash. 3 Size in bytes is not available for all collections. This ratio of 50 to 80% has been computed from LC selective crawls (2005 to 2009), NLA domain crawl (2009), BnF 2009 domain crawl and 2007-2009 selective crawls.

	[1] Abrams, S., Owens E. and Cramer, T. "What? So
	what?':	The	Next-Generation	JHOVE2
	Architecture for Format-Aware Characterization",

© 2010 Austrian Computer Society (OCG).

This study compiles data coming from Austria, Brazil, Chile, Greece, Indochina, Italy, Portugal, South Korea, Spain, Thailand, UK, and from two general analyses of several African countries performed in

and 2004.

Thanks to the Pronom database[START_REF]Pronom technical registry[END_REF], we converted the LAC answer, which used Pronom identifiers, into MIME types.

Note as an exception a surprisingly low number of gif files in the 2005 collection (only 0.8% of the collection against an average percentage of 7%).

In 2005 and 2007, twice the percentage of pdf and word files, five times the percentage of excel files.

TIFF, WAV, AVI, MPEG (2) and MDB files are neither in the domain nor the selective crawls list. BMP is only in the domain crawls list. XLS and PPT are only in the selective crawls list.

Note that there is a specific MIME type for xhtml documents: application/xhtml+xml. However, this MIME type is very rarely used, and commonly replaced for convenience reasons by text/html. Even W3C recommends doing so. See http://www.w3.org/TR/xhtml-mediatypes/.

2 These categories are defined in[1].


Ludovic Rifford
email: rifford@math.unice.fr

REFINEMENT OF THE BENOIST THEOREM ON THE SIZE OF DINI SUBDIFFERENTIALS

Keywords: 2000 Mathematics Subject Classification. Primary: 49J52, 26B35; Secondary: 26BXX Nonsmooth analysis, generalized differentials

 Theorem 3.3]), and as a corollary we obtain a classical result concerning the singular set of locally semiconcave functions.

1. Introduction. Let f : R n → R ∪ {+∞} be any lower semicontinuous function, the lower Dini subdifferential of f at x in the domain of f (denoted by dom(f )) is defined by

∂ -f (x) = ζ ∈ R n | lim inf y→x f (y) -f (x) -ζ, y -x y -x ≥ 0 .
As it is well-known, for every x ∈ dom(f ), the set ∂ -f (x) is a possibly empty convex subset of R n . Now let k ∈ {1, • • • , n} be fixed; we call k-dimensional Dini singular set of f , denoted by D k (f ), the set of x ∈ dom(f ) such that ∂ -f (x) is a nonempty convex set of dimension k. Moreover, we call Dini singular set of f , the set defined by D(f

) := k∈{1,••• ,n} D k (f ).
Before stating our result, we recall that, given r ∈ {0, 1, • • • , n}, the set C ⊂ R n is called a r-rectifiable set if there exists a Lipschitz continuous function φ : R r → R n such that C ⊂ φ(R r ). In addition, C is called countably r-rectifiable if it is the union of a countable family of r-rectifiable sets. The aim of the present short note is to extend a result by Benoist, who proved that D(f ) is countably (n -1)-rectifiable (see [START_REF] Benoist | The size of the Dini subdifferential[END_REF]Theorem 3.3]), and to obtain as a corollary a classical result on locally semiconcave functions. We prove the following result.

Theorem 1.1. Let f : R n → R ∪ {+∞} be a lower semicontinuous function. Then for every k ∈ {1, • • • , n}, the set D k (f ) is countably (n -k)-rectifiable.
Let us now recall briefly the notions of semiconcave and locally semiconcave functions; we refer the reader to the book [START_REF] Cannarsa | Semiconcave functions, Hamilton-Jacobi equations, and optimal control[END_REF] for further details on semiconcavity (see also [START_REF] Rifford | Nonholonomic Variations: An Introduction to Sub-Riemannian Geometry[END_REF]). Let Ω be an open and convex subset of R n , u : Ω → R be a continuous function, and C be a nonnegative constant. We say that u is C-semiconcave or semiconcave on Ω if

µu(y) + (1 -µ)u(x) -u(µx + (1 -µ)y) ≤ µ(1 -µ)C 2 |x -y| 2 , (1)
∈ {1, • • • , n}, the set Σ k (u) is countably (n -k)- rectifiable.
Our proofs combine techniques developed by Benoist in [START_REF] Benoist | The size of the Dini subdifferential[END_REF] and Cannarsa, Sinestrari in [START_REF] Cannarsa | Semiconcave functions, Hamilton-Jacobi equations, and optimal control[END_REF].

Notations: Throughout this paper, we denote by •, • and | • |, respectively, the Euclidean scalar product and norm in R n . For any x ∈ R n and any r > 0, we set 2. Preliminary results. Let k ∈ {1, • • • , n-1}, we call k-planes the k-dimensional subspaces of R n . Given a k-plane Π, we denote by Π ⊥ its orthogonal complement in R n . Given x ∈ R n , we denote by p Π (x) and p Π ⊥ (x) the orthogonal projections of x onto Π and Π ⊥ respectively. If Π, Π ′ are two given k-planes, we set

B(x, r) := {y ∈ R n | |y -x| < r} and B(x, r) := {y ∈ R n | |y -x| ≤ r}.
d(Π, Π ′ ) := p Π -p Π ′ ,
where • is the operator norm of a linear operator in R n . We notice that the set of k-planes, denoted by P k , equipped with the distance d, is a compact metric space. Hence it admits a dense countable family {Π k i } i≥1 . In the sequel, we denote by

B k d (Π, ǫ) the set of Π ′ ∈ P k such that d(Π, Π ′ ) ≤ ǫ. Given a compact set K ⊂ R n , we recall that the support function σ K of K is defined by ∀h ∈ R n , σ K (h) := max { w, h | w ∈ K} .
We notice that if conv(K) denotes the convex hull of K, then we have

σ conv(K) = σ K . Moreover if K, K ′ are two compact sets such that K ⊂ K ′ , then σ K ≤ σ K ′ .
Given a k-plane Π, we define the function σΠ : R n → R by

∀h ∈ R n , σΠ (h) := max w, h | w ∈ Π ∩ B .
The following result is useful for the proof of our theorem.

Lemma 2.1. Let Π, Π ′ be two k-planes and h ∈ R n , then we have

|σ Π (h) -σΠ ′ (h)| ≤ d(Π, Π ′ )|h|. ( 2
)
Proof. There is w ∈ Π ∩ B such that σΠ (h) = w, h . Set

d := |p Π ′ (w)| .
Notice, that since w ∈ B, we have necessarily d ≤ 1, which means that p Π ′ (w) belongs to Π ′ ∩ B. Hence we have

σΠ ′ (h) ≥ p Π ′ (w), h = p Π ′ (w) -p Π (w), h + w, h ≥ -p Π ′ (w) -p Π (w) |h| + σΠ (h) ≥ -p Π ′ -p Π |w||h| + σΠ (h) ≥ -d(Π, Π ′ )|h| + σΠ (h).
We deduce that σΠ ′ (h) -σΠ (h) ≥ -d(Π, Π ′ ) h . By symmetry, we obtain the inequality (2).

3. Proof of the theorem. Let k ∈ {1, • • • , n} be fixed. Let us choose a sequence (v j ) j≥1 which is dense in R n and let us define, for ω = (r, i, j, l) ∈ I := (N * ) 4 , the set D ω constituted of elements x belonging to the closed ball Br such that f (x) ≤ r, and such that there exist Π ∈ B k d Π i , 1 4r , ρ ≥ 9 r and ζ ∈ B v j , 1 2r satisfying:

∀y ∈ B x, 1 l , f (y) ≥ f (x) + ζ, y -x + ρσ Π (y -x) - 1 2r |y -x|. (3)
Lemma 3.1. We have the following inclusion:

D k (f ) ⊂ ω∈I D ω . Proof. Denote by e k 1 , • • • , e k k the standard basis in R k and choose a constant ν k > 0 such that Bk ν k ⊂ conv ±e k 1 , • • • , ±e k k , (4)
where Bk ν k denotes the closed ball centered at the origin with radius ν k in R k . Let x ∈ D k (f ); there are ζ ∈ R n and µ > 0 such that the convex set ∂ -f (x) contains the k-ball B defined as, B := B(ζ, µ) ∩ H, where H denotes the affine subspace of dimension k which is spanned by ∂ -f (x) in R n . Choose r ≥ 1 such that |x| ≤ r, f (x) ≤ r, and µ ≥ 9 ν k r . By (4), there are k

vectors e 1 , • • • , e k ∈ R n of norm 1 such that Bν k µ ∩ Π ⊂ µE ⊂ Bµ , (5)
where Π and E are defined by

Π := SPAN{e 1 , • • • , e k } and E := conv(±e 1 , • • • , ±e k ).
For every m ∈ {1, • • • , k} and every ǫ = ±1, the vector ζ + µǫe m belongs to B, then there exists a neighborhood V m,ǫ of x such that

∀y ∈ V m,ǫ , f (y) ≥ f (x) + ζ + µǫe m , y -x - 1 2r |y -x|.
Hence we deduce that for every y ∈ m∈{1,••• ,k},ǫ=±1 V m , we have

f (y) ≥ f (x) + ζ, y -x + max {µ ǫe m , y -x | m = 1, • • • , k, ǫ = ±1} - 1 2r |y -x|.
But by (5), we have for every

h ∈ R n , max {µ ǫe m , h | m = 1, • • • , k, ǫ = ±1} = σ µE (h) ≥ σ ( Bν k µ ∩Π) (h) = ν k µσ Π (h).
We conclude easily by density of the families {Π k i } i≥1 , {v j } j≥1 .

Set for every i ≥ 1, the cone

K i := h ∈ R n | σΠi (h) ≤ 1 2 h .
We have the following lemma.

Lemma 3.2. For every ω = (r, i, j, l) ∈ I and every x ∈ D ω , we have

D ω ∩ B x, 1 l ⊂ {x} + K i . Proof. Let y ∈ D ω ∩ B x, 1 l be fixed. There are Π y ∈ B k d Π i , 1 4r , ρ y ≥ 9 r and ζ y ∈ B v j , 1 2r such that ∀z ∈ B y, 1 l , f (z) ≥ f (y) + ζ y , z -y + ρ y σΠy (z -y) - 1 2r |z -y|.
In particular, for z = x, this implies

f (x) ≥ f (y) + ζ y , x -y + ρ y σΠy (x -y) - 1 2r |y -x| ≥ f (y) + ζ y , x -y - 1 2r |y -x|. (6)
But since x ∈ D ω , there are

Π x ∈ B k d Π i , 1 4r , ρ x ≥ 9 r and ζ x ∈ B v j , 1 2r such that f (y) ≥ f (x) + ζ x , y -x + ρ x σΠx (y -x) - 1 2r |y -x|. (7)
Summing the inequalities (6) and (7), we obtain

0 ≥ ζ x -ζ y , y -x + ρ x σΠx (y -x) - 1 r |y -x|. But |ζ x -ζ y | ≤ 1 r , hence ρ x σΠx (y -x) ≤ 2 r |y -x|.
Which gives by ( 2)

σΠi (y -x) = (σ Πi (y -x) -σΠx (y -x)) + σΠx (y -x) ≤ d(Π i , Π x )|y -x| + 2 ρ x r |y -x| ≤ 1 4r |y -x| + 1 4 |y -x| ≤ 1 2 |y -x|.
Lemma 3.3. Let ω = (r, i, j, l) ∈ I and x ∈ D ω be fixed; set

A := p Π ⊥ i D ω ∩ B x, 1 2l .
For every y ∈ A, there exists a unique z = z y ∈ Π i such that

y + z ∈ D ω ∩ B x, 1 2l .
Moreover, the mapping ψ ω : y ∈ A → z y is Lipschitz continuous.

Proof. First of all, for every y ∈ A, there is, by definition of A, some x ∈ D ω ∩ B x, 1 2l such that y = p Π ⊥ i (x). Since xy ∈ Π i , this proves the existence of z y . To prove the uniqueness, we argue by contradiction. Let y ∈ A, assume that there are z = z ′ ∈ Π i such that y + z and y + z ′ belong to D ω ∩ B x, 1 2l . Since y + z ∈ D ω , by the previous lemma, we know that

D ω ∩ B y + z, 1 l ⊂ {y + z} + K i .
But since both y + z and y + z ′ belong to B x,

1 2l , y + z ′ belongs clearly to D ω ∩ B y + z, 1 l . Hence y+z ′ ∈ {y+z}+K i . Which means that (y+z ′ )-(y+z) = z ′ -z belongs to K i . But since z ′ -z ∈ Π i , we have that σΠi (z ′ -z) = |z ′ -z| > 1 2 |z ′ -z|.
We find a contradiction. Let us now prove that the map ψ ω is Lipschitz continuous. Let y, y ′ ∈ A be fixed. By the proof above we know that ψ ω (y) = xy (resp. The proof of the lemma is completed.

ψ ω (y ′ ) = x ′ -y ′ ) where x is such that y = p Π ⊥ i (x) (resp. y = p Π ⊥ i (
From the lemma above, for every ω = (r, i, j, l) ∈ I and every x ∈ D ω , the map φ : A → R n defined as, ∀y ∈ A, φ(y) = y + ψ ω (y), is Lipschitz continuous and satisfies

D ω ∩ B x, 1 2l ⊂ φ(A).
Since A ⊂ Π ⊥ i , such a map can be extended into a Lipschitz continuous map from Π ⊥ i into R n . Since Π ⊥ i has dimension (nk), we deduce that the set D ω ∩ B x, 1 2l is (nk)-rectifiable. The fact that any set D ω can be covered by a finite number of balls of radius 1 2l completes the proof of the theorem.

  Finally, we use the abbreviations B r := B(0, r), Br := B(0, r), B := B 1 , and B := B1 .

  x)). Set z := ψ ω (y), z ′ := ψ ω (y ′ ) and h := x ′ -x. Since x = y+y and x ′ = y ′ +z ′ where y, y ′ ∈ Π ⊥ i and z, z ′ ∈ Π i , we have|h| 2 = |z ′ -z| 2 + |y ′ -y| 2 . But σΠi (h) = |z ′ -z| ≤ 1 2 |h|. Hence we obtain that |z ′ -z| ≤ |x ′ -x| = |h| ≤ 2 √ 3 |y ′ -y|.

Acknowledgements. The author is very indebted to Joël Benoist for fruitful discussions.