Denis Roegel

Denis Roegel Sustainable digitization

Examples of bad digitizations (cont'ed) Archive.org : 數理精蘊 (Shuli Jingyun, 1723) The same, but better:

五 五 五 三 三 三 一 一 一 五 五 五 二 二 二八 八 八一 一 一 五 五 五 二 二 二六 六 六一 一 一 五 五 五 二 二 二四 四 四一 一 一 五 五 五 二 二 二二 二 二一 一 一 五 五 五 二 二 二 一 一 一 五 五 五 一 一 一八 八 八一 一 一 五 五 五 一 一 一六 六 六一 一 一 二 二 二 四 四 四 三 三 三 二 二 二 七 七 七 六 六 六 五 五 五 三 三 三 七 七 七 七 七 七 一 一 一 六 六 六 七 七 七 四 四 四 七 七 七 三 三 三 二 二 二 九 九 九 五 五 五 三 三 三 一 一 一 七 七 七 三 三 三 八 八 八 九 九 九 一 一 一 二 二 二 九 九 九 四 四 四 八 八 八 七 七 七 一 一 一 三 三 三 五 五 五 三 三 三 二 二 二 五 五 五 二 二 二八 八 八二 二 二 五 五 五 二 二 二六 六 六二 二 二 五 五 五 二 二 二四 四 四二 二 二 五 五 五 二 二 二二 二 二二 二 二 五 五 五 二 二 二 二 二 二 五 五 五 一 一 一八 八 八二 二 二 五 五 五 一 一 一六 六 六二 二 二 三 三 三 五 五 五 九 九 九 三 三 三 一 一 一 四 四 四 八 八 八 一 一 一 一 一 一 六 六 六 二 二 二 八 八 八 三 三 三 七 七 七 七 七 七 六 六 六 二 二 二 五 五 五 一 一 一 二 二 二 一 一 一 二 二 二 二 二 二 五 五 五 一 一 一 一 一 一 一 一 一 二 二 二 二 二 二 七 七 七 八 八 八 九 九 九 一 一 一 八 八 八 二 二 二 二 二 二 八 八 八 一 一 一 二 二 二 二 二 二 三 三 三 五 五 五 八 八 八 三 三 三 一 一 一 四 四 四 五 五 五 三 三 三 三 三 三 五 五 五 二 二 二八 八 八三 三 三 五 五 五 二 二 二六 六 六三 三 三 五 五 五 二 二 二四 四 四三 三 三 五 五 五 二 二 二二 二 二三 三 三 五 五 五 二 二 二 三 三 三 五 五 五 一 一 一八 八 八三 三 三 五 五 五 一 一 一六 六 六三 三 三 二 二 二 六 六 六 九 九 九 一 一 一 八 八 八 七 七 七 三 三 三 三 三 三 三 三 三 一 一 一 五 五 五 一 一 一 一 一 一 六 六 六 九 九 九 四 四 四 七 七 七 一 一 一 六 六 六 七 七 七 四 四 四 一 一 一 三 三 三 八 八 八 二 二 二 三 三 三 六 六 六 一 一 一 四 四 四 六 六 六 九 九 九 一 一 一 七 七 七 七 七 七 二 二 二 七 七 七 六 六 六 九 九 九 五 五 五 三 三 三 四 四 四 五 五 五 二 二 二八 八 八四 四 四 五 五 五 二 二 二六 六 六四 四 四 五 五 五 二 二 二四 四 四四 四 四 五 五 五 二 二 二二 二 二四 四 四 五 五 五 二 二 二 四 四 四 五 五 五 一 一 一八 八 八四 四 四 五 五 五 一 一 一六 六 六四 四 四 二 二 二 六 六 六 二 二 二 一 一 一 九 九 九 二 二 二 九 九 九 六 六 六 七 七 七 五 五 五 二 二 二 二 二 二 四 四 四 四 四 四 二 二 二 六 六 六 二 二 二 三 三 三 七 七 七 二 二 二 一 一 一 二 二 二 二 二 二 九 九 九 二 二 二 一 一 一 七 七 七 二 二 二 三 三 三 八 八 八 一 一 一 六 六 六 三 三 三 二 二 二 四 四 四 六 六 六 二 二 二 四 四 四 一 一 一 二 二 二 五 五 五 四 四 四 一 一 一 四 四 四 五 五 五 三 三 三 五 五 五 五 五 五 二 二 二八 八 八五 五 五 五 五 五 二 二 二六 六 六五 五 五 五 五 五 二 二 二四 四 四五 五 五 五 五 五 二 二 二二 二 二五 五 五 五 五 五 二 二 二 五 五 五 五 五 五 一 一 一八 八 八五 五 五 五 五 五 一 一 一六 六 六五 五 五 一 一 一 六 六 六 一 一 一 五 五 五 四 四 四 四 四 四 五 五 五 一 一 一 一 一 一 三 三 三 一 一 一 一 一 一 一 一 一 七 七 七 四 四 四 五 五 五 七 七 七 七 七 七 三 三 三 六 六 六 五 五 五 二 二 二 四 四 四 五 五 五 二 二 二 五 五 五 三 三 三 三 三 三 四 四 四 七 七 七 一 一 一 五 五 五 一 一 一 三 三 三 七 七 七 五 五 五 三 三 三 九 九 九 五 五 五 一 一 一 二 二 二 七 七 七 五 五 五 三 三 三 六 六 六 五 五 五 二 二 二八 八 八六 六 六 五 五 五 二 二 二六 六 六六 六 六 五 五 五 二 二 二四 四 四六 六 六 五 五 五 二 二 二二 二 二六 六 六 五 五 五 二 二 二 六 六 六 五 五 五 一 一 一八 八 八六 六 六 五 五 五 一 一 一六 六 六六 六 六 二 二 二 五 五 五 一 一 一 五 五 五 三 三 三 二 二 二 二 二 二 八 八 八 九 九 九 一 一 一 七 七 七 四 四 四 六 六 六 一 一 一 三 三 三 八 八 八 二 二 二 二 二 二 五 五 五 九 九 九 一 一 一 九 九 九 四 四 四 七 七 七 六 六 六 一 一 一 六 六 六 六 六 六 一 一 一 九 九 九 三 三 三 一 一 一 二 二 二 六 六 六 一 一 一 九 九 九 一 一 一 四 四 四 六 六 六 九 九 九 二 二 二 九 九 九 五 五 五 四 四 四 五 五 五 三 三 三 七 七 七 五 五 五 二 二 二八 八 八七 七 七 五 五 五 二 二 二六 六 六七 七 七 五 五 五 二 二 二四 四 四七 七 七 五 五 五 二 二 二二 二 二七 七 七 五 五 五 二 二 二 七 七 七 五 五 五 一 一 一八 八 八七 七 七 五 五 五 一 一 一六 六 六七 七 七 四 四 四 九 九 九 一 一 一 二 二 二 三 三 三 三 三 三 一 一 一 一 一 一 六 六 六 七 七 七 一 一 一 八 八 八 六 六 六 一 一 一 二 二 二 七 七 七 一 一 一 六 六 六 七 七 七 二 二 二 九 九 九 三 三 三 二 二 二 二 二 二 七 七 七 二 二 二 二 二 二 一 一 一 五 五 五 三 三 三 八 八 八 五 五 五 二 二 二八 八 八八 八 八 五 五 五 二 二 二六 六 六八 八 八 五 五 五 二 二 二四 四 四八 八 八 五 五 五 二 二 二二 二 二八 八 八 五 五 五 二 二 二 八 八 八 五 五 五 一 一 一八 八 八八 八 八 五 五 五 一 一 一六 六 六八 八 八 一 一 一 二 二 二 五 五 五 七 七 七 七 七 七 四 四 四 四 四 四 四 四 四 九 九 九 一 一 一 一 一 一 二 二 二 二 二 二 三 三 三 六 六 六 二 二 二 一 一 一 三 三 三 五 五 五 七 七 七 一 一 一 八 八 八 八 八 八 四 四 四 三 三 三 三 三 三 一 一 一 一 一 一 六 六 六 五 五 五 二 二 二 三 三 三 九 九 九 六 六 六 一 一 一 二 二 二 五 五 五 四 四 四 七 七 七 四 四 四 六 六 六 二 二 二 七 七 七 一 一 一 八 八 八 五 五 五 三 三 三 九 九 九 五 五 五 二 二 二八 八 八九 九 九 五 五 五 二 二 二六 六 六九 九 九 五 五 五 二 二 二四 四 四九 九 九 五 五 五 二 二 二二 二 二九 九 九 五 五 五 二 二 二 九 九 九 五 五 五 一 一 一八 八 八九 九 九 五 五 五 一 一 一六 六 六九 九 九 七 七 七 一 一 一 八 八 八 七 七 七 七 七 七 一 一 一 六 六 六 七 七 七 六 六 六 三 三 三 三 三 三 二 二 二 九 九 九 五 五 五 七 七 七 一 一 一 七 七 七 四 四 四 六 六 六 一 一 一 一 一 一 九 九 九 五 五 五 五 五 五 八 八 八 一 一 一 九 九 九 八 八 八 五 五 五 一 一 一 五 五 五 九 九 九 一 一 一 六 六 六 一 一 一 九 九 九 三 三 三 一 一 一 二 二 二 三 三 三 八 八 八 九 九 九 二 二 二 一 一 一 五 五 五 三 三 三一 一 一 五 五 五 二 二 二九 九 九 五 五 五 二 二 二七 七 七 五 五 五 二 二 二五 五 五 五 五 五 二 二 二三 三 三 五 五 五 二 二 二一 一 一 五 五 五 一 一 一九 九 九 五 五 五 一 一 一七 七 七 二 二 二 三 三 三 四 四 四 二 二 二 三 三 三 五 五 五 四 四 四 五 五 五 七 七 七 二 二 二 五 五 五 五 五 五 二 二 二 三 三 三 五 五 五 二 二 二 一 一 一 二 二 二 三 三 三 九 九 九 二 二 二 九 九 九 Denis Roegel Sustainable digitization
The quality of the digitizations → Are these digitizations sufficient?

Are these digitizations sufficient?

if the purpose is to have an overview of a document, these digitizations are sufficient;

for the last example, we can recognize that it is in Chinese, and most people will be content with it;

but in order to really read what is written, it is not enough; moreover, it is easy to see that the problem is not a display problem, but a problem at the source;

with some digitizations, reading becomes archeology! should reading a digital document be painful?

Denis Roegel Sustainable digitization

The quality of the digitizations → Quality level and digitizer

Quality level and digitizer

The average quality is clearly correlated with the institution which is responsable of digitizing.

Google books and Internet Archive:

average quality, and sometimes very low quality never very good

German libraries

average quality very good sometimes less good on specific aspects some institutions produce digitizations and interfaces with a very low or suspicious quality:

Digital Library of India Universal Digital Library (Carnegie Mellon, digitizations subcontracted in India and China)

for many sites, the quality increases with time

Denis Roegel Sustainable digitization

The case of Google books (1) The case of Google books (2)

Books digitized by Google have numerous flaws: some pages may be blurred, fingers can be visible, etc.

folded plates of older books are never unfolded; some of these defects are not accidental, but are the result of instructions;

those who digitize for Google have instructions not to unfold the plates; if that were not the case, we would sometimes see unfolded plates;

Google doesn't mind to sell books which have been incompletely digitized.

The case of Google books (7)

There is a rather general ignorance of Google's technical problems, even in libraries:

in their books on Google, neither Jean-Noël Jeanneney (former head of the BNF, French National Library, critical towards Google), nor Bruno Racine (current head of the BNF, rather open to a dialog with Google), do mention Google's quality problems, and probably do not know them; in general, independently from digitization errors, the matter of insufficient resolution is never raised.

in his book on Google books (Google Livres et le futur des bibliothèques numériques, 2010), Alain Jacquesson (former head of the library of Geneva) devotes two pages to the control of quality; for the author, Google's errors are all accidental and he believes Google's promise to correct them; he seems to believe that books with foldouts do not enter the digitization chain; he also seems to ignore that badly digitized books are still put on sale;

Jacquesson mentions the "manipulation errors when digitizing" and adds that "Google gives precise instructions to its digitization operators."

Denis Roegel Sustainable digitization

The quality of the digitizations → Google books problems

The case of Google books (9)

Google has introduced a number of technical innovations, but they are no wonders:

Google has filed a patent for correcting curvature effects resulting from thick non-plane books;

on certain views, we can observe obvious image corrections; these corrections produce such deformations that the result is almost worse than without corrections;

the least that can be said is that Google's patent is not satisfactorily, or not well applied! It appears that libraries and other book professionals are very little aware of Google's technical problems.

The case of Google books (10)

Concerning foldouts:

according to Alain Jacquesson, the books which contain foldouts are excluded from digitization the fact that such books are digitized is then either an error of Google which takes them nevertheless, or an error of the providing library;

for consistency, Google should destroy a digitization whenever it contains foldouts that would lead to an incomplete digitization;

Google therefore prefers to digitize badly (and on purpose), than not at all.

The problem of pseudo-publishers (book-recyclers) (1)

probable contracts between Google and "fake" publishers such as Bibliobazaar = BiblioLife = Nabu Press, Kessinger, Pranava, Woods Press, etc.;

the digitizations are those of Google, but without the Google logo; these uses are therefore authorized;

the printing is on demand, but the catalogues are filled of these non-existing books;

Google is possibly behind the above names;

the recycled versions are shadowing the original versions;

the recycled versions are put in front of searches, in contradiction with Google's will to fight "content farms" and low-value pages, but in accordance with the probable collusion;

Denis Roegel Sustainable digitization the above "fake" publishers do not seem to overlap, and they seem to divide the market; a given book is not published by more than one of these publishers; durability problem: when an old version is reprinted by one of these pseudo-publishers, the original version may vanish from Google books (for an example, see the Wikipedia page of Kessinger Publishing);

a SUDOC record for a book recycled by Kessinger little by little, the real books will be replaced by bad quality recycled copies (invasion of the body-snatchers?)

The case of Cambridge University Press

In 2010, Cambridge University Press reprinted Henderson's book, Life of James Ferguson, 1867.

There are two problems here:

it is not clear why CUP chose to reprint the 1867 edition, since a corrected and extended edition was published in 1870;

a large number of drawings come out blurred, as if their treatment had used an insufficient resolution; oddly, this is not the case of all figures.

Denis Roegel Sustainable digitization

The case of Cambridge University Press (2)

We can compare the figures from 1870 and from the 2010 reprint on Google Books, and also compare the 1870 figures online and in PDF, as these are not exactly the same.

Overdigitization problems (3)

Close-up of the previous image:

to the left of the characters: some green; to the right: some yellow; chromatic aberration phenomenon. the green color is always on the same side of the characters, and this excludes a JPEG compression artefact.

Overdigitization problems (4)

as demonstrated by the previous example, certain libraries digitize with a resolution which is too large (that is, beyond their possibilities)

the digitizations then introduce defects, for instance due to the lenses (chromatic aberrations, etc.)

the material imposes a limit the digitizer must therefore have some physical notions, and this is seldom the case overdigitization = false quality

Denis Roegel Sustainable digitization

Evaluating the quality of digitization

How can the quality of digitization be evaluated? Idea: ask the users.

there are different types of users, which mainly split in two groups:

those who browse, and those who really use a book;

however, the digital libraries are seen as extensions of the web and the way of consulting them is the same: it is the reign of superficiality;

the users who browse therefore represent the majority and are in general satisfied; does that mean that we should rely on them for choices of quality?

Denis Roegel Sustainable digitization
the satisfaction of a user should be modulated by his/her needs;

the needs can be known or not; some needs only appear during usage, in particular because these needs are seen as granted;

for instance, a need for speed only becomes explicit when the service provided is slow; it is not because a user does not express a certain need that he/she does not have it;

knowing the needs of the users and their evolutions is therefore not always simple.

Denis Roegel Sustainable digitization

The quality of digitization can also be evaluated using the reading comfort. The (visual) reading comfort is mostly independent of the support: parchment, papyrus, clay tablets, paper, etc. We always look with the same eyes! In the case of digitization:

the comfort is more or less close to the comfort of the original;

if some characters have to be guessed, it is not sufficient; each character should be identified without ambiguity;

the document must allow for extraction of all informations;

the images and drawings must show all the details and be sufficient for a republication

Denis Roegel Sustainable digitization

Storage quality and visible quality (1)

it may be necessary to distinguish the storage quality from the quality shown to the user;

it is for instance possible to digitize at 600dpi, and to put online only 300dpi (or lower) images; this is possibly what Google does, but I doubt that the possibly higher-resolution images are exempt of the errors visible online; later on, when the accesses become faster, it might be possible to switch to a greater quality;

Denis Roegel Sustainable digitization

Storage quality and visible quality (2)

there should be a certain transparency; the user should be informed of the storage quality, even if he/she only occasionally has access to it;

Denis Roegel Sustainable digitization

The quality of the digitizations → Storage quality and visible quality

Storage quality and visible quality (3)

When the quality of an online image is too low, we should not say "it is normal, the online version is downgraded." Because then, one could ask "why was it put online?" So that the users pay for a better access?

The real question is: why have those who put the image online not understood 1) that a better online version was needed, and 2) that it is possible?

The answer is that those who digitize and put online do not understand the needs of the users. Moreover, putting online a better version is not necessarily more constraining for the user, when the user is only online. The quality should only have a cost when downloading.

Denis Roegel Sustainable digitization

The quality of the digitizations → Storage quality and visible quality

Storage quality and visible quality (4)

The various levels of quality naturally entail a structuration of the documents:

it is possible to make an entire document accessible with a good quality, but with a high cost (volume, download time);

(in addition) it is also possible to segment the document in order to allow for partial (predefined or on demand) downloads;

finally, it is possible to prepare variable qualities, depending on the parts of the document; plates might for instance be digitized with a better quality than text.

In any case, libraries can not be content with putting online downgraded versions without having appraised the needs of the users.

Denis Roegel Sustainable digitization

Storage quality and visible quality ( 5)

We can observe that some sites are increasing their image quality with time, sometimes using the same internal digitizations. An interesting example is JSTOR.

Here are excerpts of Arthur Baragar's article published in 2002 in the journal American Mathematical Monthly. The PDF was generated in 2007, at the time the file was downloaded (http://www.vcharkarn.com/uploads/2/2557.pdf).

Required resolution

How can the required resolution be determined?

the digitizer must understand the relationship between several entities, and above all between the reader and the book;

the characters have a certain smoothness;

the user reads with his eyes;

a text printed at 300dpi displays irregularities which are visible with the naked eye at 10 or 20cm;

in the same conditions, a text printed at 600dpi appears smooth;

this is compatible with the angular resolution of the eye, which is about 1 ; at 50 cm, this corresponds to about 300dpi;

this justifies a sustainable resolution of 600dpi for text;

Denis Roegel Sustainable digitization

Required resolution and information content

The required resolution mostly depends on the information content of the document: a text is aimed to be read at 50 cm, 600dpi should therefore suffice;

if the text must be increased and if other details are important, a greater resolution may be necessary;

Denis Roegel Sustainable digitization

The quality of the digitizations → Required resolution

Required resolution and information content a book photography can usually be digitized at 600dpi, even if it is a halftone; however, in that case, a smaller resolution will not be a problem, because the user will not see the impact of the low digitization quality on an original which is already of low quality; the question at stake is the conservation of the quality of the original images, even if it is bad; certain photographies contain a lot more details than 600dpi can capture; certain old contact prints may require 1000dpi or more, and slides may require 3000dpi;

the required resolution also depends on the use; a photography is not necessarily aimed at being seen in real size at a distance of 50 cm from the eyes;

but for prints, 600dpi should almost always be sufficient.

Denis Roegel Sustainable digitization

Is there a need limit? Many managers seem to view the evolution as an eternal replacement of technologies, with increasing lifespans and capacities.

Denis Roegel Sustainable digitization

Is there a need limit?

What the previous list suggests is that the needs increase.

Libraries and digitization managers therefore became accustomed of thinking that non-sustainability was the norm.

For some digitization managers, the concept of sustainability does consequently not have a very precise meaning.

But, as a doctor recently reminded me: the capacities of computers increase, but it is not the case, or very little so, with our brains. And our eyes also remain the same.

The eyes and the writing are fixing the limiting needs.

Denis Roegel Sustainable digitization

The quality of the digitizations → Limit of needs

There is a limit of needs! Limit = equivalence (general case) if the purpose is to read a text: When the digitization of a text reaches a resolution such that this text, when examined in normal reading conditions, does no longer appear rugged, then it is no longer necessary to go beyond for the reading comfort. This limiting resolution only depends on the angular resolution of the eye.

if the purpose is to extract another useful information: It is necessary to define what is this useful information and to deduce the required resolution (or any other parameter).

These are the considerations which make it possible to define the minimum criteria of a sustainable digitization.

Denis Roegel Sustainable digitization

The quality of the digitizations → Reading comfort Do we need reading comfort?

We can do a parallel with screen technologies:

we are evolving towards HD television, larger computer screens, thinner laptop screens, blu-ray movies, etc.

these technologies will more and more marginalize low quality digitizations;

today's and tomorrow's readers will more and more want a better image quality.

There is therefore no doubt that if comfort is not taken into account, sustainability will not follow.

Back to the storage and displayed qualities (1)

we do now know that there is a limit resolution for common usages, that is for books read with the eyes (and not a lens); it is therefore not necessary to go beyond this resolution (for this usage);

this does therefore define the sustainable storage quality; below this quality, the digitization will one day have to be redone;

the quality provided to the user depends on the comfort that we can (or are willing) to provide; some contents render it necessary that the provided quality is closer to the storage quality than it is with other contents; this is in particular the case with the examples seen above.

Denis Roegel Sustainable digitization

Color or black & white?

When a document is only printed in black and white, we can ask if a B&W digitization would not be sufficient.

this is true, but the paper being seldom white, the appearance of the digitized page does not correspond to that of the actual document; there is therefore an impact on the feeling of identity;

if only certain pages are digitized in color, this creates a consistency problem; these digitizations will not be viable in the long term;

Google puts the color digitizations online since 2009 (but downloading remains B&W).

Most of the large German libraries are also digitizing in color, and downloading is also in color.

Denis Roegel Sustainable digitization

The quality of the digitizations → Quantity or quality?

Quantity or quality?

As often is the case, quality is opposed to quantity:

those who digitize little tend to digitize better;

those who digitize a lot tend to digitize less well.

This raises the question: is it really more interesting for the user to have a lot of books digitized with an average quality, or less books digitized with a better quality?

I think that one tries to make us believe that we have a need for quantity.

Quantity is an economical need (2)

to go fast (and bad) is an economic necessity it is not a user's need but if an actor goes fast (and bad) and another slowly (and well), the user does not perceive the qualitative advantage;

Google therefore forces the other projects to go fast and to have a commercial attitude, although this is not justified from a scientific point of view.

Quality and sustainability

Let us focus on quality and sustainability.

And let us get down to the digitization criteria.

Denis Roegel Sustainable digitization

we will now give a little overview of the interfaces;

and take note of positive and negative features.

Denis Roegel Sustainable digitization

Gallica: start page too many things unnecessary informations for 99% of people the average user comes for a specific request 90% of this page is a distraction the distraction might be accessible from a link "Gallica's buzz" and that would be enough Hathi Trust (2)

Unless one is a member of a privileged institution, some documents (here from 1761) have such access restrictions that we were not even able to navigate the thumbnails without regularly reaching a screen whose purpose was to slow down the search. be ergonomic and as much as possible be invisible;

A good interface should be natural and discrete, like a good font. It is not sufficient for an interface to provide all functionalities.

Denis Roegel Sustainable digitization

The matter of downloading full download is useful, because not every person can (or wants to) read a book online, or on another screen the downloadable quality may differ from the real quality the downloadable quality should be sufficient for a comfortable reading, and this is not always the case (what is the purpose of an insufficient quality?) it is possible that some libraries intentionally chose an interface making downloading difficult, but the lack of transparency of many libraries makes it impossible to be sure.

Denis Roegel Sustainable digitization

The quality of interfaces → The matter of downloading

The matter of downloading (2) The job of librarians has changed (1)

until recently, libraries created little: the books were not made by the libraries;

a library was an intermediate between the publisher and the reader;

creation was limited to catalogues (cards, then computerized);

the book borrowed by a reader was the same as the one borrowed elsewhere.

Denis Roegel Sustainable digitization

The job of librarians has changed (2)

today, libraries create more: they digitize; they do it essentially for three reasons:

they own the books; the digitization material is relatively simple and rather unexpensive (camera and computer) it is easy to make an acceptable digitization if the digitizations were more difficult, they would be much more centralized

Denis Roegel Sustainable digitization

The wrong priorities of the libraries some libraries swear too much by metadata the first priority should be the faithfulness of the content, the title, the author, the year OCR can be applied, but in that case it is advisable to correct it (and OCR can be externalized)

OCR should not be a precondition for putting a document online (and this, some libraries do) one should not digitize for the sole purpose of showing that one has a digitization activity by focussing on metadata, the libraries miss the problems of quality of digitization

Denis Roegel Sustainable digitization

The heavy legacy of bad digitization the bad digitizations block the production of better ones in Strasbourg, for instance, a digitization which has been made elsewhere reduces the priority of a new digitization of the same book this penalty is all the more important that the digitization managers do not understand the quality requirements

Specifications

The specification prepared by an organization which plans to digitize is beyond the scope of this presentation, but it has to give: the conditions by which the collection's integrity will be ensured (for instance the order and layout of the documents) the technical conditions of the digitization the perennial storage conditions the restrictions to user access It is also advisable that the specifications are not confidential and are freely available to the users, so that they can comment upon them.

Denis Roegel Sustainable digitization

Are there things that should not be digitized?

This question is often raised, but:

it is an ill-posed question;

everything that has been kept in form of paper, parchment, papyrus, etc., manuscript or printed, is entitled to be digitized; the amount of documents which do not natively exist in digital form will certainly decrease in libraries and archives, and this will eventually limit the amount to digitize; the problem of non-digitization must be brought back to that of non-conservation; the digitizer should not ask him/herself if a certain document should or should not be digitized; if it has been kept, it should be digitized.

Denis Roegel Sustainable digitization

The example of Strasbourg University the library of the University of Strasbourg has been digitizing books since approximately 2005 the scientific collection is very rich the books are chosen by persons of different domains, often retired professors nevertheless, the selections seem to be made without a real knowledge of the collection; in 2010, in order to make up for this problem, I have gathered a 250 pages overview of the collection of scientific books of the University; this overview was initially meant to help better select the books that had to be digitized, but it was also meant to avoid falling in an abusive subcontracting however, this work highlighted the limits of the thematic approach followed at Strasbourg;

Denis Roegel Sustainable digitization

The problems of digitization policies

The limits of a thematic approach By digitizing only thematically and with limited means (100 to 200 books a year):

a rich collection can only be touched upon;

the sampling is subjective and the choices do not necessarily correspond to needs; the thematic approach (quotas for each field) is satisfying for those who are distantly interested (one pleases every one), but the truth is that there is a standstill everywhere; by trying to favor the themes, the nature and the needs of the documents are neglected; a thematic approach on books which do exist elsewhere is often a waste of public money.

On-demand digitizations (1) The example of mathematics there is a community interested in the digitization of mathematics it is estimated that the total number of pages of the mathematical corpus is about 100 millions "Digital Mathematics Library" (DML) project the last conference on this theme took place in July 2011.

Denis Roegel Sustainable digitization

The problems of digitization policies → The example of mathematics Digital Mathematics Library (DML) The chaos and its solutions: digitize better ideally, we should everywhere stop to digitize and start thinking notion of "layer" in order to integrate the multiple versions of a digitization (integration of quality) take the quality into account: doing bad is acceptable, provided it is taken into account in the production chain definition of a quality scale stop doing local actions think about the sustainability of the digital object, and favor the quality and the reusability to the quantity understand the needs of the users be interested in the book

Denis Roegel Sustainable digitization

We propose to give a letter from A to E for the quality of digitization. It is possible to distinguish the stored quality and the provided quality, but in general the user can only evaluate the provided quality.

This scale only considers the images, and not the quality of the text or the metadata. This is a different problem.

It is illusory to believe that everything will be OCR'ed anytime soon, especially mathematical texts and there is therefore a real need to evaluate the quality of the images.

Denis Roegel Sustainable digitization

A word on image rotations many digitized images are not aligned;

it is possible to rotate the images;

this rotation almost always produces a loss of quality (see our article on the topic) it is therefore essential to minimize this step, either by a better alignment, or by a greater resolution.

it is not forbidden to have graduated from a school for archivists, or for librarians, or to have a PhD in history, or even to know dead languages, but this plays only a minor role in the work of a digitization manager on the other hand, if the manager lacks certain competences, he/she must be able to surround him/herself with people having these competences, and not venture alone in such actions some of the current insufficiencies find their source in the inadequacies between the profiles of certain managers and the required competencies (problems of legitimity)

Denis Roegel Sustainable digitization

  are not taken into account, or the users are not very demanding A good interface should: provide a number of basic functionalities;

  The original books remain the natural property of the libraries. The digital versions should be distributed as widely as possible.What are the causes of all these problems?

	The quality of interfaces → Minimum criteria The sources of the technical problems
	Minimum criteria for a good interface (1)
	Downloading induces a fear:
	libraries are attached to their books; they often would like not to give the digital versions entirely; they would/will like to introduce a traceability: who downloads zooming in should not restrict the navigation area, nor require simple pages, without useless information special operations (as in Contentdm) title, author, year on every page moving (without restrictions) within a zoom window should be common navigation functionalities done by pulling the image
	what? navigation through groups of thumbnails the number of clicks should be minimized
	These fears correspond to an outdated perception of how libraries work. The library's activities will little by little have to be redefined: conservation of originals creation of digital versions enrichment of digital versions navigation on full pages, by steps of 5, 10, 50, 100 (very buttons should be near the images useful, but seldom available) thumbnails should be neither too big, nor too small functionalities should be grouped (no scattering) it must be easy to navigate with a constant zoom factor less important functionalities → separate page buttons should not be too close to each other avoid white zones (waste) image (page) download and PDF download internationalized interface, with at least the English option resolution downgrading is acceptable in the PDF, but it should quick access be possible to bypass the problem
	Denis Roegel Denis Roegel Denis Roegel Denis Roegel	Sustainable digitization Sustainable digitization Sustainable digitization Sustainable digitization

  , the Strasbourg National Library digitized for me (and for it!) several books which were charged to me (and which are now available online at other places. . . ); this was then natural, because the library does not have a digitization policy as developed as the German libraries do

	in 2009 or 2010, I suggested to a German library to digitize an
	old book which had not yet been digitized
	the library answered that it was possible, and told me at which
	cost question: who decides about digitizations if the researchers are not listened to? the good solution would have been either to submit this suggestion to those who are responsible for the selections, in order to see if the suggestion is consistent with the digitization policy and to integrate it; if not, then the user may be charged or to plan a slot for on-demand digitizations, without charge eventually, I gave up, and decided to wait that another library digitizes the book Denis Roegel Sustainable digitization in 2009 or 2010Denis Roegel Sustainable digitization

  Themes of the DML 2011 conference: o search, indexing and retrieval of mathematical documents o ranking of mathematical papers, similarity of mathematical documents o math OCR with MathML/TeX output o document conversions from/to MathML, OpenMath, ..., PDF o conversions between various mathematical formalisms o mathematical document compression, processing of scanned images o algorithms for crosslinking of bibliographical items, intext citations search o mathematical document classification, MSC 2010 o mathematical text mining o mathematical documents metadata exchange o long term archiving, data migration o reports and experience from math digitization projects o math publishing with long term archival goal o software engineering aspects of creating, handling MathML, OMDoc, OpenMath documents, and displaying them in web browsers

	Denis Roegel	Sustainable digitization

Denis RoegelSustainable digitization

Evaluating the quality of digitization

Asking the user is therefore dangerous: it is tempting to evaluate its needs as being those of the majority; the users which are critical are de facto a minority, as there is a majority of users who browse (how many are we to contact webmasters when the layout or the content of a page is not good?); the critical users are then marginalised: they are being told that their opinion is not representative.

Storage quality and visible quality (7)

The examination of the PDF file information reveals that they were produced differently, perhaps from the same source: in 2007 (produced when downloaded) in 2011 (produced three years before) Moreover, we observe that the two PDFs have similar sizes.

Denis Roegel

Sustainable digitization

Back to the storage and displayed qualities (2)

The user often does not know the storage quality, but if what the libraries provide is a useless quality (as in the examples seen above), one can ask if the libraries understand what they do and if they understand the needs of the users;

if the internal storage (in case it is different from the displayed one) is really of sufficient quality.

In order to avoid these doubts, the libraries owe to the users a total transparency on the means and choices, both technical and political.

Denis Roegel Sustainable digitization

Information explosion

Within a few years, a large amount of documents have become accessible.

In 10 years, we have gone from practically 0 book to 10 millions or more.

But the needs are not yet there.

And we do not have the time to exploit these resources! Who needs 10 millions books?

And moreover, not sustainable?

We are been sold quantity (and bad quality), when we actually would be satisfied with a lot less.

Denis Roegel Sustainable digitization

Quantity is an economical need (1)

We have heard statements such as:

Google can digitize in 5 years what would have taken us 50 years

But is it necessary to go that fast?

We (almost) all have a cellphone: but do we really need it?

Denis Roegel Sustainable digitization

Minimum criteria for a sustainable digitization The quality of interfaces

The quality of interfaces

Denis Roegel Sustainable digitization

Analysis of several interfaces

The problem of quality short critical overview Gallica, Strasbourg, Berlin, Goettingen, Dresden, HAB, Munich, ETH, Swiss manuscripts, etc. specialized interfaces, for instance for a particular manuscript (Madame Bovary by Flaubert) there are a certain number of works on evaluation criteria, but they will not be mentioned here Contentdm: the zoom mode (3) Some of the advertised functionalities do not work at all, or are dependent on certain platforms.

For instance, the button at the upper right of the interface is for "cutting the image in a new window."

But in that window, it is supposedly possible to select a part, then to select an operation with the right button of the mouse. However, this does not work everywhere!

In general, the zoom can not be positioned with sufficient accuracy, even with the thumbnail visualizing the zooming frame (only certain positions are allowed). The job of librarians has changed (3)

Digitization naturally ended up in the hands of libraries:

the libraries are responsible for the choices the user is almost absent from this chain:

mainly because the digitizations do not correspond to a pre-existing need; the users are not demanding.

the libraries had and still have full powers.

but . . . this leads to problems.

Part of these problems comes from a breakdown of roles.

Denis Roegel Sustainable digitization

The breakdown of roles in digitization Those who digitize do not listen to the users

Traditionally, the libraries have only consulted the users little, or not at all, for their choices.

This habit has been kept.

But it is no longer adapted to the digital world.

Moreover, listening to the users is biased:

for some libraries, the absence of feedback legitimates the choices for some libraries, a little bit of feedback also legitimates the choices

Denis Roegel Sustainable digitization

The sources of the technical problems → The breakdown of roles in digitization

Those who digitize are no technicians and they are too low in the chain to transmit possible observations when errors are noticed, it is too late, the book has been shelved, and it is too costly to go back to it.

Denis Roegel Sustainable digitization

The sources of the technical problems → Choice problems

Interactions between the libraries and Google

The libraries have been abused by Google:

is it possible that so many prestigious libraries have accepted that Google makes use of such a low level of quality?

the library heads are actually unaware of these problems! a serious patrimony curator should actually refuse that digitizations are that much botched Google knows about the errors (and could correct them), but still manages to get accepted Google even sells books which are incomplete for many libraries, an average quality is compensated by a large quantity at an almost zero cost the Lyon municipal library started to be digitized in 2009, and we can anticipate the same catastrophe

Denis Roegel

Sustainable digitization

Hindsight problems

The librarians:

for them, the users are satisfied (in fact, the users do not use) The wrong priorities of the libraries (2)

The feeling that one gets is that libraries view digitization as subcontracting consider that their work is about the "coating" (metadata, OCR, interface, etc.)

This way of viewing things creates a distanciation between the libraries and the books.

Bad digitization and duplicates An heterogeneous repository (for instance the archives of a writer) may contain documents which have already been digitized elsewhere, but that one will still wish to include in the digitized repository, for reasons of coherency.

Duplicates should not be fortuitous.

Denis Roegel Sustainable digitization

Aims of digitization

At the beginning, we have assumed that the aim was the quality. This is still our aim.

We have examined the following points:

what should the quality of the digitizations be?

how should the digitizations be displayed?

what are the causes of the encountered problems?

We now have to examine the choice of contents:

what should be digitized?

The problems of digitization policies

How should a repository be selected for digitization?

Usually, digitizations are done repository by repository. Various criteria can be considered:

unique collection, for instance manuscripts;

very demanded collection (essentially manuscripts); etc.

In any case, it is necessary to have a knowledge of the collection before choosing;

to have a long-term policy, and not a local one;

to implicate the users.

Denis Roegel Sustainable digitization

Strategies for a small digitization center

Digitization as affirmative action

It is a valorization strategy:

the purpose is here to highlight a collection, by favoring it for digitization;

this is therefore what can be called an affirmative action.

Such a digitization encourages research in a given collection.

Denis Roegel Sustainable digitization

The problems of digitization policies

Sustainability in digitization

digitization must produce a permanent object the content may have a much longer lifespan than the interface examples of non sustainable digitizations:

all the digitizations which have not been made from the original documents some of Gallica's digitizations originating from microfilms digitizations of vital records in Archives (in some places, made from microfilms)

when the object is a book, we have the option to wait that another library performs a better digitization but for manuscripts, it is up to the library which owns them sustainability will be reached when the digitization quality makes it possible to reproduce the quality of the original document insufficient digitizations waste money

Denis Roegel Sustainable digitization

Sustainability in digitization

With a sustainable digitization:

the digitization is comfortable the use of the original document is no longer necessary

Denis Roegel Sustainable digitization

The example of mathematics

The previous list shows that those who are interested in the digitization of mathematics are mainly the mathematicians the aim is to make accessible useful and not yet digitized mathematics paradoxically, the conference themes refer only very little to old documents in fact, this conference does not reflect very well the interests of the community of historians of mathematics consequently, we should fear that the digitization of ancient mathematics will not be that well defended, because these books are practically useless nowadays such problems occur in other domains, in particular in astronomy

Denis Roegel Sustainable digitization

The problems of digitization policies → Questions for a digitization

Questions for a digitization

Several questions are raised in case of digitization:

who selects the collection to digitize? why this collection? is there a demand? is this digitization sustainable? is it a window-digitization? who will digitize? do these persons understand the limits of digitization? will the digitization be adapted to the needs of the contents (formulae, images, figures, etc.)? who chooses the interface? how does this interface compare to the other interfaces? will there be downloading? if not, why not? will the text be given? if yes, will it be indexed on the web? will it be possible to put the digitized collection back into a larger collection in an homogeneous way?

Denis Roegel Sustainable digitization

Solutions

Denis Roegel Sustainable digitization

We have already seen specific solutions for two problems:

problems of the quality of digitization;

problems of the quality of interfaces.

There now mainly remains to solve the political problems.

The chaos of digitization

Selection mechanism

Some problems can get solved alone, merely using a natural selection mechanism.

the evolution can occur by means of national or international dialogue;

it can also occur by mere natural selection; the multiplicity of digitizations will produce a selection; but these two possibilities will not produce the same result; liberalism is not the solution! the cost is certainly not the same, for there are great losses with chaos;

digitizations being made for all, do we have the right to do anything that pleases us?

Denis Roegel Sustainable digitization

The user has changed Today's user is no longer the same as in the past:

in the past, the books of a library were mainly meant for the immediate readers of this library (city, university, etc.) today, the libraries produce for a national, and even international, audience

Today, the libraries have much higher duties and responsibilities than before. The frontiers of a digitization are no longer those of a university, and yet some digitization managers still have this mindset! Gallica, the German libraries, etc., are all accountable to all.

Denis Roegel Sustainable digitization

Different kinds of users the average user, who may be demanding, does not want to look for a configuration menu;

the average user is an occasional one; he/she comes for only one book;

it is possible to please this average user and many sites are rather satisfying; but some of them, in particular those based on Contentdm, are not a user may want to study a book at home, without an internet connection; he/she must be able to download or to print a good quality version.

Denis Roegel Sustainable digitization

Overcautious libraries?

As we said above, the libraries do now give a feeling of timeworn institutions:

will to control online documents (tatooing, steganography, etc.), in order to track the usages restrictions to downloading attachment to the lending model

Denis Roegel Sustainable digitization

The redefinition of the role of the libraries the libraries have a new role to play: what will the libraries do when everything will have been digitized? they hold the original documents they should not fear giving away the digital versions the new role of the libraries might be to enrich the documents (OCR, documentary files, etc.)

Denis Roegel Sustainable digitization

Document enrichment

can be split from digitization, both in time and in space drift towards the penetration into the documents it is the added value provided by the libraries (like the catalogs)

Among the enrichment forms which will develop:

the entire transcription of books, with typesetting (Gutenberg project and more, LOCOMAT project, etc.)

the structured representation of books (XML, TEI, MathML, etc.) the production of additional files for anchoring a book on the web; this is the role of a documentation center, but such a role will necessarily penetrate that of libraries.

Denis Roegel Sustainable digitization

Managing quality

The quality of digitization requires a precise management:

each library should maintain a list of digitizations which have to be redone the digitization quality can be appraised right away, and it can also be appraised outside of the library Gallica should for instance plan a new digitization of the book seen above (and many others)

certain libraries possibly do it, but it is not visible

Denis Roegel Sustainable digitization

A: the digitization might replace the original document (resolution equal or greater than 600dpi, in color, well aligned, plates well scanned, no defect); B: correct, but not comfortable enough, or does not respect the colors, or other small defects which do not prevent the reading; plates well scanned; copying the digitization does not require understanding it;

C: only one serious defect (insufficient resolution, unfolded plates, deformations, etc.); D: two serious defects; E: three serious defects; etc.

Denis Roegel Sustainable digitization

Normalization of online documents today, it is great time to think about normalizing the uploading of digitizations; precise rules should be followed by the major digitizers, both for the quality of digitizations as for the quality of the interfaces;

this is unfortunately still far from being the case, as the examples in this document prove;

in the future, the norms can of course evolve.

Denis Roegel Sustainable digitization

Solutions → Manual or automatic digitization?

Manual or automatic digitization?

There are digitization machines which turn the pages automatically, but:

old documents can not be handled without the risk to damage them;

Google does not use entirely automatic machines; a quality management can currently not be obtained automatically, because in order to digitize well, one must be able to read the content (for alignment, for adapting the resolution, etc.);

Part of the bad digitizations are due to the fact that those who digitize are almost machines.

A good digitizer should understand the needs of the book.

Denis Roegel Sustainable digitization

Solutions → The death of digitizations

The death of digitizations the digital world will be populated by a growing number of digitizations and there will be many multiple digitizations (the same original, or the same book several times) in different qualities this profusion of information will harm research, in the same way as "content farms" have made internet searches almost useless "content farms" drive the users to fall back on a few well-known places, such as Wikipedia the same might happen with digitizations, especially if illegal copies of whole libraries appear, which is likely if the quality is handled, then it is necessary to plan for the elimination of bad digitizations

Denis Roegel Sustainable digitization

Solutions → The problem with microfilms

The problem with microfilms

There is a fundamental difference between microfilms and the current digitizations:

microfilms have never had the purpose to replace the originals, but to protect them or to make them accessible microfilms are almost always in B&W and of bad quality current digitizations are in color and can reach resolutions which make it almost superfluous to access the original documents, except in some rare cases This difference of purpose explains that it was historically acceptable to produce bad microfilms.

Denis Roegel Sustainable digitization

The problem of cost there is currently often an obsession with the TIFF format it is a format which can keep all the data from the original image, but it results in large files in the context of a "preservation" approach, it often seems natural for libraries to use this format internally but, it is a wrong vision of the things! if we select the format by reasoning in a binary way (losses or no losses), the digitization chain is not taken into account the TIFF is a lossless representation of the result of chain which has perturbated the object (book or else)

Denis Roegel Sustainable digitization

Storage formats: with or without losses? (2)

the subcontractors for digitization will advise the use of TIFF, often because of ignorance of the other parameters some users also want TIFF, because their publishers ask for it by requesting TIFF, the publishers guard themselves from errors due to compression, but not from errors upstream in the digitization in almost all cases, libraries do not need to use this format; it doesn't do harm (except in storage), but it is not necessary the real challenge is to determine which losses are acceptable given the limits introduced by the production chain (in particular optical limits) and by the physiological limits (eye resolution)

the absence of losses is in fact a chimera

Several big problems

We can currently distinguish several problems: it is often difficult to know when a book has been digitized somewhere; some books have been digitized in several copies, although this wasn't always necessary;

digitizations are scattered on many sites and presented in a heterogeneous way

Denis Roegel Sustainable digitization

Census of digitizations

In order to have a better grasp on the existence of a digitization, it is essential to have a global census of everything that has been digitized. some censuses do exist, but they are not complete (example: http://digreg.mathguide.de in Goettingen) another example is http://www.zvdd.de/startseite these censuses are based on lists provided by the libraries involved;

there certainly is no notion of a unique identifier for a given copy of a book covering all libraries, as well as all uncatalogued collections outside of libraries it is however likely that the constitution of such a census will take place in a near future.

Denis Roegel Sustainable digitization

List of digitization tasks

The census of digitizations does not prevent the existence of multiple duplicates, because the census is not necessarily used, and that duplicates can be simultaneous.

In order to solve this problem, there should be a list of digitization tasks:

unique identifier for items to digitize (books or parts of books) assignment of tasks identification of books with "marginalia" whose annotations require a new digitization, even if the book has already been digitized elsewhere

Denis Roegel Sustainable digitization

Central digital server

One of the advantages of Google is that its interface is always the same, no matter where the books are located.

the libraries which are digitizing should deposit all their digitizations on a central server the presentation of all digitizations could then be unified Europeana is only a portal and only redirects towards the various libraries we can assume that this centralized server will one day see the light of day Google has forbidden certain of its libraries under contract to do such deposits using Google's digitizations, but it is certainly not forbidden with the new digitizations

Denis Roegel Sustainable digitization

The training of digitization managers And . . . if quality is not aimed?

It is however possible to live with average digitizations, provided that:

free and fast sustainable digitization is available on demand digitization errors can be quickly corrected.

At this point, no library, and certainly not Google, provides such a service.

Denis Roegel Sustainable digitization

Solutions → Should one come to terms with Google?

Should one come to terms with Google?

a contract with Google makes it possible to have quickly a large number of average quality digitizations;

it is possible to improve this situation by asking that Google follows more restrictive clauses:

immediate detection of errors (almost all errors can be identified immediately); either a book is digitized entirely (with unfolded plates), or not at all, but not partially; Google must commit not to put online (and not to sell) incomplete versions;

in any case, even with these caveats, the contracting libraries must understand that the digitization will have to be redone some time in the future (possibly a very distant one).

Denis Roegel Sustainable digitization

Perspectives

Today, a large majority of the digitizations produced in libraries will have to be redone. We are far from sustainability!

The quality of digitizations will increase when the competencies will be better distributed and when the needs will be well understood by all.

The quantity and order of digitizations will grow when the digitization will be better organized, nationally, and then at the European and world level, and not only organized by random deposits of uncontrolled digitizations.

Denis Roegel Sustainable digitization

An optimistic note for the end?

The fact that today's digitizations are not sustainable does not mean that they should not be done: they help to make aware of the problems and they help to reach a better definition of the needs (but do we have to first have non sustainable digitizations?); one might almost draw a parallel with the generations of machines, generations of mobile phones (1G, 2G, 3G, etc.), etc., each generation being used to access or prepare the next one; but is it really the same with digitizations?

Denis Roegel Sustainable digitization

Thanks to all for your input and have fun digitizing! Denis Roegel Sustainable digitization