Marçal Rusiñol

Joseph Chazalon

Jean-Marc Ogier

Josep Lladós

Marc ¸al Rusiñol

A comparative study

come

I. INTRODUCTION

Document image classification is one of the main topics of interest among the document image analysis community [START_REF] Chen | A survey of document image classification: problem statement, classifier architecture and performance evaluation[END_REF], mainly because it is a required step in many different contexts, from retrieval to document understanding. Depending on the particular application, document classes are defined either in terms of textual contents, document structure or visual similarity.

A document image classification system based on textual contents means that document images will be casted to the same class if they "talk" about the same topics. Obviously such systems require an initial OCR step in order to automatically extract the text from the document images. A subsequent step devoted to transform text strings to a feature vector is applied. The bag of words technique [START_REF] Sebastiani | Machine learning in automated text categorization[END_REF] is the most common approach, although more complex topic modeling approaches such as latent semantic analysis [START_REF] Deerwester | Indexing by latent semantic analysis[END_REF] or latent Dirichlet allocation [START_REF] Blei | Latent dirichlet allocation[END_REF] provide better results. From this numeric representation, any statistical classifier can be trained to finally decide to which class assign an incoming document.

A second family of document image classifiers would be the approaches that are based on document structure similarity. Such methods allow to assign a document class whenever the documents share the same physical or logical structure, regardless of their contents. A first layout analysis step is devoted to decompose the document images into blocks so the document similarity can be expressed in terms of the spatial relationships among those blocks. Such spatial relationships are represented by particular data structures such as attributed relational graphs [START_REF] Bagdanov | Fine-grained document genre classification using first order random graphs[END_REF] or X-Y trees [START_REF] Cesarini | Encoding of modified X-Y trees for document classification[END_REF]. Although such representations have an important discriminatory power, they present the drawback that computing a mapping between two layout structures is computationally expensive.

Finally, document image classifiers based on visual similarity will group document images into the same class if they "look" similar. Usually the proposed descriptors use statistics computed over low-level features in order to encode such visual appearance of the document. Such low-level features are directly the average pixel intensity as in the works proposed in [START_REF] Héroux | Classification method study for automatic form class identification[END_REF], [START_REF] Sidiropoulos | Content-based binary image retrieval using the adaptive hierarchical density histogram[END_REF], or a little more elaborate strategies such as multioriented run-lengths as proposed in [START_REF] Gordo | Large-scale document image retrieval and classification with runlength histograms and binary embeddings[END_REF]. Such representations have the advantage that they are normally cheap to extract and to embed in a statistical classifier. However they offer in general a limited discriminative power when it comes to a more fine-grained document image classification.

Although those three families try to solve different user definitions of what a document class or similarity means, they all present the same shortcoming. Such document image descriptions are in general global, due to the fact that historically the document images to process in our community came from a digitization device such as a scanner or a fax machine. Even lately, we started to directly process digitally-born documents directly in electronic format. However, with the quick growth of the camera quality in mobile phones and the ubiquity of such devices, in the last years, to deal with mobile captured document images has become an interesting research topic.

When acquiring a document image with a mobile gimmick we can not expect the same kind of document images to process. Such mobile digitization process entails illumination problems, geometric distortions due to the perspective effect, occlusions or cluttered background, etc. In such scenarios, it is doubtful that the classical global document image descriptors would be able to perform properly. However, some renowned object recognition techniques from the Computer Vision field, based on local descriptors, have been designed to be able to tackle such inconveniences, so researchers from the Document Analysis field started to apply such techniques to deal with document images.

One might still wonder whether those object recognition frameworks are really adaptable to processing document images, and more specifically, which local key-points detectors and local descriptors from the vast plethora that has been proposed in the last decade are the most promising and have the best performances when dealing with the particular case of document images.

In this paper we focus on the specific problem of document image classification with document images acquired with mobile devices. We propose to tackle such problem with an approach founded on a direct matching of local descriptors followed by a voting strategy. We analyze the performance of different state-of-the-art local key-point detectors and descriptors depending on a number of different factors such as the document type or the acquisition conditions. Such analysis is conducted in a large-scale scenario with more than 36.000 images in the test set.

The organization of the rest of the paper is as follows. In Section II we overview the document classification strategy which is based on matching local descriptors. In Section III, we overview the set of different state-of-the-art local keypoint detectors and descriptors that we use in this analysis. Section IV reports the experimental results, while finally in Section V we draw our conclusions.

II. DOCUMENT CLASSIFICATION BY MATCHING LOCAL DESCRIPTORS

In order to perform the document image classification by matching local descriptors we followed a similar approach than the one previously presented in [START_REF] Rusiñol | Logo spotting by a bag-of-words approach for document categorization[END_REF].

In the "training" phase, we need an example of each document class in order to extract and index their local descriptors. Given a collection of documents D and a sample document d i ∈ D of a class i, we extract their local key-points K i . Each key-point k j i ∈ K i consists of the coordinates of the key-point, a scale factor and an orientation. We then compute the local descriptors F i obtaining a feature vector f j i ∈ F i for all the key-points k j i . All local descriptors F i are then stored in an indexing structure with their associated document index i. In this paper we use the FLANN [START_REF] Muja | Fast approximate nearest neighbors with automatic algorithm configuration[END_REF] indexing framework in order to compute similarities between the stored descriptors and the local descriptors from an incoming image. When using binary descriptors the Locality Sensitive Hashing indexing structure is used within FLANN. For integer or floating point descriptors we use a KD-tree indexing structure.

When an incoming document image arrives, we compute its local key-points and associated local descriptors. Such local descriptors are then matched against the whole indexing structure. We accumulate votes to the matched document class i. When all the local descriptors from the incoming image have been matched against our indexed corpus, the document class receiving more matches is the one taken as the corresponding class.

Two additional points are considered in order to make the results more reliable. On the one hand, we use a ratio-test, as in the original SIFT paper [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], in order to just keep the local matches that are really discriminative. That is, a match is considered only if the distance to the nearest local descriptor and the distance to the second local descriptor is sufficiently different. On the other hand, a RANSAC step [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] allow to filter out matches that might be correct in terms of similarity of local descriptors but that do not agree with the rest of matches in terms of the projective transform that we should apply to go from a set of local key-points in the incoming image to the set of local key-points in the model document image.

III. LOCAL DETECTORS AND DESCRIPTORS

We will overview in this section the off-the-shelf local detectors and local descriptors that we used in our study. We have used the baseline OpenCV's implementation1 of the local detectors and descriptors.

A. Local Key-point and Region Detectors

Let us overview the four different key-point detectors that we have used in our comparison.

• SIFT: Key-point detector proposed by D. Lowe in [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] in which key-points are extracted as maxima of the Difference of Gaussians over a scale space analysis at different octaves of the image. Dominant orientations are assigned to localized key-points.

• SURF: Key-point detector proposed by H. Bay et al. in [START_REF] Bay | SURF: Speeded up robust features[END_REF] which detects blobs based on the determinant of the Hessian matrix.

• ORB: Key-point detector proposed by E. Rublee et al. in [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF] which uses an orientation-based implementation of the FAST corner detection algorithm.

• MSER: Key-region detector by J. Matas et al. in [START_REF] Matas | Robust wide baseline stereo from maximally stable extremal regions[END_REF] which is based on finding stable and extremal regions when applying a connected component analysis after iteratively thresholding the image.

We present in Figure 1 an example of the extracted local key-points within a portion of a document image. It is worth noting the differences in terms of which kind of information is retained by the different key-point extractors, as well as the amount of key-points issued by the different algorithms with their default parameters. Obviously, such difference in the amount of key-points will result in higher matching times for the algorithms yielding high amounts of key-points when analyzing document images.

B. Local Descriptors

Let us overview the four different local descriptors we have used in our comparison.

• SIFT: is a local descriptor proposed in [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] which coarsely describes edges appearing in key-point frames by an orientation histogram over the gradient image.

• SURF: is a local descriptor proposed in [START_REF] Bay | SURF: Speeded up robust features[END_REF] which is based on the computation of Haar wavelet responses in a dense fashion within the key-point frames.

• ORB: is a binary local descriptor proposed in [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF] which is a rotation aware version of the BRIEF descriptor [START_REF] Calonder | BRIEF: Computing a local binary descriptor very fast[END_REF]. It basically encodes the intensity differences among a set of pairs of pixels.

• BRISK: is a binary local descriptor proposed by S. Leutenegger et al. in [START_REF] Leutenegger | BRISK: Binary robust invariant scalable keypoints[END_REF] which is also based on a pair-wise comparison of pixel intensities. It is worth noting that both SIFT and SURF yield an integer-valued histogram while ORB and BRISK produce binary strings. Such binary descriptors are matched against each other with a Hamming distance which entails a much faster distance computation than the Euclidean distance calculation done for SIFT and SURF descriptors.

IV. EXPERIMENTAL RESULTS

A. Dataset

To build our dataset, we took six different document types coming from public databases and we chose five document images per document type. We have chosen the different types so that they cover different document layout schemes and contents (either completely textual or having a high graphical content). In particular, we have taken data-sheet documents and patent documents retrieved from the Ghega dataset [START_REF] Medvet | A probabilistic approach to printed document understanding[END_REF]. Title-pages from medical scientific papers from the MARG dataset [START_REF] Ford | Ground truth data for document image analysis[END_REF]. Colour magazine pages from the PRIMA layout analysis dataset [START_REF] Antonacopoulos | A realistic dataset for performance evaluation of document layout analysis[END_REF]. American tax forms from the NIST Tax Forms Dataset (SPDB2) [START_REF] Dimmick | Structured forms database[END_REF]. And finally typewritten letters from the Tobacco800 document image database [START_REF] Lewis | Building a test collection for complex document information processing[END_REF]. We show an example of each of those six different document types in Figure 2. We removed some small noise and margins from the original document images and finally rescaled them to all have the same size and fit an A4 paper format.

Each of these document models were printed using a color laser-jet and we proceeded to capture them using a Google Nexus 7 tablet. We recorded small video clips of around 10 seconds for each of the 30 documents in six different background scenarios. The videos were recorded using Full HD 1920 × 1080 resolution at variable frame-rate. Since we captured the videos by hand-holding and moving the tablet, the video frames present realistic distortions such as focus and motion blur, perspective, change of illumination and even partial occlusions of the document pages. In addition of the video clips, we also have captured an 8Mp picture of each of the documents to be used as models for the matching classification. We present an example of the different scenarios in Figure 3. Summarizing, the database consists of 180 video clips comprising 36.444 frames.

The classification task consists in automatically recognizing for each frame which of the thirty different documents appears.

B. Results

We report in Tables I and II the obtained classification accuracies and the required processing times for all the possible combinations of local key-point detectors and local descriptors. The best performances are reached when using the SIFT descriptor over SIFT key-points. The performance of the descriptor drops quite dramatically when it is computed over other key-points. Concerning the SURF key-point detector and descriptor, we appreciate that the SURF framework does not seem very suited for the specific task of dealing with document images. One the one hand, when computing local descriptors over SURF key-points, it is the BRISK descriptor that yields the best performances. On the other hand, the SURF descriptor achieves the best performances when used over ORB key-points. In both cases, the reached performances are quite far from the ones reached by SIFT. In addition, the processing times for the SURF descriptors are also important. Despite the fact that the SURF framework was proposed as a faster alternative to SIFT, the vast amount of key-points found when used over document images with its default parameters (we can appreciate it in Figure 1) provokes that using SURF might not be suitable in applications with realtime requirements. Computing SURF descriptors over SURF key-points is almost as costly as computing SIFT descriptors over SIFT key-points, with an important drop in accuracy. However, the SURF key-points were the ones which gave the best performance for the BRISK local descriptor. BRISK being a binary descriptor, achieves very good classification accuracies with much faster processing times than SIFT or SURF. However, being computed over such large amount of key-points, makes that the ORB framework is almost 10 times faster. Finally, we can appreciate that whatever descriptor we use, the MSER key-regions do not reach very good results.

It is worth to note that MSER does not provide a region orientation per se, so in principle computing local descriptors over MSER key-regions would not reach invariance to rotation. However, in OpenCV's implementation, SIFT, SURF and BRISK descriptors actually compute a dominant orientation in order to do an orientation normalization whereas ORB expects the key-regions to have already their angular information. This fact would explain the important drop in performance that we observe when computing ORB descriptors over MSER keyregions. Summarizing, the best performances are obtained by using the SIFT key-point detector with the SIFT descriptor, but it is worth to note that in scenarios requiring real-time computation, the performances reached when using the ORB framework are also very promising, since although there is a drop of nearly 15% classification accuracy, the processing times are trimmed by two orders of magnitude.

We report in Tables III and IV the averaged classification accuracies for the different backgrounds and document types respectively. We report the performances reached with the most promising configurations from Tables I and II. We show that the acquisition conditions have a strong impact on the final performances. The classification abilities are hindered when dealing with images acquired in low-light conditions, as in backgrounds 3 and 6 or in scenarios with severe clutter and occlusions, as the background 5. The performance drop due to low-light conditions (backgrounds 3 and 6) affects in a more severe way the BRISK and ORB descriptors than the SIFT descriptor, in which such performance drop is not that severe. Such effect is easy to understand taking into account that both BRISK and ORB are based on pixel intensities comparison, and thus quite sensitive to illumination changes, whereas the SIFT descriptor is based on gradient orientations which is more robust to such distortions. We also observe that the performance of the detectors and descriptors under analysis can also be severely hindered depending on the nature of the document to deal with. Documents which contain mostly text with uniform layouts such as the papers, patents or datsheets are harder to describe than the other documents presenting either more graphical information or having a more "textured" layout, as in the case of the letters, magazines and tax forms. V. CONCLUSIONS

We have presented in this paper a comparative study of local key-point detectors and local descriptors for the specific task of mobile document classification. The experiments, conducted in a database consisting of 30 model documents acquired on 6 different backgrounds, totaling more than 36.000 test images, show the dominance of the SIFT framework over other detectors and descriptors. Despite their lower performance, binary descriptors are also a good choice when the requirements of the application impose to have real-time responses. In such scenarios, both BRISK and ORB descriptors perform well, presenting processing times between one and two orders of magnitude difference in comparison with SIFT.

Finally, the detailed analysis of the results shows that the performance of such detectors and descriptors can be severely hindered depending on the nature of the document to deal with. Mostly textual documents are harder to describe than other documents presenting more graphical information or with a more "textured" layout. The acquisition conditions have also a strong impact on the final performances. The classification abilities are distorted when dealing with low-light conditions or with severe clutter and occlusions.

Fig. 1 .

 1 Fig. 1. Example outputs of the different key-points detectors for the same portion of a document image.

Fig. 3 .Fig. 2 .

 32 Fig. 3. Sample backgrounds used in our dataset when capturing the same magazine document.

TABLE I

 I

	.	CLASSIFICATION ACCURACIES
	Descriptor	Detector
		SIFT	SURF	ORB	MSER
	SIFT	85.16	67.81	36.07	55.32
	SURF	53.95	61.10	64.26	53.65
	ORB	29.67	25.00	70.81	9.01
	BRISK	69.51	79.66	39.20	63.23

TABLE II .

 II PROCESSING TIMES (SECS.) PER IMAGE

	Descriptor	Detector
		SIFT	SURF	ORB	MSER
	SIFT	2.974	5.166	3.886	0.273
	SURF	1.824	2.439	0.593	0.100
	ORB	0.334	0.222	0.033	0.065
	BRISK	0.420	0.339	0.057	0.070

TABLE III .

 III

		CLASSIFICATION ACCURACIES FOR THE DIFFERENT
			BACKGROUNDS
	Detector &	Background
	Descriptor	1	2	3	4	5	6
	SIFT / SIFT	98.79	95.69	81.47	89.22	61.94	83.82
	SURF / BRISK	97.42	95.77	54.07	91.91	68.44	70.37
	ORB / ORB	96.85	85.63	55.03	81.67	40.75	64.90

TABLE IV

 IV

	.	CLASSIFICATION ACCURACIES FOR THE DIFFERENT
			DOCUMENT TYPES
	Detector &	Document Type
	Descriptor	Datasheet	Letter	Magazine	Paper	Patent	Tax
	SIFT / SIFT	79.79	86.80	98.67	68.09	80.63	96.96
	SURF / BRISK	65.51	79.95	98.03	64.76	74.25	95.48
	ORB / ORB	51.47	73.06	94.68	45.57	77.03	83.02

http://www.opencv.org

ACKNOWLEDGMENT

This work has been partially supported by the Spanish Ministry of Education and Science under projects TIN2012-37475-C02-02 and by the People Programme (Marie Curie Actions) of the Seventh Framework Programme of the European Union (FP7/2007-2013) under REA grant agreement no. 600388, and by the Agency of Competitiveness for Companies of the Government of Catalonia, ACCI Ó.


Joseph Chazalon

Marc ¸al Rusiñol

Jean-Marc Ogier

Josep Lladós

A Semi-Automatic Groundtruthing Tool for Mobile-Captured Document Segmentation

This paper presents a novel way to generate groundtruth data for the evaluation of mobile document capture systems, focusing on the first stage of the image processing pipeline involved: document object detection and segmentation in lowquality preview frames. We introduce and describe a simple, robust and fast technique based on color markers which enables a semi-automated annotation of page corners. We also detail a technique for marker removal. Methods and tools presented in the paper were successfully used to annotate, in few hours, 24889 frames in 150 video files for the smartDOC competition at ICDAR 2015.

I. INTRODUCTION

In the past few years, the smartphone market expanded very quickly: according to the International Data Corporation [START_REF]Worldwide Smartphone Shipments Top One Billion Units for the First Time, According to IDC[END_REF], vendors sold a total of 1,004.2 million smartphones worldwide during the year 2013, which accounted for 55.1% of all mobile phone shipments. As a consequence of users having at nearly any moment of their life a powerful and handy device capable of taking increasingly better pictures triggered a new (and somehow unexpected) use: smartphones are progressively replacing desktop scanners, for both personal and professional use. This emerging trend is a source of new challenges for the document analysis and recognition community.

As a camera-based acquisition process, mobile document capture entails challenges which were already identified and studied by Liang et al. [START_REF] Liang | Camera-based analysis of text and documents: a survey[END_REF], but the most frequent or though ones may not be the same as camera-based document capture. Given the final goal of producing a document image suitable for many later uses (human readable, noise-free, OCR-able, indexable, etc.) like a scanned document page would be, we found that the most important tasks are, in decreasing order: background removal, perspective correction, lighting normalization, focus and motion blur avoidance. On the other hand, our recent experiments learned us that low resolution, non-planer surfaces, lens distortions, intensity and color quantization, sensor noise and compression are not majors issues for most applications. Regarding the need for lightweight algorithms, even if battery saving and real-time processing are mandatory, the processing power of smartphones makes those devices absolutely suitable for fast image processing.

Another important difference between camera-based document capture and mobile document capture is due to the intrinsic properties of the mobility situation, which induces specific constraints on the capture process. In the case of a receipt, for instance, a customer would rather digitize such document right after it is printed, instead of eventually losing it before checking his or her account balance, or getting refunded for a business expense. Therefore, we believe that mobile document capture solution should:

• be fast and simple to use;

• ensure that the quality of produced images are suitable for later processing: human reading, OCR, classification, etc.

We also believe that such features can only be obtained by assisting the user during the capture process, while he or she is pointing at the document to digitize. Ideally, the capture should be triggered at the best moment, as soon as the document is detected.

While working on a method for detecting and segmenting a page object within preview frames acquired during the capture process, we realized that evaluating such kind of approaches required a specific dataset, metrics and evaluation tools. Metrics based on overlapping areas are easily implemented, but to generate a large-scale and realistic dataset at minor costs required a new approach.

The contribution of this paper is to propose such approach, capable of generating large datasets of realistic preview videos and the associated segmentation ground-truth for document capture sessions. We introduce and describe a simple yet powerful technique based on color markers which enables a fast and robust annotation of objects corners, as illustrated in Figure 1.

Our method is based on two simple steps after acquiring video samples: 1) a semi-automated page segmentation step based on color markers, along with the removal of such markers; 2) a manual inspection and correction step to eliminate small errors and ensure a high quality dataset.

The method described in this paper was successfully implemented and used to annotate, in few hours, 24889 frames in 150 video files for the challenge 1 ("page object detection and segmentation in preview frames") of the smartDOC competition at ICDAR 2015. This method exhibited 3 main advantages:

1) it is very simple to set up and use, both for the semiautomated step and the inspection step; 2) it is inexpensive, both in terms of required material and manual workload; 3) it is robust to many distortions and is therefore suitable for the generation of realistic data.

This paper is organized as follows: Section II reviews similar datasets and approaches; Section III introduces the semi-automatic ground-truth generation method; Section IV details how color markers are removed from source videos; Section V discusses user experience and annotation costs; Section VI summarizes our contributions.

II. RELATED WORK

Several datasets related to mobile document capture have been proposed, but none of such approaches can be used to generate the ground-truth for the preview videos we introduced in the previous section.

The first dataset, to our knowledge, of camera-captured document images is the "DFKI-1 dataset", proposed by Shafait et al. [START_REF] Shafait | Document image dewarping contest[END_REF] and its evolution, the "IUPR dataset" [START_REF] Bukhari | The IUPR dataset of cameracaptured document images[END_REF]. Each dataset is composed of approximately 100 images, and represent many different documents classes captured in different situations with various distortions: perspective, warping, lighting, curl, skew, etc. The ground truth associated with each image is manually created and contains, along with a scanned version of each page, pixel-level accurate tagging of lines, text zones, and content type. ASCII text is also available. During the dewarping contest at CBDAR 2007 [START_REF] Shafait | Document image dewarping contest[END_REF], dewarping methods were compared against what a commercial OCR system would produced on dewarped images. Such approach is not viable in our case, as we cannot rely on the presence of a majority of textual content within each document. Furthermore, the manual annotation of each document image is not feasible when dealing with tens of thousands of video frames.

Bukari et al. [START_REF] Bukhari | Business forms classification using earth mover's distance[END_REF] recently proposed a method which makes use of the pixel-level content of images to compute a matching score using the Earth Movers Distance. While such technique was proved to perform well on scanned images for document classification, its application to evaluate the quality of the dewarping of some camera-captured document image seems difficult. Indeed, sensors noise and perspective distortion produce uneven image qualities after perspective correction, and we fear that it may not be possible to discriminate a wellsegmented low-quality frames from badly-segmented highquality frames because the pixel-level matching measure may confuse them.

Another direction was taken with the approach of Liang et al. [START_REF] Liang | Geometric rectification of camera-captured document images[END_REF]: the authors proposed a method for the geometric rectification of camera-captured document images, and used a synthetic dataset to evaluate their system. To avoid pixel-level comparison, they evaluated their method using the OCR accuracy after image rectification. While artificial image generation is attractive, it limits the complexity of the capture conditions, mostly in terms of lighting and device reaction to hand motion with six degrees of freedom.

Finally, one last approach, which specifically targets mobile document capture, was recently proposed by Kumar et al. [START_REF] Kumar | A dataset for quality assessment of camera captured document images[END_REF]. To evaluate their system for image quality assessment, the authors created a dataset of images containing various amounts of out-of-focus blur. Using a transparent support, the mobile was kept at a fixed position relative to each document, and variations of the focal length were used to generate the images. While the fixed position of the device could allow for the replication of the page coordinate within each image, such setup does not support enough distortions and motion for our situation.

None of those methods was suitable to generate a dataset of preview videos with realistic content and reasonable annotation time. During the design of a new approach, we decided to avoid this risk and the computational complexity of methods based on pixel-level matching, and resorted to use intersection-based metrics which are content-agnostic. This forced us to make two compromises in order to enable a semi-automated generation of segmentation ground truth: 1) we introduced color markers in the scene to define a referential which permits to recover the page position; 2) we dropped the support for warped documents in our method, as it would have prevented us from using simple quadrilaterals when detecting page content and matching results.

The next sections will demonstrate that, first, such approach produces high quality segmentation with very little manual work, and second that the color markers can generally be safely removed, avoiding any bias in segmentation methods and resulting in a minor perturbation in the end. The acquisition step, which will not be detailed, is very simple: as illustrated in Figure 1, we position markers around the document to capture, and start recording a video while trying to center the page like we would with a scanning application.

III. SEMI-AUTOMATIC GROUNDTRUTHING

In order to evaluate the performance of different document capture methods in video streams, we need to generate the ground-truth of our collection consisting of a quadrilateral that defines the borders of the paper sheet in the scene for each frame of the video. Since manually annotating each video frame is a tedious and expensive task, we decided to use a semi-automatic groundtruthing approach. One of the assumptions we made to generate the dataset is that the paper sheet lies on a flat surface and does not move. By placing color markers in the table that can be easily segmented, we could bootstrap the page segmentation process by computing the transformation that exists between the markers position and the paper sheet with a minimum human intervention.

The proposed semi-automatic approach works as follows. First we have to correctly segment the color markers that we see in Figure 2a). Given a video stream composed of n frames F 0 , ..., F i , ..., F n-1 , we present the first frame F 0 to the user and ask him to click on the four markers. The RGB values of each of the selected points together with a tolerance parameter are then used as thresholds in order to obtain the marker segmentation that we can see in Figure 2b). In order to be tolerant to illumination changes, the RGB values that serve as thresholds are updated iteratively for each frame, i.e. the RGB value of each of the markers centroid at the ith frame is used to segment the i + 1th frame.

Let us denote M i the quadrilateral formed by the four marker centroids M i = {A i , B i , C i , D i } and P i the quadrilateral formed by the four page corners P i = {W i , X i , Y i , Z i } for the ith frame, as we can see in Figure 3. A reference coordinate system described by four points M ′ = {A ′ , B ′ , C ′ , D ′ } is defined and we can then compute for each frame a perspective transform [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] H i that transforms the point set M to M ′ using

M ′ = H i M i .
When we compute H 0 for the first frame of the video we build the wrapped image F ′ 0 that is presented to the user for selecting the four corners P ′ = {W ′ , X ′ , Y ′ , Z ′ } of the page (see Figure 2c)). Since there is no movement between the page and the markers in the real world, ideally the wrapped images F ′ will all look alike and no matter the camera position and the perspective effect of the frame F i , the page will always be located at the same P ′ corners in the reference coordinate system. By backwards projecting the points from P ′ using the inverse perspective transform H -1 i , P i = H -1 i P ′ , we will find the corners of the page P i at any frame i.

Using such approach, we were able to annotate the whole dataset by just asking the user eight clicks at the first frame of each video. Four clicks for pointing out the color of the markers and four other clicks for determining the corners of the paper page in the wrapped image.

Finally, we can use the marker segmentation mask in order to "erase" the markers from each frame using an inpainting technique.

IV. MARKER REMOVAL

We have used the approach by Telea [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF] to remove the markers from the original video frames. Figure 4 illustrates the marker removal workflow: from a given frame containing color markers to be removed (Figure 4a), and the mask of the previously detected color markers (Figure 4b), we produce the final frame, for which the segmentation is known, and the markers are removed by an "inpainting" step (Figure 4c).

According to [START_REF] Telea | An image inpainting technique based on the fast marching method[END_REF], digital inpainting provides a means for reconstruction of small portions of an image. Like most inpainting approaches, Telea's approach aims at progressively propagating color information (values and gradients) toward the center of the regions to reconstruct. The inpainted value of a point p is computed using a weighted sum of the extrapolated intensities from all the points q in the neighborhood of p.

Telea's methods has several advantages which encouraged us to use it over other methods: is it fast enough to be capable of processing video frames in real time on a modern laptop; and due to its simplicity and publicly available code, many implementations are available.

In our case, the application of Telea's method requires a mask of the areas of the image to restore, and the size of the neighborhood to take into consideration for each pixel to restore. For the mask, we used a slightly dilated version of the mask produced by the color detection presented in Section III. It permits to tolerate a certain amount of blur, as well as some of the compression artifacts and noise. Regarding the size of the neighborhood, a few pixels (around 5) are usually sufficient to get results which are visually unnoticeable.

Figure 5 presents some details of results obtained with this marker removal method.

V. USER EXPERIENCE

The proposed method was successfully used to generate the sample and test sets for the Challenge 1 ("page object detection and segmentation in preview frames") of the SmartDOC competition 1 at ICDAR 2015, generating 150 inpainted videos (for 5 different backgrounds) with the associated segmentation ground-truth, giving a total of 24889 frames. Another set of 30 videos was processed, but its edition costs were too important, and we stopped the process to keep only a few valid outputs which were released as a sample of the test set.

In this section, we describe the evaluation of the user workload for each step of the process (semi-automatic groundtruthing, and manual inspection and correction) and give an overall estimation of the time required to generate the full dataset.

A ′ B ′ C ′ D ′ (0, 840) (0, 0) (700, 0) (700, 840) W ′ Y ′ X ′ Z ′ H i H -1 i F ′ F i A i B i C i D i W i Y i X i Z i

A. Semi-automatic groundtruthing

This first stage requires from the user, for each video file to process, to select from the first frame of the video (by clicking) the positions of the four markers, as well as the position of the four corners of the page.

During the frame-by-frame automatic processing, when some marker cannot be found in a frame, the system asks for a manual correction and pauses the processing of the current video. The user has to manually select again which are the marker positions. This is used to generate a new color mask and resume the process. We processed videos in parallel and pooled interaction requests at each stage to prevent the user from waiting for work.

Table I summarizes the cost of this first step. First frame actions refer to mandatory coordinate selections in the first frame of each video, and error correction actions refer to coordinate selections in an erroneous frame. It shows that we were able to reduce, at this step, the manual annotation of 24889 frame to 1232 coordinates selections (clicks). However, in order to produce a high quality ground-truth, manual inspection was still required.

B. Manual inspection and correction

Manual inspection and correction of the inpainted frames and the associated ground-truth required three kind of actions:

1) navigation actions: keystrokes or mouse wheel for forward and backward frame advance; 2) segmentation correction actions: choose four new coordinates for each frame where the segmentation page corner coordinates were erroneous; 3) inpainting correction actions: select four quadrilaterals to surround marker regions and define explicitly the inpainting mask (resulting in 16 coordinate selections).

Table II summarizes the costs for this second step. While navigation actions exhibit an important amount of user action, they are very fast and simple to perform. Segmentation corrections were simple and non frequent actions (only 21 frames were corrected). Inpainting actions were the costliest, requiring around 1500 clicks for the edition of 94 frames.

C. Overall evaluation

The global process of semi-automatic groundtruthing and inpainting, and a subsequent manual inspection and correction step, was really fast. For the first semi-automatic step, half an hour was sufficient to process all videos on a modern laptop with 8 cores and 8 GB of RAM. The second step lasted longer: each group of 30 videos took between 30 and 45 minutes to be reviewed and eventually corrected. It represents roughly 1 minute of manual inspection and edition for video recordings of around 10 seconds each. As it was mentioned in introduction of this section, another set of 30 videos was discarded because of the important processing cost it would have required.

VI. CONCLUSIONS

We presented a novel method based on color markers to generate a realistic dataset of preview frames, along with the associated segmentation ground-truth, suitable for the evaluation of assisting methods in the context of mobile document capture.

While our method does not permit to handle curled pages, it exhibits three main strengths. First, it is very simple to set up and use, both for the semi-automatic and the inspection steps: as soon as processing tools are available, unexperienced users can easily capture and process video samples without any particular knowledge. Second, it is inexpensive, both in terms of required material and manual workload: a smartphone and only a few color markers are required, and the overall edition cost is very reasonable regarding the important amount of frames which are automatically annotated. Third, it is robust to many distortions and is therefore suitable for the generation of realistic data: even in the context of low light or rapid motion, with various backgrounds, this method was capable of localizing the color markers and efficiently segment the page objects. Multiple orientations and size of pages can be captured, and there is no restriction on the content. This method was successfully used to generate the full dataset for the Challenge 1 ("page object detection and segmentation in preview frames") of the SmartDOC competition at ICDAR 2015, proving the accuracy and the efficiency of the overall process.

Fig. 1 .

 1 Fig. 1. Our annotation method allows for fast and reliable generation of segmentation ground-truth in videos based on color markers. a) Sample input frame. b) Sample output frame: markers are removed and segmentation is represented as a colored quadrilateral.

Fig. 5 .

 5 Fig. 5. Sample results for marker removal. a) Original content. b) Inpainted content. First line shows a sharp marker, second line a blurry marker, and third line a marker with highlight on it.

Fig. 2 .

 2 Fig. 2. Semi-automatic groundtruthing approach. a) Original frame. b) Marker segmentation. c) Wrapped frame in which the user marks the four document corners. d) Intermediate result with the document segmentation indicated as a red quadrilateral.

Fig. 3 . 1 iFig. 4 .

 314 Fig.3. Markers-to-corners approach. From any frame F i , marker centroids {A i , B i , C i , D i } are detected, and the forward transformation H i which maps those points to know coordinates {A ′ , B ′ , C ′ , D ′ } is found. The position of the page corners {W ′ , X ′ , Y ′ , Z ′ } within this referential F ′ are constant, and the inverse transformation of those coordinates with H -1 i gives the real coordinates {W i , X i , Y i , Z i } of the corners in frame F i .

TABLE I .

 I MANUAL COST FOR SEMI-AUTOMATIC GROUNDTRUTHING.

	Background	Frames		Actions
			First frame	Error correction
	background01	6180	240	0
	background02	6011	240	4
	background03	5952	240	24
	background04	4169	240	0
	background05	2577	240	4
	TOTAL	24889	1200	32

TABLE II .

 II MANUAL COST FOR INSPECTION AND CORRECTION.

	Background		Actions
		Navigation	Inpainting	Segmentation
	background01	6993	320	0
	background02	6254	224	4
	background03	6647	320	60
	background04	4251	144	0
	background05	2918	496	20
	TOTAL	27063	1504	84

https://sites.google.com/site/icdar15smartdoc

ACKNOWLEDGMENT

This work has been partially supported by the Spanish Ministry of Education and Science under projects TIN2012-37475-C02-02 and by the People Programme (Marie Curie Actions) of the Seventh Framework Programme of the European Union (FP7/2007-2013) under REA grant agreement no. 600388, and by the Agency of Competitiveness for Companies of the Government of Catalonia, ACCI Ó.


Joseph Chazalon

Marc ¸al Rusiñol

Jean-Marc Ogier

Improving Document Matching Performance by Local Descriptor Filtering

In this paper we propose an effective method aimed at reducing the amount of local descriptors to be indexed in a document matching framework. In an off-line training stage, the matching between the model document and incoming images is computed retaining the local descriptors from the model that steadily produce good matches. We have evaluated this approach by using the ICDAR2015 SmartDOC dataset containing near 25 000 images from documents to be captured by a mobile device. We have tested the performance of this filtering step by using ORB and SIFT local detectors and descriptors. The results show an important gain both in quality of the final matching as well as in time and space requirements.

I. INTRODUCTION

Exact document image matching in mobile environments using local descriptors has become in the latest years quite a trend in our community. Applications ranging from modelguided mobile document capture and classification [START_REF] Rusiñol | A comparative study of local detectors and descriptors for mobile document classification[END_REF], markerless augmented reality in documents and document retrieval [START_REF] Nakai | Hashing with local combinations of feature points and its application to camera-based document image retrieval-retrieval in 0.14 second from 10,000[END_REF] or document stitching and mosaicking [START_REF] Luqman | Mobile phone camerabased video scanning of paper documents[END_REF] are powered by such techniques. At its core, the proposed techniques in the document image analysis domain are no different from classical object recognition scenarios, although many research efforts have been devoted to design accurate and discriminative local descriptors adapted to the specific problem of matching documents.

For instance, Nakai et al. proposed in [START_REF] Nakai | Hashing with local combinations of feature points and its application to camera-based document image retrieval-retrieval in 0.14 second from 10,000[END_REF], [START_REF] Nakai | Camera-based document image retrieval as voting for partial signatures of projective invariants[END_REF], the Locally Likely Arrangement Hashing (LLAH) method that encodes the word distribution by computing a set of perspective invariant geometric descriptors. In [START_REF] Moraleda | Toward massive scalability in image matching[END_REF], Moraleda and Hull proposed a similar description based also on the way neighboring words are arranged and geometrically distributed.

However, despite the importance of designing new and performant local descriptors specifically tailored for document images, one should also consider the efficiency of such frameworks. Specially in the case in which such applications have to run in mobile devices at real time. Although LLAH performs queries in milliseconds in an indexed dataset of millions of pages, it also requires an incredible amount of RAM memory to store such descriptors which is an unrealistic requirement within today's mobile devices specifications.

Binary local descriptors such as BRIEF [START_REF] Calonder | BRIEF: Computing a local binary descriptor very fast[END_REF] or ORB [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF], in combination with efficient indexing schemes like Locality Sensitive Hashing [START_REF] Andoni | Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions[END_REF] (LSH), allow to run real-time document matching applications in recent smartphone devices. However, such approach presents a couple of drawbacks. On the one hand, such binary descriptors are not so discriminant when compared with state-of-the-art local descriptors like SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] or SURF [START_REF] Bay | SURF: Speeded up robust features[END_REF] for example. On the other hand, even though LSH allows a direct matching approach in real-time, it might not be that scalable when considering large amounts of documents to index in the dataset.

It is worth to note that usually the main bottleneck in terms of memory requirements is the vast amount of extracted local keypoints. Using default parameters, detecting local keypoints in a 300dpi scanned A4 page with either SIFT, SURF or ORB, usually results in thousands and even tens of thousands of keypoints to be indexed. Obviously, one could trim down such amount of keypoints by adjusting the goodness / saliency parameter of the local detector. However, reducing the amount of local keypoints by adjusting such parameter, does not guarantee that the really discriminative local descriptors are kept, just that the most reliable keypoint locations are retained. One can quickly realize that the most reliable keypoint locations in terms of cornerness, contrast or curvature (delivered by ORB, SIFT and SURF respectively) might not correspond to the most discriminative local descriptors. Specially when applying such frameworks in the document domain, where a lot of repetitive patterns (mostly coming from text) have to be expected.

In this paper we propose a simple but effective method aimed at reducing the amount of local descriptors to be indexed in a document matching framework. Inspired by the work by Kurz et al. [START_REF] Kurz | Representative feature descriptor sets for robust handheld camera localization[END_REF], we propose to build a "training" sequence of images containing one specific instance of the model document to index. In such sequence, the matching between model and images is computed retaining the local descriptors from the model that steadily produced good matches against this training sequence of images. We have evaluated this approach by using the ICDAR2015 SmartDOC dataset containing near 25 000 images from documents to be captured by a mobile device. We have tested the performance of our proposal against the use of both ORB and SIFT local detectors and descriptors. The results show an important gain both in quality of the final matching as well as in time and space requirements.

The paper is organized as follows. In Section II we briefly overview the document matching framework when using local descriptors. Section III introduces the proposed local descriptor filtering stage. In Section IV, we present our experimental setup and the associated results in Section V. Finally, Section VI presents our conclusions.

II. DOCUMENT MATCHING

We followed a standard architecture for document matching with local descriptors. Given a set of model documents D = {d 1 , d 2 , ..., d M } to index, we compute local detectors to end up with N keypoints K = {k 1 , k 2 , ..., k N } for each model document from D. Since we have run our experiments using the SIFT and ORB detectors, the keypoints from K are at some extend invariant to rotation, scale, illumination and perspective changes. Each keypoint k i is then described by a feature descriptor f i from either SIFT or ORB descriptors. Such descriptors are then indexed in an inverted file efficiently implemented with the FLANN architecture [START_REF] Muja | Fast approximate nearest neighbors with automatic algorithm configuration[END_REF], which either uses KD-trees for SIFT descriptors or LSH [START_REF] Andoni | Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions[END_REF] for ORB binary descriptors.

For any incoming image from the mobile device, keypoints and local descriptors are extracted as well and matched against the inverted file. In order to produce reliable matches, we use the ratio-test proposed by Lowe [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], in which a match is considered to be correct if the ratio between the nearest and the second nearest local descriptor is above a certain threshold.

From this set of putative matches, a RANSAC [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] step is performed in order to filter out the outlier matches that to not agree geometrically and to find the homography between the recognized model document and its instance appearing in the scene.

III. LOCAL DESCRIPTOR FILTERING

As mentioned in the previous section, the document matching framework relies heavily on the keypoints extracted from each document model. Such approach suffers from two major drawbacks. First, the heuristics used by keypoint detectors are based on local criteria (cornerness, contrast or curvature) and do not guarantee a stable matching over various perspectives or illumination conditions. Second, common keypoint detectors tend to extract many ambiguous elements which are usually filtered along the steps of the process.

Those two claims are illustrated with Figure 1. Figure 1a) shows a document model and its keypoints (small circles) originally extracted from it. We appreciate the presence of keypoints covering text areas and other ambiguous parts of the image. Furthermore, this illustration shows the usage frequency of each keypoint with a color scale: blue and green indicate keypoints which are rarely used to estimate the perspective transformed, while orange and red indicate good supports for this estimation. We can see that an important part of the keypoints retained by the local detector are not relevant for the document matching problem. This is is confirmed by Figure 1b), the histogram of usage for each keypoint as a support for the estimation of the perspective transform (i.e. not filtered by the ratio-test and the RANSAC stage).

To improve the selection of robust local descriptors, we followed an approach inspired by the work of Kurz et. al [START_REF] Kurz | Representative feature descriptor sets for robust handheld camera localization[END_REF]: we propose to build a "training" sequence of images containing one specific instance of the document model to index. In such sequence, the matching between a document model and sample images is computed retaining the local descriptors from the model that steadily produced good matches against this training sequence of images.

More specifically, the steps of the method are the following, given a document model d i and a train sequence S i,j :

1) use a local detector to extract an important amount of keypoints K i candidates from d i ; 2) build the histogram h i,j of keypoints usage (inliers) when matching d i (using K i ) with each image of S i,j ; 3) build a new filtered keypoint set K i \t with the t keypoints which exhibit the highest histogram values in h i,j ; 4) use the filtered keypoint set K i \t.

Is is worth noting that the training sequence does not require any special ground truth, except the reference of the document model captured. The selectivity of the matching ensures that only the best local descriptors are retained.

IV. EXPERIMENTAL SETUP

This section presents the dataset, evaluation protocol, metrics we used to evaluate the usefulness of local descriptor filtering.

A. Dataset

Our test dataset is the SmartDOC database for document capture (challenge 1) [START_REF] Burie | ICDAR2015 competition on smartphone document capture and OCR (smartdoc)[END_REF]. It consists of six different document types coming from public databases and five document images per class. The different types have been chosen so that they cover different document layout schemes and contents (either completely textual or having a high graphical content). In particular, the dataset consists of data-sheet documents and patent documents retrieved from the Ghega dataset [START_REF] Medvet | A probabilistic approach to printed document understanding[END_REF], title-pages from medical scientific papers from the MARG dataset [START_REF] Ford | Ground truth data for document image analysis[END_REF], color magazine pages from the PRIMA layout analysis dataset [START_REF] Antonacopoulos | A realistic dataset for performance evaluation of document layout analysis[END_REF], american tax forms from the NIST Tax Forms Dataset (SPDB2) [START_REF] Dimmick | Structured forms database[END_REF], and finally typewritten letters from the Tobacco800 document image database [START_REF] Lewis | Building a test collection for complex document information processing[END_REF]. An example of each of those six different document types is shown in Figure 2. Some small noise and margins from the original document images were removed and finally the images were rescaled to all have the same size and fit an A4 paper format.

Each of these documents were printed using a color laserjet and captured using a Google Nexus 7 tablet. Small video clips of around 10 seconds for each of the 30 documents in five different background scenarios were captured (details on each of the backgrounds and capture conditions are given in Table I and Figure 3). The videos were recorded using Full HD 1920x1080 resolution at variable frame-rate. Since the videos were captured by hand-holding and moving the tablet, the video frames present realistic distortions such as focus and motion blur, perspective, change of illumination and even partial occlusions of the document pages. Summarizing, the database consists of 150 video clips comprising near 25 000 frames. In addition of the video clips, an 8Mp picture of each of the documents was captured to be used as models.

This collection was ground-truthed by semi-automatically annotating the quadrilateral coordinates of the document position for each frame in the collection (see details in [START_REF] Chazalon | A semi-automatic groundtruthing tool for mobile-captured document segmentation[END_REF], [START_REF] Burie | ICDAR2015 competition on smartphone document capture and OCR (smartdoc)[END_REF]).

B. Evaluation Protocol

The evaluation we conducted compares the original keypoint selection scheme based on local detectors heuristics (hereafter named baseline), and the filtering scheme under investigation, detailed in the Section III. We describe here the generation of the test results for each scheme, that we later evaluated with performance measures presented in Section IV-C. 1) Baseline Scheme: The generation of document matching results using local descriptors selected with default heuristics is performed with the following algorithm, where D is the set of document models, T the set of the different numbers of keypoints to extract, S i is the set of video sequences (one for each background) for a given document model d i .

for each document d i ∈ D do for each keypoint value threshold t ∈ T do build K t i , a set of t keypoints extracted from d i with default heuristics for each test sequence s i,j ∈ S i do process s i,j , matching each frame against d i using K t i end for end for end for

2) Filtering Scheme: The generation of document matching results using local descriptors selected by a filtering based on there usage frequency on a training set is performed with the following algorithm, where D is the set of document models, T the set of the different numbers of keypoints to select, S i is the set of video sequences (one for each background) for a given document model d i .

for each document

d i ∈ D do extract K init i
, a set of init keypoints extracted from d i with default heuristics for each training sequence s i,j ∈ S i do build h i,j , the histogram of matches of K init i against each frame of s i,j for each keypoint value threshold t ∈ T do build K init i \t, a set of the t most useful keypoints from h i,j for each test sequence s i,t ∈ S i , s i,t = s i,j do process s i,t , matching each frame against d i using K init i \t end for end for end for end for

3) Tested Methods: To conduct our experiments, we studied the behavior of the ORB and SIFT detectors and descriptors. We used the following parameters.

• for ORB: init = 2 000, and T = {200, 400, 600, . . . , 2 000}

• for SIFT: init = 4 000, and T = {1 000, 2 000, 3 000, 4 000}

For each method, we extracted 1 000 keypoints from each frame of the train and test sequences.

C. Performance Measures 1) Segmentation Accuracy:

To assess the quality of the results produced by the variations of each method, we used the Jaccard index measure [START_REF] Everingham | The PASCAL visual object classes (VOC) challenge[END_REF] that summarizes the ability of the different methods at correctly segmenting page outlines while also incorporating penalties for methods that do not detect the presence of a document object in some frames.

Using the document size and its coordinates in each frame, we start by transforming the coordinates of the quadrilaterals of a matching method S and of the ground-truth G to undo the perspective transform and obtain the corrected quadrilaterals S and G . Such transform makes all the evaluation measures comparable within the document referential. For each frame f , we compute the Jaccard index (JI) that measures the goodness of overlapping of the corrected quadrilaterals as follows:

JI(f ) = area(G ∩ S ) area(G ∪ S )
where G ∩ S defines the polygon resulting as the intersection of the detected and ground-truth document quadrilaterals and G ∪ S the polygon of their union. The overall score for each method will be the average of the frame score, for all the frames in the test dataset.

2) Processing Speed: The processing speed is estimated using the average time required to process each frame of the test set. This later is obtained using instrumented Python code and running all the tests on the same dedicated machine.

3) Memory Impact: The memory gain can be directly deducted from the relative reduction of the size of the set of local descriptor for each document model.

V. EXPERIMENTAL RESULTS

This section presents experimental results for ORB and SIFT local detectors and descriptors, and discusses how the training stage should be performed.

A. ORB

Experimental results exhibit an important performance gain for ORB when filtering local descriptors. Figure 4a) shows an important improvement when progressively removing unnecessary keypoints from the model, until a peak after which the discriminative power of the model starts being hindered. Table II summarizes the values obtained for the Jaccard Index measure. We can see here that a 5-10 factor reduction of the descriptor set size can be achieved almost without harming results' quality. The processing time is also reduced significantly, with a gain of 25% for thresholds of 200 and 400.

B. SIFT

Contrary to ORB, SIFT benefits less from local descriptor filtering which only slows the performance drop, as illustrated by Figure 4b), and summarized in Table III. The slight gain in computing time may not balance the quality loss, and this can make ORB a competitive challenger for some situations.

C. Training Set Influence

The influence of the training set over the performance of the filtered approach is significant, as illustrated by Figure 5. The average quality difference can be explained by the uneven difficulty of each background, as training on a harder background makes testing easier. We note here, for both ORB and SIFT, that training on videos with extra objects should be avoided (background05), as it leads to poorer performance. In the case of ORB, Figure 5a) shows that almost any other training conditions produces interesting improvements. SIFT, on the other hand, seems to suffer from training with high illumination (background02) and to benefit from low-light or blurry conditions (background03). Standard conditions (back-ground01) also seem an acceptable choice for training SIFT.

VI. CONCLUSIONS

Inspired by the work by Kurz et al. [START_REF] Kurz | Representative feature descriptor sets for robust handheld camera localization[END_REF], we studied how a filtering step of local descriptors could influence the performance of a document matching framework. The evaluation was conducted using the ICDAR2015 SmartDOC dataset containing near 25 000 images from documents to be captured by a mobile device, and showed that the selection of local descriptors based on their response to a training sequence could lead to important performance improvements in the case of ORB, and slow the performance drop when reducing the number of descriptors in the case of SIFT. This experiments proved that, as long as the training is performed on sequences with well isolated document models, the framework can benefit from the filtering stage we proposed as it will both reduce processing time and memory requirements.

Fig. 1 .

 1 Fig. 1. Actual use of local descriptors when matching a document model with some test images. a) Visualization of keypoints, b) histogram of keypoints usage.

Fig. 2 .

 2 Fig. 2. Sample documents used in our dataset. a) Data-sheet from Ghega, b) letter from Tobacco800, c) magazine from PRIMA, d) paper from MARG, e) patent from Ghega and f) tax form from NIST.

Fig. 3 .

 3 Fig. 3. Sample backgrounds used in our dataset when capturing the same magazine document.

Fig. 5 .

 5 Fig. 5. Influence of training set over result quality

TABLE I .

 I

			SMARTDOC DATASET DETAILS
	Back-	Illumination Background	Blur and	Extra
	ground		color	artifacts	objects
	01	Ambient	Brown	Low motion blur No
	02	Ambient	Gray	Low motion blur No
	03	Low light	Light Salmon Motion	No
				and out-of-focus
				blur
	04	Intense	Light Gray	Motion blur	No
				and highlights
	05	Ambient	Light Gray	Motion blur	Yes

TABLE II

 II

				Influence of training background -ORB					Influence of training background -SIFT
		0.94			background01		0.94
	mean JI	0.92 0.90 0.88			background02 background03 background04 background05 All	mean JI	0.92 0.90 0.88					. 2000 LOCAL DESCRIPTORS, WITH AVERAGE PROCESSING TIME PER FRAME AVERAGE PERFORMANCE COMPARISON WHEN FILTERING (ORB)
		0.86						0.86			background01 background02	# descr.	Time (s)	Mean JI
		0.84						0.84			background03 background04			Baseline	Filtered
											background05
		0.82	500	# of keypoints kept in model 1000	1500	2000		1000 0.82	1500	# of keypoints kept in model 2000 2500 3000	3500 All	4000		200 400	0.127 0.127	0.637 ± 0.006 0.771 ± 0.005	0.855 ± 0.002 0.875 ± 0.002
														600	0.132	0.818 ± 0.004	0.876 ± 0.002
														800	0.137	0.837 ± 0.004	0.875 ± 0.002
														1000	0.143	0.846 ± 0.004	0.873 ± 0.002
														1200	0.149	0.852 ± 0.004	0.870 ± 0.002
														1400	0.155	0.856 ± 0.004	0.868 ± 0.002
														1600	0.161	0.857 ± 0.004	0.865 ± 0.002
														1800	0.166	0.860 ± 0.004	0.863 ± 0.002
														2000	0.171	0.861 ± 0.004	0.861 ± 0.002
														0.90	Influence of optimization -ORB
														0.85
														0.80
													mean JI	0.75
														0.70
														0.65
															optimized
															baseline
														200 0.60	400	600	800	1000 1200 1400 1600 1800 2000
															# of keypoints
															a) ORB
															Influence of optimization -SIFT
															optimized
														0.89	baseline
														0.88
													mean JI	0.87
														0.86
														0.85
														1000	1500	2000	2500	3000	3500	4000
															# of keypoints
															b) SIFT
													Fig. 4. Influence of filtering over result quality.
													TABLE III.	AVERAGE PERFORMANCE COMPARISON WHEN FILTERING
													4000 LOCAL DESCRIPTORS, WITH AVERAGE PROCESSING TIME PER FRAME
															(SIFT)
													# descr.	Time (s)	Mean JI
															Baseline	Filtered
														1000	1.488	0.847 ± 0.004	0.878 ± 0.002
														2000	1.624	0.877 ± 0.004	0.888 ± 0.002
														3000	1.718	0.888 ± 0.004	0.891 ± 0.002
														4000	1.781	0.892 ± 0.004	0.892 ± 0.002

ACKNOWLEDGMENTS

This work is partially supported by the People Programme (Marie Curie Actions) of the Seventh Framework Program of the European Union (FP7/2007-2013) under REA grant agreement no. 600388, and by the Agency of Competitiveness for Companies of the Government of Catalonia, ACCI Ó, and the Spanish project TIN2014-52072-P.