3. Max-pooling based Scene Text Proposal for Scene Text

Keywords: Scene text proposal, scene text detection, end-to-end scene text reading, word spotting system, scene

Experiments on various public scene text datasets show that the proposed scene text proposal techniques outperform other state-of-the-art scene text proposals under different evaluation frameworks. The designed end-to-end systems also outperforms other scene-text-proposal based end-to-end systems and are competitive to other systems as presented in the robust reading competition community. It achieves

which I can solve my difficulties during my studies. I also want to say thank you to Martin K. for his help in setting the server connection for my developed online scene text searching system and to Matthew for his advice on web based application development.

Finally, please allow me to express my genuine thanks to my thesis reviewers, Professor François BRÉMOND, and Professor Frédéric LERASLE, and the whole jury members. Their judgements and comments were very helpful in edifying this thesis. v "Education is the key to unlock the golden door of freedom" George Washington Carver For Dinh NGUYEN VAN

Resume

L'intérêt porté à la détection de contours pour la compréhension de texte dans une scène visuelle a été croissant au cours des dernières années comme en témoignent un grand nombre d'applications telles que les systèmes de reconnaissance de plaque d'immatriculation de voiture, les systèmes de navigation, les voitures autonomes basées sur la reconnaissance des panneaux de signalisation, etc. Dans cette recherche, nous abordons les défis de la conception de systèmes de lecture de texte de scène automatique robustes et fiables. Deux étapes majeures du système, à savoir, la localisation de texte dans une scène et sa reconnaissance, ont été étudiées et de nouveaux algorithmes ont été développés pour y remédier.

Nos travaux sont basés sur l'observation qu'indiquer des régions de texte de scène primaire qui ont forte probabilité d'être des textes est un aspect important dans la localisation et la reconnaissance de cette information. Ce facteur peut influencer à la fois la précision et l'efficacité des systèmes de détection et de reconnaissance.

Inspirées par les succès des recherche de proposition d'objets dans la détection et la reconnaissance objet général, deux techniques de proposition de texte de scène ont été proposées, à savoir l'approche Text-Edge-Box (TEB) et l'approche Max-Pooling Text Proposal (MPT). Dans le TEB, les fonctionnalités bottom-up proposées, qui sont extraites des cartes binaires de contours de Canny, sont utilisées pour regrouper les contours connectés et leur attribuer un score distinct. Dans la technique MPT, une nouvelle solution de groupement est proposée, qui est inspiré de l'approche Max-Pooling. À la différence des techniques de regroupement existantes, cette solution ne repose sur aucune règle heuristique spécifique liée au texte ni sur aucun seuil pour fournir des décisions de regroupement. Basé sur ces résultats, nous avons conçu un système pour comprendre le texte dans une scène visuelle en intégrant des modèles vii viii RESUME a l'état de l'art en reconnaissance de texte, où une suppression des faux positifs et une reconnaissance de mot peut être traitée simultanément. De plus, nous avons développé un système assisté de recherche de texte dans une scène en construisant une interface web en complément du système de compréhension de texte. Le système peut être consulté via le lien : dinh.ubismart.org:27790.

Des expériences sur diverses bases de données publiques montrent que les techniques proposées surpassent les méthodes les plus modernes de reconnaissance de textes sous différents cadres d'évaluation. Le système complet propose surpasse également d'autres systèmes complets de reconnaissance de texte et a été soumis à une compétition de lecture automatique dans laquelle il a montré sa performance et a atteint la cinquième position dans le classement (Dec-2017) : http ://rrc.cvc.uab.es/ ?ch =2&com=evaluation&task=4.

Abstract

Scene texts have been attracting increasing interest in recent years as witnessed by a large number of applications such as car licence plate recognition systems, navigation systems, self-driving cars based on traffic sign, and so on. In this research, we tackle challenges of designing robust and reliable automatic scene text reading systems. Two major steps of the system as a scene text localization and a scene text recognition have been studied and novel algorithms have been developed to address them.

Our works are based on the observation that providing primary scene text regions which have high probability of being texts is very important for localizing and recognizing texts in scenes. This factor can influence both accuracy and efficiency of detection and recognition systems. Inspired by successes of object proposal researches in general object detection and recognition, two state-of-the-art scene text proposal techniques have been proposed, namely Text-Edge-Box (TEB) and Max-Pooling Text Proposal (MPT). In the TEB, proposed bottom-up features, which are extracted from binary Canny edge maps, are used to group edge connected components into proposals and score them. In the MPT technique, a novel grouping solution is proposed as inspired by the max-pooling idea. Different from existing grouping techniques, it does not rely on any text specific heuristic rules and thresholds for providing grouping decisions. Based on our proposed scene text proposal techniques, we designed an end-to-end scene text reading system by integrating proposals with state-of-the-art scene text recognition models, where a false positive proposals suppression and a word recognition can be processed concurrently. Furthermore, we developed an assisted scene text searching system by building a web-page user interface on top of the proposed end-to-end system. The system can

List of figures

Scene texts and applications

Text is an effective tool in transferring information among people and between human and machines. Nowadays, it is worldly used for marking and describing many aspects in environments such as objects, events, shop panels, traffic signs, house plate numbers and so on, which are shown in Figure 1.1. Text information can be used for capturing rich semantic contexts for situational awareness applications such as a scene understanding, an action recognition, a navigation, and further for analysis purposes. Reading scene texts therefore becomes an increasing interest research topic. A lot of datasets and competitions in the scene text reading research have been announced [START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF][START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF][START_REF] Wang | Word Spotting in the Wild[END_REF][START_REF] Shi | ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)[END_REF]. Many productive applications have adopted scene text reading systems in practice. Google Translation [START_REF] Google | Google Translation Engine[END_REF] and Microsoft Translator [START_REF]Microsoft Translation Engine[END_REF] are two very popular applications which translate detected scene texts from one language to another language. Orcam [START_REF] Orcam | Orcam my eye[END_REF] is a recent application which is developed to assist visual impairment people in reading magazines, mails, and different texts in environment, such as texts in notification boards, street names. This product is still in a development phase and a lot of other visual based functions are targeted to be integrated, for example a general object recognition, a face identification. Scene text reading is also considered to adopt for a navigation application [START_REF] Joseph | Semantic Indoor Navigation with a Blind-User Oriented Augmented Reality[END_REF], a street house number reader [START_REF] Netzer | Reading Digits in Natural Images with Unsupervised Feature Learning[END_REF], a traffic sign recognition [START_REF] Greenhalgh | Recognizing Text-Based Traffic Signs[END_REF], and a license plate recognition [START_REF] Openalpr | License Plate Detection and Recognition[END_REF].

The Figure 1.2 shows some commercialized products which have been developed

base on scene text reading systems.

Even though there are many scene text reading based applications released, these applications are only reliable in fine arrangement of camera holders and standard text fonts. For example, the licence plate reader based car tracking system performs well only with standard fonts assigned to licence plate numbers. The OrCam is developed to mainly support visual impairment patients in reading printed texts such

Challenges

Different from scanned document texts which are in fine alignment, texts in scenes are more diverse in appearance due to variants of fonts, sizes, perspectives, colours, orientations and so on. In addition, their appearance is also distorted under various environmental impacts, such as light, haze, motion, illumination.

There is no standard text appearance in scene images. It could be printed texts as texts in panels, information boards, posters (Figure 1.3(a, b, c)) which could be detected by traditional OCR techniques [START_REF] Smith | An Overview of the Tesseract OCR Engine[END_REF]. On the other hand, text forms could be modified due to decoration purposes, for example texts in restaurant and shop panels Poles are similar to character 'l', character 'i' or character 't' (Figure 1.3 q-r).

(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) (o)
In normal usages, standard fonts printed in posters, dashboards, menus, panels are mostly considered. Recent developed applications such as Google translator, Microsoft translator and so on can solve this challenge properly even though texts are in difference perspective. In fact, reading texts algorithms which developed for scanned documents [START_REF] Smith | An Overview of the Tesseract OCR Engine[END_REF] also can handle. However, it requires users' experiences in handling their cameras for capturing quantitative pictures. In order to be independent from this requirement, researchers in robust reading community are moving forward to improve their systems so that they can perform properly on more challenging cases.

Scene text processing systems

In past two decades, many scene text reading/spotting systems have been proposed, observed by a major number of review reports [START_REF] Ye | Text detection and recogni-tion in imagery : A survey[END_REF][START_REF] Jung | Text information extraction in images and video : a survey[END_REF][START_REF] Liang | Camera-based analysis of text and documents : a survey[END_REF][START_REF] Uchida | Text Localization and Recognition in Images and Video[END_REF]. Traditionally, the system consists of two main tasks including a scene text detection and a scene text recognition. The scene text detection task aims to localize texts in scene images.

The scene text recognition task targets to recognize actual texts in located regions.

Detected regions were usually seen as parts of texts, characters, words, and text lines. In recent years, detected regions are more considered as word regions for boos-ting up scene text reading/spotting systems' performance because the word-level region is a standard input of recent developed recognition models. Each task has to tackle different difficulties and raises independent competitions as observed from recent competitions [START_REF] Computer Vision | Robust Reading Competition[END_REF][START_REF] Veit | CoCo-Text : Dataset and Benchmark for Text Detection and Recognition in Natural Images[END_REF]. Scene text proposal, which is inspired by success of object proposal in detecting non-labelled objects, has been proposed to substitute for scene text detection. In fact, it provides a larger chance of localizing different texts variants in scene images. A new scheme of scene text reading/spotting systems therefore is proposed including a scene text proposal and a scene text recognition, and it has been adopted in recent state-of-the-art scene text reading/spotting systems [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF][START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF][START_REF] Tian | Detecting Text in Natural Image with Connectionist Text Proposal Network[END_REF].

In this section, we are going to overview a major number of recent techniques and systems, which have been developed to address different reading-scene-text chal-

Scene text proposal techniques

The scene text proposal aims to generate region proposals with high recall. Despite the cost of a large number of false positive detections, it has a contribution toward reducing dramatically search spaces compared to sliding window. It's often evaluated according to the recall rate as well as the number of needed proposalstypically the smaller the better at a similar recall level [START_REF] Jan | How good are detection proposals, really ?[END_REF]. False positive scene text proposals are eliminated by either a text/nontext classifier [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF][START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF] or a scene text recognition model [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Nguyen | Text-Edge-Box : An Object Proposal Approach for Scene Text Localization[END_REF] in end-to-end scene text reading systems.

Different scene text proposal approaches have been explored. One widely adopted approach combines generic object proposal techniques with text-specific features for a scene text proposal generation. For example, EdgeBoxes [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF] is combined with Aggregate Channel Features (ACF) and an AdaBoost classifier to search for text regions in [START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF]. In [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF], Selective Search [START_REF] Uijlings | Selective Search for Object Recognition[END_REF] framework is adapted for scene text proposal, where Maximal Stable External Regions (MSER) [START_REF] Jiri | Robust Wide Baseline Stereo from Maximally Stable Extremal Regions[END_REF] is used for generating atomic regions which are later grouped together to form dendrogram based on a vast number of region features : a mean gray value of a region, a mean gray value of intermediate outer boundary, a region major axis, a mean stroke width, a mean of gradient magnitude at region borders and their coordinators. A text-specific symmetry feature is explored by using symmetry filters in [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF] to search for text line proposals directly, where false positive text lines are removed by training a CNN classifier. Deep features have also been used for scene text proposal due to its superior performance and efficiency on general object detection and object proposal in recent years. For example, inception layers are built on top of the last convolution layer of the VGG16 for generating text proposal candidates in [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF]. The Region Proposal Network (RPN) in Faster R-CNN [START_REF] Ren | Faster R-CNN : Towards Real-Time Object Detection with Region Proposal Networks[END_REF] structure is adopted for scene text proposal generation in [START_REF] Tian | Detecting Text in Natural Image with Connectionist Text Proposal Network[END_REF][START_REF] Jiang | R2CNN : Rotational Region CNN for Orientation Robust Scene Text Detection[END_REF]. In [START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF], a class activation map (CAM-conv) has been proposed and built on top of the convolutional state-5 of the VGG16 network, which is generated by using global average pooling and spatial pyramid pooling.

Scene text detection techniques

A large number of scene text detection techniques have been reported in the literature. Sliding window has been widely adopted [START_REF] Shangxuan | Text Flow : A Unified Text Detection System in Natural Scene Images[END_REF][START_REF] Siyu | A Text Detection System for Natural Scenes With Convolutional Feature Learning and Cascaded Classification[END_REF][START_REF] Wang | End-to-end Text Recognition with Convolutional Neural Networks[END_REF][START_REF] Wang | Word Spotting in the Wild[END_REF] [START_REF] Yin | Multi-Orientation Scene Text Detection with Adaptive Clustering[END_REF][START_REF] Qin | A Fast and Robust Text Spotter[END_REF]. On the other hand, segmented regions can be classified into text or nontext class before being grouped into words and text lines [START_REF] Yin | Robust Text Detection in Natural Scene Images[END_REF][START_REF] Kang | Orientation Robust Text Line Detection in Natural Images[END_REF]. In some researches, MSER is substituted by the External Regions (ER) that is more robust to blur, illumination [START_REF] Cho | Canny Text Detector : Fast and Robust Scene Text Localization Algorithm[END_REF][START_REF] Sung | Scene Text Detection with Robust Character Candidate Extraction Method[END_REF]. Various hand-craft text-specific features have also been extensively investigated. Stroke Width Transform (SWT) [START_REF] Epshtein | Detecting Text in Natural Scenes with Stroke Width Transform[END_REF] is proposed based on the hypothesis that characters in the same word have the proposed in [START_REF] Buta | FASText : Efficient Unconstrained Scene Text Detector[END_REF]. They are designed to search for terminals or corners of objects which are high-frequency features of text. Gray values of pixels at these key points are used to calculate local MSER thresholds which are used to segment images into MSER regions. Convolution Neural Networks (CNNs) are also trained for atomic regions initialization. In [START_REF] Yao | Scene Text Detection via Holistic, Multi-Channel Prediction[END_REF], a CNN network is deployed to generate three different maps including a text region map, a character map and a linking orientation map, on which scene text bounding boxes are produced. On the other hand, a pre-trained network is only used to provide scene text region maps on which atomic text regions are extracted. MSER and heuristic rules are adopted for grouping and splitting atomic regions into words [START_REF] Zhang | Multi-Oriented Text Detection with Fully Convolutional Networks[END_REF][START_REF] He | Text-Attentional Convolutional Neural Network for Scene Text Detection[END_REF].

Different from CNN models trained for atomic regions generation, several other CNN models have been developed for detecting text bounding boxes directly, including horizontal and quadrilateral shapes. For example, the DeepText [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF] adopts the VGG16 convolutional layers [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] for deep features extraction and inception layers for horizontal bounding boxes predictions including boxes' top-left corners coordinates (x,y) and their heights (h) and widths (w). Predicted boxes are then fed into last convolution layers of the network for eliminating false position ones. The TextBox [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF] adapts the Single Shot Multiboxes Detector (SSD) [START_REF] Liu | Single Shot MultiBox Detector[END_REF] architecture. Text bounding boxes are predicted from outputs of different intermediate convolution layers to address the multi-scale texts problem. In TextBox, they modified anchor boxes aspect ratios for predicting better bounding boxes which overlap ground truth boxes with higher IoU due to original anchor initialization only for general objects. Quadrilateral anchor boxes have also been proposed for detecting tighter scene text boxes [START_REF] Liu | Deep Matching Prior Network : Toward Tighter Multioriented Text Detection[END_REF]. In addition, a direct regression solution has also been proposed [START_REF] He | Deep Direct Regression for Multi-Oriented Scene Text Detection[END_REF] to remove the hand-crafted anchor boxes, which are usually initialized with certain number of boxes and aspect ratios in traditional CNN based regression models [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF][START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF][START_REF] Liu | Deep Matching Prior Network : Toward Tighter Multioriented Text Detection[END_REF]. In the proposed idea, bounding box coordinates are regressed from a single point such as a center of a studying region. Different other CNN based detection approaches have also been explored. For example, some works adopt a bottom-up approach that first detect characters and then group them to words or text lines [START_REF] Shi | Detecting Oriented Text in Natural Images by Linking Segments[END_REF][START_REF] Tian | Detecting Text in Natural Image with Connectionist Text Proposal Network[END_REF][START_REF] Hu | WordSup : Exploiting Word Annotations for Character Based Text Detection[END_REF].

Some systems instead define a text boundary class for pixel-level scene text detection [START_REF] Polzounov | Word-Fence : Text Detection in Natural Images with Border Awareness[END_REF][START_REF] Wu | Self-Organized Text Detection With Minimal Post-Processing via Border Learning[END_REF]. In addition, weakly supervised and semi-supervised learning approach [START_REF] Papandreou | Weaklyand Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation[END_REF] have also been studied to address the image annotation constraint [START_REF] Tian | WeText : Scene Text Detection Under Weak Supervision[END_REF].

Scene text recognition techniques

Scene text recognition assumes that text regions have been detected correctly, and it aims to read actual words in located regions. Traditional recognition models usually localize and recognize individual characters consequentially, and then sort them in proper orders to provide final recognition results. In recent approaches, convolution neural network (CNN) are employed. Almost CNN models consider word images as characters sequences [START_REF] Su | Accurate Scene Text Recognition based on Recurrent Neural Network[END_REF][START_REF] Boy | Scene Text Recognition and Retrieval for Large Lexicons[END_REF][START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF][START_REF] Shi | Robust Scene Text Recognition with Automatic Rectification[END_REF][START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF][START_REF] Gao | Reading Scene Text with Attention Convolutional Sequence Modeling[END_REF]. On the other hand, Jaderberg's model [START_REF] Jaderberg | Deep Features for Text Spotting[END_REF] converts a recognition task into a classification task and considers each word as a class.

In [START_REF] Bai | Strokelets : A Learned Multi-Scale Mid-Level Representation for Scene Text Recognition[END_REF], strokelets feature for character identification is proposed and a combination between Random Forest model and feature vectors based on Bag of Strokelets as well as Bag of HoG is implemented for character recognition. Recognized characters are later linked together from left to right into words. This traditional method is improved by integrating with top-down and button-up cues and the Conditional Random Field model [START_REF] Boy | Scene Text Recognition and Retrieval for Large Lexicons[END_REF][START_REF] Mishra | Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues[END_REF]. The button-up cues are texture features and the topdown cues are usually initialized lexicons. In [START_REF] Boy | Scene Text Recognition and Retrieval for Large Lexicons[END_REF], potential character regions are pointed by image binarization and false positive regions are removed by heuristic rules. In the [START_REF] Mishra | Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues[END_REF], a combination of sliding windows and HoG feature vectors is implemented for a potential character regions generation and Support Vector Machine (SVM) model is employed for a recognition. In [START_REF] Shi | Scene Text Recognition using Path-based Tree-structured Character Detection[END_REF], Conditional Random Forest model is used as a language model for eliminating strange detected characters by minimizing its cost function, where potential characters are detected base on their proposed part-based tree-structure.

In [START_REF] Su | Accurate Scene Text Recognition based on Recurrent Neural Network[END_REF][START_REF] Boy | Scene Text Recognition and Retrieval for Large Lexicons[END_REF][START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF][START_REF] Shi | Robust Scene Text Recognition with Automatic Rectification[END_REF][START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF][START_REF] Gao | Reading Scene Text with Attention Convolutional Sequence Modeling[END_REF], convolution neural networks are trained to treat words like characters sequences. They convert input images into sequences of feature vectors which are then decoded to transcripts. In [START_REF] Su | Accurate Scene Text Recognition based on Recurrent Neural Network[END_REF], they implemented sliding win- image width [START_REF] He | Reading Scene Text in Deep Convolutional Sequences[END_REF] or on whole image space [START_REF] Boy | Scene Text Recognition and Retrieval for Large Lexicons[END_REF]. In [START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF], an end-to-end trainable network has also been proposed by linking a convolution neural network, a LSTM, and a transcription layer into one network. Therefore, its feature map generation and transcript decoder can be trained concurrently. Weird text forms are considered in the [START_REF] Shi | Robust Scene Text Recognition with Automatic Rectification[END_REF][START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF]. In [START_REF] Shi | Robust Scene Text Recognition with Automatic Rectification[END_REF], a Spatial Transformer Network (STN) is employed to convert non-horizontal texts into horizontal ones before feeding them into a Sequence Recognition Network (SRN), which consists of two combinations : one is a combination between the CNN with the LSTM for sequence encoder and one is a combination between the GRU and the attention structure for a sequence recognition. In contrast, a model in [START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF] does not employ STN. Instead, a recurrent network is applied from left to right on sequences of attended regions generated by CNN networks for a word recognition. In [START_REF] Gao | Reading Scene Text with Attention Convolutional Sequence Modeling[END_REF], CNN network has been suggested to use for a sequence decoder instead of RNN. The target is to perform parallel operations and to be easier in training.

dow

Automatic scene text reading/spotting systems

Automated scene text reading system aims to detect and recognize texts in scene images. Traditional framework consists of a scene text detection and a scene text recognition [START_REF] Bissaco | PhotoOCR : Reading Text in Uncontrolled Conditions[END_REF][START_REF] Wang | End-to-End Scene Text Recognition[END_REF][START_REF] Neumann | Real-Time Scene Text Localization and Recognition[END_REF][START_REF] Neumann | Efficient Scene Text Localization and Recognition with Local Character Refinement[END_REF][START_REF] Yao | Un Unified Framework for Multioriented Text Detection and Recognition[END_REF]. In recent year, scene text proposal is proposed to substitute a scene text detection step, largely due to the fact that it is able to locate more text regions as compared with the text detection [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF]. Convolution neural network based techniques are also proposed to replace each single steps in the tradition framework by CNN models [START_REF] Wang | End-to-end Text Recognition with Convolutional Neural Networks[END_REF][START_REF] Li | Reading Car License Plates Using Deep Convolutional Neural Networks and LSTMs[END_REF][START_REF] Jaderberg | Deep Features for Text Spotting[END_REF][START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF] or develop single models handling both scene text detection and scene text recognition [START_REF] Busta | Deep TextSpotter : An End-To-End Trainable Scene Text Localization and Recognition Framework[END_REF].

A popular system is a Google-Translation [START_REF] Bissaco | PhotoOCR : Reading Text in Uncontrolled Conditions[END_REF] which performs end-to-end scene text reading by fusing a large number of techniques including three scene text detection methods, three scene text segmentation and grouping methods, deep neural network on Histogram of Oriented Gradient (HOG) features for character classification, and language models for post-processing. In [START_REF] Wang | End-to-End Scene Text Recognition[END_REF], sliding window is combined with HOG feature extraction and Random Ferns Classifier to compute text saliency maps where words are extracted using External Regions (ER) and further re-scored using Support Vector Machine (SVM). In [START_REF] Neumann | Real-Time Scene Text Localization and Recognition[END_REF], Adaboost classifier and SVM are applied on text regions that are extracted using ER to localize scene texts which are further recognized under an Optical Character Recognition (OCR) framework.

Similar approach was also adopted in [START_REF] Neumann | Efficient Scene Text Localization and Recognition with Local Character Refinement[END_REF], where Maximal Stable External Regions (MSER) instead of ER was implemented for scene text region localization. In [START_REF] Yao | Un Unified Framework for Multioriented Text Detection and Recognition[END_REF],

Stroke Width Transform [START_REF] Epshtein | Detecting Text in Natural Scenes with Stroke Width Transform[END_REF] is adopted for scene text region detection and Random Forest is used for character recognition and words are further recognized by a component linking, a word partition and a dictionary based correction. In [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF], potential text regions are first localized using EdgeBox (EB) [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF] or adapted simple selective search for scene text [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF] and scene texts are further recognized using Jaderberg's scene text recognition model [START_REF] Jaderberg | Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition[END_REF].

Quite a number of CNN based end-to-end scene text reading systems have been reported in recent years. In [START_REF] Wang | End-to-end Text Recognition with Convolutional Neural Networks[END_REF][START_REF] Li | Reading Car License Plates Using Deep Convolutional Neural Networks and LSTMs[END_REF], a CNN based character recognition model was developed where word information is extracted from text saliency map using sliding windows. The same framework has been implemented in [START_REF] Jaderberg | Deep Features for Text Spotting[END_REF], where a more robust end-to-end scene text reading system is developed by training a model handling three functions including a text and non-text classification, a case-insensitive characters recognition and a case-sensitive characters recognition. In [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF], an advanced endto-end scene text reading system is designed where the Single Shot Multi-boxes Detector (SSD) was employed for scene text detection and a scene text recognition model proposed in [START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF] was adopted for recognition. End-to-end trainable scene text reading system has also been proposed where it can provide texts location and text transcription concurrently by a single model [START_REF] Busta | Deep TextSpotter : An End-To-End Trainable Scene Text Localization and Recognition Framework[END_REF].

Objectives

This thesis aims to develop state-of-the-art automatic scene text reading systems which can be integrated into relevant applications supporting visual impairment people, especially elderly in their outdoor activities in the future phase of our project.

For developing, we mainly focus on challenges of scene text localizations when there is not any prior knowledge about texts known in advance, for example text font, text size, text color. The first goal is to provide high-quality proposal regions. The better performance of this step can support the entire system in providing more precise text localization and recognition. Based on developed scene text proposal techniques, our second goal is to develop a state-of-the art automatic scene text reading system and deploy a relevant application.

In scene text proposal techniques, the number of proposals is an essential parameter that can affect automatic reading systems' efficiency. A huge number of regions will burden further system steps because they have to iterate a lot of times along proposals. In our approach, we are intent on limiting the number of proposals provided by our proposed techniques while still gaining state-of-the-art recall rate by ranking our proposals and collecting top n proposals in the ranked list. In the first approach, we focussed on heuristic rules which gain enormous attention of researchers in developing scene text detection techniques. We did study on these rules and proposed two effective heuristic features for evaluating edge pixels, grouping edge connected components into proposals and scoring them. In the second approach, we are intent on being independent from text specific heuristic rules by proposing a max-pooling based grouping strategy. The proposed technique groups edge connected components into proposals naturally based on their internal distances. Effectiveness of the proposed features are indicated in comparisons with other state-of-the-art techniques.

In order to develop automatic scene text reading systems from scene text proposal, scene text classification and scene text recognition models are usually adopted to remove false positive proposals and recognize their actual texts. There is an observation that we can lean on recognition models for handling a classification task. For example we can use a threshold of recognition scores for eliminating false positive proposals. This idea is also implicitly embedded in deep-learning based object detection systems [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF][START_REF] Girshick | Fast R-CNN[END_REF], where proposals generated by a region proposal network (RPN) are scored by recognition layers and low recognition confident score ones are eliminated. In this thesis, we did evaluations on state-of-the-art scene text recognition model developed by Jardeberg [START_REF] Jaderberg | Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition[END_REF] and integrated it on top of our proposed scene text proposals as a classification and recognition model for designing our automated reading system.

In scanned document analysis, there are numerous applications developed to support people in reading such as : a searching-word, a translating-word, an automatic pronouncing-word. Developing an automatic scene text reading system will bring those effective tools into our daily life. Orcam [START_REF] Orcam | Orcam my eye[END_REF] is an application that supports visual impairment patients in reading articles by interacting with users fingers. Google translation [START_REF] Google | Google Translation Engine[END_REF] is proposed to detect words in scene images and translate them into expected languages. Inspired from the searching-word tool, we prefer to build the first application supporting people in searching their keywords in scene images. It could help people by saving their time in searching specific items in menus, aware environment around them based on expected words and so on.

Contributions

In this thesis, three major contributions are produced : scene text proposal techniques, an automatic scene text reading system and a scene text searching application.

Scene text proposal techniques

We

A scene text searching application

In order to evaluate reliability of the proposed automated reading system, we developed a demo of a scene text searching application and it is applied on real images captured by users' cameras. The demo has been developed based on web platform, and it can be accessed by several devices such as smart phones, tablets, laptops. Users can provide searching keywords by using microphones or keyboards.

The goal of this demo is to provide a reliable and accurate scene text searching system. It has been compared with the Google Trans in term of finding keywords in scene based on our evaluation scheme. In further phase of this project, we expect to integrate this system into more relevant applications interacting with various services such as navigation, circumstance alert, and so on.

Evaluation criteria

In this research, we developed two scene text proposal techniques and an automatic scene text reading system. Two evaluation frameworks therefore have been adopted in order to evaluate these techniques and system. For evaluating scene text proposal techniques, we utilize a framework which is widely used in an object proposal evaluation [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF][START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Jan | How good are detection proposals, really ?[END_REF]. For evaluating an automatic scene text reading system, we follow a framework provided by the robust reading competition community [START_REF] Computer Vision | Robust Reading Competition[END_REF].

Datasets

The In recent years, there are many other scene text datasets proposed and interested by research community. Each of them has been targeted for different purposes.

COCO Text :[18]

The dataset is produced based on the MS COCO dataset, which contains images of complex everyday scenes. The images were not collected with text in mind and thus contain a broad variety of text instances including machine printed texts, hand written texts and others. English is the most common language in this dataset, it also covers Western script such as German, Spanish, French, an so on. Illegible terms of texts are also annotated for advance evaluations.

Evaluation of scene text proposal techniques

There are three parameters needed to be measured when evaluating scene text proposal techniques : a recall rate, a number of proposals and processing time. The recall rate presents a technique ability to localize correctly texts in scene images. The better scene text proposal technique is one that can provide the higher recall rate using the smaller number of proposals and the faster process. In order to evaluate our technique as well as other state-of-the-art techniques, we adopted a general evaluation framework which is utilized widely to evaluated object proposal techniques, as are described in [START_REF] Jan | How good are detection proposals, really ?[END_REF].

In order to decide whether text objects have been localized successfully, the intersection over union (IoU) has been applied. It measures a fraction between an overlapped area of two boxes and an entire area covered by them, as illustrated in the first row in Figure 1.5. The higher IoU value presents the better overlapping between two boxes, meaning that a proposal box is the more similar to a ground truth box. Note that the IoU in this evaluation considers only a case of one-to-one overlapping instead of cases of one-to-many or many-to-one overlapping, which are mentioned in the evaluation framework for scene text detection techniques [START_REF] Wolf | Object count/Area Graphs for the Evaluation of Object Detection and Segmentation Algorithms[END_REF] (the second row in Figure 1.5). It is because state-of-the-art scene text recognition models mostly focus on a word recognition. Good scene text proposals on many-to-one is useless for scene text recognition models and good scene text proposal techniques on one-to-many need post-processes in order to split text lines into words before feeding into recognition models. IoU of 0.5 is usually considered as threshold of an overlapping condition to decide whether objects are detected in the object detection system. On the other hand, higher IoU is expected for evaluating scene text objects, due to the fact that proposal boxes achieving IoU of 0.5 sometime are not good enough for scene text recognitions, for example, proposal boxes are shown in the third row in Figure 1.5. The first case is a good IoU-of-0.5 proposal and the rest cases in the right are poor IoU-of-0.5 proposals. Therefore, the higher IoU threshold will be utilized to evaluate and compare scene text proposal techniques in this thesis.

Performances of scene text proposal techniques are presented by recall rates under various situations generated by different combinations between a number of proposals and an IoU threshold. When a number of proposal is fixed, the IoU threshold is varied and vice versa. The better technique will provide the higher recall rate at every evaluation scenario, especially when the number of proposal is small and the IoU threshold is high. In addition, this evaluation framework considers an average number of proposals each technique needs to perform their performances.

The better technique is one that requests the smaller number. There is a difference between our evaluation and the tradition object proposal evaluation framework that we limit the number of proposals at 2000 instead of 10k proposals as usual, which is mentioned in [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF] due to trade off between system efficiency and system accuracy.

Evaluation of automatic scene text reading systems

The evaluation framework for an automated scene text reading system is adopted from the evaluation framework of the scene text reading competition community [START_REF] Computer Vision | Robust Reading Competition[END_REF], which is embodied in [START_REF] Wolf | Object count/Area Graphs for the Evaluation of Object Detection and Segmentation Algorithms[END_REF]. There are three contextualizations (strong, weak and generic evaluation) and two kinds of systems (an end-to-end scene text reading and a scene text spotting) involved. In this evaluation, only words which are alphanumerical words and containing at least three characters are considers, others are ignored. Generally, a word is decided to be detected if there is at least one proposal box overlapping with its' ground truth box over IoU of 0.5 and it is recognized successfully.

Three contextualizations of an automatic scene text reading system are illustrated as a strong contextualization, a weak contextualization and a generic contextualization. They are clarified based on accompanied lexicon lists as a per-image vocabularies of 100 words, whole vocabularies in the testing set and a generic vocabularies of about 90k words, respectively. These lexicons are used to guide proposed automatic scene text reading system to filter out false positive proposals. In each contextualization, the F-measure value is calculated to evaluate system performance.

The better system is one that achieves the higher F-measure value. This value is calculated based on the system recall and the system precision following the below equation :

F _measure = 2 P recision • Recall P recision + Recall (1.1)
The end-to-end scene text reading system is more general than the scene text spotting system, where it considers alphanumerical words that consist of characters and numbers instead of only-character words on which the scene text spotting system focusses. In this chapter, we are going to present in detail recent state-of-the-art research works in proposing scene text proposal techniques and automatic scene text reading systems which are also targeted goals of this thesis. Four scene text proposal techniques including the adaptive selective search for scene text proposal [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF], the symmetry text line based technique [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF], the deep texts [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF] and the weakly supervised text attention network [START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF] as well as three automatic scene text reading systems including the TextBoxes [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF], the DeepTextSpotter [START_REF] Busta | Deep TextSpotter : An End-To-End Trainable Scene Text Localization and Recognition Framework[END_REF] and the Edgeboxes based scene text reading system [START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF] are described. Comparisons of our proposed techniques and systems to these research works are discussed in Chapters 3, 4, and 5.1.

Thesis outline

Scene text proposal techniques

The general framework of scene text proposal techniques is inherited from object proposal techniques. It includes two main steps : a proposals generation and a proposals scoring/ranking. In this review section, we will project each state-ofthe-art technique into this framework and clarify algorithms which they have used.

Note that these scene text proposal techniques are designed to provide word level proposals, which is the standard input of recent scene text recognition models.

Adaptive selective search for scene text proposals a. Proposal generation

This technique starts with the maximal stable external regions (MSER) algo- base on a single similarity cue i th is calculated using Equation 2.1 :

d (i) (r a , r b ) = (f i (r a ) -f i (r b )) 2 + λ(x a -x b ) 2 + (y a -y b ) 2 (2.1)
The f i (r a ) refers to a similarity cue i th of a cluster a. [x a , y a ] and [x b , y b ] are centre coordinates of cluster r a and cluster r b respectively. The λ is a horizontal priority parameter that controls a priority of horizontal grouping, and it is in a range of (0, 1).

For minimizing processing time, seven chosen similarity cues are all simple and low computational cost features, including two cues in the Lab colour space (a mean colour of pixels in the region (F lab ), and a mean colour of pixels in the immediate outer boundary of the region (B lab )), a mean intensity value (F ), a mean intensity value in the immediate outer boundary of the region (B), a mean value of the distance transformed connected component mask generated by stroke width transform (S), a mean of gradient magnitude on the bounder of the region (G), a major axis of their fitting ellipse (D).

In order to find proper proposals for localizing words, several variants of the proposed technique have been evaluated, using different complementary similarity cues, different colour channels and different spatial pyramid levels. In total, there are 84 technique variants. The optimal one has been found at a combination of DF BGS with two different image scales (1 and 0.5) in three colour channels as RGB.

b. Proposal scoring/ranking

The Real Adaboost classifier has been used to score generated proposals due to low computational cost. Feature vectors of proposals are extracted based on both individual similarity cues f i (r a ) and proposals' bounding boxes properties .

At each type of similarity cue, a feature is calculated using Equation 2.2.

F i (G) = σ i µ i (2.2)
where σ i and µ i are respectively the mean and standard deviation of the i th similarity cue in a particular group G, {f i (r) : r ∈ G}. For example, the optimal variant of this technique includes 30 (=5*2*3) similarity cues, so its proposal's feature vector contains 30 dimensions.

Proposal bounding boxes based feature dimensions are extracted base on relationship between the bounding box of the group constituent regions and the bounding box enclose only the regions' centres. A set of five simple features are originated from calculating : [START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF] the ratio between the areas of both bounding boxes, (2) the ratio between their widths, (3) the ratio between their heights, (4) the ratio between the difference of their left-most x coordinates and the difference of their right-most

x coordinates, [START_REF] Google | Google Translation Engine[END_REF] the ratio between the difference of their top y coordinates and the difference of their bottom y coordinates.

The Real Adaboost classifier has been trained on training images in the IC-DAR2013 dataset. In testing phase, the Logistic Correlation is applied on the raw classifier's outputs to obtain a probability value for each proposal. The output of this scoring algorithm is normalized in the range [0 ;1]. Proposals with their confident scores are sorted in the descending order, where the front-most proposals of the list have the highest probability of being texts.

Symmetry text line

This technique is developed base on a hypothesis as text regions usually exhibit is defined as a template scale. Two types of features are employed as a symmetry feature and an appearance feature.

-Symmetry features consist of two groups of features as a self-similarity feature and a contrast feature. The self-similarity is developed based on a hypothesis as adjacent characters bear similar color and structure while the contract feature is inspired by a high dissimilarity of text foreground to background. The self-similarity type feature is calculated based on pixels in the two middle rectangles (RMT and RMB). The contrast feature type includes two features as a top-contrast feature calculated from a RT and RMT, and a bottom-contrast feature calculated from a RB and a RMB. Following three equations : 2.3 for the self-similarity feature and two equations 2.4 and 2.5 for two types of contrast features from symmetry templates, respectively.

S c x,y = X 2 (h c x,y (RM T ), h c x,y (RM B)) (2.3) Ct c x,y = X 2 (h c x,y (RT ), h c x,y (RM T )) (2.4) Cb c x,y = X 2 (h c x,y (RB), h c x,y (RM B)) (2.5)
where, X 2 is a distance function, h c x,y (RP ) with P ∈ (T, B, M T, M B) denotes the histogram of a low feature cue c in the corresponding regions. There are four kinds of low cues adopted as brightness, color, texture, and gradient. The brightness and two color cues are extracted from Lab color space. The texture cue is extracted using implemented textons and the gradient cue is computed from pixels' gradient magnitudes. In total, there are (3*5=) 15 symmetry features extracted from one scale symmetry template at the location (x, y).

-Appearance feature is extracted by applying the Local Binary Pattern (LBP) feature [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] on the middle rectangles (RMT+RMB). It provides 59 features All features are concatenated into a 74-dimension feature vector (=15+59) representing the pixel at location (x, y) in where the corresponding template is centred.

Pixels are then classified into two classes as a symmetry axis pixel and a non symmetry axis pixel by using the Random Forest classifier. It forms a symmetry probability map. Due to implementing multi-scale input images, multiple symmetry probability maps are provided.

From a symmetry probability map, symmetry pixels are grouped together to produce symmetry axis fragments if distances between them are less than three pixels. Symmetry axis fragments are then merged together into text line proposals if they are satisfied following two geometric constraints :

-Angular different constraint : Two fragments are able to grouped together if their angular difference is less than π 16 , and their angular difference is calculated by Equation 2.6, where, A, B are two fragment, φ(A), φ(B) are their direction.

Φ(A, B) = |φ(A) -φ(B)|, (φ(A), φ(B) ∈ (- π 2 , π 2 ) (2.6)
-Distance constraint : Two fragments are able to grouped together if the minimal distance between them is less than maximal height of the two fragment heights where fragment height is the scale of the corresponding template. The minimal distance between two fragments is computed by Equation 2.7, where p, q are two given points in the two fragments A, B respectively, and an operation ||.|| is a distance calculation between two points.

D(A, B) = min(||p -q||), p ∈ A, q ∈ B (2.7)
A bounding box of a text line proposal is estimated as follow : The width is determined by the horizontal axis coordinates of the axis pixels belong to the text line proposal and the height is the scale of the corresponding template. The Inception RPN is applied on the top of the convolution feature map (Conv5_3) in the VGG16 model [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF]. It is inspired by the Inception block in the GoogleLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF], and it consists of a 3 × 3 convolution layer, a 5 × 5 convolution layer and a 3 × 3 max pooling layer. A 1×1 convolution layer is deployed on top of the 3×3 max pooling layer for dimension reduction. The output of these layers are concatenated into 640-dimension feature vectors which are then fed into two sibling fully connected layers to predict text confidence scores (a classification layer) and proposal regions (a regression layer). Note that there are 24 anchor boxes designed with four scales (32, 48, 64, and 80) and six aspect ratios (as 0.2, 0.5, 0.8, 1, 1.2, and 1.5) at each point in the final feature map. The regression layer is trained to regress these anchor boxes to find the best proposal bounding boxes. Proposal boxes are then pass through non maximal suppression (NMS) to provide a final set of proposals. In scene text proposal evaluation, only 500 proposals are kept from the provided list. The same as existing RPN models, the Inception-RPN model is trained using a multi-tasks loss function which is presented in Equation 2.8.

DeepTexts

L(p, p * , t, t * ) = L cls (p, p * ) + λL reg (t, t * ) (2.8)
where classification loss L cls is a soft-max loss and p and p * are given as the predicted and true labels, respectively. Regression loss L reg applies the smooth-L1 loss. The

t = {t x , t y , t w , t h } and t * = {t * x , t * y , t * w , t * h }
stand for predicted and ground-truth bounding-box regression, respectively. The t * is encoded as below :

t * x = (G x -P x ) P w , t * y = (G y -P y ) P h , t * w = log( G w P w ), t * h = log( G h P h ) (2.

9)

The P = (P x , P y , P w , P h ) and

G = (G x , G y , G w , G h ) are centre coordinates (x, y),
width (w) and height (h) of an anchor P and a ground truth G, respectively. The λ is a loss balance parameter and it is set to 3 in the training process of the Inception-RPN.

Weakly supervised text attention network

This technique adopts the deep learning based framework to produce text confidence score maps in which text pixels obtains higher scores than background pixels.

Text confidence score maps are then binarized by using a score threshold.

Discussion

Most existing scene text proposal techniques have various limitations. For example, the symmetry text line and the adaptive selective search for scene text proposal are efficient but often generate a large number of false positive proposals. The deep learning based techniques [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF][START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF] produces a small number of proposals but the recall rate becomes unstable when the Intersection over Union (IoU) threshold increases.

In addition, deep learning based scene text proposals require a huge amount of trai-

Automatic scene text reading systems

In this review, structures of state-of-the-art scene text reading systems are projected into a standard scene text reading framework including two major tasks as a scene text detection and a scene text recognition, excluding the DeepTextSpotter system which successfully combines these two tasks into a model. Relevant algorithms employed in each task are then described.

Edgeboxes based scene text reading system

This technique is developed following the standard framework of a scene text reading system, however two tasks are not wholly distinct. Scene text detection performance is still improved base on information gained from a word recognition model, leading to a stronger holistic text spotting system. Entire system is depicted in Figure 2.4

a. Scene text detection

The scene text detection model consists of three steps : a proposal generation, a proposal filter and a proposal bounding boxes regression.

In the proposal generation step, two object proposal techniques are implemented :

the EdgeBox technique [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF] and the proposed Aggregate Channel Feature Detector

b. Scene text recognition

The recognition model is designed base on the hypothesis that a word recognition problem can be solved similarly a multi-class classification problem. Each word in the pre-defined dictionary therefore is considered as a class. The model consists of five convolutional layers : [START_REF] Google | Google Translation Engine[END_REF][START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF], [START_REF] Google | Google Translation Engine[END_REF]128], [START_REF] Wang | Word Spotting in the Wild[END_REF]256], [START_REF] Wang | Word Spotting in the Wild[END_REF]512], [START_REF] Wang | Word Spotting in the Wild[END_REF]512] representing as [filter-size, the number of filters] and followed by three fully connected layers : 4000, 4000, 90000 for the number of units. The 90000 units in the last fully connected layer correspond to 90000 pre-defined words in dictionary. Each hidden layers are followed by a rectified linear non-linearity activation (ReLU) and a 2 × 2 max pooling. Due to integrating with fully connected layers, input image size has to be fixed and it is set at 32 × 100. The model structure is illustrated in box layer is integrated at the end of the network to regress text bounding boxes from network's default boxes and predict bounding box confidence scores. This last layer is inspired by the single shot multibox detector (SSD) [START_REF] Liu | Single Shot MultiBox Detector[END_REF] which is able to detect multi-scales objects with a single scale input image. Different from general objects, words have large aspect ratios. Therefore, large aspect ratio default boxes are included, containing 6 aspect ratios 1, 2, 3, 5, 7, and 10. Moreover, irregular 1 × 5 convolutional filters instead of the standard 3 × 3 ones are adopted, which fit better with larger aspect ratio words.

TextBox a. Scene text detection

The text-box layer stacks up the last feature maps generated by the conv11 layer with some other intermediate feature maps of the network to deal with the multi-scales problem. At each location in a feature map, it predicts bounding boxes confidence scores and regression values which is used to adjust associated default boxes to proper object bounding boxes. For example, at a location (i, j) in a feature map which associates with a default box (x 0 , y 0 , w 0 , h 0 ), the output of the Text-box layer is a set of values as (δ x , δ y , δ w , δ h , c), indicating that a box (x, y, w, h) is detected with a confident score c :

x = x 0 + δ x y = y 0 + δ y w = w 0 exp(δ w ) h = h 0 exp(δ h )
The predicted bounding boxes at all collected feature maps are aggregated and undergo a non maximal suppression process to eliminate false positive proposals.

The TextBox network is first trained on the SynthText dataset [START_REF] Gupta | Synthetic Data for Text Localisation in Natural Images[END_REF] for 50k iterations then fine-tuned on the ICDAR2013 training set [START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF] for 2k iterations. The multi-tasks loss function is implemented as below :

L(x, c, l, g) = 1 N (L conf (x, c) + L loc (x, l, g)) (2.10)
where N is the number of default boxes that match ground-truth boxes, x is the match indication matrix, c is the confidence, l is the predicted location, g is the ... Labels are all Latin characters (36 characters) as well as a "blank" label denoted by "-". Label probability map therefore has size of 37-by-[the number of feature vectors in sequence]. Connectionist Temporal Classification (CTC) is adopted for mapping labels distribution map into predicted words including two steps. The first step is to map a labels distribution map (y) into a labels sequence (l), where each column is presented by a label which has the highest probability in the column. The second step is to map generated labels sequence l into a predicted word (w) where repeated labels and "blank" labels are removed. For example, the second step will map a label sequence of "-hh-ee-l-lll-ooo-" into a word "hello". The conditional probability of predicted word w is defined as the sum of probabilities of characters in the corresponding labels sequence l.

p(w|y) = i p(l i |y) (2.11)
where, p(l i |y) is defined as following equation :

p(l i |y) = T t=0 l t i (2.12)
with l t i is a probability of having label l i at time stamp t.

CNN Bilinear Sampler CNN -RR-I-VE-R-S--III-D---EE- --------------------- --------------------- --W---AA---LLL-L--K---- --------------------- --------------------- 0.

DeepTextSpotter

The novel idea in this state-of-the-art scene text reading system is an end-to-end trainable network for both a scene text detection task and a scene text recognition task. The proposed network is depicted in Figure 2.8.

In the scene text detection task, they adapted the YOLOv2 architecture [START_REF] Redmon | YOLO9000 : Better, Faster, Stronger[END_REF] for

x = σ(r x ) + c x y = σ(r y ) + c y w = a w exp(r w ) h = a h exp(r h ) θ = r θ (2.13)
The scene text recognition network exploits a fully-convolutional network for predicting distribution of characters in time sequence. The network consists of four types of layers as a convolution layer, a max-pooling layer, a batch normalization layer and a recurrent convolution layer. Network input images have a size of (H × W ), where the height H is fixed at 32 and the width W is flexible depending on image length. Network output shape is designed as 1 × W 4 × | Â| which is compressed into a 2D matrix with a shape of W 4 × | Â|. The | Â| is length of character set  including all English characters, ten numbers, and a symbol " -" representing a non-text class. At the end, the connectionist temporal classification (CTC) network is applied to transform network outputs into conditional probability distribution over label sequences which is later used to compute probability of a predicted word, please look at section 2.2.2 for a detail of how the CTC network is applied. The recognition network has been trained on the synthetic word dataset [START_REF] Jaderberg | Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition[END_REF].

In order to join two above networks into a network, the bilinear sampling process is applied. For each region of interest (ROI) with a shape of w × h × C, it is mapped into a fixed-height tensor with a shape of wH h ×H ×C (H = 32) following Equation 2.14. The κ is a bilinear sampling kernel κ(v) = max(0, 1 -|v|) and T is a point-wise coordinate transformation, which projects coordinates x and y of the fixed-sized tensor V to the coordinates x and y in the detected region features tensor U .

V c x ,y = w x=1 h y=1 U c x,y κ(x -T x (x ))κ(y -T y (y )) (2.

Discussion

Recent state-of-the-art scene text reading systems are developed using deep learning architecture. The TextBox system [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF] consists of two deep learning models for two different task as scene text detection and scene text recognition. The Deep-TextSpotter [START_REF] Busta | Deep TextSpotter : An End-To-End Trainable Scene Text Localization and Recognition Framework[END_REF] provides an advantage design that combines two independent tasks of scene text reading system into an end-to-end trainable deep learning model. In order to provide state-of-the-art performances, these techniques need to be trained on huge amount of data such as ICDAR, SVT, COCO Text, SynthText and so on.

Our developed automatic scene text reading system adopts a framework presented in the EdgeBoxes based scene text reading system that exploits a scene text proposal technique for localizing texts in scene images and a scene text recognition model for both recognizing actual words in proposal regions and improving scene text detection performance. An advantage feature of our developed system is that we only used a top of 2000 proposals to pass to a recognition model instead of 10k proposals as mentioned in recent scene text proposals based systems [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF].

Conclusion

In this chapter, we reviewed state-of-the-art works relating to this thesis scope, including scene text proposal generation and automatic scene text reading systems in both hand-craft features based works and deep learning based works. Their performances are observed by their positions in the ranked list of algorithms in the robust reading competition website and scientific publications, which can be considered as proper benchmarks for validating our proposed techniques and systems. In the ensuing chapters, our proposed techniques and systems will be described and compared with these state-of-the-art works. dataset [START_REF] Lucas | ICDAR 2003 Robust Reading Competitions : Entries, Results and Future Directions[END_REF]. In this technique, a large number of proposals is required to avoid loosing a certain number of challenging texts. However, it will influence efficiency of entire scene text detection/recognition systems when classification/recognition models have to repeat its processes on every proposals. Reducing a number of proposals while maintaining system performance therefore is an essential contribution.

We suggest to integrate object proposal techniques with text specific features to filter out coarsely false positive proposals, as represented in recent developed scene text proposals [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF][START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF].

A lot of text specific heuristic rules have been proposed to group atomic regions, such as edge pixels and MSER regions, into scene text proposals [START_REF] Lu | Scene text extraction based on edges and support vector regression[END_REF][START_REF] Epshtein | Detecting Text in Natural Scenes with Stroke Width Transform[END_REF][START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF][START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Shangxuan | Text Flow : A Unified Text Detection System in Natural Scene Images[END_REF][START_REF] Mishra | Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues[END_REF]. For example, the stroke width transform (SWT) [START_REF] Epshtein | Detecting Text in Natural Scenes with Stroke Width Transform[END_REF] proposes stroke width features measuring distances between opposite orientation edge pixels in edges connected components based on an assumption that text edge connected components have constant stroke width. In [START_REF] Mishra | Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues[END_REF], a bunch of geographical text specific features are utilized for analysing MSER regions and eliminating false positive ones before moving on a grouping step.

Inspired by heuristic rules of text edges, we proposed two edge based text specific features for estimating text probability of edge connected components. These probabilities are also used to score our scene text proposals. True text proposals are supposed to achieve high scores and vice versa. This technique is developed based on the EdgeBox framework with our adapted grouping solution and scoring function, specifying for scene text objects.

Methodology

The proposed Text Edge Box (TEB) technique is designed to produce wordlevel text proposals in scenes. The framework is shown in Figure 3.1 including two main steps as a text edge map generation and a grouping-scoring-ranking proposal.

Firstly, we exploit the Canny edge detector [START_REF] Canny | A Computational Approach to Edge Detection[END_REF] to generate a binary edge map. A gradient map, an orientation map are also collected from the Canny's immediate

Specific text edge features

In this research, we define two specific text edge features as an Edge Pair Feature (EP) and an Edge Variance Feature (EV). They are our key contributions which are proposed to clarify text edges from non-text edges and evaluate how possible connected components (CCs) are text CCs. The evaluated probabilities are also CCs' scores and they are stored in a Text Edge Map (TEM).

The edge pair feature

The first feature is an edge pair (EP) which is inspired by the Stroke Width Transform method [START_REF] Epshtein | Detecting Text in Natural Scenes with Stroke Width Transform[END_REF]. It is developed based on a hypothesis that CCs of text objects are likely to contain the high portion of couples pixels that have opposite orientations, like the connected component A illustrated at the path c in Figure 3.2.

From now, we name these pixels as edge pair pixels. In order to detect them, we start at each given pixel in a CC and its orientation is used to decide a searching line (dash lines in Figure 3.2.c). If an opposite orientation pixel in the same CC has been found in the searching line, the considered pixel and the searched one with an opposite orientation are defined as edge pair pixels. The EP feature of a given CC is defined as a fraction of edge pair pixels in the CC as follows :

EP (CC) = N pp (CC) N p (CC) (3.1)
Where N pp (CC) and N p (CC) denote the number of edge pair pixels and the number of edge pixels belonging to a CC under study, respectively. The value of this feature is in the range of [0, 1]. The CC having higher EP value is more likely to be a text CC.

The edge variance feature

The second feature is an edge variance (EV) that measures the variance of gradient magnitudes of pixels in a CC. This parameter is useful because the gradients of pixels in the boundary of an individual character (or boundaries of characters in a same word) are often monotonous. Therefore, their variances are expected to be small. We utilize an exponential function of the gradient variance in order to normalize these values into the range of [0, 1] and produce high values for text CCs as below :

EV (CC) = e -var(CC) (3.2)
Where the var(CC) denotes the variance of the gradient of pixels in a CC.

The text edge map

The text edge map is a score map that shows the being-text probability of each perceive that most of non-text edges have been removed, meaning that text edges pixels have been scored higher values than others.

Scene text proposal generation

Generated CCs are then merged into text lines which are later split into small subgroups referring to word-level proposals. The proposed grouping strategy is illustrated in Figure 3.4.

As Figure 3 A searching space is estimated from a bounding box of a starting connected component (CC). A given CC in the searching space will be grouped with the starting CC if it satisfies certain grouping rules.

between min(w A , w B ) and max(w A , w B ) is higher than τ w , (3) the ratio between min(h A , h B ) and max(h A , h B ) is higher than τ h . The parameters that are γ and τ s are sensitive to horizontal texts. The τ w and τ h are sensitive to size relationship between characters in a word. How to set values for these parameters is discussed in Section 3.2.1.

In order to divide text line proposals into small subgroups which refer to wordlevel proposals, an average of distances between adjacent CCs' boxes in the horizontal direction is estimated. Dividing positions are decided at the location where the distance is larger than the average value. Generally, a proposal has a high likelihood of being word if it satisfies (1) its CCs should have similar scores, (2) the heights of CCs should be approximately stable and

Scene text proposal ranking

(3) the connection lines between the CCs should be appropriate in same direction, referring to text lines. Therefore, the variances of these measures are expected to be small for a word region proposal. In order to derive a high S c , S h , S o value for a group of CCs which is likely to be a word, and also normalize these measures in the range of [0, 1], the arctan functions is implemented for each measure as follows :

S x = 2 • arctan(k x /var x ) π (3.4)
where the symbol "x" represents for the c, h, and o, and the var x refers to the variances of CC's score, height, and angle respectively. The parameter k x is set at the middle of each measure range, i.e. 0.5, a half of bounding box height and π/2, respectively for S c , S h , and S o . The score function of a proposal region S p is computed as :

S p = S a • arctan(k 1 S x ) (3.5)
where, the function arctan(k 1 S x ) is used to control the relationship between the S p and the S a . If the S x makes the arctan(k 1 S x ) value higher than 1, we say that the S x has a supporting effect (S p > S a ). If the S x makes the function value smaller than 1, it has a penalizing effect (S p < S a ). It means that : although a proposal has a high value of the S a , it is unlikely to be a word proposal if its CCs provide low S x values (referring to the penalizing effect, see the low score candidate at the grouping-scoring-ranking step in Figure 3.1). The parameter k 1 adjusts the role of the S x measures. In particular, if the k 1 increases, the role of the S x measures is reduced. In this research, we expect that if the S x is higher than the middle value of its' range as 0.5, the function arctan(k 1 S x ) has the supporting effect and vice versa. So, the parameter k 1 is set at 3 as the arctan(3•0.5) 1. Note that, the S x are considered only when the number of CCs in a proposal is larger than 3. Otherwise, the S p is calculated based on the S a only.

TextEdgeBox implementation

The Text Edge Boxes (TEB) technique is implemented by two programming languages as Matlab and C++. Matlab programming language is used to read and pre-process images, generate a Canny binary edge map, a magnitude gradient map as well as an oriented gradient map, evaluate technique performance and save evaluated results. C++ programming language is used to implement major contributions of the TEB. The TEB class has been shown in Appendix. In the below TEB pseudo code, Matlab functions are highlighted in white backgrounds and C++ functions are highlighted in gray backgrounds.

- ------------- -------------- The bounding boxes generation task is applied to provide corresponding bounding boxes for text lines, words and single connected components. Concurrently proposal scores are also calculated using the proposal scoring function which is implemented inside the boxlist2Box function.

---------------Function : ClusteringEdges

- -------------

Evaluation

In this section, we describe how we optimize the proposed technique and compare its performances to other state-of-the-arts. In order to improve the robustness of the proposed system under a wide diversity of text appearance, these five parameters (γ, τ s , τ w , τ h and α) are determined based on the joined training sets of the ICDAR2013 dataset (for high contrast texts) [START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF] and the Street View Text (SVT) (for blur texts) [START_REF] Wang | Word Spotting in the Wild[END_REF]. An optimal combination of those parameters will be used to set-up the proposed system. The proposed technique has been compared to other state-of-the-art techniques including the simple text specific selective search (TP) [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF], the Symmetry-Text Line (STL) [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF], the DeepText (DT) [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF], the EdgeBox (EB) [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF], the Geodesic (GOP) [START_REF] Krahenbuhl | Geodesic Object Proposals[END_REF], the RandomizedPrim (RP) [START_REF] Manen | Prime Object Proposals with Randomized Prim's Algorithm[END_REF] and the Multiscale Combination Grouping (MCG) [START_REF] Ponttuset | Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation[END_REF] on the two scene text datasets by following scene text proposal evaluation framework as presented in Section 1.6.2.

Parameters tuning

We first focus on generating high quality set of proposals, which maximizes overlap with the ground truth, by varying the four parameters γ, τ s , τ w , τ h . The ranking step is ignored and all number of generated proposals used for evaluation.

After obtaining good proposals, we then concentrate on scoring proposals to be able to shift likely-to-be-text proposals to the top of the list by arranging the found group in the descending order, so the proposed technique can perform well under a limitation of number of proposals.

In the first optimization step, the parameter γ is tuned in the range of [START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF][START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF] with an internal step of 0. 3.1, when we remove the EP feature, performance of the TEB technique drops dramatically. In contrast, its' performance is just slightly lower than optimal performance when the EV feature is removed. Therefore, the EP feature seems more reliable than the EV features. In comparison with other state-of-the-art techniques, the α is set at 0.7. Finally, in evaluation and comparison to other state-of-the-art techniques, the five heuristic parameters of the proposed technique is set as :γ = 1.5, τ s = 0.7, τ w = 0.3, τ h = 0.7 and alpha = 0.7. the Symmetry-Text Line (STL) [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF], the DeepText (DT) [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF], the EdgeBox (EB) [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF], the Geodesic (GOP) [START_REF] Krahenbuhl | Geodesic Object Proposals[END_REF], the RandomizedPrim (RP) [START_REF] Manen | Prime Object Proposals with Randomized Prim's Algorithm[END_REF] and the Multiscale Combination Grouping (MCG) [START_REF] Ponttuset | Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation[END_REF] In addition, we also track the average number of proposals that each method provides to achieve their best performance. Hosang [START_REF] Jan | How good are detection proposals, really ?[END_REF] shows that this criterion correlates well with the detection performance and it has been used to evaluate quality of the proposals in the TP [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF]. Table 3.2 and 3.3 show the experimental results on the two datasets. On the ICDAR2013 dataset, the TEB algorithm provides less number of proposals than competitive techniques as TP and EB. On the SVT dataset, the TEB performs slightly lower in comparison with the EB algorithm, but better than other state-of-the-art algorithms. On the other hand, the average number of proposal regions required are clearly more than those for the ICDAR2013 dataset. Besides the poorer image quality in the SVT dataset, one important reason of the lower performance is due to the ground truth of the SVT dataset where the manually labelled bounding boxes are often much larger than the actual boxes. This is illustrated in Figure 3.6 where the ground truth boxes in the red color are clearly much larger than the boxes produced by the proposed TEB in the green color.

Experimental results

The efficiency of the proposed technique is also evaluated based on the processing time. All above techniques are evaluated on the same computer and executed in one thread as the Xeon CPU E5-1650 v2 @ 3.5GHz. As presented in Table 3.2 and 3.3, the proposed TEB is comparable to the most efficient methods except the original EB method. However, the original EB method does not perform well in term of the number of proposals and the maximum recall obtained. For the DeepText method [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF], the authors have not released their program yet, and we do not have a processing time report in our device. According to their report, their algorithm takes average 1.7 second for processing an image in the ICDAR2013 dataset in their device using the single GPU K40 which is much more powerful than what we used. They are presented in Figure 3.7 as the RegModel's performances. This is the maximum performance that each proposal technique might obtain if they can provide good proposals that match perfectly to the ground truth boxes. As shown in Figure 3.7, the TEB method produces the largest number of good proposals which help the recognition model read contained words correctly. In addition, the performance of Besides, while testing on the SVT dataset that includes a certain amount of non-horizontal text lines, the TEB is still competitive as compared with the TP, which is designed without horizontal restriction. It is observed that the proposed algorithm can handle multi-orientation text lines. As discussion in Section 3.1.3, the horizontal text line assumption can be relaxed by increasing the parameter γ for a larger searching space and reducing the parameter τ s for retaining more CCs. Then, the scoring function should be upgraded to handle a huge number of CCs in the merged groups.

Conclusion

In this chapter, we proposed a text-specific proposal algorithm to search text regions in scenes. Two text-specific features, namely, an edge pair and an edge variance, were designed to search for more likely text components. In order to measure the text likelihood of the proposal boxes, we designed a scoring function that com- The Text-Edge-Box (TEB) technique and other existing scene text proposal techniques usually creates proposals by merging atomic elements of images (edge connected components, MSER regions) based on a bunch of heuristic rules with a lot of parameters [START_REF] Shangxuan | Text Flow : A Unified Text Detection System in Natural Scene Images[END_REF][START_REF] Lu | Scene text extraction based on edges and support vector regression[END_REF][START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Epshtein | Detecting Text in Natural Scenes with Stroke Width Transform[END_REF][START_REF] Papandreou | Weaklyand Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation[END_REF]. Optimizing those parameters could cost huge amount of time. For example, the TEB technique has to optimize a combination of five parameters, including four parameters in a grouping step and one parameter in a scoring step. The TextFlow (TF) [START_REF] Shangxuan | Text Flow : A Unified Text Detection System in Natural Scene Images[END_REF] has three parameters as a horizontal distance, a vertical distance and a size similarity which are needed extensive tests to search for optimal values. Generally, those parameters evaluated on geographical relation among atomic elements, and optimal values are thresholds applied on those parameters for providing grouping decisions. In this chapter, we are intent on providing a proper grouping solution that are independent of text heuristic parameters.

Studying on scene text geographic appearance, we observe that texts usually appear in words and text lines, and internal distances between characters are usually extracted as a feature and used for grouping atomic regions into text lines and splitting text lines into words [START_REF] Shangxuan | Text Flow : A Unified Text Detection System in Natural Scene Images[END_REF][START_REF] Nguyen | Text-Edge-Box : An Object Proposal Approach for Scene Text Localization[END_REF][START_REF] Lu | Scene text extraction based on edges and support vector regression[END_REF]. Particular, internal distances between characters in a word are smaller than gaps between words and external distances from outer characters to non-text objects. Mean, variance and standard variance of the distances are usually estimated based on large cases of texts in scenes and used as thresholds for grouping and splitting. There is an idea that proposals can be generated from a dendrogram built based on distances, where the closest atomic regions are grouped first and followed by farther ones. In [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF], this idea has been adopted, and the internal distances has been combined with other geographic features.

In this chapter, we propose a solution that can provide a dendrogram based on internal distances between atomic regions without calculating their distances. The idea is inspired by the process of a max-pooling layer in the deep learning network architecture. Two components have been adopted as a pooling window and strides.

The stride what helps to shrink size of feature maps is adopted to shift atomic regions closer to each others. The pooling window is applied to select features for a new feature map and make grouping decisions concurrently. Atomic regions are grouped together if their pixels exist under a given pooling window. An iteration process is also employed to build a dendrogram grouping solution based on which scene text proposals have been generated.

Methodology

The proposed scene text proposal technique is developed following the general object proposal framework including two main steps as a proposal generation and a proposal scoring/ranking. In the proposal generation, texts are first localized base on their edges. We adopted the Canny edge detection method [START_REF] Canny | A Computational Approach to Edge Detection[END_REF] for generating a binary edge map. These edges could be a path of character, a character, a group of characters, a word, or text lines. Edges are then grouped together into proposals by an iterating max pooling process. Bounding box of each group is estimated by covering entire edges in a group. The proposal scoring/ranking strategy is proposed by using histogram of oriented gradients on edge pixels extracted from oriented gradient map as proposals' feature vectors and adopting Euclidean distance function for scoring. Scored proposals are then sorted in the descending order in which the high probability of text proposals are usually in the top of the list. Figure 4.1 depicts the proposed framework on an example image. 1 1 0 2 0 0 0 0 0 1 0 0 2 0 0 0 3 0 0 0 0 2 0 0 3 3 0 1 0 0 2 0 0 0 3 0 3 1 2 0 3 0 2 3 0 1 1 0 2 0 0 0 0 1 0 0 2 0 0 0 3 0 0 0 2 0 0 3 3 a CC searching process. For example, the first found CC is labelled by a number of 1. The number increases after finding a new CC, and the last found CC will be labelled by the highest number. In a labelled feature map, edge pixels in the same CC are assigned the same number as the label of that CC, and non-edge pixels are zeros. Under a given pooling window, only the highest labelled pixel is remained for generating a next labelled edge map, other lower labelled pixels are discarded, including zero-label pixels. Consequently, CCs are gradually shifted to each others.

Max-pooling based scene text proposal generation

1 1 1 0 2 2 2 0 3 3 3 1 1 0 2 1 0 0 2 0 0 0 2 2 0 0 0 0 2 0 0 0 3 2 0 0 3 3
A pooling processes is exploited in multi-iteration. CCs that are neighbours to each other are grouped first and further CCs are merged into groups at later pooling iterations. The iteration process terminates when there is no zero pixel existing in the labelled map, meaning that there is no more gap between CCs. CCs are grouped first and followed by the further CCs. For example, we have a group of CC1-CC2 at the second iteration and then a group of CC2-CC3 at the third iteration when we launch the pooling process on the second row of the labelled edge map. In order to make the max-pooling process work properly, we provide a zero-padding at the right most position when the number of columns is even. Since the max-pooling process finished, found groups are collected and duplicate ones are removed. Based on grouped CCs, bounding boxes are generated, which are the smallest rectangle boxes that cover whole edge pixels in corresponding groups.

Even-though a pooling and stride are in horizontal direction as 1-by-3 and [START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF][START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF] (row x column) respectively, the proposed max-pooling based grouping is able to capture non-horizontal texts that include CCs overlapping a path in the vertical direction. As the synthetic example is shown in Figure 4.3, a curve text (1-2-3-4-5-6) can be localized easily due to their vertical overlapping. CCs which do not contain vertical overlap also can be grouped by overlapping with other CCs proposal regions.

For example, the CC1 and CC4 are grouped together because both CCs are grouped with the CC2. This strategy is only false when texts are totally vertical which is shown in the right synthetic labelled map.

Proposal ranking

Histogram of oriented gradient is a proper feature vector for object detection and classification [START_REF] Wang | Word Spotting in the Wild[END_REF][START_REF] Mishra | Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues[END_REF]. In the proposed ranking strategy, this concept has been adopted with a variance that an orientation histogram is estimated only on edge pixels, instead of whole pixels in the analysing areas. To clarify, we named it as template vectors, the numerator will be larger and the denominator will be smaller.

s = C i_id • k C i_idx -1 D(C i , F ) (4.1) D(C i , F ) is

Maxpooling based scene text proposal implementation

The MPT technique is implemented using Matlab for reading images, generating - --------------------------- ---------------------------- -------------------- ---------------------//An input of the max-pooling grouping method is a labelled map (LabelledM) in which pixel is labelled as an ID of a connected component to which it belongs. on training images of the ICDAR2013 dataset [START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF] and the ICDAR2003 dataset [START_REF] Lucas | ICDAR 2003 Robust Reading Competitions : Entries, Results and Future Directions[END_REF].

Maxdis

The Street View Text (SVT) dataset [START_REF] Wang | Word Spotting in the Wild[END_REF] is not used because its ground truth boxes are not close enough to true text objects for the IoU based recall evaluation, as example images are shown in Figure 3.6. The following subsections discuss these two optimization processes in detail.

Proposal generation optimization

In this optimization process, a limitation on a number of proposals is relaxed.

Whole generated proposals are collected for evaluation. The goal is to find the best combination between size of pooling window and strides that can provide the maximal number of good proposals. Since this process does not need a training phase, all training images in the two ICDAR datasets are utilized as a validation set, containing 479 images. The sizes of a pooling window and strides are varied from 1 to 5 in a row and a column. The pooling window size of 1 × 1 does not provide any grouping, hence this case is ignored. Therefore, there are (24x25=) 600 combinations evaluated. Figure 4.5 presents performance of the proposed proposal techniques in a heat-map, where columns present pooling window sizes, and rows present strides.

Pixel values are average system's recall in three different IoU thresholds : 0.5, 0.7,

Size of pooling windows

Stride values Performance of the technique on training set with a stride of 1-by-2 and a pooling-window of 1-by-3 distance between connected components in a labelled map, which can be captured by the proposed technique. For example, a large pooling window such as 5×5 cannot find differences between distances of two, three, four pixels, while a small pooling window such as 2 × 2 can provide different groups for connected components having distances of one pixel first and later for connected components having distances of two pixels, three pixels and so on. The second is shape of pooling window and strides.

The horizontal direction of both components is more productive than the vertical direction. It could be because most texts are in horizontal direction in causal scenes.

As shown in the performance heat-map, the recalls of the proposed technique are extremely low at the vertical pooling window (the pooling windows have a width of one pixel). At fixed pooling windows width, the proposed technique's performance is usually decreasing following the increase of pooling window heights. These phenomena are similar to stride values. A recall rate is decreasing when a vertical stride turns larger than a horizontal stride and pooling windows are fixed. Furthermore, among horizontal pooling windows, larger-than-one-height windows can increase an ability to handle vertical texts which is a weakness of the proposed technique as mentioned in Section 4.1.1. However, it contemporarily merges text connected components with other non-text connected components in its vertical direction. It is conceivable to note that this capacity will pull recall rates down when there are a lot of non-text connected components around text connected components. It is the reason why one-height pooling windows usually provide the best recall rate at different window width. The third is the overlap between pooling windows in an iteration. The overlap is happen when stride values are smaller than pooling windows at corresponding directions. Usually, at a pooling widow size, the smaller stride provide the better recall rate, meaning that larger overlap between pooling windows provide the higher number of good proposals. In a case of non overlap, recall rates are extremely low in comparison with the best performance.

Proposal ranking optimization

This optimization step assumes that we already have the best proposal list and it tunes a number of dimensions of the HoGe feature vector and a number of templates in each class to search for optimal values that provide the best sorting solution giving the best recall rate under limited number of proposals. The training images are then Selected area

Recall at IoU = 0.5 Recall at IoU = 0.8

Experiments and results

Evaluation set-up

The proposed max-pooling based scene text proposal technique is evaluated and compared to state-of-the-art techniques on the two public scene text datasets including the robust reading competition 2013 (ICDAR2013) [START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF] and the street view text (SVT) [START_REF] Wang | Word Spotting in the Wild[END_REF] which are described in Section 1.6.1. During our evaluation, there is only 2000 proposals selected due to a trade-off between system efficiency and accuracy mentioned in [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF]. According to optimization in Section 4.2, a pooling window size of 1×3 and a stride of 1 and 2 in horizontal-vertical directions are set-up to generate proposals, and a feature dimension of 30 and a number of template of 600 are set-up to score proposals, in the proposed technique.

Comparing with state-of-the-art object proposal methods

Performance of the proposed technique is compared to state-of-the-art scene text proposal techniques, including Text Edge Box (TEB), Simple Selective Search for Text Proposal (TP) [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF], Symmetry Text Line (STL) [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF], DeepText (DT) [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF], and

Weakly Supervised Text Attention Network (WeakCNN) [START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF]. In addition, we also compare to state-of-the-art generic object proposal methods that are EdgeBox (EB) [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF], Geodesic (GOP) [START_REF] Krahenbuhl | Geodesic Object Proposals[END_REF], Randomized Prime (RP) [START_REF] Manen | Prime Object Proposals with Randomized Prim's Algorithm[END_REF], and Multiscale Combination Grouping (MCG) [START_REF] Ponttuset | Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation[END_REF]. Parameters of those comparison techniques are adopted Selective Search for Scene Text Proposal (TP) [START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF], Symmetry Text Line (STL) [START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF],

Deep Text (DT) [START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF], and Weakly Supervised Text Attention Network (WeakCNN) [START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF] ; and generic object proposal techniques : EdgeBoxes (EB) [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF], Geodesic (GOP) [START_REF] Krahenbuhl | Geodesic Object Proposals[END_REF], Randomized Prime (RP) [START_REF] Manen | Prime Object Proposals with Randomized Prim's Algorithm[END_REF], and Multiscale Combination Grouping (MCG)

[93] The most attribution of this impressive achieved performance is a grouping strategy that focuses on popular text appearance in scenes : characters usually stay in a group to form words or sentences, and internal distances between characters are usually smaller than distances between outer characters to other objects round texts.

The proposed grouping strategy also supports linking broken connected components of the same text object. In comparison to another edge based proposal technique as EB [START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF], the two techniques have two similar processes that connected components are searched on an edge map, and bounding boxes are generated based on outer edge pixels of each group. The difference is only a grouping strategy and this difference provides a huge gap performance between two techniques. Due to a simple implementation that does not contain either distance calculation or threshold comparison, the proposal generation executes efficiently. The HoGe based proposal ranking participates to shift good proposals to the top of a sorted list. It is also a significant attribution for the state-of-the-art performance of the proposed technique when a number of proposals is limited.

Conclusion

This chapter presents a max-pooling based scene text proposal technique. The proposed technique is inspired by the CNN max-pooling layer, which is capable of grouping image edges into words and text lines accurately and efficiently. A novel score function is also designed, which is capable of ranking proposals according to their probabilities of being text and accordingly helps to reduce the number of false positive proposals greatly. Further, the proposed proposal technique does not rely on those heuristic thresholds/parameters such as text sizes, inter-character distances, and so on, which are widely used in many existing techniques. Extensive experiments show that the max-pooling based proposal technique achieves superior performance as compared with state-of-the-arts, including techniques developed based on the deep learning framework.

Chapter 5

Automatic scene text reading systems and applications Further, we design an online scene text spotting system aiming to assist people in searching their keywords in given images which are captured or uploaded by their own devices.

A traditional framework of a scene text reading system consists of two main tasks as a scene text detection and a scene text recognition as described in Section 1.3. At least, two machine learning models are needed to handle these two tasks sequentially.

It will consumes a huge processing time and computing resources. In recent years, a hybrid framework is a raising interest, and it is proposed to overcome this weakness [START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF][START_REF] Gomez | TextProposals : a Text-Specific Selective Search Algorithm for Word Spotting in the Wild[END_REF][START_REF] Zhang | Symmetry-Based Text Line Detection in Natural Scenes[END_REF], in which texts have been detected and recognized concurrently. Inspired by this advantage, the hybrid framework has been adopted to develop our scene text reading systems. State-of-the-art scene text recognition models are employed and built on top of scene text proposal techniques to clarify true text proposals from the rests and recognize actual words in proposals.

Designing an automatic scene text reading system is not only to evaluate influence of scene text proposal performances on performances of end-to-end scene text reading systems but also targeted to archive state-of-the-art scene text reading performance in competition with other existing systems. In fact, the proposed system outperforms other recent systems including hands-craft features based [START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF] and deep learning based [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF][START_REF] Li | Reading Car License Plates Using Deep Convolutional Neural Networks and LSTMs[END_REF][START_REF] Busta | Deep TextSpotter : An End-To-End Trainable Scene Text Localization and Recognition Framework[END_REF] on public scene text datasets.

The online scene text searching application utilizes the proposed automatic scene text reading system as a back-end program. The interface between users and the back-end program has been designed using web design language. The application allows users to capture images using integrated cameras on their smart devices as input images and to provide keywords which they want to search on the captured images. In contextualization of searching targeted keywords, the proposed online scene text searching application is compared with the google translate application [START_REF] Google | Google Translation Engine[END_REF]. As comparison results, the proposed application outperforms the google application ; however, its processing time is less efficient than the application of google.

Text Edge Box or Max Pooling

Input image

Scene Text Proposal

Filtered by: Recognition score Lexicon

Proposal Filtering Phase I

Filtered by: Non Maximal Supression

Proposal Filtering Phase II than a threshold or their recognized words are not in the corresponding lexicon lists. In term of unavailable lexicon list, proposals are discarded using a recognition threshold only. In the second phase, the non-maximum suppression algorithm (NMS) is adopted to remove low recognition score proposals when they overlap with the higher recognition score ones larger than the Intersection over Union threshold (IoU).

The smaller IoU threshold will let the proposed system remove the larger number of proposals. However, too small IoU threshold can degrade system performance because some true text proposals also can be eliminated when they overlap with other higher score true text proposals. In the proposed system, the IoU threshold is implemented at 0.1 for removing a vast number of non-text proposals while being able to maintain true text proposals. We also proposed a word-based NMS algorithm as the third phase of the proposal filter process. It considers proposals that overlap each others and contain the same recognized words, and a proposal that has the highest recognized score is maintained. In designing this framework, we also consider to adopt other state-of-the-art scene text recognition models including the CRNN model [START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF], the IrregularTex-tReader [START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF], the CNN Text Attention model [START_REF] Gao | Reading Scene Text with Attention Convolutional Sequence Modeling[END_REF]. These models are developed base on the deep learning framework consisting of several convolution layers for converting input images into feature sequences, and bilateral Recurrent Neural Network (RNN) [START_REF] Yang | Learning to Read Irregular Text with Attention Mechanisms[END_REF] or Long Short Term Memories (LSTM) [START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF] or another CNN model [START_REF] Gao | Reading Scene Text with Attention Convolutional Sequence Modeling[END_REF] for predicting a label distribution of each feature vector. Building on top of these models' architecture is a transcription layer which adopts the Connectionist Temporal Classification (CTC) network to convert sequences of label distributions into predicted words. In comparison with the Jaderber's recognition model, these models have an advantage that they can predict strange words which do not exist in the 90k-word dictionary on which they are trained because they treat an input image as a sequence of characters. However, this ability leads these models to be false in recognizing word due to wrong recognition, miss or false recognition of one or some characters in a word. It could be a reason why they achieve inferior performance to the Jardeberg's model in none-lexicon mode while provides superior performance in lexicon mode, as shown in Table 5.1. Therefore, the Jaderberg's model has been selected as a recognition model of the proposed end-to-end scene text reading system.

Scene text reading system implementation

The automatic scene text reading systems consists of a proposed scene text proposal technique (TEB or MPT) and the state-of-the-art scene text recognition models proposed by Jaderberg [START_REF] Jaderberg | Deep Features for Text Spotting[END_REF]. The system is almost implemented in Matlab, excluding the scene text proposal technique which was already developed in C++.

The pseudo code of the proposed automatic scene text reading system is exposed below. Lexicon is a list of suggested words which are provided along with each image and used to guide the scene text recognition model to predict words.

- ---------------------System : An automatic scene text reading system --------------------- in equation 1.1. The best performance is found around the threshold of 0.7. The increment of system performance in the range of (0.1,0.7) is because that system precision improvement is much faster than system recall degradation. On the other hand, system performance decreases in the range of (0.7, 0.9) due to effect of recall degradation. In the full range of the recognition score threshold, the MPT based scene text reading system always outperforms the TEB based system. Therefore, in comparisons with recent state-of-the-art systems, the MPT based system with the threshold of recognition score of 0.7 is utilized.

Comparison with state-of-the-art systems

In order to analyse the influence of scene text proposal performance on the endto-end scene text reading performance, we provided a series of scene text proposals based systems which are constructed by applying our proposed automatic scene text reading framework on the state-of-the-art scene text proposal techniques including MPT, TEB, TP, STL, EB. Their performances are evaluated by using the ICDAR competition contextualizations including a word spotting evaluation and an endto-end evaluation as discuss in Section 1.6.3. These performances are presented in and scene text reading performances where the better scene text proposal technique usually provides the better automated scene text reading system.

The best proposed automatic scene text reading system as the MPT based system is then compared with state-of-the-art systems including the convLSTM [START_REF] Li | Reading Car License Plates Using Deep Convolutional Neural Networks and LSTMs[END_REF],

the DeepTextSpotter [START_REF] Busta | Deep TextSpotter : An End-To-End Trainable Scene Text Localization and Recognition Framework[END_REF], the TextBoxes [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF] and the Jar_E2E [START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF]. The TextBoxes system is the most state-of-the-art system in the scene text reading competition 2017 Please refer to section 2.2 for more details about these state-of-the-art systems.

As shown performance in two Tables 5.4 and 5.5, the proposed scene text reading system provides competitive performances to state-of-the-art systems, even provides superior performance to CNN based systems as DeepTextSpotter and convLSTM in F-score measurement in two datasets. The TextBoxes is the most competitive system where it provides better performance in almost measurements. However, it provides less recall rate than the proposed system in the weak contextualization scenario of the word spotting evaluation. Note that CNN based systems are trained on a large number of images including 800.000 images in SynthText [START_REF] Gupta | Synthetic Data for Text Localisation in Natural Images[END_REF], 90k word images in the Synthetic text dataset [START_REF] Jaderberg | Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition[END_REF], and training images in the ICDAR2011, the ICDAR2013, and the Street View Text, while the proposed system is only optimized using 479 images in the ICDAR2003 and ICDAR2013 datasets.

Online scene text searching application

Scene texts based applications have been taken in consideration for aiming human life activities as discussions in Section 1.1 due to containing rich semantic context information. The Orcam product is developed to support visual impairment patients in reading books, magazines, menus, and so on. Google and Microsoft translators are very useful for tourists in understanding local or unlearned languages.

Inspired by the usefulness of texts searching tools in localizing keywords in documents, which supports users in finding quickly relevant paragraphs in documents, we are intent on developing a similar text searching tool but it is applied for scene texts instead. It could be useful for quickly pointing out relevant regions in images, from which related information can be extracted and applied for semantic analysis systems such as scene understanding, navigation, and so on. For example, when customers are reading a restaurant menu and looking for specific foods such as chickens, the developed system will capture an image of the menu, search for the word "chicken", then localize where word "chicken" exist. Users then can find other relevant information such as dish names, food recipe as well as its price (Figure 5.4 (a)). For another example, we need a system that can detect dangerous zones and notify users in advance. "Dangerous", "Caution", "Stop", and so on will be relevant keywords which a danger detection system has to search in captured images. This system will be useful for people who need special cares such as dementia patients or elderly (Figure 5.4(b)). The scene text searching system is also useful for navigation which can be used to estimate direction based on detected texts and GPS data (Figure 5.4(c))

In this Section, we will describe our proposed scene text searching system developed base on our proposed automated scene text reading system, which is used as a back-end program. It is evaluated based on images captured by users and compared with the Google translator application in contextualization of spotting keywords in captured scenes. The application can be accessed by a web browser application in any smart devices such as smart phones, tablets, and computers at the link :

dinh.ubismart.org:27790. The developed application can be further integrated in another meaningful application, for example one has been described in Section 6.2.3 (Future Work).

System framework

Nowadays, smart devices such as mobile phones, tablets which have built-in cameras become more and more popular and people is easy to capture their own photos. In addition, internet network is more and more easy to access. Based on this infrastructure, we adopt the server/client framework to deploy our proposed scene text searching application. It is because the back-end program requires a high computing system which mobile device hardware is not satisfied. The framework of our proposed application is shown in using built-in cameras in client devices and keywords are provided by filling in textboxes or speaking to micro-phones. Captured images and keywords are then sent to our server where the back-end program is executed. Detected texts are drawn on captured images and returned to customers. The proposed application is developed base on the web design platform and can be accessed by using web browsers. There is advantages that it becomes independent of operation systems installed on customers' devices such as window, ubuntu, android, iOS, and so on, and customers do not need to install additional packages on their devices.

Web interface

The interface pages of the proposed application is presented in Figure 5.6 including four pages : an introduction page which explains how the proposed application work, a data acquisition page where users process to take a photo and provide keywords, a result page that shows system outputs, and an error page that gives suggestions of errors. In order to provide keywords using speech, web-browsers in customers' devices have to support the google speech recognition system. In fact, they have to use the chrome web-browser with a version over 25 or the safari webbrowser (for devices running iOS). Customers can activate this function by pressing For capturing an image, users can process through the "TAKE YOUR PHOTO" button which support accessing cameras functions in their devices (Figure 5.6b(2)).

For their convenience, image orientation has also been recorded for refining image later. Users do not need to have too much experience in capturing fine images. When they touch on the "SEND" button (Figure 5.6b(3)), all required data including an image, keywords, and an image orientation are packed into a form and sent to server.

Concurrently, web browser switches to the result page and waits for the returned image (Figure 5.6c). If users forget to take a photo, the error page will be shown up to remind them that they need to provide an input image (Figure 5.6d). The "TRY AGAIN" button is provided to help them go back to the data collection page.

If users forget to provide keywords, the proposed application still works properly ; however, it will shown up all detected words in upload images.

Back-end program

The back-end system of the proposed scene text searching application is the proposed automatic scene text reading system which has been built by integrating our proposed scene text proposal technique with an existing scene text recognition system. The RCNN scene text recognition model [START_REF] Shi | An End-to-end Trainable neural network for image-based sequence recognition and its application to scene text recognition[END_REF] which is presented in Section 2.2.2 is adopted for this back-end system instead of the Jaderberg's model [START_REF] Jaderberg | Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition[END_REF]. Despite this change can degrade system performance of finding common words (which are in 90k-word dictionary used to train scene text recognition models), it widens a chance of finding untrained words, especially non English words in Latin-character based languages such as French, German, and so on because it treats word images as character sequences. The conditional probability of a predicted word is considered as a recognition score, and the rest steps of the proposed scene text reading system including a recognition score based proposal filter, a non maximum suppression, and a word based non maximum suppression are maintained as described in Chapter 5

Scene text searching application implementation

The online scene text searching application is implemented in Python and C++ programming languages. It consists of two main parts : a web browser interface and a back-end scene text spotting system. The web browser interface provides users a convenient method to access our application. The flask python package which can interact with html, css and Java-script languages is adopted to design our web pages including a welcome page, a data collection page, a result page, and a fault alert page. This package also handles a submitting action in which users can send to our server their keywords and captured images. The pseudo codes of the web browser interface are presented below. The back-end scene text spotting system is implemented similarly to the proposed automatic scene text reading system, but in Python programming language instead of Matlab language.

- ----------------------------System : Web interface of the online scene text searching application ----------------------------

Evaluation

In this evaluation, the proposed application has been evaluated on a set of images collected from the incidental scene text dataset [START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF], which are captured by using a google glass without focussing on scene text objects. In evaluation, we simulate human behaviour when they are going to search for specific words in natural scenes.

Each image is considered as a captured image taken by users and corresponding keywords are provided. A fraction of keywords which are found successfully is used to evaluate application performance. Following this evaluation scenario, the proposed application is compared with the GoogleTrans application [START_REF] Google | Google Translation Engine[END_REF] which has been developed to read texts in scene images and released onto the Google Play store and the Apple store.

From the incidental scene text dataset [START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF] which contains 1500 images including training images and testing images, 142 images are selected to build a dataset for this evaluation. Selected images contain texts suffering a lot of degradation types such as motion blur, non-horizontal orientation, decoration, perspective, illumination and so on. Contained texts are mostly shops and restaurants name, traffic panels as well as words referring dangerous and cautious situations which are useful for navigation or danger detection applications. 217 keywords have been assigned for this dataset provided by volunteers who look at images and give keywords they are interested In evaluation, images and keywords are provided to our proposed application by using our data collection page. To evaluate the google translation application, images are uploaded to the google scene text reading engine using the image upload option as shown in Figure 5.7 following step 1 and 2 sequentially. For each image, corresponding keywords are decided as successful search if and only if the keywords are exist in detected words provided by this app. lyses scene text proposals sequentially. In addition, computing system is also a reason where the google scene text reading engine executes on a high computing system while the proposed application launches on a personal desktop computer.

Three figures as Figure 5.8, Figure 5.9 and Figure 5.10 present performance of two scene text searching applications on several example images in the evaluation dataset. Figure 5.8 shows images where the proposed application outperforms the GoogleTrans. In contrast, Figure 5.9 shows images where the GoogleTrans performs better than our proposed application.

Conclusion

In this chapter, an automatic scene text reading system has been proposed. It A scene text searching application is also developed by using the proposed automatic scene text reading system as a back-end program. The web based interface has been designed aiming for help users interact conveniently with the back-end program. Due to using web based scheme, our application can be easily accessed by different devices such as smart phones, tablets, computers, in different operation systems (Window, Linux, Android, and iOS). It also integrates with a speech recognition engine developed by Google to provide a convenient data entry solution.

In comparison with a released application as GoogleTrans, the proposed application provides better performances in term of searching for targeted keywords.

YoLo network

Yolo is you-only-look-one. As its name, the model is developed to localize and recognize objects with only one time scanning input images. An unique character of YoLo network among most recent deep learning frameworks such as R-CNN [START_REF] Girshick | Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation[END_REF],

The YoLo model has been trained on the join dataset including the COCO dataset [START_REF] Lin | Microsoft COCO : Common Objects in Context[END_REF] and the ImageNet dataset [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] to classify among 9000 classes. Multiscale training is also implemented for improving the network robustness to object size variant. This flexibility is gained by converting last two fully connected layers in the older YoLo version [START_REF] Redmon | You Only Look Once : Unified, Real-Time Object Detection[END_REF] into two convolution layers. Due to combining the object localization task and the object detection task, a multi-part loss function is applied for training and it is shown in Equation 6.1.

L = λ coord S 2 i=0 B j=0 ¶ obj ij (x i -xi ) 2 + (y i -ŷi ) 2 + λ coord S 2 i=0 B j=0 ¶ obj ij ( √ w i -ŵi ) 2 + ( h i -ĥi ) 2 + λ coord S 2 i=0 B j=0 ¶ obj ij (C i -Ĉi ) 2 + λ noobj S 2 i=0 B j=0 ¶ noobj ij (C i -Ĉi ) 2 + S 2 i=0 ¶ obj i c∈Classes (p i (c) -pi (c)) 2 (6.1)
Where ¶ obj i denotes if object appears in cell i and ¶ obj ij denotes that the jth bounding box predictor in the cell i is "responsible" for that prediction.

There are two weaknesses of the YoLo model which can be considered for future developments. Firstly, the YoLo model struggles in detecting small objects that stay nearby each other and inside a grid cell, such as folks of birds. It is because there is only one object detected in one grid cell. Secondly, a weakness comes from the training process where it treats equally errors in small bounding boxes and in large bounding boxes. In fact, small errors in large bounding boxes are generally benign while small errors in small bounding boxes are great effect on the Intersection over Union (IoU) loss.

YoLo based scene text proposal

In this proposed idea, the YoLo architecture is adopted to generate scene text In evaluation, we also apply a multi-scale input image solution to get a larger bounding boxes as the system post-processes. In [START_REF] He | Accurate Text Localization in Natural Image with Cascaded Convolutional Text Network[END_REF], two deep learning networks are deployed following a cause-to-fine strategy where a cause network provides a text attention map and a fine network processes on cropped text regions to generate a text line heat-map and a text line area heat-map. Two maps are then binarized by threshold of 0.5 and quadrilateral bounding boxes are estimated base on detected regions using the minimum area rectangle estimation method [START_REF] Freeman | Minimum-Area Encasing Rectangle for an Arbitrary Closed Curve[END_REF]. In the other hand, quadrilateral text regions are searched by using quadrilateral anchor boxes and a regression deep learning network [START_REF] Liu | Deep Matching Prior Network : Toward Tighter Multioriented Text Detection[END_REF]. Those strategies can be applied on the developed YoLo based scene text proposal system which is discussed in Section 6.2.1.

The proposed solution consists of a MPT technique based anchor initialization and a modification of the YoLo output layer for estimating quadrilateral coordinators. The model can be trained for regressing MPT proposal boxes into quadrilateral boxes.

A navigation application for aiming elderly in their outdoor activities

The world population is ageing due to life quality improvement with a lot of caring services in healthy and safety. In 2017, there is an estimation of 962 million people aged 60 or above in the world, comprising 13 percent of the global population, and that age population is growing at a rate of about 3 percent per year. By 2050 all regions of the world except Africa are predicted to have nearly a quarter or more of their populations at ages 60 and above. According to age-in-place project [START_REF]Aging in Place : Growing Old at Home[END_REF],

nearly 90% of people over age 65 want to stay at their home as long as possible. They also prefer to show up that they have abilities to stay safe and independent. However, staying alone for elderly, especially people with ageing diseases such as parkinson and dementia is unsafe due to a lot of risks, such as fail in walking, eating, using kitchen tools, and so on. Allowing their family members and caregivers to keep taking care of their activities while giving them a feel of independence will be a hard deal but it is an interesting and high demand mission. In fact, many proposed systems have been investigated, and most of applications are developed for assisting elderly in indoor activities which are known as nursing house projects including fall detection systems [START_REF] Igual | Challenges, issues and trends in fall detection systems[END_REF], tasks reminder applications, and so on [START_REF]The Top Medication Reminder Apps for Patients[END_REF].

In our observation, outdoor activities are also very helpful for the elderly as they aim to reduce elderly cognitive weakness by increasing interactions with social activities and communities. However in an outdoor environment, elderly have to deal with much more dangerous problems such as getting loss, falling in the street, being subject to crimes, being hit by vehicles, and so on. To avoid these incidents, the market demand is high. However, the number of recent applications supporting elderly in their outdoor safety is small. There is only the Global Position System (GPS) based elderly tracker which is deployed to support caregivers and family members in localizing elderly position and tracking their activities base on their movements in maps [104]. In this section, we are going to propose an application scenario based on our developed scene text searching system and the GPS system to assist elderly in their daily outdoor activities.

Proposed application framework

The GPS based navigation system is currently adopted for many tracking systems and the most favourite application is the google map which helps people to track their location and search directions. The elderly tracker system is also developed base on this google map API and supports in tracking elderly locations, arranging safety zone, elderly moving speed. On the other hand, this system does not provide extensive information regarding environment around elderly, which could be very useful for recognizing their situations and providing supporting suggestions. In this future work proposal, we are intent on integrating the GPS tracking module with our developed scene text searching system to enrich capability of our system. Besides an elderly tracking function, the system can support the elderly in finding their direction, navigating their way to their target destinations, searching for specific words, alerting dangerous conditions, and so on. Figure 6.9 depicts an infrastructure of the proposal system, including client wearable devices, which consists of a camera, microphone and a GPS module, and a server in which our back-end program and a database of local areas where elderly is living are executed and stored, respectively.

Elderly with wearable device: GPS, camera, microphone, speaker In order to provide a navigation function, the system needs to know a current location and a target location. The current location can be estimated from the GPS device, and the target location is provided by users using their microphone or keyboard. After receiving required information, additional information related to two locations is queried from the system database such as a local map, local scene texts around two locations. The shortest route is defined in the local map and local scene texts are used for correcting users' direction. Note that directions estimated from the GPS device are also collected. For example, in Figure 6.10 when users' locations are figured out, their local texts are loaded and used as keywords. When one of texts in the keyword list appears in images captured by wearable cameras, user's direction is calibrated so that the system can provide more precise advices.

In a case that target destination contains texts, these texts are also used to guide elderly reach their destinations. objects which can appear in captured images taken by users wearable cameras.

The proposed system also employs a danger alert function. Specific keywords which are usually used to present dangerous situations in modern cities such as "construction in process", "electric shock", "dangerous", "caution", "work ahead", and so on are added to the system keyword list. When those words appear in captured images, a danger alert function is activated and it could notice to users as well as according people to let them take a look on users. Depending on where those words are in images, and word sizes, different levels of alert could be generated. For

1. 1

 1 Some examples of texts in scenes. They can contain (a) semantic information, (b) branch names, (c) shop names, (d) traffic information. 1.2 Examples of applications that have been developed base on scene text reading systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Detecting and recognizing scene texts challenges including intra-class and extra-class variations . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Example images in the ICDAR2013 dataset (the first row) and the Street View Text dataset (the second row) . . . . . . . . . . . . . . . 1.5 Some features of the adopted scene text proposal evaluation framework including : (a) how Intersection over Union (IoU) is calculated, . 2.1 The proposed symmetry template consists of four rectangles with equal size of s × 4s, denoted by RT, RB, RMT and RMB . . . . . . . 2.2 Structure of a scene text proposal generation network (Inception-RPN) in the DeepText network which is developed to detect texts in scene images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Structure of a text attention network which is developed to provide a text confidence score map from an input color image. . . . . . . . . 2.4 A framework of an automatic scene text reading system developed based on scene text proposals and a deep learning based scene text recognition network. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 A schematic of Jaderberg's recognition model which handles a scene text recognition task as a multi-class classification . . . . . . . . . . . xvii xviii LIST OF FIGURES 2.6 Structure of the TextBox network, which inherits the first part of the popular VGG16 network and a proposed Text-Box layer . . . . . . . 35 2.7 A framework of the CRNN scene text recognition model. It consists of three major segments including a convolution neural network, a recurrent neural network and a connectionist temporal classification . 37 2.8 Structure of a Deep Text Spotter network which successfully combines two single tasks of an automatic scene text reading system into one trainable network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.1 A framework of the proposed Text-Edge-Box technique, including two main tasks as a text edge map generation and a grouping-scoringranking proposals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2 The path a and b are respectively an example image and its own binary edge map. The path c shows examples of connected components including . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3 Effectiveness of the two proposed heuristic text specific features on clarifying text edges from others by scoring them high score values. . 48 3.4 Illustration of the proposed connected components grouping strategy. A searching space is estimated from a bounding box . . . . . . . . . . 49 3.5 The detection rate evaluation vs the number of proposals (top row) and the IoU threshold (bottom row) of the Text Edge Box (TEB) . . 56 3.6 There are some examples of the SVT ground truth boxes which our proposals cannot localize with the IoU threshold of 0.7 . . . . . . . . 59 3.7 The performance of the end-to-end word spotting systems which are constructed by the comparison techniques and the word recognition . 60 3.8 Examples from the ICDAR2013 dataset that our algorithm failed to localize. The red boxes are ground truths and the green boxes are our proposal regions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.1 The framework of the proposed scene text proposal technique, including a max-pooling based grouping strategy for . . . . . . . . . . . . 65 LIST OF FIGURES xix 4.2 Illustration of the proposed max-pooling based scene text proposal generation with max-pooling window size of 1-by-3 and . . . . . . . . 4.3 A synthetic example explains how horizontal pooling window and horizontal stride can group non-horizontal texts . . . . . . . . . . . . 4.4 Illustration of the designed scoring function. A proposal is ranked based on the distance between its feature vector . . . . . . . . . . . . 4.5 The heat-map presents performances of the proposed proposal technique on the validation set constructed from training images of the two datasets at different combinations between sizes of pooling window and strides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 The heat-map presents performances of the proposed technique on the evaluation set achieved from the ICDAR2013 and ICDAR2003 dataset on different combinations of a number of templates and a number of HoGe bins . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Comparison of the proposed Max-pooling based scene text proposal technique (MPT) to other state-of-the-art techniques on the IC-DAR2013 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 Performance of the proposed technique MPT and two competitive techniques STL and TP on some images in the two generic scene text datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 A framework of the proposed automatic scene text reading system consists of proposed scene text proposal techniques for searching text locations in scene images and the scene text detection/recognition for eliminating false positive proposals and reading words . . . . . . . . . 5.2 Variants of the proposed scene text reading system under different thresholds of recognition score and scene text proposal techniques. . . 5.3 Several scene text detection and recognition examples where the proposed scene text reading system succeeds (the first three rows) and fails (the last row) . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx LIST OF FIGURES 5.4 Several meaningful applications for a scene text searching application including a searching keywords in menus (a), a danger alarm (b), and a navigation (c). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.5 A framework of the proposed scene text searching application based on a client/server architecture. . . . . . . . . . . . . . . . . . . . . . . 97 5.6 Web interface of the proposed application including four pages : an introduction page, a data acquisition page, a result page and an error page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.7 GoogleTrans interface and a solution to evaluate GoogleTrans performance on our dataset. Step-by-step from 1 to 2 is a path to upload collected images to the google search engine. . . . . . . . . . . . . . . 102 5.8 Several image examples on which the proposed scene text searching application outperforms the GoogleTrans app. . . . . . . . . . . . . . 105 5.9 Example images on which the GoogleTrans app provides better performance than the proposed app. . . . . . . . . . . . . . . . . . . . . 106 5.10 Example images on which both proposed scene text searching app and the GoogleTrans app fail to detect and recognize words. . . . . . 107 6.1 YoLo framework consists of 19 convolution layers. Image space is divided into 49 cells as a square grid of 7 × 7 . . . . . . . . . . . . . . . 112 6.2 Two example images with different scales and according grid sizes. With a size of 96x96, two images in the left will be downscaled in to a grid size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 6.3 Comparison of the baseline of YoLo based scene text proposal techniques (M_YoLoModel, and O_YoLoModel) to other state-of-the-art techniques on the ICDAR2013 dataset . . . . . . . . . . . . . . . . . 117 6.4 Example images where texts are detected by YoLo_sys techniques and eliminated by the recognition model. . . . . . . . . . . . . . . . . 121 6.5 The proposed framework for future development of maxpooling based scene text proposal by adopting a YoLo model for scoring proposals . 122 LIST OF FIGURES xxi 6.6 The proposed framework for future development of YoLo based network by adopting the MPT scene text proposal technique as an anchor box generation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Some examples of non-horizontal texts which have been detected successfully by the MPT scene text proposal technique while decided as miss detection in the IoU based evaluation . . . . . . . . . . . . . . . 6.8 Several methods have been studied and applied to generate quadrilateral bounding boxes for better enclosing to oriented scene texts . . . 6.9 A proposed application framework aiming to help elderly in their outdoor activities, including a direction estimation, a specific address searching, a danger detection, a navigation, and so on. . . . . . . . . 6.10 An example solution for observing users direction using scene text objects which can appear in captured images taken by users wearable cameras. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of tables 3.1 The detection rate (in%) of the proposed technique (TEB) with the variation of the α value from 0 to 1 with an inner step of 0.1, and the difference of the IoU threshold . . . . . . . . . . . . . . . . . . . . . .3.2 Recall (%) and processing time (in second) of the proposed TEB and other state-of-the-art techniques under different IoU thresholds on the ICDAR2013 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. 3 1 1 1. 2 3 1. 3 5 1. 4 12 1. 5 14 1. 6 20 1. 7

 31123354125146207 Recall (%) and processing time (in second) of the proposed TEB and other state-of-the-art techniques under different IoU thresholds on the SVT dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Recall (%) and processing time (in second) of the proposed MPT and other state-of-the-art techniques under different IoU thresholds on the ICDAR2013 dataset. . . . . . . . . . . . . . . . . . . . . . . . 4.2 Recall (%) and processing time (in second) of the proposed MPT and other state-of-the-art techniques under different IoU thresholds on the SVT dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Recognition performance comparison between the Jaderderg's model and other CRNN models on three scene text datasets : SVT, IC03, and IC13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Word spotting performance of the proposed automatic scene text reading system MPT-based and other scene text proposals based systems on Robust Reading Competition 2013 Dataset (ICDAR2013) in the two contextualizations. . . . . . . . . . . . . . . . . . . . . . . . . . . Scene texts and applications . . . . . . . . . . . . . . . . . Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scene text processing systems . . . . . . . . . . . . . . . . Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . Evaluation criteria . . . . . . . . . . . . . . . . . . . . . . . 15 1.6.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.6.2 Evaluation of scene text proposal techniques . . . . . . . 18 1.6.3 Evaluation of automatic scene text reading systems . . . . Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Figure 1 . 1 -

 11 Figure 1.1 -Some examples of texts in scenes. They can contain (a) semantic information, (b) branch names, (c) shop names, (d) traffic information.

Figure 1 . 2 -

 12 Figure 1.2 -Examples of applications have been developed base on scene text reading systems including scene text translators (a,b), a licence plate reader based car tracking system (c), a visual aid camera (d). The applications a and b execute on smart devices as smart phones and tablets, while the applications c and d require specific hardware platforms.

Figure 1 . 3 -

 13 Figure 1.3 -Detecting and recognizing scene texts challenges including intra-class variation due to text fonts, text size, illumination, color, occlusion, and so on (the first five rows) and extra-class variance due to a lot of like-text objects in scenes (the last row)

  same stroke width. The SWT output is a stroke image having the same size with an input image and contains stroke width associated with pixels. Pixels having the same stroke are grouped into characters and then text lines. Text specific heuristic rules are employed for filtering out false positive candidates. Stroke Feature Transform (SFT) [43] is a variant of the SWT. It integrates SWT with two additional cues as a color uniformity and a local relationship of edge pixels during a stroke width search process. In [44], three edge based text specific features are designed to score edge connected components. Non-text connected components are coarsely eliminated by a threshold of 0.5 and remaining connected components are grouped into word candidates. False positive words are filtered out by a trained support vector regression. Stroke End Keypoints (SEK) and Stroke Bend Keypoints (SBK) are

  integrating with the HoG features for generating features sequences and Long Short-Term Memory (LSTM) model for transforming features sequences into labels sequences which are later converted to transcripts. In order to remove hand-craft features as the HoG which fails to deal with extremely blur or fragmented texts, deep feature extraction models have been implemented. Feature maps are produced by applying deep features extraction models either on sliding windows shifting along

Figure 1 . 4 -

 14 Figure 1.4 -Example images in the ICDAR2013 dataset (the first row) and the Street View Text dataset (the second row)

IoU= b. 1 . 5 AFigure 1 . 5 -

 1515 Figure 1.5 -Some features of the adopted scene text proposal evaluation framework including : (a) how Intersection over Union (IoU) is calculated, (b) the difference of one-to-one overlap to other types of overlaps and (c) examples of good and poor proposals with IoU of 0.5.

  rithm for generating atomic regions and the single linkage clustering (SLC) algorithm for grouping generated atomic regions into clusters, which are later referred to proposals. At the first iteration of their grouping strategy, atomic regions are already considered as clusters. These clusters are then grouped together base on their similarity iteratively. Pairs of the most similar clusters are grouped first and pair of the least similar clusters are grouped at the end. In order to measure similarity among clusters, they defines a distance metric d(r a , r b ) which is calculated from seven similarity cues extracted from local pixels belonging to clusters and coordinates of cluster centres. A similarity feature d (i) (r a , r b ) of a pair of two clusters a and b

  high self-similarity to itself and strong contrast to its local background. Symmetry axes are first extracted via the proposed symmetry detector. In order to deal with text size variants, multi-scale input images are implemented. Base on detected text line axes, bounding boxes of text line level proposals are estimated and later split into word level proposals using an internal distance threshold. In this technique, a proposal scoring/ranking step is not implemented. All generated proposals are fed into a word classifier to finalize their scene text detection system. The proposed symmetry detector consists of a features extraction and a random forest based symmetry axis classifier. In the features extraction, a symmetry template is proposed. It consists of four rectangles with equal size of s × 4s, denoted by RT(a top rectangle), RB(a bottom rectangle), RMT(a top middle rectangle), and RMB(a bottom middle rectangle) as shown in Figure 2.1 The height of rectangle "s"

Figure 2 . 1 -

 21 Figure 2.1 -The proposed symmetry template consists of four rectangles with equal size of s × 4s, denoted by RT(a top rectangle), RB(a bottom rectangle), RMT(a top middle rectangle) and RMB(a bottom middle rectangle)

:

  Region proposal network based scene text proposal This is a deep learning based scene text detection network resembling to the Fast R-CNN network [80]. A scene text proposal generation and a scene text proposal scoring/ranking are process concurrently. Scene text proposals are first generated by their developed region proposal network named as Inception-RPN. 300-top proposals are then passed to a text detection network for clarifying text proposals from nontext ones. In this review, the Inception-RPN network is concentrated due to relating to a scene text proposal generation task what we are focussing on. The structure of their scene text proposal generation is depicted in Figure 2.2.

Figure 2 . 2 -

 22 Figure 2.2 -Structure of a scene text proposal generation network (Inception-RPN) in the DeepText network which is developed to detect texts in scene images

  Text regions which have superior confidence scores are segmented coarsely. On each segmented region, the scene text proposal generation framework of the adaptive selective search for scene text proposal technique (TP) [19] is employed to refine segmented regions into scene text proposals. The maximal stable external regions (MSER) is applied to generate MSER regions which are grouped together into scene text proposals by applying the single linkage criterion (SLC) and different complementary distance metrics. Scene text proposals are then scored and ranked by a trained Real Adaboost classifier. In fact, this technique can be considered as an upgraded version of the TP technique since the TP framework is applied on segmented text regions instead of whole image space as the original set-up. The major contribution of this technique is the text attention network which provides text confidence score maps. It is similar to the discriminate localization network [87] with some of modifications : (1) the global average pooling layer is substituted by the spatial pyramid pooling layer (SPP), (2) a new convolution layer is added on top of the Conv5-3 layer of the VGG16 network and named as class activation map convolution (CAM-conv), (3) outputs of the Conv5-3 layer and the CAM-conv layer are concatenated and passed to the SPP layer for predicting probability of binary classification. Structure of the proposed network is shown in Figure 2.3.

Figure 2 . 3 -

 23 Figure 2.3 -Structure of a text attention network which is developed to provide a text confidence score map from an input color image.

Figure 2 . 4 -

 24 Figure 2.4 -A framework of an automatic scene text reading system developed based on scene text proposals generated by EdgeBox [26] and Aggregate Channel Feature Detector (ACF), and a deep learning based scene text recognition network.

(

  ACF). The ACF is a conventional sliding window detector in which proposals are localized by sliding windows and scored by passing its ACF features to an Ada-Boost classifier. ACF features are channel similarity features including a normalized gradient magnitude, a histogram of oriented gradient (in 6 channels) and a raw grayscale input. These features in different input images scales are also taken in an account for scoring proposals. The different scale features approximation solution[START_REF] Dollár | Fast Feature Pyramids for Object Detection[END_REF] is adopted to speed up this extraction process. Thresholding on scored ACF based proposals gives a set of word proposal bounding boxes of this detector. Finally, the generated scene text proposals are formed by combining proposals from the EB and the ACF detector.In the proposal filter step, they are rescored by a random forest classifier applied on their Histogram of oriented gradient features. Proposals having scores falling below a certain threshold are rejected. Bounding boxes of remaining pro-posals are then refined to overlap better with ground truth boxes in the proposal bounding boxes regression step. A deep learning based regression network is developed consisting of four convolution layers ({f ilter_size, number_of _f ilter} : {5, 64}, {5, 128}, {3, 256}, {3, 512}) with a stride of 1 and two fully connected layers (4k units and 4 units). An original bounding box is parametrised by its top-left and bottom-right corners coordinates, such as (x 1 , y 1 , x 2 , y 2 ). The regression model is trained to predict more precise coordinates ((x 1 , y 1 , x 2 , y 2 )) which are tighter to the true text objects and overlap with ground truth boxes in higher IoU values.The detection set still contains a huge number of false positive proposals and they will be eliminated later based on performance of the scene text recognition model.

Figure 2 . 5

 25 It has been trained using Stochastic Gradient Descent (SGD) with Dropout regulation on Synthetic text training data [78]. The final step as a false positive detection elimination is applied using recognition confidence scores (s b ) and predicted words (w b ) of proposal boxes. Non maximum suppression (NMS) is first applied on the same word label proposals to obtain the best proposals at each location. Subsequently, NMS performs to suppress nonmaximal detections of different word label proposals with some overlap. Finally, multiple rounds of bounding box regression using the regression model mentioned

Figure 2 . 5 -

 25 Figure 2.5 -A schematic of Jaderberg's recognition model which handles a scene text recognition task as a multi-class classification

Figure 2 . 6 -

 26 Figure 2.6 -Structure of the TextBox network, which inherits the first part of the popular VGG16 network and associates with the proposed Text-Box layer inspired by the Single Shot multibox Detector network (SSD)

Figure 2

 2 Figure 2.7 -A framework of the CRNN scene text recognition model. It consists of three major segments including a feature extraction based on convolution layers, a character distribution generation based on a recurrent neural network and a word generation based on a connectionist temporal classification.

  developing a region proposal network. The first 18 convolutional layers and five maxpooling layers are inherited with proposed modifications : doubling the number of channels after every pooling steps, adding 1 × 1 filters to compress the representations between 3 × 3 filters, adding a bounding box rotation parameter into predicted proposals. At each point in an output feature map plane, proposal boxes are regressed from 14 different anchor boxes estimated from aggregated training images in the SynthText dataset [82]. Regression function is the logistic activation function presented in Equation 2.13, where, (x, y, w, h, θ) and (r x , r y , r w , r h , r θ ) are an actual bounding box position and a predicted bounding box position, respectively. The c x , c y are offset of the cell in the last convolution layer, and a w , a h are predefined height and width of the anchor box a. The rotation θ ∈ (-π 2 , π 2 ) of a bounding box is predicted by r θ . Text proposals which have confidence scores less than threshold p min = 0.1 are eliminated. Remaining text proposals are then passed to the recognition network.

14 )

 14 In a training process, two networks are first trained separately to achieve their own task and a joined network is then trained on training images of the SynthText dataset, the Synthetic Word dataset, and the ICDAR datasets[START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF][START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF][START_REF] Lucas | ICDAR 2003 Robust Reading Competitions : Entries, Results and Future Directions[END_REF].

Figure 3 .

 3 Figure 3.1 -A framework of the proposed Text-Edge-Box technique, including two main task as a text edge map generation and a grouping-scoring-ranking proposals.In the output image, only good proposals are presented for an illustration purpose

Figure 3 . 2 -

 32 Figure 3.2 -The path a and b are respectively an example image and its own binary edge map. The path c shows examples of connected components including one for text connected component (connected component A) and one for non-text connected component (connected component B). The red arrows are orientations of the considered pixels in the connected components. The dash lines are searching lines corresponding to the orientations of pixels. The pixels in the couple pixels shown in the connected component A are defined as edge pair pixels. Obviously, a text connected component includes much more number of edge pair pixels than a non-text connected component.

Figure 3 . 3 -

 33 Figure 3.3 -Effectiveness of the two proposed heuristic text specific features on clarifying text edges from others by scoring them high score values. By applying threshold on a text edge map, we can perceive that most non-text edges are eliminated.

Figure 3 . 4 -

 34 Figure 3.4 -Illustration of the proposed connected components grouping strategy.

  This section elaborates a strategy to provide a ranked list of proposals in the decreasing priority order. Four measures are defined as S a , S c , S h and S o , which are all normalized in the range of [0,1]. The S a is an averaging scores of CCs within a word-level proposal region, where the score of each included CC is defined in Equation 3.3. The S c , S h and S o indicate affinity among grouped CCs. These measures are designed so that a proposal covering a word will have a high value. In particular, they are calculated from the variance of scores of grouped CCs, the variance of bounding box's height of CCs in a proposal region and the variance of angles between lines linking centroids of two neighbourhood CCs and the horizontal axis, respectively. These angles are adjusted in the range of [0,π].

  -Algorithm : Text Edge Box (TEB)---------------//Import image Img = imread('Image address') ; //Create a binary edge map (E), a gradient map (G), and an oriented gradient map proposal boxes in descending order ppb = boxsort(ppb) ; //Evaluate scene text proposal performance. The gtb is ground truth of a corresponding image. Recall = PPBEval(ppb,gtb) ; //Save evaluated performance save('Recall.mat','Recall') ; A TEB technique is inspired by the EdgeBox technique [26] and two EdgeBox components have been adapted for scene text objects, including ClusteringEdges and BoxesGenerator. They have been rewritten by our implemented functions such as two edge based text specific feature generation (EPCC and EVCC func-tions), a connected component grouping (the Group function), a text line splitting (the BoxesBreakDown function), a bounding boxes generation (the boxlist2Box function). Before applying these two components, 8-connected components are searched on a Canny binary edge map. The ClusteringEdges component focuses on scoring connected components using two proposed text specific features, and generating text edge map. In the generated text edge map, edge pixels of a connected component are assigned the same value which is a connected component score.

  -//Inputs of the ClusteringEdge function are a magnitude of gradient map (G), an oriented gradient map (O), and a list of 8-connected components (CCs). //Score connected component. The EPmap and EVmap contain connected component scores calculated by using EP and EV features, respectively EPmap = EPCC(CCs,O) ; EVmap = EVCC(CCs,G) ; //Generate text edge map (TEM) TEM = alpha × EPmap + (1-alpha) × EVmap ; return(TEM) ;The BoxesGenerator component is divided into three tasks : a connected components grouping, a group splitting, and a bounding box generation. The proposed grouping solution is implemented in the connected component grouping task (the Group function). Its outputs are hypothesized to be text lines and they are then divided into sub-groups targeted to be words (the BoxesBreakDown function).

  [START_REF] Google | Google Translation Engine[END_REF] and other three parameters (τ s , τ w , τ h ) are tuned in the range of [0.5, 1], [0.1, 1], [0.5, 1] with an internal step of 0.1 respectively. All generated proposals were collected for evaluating detection rate. The best set of these values are found as γ = 1.5, τ s = 0.7, τ w = 0.3, τ h = 0.7.In the second optimization step, the parameter α has been estimated to find the best sorting solution to be able to shift good proposals boxes to the front of the list. This parameter controls the contribution of two proposed features (EP and EV) which are reflected into values of S a , S c and S p in the scoring function (Equation 3.4, 3.5). The number of proposals is set to maximum as 2000, the α value varies from 0 and 1 with an internal step of 0.1. The optimal performance on the joined training set under many thresholds of IoU is found at the α = 0.7. When the α is 0 or 1, it means that connected components are scored based on only the EV feature or the EP feature, respectively. As the results presented in Table

Figure 3 .

 3 Figure 3.5 illustrates the performance of the proposed technique as well as the comparison with state-of-the-art techniques. In the left column, the detection rate

Figure 3 . 5 -

 35 Figure 3.5 -The detection rate evaluation vs the number of proposals (left column) and the IoU threshold (right column) of the Text Edge Box (TEB) and other stateof-the-art algorithms, including the simple text specific selective search (TP) [19],

  on the ICDAR 2013 dataset vs the number of proposals on the ICDAR 2013 dataset has been calculated under the three different IoU thresholds, i.e 0.5, 0.7 and 0.8. The TEB algorithm obviously outperforms other methods at the different IoU values when the number of propo-

  proposal regions. Its' performance is therefore very competitive at the small number of proposals, especially with the IoU = 0.5. This is due to the deep learning model that has advantage in recognizing false positive regions and eliminating them from the generated proposal list. However, our proposed system localizes scene texts more successfully when the IoU threshold increases to 0.7 and 0.8, which is targeted for scene text objects as discussed at Section 1.6.2. The TP is the most competitive technique when a huge number of proposals are accepted. The EB shows better result than the TP when the number of proposals is smaller than 1000. On the other hand, its performance deteriorates when the number of proposals increases. The right column shows the second experiment that estimates the detection rate vs the IoU threshold for the different set of proposals : 100, 500 and 1000. The TEB outperforms other methods (excluding the DT) significantly under the different bunch of proposals. When the number of proposals increases and the IoU requirement is more constrained, the proposed TEB performs better than the DT. Note that the DT only provides maximally 500 proposals on an image.

Figure 3 . 6 -

 36 Figure 3.6 -There are some examples of the SVT ground truth boxes which our proposals cannot localize with the IoU threshold of 0.7. The red boxes are the ground truths and the green boxes are our proposals. The proposal boxes are much more smaller and closer to the scene text objects than the ground truth boxes.

Figure 3 . 7 -

 37 Figure 3.7 -The performance of the end-to-end word spotting systems which are constructed by the comparison techniques and the word recognition model [78]. The performance of RegModel here is the result of the word recognition model [78] tested on the ground truth of the testing sets of two scene text datasets as the ICDAR2013 and the SVT

Figure 3 .Figure 3 . 8 -

 338 Figure 3.8 illustrates several typical scenarios where our algorithm often failsto provide good proposals including ultra-low contrast (a.1), complex background (a.2), very small text size (a.3) and uneven illumination (a.2, a.4). In particular, the edges of texts in a complex background are often connected with edges of other objects where the edge pair feature may not be extracted reliably. Similarly, when the text objects are covered by shadow or uneven illumination, the forms of texts

1 65 4. 1 . 2 68 4. 1 . 3 69 4. 2 72 4. 3 76 4. 3 . 1 77 4. 4

 1651268136927237631774 putes word probability based on correlations of connected components in their score, height, and orientation of connections. The effectiveness of the proposed technique has been demonstrated by its superior performance as compared with other stateof-the-art algorithms. Max-pooling based scene text proposal generation . . . . Proposal ranking . . . . . . . . . . . . . . . . . . . . . . . Maxpooling based scene text proposal implementation . . Parameters optimization . . . . . . . . . . . . . . . . . . . Experiments and results . . . . . . . . . . . . . . . . . . . Evaluation set-up . . . . . . . . . . . . . . . . . . . . . . . 76 4.3.2 Comparing with state-of-the-art object proposal methods Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Figure 4 . 1 -

 41 Figure 4.1 -The framework of the proposed scene text proposal technique, including a max-pooling based grouping strategy for scene text proposal generation and a low-level feature based proposal ranking (CCL : Connected Component Labelling, MPG : Max-Pooling based Grouping, FG : Feature Generation, PSR : Proposal Scoring Ranking)

Figure 4 . 2 -

 42 Figure 4.2 -Illustration of the proposed max-pooling based scene text proposal generation with max-pooling window size of 1-by-3 and a stride of 1 in vertical direction and a stride of 2 in horizontal direction on the synthetic labelled map, detail in the second row. Zeros-padding was performed when the number of column is even. Duplicate proposals removed after the pooling process and six proposals have been generated, including three connected components itself and three found groups.

Figure 4 . 3 -

 43 Figure 4.3 -A synthetic example explains how horizontal pooling window and horizontal stride can group non-horizontal texts. The red two-direction arrows are links found in a max-pooling process to group connected components into proposals. This strategy is only false when connected components are not overlapping a part in vertical direction as shown in the right labelled map.

Figure 4 .

 4 Figure 4.2 is a visualization of the max pooling process on a synthetic labelled edge map containing 3 CCs which are labelled by 1, 2, and 3. The zero-label pixels are non edge pixels and they are eliminated gradually after iterations. The closest

  Histogram of Oriented Gradient on edges (HoGe). Proposal scores are estimated based on correlation between their HoGe feature vector and text/nontext HoGe template feature vectors extracted from training sets, referring to Section 4.2 for a detail discussion how the templates were generated. The score function is shown in Equation 4.1.

Figure 4 . 4 -

 44 Figure 4.4 -Illustration of the designed scoring function. A proposal is ranked based on the distance between its feature vector and a set of pre-constructed text and non-text feature templates labelled by 1 (a text class) and 0 (a non-text class). These distances d i are sorted in an increasing order (c) based on which the preconstructed templates are then sorted (d)

binary

  Canny edge maps, evaluating technique performances, and saving evaluated results, and using C++ for generating and scoring scene text proposals. Its main functions include the connected component grouping and the proposal boxes generator. The detail of the MPT class is depicted in Appendix. In the below MPT pseudo codes, Matlab based functions are highlighted in white backgrounds and C++ based functions are highlighted in gray backgrounds.

-

  Algorithm : Max-pooling based scene text proposal (MPT)

  -//Import an image and text/non-text centroids Img = imread('image address') ; Centroids = load('centroids address') ; //Create a binary edge map (E), an oriented gradient map (O) [E,O,-] = CannyEdgeDetector(Img) ; //Generate scene text proposals ppb = MPT(E,O,Centroids) ; //Sort proposal boxes in the descending order ppb = boxsort(ppb) ; //Evaluate scene text proposal performance. The gtb is the ground truth of a cor-

-

  Function : Max-pooling based grouping

4 . 2

 42 TemM having a size calculated from a size of LabelledM and stride va -lues (Hstride, Vstride) Parameters optimization There are four variable parameters in the proposed scene text proposal technique : size of pooling window, and stride values for the proposal generation, as well as a dimension of HoGe feature vectors and a number of templates in each class for the proposal ranking. They are essential parameters and impact significantly on performance of the proposed technique. The optimization process for these four parameters is simplified into two steps. The first is to optimize sizes of a pooling window and stride values to achieve the best recall rate without considering a number of proposals. The second is to optimize a number of dimensions of feature vectors and a number of templates to sort the best proposals to the top of a proposal list. Hence, the proposed technique can achieve the best performance under a limited number of proposals. It means that the first step focuses on providing as much as possible good proposals and the second step focusses on sorting proposals. Both steps are deployed

Figure 4 . 5 -

 45 Figure 4.5 -The heat-map presents performances of the proposed proposal technique on the validation set constructed from training images of the two datasets :ICDAR2003 and ICDAR2013, at different combinations between sizes of pooling window and strides. The limitation on a number of proposals is relaxed in this evaluation. The heat-map shows that the optimal combination is a horizontal pooling window of 1 × 3 and a horizontal stride of[START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF][START_REF] Karatzas | IC-DAR 2013 Robust Reading Competition[END_REF].

  Number of templates in each classNumber of histogram bins (feature vector dimension)

Figure 4 . 6 -

 46 Figure 4.6 -The heat-map presents performances of the proposed technique on the evaluation set achieved from the ICDAR2013 and ICDAR2003 dataset on different combinations of a number of templates and a number of HoGe bins. Its contrast has been enhanced by subtracting pixel values by 0.9 and normalizing into a range of [0,1]. The best performance has been found when a number of templates and a number of HoGe bins are (600, 30), (650, 30), and (550, 40). Their recall rates at different IoU thresholds are presented in detail in the selected area.

Figure 4 .

 4 Figure 4.6 shows a heat-map which is generated by system performances at these different combinations. Note that the heat-map has been enhanced contrast for better visualization by subtracting pixel values by 0.9 and normalizing into a range of [0,1]. Each pixel is an average performance of the proposed technique at three difference IoU thresholds. The whiter pixels present the better performances, and they are found at three combinations (600, 30), (650, 30), and (550, 40). The first number is a number of templates and the second number is a dimension of feature vectors. The combination of (600, 30) is selected to implement in the proposed technique due to the smallest number of computations.

  from their published codes and papers. Most of compared techniques is programmed in Matlab, excluding TP and STL which are implemented in C++. All techniques are executed in a HP workstation computer with Intel Xeon 3.5GHzx12 CPU, and 32 GB Ram.

Figure 4 .

 4 Figure 4.7 shows comparisons in the variations of the number of proposals at different IoU threshold at 0.5, 0.7 and 0.8 (the left-column graphs) and the variations of an IoU threshold at a limitation on a number of proposals at 100, 500, and 1000(the right-column graphs) on the ICDAR2013 dataset. In the left-column graphs, the proposed technique outperforms other techniques, excluding DT at IoU of 0.5, which was built on a deep learning frame work. In the right-column graphs, the DT technique is a competitive technique when the IoU threshold is in a range of (0.5, 0.6). However, its performance drops dramatically following an increasing IoU threshold. The TP, STL, and WeakCNN are more stable than the DT because of their included hand-craft text specific features. Nevertheless, their performances are lower than the proposed technique until the IoU threshold of 0.9, which is unusually used in real applications. The MPT also outperforms our previous technique as the TEB in different cases of number of proposals and IoU thresholds. Note that the DT technique only provides maximally 500 proposals per image. In those graphs, there is no generic object proposal technique exceeding performances of the proposed MPT technique.We also evaluate and compare scene text proposal techniques based on an average number of proposals and processing time on the two testing datasets. Those results

Figure 4 . 7 -

 47 Figure 4.7 -Comparison of the proposed Max-pooling based scene text proposal technique (MPT) to other state-of-the-art techniques on the ICDAR2013 dataset [2], in difference of a number of proposals and IoU thresholds. Comparable techniques include both scene tex proposal techniques : Text Edge Boxes (TEB), Simple

  comparable and processes slower than the proposed technique. The EB is the most efficient technique that need around a second to process an image. The DT and MCG provide the smallest number of proposals as around 500. Nevertheless, their performances are much lower than the MPT's performance.Performances of the proposed technique and two other state-of-the-art scene text proposals on some example images from the SVT and ICDAR2013 datasets have been illustrated in Figure4.8. In various challenges of text distortions such as text size variation (the first column), certain image blurs (the third columnthe fourth column), uneven illumination (the second column), and non-horizontal orientation (the fifth column), the proposed technique always handles better than other competitive techniques. It may fails when text objects are ultra-low contrast, strong blur as showed in the right most image. These texts could be lost due to a edge detector capacities. Indexes of good proposals, which overlap with corresponding ground truth boxes over 80%, in the sorted proposal list are tracked and presented

Figure 4 . 8 -

 48 Figure 4.8 -Performance of the proposed technique MPT and two competitive techniques STL and TP on some images in the two generic scene text datasets. The numbers under each image are the smallest orders where good proposals are found in the list respecting to each ground truth. The good proposals are proposals that overlap with ground truth boxes over 80%. Noted that there are many good proposals respecting to a ground truth in the proposal list. However, only the top proposal respecting to each ground truth box in each image is shown because of illustration purpose.

Contents 5 . 1 5 . 2

 5152 Text reading system . . . . . . . . . . . . . . . . . . . . . . 5.1.1 System framework . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Scene text reading system implementation . . . . . . . . . 5.1.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . Text searching app . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 System framework . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Scene text searching application implementation . . . . . 5.2.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . In the two preceding chapters, we proposed two state-of-the-art scene text proposal techniques including the Text Edge Box (TEB) and the Max Pooling Text (MPT), which are able to localize texts appearing in different fonts, sizes, colors and under different distortions due to environment conditions such as uneven light, occlusion, perspective, and so on. In this chapter, we are going to develop end-to-end scene text reading systems based on those proposed scene text proposal techniques.

Figure 5 . 1 - 5 . 1 Automatic scene text reading system 5 . 1 . 1

 5151511 Figure 5.1 -A framework of the proposed automatic scene text reading system consists of proposed scene text proposal techniques for searching text locations in scene images and the scene text detection/recognition for eliminating false positive proposals and reading actual words in proposals.

Figure 5 . 2 -

 52 Figure 5.2 -Variants of the proposed scene text reading system under different thresholds of recognition score and scene text proposal techniques. System performances are evaluated on the ICDAR2013 dataset. The best variant is the system developed based on max-pooling scene text proposal (MPT-based) at the threshold of 0.7.

Tables 5 .

 5 2 and 5.3 for a word spotting systems and an end-to-end systems respectively. In this comparison, scene text proposal based systems are named following the template [Scene Text Proposal method]_Sys.

(

  http://rrc.cvc.uab.es/ ?ch=2&com=evaluation&task=4). It is a cascade of two deep learning models : one is a scene text localization model inspired by the SSD network [51] and one is a scene text recognition model developed by combining a convolution neural network, a long-short term memory model and a connectionist temporal classification network. The convLSTM system applies a CNN based character recognition model on multi-scale input images to generate character saliency maps based on which licence plates are detected and its' bounding boxes are refined based on edge information. Characters in detected plates are finally recognized by a CNN model consists of CNN layers for feature extraction and LSTM layers for sequence recognition. The DeepTextSpotter is a trainable scene text reading system which combines a scene text detection and a scene text recognition into a single CNN network. The Jar_E2E system has been developed following the traditional scheme that includes a scene text detection task and a scene text recognition task. The scene

Figure 5 .Figure 5 . 3 -Figure 5 . 4 -

 55354 Figure 5.3 shows a number of sample images that illustrate the performance of our developed end-to-end scene text reading system. As Figure 5.3 shows, the proposed technique is capable of detecting and recognizing challenging texts with

Figure 5 . 5 .Figure 5 . 5 -

 5555 Figure 5.5 -A framework of the proposed scene text searching application based on a client/server architecture. Clients can use a web browser application installed in their smart devices to access our server for sending images and keywords and receiving results. The proposed automatic scene text reading system is implemented on a server as a back-end program to process clients' requests.

Figure 5 . 6 -

 56 Figure 5.6 -Web interface of the proposed application including four pages : an introduction page, a data collection page, a result page and an error page. Button 1.a is to activate the speech recognition function. Text box 1 is to fill keywords or to correct recognized keywords from the speech recognition function. Button 2 and 3are to take a photo and send request to server. The error page is used to notify users that they forgot to select a photo before sending and the button "TRY AGAIN" helps them to navigate back to the data collection page.

  -//Call a welcome page. It is executed when users access the application link def welcome() : return render_template('Welcome.html') //Call a data collected page. It is executed when users press a button in the welcome // page def Demo() : return render_template('Demo.html') //Call a data collected page. It is executed when users press a button in the fault //alert page def Comeback() : return render_template('Demo.html') //Call a uploading task. It is executed when users press the button "SEND" in the // data collection page def upload() : try : //'ImageOrientation' is a parameter generated by the data collection page ImageOrientation = request.form('ImageOrientation') //Write the image orientation to a txt file which later is accessed by the // back-end program txtWrite(ImageOrientation,'Orientation.txt') //'Keywords' is a parameter generated by the data collection page contai // -ning keywords provided by users keywords = request.form('Keywords') //Write the keywords to a txt file which later is accessed by the back-end // program txtWrite(Keywords,'Keywords.txt') //Get an uploaded image Images = request.files.getlist('file') : for I in Images : //Save images to files. They are later accessed by the back-end pro //gram I.save('File name') //Call a result page in which the back-end program outputs will be shown return render_template('Result.html') except : //Call a fault alert page giving a suggestion that users forgot to provide //an image return render_template('Error.html')

1 2Figure 5 . 7 -

 157 Figure 5.7 -GoogleTrans interface and a solution to evaluate GoogleTrans performance on our dataset. Step-by-step from 1 to 2 is a path to upload collected images to the google search engine.

Figure 5 .

 5 [START_REF] Greenhalgh | Recognizing Text-Based Traffic Signs[END_REF] illustrates images where both applications fail to detect and recognize keywords. In these figures, words presented under images are keywords corresponding to images. As performance shown in Figure 5.8, the proposed application performs better than the GoogleTrans on texts degraded by illumination (image (a) and (e)), non-horizontal texts (image (b)), blur and decorated texts (image (c) and (d)). In contrast, the GoogleTrans is more reliable than the proposed application when scene texts are printed in standard fonts as images in Figure5.9, even ultra-low contrast texts as texts in Figure5.9(f). Note that two scene text searching applications are totally false when texts are printed using art fonts (Figure 5.10(c,e)), under affected by both illumination and perspective (Figure 5.10(b,d,f)). These weaknesses are still open challenges for future developments.

  has been developed based on our proposed scene text proposal techniques and the Jaderberg's scene text recognition model. Due to robustness of the two models, the developed scene text reading system can detect and recognize texts under different degradations. In comparison to state-of-the-art systems evaluated on two public scene text datasets, it provides competitive performances in both an end-to-end and a word-spotting contextualization evaluation.

Figure 5 . 8 -

 58 Figure 5.8 -Several image examples on which the proposed scene text searching application outperforms the GoogleTrans app. Detected boxes are drawn on images, the green recognized words are from our proposed app and the yellow recognized words are from the GoogleTrans app. The white boxes describe from where images are cropped.

Figure 5 . 9 -

 59 Figure 5.9 -Example images on which the GoogleTrans app provides better performance than the proposed app. The green recognized words are generated by the proposed app and the yellow recognized words are generated by the GoogleTrans.The white boxes describe from where images are cropped.

Figure 5 . 10 -

 510 Figure 5.10 -Example images on which both proposed scene text searching app and GoogleTrans app fail to detect and recognize words. Those words are degraded uneven illumination, decoration purpose, and perspective. The white boxes describe from where images are cropped.

  proposals. Three issues are addressed including a training process, model modifi-cation, and multi-scale prediction. Training process is to adjust pre-trained YoLo weights from general objects sensitivity to scene text objects sensitivity. Model modification focuses on changing the output layer for providing a larger number of proposal boxes and being suitable for two classes classification as text/non-text instead of 9418 classes. Multi-scale prediction is implemented to support a model in handling text size variants. Network training process is implemented using the Darknet library developed by authors of YoLo network, published at the link : https://pjreddie.com/darknet/ Due to the output layer of YoLo network is designed to predict three bounding boxes at each cell, the original YoLo network provides maximally 147(=7x7x3) bounding boxes because some of generated boxes are eliminated by a confidence score threshold as 0.5 and a non maximal suppression with IoU of 0.4. In order to adapt this layer to generate scene text proposals, two modifications have been addressed. First, we change the number of classes from 9418 to 2 referring two classes as text and non-text. Second, a number of predicted boxes at each grid cell is increased from 3 to n and it is set at 10 in the baseline model implementation. Therefore, the third dimension of the output layer is ((5+2)*10 = )70 parameters. A confidence score threshold and a non maximum suppression IoU threshold are also changed, which are decreased from 0.5 to 0.1, and increased from 0.4 to 0.95 respectively. In the training process, the developed model is first trained on a join dataset consisting of training images in the ICDAR2013 dataset [2] and training images in the Synthetic data [82]. Then, it is fine tuned again on the ICDAR2013 training images. In order to enrich training datasets, which is very useful for training a deep learning network, training images are also modified by adding noise, random cropping, flipping, and rotating before forwarding to the network. Resolution of input images is also varied to improve model's robustness to different text sizes. However, the input image size has to be a multiple of 32. Training hyper-parameters are adopted from the YoLo training configuration, including batch size of 1, momentum of 0.9, weight decay of 0.0005, initial learning rate of 0.01, learning rate decay of 0.1 after constant epochs, maximum number of epochs of 10 8 .

Figure 6 . 2 -

 62 Figure 6.2 -Two example images with different scales and according grid sizes. With a size of 96 × 96, two images in the left will be downscaled into a grid size of 3 × 3 after going through the YoLo network. Similarly, two images in the right are downscaled to a grid size of 5 × 5. Square grids are presented with red lines.For a grid size of 3 × 3, objects in two images are located inside a grid and can not be detected separately. In contrast, a grid size of 5 × 5 can split them into single detection.

Figure 6 .Figure 6 . 3 -

 663 Figure 6.3 presents performances of two versions of Yolo based scene text proposals and three state-of-the-art techniques including the Text-Edge-Box (TEB), the Maxpooling base Text Proposal (MPT) and the DeepText (DT) [21], using the scene text proposal evaluation framework which considers a system recall rate under different number of proposals and different IoU thresholds. As results shown in Figure 6.3, these YoLo based techniques outperform state-of-the-art techniques when a number of proposals and an IoU threshold are small. In the left column, performances of YoLo based techniques improve dramatically when the number of proposals increases within a range of (0,300) at IoU of 0.5 and 0.7. When IoU is 0.8, state-of-the-art techniques, which are TEB and MPT, provide superior performances in a full range of a number of proposals. In the right column, recall rates are evaluated under difference of IoU at three numbers of proposals as 100, 500, 1000. YoLo based techniques usually provide the best performance when IoU thresholds

  sed techniques are not close to text edges which are usually considered by human during labelling ground truth boxes. Yolo proposal boxes are therefore fine match with ground truth boxes in the SVT dataset where human labels are not strongly following text edges, and it fails to outperform state-of-the-art techniques in the ICDAR2013 dataset where human labels are required to be as close as possible to text edge pixels. YoLo proposal boxes are also more precise than DeepText proposal boxes because YoLo proposal boxes are estimated based on entire image contents which combines both content features inside a studied cell and content features in other neighbour cells. In contrast, DeepText boxes are predicted base on content features inside a studied cell only. The effects of YoLo based scene text proposals to scene text reading system performance are also evaluated by embedding them into our propose scene text reading framework. Proposal boxes are then forwarded to the scene text recognition model for recognition and classification. False positive proposals are eliminated by a recognition score threshold, lexicons, and non maximum suppression. These systems performances are then compared with other developed systems, including TEB_based system and MPT_based system which achieve state-of-the-art performances in the robust reading competition in 2017. These results are shown in two Tables 6.3 and 6.4. Even though YoLo based scene text proposal techniques process

Figure 6 . 5 -Figure 6 . 7 -Figure 6 . 8 -

 656768 Figure 6.5 -The proposed framework for future development of maxpooling based scene text proposal by adopting a YoLo model for scoring proposals. CCL is connected component labelling and MPG is maxpooling based grouping.

Figure 6 . 9 -

 69 Figure 6.9 -A proposed application framework aiming to help elderly in their outdoor activities, including a direction estimation, a specific address searching, a danger detection, a navigation, and so on. The system also connects to related people of elderly such as their nurses, caregivers, family members

3 His direction: 4 5 7 Figure 6 . 10 -

 3457610 Figure 6.10 -An example solution for observing users direction using scene text

An automatic scene text reading system

	name M P T _sys.
	have proposed two state-of-the-art scene text proposal techniques that pro-
	vide high-quality text proposals. They are developed from binary edge maps, gra-
	dient maps and oriented gradient maps generated by the Canny edge detector. In the
	scene text proposal generation step, two edge based features are determined to cla-
	rify text edges from other edges in binary edge maps. Two grouping algorithms are
	proposed to merge effectively connected components into proposals. The first algo-
	rithm is developed from text specific heuristic relationship rules between connected
	components. The second algorithm provides a natural dendrogram grouping based
	on internal distances between connected components, which is inspired by a process
	of a max-pooling layer. The proposal scoring/ranking problem is also addressed.
	Two proposed scoring functions are developed from edge based features. It supports
	the proposed techniques in providing state-of-the-art performances while a number
	of proposals is limited.
	Our automatic scene text reading system has been developed by combining our
	scene text proposal techniques and existing scene text recognition models. False
	positive proposals are eliminated based on a threshold of recognition confidence
	score, non maximal suppression and provided lexicon lists. Evaluated by the evalua-
	tion framework published in the ICDAR competition, the proposed system out-
	performs other scene text proposal based systems and be competitive to other
	state-of-the-art scene text reading systems, including systems are developed based
	on deep learning frameworks, observed by its performances in the ICDAR compe-
	tition website (http://rrc.cvc.uab.es/ ?ch=2&com=evaluation&task=4) under the

ICDAR dataset and the Street View Text dataset : These two

	datasets are widely used in a community working on developing scene text analysis
	systems and they are utilized in this research evaluation. The ICDAR is the dataset
	provided by the International Conference on Document Analysis and Recognition
	(ICDAR) when they launch their robust reading competitions. This dataset was first
	announced in 2003 and rearranged in 2011, 2013, and 2015 [81, 2]. Another dataset
	is the Street View Text dataset (SVT) [3] which contains images from the google

Chapter 2 State-of-the-art scene text proposal techniques and scene text reading systems Contents 2.1 Scene text proposal techniques . . . . . . . . . . . . . . . 24

	sed text specific edge features which are used for clarifying text edge pixels from
	non-text edge pixels and scoring scene text proposals. These two features are major
	contributions of the proposed technique named as Text-Edge-Box (TEB). Chapter
	4 concentrates on grouping-edges solutions which are used to merge single edges into
	proposals. The unique of this proposed grouping solution is that it does not rely on
	any text heuristic rules which are widely utilized by traditional scene text detection
	techniques. The grouping solution is inspired by a max-pooling process in the deep
	learning framework, and the technique proposed based on this grouping solution
	is named as max-pooling based scene text proposal (MPT). An automatic scene
	text reading system described in Chapter 5 is constructed by applying scene text
	recognition models on top of the proposed scene text proposal techniques for both
	a false positive proposals elimination and a scene text recognition. The proposed
	system provides state-of-the-art performances on the robust reading competition.
	Based on success of this system, a scene text searching application has been imple-
	mented and its performance is competitive to the commercialized product known
	as Google Translator in term of searching for specific words in scene images. In this
	thesis, we also go through state-of-the-art research works in developing scene text
	proposal techniques and automatic scene text reading systems. They are described 2.1.1 An adaptive selective search . . . . . . . . . . . . . . . . . 24
	and discussed in Chapter 2. Future works are discussed in Chapter 6. It includes an 2.1.2 Symmetry text line . . . . . . . . . . . . . . . . . . . . . . 26
	idea to employ the YoLo network for scene text proposal generation and an idea to 2.1.3 DeepTexts . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
	generate quadrilateral bounding boxes for detected text regions which is a require-2.1.4 Weakly supervised text attention network . . . . . . . . . 31
	ment for system outputs in the recent scene text datasets as the CoCo text dataset 2.1.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
	and the incidental scene text dataset. A relevant application aiming to support vi-
	sual impairment patients and elderly in their outdoor activities is also described in
	this Chapter.
	This thesis goes through our contributions in developing scene text processing
	systems including scene text proposal techniques, an automatic scene text reading
	system and a scene text searching application. Chapter 3 focuses on two propo-

2.2 Automatic scene text reading systems . . . . . . . . . . . 32

	2.2.1 Edgeboxes based scene text reading system . . . . . . . . 32
	2.2.2 TextBox . . . . . . . . . . . . . . . . . . . .

. . . . . . . . 35 2.2.3 DeepTextSpotter . . . . . . . . . . . . . . . . . . . . . . . 39 2.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

  Figure 2.8 -Structure of a Deep Text Spotter network which successfully combines two single tasks of an automatic scene text reading system as a scene text detection and a scene text recognition into one trainable network

	Source Image	Fully Connectional Net	Region Proposals Region Scores	Best Proposals	Geometric Normalization	CTC	Region Transcription
			98
			0.66
			0.04
			0.07
			.
			.
			.
			0.22

Chapter 3 Scene text proposal based on heuristic text specific features Contents 3

.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.1.1 Specific text edge features . . . . . . . . . . . . . . . . . . 46 3.1.2 Scene text proposal generation . . . . . . . . . . . . . . . 48 3.1.3 Scene text proposal ranking . . . . . . . . . . . . . . . . . 49 3.1.4 TextEdgeBox implementation . . . . . . . . . . . . . . . . 51 3.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.2.1 Parameters tuning . . . . . . . . . . . . . . . . . . . . . . 54 3.2.2 Experimental results . . . . . . . . . . . . . . . . . . . . . 55 3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

Object proposal techniques well perform in localizing non-class objects. However, they fail to deal with scene texts. It is because scene texts have more diverse appearance than general objects due to not only texts' intra-class variations but also texts' distortions under environment impacts as discussed in section 1.2. There is only the EdgeBox

[START_REF] Zitnick | Edge Boxes : Locating Object Proposals from Edges[END_REF]

technique (EB) applicable for a scene text localization as described in the automatic scene text reading framework

[START_REF] Jaderberg | Reading Text in the Wild with Convolutional Neural Networks[END_REF]

. However, the EB has to integrate with another scene text proposal technique consisting of a Aggregate Channel Features (ACF) and an AdaBoost classifier to provide a proper set of pro-posals, which achieves a recall rate at 96% with 10000 proposals in the ICDAR2003

  .[START_REF] Shi | ICDAR2017 Competition on Reading Chinese Text in the Wild (RCTW-17)[END_REF] shows, starting with a given CC (called candidate A -a dark blue box), three properties of its bounding box (bb A ) are exploited including box height (h A ), box width (w A ) and box size (s A ). A corresponding search area is designed by expanding the bb A , where the search area width (w search ) is equal to the image width, and the search area height (h search ) is γ times bigger than the h A determined by expanding the h A equally in both sides in the vertical direction (a red box). CC

	Starting CC
	Given CC
	Text Edge Map	Searching space
		Proposal Searching Process

candidate B (a green box and its properties are bb B , w B , h B , and s B ) is merged with the CC candidate A to form a group if the bb B satisfies :

[START_REF] Karatzas | ICDAR 2015 Competition on Robust Reading[END_REF]

The ratio of intersection between the bb B and the A's search space to the s B is higher than τ s , (2) the ratio

Table 3 .

 3 IoU = 0.5 70.[START_REF] Huang | Text Localization in Natural Images using Stroke Feature Transform and Text Covariance Descriptors[END_REF] 87.34 90.51 93.13 93.94 94.21 94.39 94.76 94.67 94.58 94.14 IoU = 0.7 55.61 72.69 75.95 81.74 82.73 83 82.91 83.09 82.82 82.64 82.82 IoU = 0.8 47.03 63.56 66.27 72.6 73.87 73.96 73.87 73.96 73.87 73.69 73.6

	α	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1

1 -The detection rate (in%) of the proposed technique with the variation of the α value from 0 to 1 with an inner step of 0.1, and the difference of the IoU threshold on the joined training sets of the two scene text datasets : ICDAR2013 and SVT. The maximum number of proposal regions is 2000.

Table 3 .

 3 2 -Recall (%) and processing time (in second) of the proposed TEB and other state-of-the-art techniques under different IoU thresholds on the ICDAR2013 dataset. The Nppb denotes an average number of proposals that the techniques need to achieve its presented recalls.

	IoU	0.5	0.7	0.8	Nppb times (s)
	TEB [25] 94.25 87.95 81.55 1777	5.4
	TP [19]	84.47 71.32 65.11 1907	5.17
	STL [24] 79.78 62.04 49.91 1034	361.3
	DT [21]	88.5	67	4	500	-
	EB [26]	76.93 48.81 27.67 1968	1.02
	GOP [91] 45.68 19.39 11.76 1040	4.3
	RP [92]	66.91 36.95	19.3	1917	10.07
	MCG [93] 56.16 36.21 27.02	550	28.92

sals is larger than 1000. The DT leverages on the deep learning model for scoring

Table 3 .

 3 3 -Recall (%) and processing time (in second) of the proposed TEB and other state-of-the-art techniques under different IoU thresholds on the SVT dataset.

	The Nppb denotes an average number of proposals that the techniques need to
	achieve its presented recalls.
	IoU	0.5	0.7	0.8	Nppb times (s)
	TEB [25] 87.64 47.91 20.09 1890	2.2
	TP [19]	74.65	42.5	21.33 1972	5.94
	STL [24] 77.13 31.07 10.36 1358	433.82
	EB [26]	76.35 47.45 23.96 2000	1.28
	GOP [91] 52.09 18.24	6.8	1117	3.78
	RP [92]	62.6	27.05 12.21 2000	8.02
	MCG [93] 54.71 24.27	8.66	557	14.97

Scene Text Proposals CCL MPG FG PSR Input Image Output Image

	Binary Edge Map	Feature Map
	Canny Edge Detector	Offline Learned Feature Templates
	Oriented Gradient Map	HoG Feature Vectors of Proposals	Ranked List of Proposals

Table 4 .

 4 1 -Recall (%) and processing time (in second) of the proposed MPT and other state-of-the-art techniques under different IoU thresholds on the ICDAR2013 dataset. The Nppb denotes an average number of proposals that the techniques need to achieve its presented recalls.

	IoU	0.5	0.7	0.8 Nppb times (s)
	MPT	96.16 88.77 82.1 1450	4.49
	TEB [25]	94.25 87.95 81.55 1777	5.4
	TP [19]	84.47 71.32 65.11 1907	5.17
	STL [24]	79.78 62.04 49.91 1034	361.3
	DT [21]	88.5	67	4	500	-
	WeakCNN [31]	90	86.5	77	1000	-
	EB [26]	76.93 48.81 27.67 1968	1.02
	GOP [91]	45.68 19.39 11.76 1040	4.3
	RP [92]	66.91 36.95 19.3 1917	10.07
	MCG [93]	56.16 36.21 27.02 550	28.92

are shown in Tables 4.1 and 4.2. The proposed MPT technique always achieves the best recall rate in most cases of IoU threshold on the two datasets. The TP is very competitive technique ; however, it requests a larger number of proposals to be

Table 4 .

 4 2 -Recall (%) and processing time (in second) of the proposed MPT and other state-of-the-art techniques under different IoU thresholds on the SVT dataset.

	The Nppb denotes an average number of proposals that the techniques need to
	achieve its presented recalls.
	IoU	0.5	0.7	0.8	Nppb times (s)
	MPT	88.41 46.99 21.48 1769	3.59
	TEB [25] 87.64 47.91 20.09 1890	2.2
	TP [19]	74.65	42.5	21.33 1972	5.94
	STL [24] 77.13 31.07 10.36 1358	433.82
	EB [26]	76.35 47.45 23.96 2000	1.28
	GOP [91] 52.09 18.24	6.8	1117	3.78
	RP [92]	62.6	27.05 12.21 2000	8.02
	MCG [93] 54.71 24.27	8.66	557	14.97

under each image. Note that there are many good proposals for a given ground truth box, and only the smallest index proposals are presented. In comparison with other techniques, the proposed technique usually sorts good proposals at the smallest order.

Table 5 .

 5

	1 -Recognition performance comparison between the Jaderderg's model
	and other deep learning based models [62, 64, 65] on three scene text datasets :
	SVT, IC03, and IC13 in different modes including a lexicon mode and a non-lexicon
	mode (xx-None). This comparison shows that the Jaderberg's model outperforms
	other models in the non-lexicon mode where recognized words are not corrected by
	lexicons.
	Model	SVT-50 SVT-None IC03-50 IC03-50k IC03-None IC13-None
	Jar's model [66]	95.4	80.7	98.7	93.3	93.1	90.8
	CRNN model [62]	96.4	80.7	98.7	95.5	89.4	86.7
	TextAttCNN[65]	97.4	82.7	98.7	96.7	89.2	88
	IrrTReader[64]	95.2	-	97.7	-	-	-

System performance vs threshold of recognition score TEB MPT Threshold of recognition score F-measurement

	90 100		84.04	87.43	89.21	89.79	89.7	89.43	88.66
	70 80	75.05	79.71	83.57	85.47	86.75	87.12	86.81	85.97
	60	70.39
	50
	40
	30
	20
	10
	0
	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9

-//Inputs of the automatic scene text reading system are an image (Img) and a lexicon list (Lexicon) when it is available.

Table 5 .

 5 2 -Word spotting performance of the proposed automatic scene text reading system MPT-based and other scene text proposals based systems on Robust Reading Competition 2013 Dataset (ICDAR2013) in the two contextualizations.

		Strong contextualization	Weak contextualization
		Recall Precision F-score Recall Precision F-score
	EB_sys	55.26	66.81	60.49	54.79	57.48	56.10
	STL_sys	61.8	85.32	71.68	61.45	81.30	69.99
	TP_sys	66.47	89.47	79.27	65.07	82.40	72.72
	TEB_sys 84.46	96.4	90.03	82.94	91.96	87.22
	MPT_sys 88.20	97.42	92.58 87.85	95.31	91.43

Table 5 .

 5

	3 -End-to-End performance of the proposed automatic scene text rea-
	ding system MPT_based and other scene text proposals based systems on Robust
	Reading Competition 2013 Dataset (ICDAR2013) in the two contextualizations.
		Strong contextualization	Weak contextualization
		Recall Precision F-score Recall Precision F-score
	EB_sys	52.67	65.80	58.51	52.24	56.69	54.37
	STL_sys 59.65	85.70	70.13	59.43	81.10	68.60
	TP_sys	63.90	88.12	74.08	62.70	81.21	70.77
	TEB_sys 80.80	94.51	87.12	79.50	90	84.42
	MPT_sys 84.08	96.13	89.70 83.86	93.89	88.59
	According performances shown in the two tables, the two systems developed ba-

sed on the two proposed scene text proposal techniques (MPT and TEB) outperform other scene text proposal based systems. Recalling scene text proposal performances in Section 4.3.2, there is a correlation between scene text proposal performances

Table 5 .

 5 4 -Word spotting performance of the proposed automatic scene text reading system (MPT_based) and other state-of-the-art systems on the two scene text datasets including the Street View Text with 50-word lexicon per an image (SVT-50)

	and Robust Reading Competition 2013 Dataset (ICDAR2013) in the two contex-
	tualizations.
		SVT-50	Strong contextualization Recall Precision F-score Recall Precision F-score Weak contextualization
	JarE2E [20]	68	86.68	94.64	90.49	-	-	-
	ConvLSTM [76]	-	84.93	98.91	91.39	84	97.29	90.16
	DTSpotter [77]	-	-	-	92.89	-	-	89
	TextBoxes [50]	84	90.77	97.25	93.90	87.38	97.02	91.95
	MPT_sys	84.63	88.20	97.42	92.58	87.85	95.31	91.43

Table 5 .

 5 5 -End-to-end performance of the proposed automatic scene text reading system (MPT_based) and other state-of-the-art systems on the Robust Reading Competition 2013 Dataset (ICDAR2013) in the two contextualizations. Aggregate Channel Features based scene text proposal for providing potential text regions. Histogram of gradient features are extracted at each region and the random forest classifier is adopted for the text/non-text classification. Two deep learning based scene text recognition models are finally applied, one for regressing bounding boxes and one for recognizing words in detected areas.

		Strong contextualization	Weak contextualization
		Recall Precision F-score Recall Precision F-score
	JarE2E [20]	82.12	91.05	86.35	-	-	-
	ConvLSTM [76] 79.39	96.68	87.19	79.28	94.91	86.69
	DTSpotter[77]	-	-	89	-	-	86
	TextBoxes [50] 87.68	95.83	91.57 84.51	95.44	89.65
	MPT_sys	84.08	96.13	89.70	83.86	93.89	88.59
	text detection is developed by combining two scene text proposal techniques inclu-
	ding EdgeBoxes [26] and a

Table 5 .

 5 [START_REF]Microsoft Translation Engine[END_REF] -Scene text searching performance and processing time of the proposed application and the GoogleTrans app.

		Proposed app GoogleTrans
	Performance	48.7%	39.48%
	Time processing(second)	17.98	2.34

Table 5 .

 5 6 presents performances of these two applications. As the results shown in Table, the proposed scene text searching application performs better than Goo-gleTrans application with 8.22% improvement. However, it is less efficient than the GoogleTrans application. The main reason is because our proposed application ana-

Chapter 6 Conclusion and future works Contents 6

  However, it performs better and sorts almost true text proposals in the top of the ranked list. This advantage is witnessed by superior recall rates of the proposed technique under a limited number of proposals The proposed grouping solution is inspired by a max-pooling process in the deep learning framework, and it merges connected components searched on binary Canny edge maps into scene text proposals. Influences of pooling window size and stride values are studied. An unique feature of the max-pooling grouping is that it does not rely on any text specific heuristic rules for merging. A novel proposal scoring function is built up based on the Histogram of Oriented Gradient on edge pixels and the K-mean cluster algorithm. The MPT technique outperforms state-of-the-art scene text proposal techniques, including our developed technique as the Text-Edge-Box in both recall rates and efficiency.In the ensuing section, we will present future works which can be developed based on proposed techniques and systems. A YoLo based scene text proposal technique is proposed, which adopts the YoLo network to provide scene text proposals.

	at 2000.
	In Chapter 4, the max-pooling based scene text proposal (MPT) has been propo-Suggestions for improving performance of proposed techniques and systems are also
	sed. A novel edge grouping solution and a proposal scoring function are developed. presented, including a new system design and a quadrilateral bounding boxes gene-
	ration. In addition, we proposed a relevant scene text searching based application
	for aiming elderly in their outdoor activities, including a navigation assistance, a
	danger alert, a scene text searching and a direction estimation.
	6.2
	In Chapter 5, we designed an automatic scene text reading system, which is intent
	on localizing and recognizing texts in scene images, and a scene text searching appli-
	cation. A state-of-the-art scene text recognition model is adopted for both scene text
	classification and scene text recognition, and it is built on top of our proposed scene
	text proposal techniques which are the TEB and the MPT. False positive proposals
	are eliminated based on a recognition confidence score threshold, lexicons, and non-
	maximum suppression. In the scene text reading competition, the proposed system
	provides state-of-the-art performance in two different evaluation contextualizations
	In Chapter 3, the proposed scene text proposal technique has been described, including an end-to-end scene text reading and a scene text spotting. Utilizing the
	which is adapted from the EdgeBox object proposal framework for scene text ob-developed automatic scene text reading system as a back-end program and adop-
	jects. Three major contributions are applied, including text specific edge-based low-ting a server-client communication framework, the scene text searching application
	cue features for discriminating text edges from non-text edges, a text geographical is developed. Users can access the application by using web browsers on their smart
	relationship based edge grouping solution for generating text proposals, and a text devices. Input images are captured by client cameras and the proposed application
	geographical relationship based scoring function for ranking proposals. The propo-returns locations of provided keywords in output images if keywords are exist in

.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.2 Future works . . . . . . . . . . . . . . . . . . . . . . . . . . 111 6.2.1 Deep learning based scene text proposal . . . . . . . . . . 111 6.2.2 Quadrilateral bounding boxes generation . . . . . . . . . 124 6.2.3 A navigation application . . . . . . . . . . . . . . . . . . . 126 6.1 Conclusion In this thesis, we have contributed several techniques and systems on the scene text detection and the scene text reading. Extensive evaluations on several scene text datasets show that the proposed works outperform recent state-of-the-art works. sed scoring function is more simple than a Adaboost classifier and a deep learning classifier which are used in the simple selective search for scene text proposal and the DeepText technique, respectively.

images. The proposed application is compared with the Google Translation application in a contextualization of scene text searching and it provides better searching performance while its efficiency needs to be improved.

Future works 6.2.1 Deep learning based scene text proposal

	nique which is constructed base on the YoLo9000 network architecture [89]. Note
	that the developed network in this section is just a baseline version and it needs
	further improvement as discussed in Section 6.2.1 to reach as well as outperform
	recent state-of-the-art scene text proposal techniques.

Deep learning framework (DL) has been applied on several computer vision applications such as object detection, object recognition, scene text detection, scene text recognition, and so on. These systems provided attractive performance in comparison with traditional frameworks developed base on hand-craft features. In recent years, this framework is also adopted for scene text proposals

[START_REF] Zhong | DeepText : A new approach for text proposal generation and text detection in natural images[END_REF][START_REF] Tian | Detecting Text in Natural Image with Connectionist Text Proposal Network[END_REF][START_REF] Jiang | R2CNN : Rotational Region CNN for Orientation Robust Scene Text Detection[END_REF][START_REF] Rong | Weakly supervised text attention network for generating text proposals in natural scenes[END_REF]

. They usually adapt the R-CNN framework for proposal generation by modifying the network to provide much more number of detection boxes at each cell in a square grid over image space. Proposal scores predicted by DL networks are seen as boxes confidence scores. An advantage of this solution is that it processes more efficient than traditional solutions. In this section, we propose a DL based scene text proposal tech-

Table 6 .

 6 1 -Recall (%) and processing time (in second) of two versions of YoLo based technique and other state-of-the-art techniques under different IoU thresholds on the ICDAR2013 dataset. The Nppb denotes an average number of proposals that the techniques need to achieve its presented recalls.

	IoU	0.5	0.7	0.8 Nppb times (s)
	M_YoloModel 95.62 80.18 53.06 1171	1.49
	O_YoloModel 81.83 63.11 37.72	16	0.34
	MPT	96.16 88.77 82.1 1450	4.49
	TEB [25]	94.25 87.95 81.55 1777	5.4
	DT [21]	88.5	67	4	500	-
	learning based techniques including YoLo based systems and DT are degrade dra-
	matically. Between two YoLo based techniques, the modified one (M_YoloModel)
	usually provides approximately 10% better than the original one (O_YoloModel) in
	different combinations of a number of proposals and an IoU threshold. This impro-
	vement is because the modified model provides a larger number of proposals which
	widen a change of localizing text regions in images.
	The second evaluation, two YoLo based baseline techniques are compared with
	state-of-the-art techniques in term of processing time and an average number of
	proposals each technique needs to provide competitive performances. This evaluation
	is shown in two Tables 6.1 and 6.2. As the shown results, YoLo based techniques
	provide competitive performance on the ICDAR2013 dataset and outperform other
	state-of-the-art techniques on the SVT dataset. Even though the O_YoloModel
	needs much less number of proposals than DT (16 comparing to 500), it performs
	much better than DT under IoU of 0.8 (37.72% comparing to 4%). Both YoLo based
	techniques are the best efficient techniques when they need less than two seconds
	in average to process an image, especially around 0.3 seconds by the O_YoloModel.
	Due to resizing images into constraint sizes, average processing time of both YoLo
	based techniques are similar on both datasets.

On the SVT dataset, YoLo based scene text proposal techniques provide attractive results in a full range of IoU from 0.5 to 0.8. Note that we do not evaluate

Table 6 .

 6 2 -Recall (%) and processing time (in second) of two versions of YoLo based technique and other state-of-the-art techniques under different IoU thresholds on the SVT dataset. The Nppb denotes an average number of proposals that the techniques need to achieve its presented recalls.

	IoU	0.5	0.7	0.8	Nppb times (s)
	M_YoloModel 97.37 82.53 53.01 1369	1.58
	O_YoloModel 78.67 56.11 32.46	18	0.34
	MPT	88.41 46.99 21.48 1769	3.59
	TEB [25]	87.64 47.91 20.09 1890	2.2
	techniques performance at the higher IoU threshold due to uncommonly used in
	real applications. The main reason is because proposal bounding boxes of YoLo ba-

Table 6 .

 6 3 -Word spotting performance of the YoLo based scene text reading systems (M_YoloModel_sys and O_YoloModel_sys) and other proposed systems (MPT_sys and TEB_sys) on the Robust Reading Competition 2013 Dataset (IC-DAR2013) in the two contextualizations.

		Strong contextualization	Weak contextualization
		Recall Precision F-score Recall Precision F-score
	M_YoloModel_sys 80.84	84.18	82.47	82.12	88.76	85.31
	O_YoloModel_sys 79.79	97	87.34	79.43	97	87.34
	TEB_sys	84.46	96.4	90.03	82.94	91.96	87.22
	MPT_sys	88.20	97.42	92.58 87.85	95.31	91.43

Table 6 .

 6 4 -End-to-end performance of the YoLo based scene text reading systems (M_YoloModel_sys and O_YoloModel_sys) and other proposed systems (MPT_sys and TEB_sys) on the Robust Reading Competition 2013 Dataset (IC-DAR2013) in the two contextualizations.

		Strong contextualization	Weak contextualization
		Recall Precision F-score Recall Precision F-score
	M_YoloModel_sys 78.95	87.65	83.07	77.86	83.02	80.36
	O_YoloModel_sys 76.33	97.35	85.57	76.11	96.11	84.91
	TEB_sys	80.8	94.51	87.12	79.49	90	84.42
	MPT_sys	84.08	96.13	89

.70 83.86 93.89 88.59 faster

  and provide smaller number of proposals, automatic scene text reading systems built on them still perform competitive results. Besides missing in detecting scene text objects, non-close-to-text-edge bounding boxes is also a reason that makes YoLo based scene text reading system miss detected texts due to false recognitions. Figure 6.4 presents examples of images in which detected texts have been removed by the text recognition model.

		Canny Edge Detector	Binary Edge Map	CCL	Feature Map	MPG
						Scene Text Proposals
			Proposal Generation
	Input Image	YoLo Model			Text Heat Map
			Proposal Scoring and Ranking
						Output Image

  example, if those words have a large size and at the centre of images, users might stay very close to dangerous areas. A high lever alert should be activated. int _segCnt ;//A number of connected components arrayi _segIds ;//Segmented edge pixels ids (-1/0 means no segment)

	vector<vectori> _GList ;//A list of groups of connected components
	//Internal functions which are our contributions
	arrayf _EPsegIds ;//A EdgePair score map void clusterEdges(arrayf &O, arrayb &BW, float thrs ) ;
	arrayf _EVsegIds ;//A EdgeVariance score map void BoxesGenerator(Boxes &boxes, arrayf &sc) ;
	arrayf _TEMsegIds ;//A TextEdgeMap //Searching for 8-neighbourhood connected components
	vector<vectori> _csIds, _rsIds ;//x,y coordinates of edge pixels void CCfound(vectori &cs, vectori &rs, vectori ids) ;
	//internal functions (our contributions) : void Grouping() ;//The proposed max-pooling based grouping solution is imple
	//Searching for 8-neighbourhood connected components //mented
	void CCfound(vectori &cs, vectori &rs, vectori ids) ; void BoxesScore(float &s, vectori &idlist) ;//The proposed scoring function is
	//Clustering edge pixels based on their score visualized by a TextEdgeMap //implemented
	//Proposal boxes generation functions
	} ;
	Utilizing a MPT class
	//Loading text/non-text centroids
	arrayf cens = <load _cens> ;
	//Set pooling window size (w_pw and h_pw) and stride values (w_str and h_str)
	vectorf psinfo ;
	psinfo.push_back{w_pw} ;
	psinfo.push_back{h_pw} ;
	} ; psinfo.push_back{w_str} ;
	Utilizing the TEB class : psinfo.push_back{h_str} ;
	C++ codes Boxes boxes ;
	//Initializing a TEB class MPTGenerator MPTGen ;
	TextEdgeBoxGenerator TEBGen ; MPTGen.ProposalBoxes(boxes, O, BW, cens, psinfo) ;
	//Initializing vector to store generated proposal boxes. Each element as a vectori is
	// a box
	vector<vectori> boxes ;

void clusterEdges(arrayf &E, arrayf &O, arrayi &BW ) ; //Generating proposals, estimating proposal bounding boxes and scoring proposals //it including many sub-functions void BoxesGenerator(Boxes &boxes) ; //Splitting proposals into word-level proposals void BoxesBreakDown(Boxes &boxesout,Boxes &boxesin) ; //Estimating boxes covering whole small boxes void boxlist2box(Box &b, Boxes &boxes) ; //Scoring proposal boxes in which our proposed scoring function is implemented void scoreBox( Box &box, arrayf &V ) ; //Two edge based text specific features estimation functions void EPCC(float &ep,vectori &cs,vectori &rs,arrayf &O) ; void EVCC(float &ev,vectori &cs,vectori &rs) ; void BoxesFromList(Boxes &boxes, vector<vectori> selectedList) ; void CleanList(vector<vectori> &cleanedList, vector<vectori> selectedList) ; void LinkList(vector<vectori> &linkedList, vector<vectori> selectedList) ; void List2Boxes(Boxes &boxes, vector<vectori> selectedList) ;

Acknowledgements

ACKNOWLEDGEMENTS vided comfortable facilities, including an office room, computers, printers, scanners and a lot of assistance from their Information Technology help desk. I would like to acknowledge all members of the IPAL laboratory in Singapore, who provided a friendly research environment : Hoa T., Ibrahim S., Martin K., Davi, Matthew, Thomas B., Joaquim B.M., Anna, Antoine M., Angela Sáenz, Christophe J., Fabien Clavier, Anssuya. I have special thanks to Hoa T., Ibrahim S., Martin K., Davi, Matthew, Thomas B., Antoine M., Angela S. for sharing their experiments on

concentrate on my research work with full energy and complete it properly. I am gratefully indebted to Professor Mounir MOKHTARI, the Director of the France-Singapore joint laboratory, Image and Pervasive Access Lab (IPAL), in Singapore as well as the Director of my thesis. He gave me a great chance to start my doctoral research and has supported me during my study. I am thankful for his patience, guidance and ideas which encouraged me and French delegate of Centre My success in this research includes his participation by a lot of useful discussions. Some of my difficulties have been solved under his support. I thank the Agency for Science, Technology and Research -Institute for Infocomm Research (ASTAR -I2R) for supporting this research work. They have pro-

 ---------------Function : boxlist2Box ---------------/ ---------------
Evaluation

Variants of the proposed systems

The recognition score threshold is an essential parameter of the proposed automatic scene text reading system. The higher threshold is better to filter out false positive proposals, however, it will degrade system performance in localizing challenging texts in scenes. We take this parameter in our consideration and evaluate its contribution to performance of our two proposed scene text reading systems, including a TEB based and a MPT based.

We vary the threshold of recognition score in a range of (0.1, 0.9) with an internal step of 0.1. Performances of two scene text proposal based scene text reading systems are evaluated on the ICDAR2013 dataset and presented in Figure 5.2. The F-measurement is calculated base on system precision and system recall as shown Fast and Faster R-CNN [START_REF] Girshick | Fast R-CNN[END_REF][START_REF] Ren | Faster R-CNN : Towards Real-Time Object Detection with Region Proposal Networks[END_REF] is that it performs coincidently both object localization and object recognition in the final prediction layers. Other frameworks mostly separate these two tasks into two different layers including proposal generation layers (forms a region proposal network (RPN)) and object recognition layers. This unique architecture brings YoLo several benefits including an extremely fast process, a global reasoning over image space for generating and scoring bounding boxes. Its architecture is shown in Figure 6.1 This architecture is inspired by the GoogleLeNet model [START_REF] Szegedy | Going deeper with convolutions[END_REF] with a difference that inception modules are replaced by proposed blocks in which a 1 × 1 reduction layer is followed by a 3 × 3 convolution layer. The output layer has a size of 7 × 7 × 28269, where the 7 × 7 space presents a square grid covering over image space. At each grid cell, the network predicts three bounding boxes. Each box contains five parameters as x,y,w,h,s, where [x,y,w,h] is box coordinates and s is a box confident score, and 9418-class probability distribution. They form the third dimension of the output layer as (3x(5+9418)=)28269 parameters.

divide an image into grid cells and assume that there is only one object in a cell if it exists. At each grid cell, anchor boxes are initialized with different sizes and aspect ratios. R-CNN neural networks are trained to score those boxes and regress their ratio and size to cover objects better. In this framework, initial anchor boxes need to have a large range of sizes and aspect ratios to be robust to objects' shapes variants. Otherwise, they will be false. For example, the SSD network [START_REF] Liu | Single Shot MultiBox Detector[END_REF], which is one of state-of-the-art models in detecting general objects, is false when its original design is applied for detecting scene text objects. The reason is because scene text objects including words, text lines have straight aspect ratios which are not initialized in SSD's anchor boxes. By redesigning anchor boxes with more suitable aspect ratios, the SSD network performs better and achieves state-of-the-art performances in scene text detection [START_REF] Liao | TextBoxes : A Fast Text Detector with a Single Deep Neural Network[END_REF]. By applying MPT as an anchor box initialization, anchor box size and aspect ratio are flexible and they can be self-adaptive to different image contents.

TextEdgeBoxes class :

The parameter _alpha is a parameter of the