Pierre-Etienne Martin
email: pierre-etienne.martin@u-bordeaux.fr

Jenny Benois-Pineau
email: jenny.benois-pineau@u-bordeaux.fr

Boris Mansencal
email: boris.mansencal@labri.fr

Renaud Péteri
email: renaud.peteri@univ-lr.fr

Julien Morlier
email: julien.morlier@u-bordeaux.fr

Siamese Spatio-temporal convolutional neural network for stroke classification in Table Tennis games

This work presents a Table Tennis stroke classification approach through a siamese spatio-temporal convolutional neural network -SSTCNN. The videos are recorded at 120 frames per second with players performing in natural conditions. The frames are extracted, resized and processed to compute the optical flow. From the optical flow, a region of interest -ROI -is inferred. The SSTCNN is then feed by RGB and optical flow ROIs stream to give a probabilistic classification over all the table tennis strokes.

INTRODUCTION

In the scope of video processing, action recognition and classification is one of the main challenge. In the Sport task of MediaEval 2019 [START_REF] Martin | Sports Video Annotation: Detection of Strokes in Table Tennis task for MediaEval 2019[END_REF], this aspect is underlined by providing a dataset of Tennis table recordings, TTStroke-21 [START_REF] Martin | Sport Action Recognition with Siamese Spatio-Temporal CNNs: Application to Table Tennis[END_REF], where strokes have to be extracted and classified with the aim of improving athletes performances. As a first step, videos are provided with temporal segmentation and the task is to classify those segments. However, contrary to the common datasets widely used in image and video processing such as UCF-101 [START_REF] Soomro | UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild[END_REF], HMDB [START_REF] Kuehne | HMDB: A large video database for human motion recognition[END_REF] or Kinetics [START_REF] Kay | The Kinetics Human Action Video Dataset[END_REF]; this task focuses on fined grained classification with the classification of strokes highly similar. The difficulty of this task is to be able to find the characteristics of each kind of stroke using a limited dataset without over-fitting it. In this paper, we present an approach aiming at providing data with enough inter-dissimilarity and focusing on intra-similarity to feed a neural network able to classify without over-fitting on a limited dataset.

APPROACH

To deal with the low inter-variability of the classes in TTStroke-21 and avoid over-fitting on this sample of the dataset, we decided to use cuboids of optical flow in addition to cuboids of RGB images with spatio-temporal convolutions processed simultaneously through a Siamese architecture as presented in [START_REF] Martin | Sport Action Recognition with Siamese Spatio-Temporal CNNs: Application to Table Tennis[END_REF].

Optical Flow estimator

As shown in [START_REF] Martin | Optimal choice of motion estimation methods for finegrained action classification with 3D convolutional networks[END_REF], flow estimators can have a strong impact on the classification, so we tested classification using two different flow estimators: DeepFlow [START_REF] Weinzaepfel | DeepFlow: Large Displacement Optical Flow with Deep Matching[END_REF] and Dense Inversive Search -DIS [START_REF] Kroeger | Fast Optical Flow Using Dense Inverse Search[END_REF].

Because of the strong motion artefacts observed on DIS flow, this one is smoothed with a Gaussian blur using a kernel of size 3 × 3 and then multiplied by the computed foreground [START_REF] Zivkovic | Efficient adaptive density estimation per image pixel for the task of background subtraction[END_REF] to keep only foreground motion.

Spatial segmentation

RGB and Optical Flow are spatially segmented using a region of interest -ROI -of center C roi = (x r oi , y r oi ) estimated from the maximum of the optical flow norm and the center of gravity of all pixels [START_REF] Martin | Sport Action Recognition with Siamese Spatio-Temporal CNNs: Application to Table Tennis[END_REF] as follows:

C max = (x max , y max ) = arдmax x ,y (||D|| 1 ) C g = (x д , y д ) = 1 δ (C) C∈Ω Cδ (C) C∈Ω with δ (C) = 1 if ||D|| 1 (C) 0 0 otherwise x r oi = α f ω x (x max , W ) + (1 -α) f ω x (x д , W ) y r oi = α f ω y (y max , H ) + (1 -α) f ω y (x д , H ) (1)
with parameters α = 0.6, Ω = (ω x , ω y ) = (320 × 180) the size of the resized video frames, (W , H ) the size of the data inputted to our network. The function f ω (u, V ) = max(min(u, Vω 2 ), ω 2 ) allows to have input data extracted within the boundaries of our data. To avoid jittering, we apply a Gaussian blur along the time dimension to average the center position using a kernel of size 40 and scale parameter σ blur = 4.44.

Data normalization

The RGB image channels are normalized by their theoretical maximum value, 255 in our case, to map them into interval [0,1]. As done in [START_REF] Martin | Optimal choice of motion estimation methods for finegrained action classification with 3D convolutional networks[END_REF] which compare different normalization methods, we decide to normalize the optical flow V = (v x , v y ) using the mean µ and standard deviation σ of the maximum absolute values distribution of each optical flow components over the whole dataset. In the following equation v and v N represent respectively one component of the OF V and its normalization.

v ′ = v µ+3×σ v N (i, j) = v ′ (i, j) if |v ′ (i, j)| < 1 SIGN (v ′ (i, j)) otherwise. ( 2
)
This normalization method maps the values into interval [-1,1] and increases the magnitude of most vectors making the optical flow easier to process for classification of very similar actions such as

SSTCNN

Our Siamese Spatio-Temporal Convolutional Neural Network -SSTCNN, see Fig. 1, is constituted of 2 branches with three 3D convolutional layers with 30, 60, 80 filter response maps, followed by a fully connected layer of size 500. They take respectively of RGB values and optical flow computed from them of size (W × H × T )= (120×120×100). The 3D convolutional layers use 3×3×3 spacetime filters with a dense stride and padding of 1 in each direction. The two branches are fused through a final fully connected layer of size 21 followed by a Softmax function to output a probabilistic classification. 6 . We also spatially augment the data by applying random rotation in the range ±10 • , random translation in range ±0.1 in x and y directions, random homothety in range 1 ± 0.1 and a 0.5 chance flip in horizontal direction and random channel swaps on the RGB data. We take extra care of applying those changing on the Optical Flow by updating its values according to the transformations. Transformations are applied and centered on the region of interest avoiding crops outside of the camera range.

Training and submitted runs

All models were trained from scratch. We used firstly 250 epochs with the data samples split randomly between all strokes and then split using only two videos for validation. However we noticed the results obtained by splitting the dataset between videos were not satisfying. After looking at the dataset in detail, this is due to the fact that most of the videos contain only one kind of stroke performed by the same player. So the model will over-fit easily to the player appearance and not the characteristics of the stroke itself. With such a limited dataset and a limited time window we preferred to focus on the random distribution of the strokes among our training and validation sets. The two first runs are the classification obtained with the model trained on the split dataset and saved on the minimum loss obtained on the validation set with two different flows presented in section 2.1. The other two runs are the same models but retrained from scratch using all data samples with the number of epochs used for obtaining best performance on the first validation set.

RESULTS

On the left side of the Table 1 we can see results of the first two runs from the models trained on the split database with 250 epochs; and on the right side two others runs obtained from the models trained with all the data. Compared to what has been obtained in previous work [START_REF] Martin | Sport Action Recognition with Siamese Spatio-Temporal CNNs: Application to Table Tennis[END_REF], the results are very low. The main differences are i) the lack of a negative class and ii) the split of the dataset in train and test sets between videos. It directly leads to an over-fitting of the dataset and makes the model much less able to do a proper classification. Best results were obtained by using DeepFlow estimator.

Figure 2: Confusion Matrix of our best run

Furthermore, if we consider the confusion matrix of our best run, Fig. 2, and group strokes in larger classes as: 'Forehand', 'Backhand' or 'Service', 'Offensive', 'Defensive' or their intersection (6 classes), we respectively get accuracies of 76.8%, 65.8% and 54.8%.

CONCLUSION

Despite a strong over-fitting, by grouping strokes together in larger classes, we can notice that some characteristics to recognize strokes are still learned. Furthermore, the work on TTStroke-21 [START_REF] Martin | Fine-Grained Action Detection and Classification in Table Tennis with Siamese Spatio-Temporal Convolutional Neural Network[END_REF] is still in progress and the enrichment of the dataset will be a big contribution in the domain of action detection and classification especially for very similar actions.

Figure 1 :

 1 Figure 1: SSTCNN architecture

  Table Tennis strokes.

	MediaEval'19, 27-29 October 2019, Sophia Antipolis, France	P-e Martin et al.

Table 1 :

 1 Runs results

	Flow	Epochs Train Val Test Train Test
	DIS	249	70.4 52.6 19.2 61.2 17.8
	DeepFlow	229	74.7 56.1 17.2 70.2 22.9

ACKNOWLEDGMENTS

This work was supported by Region of Nouvelle Aquitaine grant CRISP and Bordeaux Idex Initiative.


Tilahun Yeshambel
email: tilahun.yeshambel@gmail.com

Josiane Mothe
email: josiane.mothe@irit.fr

Yaregal Assabie
email: yaregal.assabie@yahoo.com

2AIRTC: The Amharic Adhoc Information Retrieval Test Collection

Keywords: Information Retrieval, Amharic test collection, Adhoc retrieval, Evaluation, Data collection, Corpus, Under-resourced language

Evaluation is highly important for designing, developing, and maintaining information retrieval (IR) systems. The IR community has developed shared tasks where evaluation framework, evaluation measures and test collections have been developed for different languages. Although Amharic is the official language of Ethiopia currently having an estimated population of over 110 million, it is one of the under-resourced languages and there is not yet Amharic adhoc IR test collection. In this paper, we promote the monolingual Amharic IR test collection that we built for the IR community. Following the framework of Cranfield project and TREC, the collection that we named 2AIRTC consists of 12,583 documents, 240 topics and the corresponding relevance judgments.

Introduction

With the Internet technology, and the growth of online text information and globalization, information retrieval (IR) has gained more attention especially in commonly used languages. Both the research community and the industry have been very active in this field for more than 50 years [START_REF] Sanderson | The history of information retrieval research[END_REF]. IR also has an old history of evaluation. After the early framework in the Crandfield project, TREC has standardized adhoc retrieval evaluation [START_REF] Buckley | TREC: Experiment and Evaluation in Information Retrieval[END_REF]. Performance evaluation of a system is indeed very crucial for scientific progress [START_REF] Ferro | CLEF 15th Birthday: Past, Present, and Future[END_REF]. There are many system evaluation criteria such as effectiveness, efficiency, usability, accessibility, utility, portability, and maintainability to complete the task. One of the major focuses of IR research evaluation is to measure the IR system effectiveness. In adhoc retrieval, where the task for the system is to retrieve the relevant documents for a given query, effectiveness looks at the ability of the system to retrieve only relevant documents for a given user's query. This implies an evaluation framework consisting of a test collection as well as metrics. The standard adhoc retrieval test collection consists of three components: a corpus of documents to be searched in, a set of users' information needs or topics, and the associated relevance judgments indicating which documents are relevant for which topics. Test collections facilitate reproducibility of results and meaningful effectiveness comparison among different retrieval techniques.

In adhoc retrieval evaluation, a corpus is usually a systematic collection of naturally occurring fixed size documents in machine readable form. Building text collections for adhoc IR is a common task although it is resource demanding. A large number of shared tasks rely on such collections. Some of the well known text collections are Cranfield project [START_REF] Cleverdon | The evaluation of systems used in information retrieval[END_REF], Text REtrieval Conference1 (TREC) and more specifically TREC adhoc [START_REF] Harman | Overview of the second text retrieval conference (TREC-2)[END_REF], Cross-Language Evaluation Forum2 (CLEF) datasets [START_REF] Ferro | CLEF 15th Birthday: Past, Present, and Future[END_REF], and NACSIS test Collection for Information Retrieval3 (NTCIR) [START_REF] Kando | The NTCIR Workshop : the First Evaluation Workshop on Japanese Text Retrieval and Cross-Lingual Information Retrieval[END_REF]. Indeed, the IR international conferences such as TREC, CLEF, NTCIR, INEX4, and FIRE5 are held based on their own test collections.

One of the common techniques to build a test collection for adhoc retrieval is pooling, where the document pool to be judged by humans is built by putting together the top N retrieved results from a set of systems [START_REF] Soboroff | A comparasion of pooled and sampled relevance judgments in the TREC 2006 Terabyte Track[END_REF]. In this technique, documents outside the pool are considered as non-relevant. Pooling is a standard technique to create relevance judgment in TREC, CLEF, and NTCIR test collections. The second technique is exhaustive relevance judgment where every and each document in a dataset is judged according to each query [START_REF] Kagolovsky | Current Status of the Evaluation of Information Retrieval[END_REF]. Cranfield and CACM test collections are built using exhaustive relevance judgment. A third way of preparing relevance judgment is crowdsourcing in which huge and heterogeneous mass of potential workers are assigned to a relevance task in the form of an open call through Internet [START_REF] Samimi | Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing :A Review[END_REF]. Whatever the way they are built, test collections are one of the pillars for testing and comparison of retrieval system performance.

While there are a lot of existing collections in different languages such as English, French, Arabic, and Asian languages, to our best knowledge, there is no test collection for Amharic IR. The lack of reference collection is a major impediment to the development of Amharic IR as well as natural language processing (NLP) tools. Indeed, Amharic is one of the under-resourced languages: computational resources such as training and test data, electronic bilingual dictionary, stemmer, tagger, morphological analyzer, etc. do not exist or the existing ones are not fully functional and the number of studies reported on Amharic is considerably limited compared to what is done for other languages such as English. Tools and resources that have been developed for other languages cannot be directly applied to Amharic because of its very specificities. We do believe that an Amharic reference collection would help to carry out more research and development works on Amharic IR and NLP. This paper describes the construction process and characteristics of the resulting Amharic IR test collection we deliver to the IR community.

Amharic Language

Amharic is an Afro-Asiatic language belonging to the South-west Semitic group [START_REF] Hetzron | Ethiopian Semitic: Studies in Classification[END_REF][START_REF] Argaw | Dictionary-Based Amharic-French information retrieval[END_REF]. It is an official language of the government of Ethiopia currently having an estimated population of over 110 million [countrymeters, 2020]. It is the secondmost commonly spoken Semitic language in the world next to Arabic [START_REF] Abate | Development of amharic morphological analyzer using memorybased learning[END_REF]. Although many languages are spoken in Ethiopia, Amharic is the lingua franca and the most literary language serving as a medium of instruction in the education system of the country for a long period. It uses Ethiopic alphabet for writing and has 34 base characters along with modifications on the respective base characters. The alphabet is conveniently written in a tabular format of seven columns. The first column represents the basic form with vowel ኧ /ä/ and the other six orders represent modifications with vowels in the order of ኡ/u/, ኢ/i/, ኣ/a/, ኤ/e/, እ/ə/, and ኦ /o/. For example, the base character መ/mä/ has the following modifications: ሙ/mu/, ሚ/mi/, ማ/ma/, ሜ/me/, ም/mə/, and ሞ/mo/. The language also uses punctuation marks such as ። (full stop), ፣ (comma), ፤ (semicolon), ፥ (colon), etc. It also adopts some other punctuation marks such as question and exclamation marks from English.

Amharic is known to have a complex morphology. A large number of words can be formed from a base form and word formation is complex involving affixation, reduplication, and Semitic stem inter-digitation. Thousands of surface words can be generated from an Amharic root and its stems by changing the shape of alphabets in a stem or root, and by adding affixes on stems [START_REF] Abate | Development of amharic morphological analyzer using memorybased learning[END_REF]. For example, the verb አሰበ/ässäbbärä/ is derived from the verbal stem ሰበር-/säbbär-/ which is itself derived from the verbal root ስ-ብ-ር /s-b-r/. Furthermore, a verb can be marked for a combination of person, case, gender, number, tense, aspect, and mood. Accordingly, the following verbs can be generated from the verbal stem ሰበር-/säbbär-/: ሰበርኩ /säbbärku 'I broke'/, ሰበርኩህ/säbbärkuh 'I broke you'/, ሰበርን/säbbärn 'we broke'/, ተሰበርኩ/täsäbbärku 'I was broken'/, ሰበረች/säbbäräc 'she broke'/, አይሰብርም/ʔäysäbrəm 'he will not break'/, etc. Thousands of words can be generated from a single verbal root making analysis, annotation and tagging of Amharic text a non-trivial task. This level of morphological complexity has significantly contributed to the difficulty of producing linguistic resources for Amharic.

3.

Related Work

Test collections and evaluation standards

To investigate the performance of a given retrieval technique, IR research community uses reference collections that have been built for different languages and many of them are publicly available and freely accessible. They are now commonly used in IR studies and helped in promoting research in IR. In this section, we present some of the text test collections.

Cranfield test collection is the first IR test collection that also grounded the evaluation framework used nowadays in IR. It was created in late 1960's and contains the abstract of 1,400 documents, 225 queries and the corresponding relevance judgment [START_REF] Cleverdon | The Cranfield tests on index language devices[END_REF][START_REF] Harman | Overview of the second text retrieval conference (TREC-2)[END_REF]. The Cranfield test collection is the base for the success of different conferences like Text Retrieval Conference (TREC). However, in the 1990's, the size of the Cranfield collection was considered as too small to generalize a given finding on it.

TREC was established in 1992 in order to support IR researches, to provide larger and more realistic collections, as well as to promote a standard for IR evaluation [START_REF] Harman | Overview of the second text retrieval conference (TREC-2)[END_REF]. Since then, the TREC conference creates series of evaluation resources specifically for adhoc retrieval. What is now considered as a standard adhoc IR TREC collection consists of documents, topics that correspond to users' needs and that can be structured in various fields, and relevance judgment. While TREC initially focused on English, it had also considered other languages as Spanish, Chinese and Arabic that went later to other conferences.

The Cross-Language Evaluation Forum (CLEF) conference is one of the known conferences that have their own large-scale evaluation test collection for European languages. With the initiative of CLEF, large test collections for languages such as English, French, German, Bulgarian, and Hungarian are now available [Peter, 2001;[START_REF] Ferro | CLEF 15th Birthday: Past, Present, and Future[END_REF]. One of the CLEF aims was to evaluate both monolingual IR and cross-lingual IR systems while it is now oriented more to other tasks such as image retrieval, health systems, etc. NII-NACSIS Test Collection for Information Retrieval (NTCIR) focuses on crosslanguage search for Asian languages such as Japanese, Chinese, and Korean [START_REF] Kando | The NTCIR Workshop : the First Evaluation Workshop on Japanese Text Retrieval and Cross-Lingual Information Retrieval[END_REF]. Since 1997, it promotes researches in IR, text summarization, information extraction, and question-answering with the aims of offering research infrastructure, forming research community and developing evaluation methodology.

Amharic corpora, resources and tools

The growth of Amharic digital data accelerates the demand for technologies and NLP tools for online data processing. Nonetheless, for experimental evaluation of applications and tools very few corpora and resources have been built. Some of the Amharic text corpora which are available digitally and utilized for the development of Amharic NLP tools, IR or other text-centered tasks are presented in what follows.

Corpora

Walta Information Center (WIC) Corpus: This corpus has been built by linguists from Addis Ababa University and it is available both in Amharic characters and Romanized form. The corpus contains 1,065 Amharic news articles with 4,035 sentences. The domain of the corpus is much diversified including topics like politics, economics, science, sport, religion, business, etc. [START_REF] Demeke | Manual annotation of Amharic news items with part-ofspeech tags and its challenges[END_REF].

Ethiopian Language Research Center (ELRC) Corpus: This is the annotated version of WIC corpus. It has been annotated with part-of-speech (POS) tags manually by ELRC at Addis Ababa University. The corpus is tagged with 30 different POS tags [START_REF] Demeke | Manual annotation of Amharic news items with part-ofspeech tags and its challenges[END_REF].

Addis Ababa University NLP Task Force Corpus: This corpus is prepared by Language Technology staff members from Information Technology Doctoral Program at Addis Ababa University. The corpus is prepared to create parallel corpus for computational linguistics and includes Amharic, Afaan-Oromo, Tigrigna, and English languages with diverse content from historical documents and newspapers. The project is still on-going and the corpus is continuously being updated [START_REF] Abate | Parallel Corpora for Bi-Lingual English-Ethiopian Languages Statistical Machine Translation[END_REF].

Amharic Corpus for Machine Learning (ACML): This corpus has been prepared by Gamback [START_REF] Gamback | Tagging and Verifying an Amharic News Corpus[END_REF]. The data set consists of free texts collected from Ethiopian News Headlines (ENH), WIC and Amharic fiction "Fikir Iske Meqabir" (FIM). It is a set of 10,000 ENH articles with a total of 3.1 million words, 1,503 words from the WIC corpus, and 470 words from FIM book.

These corpora are simply collections of documents and as such are not appropriate to evaluate adhoc retrieval since there is no query set nor the associated relevance judgment. They are mainly collections of documents from which formatting tags are removed and additional semantic annotations are provided. Moreover, most of the corpora are not publicly accessible.

It is also worth to mention the non-European languages multi-lingual IR track at CLEF where queries in Amharic language were used; although document collections were in European languages [Di Nunzio et al., 2005;Di Nunzio et al., 2007]. For Amharic-English IR, one of the most successful approaches was a dictionary-based one [START_REF] Argaw | Dictionary-Based Amharic-French information retrieval[END_REF].

Resources

Amharic Machine Readable Dictionaries: Some of the commonly used dictionaries involving Amharic are Amharic-English dictionary containing 15,000 Amharic words [START_REF] Amsalu | Amharic-English Dictionary[END_REF], Amharic-French dictionary containing 12,000 Amharic entries [START_REF] Berhanu | Amharic-Français Dictionnaire[END_REF], and Amharic-Amharic dictionary containing 56,000 entries [START_REF] Kesatie | YeAmarinja Mezgebe Qalat. Ethiopian Languages Research Center[END_REF]. Entries of the Amharic machine-readable dictionaries are represented by their citation forms.

Stopword lists: Few researches were conducted on building Amharic stopword list. Researchers who conduct studies on Amharic IR usually build their own list of stopwords. For example, Mindaye et al. [2010] built the stopword list with 77 entries while [START_REF] Eyassu | Classifying Amharic news text using self-organizing maps[END_REF] created the stopword list with 745 entries. They collected stopwords from different sources but these lists have not been evaluated by linguists. Recently, [START_REF] Yeshambel | Construction of Morpheme-Based Amharic Stopword List for Information Retrieval System[END_REF] built morpheme-based stopword list by systematically analyzing the morphology of the language and their distribution in the corpus. The list consists of 222 stopwords.

Tools: Amharic faces challenges in the development of NLP tools and applications. The major obstacles that hinder the progress on the development of Amharic NLP applications are complex morphology of the language, lack of sufficient corpora, and lack of standards in resource construction and application development. As a result, only few Amharic NLP tools have been developed thus far using rule-based and machine learning approaches. Among the available tools are morphological analyzers [START_REF] Argaw | Amharic-English information retrieval[END_REF][START_REF] Gasser | HornMorpho: a system for morphological of Amharic, Oromo, and Tigrinya[END_REF][START_REF] Abate | Development of amharic morphological analyzer using memorybased learning[END_REF], stemmer [START_REF] Alemayehu | The effectiveness of stemming for information retrieval in Amharic[END_REF], and parser [START_REF] Argaw | Amharic-English information retrieval[END_REF][START_REF] Tachbelie | Part-of-speech tagging for underresourced and morphologically rich languages-the case of Amharic[END_REF]. However, the development of these tools is at prototype stages. They are also limited in scope.

4.

Amharic IR Test Collection

Methodology to build the 2AIRTC collection

The document collection creation has been carried out in two steps: initial corpus and Web corpus. We initially collected documents from various sources that produce documents in Amharic without considering any specific topics. We then complement this collection with web documents retrieved considering our target topics. To do this, we run the query part of our topics on a Web search engine and gather the retrieved documents. The topic set was created by considering both current issues but also considering the topics that were likely to be treated in our initial sources. The document relevance judgments were manually done using a precise guideline. For each topic, we run Lemur (http://www.lemurproject.org/) on the initial document set and fuse these results with the ones from the web search engine. The fused lists were then manually assessed with binary relevance for each topic. These steps are described in more details in the next sub-sections.

Document set

Since we wanted documents to be diverse enough, we initially collected documents from different sources. More precisely, we collected 777 documents from news agencies sites (Walta Media and Communication Corporate6 , Fana Broadcasting Corporate7 , Amhara Mass Media Agency8 ) and social media (Facebook), 701 historical documents from blogger (Daniel Kibret9 ), and 15,000 documents from Amharic Wikipedia10 . In addition, 8,522 news articles were collected from Walta Information Center and we also collected 1,189 religious documents, 1,773 news articles, and 772 documents (letters, opinions and reports) from various sources. Accordingly, the total number of documents collected is 28,734. The document collection represents various topics about business, sport, entertainment, education, religion, politics, technology, health and culture.

After the topics were created, to select the documents to be assessed, we ran the title fields of these topics on our initial corpus using Lemur toolkit. We also ran the same queries (topic title) using Google on the Web. Here our idea was to use not only the documents from our initial collection but also to enrich the collection with documents that were retrieved on the Web. Our aim was to complement the document collection and to avoid topics with either no or a very few relevant documents. For the search engine documents, we considered a maximum of 50 documents per topic and we collected 2,880 documents in that way on the Web. For relevance assessment, we fused both retrieved document list (Lemur on our initial collection and Web documents for the complementary documents). Each document from the fused list was then judged for relevance.

Finally, the document collection consists in two sub-collections: documents that have been assessed for at least one topic (either relevant or non-relevant) and the entire document collection.

Assessed document sub-collection: This collection is created only from the judged documents in the initial corpus and Web documents. The top retrieved documents from both Lemur and Google of each query are fused and organized in separate files and then assessed independently. This sub-collection consists of 12,583 assessed documents though the collected documents were more than this. Out of these, 6,960 documents have been assessed as relevant for at least one topic and the remaining 5,623 documents have been judged as non-relevant. These documents are full length, processed to remove unnecessary parts such as tags and English alphabets, and plain text form. All the documents are stored in a single text file using TREC-like format. Each document has a unique document identification number. The content of each document is enclosed with <TEXT> and </TEXT> tags. One document is delimited from the other by "DOC" and "</DOC>" tags. As shown in Table 1, the relevant document set contains various document lengths from small to very large. The same holds for documents that have been judged as non-relevant. We created the topic set using Amharic language statements from our search experience. The topics were built in such a way to reflect real word information need and cover diverse issues. The assumption considered during topic creation was that words in the topic titles are expected in document collection. The 2AIRTC contains a set of topics which prescribe information needs to be met. We manually created 240 topics for the adhoc IR task where the topics are about specific entities (e.g., people, places or events). Many of these topics were created by skimming some documents in the initial corpus, few of them were created by considering current issues, and the other were simply made up. The topic set is written both in Amharic and its translated version of English. Both the corpus and topic set are coded in UTF-8 (see Figure 1). Each topic has a unique identification integer number. The title field contains fewer search words which describe a topic and could be a typical query to be submitted to a retrieval system. Topic titles vary in terms of length and types. The topic titles include short topics, medium topics, and collocation. Since Amharic is a morphologically rich and complex language, the topic titles were designed to reflect real operational environment. The base of Amharic words might be stem or root. For Amharic retrieval, on top of stems, the roots of words which are derived from verbs are important rather than stems. Therefore, various types of words are included in the titles. Some of them consist of primary words and the others are from derived words. The description field contains the description of the topic area in one or two sentences. It is the description of the user's information need. Conceptually, it is consistent with the topic title and states its purpose in a sentence form. The narrative field provides further explanation about each title to decide which types of documents are relevant and which are not. It consists of more than two sentences. Assessors judge document relevance based on this field. Table 2 presents the detailed information on the topic set.

Relevance judgments

The relevance judgment is the third element of an adhoc IR test collection. It indicates the set of relevant documents to each topic. With regard to the document list to be reviewed by assessors, as mentioned previously, we ran the title field of the topics on our initial corpus using Lemur toolkit to get the first top 50 retrieved documents list per topic. We also ran the same topic using Google and considered a maximum of 50 documents. The topics and the associated retrieved results of both were distributed to assessors who judged them. Duplicated documents were removed. The relevant assessment was made manually by reading documents and using the narrative part of each topic. In addition, exhaustive relevance judgment was used on some topics in the initial document collection to get a larger number of relevant documents.

A document is marked as relevant based on the narrative information in the topic; thus it should not simply contain words from the query but rather fulfill the information need. Document relevance assessment has been done by students taking the IR course at University of Rift Valley. The students formed groups in which each group consisting of five students was given 20 topics and the top 50 retrieved documents for each topic by Google and Lemur. The students judged the relevance of each document for the given topic based on its narrative information and their satisfaction as users. There was one assessor group per topic. However, the five students in the group needed to agree to decide the relevance of each document. Therefore, a judgment represents the shared information needs of a group of students.

While assessing the top 50 documents retrieved by Lemur, sometimes we could not get any relevant document from our initial corpus of 28,734 documents. Therefore, some documents which were not retrieved for any of the topics using Lemur were assessed carefully in exhaustive relevance judgment during the second phase. For those documents, students had read each of them and judged as relevant or non-relevant to each topic.

As a result of manual relevant assessment, some topics have many relevant documents, while other topics have fewer relevant documents. Each topic has at least 10 relevant documents. This indicates that for some topics it will not be possible to measure effectiveness above rank ten; this is also the case in TREC or alike test collections.

Finally, the 2AIRTC relevance file was produced using the TREC format, as follows: topic ID, 0, document ID and relevance fields. Topic and document IDs are unique identification numbers of topics and documents, respectively. The number zero (0) is common to all topics and documents. The relevance indicates the relevance value of the considered document/topic pair and is 1 if the document is relevant to a topic, 0 otherwise.

Conclusion

Using standard test collections in IR is a common experimental practice. Various test collections for different languages have been built and are used by many research groups. Amharic IR test collection is an under explored research area. In comparison with other resourceful languages, few resources and tools have been built for Amharic. Even the existing Amharic corpora do not have any associated Amharic topics and relevance judgments. Furthermore, most of the existing corpora and resources are small in size and not publicly accessible. However, the importance of building and sharing test collections is well acknowledged. We built the first reusable test collection for IR system benchmarking. This collection will be made available to research communities. It will be accessible on request at tilahun.yeshambel@gmail.com. Our Amharic test collection is reproducible and contains representative documents and topics. This collection, named 2AIRTC 11 , can serve as a reliable resource for the evaluation and comparison of various Amharic IR systems. We do believe this collection will help in enhancing new research on Amharic IR.

Table 1 :

 1 Statistics of the 2AIRTC relevant judged document corpus While the previous sub-collection contains the documents that have been assessed for at least one topic, the entire collection consists of our initial documents plus the ones retrieved from the Web making a total of 31,614 documents. The documents from this collection are not formatted.

	Parameter	Size
	Number of documents	6,960
	Number of sentences	63,081
	Total number of words in the documents	2,243,372
	Number of unique words	6,446
	Minimum number of words per document	43
	Average number of words per document	1,357
	Median of words per document	219
	Maximum number of words per document	74,804
	Size of the relevant judged document	28.8MB
	Entire document collection: 4.3. Topic set
	<top>
	<num>2</num>
	<title_A> የኢትዮጵያዊያን የዘመን አቆጣጠር </title_A>
	<title_E> Ethiopian calendar </title_E>
	<desc_A> ስለኢትዮጵያ ዘመን አቆጣጠር ሥርአት የሚያትቱ ሰነዶችን መለየት፡፡ </desc_A>
	<desc_E> Identifying documents discussing on Ethiopian calendar system. </desc_E>
	<narr_A> ስለ ኢትዮጵያ የዘመን አቆጣጠር ታሪክና አመሰራረትት የሚያትቱ ሰነዶች ጥሩ የመረጃ ምንጮች ናቸው፡
	፡ ከዚህ በተጨማሪ የበአላት ቀናት እና የአቆጣጠር ስሌት የሚያትቱ ሰነዶች ጠቃሚ የመረጃ ምንጮች ናቸው፡፡
	ይሁን እነጅ ስለአውሮጳውያን የዘመን አቆጣጠር ወይም ሌሎች ሀገሮች የቀን አቆጣጠር የሚገልጹ ሰነዶች ጠቃሚዎች
	አይደሉም፡፡ እንዲሁም ስለአዲስ አመት የሚያትቱ ሰነዶች ጠቃሚ የመረጃ ምንጮች አይደሉም፡፡</narr_A>
	<narr_E> Documents discussing the origin and history of Ethiopian calendar are good
	sources of information. In addition, documents explaining about holidays and methods for
	finding the dates and day in each year are relevant. However, documents discussing on
	Gregorian calendar or other calendars are not relevant. Moreover, documents discussing on
	new year are not relevant. </narr_E>
	</top>
	Fig. 1: 2AIRTC topic number 2

Table 2 :

 2 Statistics of the topic set

	Parameter	Size
	Number of topics	240
	Minimum number of words per topic title	1
	Average number of topic title's words per topic	4
	Median of topic title's words per topic	3
	Maximum number of topic title's words per topic	7

Table 3 :

 3 2AIRTC relevance judgment statistics

	Parameter	Size
	Total number of topics	240
	Average number of relevant documents per topic	22
	Minimum number of relevant documents per topic	10
	Maximum number of relevant documents per topic	172

Text REtrieval Conference http://trec.nist.gov/

Cross Language Evaluation Forum http://www.clef-initiative.eu/

NII Test Collection for Information Retrieval http://research.nii.ac.jp/ntcir/

Initiative for the Evaluation of XML Retrieval http://inex.mmci.uni-saarland.de.

Forum for Information Retrieval Evaluation http://fire.irsi.res.in

http://www.waltainfo.com/

https://www.fanabc.com/

https://www.facebook.com/AmharaMassMediaAgencyAMMA/

http://www.danielkibret.com/

https://am.wikipedia.org/wiki/

Acknowledgement

We would like to thank students who participated in the creation of the test collection. We also thank the owners and sources of documents (Walta Media and Communication Corporate, Fana Broadcasting Corporate, Amhara Mass Media Agency and Daniel Kibret) who provide and permit their documents to carryout academic research experiments and annotate, use and share them for research communities.