Kevin Chappellet

Guillaume Caron

Fumio Kanehiro

Ken Sakurada

Abderrahmane Kheddar

Benchmarking Cameras for OpenVSLAM Indoors

la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

In the context of a closed-known-environment (i.e. indoor), where an exploration phase for a robot is not necessary, visual Simultaneous Localization and Mapping (vSLAM) algorithms can be used in two phases. The first one is to create a map with a sensor commonly used by the robot. This pre-built map will be used for further localization only. This recently proved to be very efficient in a robotic industrial context, e.g. [START_REF] Kheddar | Humanoid robots in aircraft manufacturing: The airbus use cases[END_REF]. Some works show that the field of view, a multiple view points, and the depth have an impact on the localization. In this work, we rather investigate for the first time, how the localization is impacted by the choice of various cameras. We use the unified feature-based vSLAM framework OpenVSLAM [START_REF] Sumikura | OpenVSLAM: A versatile visual SLAM framework[END_REF] which allows using various cameras as input. To our best knowledge, the only work evaluating practically the impact of the camera type on vision-based 3D motion estimation concerns visual odometry (hence no relocalization) [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF]. The latter focuses on single cameras of conventional and panoramic field-of-views, both indoors and outdoors, and in simulated and actual environments. It is found that a small FoV is preferred for large scale scenario while a large FoV is more suited for small confined environments. Our paper goes beyond the previous study by (i) considering the relocalization problem under varying lighting conditions and scene content, after a first mapping of the scene, and (ii) considering more camera types, including RGB-D, full spherical and stereoscopic types.

Our paper is organized as follows: Section II reviews various vSLAM with their specifications. Section III recalls impacts of camera types on vSLAM. Section IV introduces the evaluation metrics use in this study. Section VI highlights the methodology concerning the data acquisitions, followed by Section VII that analyzes the obtained results. Finally, Sections VIII and IX conclude our study.

II. RELATED WORKS

Simultaneous Localization And Mapping is a rich research area ranging from filtering algorithms to loop-closure detection, exploiting various sensors as lidars, sonars, inertial measurement units (IMU) and cameras, to cite a few [START_REF] Bresson | Simultaneous Localization And Mapping: A Survey of Current Trends in Autonomous Driving[END_REF]. In this paper, we deal with visual SLAM [START_REF] Taketomi | Visual SLAM algorithms: A survey from 2010 to 2016[END_REF] and vision-based only SLAM, i.e. without considering other types of sensors (e.g. IMU). Furthermore, we review the state-of-the-art within the spectrum of camera types including monocular, stereoscopic, multi-camera, either passive RGB (Red Green Blue) or active RGB-D (RGB plus Depth), of conventional, panoramic and full spherical field-of-views. Such line is complementary to existing vSLAM surveys and benchmarks [START_REF] Nardi | Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM[END_REF]- [START_REF] Liu | Evaluation of different SLAM algorithms using google tangle data[END_REF], mostly focusing on visual feature types, filtering or optimization methods and map structures. Table I gathers and classifies a restricted number of key existing vSLAM, highlighting the most versatile ones in terms; i.e. those dealing with various field-of-views and combinations of cameras.

To the best of our knowledge, vSLAM technology started with the first visual SLAM running in real-time based on keypoints detected in images of a conventional camera [START_REF] Davison | MonoSLAM: real-time single camera SLAM[END_REF]. Since then, one of the key contributions have been architectural, by running in parallel the features tracking and the mapping stages [START_REF] Klein | Parallel tracking and mapping for small AR workspaces[END_REF]. They were later renamed as front-end and backend, respectively. The other type of key contribution is related to the visual feature considered such as very close to the camera measurement, i.e. pixel intensities [START_REF] Forster | SVO: Fast semi-direct monocular visual odometry[END_REF], or much more abstract as Oriented fast and Rotated Brief (ORB) features [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF]. All these works are done with the target of robustness and precision while keeping real-time localization and mapping, i.e. at the conventional RGB camera frame rate.

Redundancy brought by stereovision [START_REF] Engel | Large-scale direct SLAM with stereo cameras[END_REF], [START_REF] Forster | SVO: Semidirect visual odometry for monocular and multicamera systems[END_REF] as well as considering the additional Depth modality (RGB-D), either in physics-based [START_REF] Bu | Semi-direct tracking and mapping with RGB-D camera for MAV[END_REF]- [START_REF] Whelan | Elasticfusion: Real-time dense slam and light source estimation[END_REF] or learning-based [START_REF] Brahmbhatt | Geometryaware learning of maps for camera localization[END_REF]- [START_REF] Zhou | Unsupervised learning of depth and ego-motion from video[END_REF] features modeling, also improve precision and robustness of localization and mapping thanks to a better handling of partial Stereo Dual [START_REF] Davison | MonoSLAM: real-time single camera SLAM[END_REF], [START_REF] Klein | Parallel tracking and mapping for small AR workspaces[END_REF], [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF], [START_REF] Forster | SVO: Fast semi-direct monocular visual odometry[END_REF] [8] [9], [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF] [10], [START_REF] Kerl | Dense visual SLAM for RGB-D cameras[END_REF], [START_REF] Whelan | Elasticfusion: Real-time dense slam and light source estimation[END_REF] (a) [START_REF] Brahmbhatt | Geometryaware learning of maps for camera localization[END_REF], [START_REF] Oliveira | Topometric localization with deep learning[END_REF], [START_REF] Wang | DeepVO: Towards end-toend visual odometry with deep recurrent convolutional neural networks[END_REF], [START_REF] Yin | GeoNet: Unsupervised learning of dense depth, optical flow and camera pose[END_REF], [START_REF] Zhou | Unsupervised learning of depth and ego-motion from video[END_REF] [18], [START_REF] Caruso | Large-scale direct slam for omnidirectional cameras[END_REF], [START_REF] Gutierrez | Adapting a realtime monocular SLAM from conventional to omnidirectional cameras[END_REF], [START_REF] Scaramuzza | Appearance-guided monocular omnidirectional visual odometry for outdoor ground vehicles[END_REF] [22], [START_REF] Labb | RTAB-Map as an open-source lidar and visual SLAM library for large-scale and long-term online operation[END_REF], [START_REF] Zhao | GSLAM: A general SLAM framework and benchmark[END_REF] (b) [START_REF] Kuo | Redesigning SLAM for arbitrary multi-camera systems[END_REF] (c) [START_REF] Im | Allaround depth from small motion with a spherical panoramic camera[END_REF] (c)

[2] (a) at least for learning, (b) as diverging optical axes, (c) as two opposite optical axes. occlusions, illumination changes and low textured (mainly indoors) environments.

Monocular cameras of very wide field-of-view, i.e. 180 deg panoramic or more with a fisheye or catadioptric optics, have also been considered for vSLAM [START_REF] Matsuki | Omnidirectional DSO: Direct sparse odometry with fisheye cameras[END_REF]- [START_REF] Gutierrez | Adapting a realtime monocular SLAM from conventional to omnidirectional cameras[END_REF]. These methods mainly benefit from higher opportunities in catching strong image features than with a conventional camera. And also, from the better conditioning of camera pose estimation that such optics bring. Both characteristics being known to be capable of estimating more reliably trajectories with visual odometry than with a conventional camera [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF]. Following a similar way, recently, vSLAM has been adapted to the combination of conventional cameras [START_REF] Kuo | Redesigning SLAM for arbitrary multi-camera systems[END_REF] as well as the combination of fisheye cameras [START_REF] Sumikura | OpenVSLAM: A versatile visual SLAM framework[END_REF], [START_REF] Im | Allaround depth from small motion with a spherical panoramic camera[END_REF], to reach a field of view of up to 360 deg, i.e. a full spherical field of view.

Among the above related works, some are generic enough in terms of visual feature representation, or in terms of camera projection model to handle natively several types of camera [START_REF] Sumikura | OpenVSLAM: A versatile visual SLAM framework[END_REF], [START_REF] Forster | SVO: Semidirect visual odometry for monocular and multicamera systems[END_REF], [START_REF] Mur-Artal | ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras[END_REF]- [START_REF] Kuo | Redesigning SLAM for arbitrary multi-camera systems[END_REF]. However, to our best knowledge, OpenVSLAM [START_REF] Sumikura | OpenVSLAM: A versatile visual SLAM framework[END_REF] is the only framework that can natively handle RGB and RGB-D modalities, monocular or multicamera setups of conventional or up to 360 deg field-of-view stereoscopic. That is why OpenVSLAM is considered in this paper to evaluate the impact of the camera type on the vSLAM results in term of localization, with the same visual feature type (ORB), estimation method (pose-graph optimization) or implementation.

III. PROPERTIES OF CAMERA TYPES AND ALGORITHMS

As OpenVSLAM is an indirect vSLAM, input frame(s) are pre-processed to extract features. These features are then processed to get 3D points used in the translation and rotation estimation of the current camera pose with respect to the map made of keypoints, whether during the mapping or the localization-only process (when the map is already available). However, depending on the camera type, frames are of different nature and the process to obtain keypoints is different as well. Thus, we briefly recall the different properties and algorithms by camera type as well as their consequences on estimations of both maps and camera poses.

A. Stereo

A stereo configuration links rigidly two monocular cameras. So, features are extracted from the stereo-frame, ie. two input frames, left and right. For each feature of the left frame, its corresponding feature is matched on the right frame assuming stereo rectified frames and using epipolar lines. A keypoint is composed of the left feature coordinate and the horizontal right match [START_REF] Mur-Artal | ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras[END_REF]. Knowing the baseline and the focal length, these keypoints are triangulated to estimate scale. Thus, maps and camera poses are estimated at scale from one stereo-frame.

B. RGB-D

Features are extracted from the RGB frame. For each of these features, the equivalent horizontal right match is obtained by using the corresponding measured depth value [START_REF] Mur-Artal | ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras[END_REF]. Then, as in the Stereo case, maps and camera poses are estimated at scale, but from measured depths, not estimated ones.

C. Monocular

Features are extracted from the RGB frame. Depth information is not observable from a single frame in case of a monocular configuration. It requires an initialization using structure from motion methods [START_REF] Mur-Artal | ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras[END_REF], ie. the triangulation of keypoints is done from several views of which poses are estimated up to scale. As scale is not constrainted by the camera itself, contrary to Stereo or RGB-D cases, the scale can drift over time, only corrected in case of loop-closure [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF]. Hence, maps and camera poses are estimated up to scale.

IV. EVALUATION METRICS

The two main outputs of a vSLAM system are the estimated camera trajectory along with a map building of the environment. Evaluating the quality of the outcome map is a challenging work that involves 3D model of the existing environment, blueprints [START_REF] Watanabe | Robust localization with architectural floor plans and depth camera[END_REF], or topological data in order to build an accurate groundtruth. Therefore, we will use the estimated camera trajectory that the vSLAM system outputs to evaluate its accuracy during both mapping and localization processes.

For the evaluation, we consider two sequences of poses: estimated trajectory P 0 , . . . , P n ∈ SE(3) and groundtruth trajectory Q 0 , . . . , Q n ∈ SE(3). As they may come from different sources such as camera or motion capture system, these sequences may have different length, sampling rates and possibly missing data that require to perform a data association as additional step. This is taken care automatically by evo framework [START_REF] Grupp | evo: Python package for the evaluation of odometry and SLAM[END_REF] based on the common timestamps between the different sources. To simplify the following notations, we assume that sequences are time-synchronized, equally sampled and have the same length. A sequence is a succession of homogeneous transformation matrices of a frame from a reference frame. The considered frame of a monocular camera is often the optical frame. For a stereo rig, the center between optical frames is the first frame of the map used during the acquisition. In case of the motion capture system, the considered frame is arbitrarily deduced from the tracked markers and the reference frame that we chose to be associated to the ground during the calibration process.

In the following part we recall two common evaluation metrics for vSLAM [START_REF] Zhang | A tutorial on quantitative trajectory evaluation for visual(-inertial) odometry[END_REF], [START_REF] Sturm | A benchmark for the evaluation of RGB-D SLAM systems[END_REF]. These metrics must be as low as possible to assess a good result.

A. Relative Pose Error (RPE)

Given a fixed increment ∆ ∈ N * of frames, the relative pose error evaluates the local accuracy of the trajectory over ∆ frames. A common and valid choice is to use ∆ = 1; which means it evaluates the drift per frame. Another choice is to use ∆ = Hz where Hz is the acquisition rate of the camera, therefore it is estimating the drift over one second. To be fair, we use ∆ = 1 for Theta S and ∆ = 2 for Azure and T265 cameras thus considering them at 15 Hz to compare them on the same traveled distance.

The relative pose error for a given time step i is defined as:

E i = (Q i -1 Q i+∆ ) -1 (P i -1 P i+∆ ). (1)
In order to have a single value representing the local accuracy of the trajectory, we compute the root mean squared error (RMSE) of the chosen operator for a sequence of length n (where m = n -∆) as:

RMSE operator (E 0:n , ∆) = 1 m m i=0 operator(E i ) 2 1/2 , (2)
with operator(E i ) defined as follow: 'trans' is the translation part such as operator

(E i ) = ||trans(E i )||.
operator(E i ) could also be 'rot', 'full' or 'angle'[cite here]. We choose 'trans' as single operator as it is already impacted by the rotation error.

B. Absolute Pose Error

The absolute pose error (APE) evaluates the global consistency of the estimated trajectory by comparing the absolute error between the estimated and groundtruth poses over a trajectory. As previously mentioned, the reference frame for different trajectories may be different. We must align them before comparison. This alignment is done by least-squares Fig. 1: 3D printed support holding cameras and markers used by the motion capture system estimation of the rigid-body transformation T between P 0:n and Q 0:n [START_REF] Umeyama | Least-squares estimation of transformation parameters between two point patterns[END_REF]. For a monocular setup, the scale cannot be recovered with vSLAM. An additional step of rescaling the estimated trajectory is necessary. This method requires as input both trajectories and n the number of frames to use in order to compute T . The estimated T slightly changes depending of n. For this study, after proceeding to the data association step, we choose to use the length of the trajectory for n. Once this transformation matrix is estimated, the absolute pose error is defined as:

E ′ i = Q i -1 T P i . (3)
Similarly to the Relative Pose Error (RPE, Sec. IV-A) we compute the root mean squared error of translation, rotation, full and angle components of

E ′ i with (2), substituting E ′ i (3) to E i in (2).

V. EXPERIMENTAL SETUP

A. Data acquisition

All the cameras' data are simultaneously recorded in onego for each acquisition with a laptop on Ubuntu 18.04 and ROS melodic. The cameras were connected to the laptop while the motion capture system, from MotionAnalysis, was running on another Windows 7 desktop computer. The Cortex software provided by MotionAnalysis allows to publish the markers data on a socket. Both computers were on the same local network linked by an Ethernet cable. It allows us to record all the data using ROS capabilities and to have a synchronized dataset between groundtruth and the various cameras we consider in the benchmark. It simplified the later association process to compare the different data within evo framework.

B. Cameras

Table II gathers the specifications of the cameras we used to create this dataset. Figure 1 shows the support made in order to attach rigidly together every camera and markers. Then, we could record every sequence in one-go, bounding experimental biases: every camera follow the same trajectory, up to a constant rigid transformation, at the same pace and sees the same environment with the same light variations. Considered cameras with properties shown in Section III are: computed from stereovision. Despite the rules described later in Section VI-A about the acquisition methodology, D435i RGB-D camera did not allow having maps for any of the runs. Its field-of-view is the narrowest among the ones considered in this benchmark (Fig. 6). Such characteristics lowers the opportunities to sense features in low textured areas of any environment, particularly when the environment is narrow like ours. This is the reason why OpenVSLAM did not produce any result with the D435i. Thus, no mapping nor localization results with it could be shown and exploited to conduct this study.

C. Motion Capture System

For each trajectory we use the MotionAnalysis motion capture system for groundtruth recording. The setup is composed of two different types of Infra-Red (IR) cameras for a total of thirteen IR cameras. There are two Kestrel 2200 and eleven Kestrel Digital IR cameras sharing the same specifications such as a 2048 × 1088 resolution and 332 FPS. The limited number of available cameras and their coverage volume constrain the size of the described environment in V-E. The motion capture system tracks the 3D position of each marker. To meet tracking reliability, a marker is considered detected only if it is seen by at least three cameras. To get the orientation, we placed several markers in an L-shape configuration at the top of the cameras-support. The rotation is computed with respect to the position of left-marker (green marker in Fig 1). A black tape was put on different reflective parts of cameras and support to avoid possible outliers with the motion capture system markers during acquisitions.

D. OpenVSLAM

The vSLAM framework OpenVSLAM is publicly available under 2-clause BSD license. It is implemented in C++ and based on well-known libraries such as Eigen for matrix computation, OpenCV for images manipulation and g2o for map optimization. As previously mentioned (Sec. II), OpenVSLAM allows using various camera models such as perspective, fisheye and equirectangular. RGB-D, perspective and fisheye camera models can be used in a monocular and stereo setups. However stereo-fisheye is still officially experimental. In addition, we defined a mask to remove undesired part of an image before features detection. The reason is that such a mask, see Fig. 2, prevents features detection on cameras support for Theta S camera.

E. Environment

VI. DATA ACQUISITION METHODOLOGY

In this section we introduce the data recorded during various acquisitions used to produce an estimated trajectory with vSLAM. Records are clustered in two groups, one per purpose: mapping or localization. Indeed, we emphasize that some data are dedicated to obtain a map of the environment prior to perform a localization process with the remaining others. Recall that the main idea behind this decoupled process is to be able to navigate later on in the entire known environment to perform tasks (see a practical example in [START_REF] Kheddar | Humanoid robots in aircraft manufacturing: The airbus use cases[END_REF]).

Whereas dense vSLAM algorithms produce maps which are often visually checked by looking at ghost walls or duplicates of the same actual environment, sparse maps of featured-based vSLAM, as OpenVSLAM, makes difficult a similar analysis. Hence, to quantify its quality we use the RMSE trans of APE between the groundtruth and the estimated trajectory at the localization only stage.

A. Mapping

For a given environment, there is an infinite number of possible paths along which one can acquire images to create a map with vSLAM. The path has usually a significant impact on estimations. For example, relatively light variations in altitude along a given path would impact the map [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF].

In this study, the cameras are always about 1.2 meters height. The almost constant height is a reasonable experimental characteristics when targeting mobile robots applications. However, to limit the experimental biases, we imposed ourselves few rules during acquisitions: slow motion, avoid rotation without translation and provoke loop closure by looking at previous recorded places. With this set of constraints in mind we recorded several maps in the environment of Fig. 4, a subset of which is illustrated in Figure 3 as MapA, MapB and MapC. A map is described by a couple (length(m), duration(s)) where length corresponds to the length of the trajectory that the camera did to record the sequences; we have MapA (18.59, 162), MapB (26.94, 93) and MapC (17.52, 120). 1.68, 0.37, -28) respectively. 2) Path: a motion between two stationary poses described by a pair (length(m), ∆θ(deg)). We consider three different paths: A to B (3.48, -7), C to B (1.66, 39) and C to D (4.58, -151). Poses A, B, C and D were defined within the environment in front of several areas of various content. For each case, we record three different sequences in order to evaluate statistically the estimated localization within a pre-built map. We repeat this procedure under three different conditions:

• Nominal: default environment in which there is no variation between mapping sequences and localization sequences; • Lighting: prior to record the images sequences we modify the lighting conditions of the environment by switching off a subset of the available ceiling lights. • Scenery: prior to record image sequences, we moved several objects of various size (Fig. 5). Note that OpenVSLAM is not designed to support these perturbations. However these conditions reflects the possible changes that may happen in any environment. Therefore, they are used in this study to quantify how considered cameras are sensitive to these perturbations.

VII. RESULTS

First, we compare three mapping runs to select the best reference for the purpose of a fair comparison for further localization-only runs.

A. Mapping

Table III shows indirectly the mapping error thanks to the APE (RMSE trans ) computed for the three MapA, MapB, MapC runs (Fig. 3). Before computing errors, estimated and measured camera trajectories are aligned to transform the former in the groundtruth reference frame. Only Theta S estimations are scaled. Maps are not modified to be used as they were estimated during the mapping phase for later localization only. To sum up, the Azure camera has the lowest RMSE trans of APE, followed by the T265 and lastly Theta S among the three runs, each having a path turning at both extremities around A and B. Among the three runs, T265 camera has the smallest deviation whereas both Azure and Thetas S cameras share their lowest error within MapA. Then, MapA is selected to serve as reference for a fair comparison between the various cameras.

B. Localization

Using MapA as input for localization, we obtain estimated poses and trajectories for the localization of each stationary and moving cases described in Section VI-B. Estimations are transformed in the groundtruth frame, for further evaluation, using transformations kept from the map alignments (Sec. VII-A). Table IV

VIII. DISCUSSION

Focusing, first, on vSLAM localization rates in narrow environment under nominal conditions, our study confirms the recommendation of using a wide FoV camera for visual odometry in confined environment [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF]. In our study, Theta S equirectangular camera is the only monocular one but it leads to better localization rates under nominal conditions than both Azure RGB-D and T265 stereo-fisheye cameras (Tab. V). Clearly, the widest FoV allows matching numerous keypoints in the pre-built map, in accordance with [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF]. Furthermore, only Theta S could localize itself at every stationary pose in nominal conditions (Tab. IV).

However, still in the nominal case, Theta S is the best, only regarding localization rates. Indeed, T265 has the smallest RPE trans (Tab. V), showing that a wide FoV alone, even the widest, is not sufficient to get the best local accuracy. The combination of depth estimation (Sec. III) with the wide FoV permits T265 to reach this result. Azure has a depth sensor, but it does not compensate its narrower FoV.

Despite the FoV importance for local accuracy, Azure has the best global accuracy, ie. the smallest APE trans (Tab. IV and V), thanks to its depth sensor. Indeed, T265's depth is estimated (Sec. III), thus more sensitive to calibration and matching errors than Azure. Theta S, after scaling its estimations, has 57.4% worse global accuracy than Azure and 19.6% worse than T265, due to scale drift.

These results and analyses must be put into perspective of the map quality. The APE trans from MapA (Tab. III) is a quantitative indicator of the "quality" of the pre-built map. It is similar to APE trans values in Tab. V thus assessing the direct impact of the mapping on the global accuracy of later localization processes.

Beyond conclusions of [START_REF] Zhang | Benefit of large field-of-view cameras for visual odometry[END_REF], as in nominal case, lighting and scenery conditions (Tab. V) lead Azure to the lowest APE trans , T265 to the lowest RPE trans and Theta S to the highest localization percentage (tied by T265 is 1 case over 9). Overall, Azure is the most sensitive to variations in the scenery whereas Theta S is the most sensitive to lighting variations (Tab. VI). Complementing that, Azure is the less sensitive to lighting condition whereas Theta S is the less sensitive to variations in the scenery. The Time-Of-Flight sensor used to acquire the depth of Azure camera is by nature tolerant to lighting conditions thus making the Azure less sensitive to light variations. However ORB features [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF] are known to be resilient to lighting variations but non-invariant. Therefore T265 and Theta S cameras, by only relying on features to estimate poses, are more impacted by high lighting variations than Azure. The decreasing trend of APE trans (Tab. VI) is correlated to the localization percentage decrease since only localized frames are, obviously, considered in APE and RPE. Theta S is the less impacted by Scenery changes thanks to its wide FoV, allowing to extract more non-altered features to estimate poses than T265 and Azure. Then, the narrowest the FoV, the highest the impact of scenery modifications on RPE trans and percentage localization.

Finally, T265's global accuracy is the least impacted by lighting and scene changes and is 2 nd for the other metrics.

IX. CONCLUSION

In this paper we evaluated the performance of Microsoft Azure (wide FoV and RGB-D), T265 (stereo fisheye), Theta S (dual-fisheye) and D435i (RGB-D) using OpenVSLAM. The environment was chosen to be representative of common robotic applications: an indoor setting where the environment is susceptible to have light variations or changes in the scene after building maps (e.g. indoor production industrial site, homes, offices, hospitals). We quantitatively evaluated localization within pre-built maps using each of these cameras. Results show that a wide field of view improves significantly the localization rate whereas depth information enhances the global consistency. Lighting variations and changes in the scene have less impact on stereo fisheye camera.

For a camera choice, when it comes to vSLAM, stereo fisheye is the best compromise thanks to its wide field of view and its depth estimation.

As future work, this benchmark will be extended to real indoor industrial and hospital setups that are confined environments, considering locomotion while manipulation by a humanoid robot [START_REF] Tanguy | Closed-loop MPC with dense visual SLAM-stability through reactive stepping[END_REF]. Thus, our methodology will be applied in these environments to assess the results while the camera is embedded on a real robot.

Fig. 2 :

 2 Fig. 2: Required mask for Ricoh Theta S camera to remove undesired part for mapping and localization. It occludes 46.2% of the image.

Figure 4

 4 Figure 4 shows the environment created in a specially equipped room of 8 meters by 5 meters with the motion capture system. It emulates a simple corridor along A-B path of approximately two meters of width and five meters of length with two separated areas. Both of them include two separate parts with different feature densities. For example, behind pose C in Fig 4 there is mainly windows and gray planes which may produce few features compared to the zone in front of pose D with several objects. While moving from A to B, the camera motions are almost limited to a straight line in order to not get too close to a wall on each side. This constrained environment implies to have rotation near pose C and B or near pose D and A.

Fig. 3 :

 3 Fig. 3: Top-down views of different done trajectories during acquisitions used to build a map.

Fig. 4 :

 4 Fig. 4: Environment within the motion capture system with poses A, B, C, D and paths illustrated. Each black arrow is the looking direction of the camera for each pose.

Fig. 5 :

 5 Fig. 5: Two areas prior and after scenery modifications (chair, bucket, tables, tables objects, slippers, sofa).

Fig. 6 : 1 )

 61 Fig. 6: Cameras field of view at pose D

TABLE I :

 I vSLAM related works with camera-type versatility.

	Viewpoints	Modality		Field-of-view
	Single Monocular	Multiple	RGB	Depth	Conventional	Hemispherical	Spherical	References

TABLE II :

 II Camera devices Theoretically the FOV is 360°×360°but due to the mask, shown in Fig 2, the FOV is estimated to be 360°×[165;315]°. 1) Intel RealSense T265 stereo fisheye camera. It has a very wide field-of-view. It can be used as a fisheye only or as a stereo-fisheye. 2) Microsoft Azure RGB-D camera. It has the particularity to have a fisheye-depth sensor. The output RGB and Depth images are configurable. We chose to record the wide field-of-view (WFOV) Depth image, 2 × 2 binned aligned on RGB image. 3) Ricoh Theta S, as an equirectangular camera. It has the widest field-of-view. It is considered as a monocular camera. 4) Intel RealSense D435i RGB-D camera. The depth is

	Camera	T265	D435i	Theta S	Azure
	Model	Fisheye	Perspective	Equirectangular	Perspective/Stereo
	Setup	Stereo	RGB-D	Monocular	RGB-D
	Resolution	848 × 800	640 × 480	1920 × 1080	1280 × 720
	Frame Rate	30	30	15	30
	FOV	163±5°69.4°× 42.5°× 77°(± 3°)	360° *	90°× 59°*

  and table V gathers estimation errors shown as the mean of RMSE trans of APE and RPE, respectively written APE trans and RPE trans hereafter, for shortness. APE trans column evaluates the absolute position within the environment. RPE trans column evaluates the position drift over the trajectory. The percentage column beside APE and RPE columns quantifies the percentage of the trajectory estimated by OpenVSLAM with respect to the trajectory recorded in the groundtruth. Table VI gathers percentage changes with respect to Nominal case computed as: 100 * (v condition -v N ominal )/v condition where condition ∈ {Lighting, Scenery} and v ∈ {AP E trans , RP E trans , %}, the closer to 0 it is the better it is.

TABLE III :

 III RMSE trans of APE in meter for MapA, MapB and MapC

		MapA	MapB	MapC
	Azure	0.0977	0.4710	0.1209
	T265	0.1640	0.1694 0.1634
	Theta S	0.1907	0.2401	0.2162
	bold highlights the best result for each camera.

TABLE IV :

 IV Mean of RMSE trans of APE and RPE in meter and localization rate w.r.t. ground truth within MapA for stationary poses

	Element	Camera	APEtrans	Nominal RPEtrans	%	APEtrans	Lighting RPEtrans	%	APEtrans	Scenery RPEtrans
		Azure	-	-	-	-	-	-	-	-	-
	A	T265	0.1961	0.0097 *	88.1868	-	-	-	0.1971	0.0117 *	43.9128
		Theta S	0.3314	0.0073	98.1345	-	-	-	0.3278	0.0088	95.4245
		Azure	0.0651	0.0042	93.8273 *	0.0590	0.0049	93.9026	0.1198	0.0125 *	92.2759 *
	B	T265	0.1676	0.0045 *	97.8788	0.1585	0.0044	97.8540	0.1621 *	0.0049	98.7390
		Theta S	0.1083	0.0045 *	96.5242	0.1020	0.0054 *	91.9277 *	0.1025	0.0053	94.5744
		Azure	-	-	-	-	-	-	-	-	-
	C	T265	-	-	-	-	-	-	-	-	-
		Theta S	0.2978	0.0158	95.0270	-	-	-	0.2700	0.0229	67.6553
		Azure	-	-	-	-	-	-	-	-	-
	D	T265	-	-	-	-	-	-	-	-	-
		Theta S	0.1601	0.0086	87.1713	-	-	-	-	-	-
				bold highlights the best result,

* marks the worst result.

TABLE V :

 V Mean of RMSE trans of APE and RPE in meter and localization rate w.r.t. ground truth within MapA for paths

	Element	Camera	APEtrans	Nominal RPEtrans	%	APEtrans	Lighting RPEtrans	%	APEtrans	Scenery RPEtrans	%
		Azure	0.0974	0.0110 *	81.9243 *	0.0825	0.0098 *	63.0214 *	0.0930	0.0152 *	70.6892 *
	A to B	T265	0.1592	0.0061	99.1146	0.1504	0.0060	72.2806	0.1601	0.0067	91.8179
		Theta S	0.2149 *	0.0077	99.1612	0.1599 *	0.0077	71.6138	0.2250 *	0.0082	98.6805
		Azure	0.0725	0.0083	69.5875 *	0.0786	0.0106 *	58.7756 *	0.0786	0.0098	63.2116
	C to B	T265	0.1681	0.0049	70.0007	0.1765 *	0.0064	60.5449	0.1738	0.0062	61.8020 *
		Theta S	0.1690 *	0.0162 *	90.0909	0.1238	0.0101	61.6005	0.1783 *	0.0171 *	83.0214
		Azure	0.0992	0.0157 *	67.6783 *	0.0849	0.0154 *	70.5691 *	0.1035	0.0259 *	69.1554 *
	C to D	T265	0.1688	0.0085	69.8565	0.1467	0.0089	70.6285	0.1714	0.0141	72.1434
		Theta S	0.2498 *	0.0110	92.0478	0.2003 *	0.0100	71.0494	0.2440 *	0.0135	91.4822
				bold highlights the best,

* marks the worst.

TABLE VI :

 VI Mean of percentage change w.r.t Nominal case within MapA for paths

	Camera	APEtrans	RPEtrans	%
		Lighting
	Azure	-7.10	4.96	-11.45
	T265	-4.54	11.23	-13.16
	Theta S	-24.05 *	-15.58 *	-27.41 *
		Scenery
	Azure	2.74 *	40.41 *	-6.90 *
	T265	1.83	34.08	-5.27
	Theta S	2.63	11.59	-2.98
	bold highlights the min,

* highlights the max.


Barbara Mcgillivray

In spite of the increasingly large textual datasets humanities researchers are confronted with, and the need for automatic tools to extract information from them, we observe a lack of communication and diverging goals between the communities of Natural Language Processing (NLP) and Digital Humanities (DH). This contrasts with the wealth of potential opportunities that could arise from closer collaborations. We argue that more efforts are needed to make NLP tools work for DH datasets so that that NLP research applied to humanities data receives more attention, leading to the development of evaluation approaches tailored towards relevant research questions. This has the potential to bring methodological advances to NLP, while at the same time confronting DH datasets with powerful state-of-the-art techniques.

Introduction [1] The recent years have witnessed an increased interest in Digital Humanities (DH) textual datasets within the Natural Language Processing (NLP) community, as several initiatives (such as the Computational Humanities group at Leipzig [2] and the Computational Humanities committee), [3] workshops (such as Computational Humanities 2014, [4] Teach4DH, [5] COMHUM 2018, [6] and the various editions of the LaTeCH-CLfL workshops), [7] and publications testify to [START_REF] Biemann | [END_REF]] [Nyahn and Flinn 2016] [START_REF] Jenset | Quantitative Historical Linguistics. A corpus framework[END_REF]McGillivray 2017] [van der Zwaan et al. 2017] [Bamman 2017] [Schulz 2018] [Hinrichs et al. 2019]. Research in this area has focussed on developing new computational techniques to analyse and model humanities data. This interest stems from the methodological and technical challenges posed by these datasets, including those related to non-standard textual or multi-modal input, historical languages, multilinguality, and the need for advanced methods to improve the quality of digitization and for semi-automatic annotation tools. A number of successful results have been achieved, especially in the area of handwriting recognition [START_REF] Christlein | Handwritten Text Recognition Error Rate Reduction in Historical Documents using Naive Transcribers[END_REF]], computational stylistics [START_REF] Boukhaled | [END_REF]], historical natural language processing pipelines [START_REF] Piotrowski | [END_REF]], authorship attribution [START_REF] Stamatatos | A survey of modern authorship attribution methods[END_REF]], and semantic change detection [Tang 2018], to name a few examples.

In spite of this growing activity, there is a real danger that NLP research on DH datasets does not take into account all the complexities of the phenomena and corpora, as others have pointed out [START_REF] Dubossarsky | Outta Control: Laws of Semantic Change and Inherent Biases in Word Representation Models[END_REF]] [START_REF] Hellrich | Bad Company -Neighborhoods in Neural Embedding Spaces Considered Harmful[END_REF]. Moreover, NLP work on DH data has often been confined within the limits of the NLP community, which leads to serious methodological limitations for its applicability to DH and humanities research. In spite of the huge potential impact of NLP for DH datasets, NLP activities aimed at applying and adapting NLP research to the needs of the humanities are still marginal. This can be explained by the standard processes that the discipline adopts. Because the emphasis is on developing new computational systems or improving existing ones, it is very important that these are evaluated on standard datasets using reproducible methods. This means that there is an incentive for NLP researchers to work on very restricted sets of datasets and languages, leading to the development of tools which are optimized for those datasets and languages. This drives research towards a very specific direction, away from the idiosyncratic features displayed by historical languages and DH data. Moreover, publication venues dedicated to NLP methods for DH are few and do not set the mainstream agenda of the field. Coupled with the challenges and the effort required to work on DH datasets, this means that engaging with this line of research appears to be a less than attractive option for most scholars.

www.digitalhumanities.org/dhq/vol/14/2/000454/000454.html# 2/11

On the other hand, a large part of humanities research involves analysing and interpreting written texts. Over the past few years large digital text collections have become available to the scholarly community, but where DH scholars confront Big Data to answer humanities questions, they often rely on methodologically un-sophisticated tools such Google Books Ngram Viewer [Greenfield 2013]. There is a real danger that these non-scientifically rigorous approaches will become state of the art [START_REF] Pechenick | Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution[END_REF].

In this article we aim to draw attention to the lack of communication between the communities of NLP and DH. In spite of its damaging effect on the progress of the disciplines, we believe this lack of communication and miscommunication are underestimated. We argue that what is needed is to bridge the gap between the highly technical state of the art in NLP and the needs of the DH community. We also offer a solution to this situation, inviting DH researchers to play a more active role in making NLP tools work for their data in order to give new insights into their questions, while at the same time advocating for a higher profile of NLP research applied to humanities data. Institutions also need to play a role in enabling better communication between the two communities, promoting interdisciplinary work and multi-author publications; publication venues welcoming such NLP/DH collaborative research also have an important role to play.

Contexts

An informal definition of the scope of DH was given by Fitzpatrick commenting on the DH 2010 conference, as "a nexus of fields within which scholars use computing technologies to investigate the kinds of questions that are traditional to the humanities [...] or who ask traditional kinds of humanities-oriented questions about computing technologies" [Fitzpatrick 2010]. Though informal, this broad characterization agrees with the variety of work described as DH in overviews of the field [START_REF] Berry | Understanding Digital Humanities[END_REF], 1-20] [Schreibman et al. 2004].

More recently, some authors [Biemann et al. 2014, 87-91] Humanities" as situated in a continuum between the humanities or the DH (according to their definition of the latter term)

and Computer Science. Therefore, should we want to adopt the Digital vs. Computational Humanities terminology sometimes proposed, the work referred to here can be considered within the Computational Humanities. However, in the rest of this paper we will use the more widely adopted term of "DH". In 2019, a heated debate emerged around the role of computational analysis in the study of literature specifically, after the publication of Nan Da's paper, which questions whether the computational analysis of literary texts can bring any additional insight in literary studies [Da 2019]. Counterarguments to such claims were offered by computational literary studies researchers [Critical Inquiry 2019] [Cultural Analytics 2020]. We do believe that computational methods can contribute to literary text analysis, and

cite some examples related to character identification below. We agree, however, with Da's emphasis on the need to use computational tools optimally [Da 2019, 604]. As we argue below, this can involve considerable work in order to adapt to the specifics of a DH-relevant dataset, going beyond the tools' default configuration and requiring at times novel evaluation procedures.

Data relevant for social sciences and humanities research often take the shape of large masses of unstructured text, which is impossible to analyze manually. For example, regarding the use of textual evidence in political science, a variety of relevant text types have been identified, such as regulations issued by different organizations, international negotiation documents, and news reports [START_REF] Grimmer | Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts[END_REF]. Grimmer and Brandon conclude that "[t]he primary problem is volume: there are simply too many political texts". In the case of literary studies, the complete text of thousands of works spanning a literary period [START_REF] Clement | How Not To Read A Million Books[END_REF]] [Moretti 2005, 3-4] are beyond a scholar's reading capacity, and researchers turn to automated analyses that may facilitate the understanding of relevant aspects of those corpora.

Because DH researchers now face volumes of data that cannot be analyzed manually, NLP technologies need to be applied and adapted to specific use cases, integrating them in user interfaces to make the technology more easily usable by domain experts from the humanities and social sciences. Besides, a critical reflection on the computational tools and methods developed must be initiated, based on an evaluation by domain experts who are expected to benefit from those technological means.

We argue that researchers in the social sciences and humanities need ways to gain relevant access to large corpora.

NLP can help provide an overview of a corpus, by automatically extracting actors, concepts, and the relations between them. However, NLP tools do not perform equally well on all texts and may require adaptation. Furthermore, the connection between these tools' outputs and research questions in a domain expert's field may not be immediately obvious and needs to be made explicit and kept in mind in the development of computational tools. Finally, evaluating the usefulness of an NLP-based tool for a domain expert is not trivial and ways to enable accurate and helpful evaluations need to be devised.

Different Datasets

Research in NLP aims to build tools and algorithms for the automatic processing of language [START_REF] Jurafsky | Speech and Language Processing: An Introduction to Natural Language Processing, Speech Recognition, and Computational Linguistics[END_REF]. In NLP, such systems are typically evaluated against baseline and existing systems with the aim to improve various measures of accuracy, coverage, and performance. Because the focus is on developing optimal algorithms, it is common practice to build and evaluate them based on existing, standard corpora. This way, it is possible to compare different approaches in a systematic way.

In the case of DH research, however, the focus is not so much on the algorithms as on the results that they lead to, which help the researchers answer their research questions. In this context, each study tends to focus on a specific and often unique dataset, with an emphasis on achieving satisfactory and insightful results using or adapting existing methods, if possible.

Several differences separate corpora typically used in NLP research from such DH datasets. First of all, size is typically very large in the former case, unless it is a particular aim of the research to optimize algorithms for small datasets.

Moreover, with the exception of the cases in which the NLP research is focussed on a specific domain (such as medical, legal, etc.), balanced corpora are typically used. This ensures that the systems developed on such corpora can be generalized to the language as a whole, in line with generally accepted definitions of a corpus as "a collection of pieces of language text in electronic form, selected according to external criteria to represent, as far as possible, a language or language variety as a source of data for linguistic research" [Sinclair 2004]. In DH research, on the other hand, if answers to more general questions are sought, they rarely concern linguistic phenomena per se, and the aim of generalization is instead replaced by the need to describe and further explore the datasets at hand. Sometimes these take the form of archives, which, unlike corpora, are not collected with the aim to represent a language variety. Therefore, it may not be an option to modify the size of the corpus without changing the scope of the research, as the research question is bound to a specific set of texts. In the case of historical languages, for example, unlike current languages, it is often not possible to increase the size of the data because the amount of transmitted texts is limited by particular historical circumstances [McGillivray 2014, 15].

DH studies normally do not involve balanced datasets according to criteria such as genre, style, register, etc. Another important difference concerns the quality and format of the texts, with particular reference to historical texts. Because almost every DH study requires its own dataset, a necessary preliminary step of the research process consists in acquiring the texts, if they are not already available. When Optical Character Recognition (OCR) is chosen, the texts will likely require a significant amount of processing before they reach an acceptable level of quality, otherwise there is a serious risk that standard NLP tools will fail to provide satisfactory results [START_REF] Piotrowski | [END_REF]]. This is sometimes complemented with other challenges inherent to historical texts and which have to do with diachronic variation, including spelling variation and other types of language change. This means that a diachronic dimension needs to be included in the text processing when applying NLP tools developed for current languages or based on synchronic corpora. We argue that DH corpora can usually be used without the privacy concerns of user-generated data. At the same time, the challenges that come with them offer a good test case not only to pursue meaningful DH questions, but also to measure the robustness of state-of-the-art algorithms.

Natively digital texts are not exempt from the caveats above. A case illustrating this is the work on the Earth Negotiations Bulletin (ENB) corpus [START_REF] Ruiz | More than Word Cooccurrence: Exploring Support and Opposition in International Climate Negotiations with Semantic Parsing[END_REF]] [Ruiz 2017]. ENB's volume 12 consists of reports summarizing participants' statements at international climate negotiation summits, where global climate agreements are handled. The goal of the study was to automatically identify where participants stand with respect to negotiation issues (support or opposition) as well as regarding other participants, thanks to syntactic and semantic role annotations obtained with an NLP toolkit. The corpus covers a period starting in 1995, and it contains what we might call "historical HTML varieties", as well as fixed column-width plain-text formats that required normalization before the content can be input to the NLP pipeline. In addition to this type of normalization, language use in the corpus has several non-standard traits, for which it was necessary to adapt the NLP toolchain (see section 4 below). In other respects, this corpus displays some of the typical features of DH datasets that we described above: it does not seek to represent a linguistic phenomenon, instead it is the set of texts required to answer domain-specific research questions. Venturini et al. argued that the corpus is a good choice to study climate negotiations, given that the corpus editors strive to cover participants in a balanced way, using a neutral and controlled language, although its specific linguistic characteristics need not generalize to political reporting texts [START_REF] Venturini | Three maps and three misunderstandings: A digital mapping of climate diplomacy[END_REF]].

www.digitalhumanities.org/dhq/vol/14/2/000454/000454.html# 4/11

When the texts are already available to the DH community, they are often encoded according to the Text Encoding Initiative (TEI) markup, which has become a standard in DH. This markup focusses on editorial aspects and structural properties of the texts themselves, and less on their linguistic features. This contrasts with the state of the art in NLP, where corpora typically have shallow metadata information and are often rich in linguistic annotation [Jenset and McGillivray 2017, 124-125, 137-138]. Some linguistic corpora using TEI are however available, e.g. the National Corpus of Polish [START_REF] Bański | Stand-off TEI Annotation: The Case of the National Corpus of Polish[END_REF]Przepiórkowski 2009] [Przepiórkowski 2009]. Although traditions have diverged since then, at its inception the TEI was sponsored by both DH-related and NLP-related professional associations [Cummings 2007]. Also, major cultural works encoded in TEI have been annotated with morphosyntactic information, e.g. Dante's works [Dante Search 2018]. Therefore, extra processing and particular attention is required when employing NLP tools on such texts.

NLP Techniques in DH and Their Challenges

The DH research process can sometimes include an NLP pipeline, such as sentence segmentation, tokenization, lemmatization or stemming, or part-of-speech tagging. These steps are often required in order to conduct further analyses, either because they contribute to "cleaning" the texts, or because they help the researchers identify individual linguistic elements of interest such as word tokens, or classes such as parts-of-speech. This is often complemented by the use of corpus linguistics techniques, such as collocation analysis or various quantitative analyses of linguistic elements. In other words, the boundary between computational and corpus linguistics blurs in the interest of the DH research questions [Jenset and McGillivray 2017, 101].

Partially related to and building on the previous point, another way in which NLP methods are used in DH research concerns those techniques that extract various types of structured information from texts, including keywords, named entities, and relations. Again, these steps usually precede further analysis, in the form of qualitative or quantitative investigations, and can rest on other levels of linguistic processing, such as syntactic or semantic parsing.

Semantic processing can also support the identification and analysis of more abstract entities, such as semantic fields or concepts, and their linguistic realization in texts. Some types of information are generally useful to understand a corpus. These include actors mentioned in it (e.g. people, organizations, characters), core concepts or notions of specific relevance for the corpus domain, as well as the relations between those actors and those concepts. This is a critical element in the analysis of the ENB, the climate diplomacy corpus mentioned in section 3 [Ruiz 2017]. As the corpus covers international negotiation reports, in order to understand its content it is essential to know not only which concepts were discussed in the negotiation, but also who discussed them and with which attitude (in a supporting or oppositional manner). Syntactic dependencies, semantic role annotations and coreference chains obtained with an NLP toolkit were exploited to that end. The agent of a reporting predicate was identified as a negotiation actor, and the message tied to that reporting predicate was considered to express negotiation issues addressed by the agent. The predicate itself (a reporting verb or noun) was seen as expressing the actor's attitude. The corpus has some non-standard linguistic features, and the NLP toolchain had to be adapted to handle them. For instance, personal pronouns "he", "she" can refer to countries in this corpus, so anaphora resolution had to be modified accordingly. Corpus sentences contain information about several participants at the same time (multiple actors mentioned within the agent role, or in adjunct roles), which also required a specific treatment in order to identify as many relations as possible of the type ⟨actor, predicate, concept⟩. A good coverage of such relations is crucial to analyses on negotiation behaviour relevant to domain experts.

A widespread approach to gain an overview of a corpus relies on network graphs called concept networks, social networks or socio-technical networks, depending on their content [Diesner 2012, 5, 84]. In such graphs, the nodes represent terms relevant in the corpus (actors and concepts), and the edges represent either a relation between the terms (like "support" or "opposition"), or a notion of proximity between them, based on overlap between their contexts.

Creating networks requires a method to identify nodes, as well as a way to extract relations between nodes or to define node proximity, such as a clustering method. Networks have yielded very useful results for social sciences and humanities research. Established techniques to extract networks from text exist, and networks offer useful corpus navigation possibilities.

However, NLP can complement widespread methods for network creation. Sequence labeling and disambiguation techniques like Entity Linking can be exploited to identify the network's nodes: actors and concepts. The automatic definition of network edges is usually based on node co-occurrence, while more detailed information about the relation between actors and concepts is not usually automatically identified for defining edges. Nonetheless, such information can also be obtained via NLP methods. As for corpus navigation, networks do not in themselves provide access to the corpus fragments that were used as evidence to create the networks; but they can be complemented with search workflows that allow researchers to access the contexts for network nodes and the textual evidence for the relations between them.

The above-mentioned techniques are, in most cases, considered solved problems in NLP research, and are normally developed and tested on a set of large standard synchronic corpora of current languages. However, when applied to "messy", noisy, and/or historical texts, these tasks prove to be significantly more challenging. This points to the need of more research to bridge the gap between the DH and NLP communities, so that NLP tools can be developed with the requirements of the former in mind. In this context, Perrone et al. [START_REF] Perrone | GASC: Genre-Aware Semantic Change for Ancient Greek[END_REF]] develop further a Bayesian learning model for semantic change detection developed by Frermann and Lapata [START_REF] Frermann | A Bayesian Model of Diachronic Meaning Change[END_REF].

Frermann and Lapata's original system was built with the aim to model the change in meaning of English words in a general corpus covering the time period 1700-2010. Perrone et al. extend this to the case of Ancient Greek. This extension is not trivial, as the differences between the two corpora are substantial. The Ancient Greek corpus covers 13 centuries and its content is highly skewed, as certain centuries are only represented by one author or very few works, and some genres only appear in certain centuries. This lack of balance means that the models need to account for the differences in the texts' features and their effect on word semantics. For example, the sense of a word is highly dependent on the genre of the text it appears in. The Ancient Greek word mus can mean "muscle", "mussel", and "mouse", but in medical texts is it more likely to mean 'muscle', independently of the era of the text. As a result, Perrone et al. modified the original system to incorporate genre information into the statistical model, thus achieving improved accuracy of the system compared to the English case.

Applying NLP for text analysis in social sciences and humanities poses some specific challenges. First of all, researchers in these domains work on texts displaying a large thematic and formal variety, whereas NLP tools have been trained on a small range of text types, e.g. newswire [Plank 2016]. Second, the experts' research questions are formulated using constructs relevant to their fields, whereas core tools in an NLP pipeline (e.g. part-of-speech tagging or syntactic parsing) provide information expressed in linguistic terms. Researchers in social sciences, for example, are not interested in automatic syntactic analyses per se, but insofar as they provide evidence relevant for their research questions: e.g. which actors interact with each other in this corpus, or which concepts does an actor mention, and which attitudes are shown towards those concepts? Adapting tools to deal with a large variety of corpora, and exploiting their outputs to make them relevant for the questions of experts in different fields is a challenge in itself.

Evaluation and Usability

In this section we first discuss the mismatch between the evaluation carried out in NLP and the needs of DH scholars in terms of tool evaluation and tool performance. Then, we present an example of a DH-relevant evaluation approach.

In the same way that exploiting NLP technologies to make them useful to experts in social sciences and humanities is challenging, evaluating the application of NLP tools to those fields also poses difficulties. A vast literature exists about evaluating NLP technologies using NLP-specific measures. However, these NLP measures do not directly answer questions about the usefulness for a domain expert of a tool that applies NLP technologies. Even less do they answer questions about potential biases induced by the technologies (e.g. focusing only on items with certain corpus frequencies), and how these biases affect potential conclusions to draw from the data [Rieder and Röhle 2012, 77] [ Marciniak 2016]. As Meeks et al. state, research is needed with "as much of a focus on what the computational techniques obscure as reveal" [START_REF] Meeks | The Digital Humanities Contribution to Topic Modeling[END_REF]].

Let us take the example of a researcher interested in the analysis of characters in different novels. Named-entity recognition is an interesting application for the task, but existing tools are not always very accurate with fiction texts. Moreover, named entities are not enough: the system must probably be coupled with an anaphora recognition and resolution system, and with other modules able to recognize titles and occupations, for example, as some characters may just be mentioned by the title they have. corpus, and evaluating this is a hard and, above all, time-consuming task for computational humanists. The characterdetection papers just cited, accordingly, developed task-specific corpora.

To take another example, McGregor and McGillivray describe a computational semantics system for information retrieval from historical texts, the Medical Officer of Health reports from London for the years from 1848 to 1972 [START_REF] Mcgregor | A distributional semantic methodology for detecting implied smell in historical medical records[END_REF]. The ultimate aim of this research was to answer a specific question in medical history, namely the nature of the relationship between smell and disease in 19th-century London. Therefore, the computational system had to be evaluated in the context of the original research question and according to metrics that assessed its suitability for the specific task at hand, which was the retrieval of smell-related sentences in a specific large collection of historical health reports, rather than on standard NLP metrics and approaches.

As we said in section 2, evaluation procedures for NLP tools are typically focussed around the aim to improve the state of the art. In the case of DH research, the objective is connected to a set of research questions, which are typically not methodological or linguistic. A computational system may perform very well according to standard NLP evaluation measures, but it is not usable in DH if it does not help the researchers answer their questions. Moreover, the converse scenario has also been attested: a system that attains low scores in an NLP task may prove useful for a DH application.

An example of the latter case is automatic keyphrase extraction. A standard evaluation task took place within the SemEval campaign [START_REF] Kim | Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles[END_REF]]. The best systems reached an F1 measure below 0.3. These scores in themselves may seem unimpressive, the possible range being between 0 and 1. However, keyphrase extraction is routinely applied in order to get an overview of a corpus in DH research [START_REF] Moretti | ALCIDE: Extracting and visualising content from large document collections to support humanities studies[END_REF][START_REF] Rule | Lexical shifts, substantive changes, and continuity in State of the Union discourse, 1790-2014[END_REF], which suggests the usefulness of this technology for corpus-based research in the humanities. The way the NLP task was defined at SemEval (a keyphrase extracted by the candidate systems had to exactly match a keyphrase in the reference set annotated by human experts for it to count as correct) does not fully reflect the way the technology is useful for a corpus overview in DH tasks, where inexact matches can still be useful. In other words, a tool's performance in a standard NLP competition like SemEval and the tool's performance with a DH-relevant corpus need not be related.

In the same way that we encourage humanities researchers to accept automatic textual analyses as complementary to manual ones, we would also like to insist on the importance of understanding the limits of computational tools. Initiatives in this direction have emerged, like Tool Criticism [START_REF] Traub | Workshop on Tool Criticism in the Digital Humanities[END_REF], which seeks to understand biases introduced by tools and digital methods on a task's results.

We will now review an example of evaluation relevant to a specific DH task [Ruiz 2017]. That study evaluates a corpus navigation application for the ENB corpus introduced above. The application relies on relation extraction, based on an NLP pipeline adapted to the corpus. Based on the pipeline's results, a user interface (UI) allows users to search separately for actors, their statements in climate negotiations, and their attitudes towards negotiation items or other actors (support or opposition). First of all, an NLP intrinsic evaluation for relation extraction was performed, comparing automatic results with human reference annotations. The difference here is that, additionally, a qualitative evaluation was carried out, based on interviews of over one hour with three domain-experts familiar with the corpus: two of them had previously published research on it, and one of them works at the organization that produces the corpus reports.

The goal of the evaluation was threefold. First, to assess to what extent the corpus navigation application developed for the ENB corpus helps experts obtain an overview of its content (i.e. an overview of actors' behaviour in the negotiations). Second, whether the tool can help experts gain new insights on the corpus: facts unknown to them that emerge from using the tool, or new research ideas made apparent to them by using the tool. Finally, another goal was to see if the quality (in F1 terms) of the NLP-based system was sufficient for this specific application. [8] As an evaluation protocol, the experts were first shown the corpus exploration functions available to them on the UI, following the same steps with each expert. The experts were then asked to come up with questions about the corpus, use the interface to obtain information relevant to their questions, and comment on the results. As said above, it was assessed whether the UI helps them gain an overview as well as new insights into the corpus. The assessment was based on verbal evidence (expert comments) or other behavioural evidence (queries and other operations performed on the UI). The sessions were recorded and later transcribed non-verbatim: expert comments were summarized, and a description of operations performed by users on the UI was written up; the reports and session audios are publicly available. Rather than asking experts explicitly for feedback, their spontaneous comments were considered as possible evidence for or against the usefulness of the system in terms of corpus overview and insight gain.

The results that emerged from this qualitative evaluation highlight those aspects of the NLP-based system that are useful or pose problems for domain experts. Dynamic extraction of speakers as agents of reporting predicates, without relying on a predefined speaker list, brought to light statements by lesser-studied negotiation participants, that our experts had no data about, like NGOs and interest groups on climate and gender or climate and indigenous peoples. A perceived shortcoming of the system was its incomplete treatment of complex predicates like "express concern", where the informative part of the predicate regarding the speaker's attitude to a negotiation issue is the noun "concern" rather than the verb "express". Such results, obtained not from researcher expectations but from actual user feedback, can help shape improved versions of the system, and even identify generalizable areas of improvement for an NLP technology itself. [9] The main point here is that a simple evaluation task as the one we just described can be informative in ways an intrinsic quantitative NLP evaluation would not be. However, note that tasks like the one just described involve several challenges, and the system just described could be improved. A first difficulty is that, as reported before [START_REF] Khovanskaya | Double binds and double blinds: evaluation tactics in critically oriented HCI[END_REF], study participants often form an opinion about the intended contribution of a study, and they believe that it helps the research if the experiment provides evidence for their expected result. These beliefs can bias participants' behaviour. The attempt by Ruiz et al. to reduce this bias relied on avoiding to ask participants for explicit feedback, but rather on recording their comments and behaviour, although it is debatable whether such bias decreases this way. A second difficulty is that collecting domain-expert feedback in the way described is time-consuming; even finding suitable experts may be difficult, hence the small number of participants in the study. Finally, as the NLP-based system was exploited via a UI, the task could be complemented by an evaluation based on Human Computer Interaction criteria, like usability or user satisfaction [Al-Maskari and Sanderson 2010] [ Kelly 2007]. Such an evaluation would involve defining tasks to perform with the UI, employing a larger sample of domain experts. The proportion of tasks completed successfully could be measured, as well as task completion time; a questionnaire could measure user satisfaction. All in all, even the simple small-sample qualitative evaluation task above was informative about concrete aspects of an NLPbased system that helped or were an obstacle to answering specific questions relevant to a DH-research task.

Conclusion

We would like to end this paper with a summary of the status quo and some recommendations for the future.

As we have seen, many DH projects are based on large or even very large textual corpora. Sometimes it is not possible for the experts to know all the texts included in their corpus, and in some other cases the corpus is complex and a specific interface would help and quicken the analysis. In such contexts, it makes sense to use advanced NLP techniques in DH research. NLP tools are now mature and accurate enough to provide, in principle, reliable and extended analyses of various corpora in literature, history, and other text-focussed humanities fields. It is possible to annotate entities (people and locations, companies and institutions), semantic concepts, technical terms and domainspecific expressions. It is also possible to extract links between entities, and derive a network of relations from the information included in the texts in an unstructured form. It is even possible to represent relevant information through navigable maps, so that the end user can navigate the corpus and find relevant pieces of information scattered in different texts.

However, despite the accuracy and robustness of current NLP techniques, these are not yet widely used in DH. And, on the other hand, even if we see a growing interest for social sciences and cultural heritage in the NLP community, this is still quite marginal. The main issue is probably that the two communities are fundamentally driven by opposite goals.

The NLP community is interested in advancing NLP techniques, which means every published experiment must be evaluated and compared to previous work and show some improvement over it. The use of gold standards is thus of paramount importance to perform this comparison. The drawback is a large standardization of the research effort: many researchers explore broadly the same methodological avenues, with the same techniques applied on the same data. In fact, the NLP community tends to focus on one paradigm at a time and to produce a phase of homogeneous research questions and methods, sometimes at the expense of leaving out potentially interesting but less mainstream work. Today it is hard to publish research in an NLP venue that does not use word embeddings, neural networks and deep learning. Deep learning methods have had a huge impact on the field, leading to a clear (and sometimes dramatic) improvement in performances for almost all kinds of NLP tasks. However, we believe diversity of methods and critical approaches to their use can only be beneficial.

On the other hand, the DH community, generally speaking, is only secondarily interested in processing techniques. The goal of DH is to shed new light on humanities problems, taking into account the advantages of digital and computerized methods. Evaluating processing techniques is a challenge, since data are by definition always project-specific and there are no gold standards in the same sense intended in NLP. Tools are thus used as they are ("off-the-shelf"), sometimes after a brief evaluation and estimation of their quality and adequacy for a given task, sometimes without any evaluation. are necessary to be able to train a new model or, in other words, to adapt the system to the corpus under study. This means that even when the team is skilled enough to adapt a system, this adaptation is not always possible in practice.

All this explains the current situation and maybe some of the missed opportunities. At this point, one conclusion could be that the two communities are just different, they have different goals and, even if we can observe some points of contact, these are marginal and not so interesting from a scientific point of view. But in fact we would like to defend the exact opposite view.

DH offers new, original and complex challenges to the experts, and these challenges require new, original and complex techniques to tackle them. DH research also offers real-world problems that are needed to prove that NLP techniques can be applied to different languages than English, to different corpora than newspapers and to different periods than just the 21st century. At the same time, NLP tools and methods are often underutilized and a more accurate choice of the techniques to use can have a strong impact on DH research. In fact, we argue that adapting and tuning NLP techniques to the corpus or domain under study is precisely where some of the most challenging, innovatively interesting and impactful research can be.

The dialogue between disciplines and the constructive and critical use of methods that we advocate for implies that most projects need a multidisciplinary approach. We acknowledge that more and more institutions support this interdisciplinary and multidisciplinary research and even create new job positions in this space, often spanning over several departments. However, more needs to be done to properly support interdisciplinary careers. An academic culture that favours single-author publications does not sufficiently support multidisciplinary (and multi-authored) research. Institutions should acknowledge this situation, which means promoting multi-author publications, and also publications related to different domains, ranging from computer science to traditional areas of the humanities.

There is a growing number of papers mixing DH and NLP presented at conferences such as the annual Digital

Humanities conference or in the series of LaTeCH workshops (Language Technology for Cultural Heritage, Social Sciences, and Humanities). This indicates that there is a community, which sometimes calls itself Computational Humanities and includes the community of NLP for DH, but also research around techniques dedicated to image, video, sounds and music, etc. However, the insufficient number of high-profile publication venues where such research can be hosted penalizes scholars active in this space, as they are less likely to be considered "successful" by academic standards. One possible solution to this is that humanities departments recognize publications in computer science venues as equally valuable as publications in venues considered more traditionally humanistic.

In addition to these academic cultural changes, we believe that changes to research practices and publication standards can support the work at the interface between NLP and DH. We have stressed that off-the-shelf algorithms are often blindly applied, but it is hard to benchmark an algorithm's performance on a DH corpus due to its unique features. We propose that reproducibility and algorithmic robustness checks are added to all NLP/DH publications, to strengthen the methodological foundations of the research results. For example, if a DH paper uses LDA topic modeling, a requirement for publication should be that the authors run the analysis using differing values of the parameter for the number of topics (k) and differing pre-processing steps.

After stressing the challenges (and some possible solutions to them), we would like to end this paper on a positive note, highlighting the interest of the research at the frontier between these domains and the opportunities it brings. The state of the art in both fields is advanced enough to contemplate combined approaches, research opportunities happen at a global scale, ethical considerations are a priority and the potential negative impact of the increasing use of artificial intelligence in propagating fake news, for example, is widely recognized. NLP for DH offers a unique opportunity to explore complex data, and to show how we can deal with complexity to get a better knowledge of our past.

By increasing the recognition and support of computational scholars within DH, NLP scholarship can become an attractive area of DH, thus creating space for scholars who might otherwise hesitate to go into DH due to poor job prospects. Just as the social sciences have successfully created space for these kinds of scholars, which has benefited the social sciences overall, DH could achieve similar results by adopting a similar strategy.

  For example, McGregor and McGillivray report on a methodology for extracting explicit occurrences of smell in historical medical records, the Medical Officer of Health Reports collected in London in the 19th and 20th century [McGregor and McGillivray 2018].Given the large size of the dataset, automatic detection of instances of smell-related words by drawing on the distributional semantic properties of a small set of seed words enables medical historians to extract relevant passages in the texts, which can then be further analyzed, for instance geographically or diachronically.

Acknowledgements

We would like to thank the anonymous reviewers for their thorough and helpful comments. Thierry Poibeau's research is partially funded by the PRAIRIE 3IA Institute, part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001. This work has been mainly done while Thierry Poibeau benefited from a Rutherford fellowship at the Turing Institute (London and University of Cambridge). This work was supported by The Alan Turing Institute under the EPSRC grant EP/N510129/1.