Elodie C T Descamps

Damien Meunier

Catherine Brutesco

Sandra Prévéral

Nathalie Franche

Ingrid Bazin

Bertrand Miclot

Philippe Larosa

Camille Escoffier

Jean-Raphaël Fantino

Jean-Raphael Fantino

Daniel Garcia

Mireille Ansaldi

Agnès Rodrigue

David Pignol

Pierre Cholat

Nicolas Ginet
email: nginet@imm.cnrs.fr

Semi-autonomous inline water analyzer: design of a common light detector for bacterial, phage, and immunological biosensors

Keywords: Bioluminescence, Chemiluminescence, On-site water analyzer, Biosensors, Bacterial pathogens, Metals, Toxins

come

Introduction

Environmental protection requires efficient and rapid methods to detect accidental or intentional spill of toxic compounds in waters as well as pathogenic bacterial contamination. Analytical methods such as mass spectrometry commonly used to detect organic or metallic compounds are very sensitive but usually performed in specialized laboratories, not in real time on field. Similarly, current detection methods of living pathogenic bacteria in environmental samples require standard clinical cultures, incompatible with rapid, on-site measurements [START_REF] Rompré | Detection and enumeration of coliforms in drinking water: current methods and emerging approaches[END_REF]. Biosensors integrated into inline water monitoring devices represent promising and complementary alternatives that could offer sensitive and rapid detection on-site used to quickly raise the alarm [START_REF] Michelini | Fielddeployable whole-cell bioluminescent biosensors: so near and yet so far[END_REF]). Significant advances have been reported in the design of whole cell-based biosensors used to detect heavy metals with genetically modified bacterial cells; these improvements concern higher sensitivity and selectivity [START_REF] Daunert | Genetically engineered whole-cell sensing systems: coupling biological recognition with reporter genes[END_REF][START_REF] Cayron | Pushing the limits of nickel detection to nanomolar range using a set of engineered bioluminescent Escherichia coli[END_REF] in this Special Issue), robustness and rapidity using cells in a dormant state until use [START_REF] Camanzi | Optimal conditions for stability of photoemission and freeze drying of two luminescent bacteria for use in a biosensor[END_REF]Prévéral et al. 2016 in this Special Issue) or adaptation to the analysis of samples in complex matrices using optimized bioluminescent reporters [START_REF] Winson | Engineering the luxCDABE genes from Photorhabdus luminescens to provide a bioluminescent reporter for constitutive and promoter probe plasmids and mini-Tn5 constructs[END_REF][START_REF] Brutesco | Bacterial host and reporter gene optimization for genetically encoded whole cell biosensors[END_REF] in this Special Issue). However, progresses are still required to translate such biosensing technologies from laboratory bench tops to technological devices offering easy-to-use, cost-effective on-site detection. To date, various technical implementations have been proposed to embed laboratory biosensors into actual monitoring of water networks. For instance, the Lumisens 1 system by [START_REF] Horry | Development of a biosensor for the detection of tributyltin[END_REF] is based on a bioreactor maintain living bioluminescent Escherichia coli cells to detect organotin compounds; this cumbersome solution was later simplified into a disposable card with immobilized cells in the Luminens 2 system [START_REF] Horry | Technological conception of an optical biosensor with a disposable card for use with bioluminescent bacteria[END_REF]). Elad et al. described the immobilization of bacterial biosensors into polymeric biochips placed in flow-through chambers; the authors validated the simultaneous detection of various pollutants (paraquat, arsenite, nalidixic acid) under continuous water flow for several days [START_REF] Elad | Online monitoring of water toxicity by use of bioluminescent reporter bacterial biochips[END_REF]. Miniaturization of integrated sensors is a promising route to develop robust, reliable and cheap monitoring devices; as an example, Moll et al. associated microfluidics and love-wave acoustic detection to design an immunological biosensor aiming at detecting online pathogenic bacteria ((Moll et al. 2007), Patent FR2878622).

In this context, the main objective of the COMBITOX project funded by the French Agence Nationale de la Recherche (ANR) lies in the integration of three different types of luminescent biosensors into a semi-autonomous inline water analyzer prototype. The Editorial to this Special Issue provides a synthetic description of the aims and reaches of COMBITOX [START_REF] Ansaldi | Toward inline multiplex biodetection of metals, bacteria, and toxins in water networks: the COMBITOX project[END_REF]. The collection of articles accepted for publication in the COMBITOX Special Issue of Environmental Science and Pollution Research summarizes the design of bacterial biosensors to detect toxic metals such as nickel, arsenite, or mercury [START_REF] Cayron | Pushing the limits of nickel detection to nanomolar range using a set of engineered bioluminescent Escherichia coli[END_REF][START_REF] Prévéral | A bioluminescent arsenite biosensor designed for inline water analyzer[END_REF][START_REF] Brutesco | Bacterial host and reporter gene optimization for genetically encoded whole cell biosensors[END_REF]), a bacteriophagebased biosensor to detect enteric pathogenic bacteria [START_REF] Franche | Substrate-independent luminescent phage-based biosensor to specifically detect enteric bacteria such as E. coli[END_REF]) and an immunological biosensor to detect microcystin-LR [START_REF] Demey | Sorption of his-tagged protein G and protein G onto chitosan/divalent metal ion sorbent used for detection of microcystin-LR[END_REF][START_REF] Bazin | Profiling the biological effects of wastewater samples via bioluminescent bacterial biosensors combined with estrogenic assays[END_REF]; the three of them rely on the emission of light in a close range of emission wavelengths, either through bioluminescence for the bacterial and bacteriophage biosensors with the bacterial luciferase [START_REF] Baldwin | Structure of bacterial luciferase[END_REF] or chemiluminescence for the immunological biosensors with the horseradish peroxidase [START_REF] Ryan | Horseradish peroxidase: the analyst's friend[END_REF]. Because the light signal is emitted in a similar range of wavelengths (c.a. 490 and 430 nm for bioluminescence and chemiluminescence, respectively), we selected a single sensor to detect the light signals. The challenge here rather lies in the design and the optimization of a common reaction cell to accommodate the various biological reactions. Indeed, for both the bacteriophage and the bacterial biosensors, the water sample is usually incubated with the bioreagent (i.e., a bacteriophage solution or lyophilized bacteria) while bioluminescence emission kinetics is measured, whereas the immunological biosensor requires multiple incubation/washing steps to remove unspecific antibody ligands after incubation with the water sample. This Technical Note describes the design of our prototype core unit associating a light detector and a reaction cell together with the operational procedures to bring the three types of biosensors into play. The water analyzer prototype resulting from the COMBITOX project can be situated at level 4 on the Technology Readiness Level (TRL) scale, i.e., BComponent and/or breadboard validation in laboratory environment.M

aterials and methods

Principals and operational details about the bacteriophage, bacterial and immunological biosensors cited in this paper can be found elsewhere in this Special Issue [START_REF] Demey | Sorption of his-tagged protein G and protein G onto chitosan/divalent metal ion sorbent used for detection of microcystin-LR[END_REF][START_REF] Prévéral | A bioluminescent arsenite biosensor designed for inline water analyzer[END_REF][START_REF] Bazin | Profiling the biological effects of wastewater samples via bioluminescent bacterial biosensors combined with estrogenic assays[END_REF][START_REF] Franche | Substrate-independent luminescent phage-based biosensor to specifically detect enteric bacteria such as E. coli[END_REF]. We selected the substrate-independent, bacteriophagebased biosensor among the three different types we devloped to optimize the core unit light detector/reaction chamber of the COMBITOX analyzer prototype because it is the one expected to require the highest sensitivity. Thus, the different light measurements performed here were done on E. coli TD2158 live cells after successful lysogenization of the genetically modified HK620 bacteriophage according to the protocol described elsewhere in this Special Issue [START_REF] Franche | Substrate-independent luminescent phage-based biosensor to specifically detect enteric bacteria such as E. coli[END_REF].

Prototype design and construction

The French company AP2E (Aix-en-Provence, France) developed electronics, mechanics, automation and realized the COMBITOX prototype assembly. We selected the H1070 photomultiplier (PMT) from Hamamatsu for its robustness and high sensitivity in the 230-700 nm range (datasheet available at http://www.hamamatsu.com). For the record, bioluminescence and chemiluminescence reactions result in photons emission with a maximum emission peak at 490 and 430 nm, respectively. The current signal issued by the PMT is pre-amplified in situ and digitally converted by a 16 bits ADC connected to the black box where the biological reactions take place. The PMT is set at its maximum gain and powered by a reference voltage. For light emission measurements, the photosensor surface is placed above the reaction chamber (5 mm distance), provisionally described as a cuvette array containing the water sample to be analyzed.

Sample concentration by filtration

In order to maximize the light collection by the Hamamatsu PMT, the light-emitting objects (bioluminescent bacteria) contained in the incubation mix were concentrated in 2D onto a membrane surface prior to light measurements. To that avail, we used 0.45 μm Millipore® HA nitrocellulose membranes (25 mm diameter), with a Millipore® XX2702550 Model 1225 Sampling Manifold Vacuum Filtration Cell Harvester. The water sample is concentrated on circular surface S equal to 4.91 cm 2 , which is 4.34 times bigger than the sensitive surface of the PMT (S PMT = 1.13 cm 2 ). Supposing a uniform deposition of the bacteria on the membrane surface, a volume Vof 2.17 ml (i.e., 4.34 × 0.5 ml) was filtered in order to obtain the same number of bioluminescent bacteria under the PMT sensitive surface than in the 0.5 ml volume inserted into the GloMax® 20/20 luminometer from Promega. We thus can readily compare the light emission of both apparatus. In order to correlate the biomuminescence intensity with the number of bioluminescent bacteria concentrated onto the membrane surface, we used the E. coli biosensor pRpoD-lux that constitutively emits light described elsewhere in this Special Issue [START_REF] Prévéral | A bioluminescent arsenite biosensor designed for inline water analyzer[END_REF][START_REF] Brutesco | Bacterial host and reporter gene optimization for genetically encoded whole cell biosensors[END_REF].

Bioluminescence measurements

After 60-min incubation at 30 °C of a known number of E. coli TD2158 cells with the bacteriophages suspension, bioluminescence emission was measured either using a GloMax® 20/20 luminometer from Promega or the COMBITOX light detection unit. Cell counting was done with a spectrophotometer according to the following calibration: at an Optical Density measured at 600 nm of 1, 1 ml of a cellular suspension of E. coli contains 10 9 cells. In the luminometer, the incubation mix (0.5 ml) was placed in a 1.5-ml Eppendorf tube inserted in an integrating sphere ensuring maximum light collection. In the COMBITOX light detection unit, bioluminescence emission was measured by lowering the PMT at the surface of a 0.45 μm Millipore HA nitrocellulose membranes after filtration of 2.17 ml of the incubation mix. The distance between the PMT and the membrane was set at 5 mm to prevent contamination of the sensitive surface of the detector.

Results

Optimized sample conditioning prior to luminescence measurements

The design of a universal light emission measurement unit is the most challenging part of the COMBITOX semiautonomous inline water analyzer prototype. Although the reading output is the same for all biosensors-emission of light between 430 and 490 nm-the way by which the water sample interacts with the biosensor can be very different. For bacterial pathogens and metal detection (bacteriophage and bacterial biosensors, respectively), the water sample and the bioreagents (a bacteriophage suspension and lyophilized bacteria, respectively) are mixed and incubated in the same liquid phase, whereas for toxin detection (immunological biosensor), they are in separate phases. For the latter, the specific antibodies used to detect the toxins are immobilized onto a chitosan foam. After incubation with the water sample, several washing/incubation steps of the chitosan foam with chemiluminescent reagents and washing buffers are necessary to initiate the chemiluminescence reaction.

The simplest way to operate the PMT is to bring its sensitive area as close as possible to the sample surface for light detection. For the immunological biosensor, this setup will be almost optimal for photon collection, especially if the chitosan foam containing the antibodies is thin. For the bacteriophage and bacterial biosensors, the sensor surface is placed above the liquid surface. Depending on the depth of the cuvette and absorbance properties of the water sample incubated with the bioreagents, only the light emitted in the top layers of the reaction volume will reach the light sensor. To prevent such effects and ensure maximal collection, we decided to concentrate all the light-emitting objects (bioluminescent bacteria) on a surface by filtrating the water sample on a 0.45 μm nitrocellulose membrane after incubation with the bioreagents. Light emission is then measured at the membrane surface as described for the immunological biosensor. This protocol allows lower detection limits by increasing the reaction volumes (for the bacteriophage and bacterial biosensors) and/or addition of higher doses of bioreagents (for the bacterial biosensors) while discarding problems linked to the volume treated, light scattering or re-absorption by the reaction medium.

The three types of luminescent biosensors of the COMBITOX project derive from academic research and were optimized with laboratory-type luminometers, such as the GloMax® 20/20 from PROMEGA. The design and the improvements of the custom-made light sensor for the analyzer 1 indicate that we can detect at most 10 4 infected bacteria per ml with the bacteriophage-based biosensor with the whereas the statutory limit is set at 10 2 cells per ml. Our main objective with the COMBITOX analyzer prototype is to gain at least two logs of sensitivity; such an improvement will also be profitable for the other biosensors (bacterial and immunological sensors). To do so, we count on the Hamamatsu PMT and the concentration of greater water sample volumes on a membrane prior to light measurement. To assess the detection limit of our laboratory-made luminometer, we repeated the The chitosan foam loaded with the anti-toxins antibodies is supported by the membrane in the case of the immunological biosensor experiment described in Fig. 1 by first filtering the bacterial sample after 1-h incubation with the phages on the nitrocellulose membrane and then reading the light signal at the surface of the membrane (distance from PMT is 5 mm). The resulting titration curve is presented in Fig. 2a. With our experimental setup, we could achieve the detection limit of 10 2 infected bacteria per ml. We also evidenced that the sensitivity of the measurements can be increased by simply varying the volume filtered on the nitrocellulose membrane, as shown in Fig. 2b where multiplying the volume filtered by a factor 4 resulted in more than a fourfold light signal increase.

Guidelines for a disposable reaction cell for all biosensors

The validation of the Hamamatsu PMT setup together with the filtration protocol allowed us to propose the design of a disposable reaction cell able to accommodate either the phage or the bacterial or the immunological biosensor, coupled to a unique light detection system. Basically, it is a 20-mL cylinder closed at one end by a filtration membrane that can be connected to a manifold vacuum system. The membrane serves both as a support for the chitosan foam (immunological biosensor) and as a reaction cell for all three types of biosensors. The cabinet includes the interface with the water network, the sampling module, the bioreagent array, the disposable incubation/filtration units, the pipetting and dispensing system, and the light sensor. b, c A closer view of the bioreagent array (b), the disposable incubation/ filtration units (i), the pipetting and dispensing system (p), and the light sensor (s). d The disposable incubation/filtration unit When incubation is required (the water sample with the phage solution, the lyophilized bacteria or the chitosan foam; the chitosan foam with the washing solutions or chemical reagents), a counter pressure is applied to avoid liquid leakage through the membrane pores. When filtration is required, the disposable cell is connected to a vacuum pump. When all the necessary steps are finished, the light detector is lowered by the open end of the cylinder as close as possible to the membrane/chitosan foam surface and the light emission intensity recorded. Figure 3 displays a blueprint of such a disposable incubation/filtration unit.

This system is simple and flexible. The length of the cylinder can be adjusted to accommodate greater volumes if need be for improved sensitivity; this reaction cell can be cheaply produced with 3D-pinters in plastics compatible with water analysis and only need to be equipped with simple, calibrated nitrocellulose membranes. This consumable part of the water analyzer can be further processed to accommodate the chitosan foam loaded with the specific anti-toxin antibodies. It is well suited to the various incubation/washing steps required to fabricate the chitosan foam as well as operate the immunological biosensor, thanks to the connection to the vacuum pumping system. Figure 5d shows an example of a disposable incubation/filtration unit obtained with a 3D-printer and equipped with a filtration membrane.

Conclusion

The scientific literature, including this Special Issue, offers various and original detection possibilities using biological transducers, most of them being validated and optimized in research laboratories. Moving now from laboratory bench with manual operator interventions to semi-autonomous processes compatible with on-site measurements represents one of the major bottlenecks for industrial applications of biosensors. The design of a universal device that can house various types of biosensors whose functioning require different actions (addition of substrate, washing steps…) is a key challenge for implementation in the automate. In this article, we contributed to this field with the development of a flexible and original light detector/reaction cell suited for bacterial (nickel, arsenite, mercury detection), immunological (microcystin-LR toxin detection) and bacteriophage (live enteric bacteria detection) biosensors without loosing sights of the technical and economical constraints. Our disposable reaction cell, coupled to a unique light detection system, is composed of a 20-mL cylinder close at one end by a filtration membrane that can be connected to a vacuum system. The water sample is dispensed in the chamber and incubated with phage biosensor or whole cell biosensor concentrated solutions. After 60 min incubation, the mixture can be filtrated before lowering the light detector as close as possible to the membrane for light emission intensity measurements. For immunological biosensors, the membrane supports chitosan foams loaded with the anti-toxins antibodies and the different washing/incubation steps required to obtain the chemiluminescence reaction can be performed via filtration prior to light measurements. These procedures used for the different biosensors can be easily automated to develop a semi-autonomous inline water analyzer where consumables are arrays of disposable reaction cells (with antibodies-loaded chistosan foams for the immunological biosensors), bioreagents for bacterial pathogens and metal detection (phage solutions and lyophilized bacterial cells) and chemical reagents for toxins detection (labeled toxins, luminol, washing solutions). Thus, the results presented here serve as a basis to develop a complete prototype of inline water analyzer prototype using biosensors (Fig. 4). In this prototype, the core unit light detector/reaction cell is connected to a sampling system plugged in the water network, a storage chamber for the reagents and the disposable reaction cells arrays and a collector for wastes coupled to a sterilization process to avoid downstream contamination by the biosensors after usage. Figure 5 presents pictures of the final COMBITOX analyzer prototype working in laboratory conditions. This machine reaches now TRL 4 and will be upgraded to TRL 6 in a following project with the development of a fully functional analyzer working in real conditions plugged to water networks.

Fig. 1

 1 Fig. 1 Detection limit observed with the bacteriophage biosensor with the GloMax® 20/20. These data were obtained by measuring light emission 60 min after infection of a known number of E. coli TD2158 live cells by the bacteriophage biosensor

Fig. 2

 2 Fig. 2 Qualification of the Hamamatsu laboratory-made luminometer for light detection after membrane filtration. a Titration curve with the bacteriophage biosensor obtained with the Hamamatsu laboratory-made

Fig. 4

 4 Fig. 4 Schematic of the COMBITOX semi-autonomous inline water analyzer using luminescent biosensors

Acknowledgments This work was funded by the French national research agency ANR on the call-for-project ECOTECH (project COMBITOX) and also supported by the Centre National de la Recherche Scientifique and the Commissariat à l'Energie Atomique et aux Energies Alternatives (program NRBC).


Fabrice Mayran De Chamisso
email: fabrice.mayran-de-chamisso@cea.fr

Boris Meden
email: boris.meden@cea.fr

Mohamed Tamaazousti
email: mohamed.tamaazousti@cea.fr

HSPA: Hough Space Pattern Analysis as an Answer to Local Description Ambiguities for 3D Pose Estimation

When performing feature-based 3D object registration, one may expect to find a unique point corresponding to the right transformation in Hough space for each object instance. However, we observed that description ambiguities of the objects or scenes create a structured pattern in the Hough space of transformations during the matching process. We argue that this pattern can be viewed as a global descriptor, as opposed to the local descriptors or features whose matching resulted in the pattern. Thus, we propose to shift the focus from finding better local descriptors to better using the Hough-space pattern. This paper introduces a methodology to compute, analyze and match said patterns in order to improve the quality of 3D pose estimation. We detail a whole framework, termed HSPA, to first generate what we call the Hough space canonical invariance pattern for any given object to register and second, take this pattern into account when assembling and pruning pose hypotheses generated by a registration algorithm. We show the benefits of this technique on object registration as well as 3D scene registration benchmarks.

Introduction

Rigid object localization aims at finding a six degree of freedom (6DoF) transformation from a rigid object in the scene to the same object in a reference configuration. Typically, the scene is a point cloud, RGB or RGB-D image captured by a sensor and the reference is a CAD model or a point cloud obtained by fusing multiple sensor acquisitions with varying point of view. There are a large number of localization approaches: template-based approaches such as [START_REF] Hinterstoisser | Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes[END_REF][START_REF] Hodaň | Detection and fine 3d pose estimation of texture-less objects in rgb-d images[END_REF], whole-image based processing using neural networks such as [START_REF] Rad | Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth[END_REF] or [START_REF] Park | Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation[END_REF] and many descriptor-based approaches, reviewed in [START_REF] Guo | 3d object recognition in cluttered scenes with local surface features: A survey[END_REF]. However, all these approaches eventually converge to a set of pose hypotheses: 6DoF transformations with a confidence score. The space of the degrees of freedom of transformations is called generalized Hough space. In the ideal case, for objects without any symmetry, there is a unique rigid transformation from any given scene to a given reference, resulting in theory in a single point in Hough space corresponding to the object's localization. c) translation parts of the Hough space. d) registrations obtained by the proposed method, with confidence scores from red to green. Even when an object has no apparent symmetry (pump object, right), some parts may still exhibit invariances.

In practice, transformation hypotheses are noisy, resulting in multiple points in Hough space forming a cluster around the correct transformation. This is accounted for in Houghspace pose grouping approaches [START_REF] Guo | 3d free form object recognition using rotational projection statistics[END_REF][START_REF] Guo | 3d object recognition in cluttered scenes with local surface features: A survey[END_REF][START_REF] Khoshelham | Extending generalized hough transform to detect 3d objects in laser range data[END_REF][START_REF] Li | A unified framework for multi-view multi-class object pose estimation[END_REF][START_REF] Manhardt | Explaining the ambiguity of object detection and 6d pose from visual data[END_REF][START_REF] Mian | On the repeatability and quality of keypoints for local feature-based 3d object retrieval from cluttered scenes[END_REF][START_REF] Tombari | Object recognition in 3d scenes with occlusions and clutter by hough voting[END_REF][START_REF] Zhong | Intrinsic shape signatures: A shape descriptor for 3d object recognition[END_REF]. What is not accounted for, however, is the fact that objects with global symmetries such as planes, cylinders, spheres or gears result in multiple and possibly disjoint clusters in Hough space. More generally, when objects have parts that resemble each other relatively to the descriptor used1 (or feature vector, template, etc.), which we call invariances, structured patterns can be observed in Hough space (see Figure 1). We call these invariance patterns because they arise due to the description of the object being locally or globally invariant. In other words, there exist transformations, which, when applied to the object, leave a subset of its descriptors invariant. Invariance patterns are implicitly considered detrimental to localization in state of the art approaches. On the contrary, in this paper, we develop a methodology to exploit them. Our contributions are as follows: (i) we build a global Hough-space descriptor called the canonical invariance pattern. (ii) We propose a Hough-space pattern matching algorithm for 3D pose estimation. (iii) We propose analgorithm to find and exploit symmetry-breaking details that we call "disambiguation". (iv) We propose a local descriptor to supply Houghspace pose hypotheses with competitive computation and matching times.

Related Work

Symmetries and Hough space in 3D geometry Mitra, Pauly et al. [START_REF] Niloy | Partial and approximate symmetry detection for 3d geometry[END_REF][START_REF] Niloy | Symmetry in 3d geometry: Extraction and applications[END_REF][START_REF] Pauly | Discovering structural regularity in 3d geometry[END_REF] detect pairs of points symmetric along a line in Hough space and perform clustering in order to find global symmetries. This allows them to symmetrize real-world objects (see Figure 15 of [START_REF] Niloy | Symmetry in 3d geometry: Extraction and applications[END_REF]) as well as, for instance, edit geometry while preserving a symmetry property. Also, by matching two point clouds representing the same object during a motion, they obtain one cluster per rigid body which allows easy segmentation thereof. All analyses are based on clustering sets of Hough points obtained by matching invariant zones. One of the key differences with our work is that Mitra et al. do not consider votes for the pose of an object, but for a symmetry axis, which prevents the use of an invariance pattern for 3D pose estimation.

Symmetries and Hough space for 3D pose estimation The methods addressing symmetry issues for 3D pose estimation can be categorized in two families. The first category assumes information on the set of object symmetries to improve scoring of poses. [START_REF] Brégier | Symmetry aware evaluation of 3d object detection and pose estimation in scenes of many parts in bulk[END_REF][START_REF] Brégier | Defining the pose of any 3d rigid object and an associated distance[END_REF] remove poses that are symmetries of each other from Hough space. A similar strategy is used in [START_REF] Park | Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation[END_REF]. [START_REF] Sergey | Leveraging symmetries to improve object detection and pose estimation from range data[END_REF] uses symmetries to reduce the Hough space used for pose clustering by only keeping one instance of each symmetrical part of the original object. Finally, [START_REF] Pitteri | On object symmetries and 6d pose estimation from images[END_REF] introduces a pipeline to handle symmetrical or quasi-symmetrical objects whose aspect is the same from multiple points of view. By modifying the loss function driving a neural network to predict symmetry-invariant data, the average recall in an object localization task is improved and the learning time is reduced. Global symmetries are handled in a binary way (symmetrical/not symmetrical pose), ignoring objects with local symmetries (only symmetrical parts). Fine localization of symmetry-breaking details (foolproofing, holes, . . . ) is not handled in [START_REF] Pitteri | On object symmetries and 6d pose estimation from images[END_REF].

The second category is approaches implicitly robust to symmetries to some extent. For instance, [START_REF] Hodan | EPOS: estimating 6d pose of objects with symmetries[END_REF] splits objects into fragments. Fragment matches and object labels are estimated separately, so that a single object label can be predicted even in the case of symmetries. The symmetries themselves are not considered during inference. During learning, symmetries are not taken into account either, with the hypothesis that, over the whole dataset, all symmetries will be represented. It is not clear how this approach is impacted by local symmetries in an object. However, this method uses the implicit assumption that some of the fragments are symmetry-disambiguating, so that matching them will result in an ambiguous pose. In the same category, [START_REF] Li | A unified framework for multi-view multi-class object pose estimation[END_REF] finds poses representing the same model but with a different symmetry by using an ICP score (average of closest-point distances) to compare poses, based on the fact that the geometry is invariant by any of its symmetries. This process is computationally expensive (at least n log n per pose pair). However, one can naturally expect the ICP score to handle local invariances to some extent.

Note that none of these approaches provides a way to find the set of symmetries. In this paper, we propose to build this set of symmetries in the form of our Invariance Hough Space Pattern as shown in Figure 1.

Feature-based 3D pose estimation In order to better understand our method, we give a quick overview of feature-based 3D pose estimation. One way to tackle the problem is endto-end learning [START_REF] Sk | Rpsrnet: End-to-end trainable rigid point set registration network using barnes-hut 2d-tree representation[END_REF][START_REF] Huang | A comprehensive survey on point cloud registration[END_REF] or [START_REF] Banani | Unsupervisedr&r: Unsupervised point cloud registration via differentiable rendering[END_REF] with RGB-D input. However, to the best of our knowledge, there is no trivial way to tackle the symmetry issues except by representing all instances of the symmetries in the input dataset. The problem seems to be ill-posed as noted in [START_REF] Pitteri | On object symmetries and 6d pose estimation from images[END_REF]. For this reason, we focus on feature-based approaches which can be summarized in three classical steps: 1) data preparation (smoothing, normal and sometimes reference frames calculations), keypoint detection, etc., 2) description and matching, and finally 3) outlier rejection and pose refinement for robust alignment.

Regarding description, there are traditional/geometrical and learning-based methods. There are currently two major geometry-based 3D registration pipelines. The first one is based on descriptors such as SHOT [START_REF] Tombari | Object recognition in 3d scenes with occlusions and clutter by hough voting[END_REF] computed on a 3D point cloud in a local reference frame centered on each keypoint. Each descriptor match generates a unique 6D pose hypothesis. The second pipeline is based on Point-Pair Features (PPF), as exposed notably in [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF]. PPF do not use reference frames but require both points of each pair to match (2 to 2 matching), which implies a quadratic sampling of points. The quadratic computation cost is avoided by smart sampling of computation points [START_REF] Hinterstoisser | Going further with point pair features[END_REF]. Otherwise, many approaches have taken advantage of Deep Neural Networks for keypoint detection [START_REF] Ao | Spinnet: Learning a general surface descriptor for 3d point cloud registration[END_REF][START_REF] Aoki | Pointnetlk: Robust & efficient point cloud registration using pointnet[END_REF] and feature description [START_REF] Bai | D3feat: Joint learning of dense detection and description of 3d local features[END_REF][START_REF] Huang | Predator: Registration of 3d point clouds with low overlap[END_REF]. All in all, state of the art approaches focus on having each pose hypothesis correct up to noise, which leads to developing more and more complex descriptors (for instance SHOT [START_REF] Tombari | Unique Signatures of Histograms for Local Surface Description[END_REF] with 352 dimensions), templates or neural networks to generate hypotheses. Mismatches will happen even with very complex descriptors, since there will always be objects or parts of objects invariant under description by the chosen descriptor. Of particular interest is the case of symmetrical objects such as cylinders, for which the notion of a "correct" pose is undefined and the symmetry will manifest in any description of the object.

Regarding outlier rejection, existing methods are either RANSAC [START_REF] Hodan | EPOS: estimating 6d pose of objects with symmetries[END_REF][START_REF] Papazov | Rigid 3d geometry matching for grasping of known objects in cluttered scenes[END_REF] or Hough based [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF][START_REF] Guo | 3d object recognition in cluttered scenes with local surface features: A survey[END_REF][START_REF] Guo | Rotational projection statistics for 3d local surface description and object recognition[END_REF][START_REF] Khoshelham | Extending generalized hough transform to detect 3d objects in laser range data[END_REF][START_REF] Li | A unified framework for multi-view multi-class object pose estimation[END_REF][START_REF] Manhardt | Explaining the ambiguity of object detection and 6d pose from visual data[END_REF][START_REF] Mian | On the repeatability and quality of keypoints for local feature-based 3d object retrieval from cluttered scenes[END_REF][START_REF] Mian | Three-dimensional model-based object recognition and segmentation in cluttered scenes[END_REF][START_REF] Tombari | Object recognition in 3d scenes with occlusions and clutter by hough voting[END_REF][START_REF] Zhong | Intrinsic shape signatures: A shape descriptor for 3d object recognition[END_REF]. Hough based approaches mainly differ by the number of dimensions considered: 3 (translations or rotations only), 6 (both) or 7 (with a scale factor) and by the clustering method (voxel-based or sparse). A comprehensive review of traditional 3D outlier rejection is proposed by [START_REF] Yang | A performance evaluation of correspondence grouping methods for 3d rigid data matching[END_REF]. Recently, some deep-learning based methods for outlier rejection have been proposed [START_REF] Bai | Pointdsc: Robust point cloud registration using deep spatial consistency[END_REF][START_REF] Choy | Deep global registration[END_REF].

3 Our Hough Space Pattern Analysis (HSPA) approach

Standard 3D registration pipeline

In order to perform invariance analysis in Hough space, we first have to populate said space with pose hypotheses. In this paper, we are not interested in how pose hypotheses are generated, so we choose a classical descriptor-based pipeline to simplify explanations. Such pipelines are reviewed in [START_REF] Guo | 3d object recognition in cluttered scenes with local surface features: A survey[END_REF]. The only requirement in order to perform invariance analysis in Hough space is to generate multiple pose hypotheses. Instead of having a very disambiguating descriptor, we keep description simple and recover the missing information from HSPA. We use a simple geometrical 6-dimension descriptor called Hexagon (illustration as supplementary material). From a point cloud with normals and BOARD [START_REF] Petrelli | On the repeatability of the local reference frame for partial shape matching[END_REF] reference frames, the descriptor is computed at each point (no keypoint extraction required) as the six point-to-point distances from the vertices of a fixed-size hexagon in the tangent plane (oriented using to the x-axis of the reference frame) to the object's surface. Normal orientation (inwards/outwards) gives each of the six distances a sign. Descriptor matching generates correspondences, which along with local referentials create pose hypotheses.

We represent poses in 6D Hough space by the translation components and an angle-axis representation of rotations (angle times unit rotation axis θ .⃗ r/||⃗ r||). This representation is optimally compact (3 parameters for rotations) and the cyclicity of the angular coordinate is much easier to handle than with Euler angles (as used for instance in [START_REF] Niloy | Partial and approximate symmetry detection for 3d geometry[END_REF][START_REF] Niloy | Symmetry in 3d geometry: Extraction and applications[END_REF][START_REF] Pauly | Discovering structural regularity in 3d geometry[END_REF]). In Hough space, the cyclicity comes down to replicating points at (θ -2π).⃗ r/||⃗ r|| whenever necessary.

We perform an agglomerative clustering in 6D Hough space to create robust pose hypotheses from non-robust input hypotheses. Agglomeration is performed by doing fixed radius searches in Hough space around each hypothesis, ignoring points with less than a fixed number of neighbors (noise filtering) and propagating clusters to the remaining neighbors. For each cluster, a 6-component ponderated average is accumulated together with a cluster weight, as the sum of input pose weights in the cluster (input pose weights are obtained for instance as descriptor similarities and are also used to ponderate the 6-component average). Extra care is taken to ensure that the ±π seam is handled correctly.

Agglomerative clustering returns clusters of arbitrary shape and size while still eliminating outliers (compared to k-means [START_REF] Mian | On the repeatability and quality of keypoints for local feature-based 3d object retrieval from cluttered scenes[END_REF] for instance). Also, compared to finding the 6D point with the most neighbors [START_REF] Guo | Rotational projection statistics for 3d local surface description and object recognition[END_REF], a very small 6D search radius can be used for neighborhood searches, which saves computation time. It should be noted that this clustering step does not take object invariances into account. The goal is to reduce the number of points in Hough space while increasing robustness of the remaining points (as the average of transformations in each cluster). In practice, keeping only the N = 100 highest-weight clusters is sufficient.

Pose refinement is then performed on the cluster's average using a simple point-to-point ICP [START_REF] Besl | A method for registration of 3-d shapes[END_REF], which we found more robust for mostly planar objects than point-to-plane ICP.

The 3D registration pipeline described in this section performs registration of objects without invariances or with a low amount of invariances. It works well for these objects since scene-to-model correspondences produce 6D transforms forming a single small cluster in Hough space for each object instance.

Invariance pattern analysis for 3D registration

Overview

Objects with invariances or symmetries cause mutiple clusters to exist in Hough space for each object instance in the scene.First, we group them into one meta-cluster per object instance using a Hough-space canonical invariance pattern computed and discretized ("beads" model) offline on the reference object or model. Then, we apply a step called disambiguation whose goal is to decide between quasi-symmetries (this should be viewed as a second order registration, refining the first order registration obtained at the previous step). Disambiguation uses a disambiguation map computed offline on the reference object or model. A visual representation of the process can be found in the supplementary material/Figure 4.

The canonical invariance pattern

Obtaining the canonical invariance pattern. We start by performing a registration of the model to a version of itself with noise added. Adding noise is necessary to emulate an object viewed by a sensor and overcome the original discretization of an object represented as a point cloud or mesh. It prevents all descriptors from only matching with themselves. Selfregistration is performed offline using the the same registration approach as online: optional keypoints detection, description and matching generate correspondences. Correspondences yield 6D transformations expressed in Hough space: the canonical invariance pattern.

Obtaining the discretized bead model. The set of correspondences in Hough space is discretized through fixed radius clustering. We call the centers of the clusters beads. The role of this discretization is mainly for efficiency for the following online computations.

Grouping clusters belonging to a single object instance

The output of a registration pipeline such as that of section 3.1 is a set of poses, each pose associated to a set of correspondences (a cluster), a weight and a model identifier, if there are multiple models to localize at the same time. The first step of invariance analysis is to group together poses belonging to the same model instance in the scene. For symmetrical objects, multiple pose hypotheses are valid. For objects with invariances but no symmetries, only one pose is correct, but some parts of the objects may match some parts of the scene. For instance, a plane may align on any plane. The process of pose grouping goes as follows (algorithmic form in supplementary material): for each pose hypothesis P i with weight w i aligning model M to the scene, let ∀ j, P ′ j = P -1 i P j . According to the chain rule, P ′ j writes as a transformation from the model to itself, and thus should belong to the canonical invariance pattern. By applying P -1 i , we transform the invariance pattern into what should be the canonical invariance pattern if P i is a correct pose hypothesis. Then, for each P i and each P j check if P ′ j = P -1 i P j belongs to the canonical invariance pattern or bead discretization thereof (6D distance lower than a threshold r s , taking into account the ±π seam). If yes, add its weight (multiplied by the canonical pattern's bead's own weight if using beads) to P i 's weight counter (initialized at zero). Then, take the highest score P i and attach all compatible P j to it, increasing the final score of P i by that of all the P j attached. Neutralize P i and its attached P j and restart the process until all poses have been considered. If poses in Hough space are considered as representatives of an underlying probability distribution, the above process consists in finding the transformation which, when applied to the current invariance pattern, maximizes its correlation with the canonical invariance pattern. Repeating the process allows identifying multiple object instances, one at a time.

This algorithm has at least an O(n 2 hypotheses ) complexity (potentially higher depending on the indexing structures used for nearest neighbor searches within the beads), which is why we execute it on poses output by our 6D clustering algorithm and not on raw hypotheses obtained from correspondences. Currently, 6D clustering finds a maximum of 100 pose hypotheses before invariance analysis but only 16 are kept after invariance analysis.

The disambiguation map

Motivation. The cluster grouping approach described in the previous paragraphs attaches, let's say, clusters B and C to cluster A. For quasi-symmetrical objects such as the star (Figure 1), we can't be sure that A is the best pose hypothesis, because descriptors may not see the small details that would allow disambiguation. So, B or C may be better options than A. However, if we have either A, B or C for the star, we know that we could obtain both others by applying a few 30 • rotations. More generally, if a pose is almost correct up to a quasisymmetry, applying one of the beads belonging to the canonical invariance pattern will bring it to a correct pose. Which bead to apply is chosen by computing a feature matching score between model and scene for each bead, possibly ponderated by the disambiguation map.

Offline computation. Once beads are computed, we can obtain a disambiguation map for a set of local features. Some such features are contours (or 3D edgelets), points with or without normals, descriptors, etc. and are typically not the descriptors used to compute the Hough pattern. Let F = { f i } be the set of features and P = {P i } be the set of poses, with each pose being the center of one bead and each feature f i associated with a 3D position r( f i ) on the model. Also, let ω i be the weight (sum of correspondence weights in the bead) of P i . Let M be a model. P i (M) is the model transformed by P i . For each f i ∈ F on M, let f ′ j i be the feature on P j (M) that is closest to f i in terms of position, that is f ′ j i = f argmin k ||P j (r( f k ))-r i || . Let w j i be the matching score of f i and f ′ j i . We define the feature disambiguation weight of f i as W i = ∑ bead j w j i ω j ∑ bead j ω j . In other words, the disambiguation weight of a feature is the sum, for all beads, of how different the feature is from the closest feature in the model transformed by the bead. The map of disambiguation weights of all features constitutes what we call the disambiguation map. An example of disambiguation maps using points as features and the distance to the closest point, ponderated by the absolute cosine of the dot product of normals, as score, is plotted in the supplementary material as Figure 2.

Online usage Let P be the current pose hypothesis and {P i } be the set of beads from the canonical invariance pattern. If the Identity transformation is not in the canonical pattern, we add it nevertheless to the set of P i . Let F = { f i } be the set of features of the scene, r( f i ) being the position of f i . Similarly, F ′ = { f ′ } is the model's feature set. We find the closest model feature to a scene feature for a transformation P.P j as f ′ j i = f argmin k ||P.P j (r( f k ))-r i || . Let w j i be the matching score of f i and f ′ j i . We can define a pose matching score for P.P j as S j = ∑ i,||r( f i )-r( f ′ j i )||<τ w j i .W i where W i is a feature f i 's weight in the disambiguation map and τ is a fixed distance (say, 2mm) corresponding to the maximum allowed discrepancy between registered model and scene. The pose P.P j leading to the highest S j is chosen instead of P.

Experiments

Our experiments are organized as follows: first we evaluate our method on an object pose estimation task, where the goal is to localize up to N instances of a known object (i.e. we have a 3D model of it) in a scene. For this, we follow the rules of the BOP challenge [START_REF] Hodaň | Bop challenge 2020 on 6d object localization[END_REF] and focus on the ITODD dataset [START_REF] Drost | Introducing mvtec itodd -a dataset for 3d object recognition in industry[END_REF] which is representative of industrial objects (shiny, texture-less, etc.). We then evaluate our approach on a scene registration task in a 3D reconstruction context, which consists in recovering the displacement of a camera between two views of the same scene. There, we use the 3DMatch dataset [START_REF] Zeng | 3dmatch: Learning local geometric descriptors from rgb-d reconstructions[END_REF] and its evaluation metrics. Experimental details as well as hyperparameters can be found in the supplementary material.

6DoF object pose estimation

Evaluation metrics. In the BOP Challenge, average recall (AR) is defined as AR = (AR V SD + AR MSSD +AR MSPD )/3, the mean between Visible Surface Discrepancy, Maximum Symmetry-Aware Surface Distance and Maximum Symmetry-Aware Projection Distance [START_REF] Hodaň | Bop challenge 2020 on 6d object localization[END_REF], and is evaluated through the BOP evaluation server. Note that quasi-symmetrical objects such as "star" (see Figure 1) are considered completely symmetrical in the challenge, so the metrics of the BOP challenge can't completely highlight the benefits of symmetry disambiguation and the disambiguation map as explained in this paper.

Method

Modality AR Timing Drost-CVPR10-Edges [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF] RGB-D 0.570 6.833s SurfEmb [START_REF] Rasmus | Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings[END_REF] RGB-D 0.538 4.942s Koenig-Hybrid-DL-PP combination of [START_REF] Fu | Retinamask: Learning to predict masks improves state-of-the-art single-shot detection for free[END_REF][19] [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF] RGB-D 0.483 0.318s Drost-CVPR10-3D-Edges [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF] Depth only 0.462 5.838s Vidal-Sensors18 [START_REF] Vidal | A method for 6d pose estimation of free-form rigid objects using point pair features on range data[END_REF] Depth only 0.435 3.419s Drost-CVPR10-3D-Only [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF] Depth only 0.316 2.0s HSPA (whitout invariance analysis) Depth only 0.412 0.535s HSPA (whith invariance but no disambiguation) Depth only 0.458 0.471s HSPA (with invariance and disambiguation) Depth only 0.511 0.567s Table 1: HSPA (with ablation study) and current top competitors on BOP/ITODD. We focus on the approaches using only the depth modality (bottom rows) but also give the overall best performers using RGB-D (top rows).

On ITODD (Table 1), the performance of our approach with Hough space clustering but no invariance analysis is average: third in terms of AR for the depth only modality. The computation time is more than six times lower than the two better approaches, showing that hexagon is a decent yet not great descriptor. It is fast to compute and match though, which justifies its use to quickly populate Hough space with many pose hypotheses.

With invariance analysis but no disambiguation, AR improves to almost that of the current best depth-only method (0.458 vs 0.462). Computation time decreases by 12% due to each cluster fused into another cluster being one less hypothesis to refine and score. With both invariance analysis and disambiguation, computation time compared to the no invariance analysis case increases by 6%. The average recall gains 0.099 points, placing the approach first among depth-only methods. When comparing to state of the art approaches that also use RGB, our approach has noticeably better average recall than Koenig-Hybrid-DL-PointPairs while being a little slower (their time gain was obtained by segmenting data first with a convolutional neural network before using Drost-CVPR10-Edges [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF], so there is a speed/accuracy trade-off). Compared with the current best approach in terms of AR [START_REF] Drost | Model globally, match locally: Efficient and robust 3d object recognition[END_REF], our approach is 12x faster (although not evaluated on the same computer) while loosing 0.059 AR. Table 1 displays the top contenders of the challenge for the depth and RGB-D modality for this dataset. PPF-CVPR-10 (Implementation of HALCON 19.05) is still the best performing method on this dataset, outperforming more recent (including 2021) approaches.

3D Scene registration

The main difference between 3D scene registration and object localization is that in the latter, the reference model is loaded and processed only once "offline", while scenes are processed online. In scene registration, the "object" is a scene. Since the "beads" algorithms was not designed to be fast, we replace it by the 6D clustering used in the online phase.

Evaluation metrics. Following DGR [START_REF] Choy | Deep global registration[END_REF] and PointDSC [START_REF] Bai | Pointdsc: Robust point cloud registration using deep spatial consistency[END_REF], we use three evaluation metrics, namely (1) Registration Recall (RR), the percentage of scenes that were successfully aligned (rotation error and translation error below some thresholds of the ground truth), (2) standard rotation error (RE), and (3) standard translation error (TE).

Method

RR (% ↑) RE ( [START_REF] Bai | Pointdsc: Robust point cloud registration using deep spatial consistency[END_REF]. Only methods based on FPFH [START_REF] Rusu | Fast point feature histograms (fpfh) for 3d registration[END_REF] have been used for fair comparison with our naive geometric descriptor hexagon and to avoid overfitting a learnt descriptor to a set of scenes. Time is only given for grouping and invariances (when present), when using keypoints only/all points. RR, RE and TE are computed in our case with all points. Results. Figure 2 shows a registration between two scenes of "redkitchen" [START_REF] Zeng | 3dmatch: Learning local geometric descriptors from rgb-d reconstructions[END_REF]. The floor is never considered highly disambiguating but most high curvature objects are. For (c), we think the chairback was not considered highly disambiguating as in (d) because of a discrete translational invariance which is incomplete due to occlusions in (d). Vertical planes were not considered disambiguating in (d) because there are many other flat surfaces.

Table 2 compares results obtained with hexagon to results obtained with FPFH, another "classical" descriptor. Without invariance analysis, our approach performs similarly to the current best approach PointDSC [START_REF] Bai | Pointdsc: Robust point cloud registration using deep spatial consistency[END_REF], with similar RR, RE and TE. Invariance analysis allows our approach to take the lead with 82% RR while preserving RE and TE. Computation times are to be taken with caution, as only correspondence processing/pruning time (including invariance analysis when present) is listed to be consistent with [START_REF] Bai | Pointdsc: Robust point cloud registration using deep spatial consistency[END_REF]. When operating on all points, our computation times are comparable to [START_REF] Bai | Pointdsc: Robust point cloud registration using deep spatial consistency[END_REF] which is operating on keypoints only. When operating on keypoints only, our computation times decrease by about a factor ten with a 4% RR decrease. Also, hexagon computations and matching are feasible on all points because of the low dimensionality of the descriptor, and the computation times would prevent the computation and matching of FPFH on all points in practical situations.

Conclusion and perspectives

In this paper, we proposed a new framework to tackle a common issue with 3D registration, where some zones in an object have the same descriptor (an invariance). When matching such descriptors, Hough space patterns appear. These can be precomputed automatically for any object and matched when trying to find the object in a scene. The Hough pattern thus becomes a global object descriptor, as opposed to local descriptors. We also exposed algorithms to automatically find which parts of an object break the object's common symmetry if any. This disambiguating information can be used in a second pass to refine registration results, which is necessary for objects with fool-proofing or asymmetric holes for instance. We demonstrated state of the art performance in object localization and scene registration benchmarks. Conceptually, our method should be agnostic to the descriptor choices. However, the practical balance between local description power and hough space invariance pattern needs an in-depth investigation. We are currently investigating extensions of invariance analysis to other parametric problems.

Figure 1 :

 1 Figure 1: Canonical invariance patterns and scene-to-model registrations for objects of the ITODD [11] dataset. a) objects with invariance and almost-invariance axes. b) rotation and c) translation parts of the Hough space. d) registrations obtained by the proposed method, with confidence scores from red to green. Even when an object has no apparent symmetry (pump object, right), some parts may still exhibit invariances.

Figure 2 :

 2 Figure 2: Registration on 3DMatch/redkitchen. (a), (b): input point clouds; (c), (d): computed disambiguation maps, from green (low disambiguation potential) to red (high disambiguation potential); (e) registration. (best viewed in color)

Table 2 :

 2 Comparison with state of the art methods on the 3DMatch dataset, based partially on

	RANSAC-100k	73.57	3.55	10.04	5.24/-
	3DRegNet [34]	26.31	3.75	9.60	0.05/-
	DGR w/o s.g. [10]	27.04	2.61	7.76	0.56/-
	DGR [10]	69.13	3.78	10.80	2.49/-
	PointDSC [6]	78.50	2.07	6.57	0.09/-
	HSPA (no invariance)	75.64	1.79	7.91	0.008/0.089
	HSPA (with invariance)	81.99	1.82	7.19	0.009/0.092

• ↓) TE (cm ↓) Time (s)

The limit between "same" and "different" descriptors or "symmetrical" and "non-symmetrical" objects is often a question of level of detail of the description.


Simon Guillot

Thibault Prouteau

Nicolas Dugué

Sparser is better: one step closer to word embedding interpretability

Keywords: SINr insulin scalar, tablespoon, vesicular, dystrophy antiserum, falsiable, experimenter, internat PBS, NC, arginine, IFN glutathione, pancreas, gastroduodenal, vitamin immunologically, hyperplasia, transgene, nociceptive insulin, sulphasalazine, interferon, cholangitis hypertriglyceridaemia, mellitus, porcine, insulin aldosterone, aminotransferase, creatinine, glycated ulcerative, sulphasalazine, colitis, sera mint scalar, tablespoon, vesicular, dystrophy cube, geranium, Berowne, curiosities polyunsaturated, misre, margarine, methile spoonfuls, parsnips, kebabs, preheat onion, basil, yogurt, coriander dial, screams, vibration, spadefoot tbsp, oregano, diced, dijon Gibson, gigged, charvel, Ibanez minted, minting, hoards, coinages oxygen scalar, tablespoon, vesicular, dystrophy herbicides, menstrual, deprave, angiotensin pou, tenascin, cytoplasm, platelet glutathione, pancreas, gastroduodenal, vitamin lipid, crypt, tris, calcium monoxide, oxides, sulphuric, nitrogen monoxide, dioxide, nitrous, oxides supplemental, hypoxaemic, electrocardiographic, gastroscopy diastolic, systolic, transfusion, transfusions

Sparse word embeddings models (SPINE, SINr) are designed to embed words in interpretable dimensions. An interpretable dimension is such that a human can interpret the semantic (or syntactic) relations between words active for a dimension. These models are useful for critical downstream tasks in natural language processing (e.g. medical or legal NLP), and digital humanities applications. This work extends interpretability at the vector level with a more manageable number of activated dimensions following recommendations from psycholinguistics. Subsequently, one of the key criteria to an interpretable model is sparsity: in order to be interpretable, not every word should be represented by all the features of the model, especially if humans have to interpret these features and their relations. This raises one question: to which extent is sparsity sustainable with regard to performance? We thus introduce a sparsication procedure to evaluate its impact on two interpretable methods (SPINE and SINr) to tend towards sustainable vector interpretability. We also introduce stability as a new criterion to interpretability. Our stability evaluations show little albeit non-zero variation for SPINE and SINr embeddings. We then show that increasing sparsity does not necessarily interfere with performance. These results are encouraging and pave the way towards intrinsically interpretable word vectors.

Introduction

Word embeddings models [START_REF] Mikolov | Efcient estimation of word representations in vector space[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF][START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] allowed tremendous evolution in natural language processing. However, they embed the lexicon in dense representation spaces with opaque dimensions. It is possible to obtain an understanding of these models via probing [START_REF] Rogers | A Primer in BERTology: What We Know About How BERT Works[END_REF] and embedding matrix analysis [START_REF] Shin | Interpreting word embeddings with eigenvector analysis[END_REF]. However such methods are subject to criticism with regard to the interpretation that can actually be drawn from them [START_REF] Hewitt | Designing and interpreting probes with control tasks[END_REF][START_REF] Ravichander | Probing the probing paradigm: Does probing accuracy entail task relevance?[END_REF][START_REF] Elazar | Amnesic probing: Behavioral explanation with amnesic counterfactuals[END_REF]. This a posteriori approach to understanding models' decisions corresponds to the explainability paradigm in machine learning.

On the other hand, interpretability [START_REF] Rudin | Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead[END_REF]) is dened for word embedding models as the possibility to nd semantic (or syntactic) consistency in the dimensions of the embedding space [START_REF] Murphy | Learning effective and interpretable semantic models using non-negative sparse embedding[END_REF][START_REF] Faruqui | Sparse overcomplete word vector representations[END_REF][START_REF] Subramanian | Spine: Sparse interpretable neural embeddings[END_REF][START_REF] Prouteau | Are embedding spaces interpretable? results of an intrusion detection evaluation on a large french corpus[END_REF]. Models such as SPINE [START_REF] Subramanian | Spine: Sparse interpretable neural embeddings[END_REF] and SINr [START_REF] Prouteau | SINr: Fast Computing of Sparse Interpretable Node Representations is not a Sin! In Intelligent Data Analysis[END_REF] meet this requirement: Table 1 illustrates the interpretability of the dimensions resulting from such methods. These inherently interpretable approaches to represent the lexicon are deemed preferable for high-stakes downstream use such as medical or legal NLP [START_REF] Rudin | Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead[END_REF]. Interpretability also eases connection between word embeddings and linguistic models of the lexicon, since consistent semantic dimensions can be grasped as semantic features, which are used in a variety of theoretical models [START_REF] Jackendoff | Semantic and Cognition[END_REF][START_REF] Pottier | Recherches sur l'analyse sémantique en linguistique et en traduction mécanique[END_REF][START_REF] Rastier | Principes et conditions de la sémantique componentielle[END_REF].

As far as we know, only the interpretability of dimensions is considered in the literature and human evaluations such as the Word Intrusion Detection [START_REF] Murphy | Learning effective and interpretable semantic models using non-negative sparse embedding[END_REF] are targeted specically towards this aspect. In this paper, we introduce vector-level interpretability and dene it as the capacity for a speaker to make sense of the set of activated dimensions in a word vector. It is possible only if the set of dimensions to describe the word is limited. The size of this set is bounded by two different kinds of psychological experiments: semantic features production [START_REF] Garrard | Prototypicality, distinctiveness, and intercorrelation: Analyses of the semantic attributes of living and nonliving concepts[END_REF][START_REF] Mcrae | Semantic feature production norms for a large set of living and nonliving things[END_REF] and features retention [START_REF] Miller | The magical number seven, plus or minus two: Some limits on our capacity for processing information[END_REF][START_REF] Peterson | Short-term retention of individual verbal items[END_REF]. This body of literature comes to an agreement at roughly ten fea-tures. We consider in this paper that this number of features is a desirable horizon for vector-level interpretability. Following this objective and to further reduce the amount of information provided to the speaker, we also consider binary word vectors as in [START_REF] Faruqui | Sparse overcomplete word vector representations[END_REF]. Moreover, this binary approach is consistent with componential analysis [START_REF] Ward | Componential analysis and the study of meaning[END_REF][START_REF] Katz | The structure of a semantic theory[END_REF].

Considering these criteria, and to tend towards more interpretability, our work offers the following contributions :

• Rene interpretability by introducing additional criteria: stability and increased sparseness for vector-level interpretability.

• Evaluate the effects of increased word vector sparseness and binarity on performance.

• Illustrate the effects of increasing vector sparseness on the embedding space.

To this end, we introduce Section 2 the criteria for interpretability and their different settings in the literature. Section 3 introduces the models considered for our experiments. In Section 4, we detail the experimental setup adopted to evaluate the impact of sparsity as well as binarity on performance and vector-level interpretability. In Section 5, we demonstrate that the trade-off between sparsity and interpretability is not as strong as one would think. Finally, Section 6 illustrates the impact of sparsity on word vectors and discusses its benets.

Related work

Interpretability : criteria and models. The seminal article of [START_REF] Murphy | Learning effective and interpretable semantic models using non-negative sparse embedding[END_REF] paves the way towards psycholinguistically plausible distributional representations. The authors x the following set of constraints on the representation space: sparseness, positivity and performance. Sparseness is justied by the difculty to cover a vast vocabulary comprised of many different topics with a small set of features. Thus, a large number of dimensions is needed, but only some of those are activated for the description of each word. Positivity is motivated by the fact that storing null or negative features for each item of the lexicon is not cognitively efcient [START_REF] Palmer | Hierarchical structure in perceptual representation[END_REF][START_REF] Lee | Learning the parts of objects by nonnegative matrix factorization[END_REF]. The performance criterion is needed since it is possible to produce interpretable representations of the lexicon (e.g raw co-occurrence matrices) with subpar performances on intrinsic or extrinsic evaluations. This sparse interpretable word model research is carried on with SPOWV [START_REF] Faruqui | Sparse overcomplete word vector representations[END_REF], SPINE [START_REF] Subramanian | Spine: Sparse interpretable neural embeddings[END_REF] and SINr [START_REF] Prouteau | SINr: Fast Computing of Sparse Interpretable Node Representations is not a Sin! In Intelligent Data Analysis[END_REF]. The rst two models transform previously trained dense representations into sparse word embeddings while the latter builds a sparse embedding space from a word co-occurrence matrix. The word intrusion tests [START_REF] Murphy | Learning effective and interpretable semantic models using non-negative sparse embedding[END_REF][START_REF] Lut | Semantic structure and interpretability of word embeddings[END_REF][START_REF] Subramanian | Spine: Sparse interpretable neural embeddings[END_REF][START_REF] Prouteau | Are embedding spaces interpretable? results of an intrusion detection evaluation on a large french corpus[END_REF] are designed to assess the internal consistency of dimensions in the embedding space. As introduced Section 1, we wish to allow interpretability at the vector level which might benet from a smaller set of activated components in word vectors.

Stability. [START_REF] Pierrejean | Qualitative Evaluation of Word Embeddings: Investigating the Instability in Neural-Based Models[END_REF] demonstrate the nondeterminism of neural models' training which lead to variations in evaluation scores and word neighborhoods. On the front of explicability, new deterministic methods are emerging (Zafar and Khan, 2021). However, [START_REF] Rudin | Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead[END_REF] encourages to prioritise interpretable approaches over explicable approaches, motivating this work.

From these observations and as stated Section 1, we rene the criteria necessary to enable vectorlevel interpretability by redening sparsity and adding stability.

Binary embeddings. Prototypicality theory [START_REF] Rosch | Cognitive representations of semantic categories[END_REF][START_REF] Rosch | Basic objects in natural categories[END_REF] introduced the paradigm of weighted features in psychology and linguistics. However, feature-based analysis preempted this theoretical framework with componential analysis. This approach based on binary features was used by anthropological linguists [START_REF] Ward | Componential analysis and the study of meaning[END_REF], in structuralist work [START_REF] Pottier | Recherches sur l'analyse sémantique en linguistique et en traduction mécanique[END_REF] and in cognitively informed generativist frameworks [START_REF] Katz | The structure of a semantic theory[END_REF]. [START_REF] Faruqui | Sparse overcomplete word vector representations[END_REF] construct binary vectors using sparse coding to sparsify dense word embeddings in more dimensions than the original space-called overcomplete vectors (SPOWV). The model is then binarized simply by setting each non-zero value to one. In computer science, another use to binary models is to reduce the memory footprint of word embeddings by replacing oats with bits and also the compute needed to exploit these representations. It is especially critical in lowresource embedded systems-e.g mobile phones. [START_REF] Tissier | Near-lossless Binarization of Word Embeddings[END_REF] and [START_REF] Navali | Word Embedding Binarization with Semantic Information Preservation[END_REF] introduce autoencoder approaches to binarize Table 1: Words with the highest values on the top three dimensions of "insulin", "mint" and "oxygen" in Word2Vec, SPINE and SINr sparsied to 100 active dimensions per vector according to the protocol described Section 4. dense representations. Both of these models optimize for non-redundancy among dimensions and conservation of semantic information. Once vectors are binary, classical evaluation tasks such as word similarity or analogy may be redened with bitwise operations [START_REF] Sokal | A statistical method for evaluating systematic relationships[END_REF][START_REF] Tissier | Near-lossless Binarization of Word Embeddings[END_REF]. These models achieve competitive results to the baseline considering their small footprint.

3 Interpretable word embeddings SPINE and SPOWV achieve close results on intrinsic and downstream evaluations but SPINE scores better in terms of interpretability [START_REF] Subramanian | Spine: Sparse interpretable neural embeddings[END_REF], we thus do not consider SPOWV in the experiments that follow. Furthermore, SINr performances and interpretability are on a par with SPINE, we thus consider both SPINE and SINr as our reference interpretable models. SPINE. SPINE, rst introduced in Subramanian et al. ( 2018) derives sparse word embeddings from a previously trained dense model such as Word2Vec [START_REF] Mikolov | Efcient estimation of word representations in vector space[END_REF] or GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. Architecturally, it is an autoencoder whose hidden layer is of higher dimension than the dense input-e.g sparsifying from 300 dense dimensions to 1000 sparse dimensions. Three losses are implemented to enforce sparsity and interpretability. The Reconstruction Loss penalizes the poor reconstruction of the input representation from the output of the hidden layer, the Average Sparsity Loss and the Partial Sparsity Loss enforce sparse representations by limiting the number of active dimensions and skew vector values towards 0 or 1. SPINE has multiple hyperparameters: the minimum sparseness, the number of epochs and the vector output dimension. [START_REF] Prouteau | SINr: Fast Computing of Sparse Interpretable Node Representations is not a Sin! In Intelligent Data Analysis[END_REF], SINr is a graph-based approach to word embeddings. From a co-occurrence matrix extracted on a cor-pus, SINr builds a weighted word co-occurrence graph-words are represented by nodes and the number of co-occurrences by edges. A community detection algorithm, the Louvain method [START_REF] Vincent | Fast unfolding of communities in large networks[END_REF], then uncovers dense groups of cooccurring words in the graph. SINr then leverages the distribution of each node over this partition to derive a sparse representation-not all words co-occur with words from each community. The representation is sparse by design, each component of the embedding space is related to a community. Community detection is an unsupervised process admitting a single parameter allowing to potentially control the number of communities detected.

SINr. Introduced in

Methodology

Models. Alongside the models presented Section 3, Word2Vec is used as a baseline. We use the Skip-gram with negative sampling (SGNS) architecture and the parameters described in [START_REF] Levy | Neural word embedding as implicit matrix factorization[END_REF]. Word2Vec embeddings have 300 dimensions with a context window of 5 words. Since SPINE's number of dimensions is adjustable when SINr's is not-it is dependent on the number of communities detected-we base the number of dimensions of SPINE on SINr. Optimal performances for SINr are observed with the hyperparameter controlling the number of communities set to 50 resulting in 4460 dimensions for OANC [START_REF] Nancy | The open anc (oanc)[END_REF] and 8454 for BNC (Consortium, 2007) -the English corpora we use in our experiments is presented at the end of the next section. SPINE embeddings are trained from the Word2Vec model previously presented. The sparsity parameter of SPINE has little impact on the sparsity of the output. Subsequently, after several rounds of training, the model selected is that which achieves the best performances on the similarity task with a sparseness-95% after 1000 epochs-allowing further sparsication according to our experimental setup described hereinafter. Experimental framework. We introduce an experimental framework allowing to evaluate word embedding interpretability. We rst consider a performance-sparsity compromise. Our hypothesis is that sparse vectors are both more interpretable and psycholinguistically plausible. To control sparseness, we introduce our sparsication method: from each embedding model, we keep only the k top strongest dimensions by value in each vector-k is in range 250 -10. Components not in the top k for the vector are set to zero. Figure 1 presents the sparseness of SPINE and SINr with regard to the active dimensions threshold. In the case of Word2Vec, we keep the top k dimensions out of the absolute values from the vectors.

In our second setup, we study the impact of switching to binary vectors. The binarization step is straightforward, we simply replace all non-zero values in each sparsied and unsparsied model by 1 as in [START_REF] Faruqui | Sparse overcomplete word vector representations[END_REF].

To evaluate the quality of the representations after sparsication and binarization, we use the word similarity evaluation-the correlation between the cosine similarity of words in our model and similarity rated by humans. Selected datasets model a variety of relations : MEN [START_REF] Bruni | Multimodal distributional semantics[END_REF], WS353 [START_REF] Agirre | A study on similarity and relatedness using distributional and WordNet-based approaches[END_REF], SCWS [START_REF] Huang | Improving word representations via global context and multiple word prototypes[END_REF]. To evaluate the stability of vectors produced by SPINE and SINr, our second criterion to interpretability, we learn 10 models and present the averaged results.

As similarity datasets are mostly available in English, we use the British National Corpus (BNC) (Consortium, 2007) and the text part of the Open American National Corpus (OANC) (Nancy et

Results

Stability. The rst property we consider with regards to interpretability is the stability of the models trained. This experiment is twofold, it allows to show whether methods are stable and also sets reference values for the similarity evaluation prior to sparsifying. Each model was run ten times on the same data with the same hyperparameters.

As reported in Table 2, the three models achieve scores in close ranges, with all models showing some degree if variability, their standard deviation being non-zero across ten runs. While Word2Vec and SINr seem more stable than SPINE, the overall observed variability on the small samples of the vocabulary present in the similarity datasets hinders reproducibility and is a aw to the three model's interpretability.

Impact of sparsity on similarity. Results presented Figure 3 show the Pearson correlation scores on the similarity evaluation with regard to the number of components activated. The similarity scores are given with regard to the maximum number of top values kept in each vector according to our sparsication procedure. First, the three models achieve comparable results to those reported Table 2 up until 50 dimensions. More surprisingly, sparsifying SINr embeddings seems to improve performances. Sparsication may lter out noise from the base SINr model. Subsequently, there is not necessarily a trade-off between sparseness and efciency. Furthermore, the fact that results remain satisfactory on our Word2Vec control model despite the sparsication is an unexpected behavior and is interesting with regard to how the semantic information is organized in its vectors.

In order to approach the sparsity objective of 10 dimensions presented Section 1, the experiment is also conducted at this level. Although we observe an overall drop in performance and especially for Word2Vec, a signicant part of the semantic information is retained within these ten dimensions. Indeed, they allow to solve at least partially the similarity task. Even though the usefulness of this representation for downstream tasks can be discussed, it still allows to build interpretable word vectors despite the drop in performance. The low number of active dimensions render these models compatible with theoretical models leveraging semantic features, thus paving the way for new empirical opportunities.

Impact of binarization on similarity. Results presented Figure 3 follow the same display than sparsity results except that all models are binarized. Overall, we observe drops in performance across all models but to drastically varying extents. While SPINE and SINr lose some semantic information compared to the sparsied weighted models, they tend to retain performances of the same magnitude. This is especially true for models trained on BNC, considering that the models trained on OANC show bigger drops in word similarity performance. On the other hand, overall Word2Vec performances crumble with binarized vectors. This result is to be expected since Word2Vec is a dense model. We can observe a common pattern across all models, where performance of binarized embeddings increases with sparsication until 100 or 50 activated dimensions. Binarizing while maintaining a lot of active dimensions attens the hierarchy between components with strong values and others with low activations, thus otherwise very weak activations may gain weight in the vector as a result of binarization. In this case, the sparsication may remove noise from representations, by restoring a hierarchy between the few strong dimensions, activated with a 1 value, and the others set to 0. This denoising behavior resulting from sparsication seems common to binarized models, and weighted SINr.

Discussion

Our results show that there is not necessarily a trade-off between interpretability and performance. On the contrary, stability and increased sparseness of interpretable models can even improve results. At thresholds close to what is described in psycholinguistics, performances may remain accept-Figure 2: Word similarity performance (Pearson correlation) against maximum number of activated dimensions per vector for Word2Vec (left), SPINE (middle) and SINr (right). Performances on OANC are reported in yellow, and performances on BNC in blue.

Figure 3: Word similarity performance (Pearson correlation) on binary models against maximum number of activated dimensions per vector for Word2Vec (left), SPINE (middle) and SINr (right). Performances on OANC are reported in magenta, and performances on BNC in cyan.

able considering the number of dimensions activated. Interpretability is hard to visualize without a set objective. In the discussion ensuing, we illustrate the interpretability of models through visualizations on selected items.

Interpretability of the dimension. Interpretability of the dimensions can be assessed after conducting a word intrusion evaluation with humans, both SPINE and SINr's dimension interpretability have been previously evaluated without prior sparsication [START_REF] Subramanian | Spine: Sparse interpretable neural embeddings[END_REF][START_REF] Prouteau | Are embedding spaces interpretable? results of an intrusion detection evaluation on a large french corpus[END_REF]. The goal is to evaluate whether dimensions are interpretable-words with highest values on a dimension should be related. We present Table 1 top dimensions for three words as a glimpse into how interpretable dimensions of SPINE and SINr are in comparison with Word2Vec. As in previous evaluations, Word2Vec does not exhibit dimensions with related terms. If we consider the term "insulin", words on the rst three strongest dimensions in the vectors are all related to medical conditions or biological functions. The word "mint" presents interesting dimensions, for SPINE, the rst two dimensions are related to food and ingredients, the third one is less interpretable as one has trouble linking "spadefoot", a frog specie to "dial". SINr captures the polysemous nature of the word "mint" with top dimensions unrelated with one another. The rst one is most probably related to mint as an aromatic, meanwhile, the sec- ond one as the adjective describing guitars in mint condition, and the third one as a verb, to mint, in the sense of producing and managing currency. The same analysis can be drawn for the word "oxygen" where the use in the medical eld is represented alongside chemical characteristics.

Interpretability of the vector. We evaluated increasingly sparsied word embeddings with the hypothesis that fewer features makes interpreting words vectors themselves easier. Our evaluations show that this gain in interpretability is not necessarily at the cost of model performances, the sparsication of representation can even increase performances up to a certain sparseness level. The following paragraphs aim to illustrate interpretability at the word vector level.

We present Figure 4 the distribution of values in the 50 most (top of each gure) and least similar (bottom of each gure) words to "mint" for SPINE (a; c) and SINr (b; d) on BNC. Lines appearing vertically across gures show shared dimensions between vectors in the embedding space. The rst two gures (a; b) represent the shared features in the model prior to sparsication. SPINE presents vertical lines spanning most similar and least similar vectors, the embeddings seemingly share a large number of dimensions. SINr, on the other hand, exhibits a clear distinction between most and least similar words. One can clearly see shared dimensions among close neighbors of SINr for the word "mint". These rst two distributions need to be compared with the distributions observed after sparsifying the vectors (c; d). At the 100 active dimensions sparsity setup, SINr seems to display more shared dimensions than SPINE for the word "mint". We assume that the performance gain in the similarity task observed for SINr Figure 2 is due to a process of noise reduction induced by the sparsication of the model.

The interesting results on similarity evaluation showed by sparsied interpretable models seems to indicate that the most important part of the semantic information is stored in the few strongest components of each vector. This observation allows us to analyze these models through the lens of our constrained version of interpretability di- rected towards the interpretation of word vectors. A speaker might want to interpret word embeddings by composing the meaning of a word with a limited subset of the features that describe it. In this case, the stability of the models becomes an increasingly important issue. Indeed, interpreting dimensions amounts to nding a consistency to a set of words that strongly interpret a dimension. However, interpreting a word vector relies both on this consistency and the strength of the activation of each dimension for a given vector. Thereby, even subtle variations in the representation across runs may induce different interpretations.

Binary representations. Our last experiment aims to quantify the benet of weighted features over binary features. Considering results Figure 4, it appears that a signicant part of the semantic information for sparse interpretable models is encoded in the mere activation of a dimension by a vector. Binarity is a means of reducing time and memory complexity of semantic models and is undoubtedly benecial in embedded applications with low latency requirements or low resource hardware. We observe with Figure 5 that a SINr weighted model tends to have fewer and more strongly activated dimensions than a SPINE weighted model, which makes the former more alike binarized representations. This property facilitates the interpretation at the vector level: for example, dimensions 12 to 15 are strongly activated for "mint" and "thyme", and not at all for the other words, in the SINr representation. Recognizing the similarity of "mint" and "thyme", and their opposition to the other words, is easier when there is a clear gap between a strong activation and no activation of the dimension considered, like in a binarized vector.

Taking a step back, the comparison between weighted and binarized vectors performances allow us to pinpoint where the information is encoded. A signicant part of the semantic information is stored in the activation of a few dimensions for each word vector, but the dimensions weights are needed to reach the most competitive performances. This assessment is coherent with the theoretical paradigm shift mentioned Section 2. Furthermore, it appears that, while binarizing embeddings represents a cost in performance, sparsifying them is not necessarily a trade-off. In some cases, it might even be benecial.

Conclusion

Previously, the interpretability of embedding spaces focused mainly on dimension, this work re-dened interpretability from the vector standpoint. We state that stability of the models and sparsity are necessary conditions to intepretability. Constraining on sparsity echoes psycholinguistic plausibility, it is essential to nd semantic coherence within dimension of the embedding space but also to describe a word with a limited set of these dimensions. We hypothesize that vectors constrained following this protocol are interpretable by a speaker, since it becomes possible to manipulate this small number of dimensions in working memory.

Interpretable word embedding models achieve good results on the intrinsic word similarity evaluation task even with higher sparseness levels. SINr even benets from being sparsied. Furthermore, we show through examples that dimensions remain interpretable even on sparsied vectors and that, indeed words that are close in the embedding space are represented by a common set of dimensions. Lastly, we show that real-valued vectors are a slight improvement upon binary representation.

These results allow to reconsider the interpretability performance for distributed representations. A following step would be to conceive an evaluation framework to measure vector-level interpretability, allowing us to investigate if and how speakers would make sense of interpretable word vectors. Such models also open up new perspectives in which theoretical models describing the lexicon benet from semantic features of word embeddings. In the eld of semantic drift detection, it would also allow to easily characterize the drift by keeping track of the few dimensions at stake.

Figure 1 :

 1 Figure 1: Sparseness of SPINE and SINr according to the maximum number of activated dimensions per vector on OANC (top) and BNC (bottom). First data point of each model is sparseness before sparsication.

  Figure 4: Shared dimensions across 50 most and least similar words to "mint" in SPINE and SINr. The models are trained on BNC both without sparsication, and with a threshold set to 100 dimensions on BNC. The top half of each gure represents the most similar words and the bottom half the least similar words.

Figure 5 :

 5 Figure5: Word vectors on the set of top 5 shared dimensions for "mint", "insulin" and "oxygen" and their respective closest neighbors for SPINE (top) and SINr (bottom) onBNC.

  al.,

		MEN		WS353		SCWS
	BNC
		Spearman σ	Spearman σ	Spearman σ
	Word2Vec 0, 72	0, 002	0, 65	0, 005 0, 57	0, 002
	SPINE	0, 65	0, 006	0, 57	0, 01	0, 60	0, 004
	SINr	0, 66	0, 0006 0, 62	0, 002 0, 54	0, 001
		MEN		WS353		SCWS
	OANC
		Spearman σ	Spearman σ	Spearman σ
	Word2Vec 0, 43	0, 002	0, 50	0, 005 0, 46	0, 003
	SPINE	0, 36	0, 009	0, 43	0, 01	0, 39	0, 01
	SINr	0, 39	0, 0008 0, 44	0, 002 0, 39	0, 002

Table 2 :

 2 Stability results for the word similarity evaluation on BNC (top), and OANC (bottom). Average Pearson correlation coefcient and standard deviation σ over 10 runs.

	2011) to train our models. BNC contains 100 mil-
	lion tokens and OANC 11 million. Both corpus are
	composite in domain and genres. Those relatively
	small corpora, considering the standards in natu-
	ral language processing, are chosen because doc-
	umented corpora allow for ner interpretations of
	dimensions. Text preprocessing was performed us-
	ing spaCy : tokenization with named-entity chunk-
	ing, deletion of words shorter than three characters,
	of punctuation and of numerical characters. The
	minimum frequency for a type is set to 20. After
	preprocessing, OANC contains 20,814 types and
	roughly 4 million tokens, 58,687 types and 40 mil-
	lion tokens for BNC.

Acknowledgments

The work has been funded by the ANR project DIGING (ANR-21-CE23-0010).