Fuwei Pi

Pierre Dillard

Ranime Alameddine

Emmanuelle Benard

Astrid Wahl

Igor Ozerov

Anne Charrier

Laurent Limozin

Kheya Sengupta

Size-Tuneable Organic Nano-Dot Arrays: A Versatile Platform for Manipulating and Imaging Cells

Keywords: nano-bio-patterning, antibody nano-dots, nano-sphere lithography, T lymphocyte

Arrays of protein nano-dots with dot-size tuned independently of spacing (e.g. ~100 to 600 nm diameter for 900 nm spacing) are fabricated. The mechanism of size control is demonstrated, by numerical simulations, to arise from shadow effects during deposition of a sacrificial metal mask.

We functionalize the nano-dots with antibodies and embed them in a polymer-cushion or in lipid-bilayers, or transfer them to soft elastomers. Their ability to influence cell architecture and local membrane organization is demonstrated in T-lymphocytes, using reflection interference contrast and total internal reflection fluorescence microscopy.

 Experiments on cellular interactions with such patterned surfaces provide insights into the mechanisms of cell adhesion, and this knowledge in turn is used to optimize the performance of artificial soft tissue grafts and prosthesis (see reviews 10-12 and references

therein). In the context of cell biology and bio-engineering, arrays of solid-supported, sub-micro/nano scaled protein dots are currently used to influence cell architecture and to decipher cell function. [1][2][3][4][5][6][7][8][9][10] Some of the most successful strategies for creating bio-chemically funtionalized nano-dots are based on gold chemistry. [1][2][3][4][5][6][7][8][9][10] However, gold interacts strongly with light and as a consequence, advanced optical imaging techniques, such as total internal reflection fluorescence microscopy (TIRFM), reflection interference contrast microscopy (RICM) or optical super-resolution microscopy, which are essential tools for cell biology, are often not practicable, especially at high dot density. Replacing gold with organic molecules can overcome this problem. 13 Organic-molecule based nano-patterning, covering large surface area, has been implemented via nanosphere lithography (NL). [13][14][15] NL is widely used in diverse applications, including for optical meta-materials, and plasmonic sensors. [16][17][18][19][20] However in NL, as applied to organic molecule deposition, so far the dot-size could not be varied independently of the dot-spacing -an important requirement for any systematic cellular study. One reported strategy to overcome this problem was heat treatment of polystyrene beads to increase the foot-print, 13,21 but this process is difficult to control and has not been implemented yet for any reported biological study. Another strategy is to reduce the bead size by reactive ion etching, but it is applicable only for areas of few tens of microns (see ref. 22 for an overview). An additional challenge in cellular context comes from the fact that cells are sensitive to the compliance of their immediate environment. 23 Further, since in cell-cell interactions, the relevant proteins are embedded in the cell membrane, synthetic supported lipid bilayer (SLBs) membranes are often used as surrogate cells. [24][25][26][27] Patterning of soft substrates at the nano-scale, [28][29][30] or an SLB even at the micro-scale, 31- 36 is a current engineering challenge.

Here we report a NL based technique to create nano-dots whose size can be tuned independently of spacing and use it to pattern not only glass but also soft substrates and SLBs. As summarized in Fig. 1, our strategy relies on successive creation of two masks -a primary bead-mask that sets the lattice spacing and a secondary sacrificial aluminium mask, deposited through the primary mask, which sets the size of the motifs. The primary bead-mask is a self-assembled, two dimensional monolayer of colloidal beads (see SI for method of preparation and Fig. S1 for an image of the mask), through which aluminium is sputter-deposited from an aluminium target doped with silicon, using a standard radiofrequency (RF) magnetron sputtering system, to create the secondary mask (see SI for details and Fig. S2a for the geometry of deposition). The thickness of the aluminium layer (Al-thickness), defined as the the thickness of the deposited aluminium layer in an area well outside the bead-mask, was experimentally determined for each deposition run and was found to depend linearly on the duration of metal deposition (SI Fig. S2b) for a given RF power. After metal deposition, the bead-mask was removed mechanically by ultrasonication, revealing the secondary aluminium mask, with a welldefined array of pits giving access to the glass underneath (Fig. 1b, 2a, Fig. S3) for further deposition. This glass supported metal-mask can be stored almost indefinitely (tested for up to several months without any degradation).

We observed that the lateral size of the pits can be tuned by tuning Al-thickness (see scanning electron microscopy images in SI Fig. S3). The mechanism that links the thickness of the deposited metal layer to the size of the pits can be understood in terms of shadow-effects during the process of deposition. In the present set-up, the mean free path of aluminium atoms in the plasma at the operating pressure is estimated to be about 10 mm -smaller than the target to sample distance. Thus, multiple collisions ensure that atoms reach the bead-mask from all directions. However, at the scale of a micro-bead, the trajectory of the sputtered aluminium atoms in the ambient argon gas is ballistic, and the rules of optical geometry applicable to standard NL 16,17 should be valid. Accordingly, the deposition pattern is calculated by summing over all the angles the shadow cast by an array of beads for a given illumination direction.

Outside the mask, there are no shadows and the metal thickness is equal to Al-thickness defined above.

Under the beads, at the contact point between the bead and the surface, the shadow is absolute and therefore there is no aluminium deposition. The theoretical pattern therefore consists of an array of pits with sloping walls (Fig. 2b, Fig. S4). The radial profiles of the pits show that the steepness of the pitwall strongly depends on Al-thickness (as defined above). In Fig. 2c, it is seen that as the Al-thickness is increased from a low (20 nm) to a high (350 nm) value, the gradient becomes more pronounced (calculated for 2  m bead size and 2.1  m spacing).

A direct comparison of the theoretical profile of metal deposition with experiments is non-trivial. On the theoretical side, the simulation ignores the discreet nature of the deposition of aluminium atoms, and any possible reorganization after the deposition, which may give rise to roughness and graininess in experiments. On the experimental side, scanning electron microscopy (SEM) usually provides a means to probe metal layers at the required spatial resolution but here is rendered difficult by the fact that the samples are non-conducting, and therefore charging by electrons can significantly distort the images, especially for very thin films. Nevertheless, high resolution SEM images of the pits (Fig. 2a) show that as expected, there are different shadow regions that strongly resemble the simulations. Comparing Fig. 2a andb, it is seen that the characteristic asymmetric shadow at the edge of the mask (white arrow) is visible both in SEM images and in the simulated image. Similarly, in the interior of the mask, the shadow is symmetric in both images.

To correlate the pit-size and geometry to the final size of the protein nano-dots, we need to consider that the aluminium is deposited atom by atom and therefore a certain thickness is needed to achieve full surface coverage. However, for the subsequent bio-functionalization steps, complete surface coverage of the aluminium layer is essential, since otherwise the underlying surface is not adequately masked. This is achieved at an average aluminium deposition of 3 nm (as explained in SI). The effective pit-size calculated with this threshold of 3 nm, has a strong dependence on Al-thickness and a weak dependence on the bead spacing (Fig. 2d). The simulated pit-size will be compared below to the experimentally determined size of the protein nano-dots obtained after bio-functionalization.

For bio-functionalization, first an organosilane (APTES) is deposited through the glass supported metal-mask from the vapour phase. It is subsequently functionalized with biotin conjugated bovine serum albumin (bBSA) from an aqueous phase. The aluminium is removed by chemical lift-off, taking care to retain the functionality of the biotin moiety of bBSA, 36 revealing size-tuneable organic nano-dot arrays of bBSA (bBSA-STONA) on glass (Fig. 1c, see Fig. S5 for atomic force microscopy images).

The bare glass background is then covered with a passive material, here chosen to be a copolymer of polyethylene-glycol and poly-L-Lysin (PEGPLL), to prevent further protein absorption. [37][START_REF] Théry | Adhesive micropatterns for cells: a microcontact printing protocol[END_REF][39] After passivation, the bBSA dots are functionalized with the biotin binding protein neutravidin (NAV), which serves as a versatile linker for further functionalization (Fig. 1f).

The NAV-STONA, created with beads of diameter 4, 2, 1 or 0.5  m, and various Al-thickness were characterized by optical microscopy (OM) or atomic force microscopy (AFM) (Fig. 3a,S6). In each case, the fast Fourier transform (FFT) of the images (insets in Fig. 3b-e) indicates long range order in the system. As expected, the dot separation is set by the diameter of the beads used and, the dot size gradually decreases with increasing Al-thickness for each type of bead mask. By correct selection of bead-size and Al-thickness, the dot-size could be varied from ~1800 nm (1800 ± 100 nm, OM data) down to ~100 nm (110 ± 20 nm, AFM data). Plotting the pit-size from simulations against the experimental dot-size shows a linear correspondence (inset Fig. 3a). Note that the theoretical pit-size is systematically larger than the dot-size because the former is the diameter of the holes in the aluminium mask whereas for the latter we report the size as full width at half maximum (FWHM) from fluorescence images. Most importantly, we show that the spacing and size can be varied independently -similar dot-sizes are obtained for different dot-spacing and different sizes are obtained for same spacing -this was not hitherto possible with NL technology, especially when applied to bio-patterning of molecules.

The ability of STONA to influence cell behaviour and their compatibility with advanced optical imaging were tested on T lymphocytes (T cells). T cells play a central role in the acquired immune system. Supra-molecular organization at the adhesive interface of T cells with antigen presenting cells is thought to have a direct impact on the subsequent signalling, activation and function of these cells. 26,27,31,32 Influence of micron scale patterning of ligands in T cell has shed light on the importance of subcellular scale molecular organization in these cells. 41,42 Recent work with gold nano-dot based patterning revealed sensitivity of cellular activation to ligand organization at nano-scale but cells could not be imaged in detail. 1,43,44 Here we imaged T cells interacting with STONA decorated with the biotinylated antibody antiCD3, which on one hand binds specifically to the NAV dots created above, and on the other hand targets the T cell receptor (TCR) complex. The antiCD3-STONA influences the membrane topography as seen in RICM, which looks roughly uniform on non-patterned surfaces 26,27 but has a clear structuring on STONA (Fig. 4a). Moreover, the cell surface TCR molecules, which appear as randomly distributed puncta on homogeneously coated antiCD3, 27,40 are rearranged to follow closely the underlying antibody pattern (Fig. 4b). To our knowledge, this is the first time that such protein reorganization on the membrane of a lymphocyte was demonstrated at the sub-micron scale. There is also an impact on the architecture of the actin cytoskeleton as seen in TIRFM, which goes from the commonly observed peripheral organization 26,27,40 to partial co-localization with the dots (Fig. 4c).

To illustrate the versatility of the technique, we present two variations. First, antibody-STONA were created in a sea of SLB (Fig. 1h,4d) mimicking a cell membrane. 24 Such SLBs have previously been patterned by fabrication of metal grids 31,32,35,36 or by protein stamping, 31,33 however these studies were mostly on micron-scale motifs. Here, we introduce the nano-scale STONA in the SLB such that the lipids in the SLB remain mobile in the plane of the bilayer even after nano-patterning (Fig. S7). As before, the STONA impacts the cell membrane topography as seen in RICM (Fig. 4e) and the TCR distribution as seen in TIRFM (Fig. 4f). The second variation is STONA supported on an elastomeric substrate, which better mimics soft cells and also lend themselves to eventual use as a tool for force measurements. 43,44 NAV-STONA were transferred by reverse-contact-printing 13 to silicone rubber coated on glass, with a measured elasticity of 300 kPa (Fig. 1g, 4g, Fig. S8). Remarkably, cells could be imaged in both RICM and TIRFM through the PDMS in presence of the STONA (Fig. 4h,i).

In all the cases, the obtained STONA are fairly homogeneous. Within each sample, or for different samples for the same deposition run, the size varied by about 5 to 10 % (Table S1). The size variation from one deposition run to the other was slightly higher since the aluminium thickness was not exactly the same for each run. Another important characteristic of the STONA is the contrast -the amount of protein on the dots as compared to the amount absorbed outside the dots. Depending on the quality of metal deposition, this contrast, defined as the ratio of fluorescence intensities inside and outside the dots, is about 2 to 5.

The STONA method presented here relies on the combination of three key factors: (a) the use of selfassembly to create the primary-mask so that large area coverage is achieved in a simple manner, (b) metal-free strategies for coupling the bio-molecules ensuring compatibility with advanced optical microscopy, and (c) the use of shadow-effect in sputter deposition to lift the interdependence of dot size and spacing which hitherto limited applications of NL. The STONA protocol removes these constrains and truly fulfils the requirements of cell biologists interested in using nanobiopatterning as a tool to probe cellular interactions. The numerical simulations elucidate the underlying principle and provide an estimation of the size of the holes in the metal mask as a function of bead separation and metal layer thickness, making the method easily adaptable to other usage. Since the secondary metal-mask can be stored indefinitely, and the subsequent functionalization can be performed in a standard wet-laboratory, the metal-masks can potentially be prepared in specialised facilities and be made available for cell biological experiments on as-required basis. The simplicity of the STONA technology, including the linking-chemistry employed, makes it very versatile in terms of choice of material for the dots as well as the background. The feasibility of transfer by reverse-contact-printing on to elastomers further broadens the potential impact of STONA in the fast growing field of nanobiopatterning.

ASSOCIATED CONTENT Details of experimental methods, simulations, imaging and analysis, table of data corresponding to Fig. 3, SEM image of primary bead mask, schematic of metal deposition and SEM images of metal mask, simulated pattern, OM and AFM images of dots, diffusion measurement on SLB, images of nano-dot transfer to PDMS, schematic to demonstrate versatility of the technique. This material is available free of charge via the Internet at http://pubs.acs.org.

Figure 1.

Schematic representation of fabrication steps to create size-tuneable organic nano-dot arrays (STONAs). A primary colloidal bead-mask is self-assembled on glass (a), aluminium (Al) is sputter deposited (d), the beads are mechanically removed to obtain the secondary Aluminium mask (b), an organic molecule -biotinylated BSA (bBSA) -is bound to the exposed bare glass (e), and Al is chemically lifted off to produce the bBSA template (c). Subsequent passivation of the bare glass with PEGPLL (f) and appropriate functionalization of the bBSA (with neutravidin -NAV and the antibody antiCD3) yields antibody STONA surrounded by a polymer cushion (i). Alternatively, a supported lipid bilayer (SLB) can be used for the passivation step (h). NAV-STONA can be transferred to a soft elastomer -PDMS (g). Note that the drawings are not to scale -the square glass slide is 2424 mm, the bead diameter ranges from 0.5 to 4 μm, the thickness of the aluminium varied from ~15 to ~500 nm and the molecular dimensions of the proteins and lipids are in the range of few nm. The spacing of the dots is roughly equal to the diameter of the beads used in the primary masks and the diameter of the dots, tuned independently during creation of Aluminium masks, could be varied from ~100 nm to ~2 μm.

Figure 2 .

 2 Figure 2. Tuning pit-size with Al-thickness. (a) SEM image of the secondary Aluminium mask after removal of the primary bead-mask (2  m beads, 350 nm Al-thickness). Hexagonal pattern of aluminium free pits are clearly visible as black circles, few remaining beads from the primary mask are also seen. High-resolution zoom of two pits are shown. (b) Simulated pattern of aluminium deposition, colour coded from dark-blue (no deposition) to dark-red (maximum deposition, outside mask region, corresponds to Al-thickness). Equivalent regions on the SEM and simulated images are marked by white box. The white arrow points to characteristic shadow effect at the edge of the mask seen both in SEM and simulations. (c) Simulated profiles of the pits close to the substrate, showing the height profile of the deposited aluminium, calculated for various Al-thicknesses. The pit-size can be read off as the radius at which the profile is at least 3 nm thick. (d) The dependence of the simulated pit-size (normalized by the radius of the beads of the mask) on Al-thickness, calculated for various bead separations.

Figure 3 .

 3 Figure 3. Tuning STONA dot-size. (a) The size of FluoNAV-STONA plotted as a function of Althickness. For bead size 4 and 2  m, the dot-size is the full width at half maximum (FWHM) of intensity profiles from fluorescence images. For bead size 900 and 540 nm, the reported dot-size is edge to edge distance measured from AFM images. Symbols are data, solid line is guide to the eye. Error bars for dot-size are standard deviations (at least 30 dots for 3 image-fields), and for Al-thickness correspond to the full range of data (three independent measurements). Inset: comparison of pit-size predicted from simulations and protein dot-size measured as above. (b,c,d,e) Examples of fluorescence images for different spacing and dot-size, corresponding letters are marked in the graph above. Insets are corresponding fast Fourier transform (FFT, performed on image fields of about 100  m). Scale bars: 4  m.

Figure 4 .

 4 Figure 4. T cells on antibody STONA. (a) RICM image of T lymphocytes interacting with antiCD3-STONA (spacing 2  m, size ~700 nm), showing modified membrane topography, as compared to cells on homogeneous antiCD3 under equivalent conditions (inset). (b) corresponding TIRFM image of labelled T cell receptors (TCR), showing that the TCR are organized according to underlying antibody STONA pattern, corresponding FFT (inset, top) indicates long range order in the TCR organization: compare with random distribution on homogeneous antiCD3 (inset, bottom). (c) TIRFM of actin shows that the cytoskeletal structure is influenced by the presence of the STONA, as compared to homogeneous antiCD3 (inset). (d) FluoNAV-STONA (inset: NAV, red) in a sea of supported lipid bilayer (inset: SLB, green). The main image shows perfect complementarity of the NAV and SLB pattern. The FFT in the inset indicates long range order. (e,f) T cell interacting with STONA/SLB shows patterned membrane topography in RICM and organized TCR in TIRFM. (g) NAV-STONA on an elastomer (PDMS, Young modulus 300 kPa). (h,i) RICM and TIRFM of a T cell interacting with PDMS supported STONA. Scale bars: 4  m.

Acknowledgements

We thank Herve Dallaporta, Dominique Chatain, Carol Handwerker, John Blendell, and Paul Wynblatt for insights into metal deposition process, Frederic Bedu for SEM, Anne-Marie Lellouch and Martine Biarnes for help with cells, and HD/DC for careful reading of the manuscript. This work was partially funded by European Research Council via grant no. 307104 FP/2007-2013/ERC-Stg SYNINTER.

AUTHOR INFORMATION

Corresponding Author

* E-mail: sengupta@cinam.univ-mrs.fr.

Author contributions

FP developed the concept, put in place the protocol, fabricated and characterized the basic STONA, was involved in all experiments and participated in writing the article, PD participated in all the cell related work, RA was responsible for patterning and characterization of PDMS, AW for cells on PDMS, EB for SLB-STONA, IO for sputtering and SEM. AC co-supervised all PDMS related work and AFM measurements, and participated in project conception, LL co-supervised all T cell and optical microscopy work, did the simulation of sputtering, participated in project conception and manuscript preparation, KS conceived and supervised the entire project and wrote the article.

Notes

The authors declare no competing financial interests.

Supplementary Information

Size-Tuneable Organic Nano-Dot Arrays: A versatile platform for controlling and imaging cells Fuwei Pi 1 , Pierre Dillard 1,2 , Ranime Alameddine 1 , Emmanuelle Benard 1 , Astrid Wahl 1 , Igor Ozerov 1 , Anne Charrier 1 , Laurent Limozin 2 , Kheya Sengupta 1 .

Materials and Methods

Preparation of glass substrates

Ultra-hydrophilic glass coverslides (thickness = 170 ± 10 µm, Assistent, Karl Hecht KG, Germany), 24×24 mm, were obtained by cleaning on a Nanoplas DSB3 (France) oxygen plasma equipped with a SEREN R301 radio-frequency power supply (200W; ~2.3 Torr; Ar, 0.75 sccm; O 2 , 1.5 sccm, SEREN IPS, USA) at room temperature for 15 min or by using the following protocol: ultrasonication in 2% (v/v) aqueous solution of Hellmanex (Sigma, France) for 20 min, rinsing 10 times with ultrapure water (resistivity of 18.2 Mcm, Elga, UK), again ultrasonication in 2% Hellmanex solution for 20 min; ultrasonication in ultrapure water (2 times 20 min) with repeated rinsing; finally blow-drying with nitrogen stream. Note that both procedures render the contact angle of water on the glass coverlslides less than 5 o . Such high hydrophilicity is essential for the subsequent bead-mask creation.

Primary colloidal bead mask

Silica colloidal beads with stated diameters of 4 µm, 2 µm, 900 nm or 540 nm and stated polydispersity of 10 to 15 % (Polysciences, Inc., Germany) were washed 10 times with ultra-pure water before utilisation. An ultra-hydrophilic glass coverslide as prepared above was set at an angle and a pre-calibrated volume of the colloidal suspension was allowed to spread on the slide. Slow evaporation at ambient conditions coupled with strategic change of the angle results in uniform and large area coating with a monolayer of colloidal beads. For details see ref. 13.

Secondary sacrificial aluminium mask

Thin aluminium (Al) films were deposited on the glass coverslips through the primary bead mask using a radio-frequency (RF) magnetron sputtering system (SMC600 tool by ALCATEL, France, refurbished but without any special modifications) from a mixed Al(99%):Si(1%) target (purity ≥ 99.99%, Kurt J.

Lesker Company, USA). First, the deposition chamber was evacuated down to a pressure of 2.6×10 -4 Pa using a turbomolecular pump. Then pure argon atmosphere (purity ≥ 99.999%) was introduced with a flux of 10 sccm at a pressure of 0.8 Pa (6.6 mTorr). The RF power was held constant in the range of 300 to 400W giving typical film deposition rates from 2 to 3 nm/min. In order to remove the native oxide layer of aluminium on the target, the target was sputtered during two minutes with the closed shutter before each deposition run. The geometry of this sputtering system is off-axis and the mean free path is ~10 mm in the operating pressure range. Samples were placed at a distance of 105 mm from the target, onto a rotating table (3-5 rpm) (Fig. S2) in order to increase homogeneity.

The thickness of the aluminium film for each condition was measured on a silicon standard which was exposed to the same conditions as the sample for patterning. Before exposure, features with lateral sizes of 200 µm were obtained by optical lithography in a 1.3 µm thick photoresist (Shipley S1813) spin-coated on the planar polished silicon standard substrate. This standard was placed next to the sample during deposition and after deposition, the resist was lifted-off to obtain a sharp step corresponding to the deposited aluminium layer. The thickness was then measured by a stylus profiler (Dektak DXT-S, Bruker). The relation between the deposition time and layer thickness is nearly linear (SI Fig. S2b).

It should be noted that the precise curve relating the duration of aluminium deposition and the thickness of the deposited aluminium layer (SI Fig. S2b) and the one relating STONA size to the Al-thickness (Fig. 3) need to be experimentally determined for a given instrument and the chosen set of parameters.

We also noted that the quality of the STONA is rather sensitive to the presence of impurity in the deposited layer. One possible concern arises if different types of metal are deposited using the same chamber. Oxidation of aluminium can also be a problem. As can be expected, the procedure adapted below for chemical removal of aluminium does not remove other metals or aluminium oxide, and remaining traces of the contaminant may show up in the final sample as background that interacts strongly with light. Preliminary aluminium deposition runs on dummy samples may be necessary to purge the chamber completely of any remaining trace of the second metal.

After deposition of Al, the colloidal beads mask on coverslip was rinsed away by ultra-sonication in ultra-pure water for several seconds. The metal masks could be stored for up to at least four months without any deterioration.

bBSA template, PEG passivation and functionalization with antibody

The alumina mask coated slides were placed in a chamber containing (3-Aminopropyl) triethoxysilane (APTES, Sigma, France) in vapour phase, at 70 o C for one hour. Next, biotin conjugated Bovine Serum Albumin (bBSA, Sigma, France) at a concentration of 20 µg/ml in phosphate buffered saline (PBS, Sigma, France) was incubated on the samples for 30 min. Finally, to remove the sacrificial aluminium mask and reveal the bBSA STONA, the coverslip was placed in an alkaline PBS buffer (pH = 11.4, pH value was controlled by adding NaOH) for 2 h at room temperature. If required, the complete removal of Al was verified by optical microscopy. The aluminium free glass slides, containing bBSA dot arrays were thoroughly rinsed with neutral PBS buffer, pH = 7.2. Biotin, being a relatively simple molecule, retained its function even if subjected to high pH values. At this stage the coverslide was covered with uniform nano-dots of functional bBSA, separated by an expanse of bare glass. The slides could be stored at 4 o C up to 30 days.

The bare glass separating the bBSA dots was filled with poly(ethylene glycol) (PEG), here via incubation in PLL(20)-g(3.5)-PEG( 5) copolymer (Susos, Switzerland) dissolved in PBS at a concentration of 100  g/ml for 30 min. The PLL moiety, being positively charged, readily binds to glass, which is negatively charged at neutral pH after cleaning. The PEG moiety remains unbound and prevents further interaction of the glass with proteins in solution. 37 For T-cell adhesion studies, bBSA STONA was further functionalized by incubation with 2 µg/ml neutravidin (or neutravidin TexasRed, Invitrogen, USA) for 30 min, followed by incubation in antiCD3 2 µg/ml (multibiotinylated mouse anti-human clone UCHT1, Beckman Coulter, France) for 30 min.

Supported Lipid Bilayers (SLB)

Glass slides already functionalized with bBSA STONA were coated with SLB with 1,2-dioleoyl-snglycero-3-phosphocholine (DOPC, Avanti Polar Lipids, USA) as the matrix, into which fluorescent tracer lipids were sometimes dispersed to improve visualization and characterization. The bilayers were deposited one monolayer at a time from an air/water interface, at a surface pressure of 27 mN/m, by Langmuir-Blodgett/Langmuir-Shaefer (LB/LS) technique, using a film-balance (Nima, UK) S1, 27 . The surface was then blocked using 0.01% BSA and the bBSA STONA were functionalized as described above.

We used continuous photo-bleaching (CPB) S1,27 to quantify the diffusion of the fluorescent labelled tracer lipids which is expected to reflect the diffusion of the matrix as well. In CPB a well defined zone is observed while bleaching. Due to the constant entry of fresh molecules at the edge, a fluorescent halo is visible around the bleached zone when the lipids are mobile (see Fig. S7). The bleaching curve at the center of the illuminated zone can be fitted to calculate the bleaching rate and the intensity distribution across the halo can be fitted to yield the recovery decay length. The diffusion constant is estimated from these two values (see Ref. S1 or 27 for details) and is found to be 6  m 2 /s for the present example. This value is comparable to equivalent DOPC bilayers without the pattern. The size of the STONA in SLB, as visualized with fluoNAV, is comparable to those obtained with PEGPLL background and could be varied as described above.

Soft elastomer (PDMS) substrate

To prepare the soft substrates, liquid silicones (PDMS, CY52-276 A and B; Dow Corning, Japan) were mixed in 1:1 ratio. The mixture was spin-coated on glass slides cleaned as described above. The PDMS coated glass was baked at 100 o C for 15 min. Finally, a 8-10 µm thick PDMS layer, supported by glass, was obtained. The refractive index of this PDMS is 1.4. S2 The PDMS surface was activated by exposure to oxygen plasma (pressure 2.6 Torr, power 60 W) followed by treatment for 10 min. with 10% (v/v) solution of APTES in absolute ethanol. The APTES treated PDMS was then rinsed with 96% ethanol, dried and then treated with 8% (v/v) solution of glutaraldhehyde (1.06 g/ml) in ultra-pure water for 1 hour at room temperature. FluoNAV-STONA on glass were prepared as described above and were transferred to the PDMS (Young modulus = 300 kPa) by reverse contact printing 13 . The FluoNAV-STONA were correctly transferred as seen in epifluorescence images (Fig. 4g,S8). These FluoNAV-STONA on PDMS were further functionalized with antiCD3 by incubation in 4 µg/ml solution of the antibody for 4 h. Note that to demonstrate the proof of the principle and ensure cell spreading, the PDMS was not a priory passivated.

The elasticity was measured, after plasma treatment, by AFM (NTEGRA system, NT-MDT, Russia) using a cantilever with a 2 microns diameter colloidal-bead tip and stated spring constant of 0.02-0.77 N/m (MikroMash, Bulgaria). The colloidal cantilevers were imaged in SEM and were individually calibrated against a standard cantilever of known spring constant. AFM force measurements were conducted under water. At least 100 curves were recorded for each PDMS substrate under exactly the same conditions. The final value of the Young modulus was obtained from the average of all the curves. Force measurements show pure elastic behavior with no adhesion in the contact region. The data was fitted with Hertz model, using self-written routines in IGOR-Pro (Wavemetrics, USA).

Cell culture, fixation and labelling

Jurkat E6 T-lymphocytes were cultivated in RPMI complete medium supplemented with 10% Fetal Bovine Serum (Life Technologies, France) and with 1% GlutaMAX (Life Technologies). Last splitting was realized 24 h before experiments to ensure a concentration of 0.6 million cells/mL at the time of the experiment. Cells were deposited on the substrates in HEPES-BSA buffer (20 mM HEPES, pH=7.2, 137 mM NaCl, 5 mM KCl, 0.7 mM Na 2 HPO 4 , 4 mM D-glucose, 2 mM MgCl 2 , 1 mM CaCl 2 , 1% BSA) and incubated for 30 min at 37 °C 5% CO 2 . Fixation was performed after deposition with 2% paraformaldehyde during 15 min at 37 °C and followed by extensive rinsing with PBS. The cells were immunostained by incubation with 5 μg/ml of FITC labeled anti-  8 (BD Biosciences, USA), which is directed against the T-cell receptor, and/or with Phalloidin conjugated with Alexa-488 or TexasRed (Life technologies, USA) to mark the actin cytoskeleton, during 60 and 30 min. respectively, followed by extensive rinsing.

Imaging and analysis

Optical Microscopy (OM): For all optical imaging, an inverted microscope (Axiovert or AxioObserver, Zeiss, Germany) equipped with appropriate oil immersion objectives (Zeiss, Germany) and an EM-CCD camera (Andor, UK), was used. The protein-dots and cells were imaged in epifluorescence (100× 1.4 NA objective) and RICM/TIRFM (custom made 100× 1.46 NA Antiflex objective) respectively. Analysis and processing of optical images were done with Fiji [S3]/ImageJ v1.49d and IGOR Pro (WaveMatrics, USA) software packages.

Scanning Electron Microscopy (SEM):

Scanning electron microscopy images were taken either with a general purpose SEM (JEOL-5910, Jeol, Japan) equipped with a tungsten filament as electron source, or a e-beam lithography system (Pioneer, Raith, Germany) equipped with a hot cathode field emission electron gun and in-lens detector (Gemini column, Zeiss, Germany). Typical operating conditions we used were obtained using a 30 µm diaphragm at accelerating voltage of 2 to 10 kV depending on the sample. For the images presented here, a thin (~10 nm) layer of titanium was deposited on the sample to facilitate imaging.

Atomic Force Microscopy (AFM):

AFM imaging was done in tapping mode on a NTEGRA system (NT-MDT, Russia) at room temperature in PBS buffer solution (pH = 7.2). Silicon tips (CSC38, MikroMash, Bulgaria) with a typically resonance frequency of 20 kHz and ~8 nm tip radius were used. AFM images were analyzed using NTEGRA Imaging Analysis 2.1.2 software packages. Images were flattened in the scanning direction following standard practice.

Measuring dot size

To measure the dot size from optical images (OM), a line was drawn across a dot and the intensity profile was analyzed to calculate the full-width-at-half-maximum (FWHM). Similarly, for AFM images, the height profile across a dot was generated from the flattened height image, and the dot diameter was calculated as the edge-to-edge distance. For bead size of 4  m and 2  m, the dot-size is larger than the optical resolution and is reported as the FWHM from OM in Fig. 3. For smaller bead size of 900 nm and 540 nm, the dot-size is comparable to or smaller than the optical resolution, and therefore data from AFM is reported in Fig. 3. The data depicted in Fig. 3 is summarized in Table S1 below.

Numerical simulations of metal mask deposition.

The amount of metal deposited during isotropic but ballistic sputtering under a hexagonal two-dimensional array of N beads of radius R and spacing p was determined numerically as follows. For a given incidence direction defined by an azimuthal angle  and a polar angle  , the shadow cast by one single bead touching the surface at point O, is an elliptic surface centred at point C such that: OC=R/sin( )*(1/cos( )-cos( )); the long axis is given by R/cos(  and the short axis is R. The shadow cast by the bead array was defined as the union of the shadows cast by the N beads of the array.

Contrarily to previous modelling at fixed angle 17 , the lateral shift OC is here crucial. We consider N=25 beads organized in a hexagonal array, with the ratio of inter-spacing to bead-radius p = 2.0, 2.05, 2.1, 2.15 (results are reported for p=2.1 unless stated otherwise). The array is defined on a grid of 2000×2000 pixels corresponding to physical dimensions of 30×30 µm. The complementary surface corresponds to the one exposed to metal deposition at given incidence ( ,  ). The process is iterated for all incidence angles  and  by adding successively the exposed surface normalized by cos( ). Numerically, at each incidence angle, the deposit is represented by a binary image (as in ref. 18). The complete metal deposit is represented by a superposition of all binary images when the angle  varies between 0 and 1.1 rad (in 45 steps) and  between 0 and 2 (in 50 sin( )+4 steps). We verified that due to total eclipse of the surface by the mask for  >  /3 17 these angles need not be considered. At the end of the process, the metal thickness on the surface outside of the array was assigned to the experimentally measured Al-thickness. The threshold of Al thickness required to ensure full coverage of the glass was determined by assuming that Al atoms of radius 0.12 nm are randomly reaching the surface and arranging roughly as a hexagonal symmetric three dimensional lattice. We find that the glass is completely covered when, on average, a layer of 20 atoms are deposited. This corresponds to an average thickness of 3 nm. The simulation was done using self-written routines in Python Software (www.python.org) and analysis of cross-section profiles were done with IGOR Pro (Wavematrics, USA).


Zhuoyao Zhong

Weishen Pan

Jin + Lianwen

Harold Mouchè
email: harold.mouchere@univ-nantes.fr

Christian Viard-Gaudin
email: christian.viard-gaudin@univ-nantes.fr

SpottingNet: Learning the Similarity of Word Images with Convolutional Neural Network for Word Spotting in Handwritten Historical Documents

Keywords: component, word spotting, similarit learning, similarity score fusion, convolutional neural network I

Word spotting is a content-based retrieval process that obtains a ranked list of word image candidates similar to the query word in digital document images. In this paper, we present a convolutional neural network (CNN) based end-to-end approach for Query-by-Example (QBE) word spotting in handwritten historical documents. The presented models enable conjointly learning the representative word image descriptors and evaluating the similarity measure between word descriptors directly from the word image, which are the two crucial factors in this task. We propose a similarity score fusion method integrated with hybrid deep-learning classification and regression models to enhance word spotting performance. In addition, we present a sample generation method using location jitter to balance similar and dissimilar image pairs and enlarge the dataset. Experiments are conducted on the George Washington (GW) dataset without involving any recognition methods or prior word category information. Our experiments show that the proposed model yields a new state-of-the-art mean average precision (mAP) of 80.03%, significantly outperforming previous results.

INTRODUCTION

With the increased demand for text understanding and semantic analysis in document images and the lack of reliable and robust optical character recognition (OCR) for specific languages, symbols or low quality images, word spotting has attracted considerable attention from the document analysis community. The goal of word spotting is to search an ordered list of word image candidates similar to the queried word in the entire image dataset during the content-based retrieval procedure. According to the different representations of the query word, there are two main approaches to word spotting, namely, Query-by-String (QBS) [START_REF] Frinken | Adapting BLSTM neural network based keyword spotting trained on modern data to historical documents[END_REF] and Query-by-Example (QBE) [START_REF] Manmatha | Word spotting: A new approach to indexing handwriting[END_REF]. If the query word is represented as an arbitrary text sequence, the technique used is QBS; alternatively, if an image of the queried word exists in the document, QBE is used.

In this work, we focus on QBE and assume that an annotated location of the word in the document images is supplied; thus, we apply the cropped word image directly without involving text localization and segmentation. That is, we mainly address the content-based retrieval process. The two key challenges of QBE word spotting are the representation of word images and the similarity measure between word image representations. Most popular algorithms are based on applying projection profiles [START_REF] Rath | Word Image Matching Using Dynamic Time Warping[END_REF][3], word geometry information [START_REF] Rath | Word Image Matching Using Dynamic Time Warping[END_REF][3], statistical features [START_REF] Fischer | Lexicon-free Handwritten Word Spotting Using Character HMMs[END_REF], SIFT [6] [START_REF] Almazán | Word spotting and recognition with embedded attributes[END_REF], or other hand-crafted features to generate fixed or variable length feature sequences for word image descriptors, while using Dynamic Time Warping (DTW) [START_REF] Rath | Word Image Matching Using Dynamic Time Warping[END_REF][3][5] [6], Hidden Markov Models (HMMs) [6][7] and BLSTM [START_REF] Frinken | Adapting BLSTM neural network based keyword spotting trained on modern data to historical documents[END_REF] [START_REF] Frinken | A novel word spotting method based on recurrent neural networks[END_REF] to measure the relevance of the word feature representations; and achieved promising results on different word spotting benchmarks.

However, effectively spotting words in image-based documents, especially handwritten or historical documents, poses a great challenge owing to the large variability and confusion of handwritten manuscripts or the low quality of historical images, resulting in inadequate performance of previous research. For example, the state-of-the-art mean average precision (mAP) of a widely used historical handwritten dataset, known as the George Washington (GW) dataset [START_REF] Rath | Word Image Matching Using Dynamic Time Warping[END_REF], is about 62.72% [START_REF] Almazán | Word spotting and recognition with embedded attributes[END_REF] in the case where recognition methods and prior word category information are not accessible in the literature.

Conversely, with the increased development of deep learning algorithms in recent years, convolutional neural networks (CNN) [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] have enabled a breakthrough in many computer vision tasks [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Jaderberg | Reading text in the wild with convolutional neural networks[END_REF][START_REF] Noh | Learning Deconvolution Network for Semantic Segmentation[END_REF][START_REF] Karpathy | Deep visual-semantic alignments for generating image descriptions[END_REF]. Many research studies have subsequently applied CNNs to learn the similarity of two input image patches. Zbontar and Lecun [START_REF] Žbontar | Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches[END_REF] proposed CNN-based methods to learn the similarity of image patches for a stereo matching problem and showed improved performance results on the KITTI datasets. Hu et al. [START_REF] Hu | Discriminative Deep Metric Learning for Face Verification in the Wild[END_REF] presented a discriminative deep metric learning method using a CNN for face verification in the wild and achieved very promising performance on the LFW and YFT datasets. Zagoruyko et al. [START_REF] Zagoruyko | Learning to Compare Image Patches via Convolutional Neural Networks[END_REF] applied several different CNN architectures to compare image patches as well as encode a general similarity function and attained the best results on several local image patch benchmarks.

In this paper, we propose a similarity score fusion method based on a deeply trained classification and regression model in a complementary, facilitative and optimized manner. Our designed models can learn the word descriptors and similarity score between word descriptors directly from a cropped word image thanks to the CNN's outstanding end-to-end mechanism. Moreover, through the presented location jitter sample generation approach, we can construct a balanced and sufficiently large dataset with positive (similar) image pairs and negative (dissimilar) image pairs of word images to overcome data imbalance problem. Inspired by [START_REF] Zagoruyko | Learning to Compare Image Patches via Convolutional Neural Networks[END_REF], we design different traditional CNN architectures, including 2-channel, Siamese, and pseudo-Siamese networks to investigate the similarity learning performance of diverse models.

The remainder of this paper is set out as follows. Section 2 gives a detailed introduction to our approaches, while Section 3 presents our experimental results and an analysis thereof. Finally, the conclusion is given in Section 4.

II. METHODOLOGY

A. Learning optimization

It is noted that we can consider the similarity comparison of two word images as a classification as well as a regression problem by collecting a dataset with positive and negative word image pairs using supervised learning. For the classifier, we can obtain a 2-dimensional label distribution, while for the regressor, a similarity score ranging from 0 to 1 can be acquired directly. Suppose we have categories and the training data for each category are denoted as , , where and are the feature vector and label, respectively. Specifically, means it is a similar image pair, and zero denotes a dissimilar pair. In addition, is represented as the model parameters, and as the weight decay for regularization.

1) Classification model: With the development of deep learning algorithms, using the softmax loss function for classification problems [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] has become popular. The loss is described as:

‖ ‖ {∑ ∑ { } ∑ }
(1) where ∑ is a factor of normalization, and is the indicator function defined as following:

{ . ( 2
)
Furthermore, as hinge loss function is also widely used for binary classification, hence we intend to evaluate the performance of the same model with an embedded corresponding loss function for optimization. The hinge loss function is given by Eq. (3).

‖ ‖ {∑ ∑ { } }
(3) where is given as:

{ } { (4)
The loss functions defined in Eq. ( 1) and ( 3) can be minimized by the stochastic gradient descent algorithm during the training process.

2) Regression model: For the regression model, we use the binary cross-entropy loss for training and obtain a similarity value mapped onto the range [0, 1] by the sigmoid function. The loss is given:

‖ ‖ {∑ }
(5) where is defined as ⁄ The output of the model is expected to be closed to the target label.

3) Similarity score fusion method: As previously mentioned, the similarity measure for word image pairs can be regarded as a classification and a regression problem, and thus we assume that these two model types can be combined to improve the performance in a joint and complementary manner. In this section, we explain the similarity score fusion method based on a deep learning classifier and regressor model. Suppose and are the similarity confidence scores obtained from the classifier and regressor model, respectively. The classifier obviously provides a 2-dimensional class distribution, and can be computed in terms of probability normalization as:

∑ . (6)
The regressor outputs 1-dimesional similarity score directly and we map it onto the range [0, 1] via a sigmoid function. Thus, is denoted as: .

Moreover, the fusion similarity score is denoted as:

( ) (8
) where is the fusion function and is the input for . Our experiments in next section will show that our proposed score fusion method yields a better mAP result than using only single classification or regression model. An overview of similarity score fusion is illustrated in Fig. 1.

B. Location jitter for data augmentation

There are not nearly as many positive pairs as negative pairs, which may result in data imbalance and incorrect network guidance. For instance, suppose there are words from classes with images per class . In this case, we can collect positive image pairs, and negative pairs, which easily leads to the conclusion that , . Under the condition , it is obvious that is much greater than . Therefore, it is necessary to artificially enlarge the dataset by a label-preserving transformation to keep the data balanced. In this work, we present a random location jitter approach to augment the data and retain the word content information integrity as far as possible, which is crucial to word image similarity learning. The proposed sample generation method is described below:

 Given an original word image of size , calculate its center point coordinate . Denote as the jittered target center point coordinate; In this way, several jitter-generated samples for each original word image can be obtained. Using this data augmentation operation, we can greatly increase the number of positive pairs and then randomly select an equal number of negative pairs to keep the data balanced.

C. Network models

Inspired by [START_REF] Zagoruyko | Learning to Compare Image Patches via Convolutional Neural Networks[END_REF], we use the following three fundamental CNN models to compare the similarity of two cropped word images, as well as to evaluate the trade-off between flexibility and performance, while taking into consideration the efficiency of the different models.

Siamese network [START_REF] Bromley | Signature Verification using a "Siamese" Time Delay Neural Network[END_REF][21] is divided into two dependent branches with each branch sharing parameters in every weighted layer. Each branch takes one of image pair as input and proceeds to feature extraction via a series of convolutional and sub-pooling layers synchronously. The equal-size feature maps extracted from each branch are concatenated across the channel and then connected to the decision network (classification or regression network) to calculate the similarity measure of this word image pair.

Pseudo-siamese network resembles the Siamese net-work, but differs in that its two branches are independent without shared parameters. This increases the number of parameters, while improving the flexibility of the network architecture for each branch.

2-channel network applies an image pair as a 2-channel input image array, with the channels jointly convolved in the first convolutional layer. This is the most flexible and fastest of the three network models with respect to the training process. The three network model architectures designed are depicted in Figs. 2 and3.

III. EXPERIMENTS AND ANALYSIS

A. Experimental data

The GW dataset [START_REF] Rath | Word Image Matching Using Dynamic Time Warping[END_REF] is a collection of 20 pages of letters and 4860 words annotated at word level written by George Washington and his assistants. The dataset was split randomly into 15 pages for training and the remaining five pages for testing, so a certain category of word, which may exist in the testing set, is not likely to appear in the training set. Having evaluated the aspect ratio (height/width) and width distribution (width is more important for the word image itself) of the dataset, we decided to normalize the word images to size 40×100. Standard mAP was used as the evaluation method.

B. Implementation details

We trained our proposed classification and regression model from scratch via back-propagation and stochastic gradient descent (SGD) in an end-to-end manner. All the weights of the new layers were initialized with a zero mean and a standard deviation of 0.01 Gaussian distribution. The base learning rate was 0.01 and divided by 10 for each 50K mini-batch until convergence. We use a momentum of 0.9 and weight decay of 0.0002. Our experiments were carried out on the popular CNN platform, Caffe [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF], using a GTX TITAN BLACK GPU card. Furthermore, for the similarity score fusion function , we simply set as: ( ) in practice after empirically observation.

C. Evaluation of three classification models embedded corresonding binary loss function for optimization

In this section, we compare the performance of the proposed three fundamental classification models with an embedded corresponding loss function on the five-page testing dataset over five runs using the same training details and strategies. The results are given in Table 1. It is generally seen that the 2-channel network is superior to the other two models and the softmax loss function is better suited to this similarity learning task. The best mean mAP and standard error (%) performance of the 2-channel model with softmax loss function is 78.46 0.18.

D. Analysis of similarity score fusion approach

An important finding from our experiments is that the 2-channel network precedes the other model architecture.

Thus, when considering this similarity learning task as a regression problem, we also trained a 2-channel CNN-based regression model with embedded binary cross-entropy loss for optimizing from scratch to obtain a 1-dimensional similarity score. In this way, we obtained an average mAP and standard error (%) of 77.51±0.27. As mentioned above, the mAP rate of the proposed 2-channel classification model with softmax loss function for training is 78.46±0.18. In this section, we evaluate these two models ensemble performance using the proposed similarity score fusion method. The corresponding experiments are denoted as 2-ch-cls, 2-ch-reg, and 2-ch-cls+2-ch-reg, respectively. Figure 5 shows the results of the three experiments over five runs. It can be seen that our score fusion method yields a better mAP result than using only a single classifier or regressor model with the best mAP of 80.03%.

E. Comparison with different methods on GW dataset

To further evaluate our proposed approaches, we compared the performance of different QBE word spotting methods on the GW dataset without involving recognition methods or any word category prior information. The compared methods include DTW [START_REF] Rodriguez-Serrano | A Model-Based Sequence Similarity withApplication to Handwritten Word Spotting[END_REF], SC-HMM [START_REF] Rodriguez-Serrano | A Model-Based Sequence Similarity withApplication to Handwritten Word Spotting[END_REF], BoVW+Cosine Distance [6], and SIFT+FV [START_REF] Almazán | Word spotting and recognition with embedded attributes[END_REF] with the results given in Table 2. It is worth noting that not all the results are directly comparable because the training and testing dataset partitions are not official and may differ from each other. However, the published results in the literature provide an evaluable measure of the proposed ideas in the expected scope. In the table, we can see that our 2-ch-cls or 2-ch-cls+2-ch-reg model outperforms previous state-of-the-art results with a 15%-38% improvement in mAP. To the best of our knowledge, this is the first time CNN-based methods have been applied to learn the similarity of two word image patches for QBE word spotting. The qualitative results for word spotting on the GW dataset are shown in Fig. 6.

IV. CONCLUSION AND DISCUSSION

In this paper, we presented a CNN based end-to-end approach for QBE word spotting in historical handwritten documents. A new similarity score fusion method based on a deep-learning classification model and regression model was proposed to achieve better retrieval performance. Furthermore, we presented a location jitter sample generation method that retains the maximum word content information, to construct a balanced and sufficiently large image pair dataset. In addition, we explored three different CNN architectures embedding two common loss functions for classification. We showed that the 2-channel network is superior to both the Siamese and pseudo-Siamese networks while the softmax loss function is better suited to this task. Using the proposed score fusion method, we achieved improved mAP performance in a complementarily optimized way. The best mAP rate we obtained is 80.03% on the GW dataset, which is a significant improvement over previous approaches

Figure 1 .

 1 Figure 1. Score fusion method based on deep-learning classifier and regressor models.

  denotes the union distribution, and represent the jitter scope on the x-and y-axes, respectively, and with used in practice. Sample and to obtain ;  Apply as the central anchor to crop the image patch of size .

Figure 2 .Figure 3 .Fig. 5 .

 235 Figure 2. Illustration of 2-channel regression network model.

Table 1

 1 Comparison of three network models with coresponding loss function in terms of mAP for five runs.

	Model+loss function 2-channel+softmax loss 2-channel+hinge loss Siamese+softmax loss Siamese+hinge loss	Mean mAP and standard error(%) 78.46 0.18 72.40 0.11 77.88 0.26 70.40 0.07	Table 2 mAP of our proposed techniques and reported meth-ods using the GW for word spotting Methods mAP(%) BoVW+Cosine Distance [6] 42.20 DTW [5] 50.00 SC-HMM [5] 53.10
	Pseudo-Siamese+	67.17 0.23	SIFT+FV [9]	62.72
	softmax loss Pseudo-Siamese+ hinge loss	62.13 0.15	Proposed 2-ch-cls Proposed 2-ch-cls+2-ch-reg	78.46 80.03

ACKNOWLEDGMENT

We gratefully appreciate the support of NVIDIA Corporation with the donation of GPUs used for this research.