Hongjun Liu

S Battiato

A L L Pellegrino

P Paoli

P Rossi

Carmen Jimenez

G Malandrino

David Muñoz-Rojas

C Jiménez

D Muñoz- Rojas

Deposition of metallic silver coatings by Aerosol Assisted MOCVD using two new silver [small beta]-diketonate adduct metalorganic precursors

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Silver is a noble metal with unique physical properties including high conductivity and plasmonic, anti-bacterial and self-cleaning effects. 1-3 Metalorganic Chemical Vapor Deposition (MOCVD) is an industrially relevant technique for obtaining functional coatings, including Ag metallic coatings. In MOCVD, the process can be largely altered by the way in which precursors are designed and delivered. Thus, depending on how precursors are fed into the deposition chamber, MOCVD is classified into several branches: Hot Wall CVD (HW-CVD), Cold Wall CVD (CW-CVD), Pulsed Injection CVD (PI-CVD), Aerosol Assisted CVD (AA-CVD), etc. 4 This big family of MOCVD systems also brings new challenges for the design of new silver precursors. In particular, AA-CVD is an interesting CVD approach since it operates at ambient pressure. For AA-CVD, alcohol solvents are preferred due to the concerns related to the use of other more hazardous solvents, thus precursors being soluble in alcohol solvents are desirable. [START_REF] Marchand | [END_REF]6 In the case of the currently known silver metalorganic precursors, solubility in alcohols is low since silver complex molecules tend to polymerise when dissolved in the solvent. This limits the number of available silver precursors suitable for AA-CVD systems. AgNO 3 has indeed been reported as precursor for Ag coatings using AA-MOCVD by N. Bahlawane et al. and Ponja et al. 7,8 But given the nonorganometallic nature ). As shown in Table 1, the different silver metalorganic precursors reported to date were mostly used in conventional or PI-CVD systems. Few examples of ALD of Ag coatings have also been reported previously. [20][21][22][23][24][25] In the cases where AA-CVD was used, nonenvironmentally friendly solvents, such as THF and toluene, were used in combination with organometallic precursors. 10,11,16 Concerning the design of Ag metalorganic precursors, previous works have introduced different systems, including β-diketonate adducts Ag(hfac))(L) 26 or Ag(fod)(L), where L were tertiary phosphines (PPh 3 ), 2,5,8,11,14pentaoxapentadecane or bis(trimethylsilyl)-acetylene) or carboxylates (Ag(O 2 C-L)). 9,11,15 Apart from the low solubility of these precursors in alcohol solvents, other issues such as low volatility or the high toxicity of the ligands have been also reported. 14, 16, 20 In addition, some of the ligands are rather expensive, contributing to a higher price of the final Ag precursor. Thus, new silver precursors with less toxic ligands, good solubility in alcohol solvents and clean rapid decomposition, are still needed, ideally using less expensive ligands.

In this work, two new silver metalorganic precursors with general formula Ag(hfac)-L were designed and synthesized using different ligands: [Ag(hfac)(phen)] (AgP, phen=1,10phenanthroline) and [Ag(triglyme) 2 ] + [Ag(hfac) 2 ] -(AgT, triglyme = 2,5,8,11-tetraoxadodecane). In addition, all the ligands Hhfac, triglyme and phenanthroline are not very expensive, thus reducing the final precursor cost. In this study we describe the synthesis and characterization of these new silver complexes and the results of their use as precursors for the deposition of Ag coatings via AA-CVD with alcohol solvents.

Experimental

Starting materials: Ag 2 O (99%, Strem Chemicals), H-hfac (99%, Strem Chemicals), triglyme (99%, Sigma Aldrich), 1,10phenanthroline (anhydrous 99%, Strem Chemicals), dichloromethane (99.8%, Sigma Aldrich), and n-pentane (99%, Sigma Aldrich) were used as received for the synthesis. Synthesis of [Ag(hfac)(phen)] (AgP): The synthesis was performed suspending 2.20 g of Ag 2 O (9.49 mmol) in 50 ml of dichloromethane. 2.54 g of Phenanthroline (14.14 mmol) were then added to the suspension. After 10 min, 2.94 g of H-hfac (14.14 mmol) were added to the mixture, which was refluxed under stirring for 1 h. Excess silver oxide was filtered off. Lastly, yellow crystals were obtained after removal of the solvent upon evaporation. A pure crystalline solid was obtained after purification with small quantities of n-pentane, in which the adduct is insoluble, and then vacuum dried. The reaction yield was 72%. The melting point of the crude product was 158-164 °C (dec.) (760 Torr Structures were solved by using the SIR-2004 package 28 and subsequently refined on the F 2 values by the full-matrix leastsquares program SHELXL-2013. 29 Geometrical calculations were performed by PARST97 30 and molecular plots were produced by the program Mercury (v3.7).

31

The crystal data and refinement parameters of the new Ag adducts are reported in In AgT all the CF 3 moieties are disordered. Such a disorder was modelled by introducing two positions for each fluorine atom bonded to C4, C5 and C10 (refined occupancy factors: 0.62/0.38 for the atoms bonded to C4; 0.83/0.17 for the atoms bonded to C5; 0.76/0.34 for the atoms bonded to C10) In addition all the atoms of C4(F 3 ) moiety were set in double position (refined occupancy factors:0.55/0.45). For all the non-hydrogen atoms in AgP and AgT, anisotropic thermal parameters were used.

For the hydrogen atoms in AgP, they were found in the Fourier difference map and their positions were freely refined, while their thermal parameters were refined accordingly to the bound atoms. The hydrogen atoms in AgT were introduced in calculated position and refined in agreement with the carbon atom to which they are bound. Chemical shifts (δ) are expressed in parts per million (ppm). Spectra were referenced to the residual proton solvent peaks; coupling constant (J) values are given in Hz. Thermogravimetric analyses were made using a Mettler Toledo TGA/SDTA 851e. Dynamic thermal analyses were carried out under purified nitrogen flow (30 sccm) at atmospheric pressure with a 5 °C/min heating rate. Weights of the samples were between 10-15 mg. Melting points were taken on very small single crystals using a Kofler hot-stage microscope.

Deposition parameters:

The depositions of Ag coatings were conducted in a home-made AA-CVD system at 320 °C, using ethanol as solvent with a concentration of 0.01 M. 2 L/min and 5 L/min of N 2 were used as carrier gas and accelerating gas, respectively. The solution consumption rate was fixed at around 1.4 ml/min and deposition duration was between 30 to 90 min, glass (Corning Glass) and Si wafers were used as substrate. The detailed parameters are shown in Table 3.

Characterization of the Ag coatings: the Ag coatings were analysed using a Scanning Electron Microscope (SEM-FEG Environmental FEI QUANTA 250). Particle diameter distribution was calculated from SEM images using , the Image J software. 32 The Ag coating were scratched for further characterization with Transmission Electron Microscopy (TEM-JEM2010-JEOL). Carbon residues were analysed with RAMAN Spectroscopy (Jobin Yvon/Horiba LabRam) equipped with a liquid nitrogen cooled charge coupled device detector.

Results and discussion

Precursor synthesis, structure and properties

The synthesis of AgP and AgT adducts was carried out through a single step reaction, from silver (I) oxide, hexafluoroacetylacetone and the neutral ligands, phenanthroline or triglyme, in dichloromethane, as described above and shown in Equations ( 1) and (2). 4). The 1 H NMR spectrum of AgP (Fig. S1) has been attributed in accordance to the assignments of references 33,34 . The spectrum shows a singlet (5.79 ) associated with the ring proton of the hfa ligand, while the phenanthroline shows peaks in the typical range of aromatic ligand from 7.6 to 9.1 . The a protons are observed as a singlet at 9.04 , the b proton as a multiplet at 7.79  , the c protons as a doublet of doublets at 8.40  and, finally, the e protons as a doublet at 7.89 . The phen peaks have been assigned following the notation reported as a note in table 4. The 1:1 (hfa:phen) ratio (Fig. S1) found through evaluation of intensities of pertinent resonances finds counterpart in the ratio obtained through single crystal X-ray diffraction. The 13 C NMR spectrum of AgP (Fig. S2) shows resonances associated with the coordinated hfa ligand, which consist of a singlet ( 86) for the CH group, quartets ( 118) for the CF 3 groups and quartets ( 177) for the CO groups. The quartets are due to first order (CF 3 ; 4) can be assigned using comparative arguments with data from the related lanthanum complex. 31 The spectrum (Fig. S3) shows a singlet at 5. C NMR spectrum of AgT (Fig. S4) is similar to that found for the AgP adduct in regard to the hfa ligand. Resonances associated with the coordinated hfa ligand consist of a singlet (  86) for the CH group, a quartet ( 117) for the CF 3 groups and a quartet ( 177) for the CO groups. The quartets are due to first order (CF 3 ; The solid state molecular structure of AgP was determined by single crystal X-ray diffraction and consists of dimers made up of two silver complexes related by an inversion centre (-x+1,y, -z+2), shown in Fig. 2 a). In each complex the silver ion is bound to the oxygen and nitrogen atoms provided by the -hfac and -phen ligands, respectively. The coordination sphere is a slightly distorted square planar (as provided by the angle between the AgO2 and AgN2 planes, 7.2(2)°, and the small displacement of the silver ion with respect to the four donor atoms, 0.1110(5) Å). Even if quite uncommon 35 this kind of geometry has been observed in similar Ag(I) complexes.

2,33

Bond distances and angles are within the expected range for this kind of compounds (Table 5). 36 Additionally, as already found in other [Ag(hfac)-L] complexes The distance separating the two metal centres (3.0843(7) Å) is definitely longer than the silver-silver one in the metal (2.888 Å). 38 As a consequence, the two metal ions are not interacting significantly. 21,[START_REF] Griffith | [END_REF] However, given the fact that there are no bridging ligands holding together the dimer, a weak stabilizing interaction between the metal ions might be postulated. 40 In addition, intra-dimer π stacking appeared between the -hfac and -phen rings (the distances separating the centroids (C T ) † is . In addition, inter-dimer π stacking interactions involving the phen molecules provided by two contiguous dimers are present (the mean phen planes are 3.5 Å apart). Finally, dimers propagates along the b axis (Fig. 2 c) through a weak C(7)H(7)...O(1)' intermolecular interaction (' =x,1+y,z; D...A and H...A distances 3.628(6) and 2.75(6) Å, respectively, DHA angle 161(6)°). The solid state structure of compound AgT was also determined by means of single crystal X-ray diffraction. In the asymmetric unit of AgT one [Ag(triglyme) 2 ]

+ cation and one [Ag(hfac) 2 ] -anion are present (Fig. 3). In the two ions, the silver atom shows different coordination spheres. In fact, in the [Ag(triglyme) 2 ] + cation, the silver atom Ag(2) is surrounded by two triglyme molecules and bound to all the eight oxygen atoms that those ligands provide. It is worth noting that, although one Ag-O distance [i.e. Ag(1)-O (8)] is longer than the others (see Table 5), it is inside the Ag-O range as retrieved in the Cambridge Structural Database (CSD v. 5.37). [START_REF] Haynes | CRC Handbook of Chemistry and Physics[END_REF] The two triglyme molecules are almost perpendicular to each other, the angle between the mean planes defined by the nonhydrogen atoms of the two molecules being 85.4(1)°. Finally the resulting coordination polyhedron could be described as a distorted dodecahedron.

Concerning the [Ag(hfac) 2 ] -anion, the silver atom Ag(1) is fourcoordinated by the oxygen atoms of two -hfac anions in a resulting pseudo-tetrahedral arrangement. This coordination arrangement resembles well that already found in a similar silver complex [[Ag(hmten)][Ag(hfac) 2 ]; hmten = hexamethyltriethylene tetra-amine]. [START_REF] Haynes | CRC Handbook of Chemistry and Physics[END_REF] The two -hfac anions lay in two Thus it can be concluded that the precursor does not present a clean decomposition, with some organic species still remaining at 400 °C. Conversely, the TGA curve related to AgT (Fig. 4 b) shows a single (see derivative) rapid weight loss in the range 150-250 °C, yielding a 22.5% residue when heating to 400 °C. The stoichiometric Ag weight portion in the precursor being 21.9%, the value obtained from TGA is thus well tuned with the expected percentage of metallic silver in the compound, implying a clean and complete decomposition of the precursor. By comparing the TGA results for those two precursors, AgT appears as a better silver metalorganic precursor.

Silver coatings by MOCVD and characterizations

To test the two silver precursors, depositions at 320 °C for different durations were carried out in a homemade Aerosol Assisted CVD system, as detailed above in Table 3. Both precursors were carefully grinded before being dissolved in ethanol, reaching 10 mM concentration at room temperature.

Comparing the results of depositions using the different precursors, AgT showed higher deposition efficiency than AgP (see SEM images in Fig. 5). For a deposition duration of 0.5 h, AgP yielded a yellow coating while AgT gave an orange colour with a higher particle density (Fig. 5 a) and c)). Increasing the deposition duration to 1.5 h, coatings from AgP and AgT both presented purple colour coatings but the grain density for the latter was still higher than for the former (Fig. 5 b) and d)). For the deposition conditions used, the particles in the coating are not connected and thus the samples present high resistance values, namely a sheet resistance of 5 x 10 8 Ω/cm 2 measured by 4 probes. As well, the adhesion between the Ag coating and glass substrate is poor and the films can be easily removed. The carbon contaminations found for AgP, both by Raman and TEM, are due to the incomplete decomposition of the precursor under the deposition conditions used, and are in agreement with the TGA results shown above. Thus, although AgP is able to yield Ag coatings when used as precursor in AA-CVD, these present carbon species which are expected to affect the properties of the final coating (i.e. conductivity) and thus can be not suitable for certain applications.

Conclusions

In this work, two new silver metal organic precursors have been synthesized using less expensive and toxic ligands, namely, [Ag(hfac)phen] (AgP) and [Ag(triglyme) 2 ] + [Ag(hfac) 2 ] -(AgT). FTIR spectroscopy has confirmed the formation of such molecular complexes. Furthermore, the crystal structures of both precursors were solved by means of single crystal X-ray diffraction. TGA analyses have shown that AgT is a precursor with a more rapid and clean decomposition than AgP. The new adducts present enough solubility in ethanol and have been tested in an Aerosol Assisted CVD system. Depositions by AA-CVD revealed that the two precursors, used under similar CVD parameters, produced coatings with different surface morphologies. Coatings deposited from AgP are made of Ag nanoparticles presenting a carbonaceous shell. Conversely, Ag coatings from AgT consist of pure Ag nanoparticles and present high crystallinity. In conclusion, our results show that the new AgT adduct is a good precursor for depositing metallic Ag coatings via AA-CVD using alcohol solvents.

27

 27

  fluorine atoms bound to the carbon atom C5 of the -hfac anion, in AgP are disordered. Such disorder was modelled by introducing three positions for each fluorine atom (refined occupancy factors: 0.44, 0.41, 0.15).

Ag 2 O 1 . 1 ,Figure 1 .

 2111 Figure 1. FT-IR spectra of AgP (a) and AgT (b) in nujol.

* 1 H and 13 C

 113 The following notation has been used for phenanthroline.† The following notation has been used for triglyme.(CH 3 -O-CH 2 -CH 2 -O-CH 2 ) 2 .a b c dwhile the C=C stretching vibration is associated with the absorptions at 1500-1550 cm -1 . In addition, the bands present in the interval 1050-1400 cm -1 can be attributed to typical C-F vibrations of the -hfac ligand present in both adducts. For AgP, the bands observed in the 700-1100 cm -1 range are considered fingerprints of the phenanthroline coordination to silver ion. In AgT, the broad band observed in the 1000-1300 cm -1 range might be associated with absorptions from polyether C-O bending and/or stretching overlapped with the C-F stretching of the β-diketonate. Furthermore, bands at 700-1100 cm -1 are related with glyme modes. The C-H glyme stretching modes, lying in the 2800-3000 cm -1 range, overlap with nujol features. NMR of AgP and AgT were recorded in CDCl 3 (Table

1 H

 1 Hz) coupling with the CF 3 fluorine atoms. The coordinated phenanthroline signals are observed in the range of 127-152 . Assignments are reported in the table 4 together with the notation used for phenanthroline. The NMR spectrum of AgT (Table

  81  associated with the ring proton of the hfa ligand. Different resonances are associated with the triglyme framework. The methyl groups (a protons) are always observed as a singlet at  = 3.60. The b and c protons are observed as a multiplet centered at  = 3.62. The protons d are associated with a singlet at  = 3.71. Evaluation of intensities of pertinent resonances points to a 1:1 (hfa: triglyme) ratio in accordance with single crystal X-ray diffraction.The13

1 J

 1 = 289 Hz) and second order (CO; 2 J = 32 Hz) coupling with the CF 3 fluorine atoms. The coordinated triglyme signals are observed at  61.21 (s, OCH 3 , a),  70.11 (s, OCH 2 , b),  70.58 (s, OCH 2 , c) and  71.42 (s, OCH 2 , d).

Figure 2 .

 2 Figure 2. a) Ellipsoid representation of the dimer found in the solid state structure of AgP (hydrogen atoms have been omitted for sake of clarity, for the

Figure 3 .

 3 Figure 3. View of the [Ag(triglyme)2] + cation and [Ag(hfac)2] -anion in AgT (ellipsoids are drawn at 30% probability). For the sake of clarity, just one model of all thehfac fluorine atoms (which were disordered and refined isotropically) is shown. For the same reason the hydrogen atoms are no t shown in the image. 3.60(1)Å) and the angle between the C T -C T line and the mean phen plane is 71.2(2)°. Centroid of the hfa ring as defined by O(1)C(2)C(1)C(3)O(2), centroid of the central ring of phen as defined by C(9)C(10)C(11)C(12)C(16)C(17)). Finally, the dimers stack along the a axis in a head-to-tail arrangement (Fig.2 b), which brings the carbon atom C(10) of a symmetry related (-x, -y, 2-z) phen molecule 3.350(8) Å apart from the silver ion, with a distance less than the sum of the Van der Waals radii (3.81 Å)41

  planes nearly perpendicular to each other (the angle between the two mean planes defined by O(1), O(2), C(1), C(2), C(3) and O(3), O(4), C(6), C(7), C(8) is 84.8(2)°). Finally, no relevant intermolecular interactions are present in the crystal packing of AgT. The thermal characteristics of the present adducts were evaluated by Thermogravimetric Analysis (TGA). The TGA profile of AgP (Fig. 4 a) presents a two-step weight loss (as clearly shown in the derivative curve), in the range 190-230 °C and 230-280 °C, respectively, with a residue of 37% up to 400 °C. This value does not match the stoichiometric Ag weight % in the precursor, namely, 21.8%.

Figure 4 .

 4 Figure 4. Thermogravimetric Analysis (TGA) and its derivative curves (DTG) of precursors, between 30 °C and 400 °C, of a) AgP and b) AgT.

Figure 5 .

 5 Figure 5. Scanning Electron Microscopy images of silver coatings deposited at 320 °C by AA-CVD using: AgP precursor with deposition duration a) 0.5 h (P1); b) 1.5 hrs (P2); and AgT precursor with deposition duration c) 0.5 h (T1); d) 1.5 hrs (T2). Scale bars in figures are 500 nm.

Figure 6 .

 6 Figure 6. Size distribution of Ag nanoparticles deposited from: P1 (a) and T1 (b).

Figure 7 .Figure 8 .

 78 Figure 7. Transmission Electron Microscopy images and electron diffractions for Ag particles deposited using AgP (P2), a), c), e) and AgT (T2), b), d), f).By analysing the different particle diameters, a narrower size distribution of Ag grains from P1 is revealed from comparing Fig.6 a) and b). The size of grains from P1 is mainly located around 19 nm, while particles obtained from T1 range broadly distributed from 10 nm to 22 nm. Such a different size distribution can be related to the thin carbon layer. As shown in Fig.7 a) and b), the coatings obtained with both precursors present well crystallized Ag nanoparticles, but from the sample deposited with AgP (P2), a thin shell surrounding the Ag nanoparticles was observed. Also, as already hinted by TEM images (Fig.7 c) and d)), Ag nanoparticles obtained from AgP

Table 1 .

 1 Reported silver metal organic precursors used in metalorganic CVD.

	Metalorganic Precursor	CVD variant	Solvents	Deposition ture / °C Tempera-	Ref
	(hfac)Ag(BTMSA)	HW-CVD	none	~ 250	9
	(hfac)Ag(SEt2)	AA-CVD	Toluene	n.a.	10
	[Ag(O2CC3F7)(PPh3)2] AA-CVD	THF	310	11
	Ag(fod)(PEt3 )	PI-CVD	None precursor) (Liquid	220	12
	(hfac)Ag(VTES)	CW-CVD	None precursor) (Liquid	220	13
	(tfac)AgP(OEt)3	CW-CVD	None precursor) (Liquid	180	14
	(C2F5COO)Ag(PMe3)	HW-CVD	n.a.	200	15
	Ag(hfac)(PPh3)	AA-CVD	THF	310	16
	Ag(O2CtBu)(PMe3)	HW-CVD	n.a.	180	17
	Silver carboxylates	PI-CVD	Mesytilene 300	18
	Notes: H-hfac: 1,1,1,5,5,5-hexafluoro-2,4-pentanedione; H-tfac: 1,1,1-trifluoro-
	2,4-pentanedione; Hfod: 6,6,7,7,8,8,8-heptafluoro-2,2-dimethyl-3,5-octanedione
	BTMSA: bis(trimethylsiyl)acetylene; SEt2: Ethyl sulphide; VTES: vinyltriethylsilane;
	PEt3: triethylphosphine; P(OEt)3: triethylphosphite; PMe3: Trimethylphosphine;
	PPh3: Triphenylphosphine; O2CtBu: 2,2,6,6-tetramethyl-3,5-heptanedionate

of AgNO 3 , the utilization of such precursor may be limited to some type of CVD reactors. In addition, the development of new organometallic Ag precursors is also interesting, for instance for application in ALD, where suitable Ag precursors are still needed, especially for thermal ALD

19

Synthesis of [Ag(triglyme) 2 ] + [Ag(hfac) 2 ]

	crystals were obtained after purification with n-pentane. The
	reaction yield was 82%. The melting point of the crude product
	was 62-65 °C (760 Torr). Elemental analysis for AgT
	). Elemental analysis for AgP
	(C 17 H 9 AgF 6 N 2 O 2 ): Calc: C, 41.23; H, 1,83; N, 5,66. Found: C,
	41,35; H, 1,71; N, 5,78.
	-(AgT): The synthesis
	was conducted following a similar procedure to that used for
	AgP from Ag 2 O (2.20 g, 9.49 mmol), triglyme (2.528 g, 14.14
	mmol) and H-hfac (2.94 g, 14.14 mmol). Pure light yellow

Table 2 .

 2 Single crystal crystallographic data and refinement parameters for AgP and

	AgT.
	(C 26 H 38 Ag 2 F 12 O 12 ): Calc: C, 31.66; H, 3.88. Found: C, 31,52; H,
	3.76.

XRD analyses and structure elucidation: The molecular and crystal structures of AgP and AgT were

	studied by means of
	single crystal X-ray diffraction. Measurements were carried out
	with an Oxford Diffraction Excalibur diffractometer using the
	Cu-Kα radiation (λ = 1.54184 Å) for AgP and Mo-Kα radiation (λ
	= 0.71073 Å) for AgT. Data collection and data reductions were
	performed with the program CrysAlis Pro. Finally, absorption
	correction was performed with the program ABSPACK.

Table 2

 2

		AgP	AgT
	Chemical formula	[Ag(hfa)(phen)]	[Ag(triglyme)2] + [Ag(hfa)2]
	Empirical formula	C17H9AgF6N2O2	C26H38Ag2F12O12
	Formula weight	495.13	986.30
	Temperature (K)	100	250
	Wavelength (Å)	1.54184	0.71073
	Crystal system,	Triclinic, P-1	Triclinic, P-1
	space group
	Unit cell	a = 7.4309(3) b =	a =11.4965(6) b = 12.9561(7)
	dimensions (Å, °)	9.2288(4)	c = 13.0344(9)
		c = 13.1426(6)	α = 96.821(5)
		α = 102.579(4)	 = 91.795(5)
		 = 105.172(4)	γ = 92.350(4)
		γ = 103.608(4)
	Volume (Å 3 )	807.10(6)	1924.8(2)
	Z, Dc (mg/cm 3 )	2, 2.037	2, 1.702
	 (mm -1 )	10.823	1.127
	F(000)	484	984
	Crystal size (mm)	0.30x0.27x0.20	0.28x0.25x0.19
	 range (°)	5.165 to 72.336	4.111 to 29.400
	Reflections	8748 / 3045	13020 / 7438
	collected / unique
	Data / parameters	3045 / 295	7438 / 519
	Goodness-of-fit on
	F 2

. Single crystals of AgT were twinned. Two domains were taken into account during

Table 3 .

 3 Deposition parameters of Ag coatings in Aerosol CVD

	Sample	Precursor used	Deposition Duration
	names		(h)
	P1	[Ag(hfac)(phen)]	0.5
	P2	(AgP)	1.5

Table 4 .

 4 1 H and 13 C NMR of AgP and AgT.

		1H NMR		13 C NMR
	complex	-CO-CH-CO	Phen*	-CO-CH-CO	-CO-CH-CO	-CF 3	Phen*
	AgP	5.79 (s, 1H)	a 9.04 (s, 2H),	86.38 (s)	176.91 (q, 2 J=32Hz)	117.63(q, 1 J=288 Hz)	a 150.99, b, 124.40,
			b 7.79 (m,				c 137.81,
			2H),				d 128.94,
			c 8.40				e 126.89,
			(dd, 2H)				f 142.13
			e 7.89
			(d, 2H)
		1H NMR		13 C NMR

Table 5 .

 5 Selected bond distances (Å) and bond angles (°) for AgP and AgT.

J. Name., 2013, 00, 1-3 | 3

J. Name., 2013, 00, 1-3 | 7

Acknowledgements

HL acknowledges the Ministère de l'Education nationale, de l'Enseignement supérieur et de la Recherche in France for a Ph.D. grant. DMR acknowledges funding through the Marie


Matias Martinez
email: matias.sebastian.martinez@usi.ch

Thomas Durieux
email: thomas.durieux@inria.fr

Romain Sommerard
email: romain.sommerard@etudiant.univ-lille1.fr

Jifeng Xuan
email: jxuan@whu.edu.cn

Martin Monperrus
email: martin.monperrus@univ-lille1.fr

Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset

Keywords: software repair, bugs, defects, patches, fixes

Defects4J is a large, peer-reviewed, structured dataset of real-world Java bugs. Each bug in Defects4J comes with a test suite and at least one failing test case that triggers the bug. In this paper, we report on an experiment to explore the effectiveness of automatic test-suite based repair on Defects4J. The result of our experiment shows that the considered state-of-the-art repair methods can generate patches for 47 out of 224 bugs. However, those patches are only test-suite adequate, which means that they pass the test suite and may potentially be incorrect beyond the test-suite satisfaction correctness criterion. We have manually analyzed 84 different patches to assess their real correctness. In total, 9 real Java bugs can be correctly repaired with test-suite based repair. This analysis shows that test-suite based repair suffers from under-specified bugs, for which trivial or incorrect patches still pass the test suite. With respect to practical applicability, it takes on average 14.8 minutes to find a patch. The experiment was done on a scientific grid, totaling 17.6 days of computation time. All the repair systems and experimental results are publicly available on Github in order to facilitate future research on automatic repair.

Introduction

Automatic software repair is the process of automatically fixing bugs. Test-suite based repair, notably introduced by GenProg (Le Goues et al (2012b)), consists in synthesizing a patch that passes a given test suite, which initially has at least one failing test case. In this paper, we say that this patch is "test-suite adequate". In this recent research field, few empirical evaluations have been made to evaluate the practical ability of current techniques to repair real bugs.

For bugs in the Java programming language, one of the largest evaluations is by [START_REF] Kim | Automatic patch generation learned from human-written patches[END_REF], but as discussed in [START_REF] Monperrus | A critical review of automatic patch generation learned from humanwritten patches: essay on the problem statement and the evaluation of automatic software repair[END_REF], this evaluation suffers from a number of biases and cannot be considered as a definitive answer to the repairability of Java bugs. For bugs in the C programming language, according to bibliometrics, the most visible one is by Le Goues et al (2012a). They reported on an experiment where they ran the GenProg repair system on 105 bugs in C code. However, [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF] have shown at ISSTA 2015 that this evaluation suffers from a number of important issues, and call for more research on systematic evaluations of test-suite based repair.

Our motivation is to conduct a novel empirical evaluation in the realm of Java bugs, in order to better understand the fundamental strengths and weaknesses of current repair algorithms.

In this paper, we present the results of an experiment consisting of running repair systems for Java on the bugs of Defects4J. Defects4J is a dataset (Just et al (2014a)) by the University of Washington consisting of 357 real-world Java bugs. It has been peer-reviewed, is publicly available, and is structured in a way that eases systematic experiments. Each bug in Defects4J comes with a test suite including both passing and failing test cases.

For selecting repair systems, our inclusion criteria is that 1) it is publicly available and 2) it runs on modern Java versions and large software applications. This results in jGenProg, an implementation of GenProg (Le Goues et al (2012b) for Java; jKali, an implementation of Kali [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]) for Java; and Nopol [START_REF] Demarco | Automatic repair of buggy if conditions and missing preconditions with smt[END_REF]; [START_REF] Xuan | Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs[END_REF]).

Our experiment aims to answer to the following Research Questions (RQs): RQ1. Can the considered repair systems synthesize patches for the bugs of the Defects4J dataset? Beyond repairing toy programs or seeded bugs, for this research field to have an impact, one needs to know whether the current repair algorithms and their implementations work on real bugs of large scale applications.

RQ2. In test-suite based repair, are the generated patches correct, beyond passing the test suite? By "correct", we mean that the patch is meaningful, really repairing the bug, and is not a partial solution that only works for the input data encoded in the test cases. Indeed, a key concern behind test-suite based repair is whether test suites are good enough to drive the generation of correct patches, where correct means acceptable. Since the inception of the field, this question has been raised many times and is still a hot question: [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]'s recent results show that most of GenProg's patches on the now classical GenProg benchmark of 105 bugs are incorrect. We will answer RQ2 with a manual analysis of patches synthesized for Defects4J.

RQ3. Which bugs in Defects4j are under-specified? A bug is said to be underspecified if there exists a trivial patch that simply removes functionality. For those bugs, current repair approaches fail to synthesize a correct patch due to the lack of test cases. Those bugs are the most challenging bugs: to automatically repair them, one needs to reason on the expected functionality below what is encoded in the test suite, to take into account a source of information other than the test suite execution. One outcome of our experiment is to identify those challenging bugs.

RQ4. How long is the execution of each repair system? The answer to this question also contributes to assess the practical applicability of automatic repair in the field.

Our experiment considers 224 bugs that are spread over 231K lines of code and 12K test cases in total. We ran the experiment for over 17.6 days of computational time on Grid'5000 [START_REF] Bolze | Grid'5000: a large scale and highly reconfigurable experimental grid testbed[END_REF]), a large-scale grid for scientific experiments.

Our contributions are as follows:

-Answer to RQ1. The Defects4J dataset contains bugs that can be automatically repaired with at least one of the considered systems. jGenProg, jKali, and Nopol together synthesize test-suite adequate patches for 47 out of 224 bugs with 84 different patches. Some bugs are repaired by all three considered repair approaches (12/47). These results validate that automatic repair works on real bugs in large Java programs1 . They can be viewed as a baseline for future usage of Defects4J in automatic repair research. -Answer to RQ2. Our manual analysis of all 84 generated patches shows that 11/84 are correct, 61/84 are incorrect, and 12/84 require a domain expertise, which we do not have. The incorrect patches tend to overfit the test cases. This is a novel piece of empirical evidence on real bugs that either the current test suites are too weak or the current automatic repair techniques are too dumb. These results strengthen the findings in [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF], on another benchmark, on another programming language. They show that the overfitting problem uncovered on the Genprog's benchmark by [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF] is not specific to the benchmark and its weakness, but is more fundamental. -Answer to RQ3. Defects4J contains very weakly specified bugs. Correctly repairing those bugs can be considered as the next milestone for the field. This result calls for radically new automatic repair approach that reasons beyond the test suite execution, using other sources of information, and for test suite generation techniques tailored for repair. -Answers to RQ4. The process of searching for a patch is a matter of minutes for a single bug. This is an encouraging piece of evidence that this research will have an impact for practitioners.

For the sake of open science and reproducible research, our code and experimental data are publicly available on Github (ExperimentalData (2016); AstorCode (2016); NopolCode (2016)).

The remainder of this paper is organized as follows. Section 2 provides the background of test-suite based repair and the dataset. Section 3 presents our experimental protocol. Section 4 details answers to our research questions. Section 5 studies three generated patches in details. Section 6 discusses our results and Section 7 presents the related work. Section 8 concludes this paper and proposes future work.

Background

In this paper, we consider one kind of automatic repair called test-suite based repair. We now give the corresponding background and present the dataset that is used in our experiment.

Buggy program

Fault localization

Patch validation

Final patch

Statement ranking

1 st Statement 2 nd Statement k th Statement ...

Yes

Next statement

One statement

No

No patch

No statement left jGenProg jKali Nopol Fig. 1 Overview of test-suite based repair. Automatic repair takes a buggy program and its test suite as input; the output is the patch that passes the whole test suite if any.

Overview of Automatic Repair Techniques

We briefly give an overview of automatic repair. For a survey on the field, we refer the reader to [START_REF] Monperrus | Automatic software repair: a bibliography[END_REF]. There are two families of repair techniques: offline repair consists of generating source code patches, online repair, aka runtime repair, consists of modifying the system state at runtime to overcome failures. The latter is the direct descendant of classical fault tolerance. Automatic repair techniques use a variety of oracles to encode the expected behavior of the program under repair: the notable oracles are test-suites, pre-and post-conditions, formal behavioral models and runtime assertions. In the experiment presented in this paper, the considered benchmark makes us considering offline patch generation using test suites as oracles.

Test-Suite Based Repair

Test-suite based repair generates a patch according to failing test cases. Different kinds of techniques can be used, such as genetic programming search in GenProg (Le Goues et al (2012b)) and SMT based program synthesis in SemFix [START_REF] Nguyen | Semfix: Program repair via semantic analysis[END_REF]). Often, before patch generation, a fault localization method is applied to rank the statements according to their suspiciousness. The intuition is that the patch generation technique is more likely to be successful on suspicious statements.

Fig. 1 presents a general overview of test-suite based repair approaches. In a repair approach, the input is a buggy program as well as its test suite; the output is a patch that makes the test suite pass, if any. To generate a patch for the buggy program, the executed statements are ranked to identify the most suspicious statements. Fault localization is a family of techniques for ranking potential buggy statements [START_REF] Jones | Visualization of test information to assist fault localization[END_REF]; [START_REF] Abreu | On the accuracy of spectrum-based fault localization[END_REF]; [START_REF] Xuan | Test case purification for improving fault localization[END_REF]). Based on the statement ranking, patch generation tries to modify a suspicious statement. For instance, GenProg (Le Goues et al (2012b)) adds, removes, and replaces Abstract Syntax Tree (AST) nodes. Once a patch is found, the whole test suite is executed to validate the patch; [START_REF] Noor | Test case analytics: Mining test case traces to improve risk-driven testing[END_REF]; Just et al (2014b)). To our knowledge, Defects4J is the largest open database of well-organized real-world Java bugs. In our work, we use four out of five projects, i.e., Commons Lang,2 JFreeChart,3 Commons Math,4 and Joda-Time. 5 We do not use the Closure Compiler project6 because the test cases in Closure Compiler are organized in a non-conventional way, using scripts rather than standard JUnit test cases. This prevents us from running with our platform and is left for future work. Table 1 presents the main descriptive statistics of bugs in Defects4J.

The advantages of using Defects4J for a automatic repair experiment are: (realism) it contains real bugs (as opposed to seeded bugs as in [START_REF] Nguyen | Semfix: Program repair via semantic analysis[END_REF]; [START_REF] Kong | Experience report: How do techniques, programs, and tests impact automated program repair?[END_REF]); (scale) bugs are in large software (as opposed to bugs in student programs as in [START_REF] Smith | Is the cure worse than the disease? overfitting in automated program repair[END_REF]); (novelty) nobody has ever studied Defects4J for repair.

Experimental Protocol

We present an experimental protocol to assess the effectiveness of different automatic repair approaches on the real-world bugs of Defects4J. The protocol supports the analysis of several dimensions of automatic repair: test-suite adequacy, patch correctness, under-specified bugs, performance. We first list the Research Questions (RQs) of our work; then we describe the experiment design; finally, we present the implementation details.

Research Questions

RQ1. Test-suite Adequate Repair

For how many bugs can each system synthesize a patch that is test-suite adequate, i.e., it fulfills the test suite? This is the basic evaluation criterion of automatic test-suite based repair research. In test-suite based repair, a bug is said to be automatically patchable if the synthesized patch makes the whole test suite pass. Test-suite adequacy does not convey a notion a correctness beyond the test suite. To answer this question, we run each repair approach on each bug and count the number of bugs that are patched according the test suite.

RQ2. Patch Correctness

Which generated patches are semantically correct beyond passing the test suite?

A patch that passes the whole test suite may not be exactly the same as the patch written by developers. It may be syntactically different yet correct. As shown by [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF] it may also be incorrect. To answer this question, we manually examine all synthesized patches to identify the correctness.

RQ3. Under-Specified Bugs

Which bugs in Defects4j are not sufficiently specified by the test suite?

In test-suite based repair, the quality of a synthesized patch is highly dependent on the quality of the test suite. In this paper, we define an "under-specified bug" as a bug for which the test cases that specify the expected behavior have a low coverage and weak assertions. Among the incorrect patches identified in RQ2, we identify the bugs for which the primary reason for incorrectness is the test suite itself (and not the repair technique). To find such under-specified bugs, we use two pieces of evidence. First, we closely look at the results of jKali. Since this repair system only removes code or skips code execution, if it finds a patch, it hints that a functionality is not specified at all. Second, we complement this by a manual analysis of generated patches.

RQ4. Performance (Execution Time)

How long is the execution time of each repair system?

It is time-consuming to manually repair a bug. Test-suite based repair automates the process of patch generation. To assess to which extent automatic repair is usable in practice, we evaluate the execution time of each repair approach.

Experiment Design

According to the inclusion criteria presented in Section 3.2.1, we select three repair systems (Section 3.2.2) on the Defects4J dataset (Section 2.3). Since the experiment requires a large amount of computation, we run it on a grid (Section 3.2.3). We then manually analyze all the synthesized patches (Section 3.2.4).

Inclusion Criteria

For selecting repair systems, we have three inclusion criteria.

Criterion #1 In this experiment, we consider a dataset of bugs in software written in the Java programming language. We consider repair systems that are able to handle this programming language. This rules out many systems that repair C code, such as GenProg (Le Goues et al (2012b)) and Semfix [START_REF] Nguyen | Semfix: Program repair via semantic analysis[END_REF]).

Criterion #2 Defects4J contains bugs written in modern Java versions (Java 5, 7 and 8). We consider repair systems that support those versions. This rules out Arcuri's pioneering prototype Jaff [START_REF] Arcuri | A novel co-evolutionary approach to automatic software bug fixing[END_REF]).

Criterion #3 The third inclusion criterion is that the repair system is available, whether publicly or on demand. This rules out for instance PAR [START_REF] Kim | Automatic patch generation learned from human-written patches[END_REF]).

Repair Systems Under Study

As of submission time, there are three repair systems that meet all those criteria: jGen-Prog (AstorCode (2016); [START_REF] Martinez | Astor: A program repair library for java (demo)[END_REF]), jKali (AstorCode ( 2016)) and Nopol (NopolCode (2016)).

jGenProg jGenProg is an implementation of GenProg for Java. It implements the Gen-Prog algorithm (Le Goues et al (2012b)) as follows. It randomly deletes, adds, and replaces AST nodes in the program. The modification point is steered by spectrum based fault localization. Pieces of code that are inserted through addition or replacement always come from the same program, based on the "redundancy hypothesis" [START_REF] Martinez | Do the fix ingredients already exist? an empirical inquiry into the redundancy assumptions of program repair approaches[END_REF]). There is a threat that this Java implementation does not reflect the actual performance of the original GenProg system for C, it is discussed in Section 6.1.2. We are the authors of jGenProg, and we make it publicly available at Github AstorCode (2016); [START_REF] Martinez | Astor: A program repair library for java (demo)[END_REF]. Our rationale for implementing jGenProg is that GenProg (which works for C code) is arguably the baseline of this research area, being the most known and visible patch generation system. jGenProg has 9.5K lines of Java code.

jKali jKali is an implementation of Kali for Java. Kali [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]) performs patch synthesis by only removing or skipping code. Kali is clearly not a "program repair" technique. However, it is a perfectly appropriate technique to identify weak test suites and under-specified bugs. We are the authors of jKali and we make it publicly available (AstorCode ( 2016)). Our rationale for implementing Kali is that is an appropriate technique to detect test suite weaknesses, yet the original Kali works for C.

Nopol Nopol is a repair system for Java (DeMarco et al (2014); Xuan et al ( 2016)) which targets a specific fault class: conditional bugs. It repairs programs by either modifying an existing if-condition or adding a precondition (aka. a guard) to any statement or block in the code. The modified or inserted condition is synthesized via input-output based code synthesis with SMT solvers [START_REF] Jha | Oracle-guided component-based program synthesis[END_REF]). Nopol is publicly available (NopolCode ( 2016)) and has 25K lines of Java code.

Open Science We note that we are authors of those three systems. This is both good and bad. This is good because we know that they are of similar quality. This is bad because it shows that as of today, although a number of automatic repair contributions have been made for Java, the current practice is not to share the tools for sake of open and reproducible science.

Large Scale Execution

We assess three repair approaches on 224 bugs. One repair attempt may take hours to be completed. Hence, we need a large amount of computation power. Consequently, we deploy our experiment in Grid'5000, a grid for high performance computing [START_REF] Bolze | Grid'5000: a large scale and highly reconfigurable experimental grid testbed[END_REF]). In our experiment, we manually set the computing nodes in Grid'5000 to the same hardware architecture. This avoids potential biases of the time cost measurement. All the experiments are deployed in the Nancy site of Grid'5000 (located in Nancy, France).

As we shall see later, this experiment is significantly CPU intensive: it takes in total 17.6 days of computation on Grid'5000. This means we had to make careful choices before coming to valid run of the experiment. In particular, we set the timeout to three hours per repair attempt, in order to have an upper bound.

For the same reason, although jGenProg is a randomized algorithm, for all but 5 bugs, we run it only once also to keep an acceptable experiment time, this important threat to validity is discussed in Section 6.1.4. This results in a conservative underestimation of GenProg's effectiveness but remains sound (found patches are actually test-suite adequate patches). Finally, although our and other's experience with repair has shown that multiple patches exist for the same bug, we stop the execution of a repair attempt after finding the first patch.

Manual Analysis

We manually examine the generated patches. For each patch, one of the authors, called thereafter an "analyst", analyzed the patch correctness, readability, and the difficulty of validating the correctness (as defined below). For sake of cross validation, the analyst then presents the results of the analysis to another co-author, called thereafter a "reviewer" in a live session. At the end of the live session, when an agreement has been reached between the analyst and the reviewer, a short explanatory text is written. Those texts are made publicly available for peer review (ExperimentalData ( 2016)). During agreement sessions, analysts and reviewers applied a conservative patch evaluation strategy: when there were a doubt or a disagreement about the correctness of a patch, is was labeled as "unknown".

Correctness Analysis

The correctness of a patch can be correct, incorrect, or unknown. The term "correct" denotes that a patch is exactly the same or semantically equivalent to the patch that is written by developers. We assume that all patches included in the Defects4J dataset are correct. The equivalence is assessed according to the analyst's understanding of the patch. Analyzing one patch requires a period between a couple of minutes and several hours of work, depending on the complexity of the synthesized patch. On the one hand, a patch that is identical to the one written by developers is obviously true; on the other hand, several patches require a domain expertise that none of the authors has.

Note that in the whole history of automatic program repair, this is only the second time that such a large scale analysis of correctness is being performed (after [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]). While other published literature has studied aspects of the readability and usefulness of generated patches (e.g. [START_REF] Fry | A human study of patch maintainability[END_REF]; Tao et al ( 2014)), it was on a smaller scale with no clear focus on correctness.

Readability Analysis

The readability of the patch can be "easy", "medium", or "hard"; and it results from the analyst opinion on the length and complexity of the patch. This subjective evaluation concerns: 1) The number of variables involved in the patch: a patch that refers to one or two variables is much easier to understand than a patch that refers to 5 variables. 2) The number of logical clauses: in a condition, a patch with a single conjunction (hence with two logical clauses) is considered easier a patch that with many sub-conjunctions. 3) The number of method calls: when a patch contains a method call, one must understand the meaning and the goal of this call to understand the patch. Hence, the more method calls, the harder the patch to understand.

Difficulty Analysis

The difficulty analysis relates to the difficulty of understanding the problem domain and assessing the correctness. It indicates the effort that the analyst had to carry out for understanding the essence of the bug, the human patch and the generated patch. The analysts agree on a subjective difficulty on the scale "easy", "medium", "hard", or "expert" as follows. "Easy" means that it is enough to examine the source code of the patch for determining its correctness. "Medium" means that one has to fully understand the test case. "Hard" means that the analyst has to dynamically debug the buggy and/or the patched application. By "expert", we mean a patch for which it was impossible for us to validate due to the required expertise in domain knowledge. This happens when the analyst could not understand the bug, or the developer solution or the synthesized patch.

Empirical Results

We present and discuss our answers to the research questions that guide this work. The total execution of the experiment costs 17.6 days.

Test-suite Adequate Repair

RQ1. For how many bugs can each system synthesize a patch that fulfills the test suite?

The three automatic repair approaches in this experiment are able to together find test-suite adequate patches for 47 bugs of the Defects4J dataset. jGenProg finds a patch for 27 bugs; jKali identifies a patch for 22 bugs; and Nopol synthesizes a condition that makes the test suite passing for 35 bugs. Table 2 shows the bug identifiers, for which at least one patch is found. Each line corresponds to one bug in Defects4J and each column denotes the effectiveness of one repair approach. For instance, jGenProg and jKali are able to synthesize a test-suite adequate patch for Bug M2 from Commons Math. As shown in Table 2, some bugs such as T11 can be patched by all systems, others by only a single one. For instance, only Nopol synthesize a test-suite adequate patch for bug L39 and only jGenProg for bug M5.

Moreover, Table 2 shows that in project Commons Lang only Nopol generates testsuite adequate patches while jGenProg and jKali fail to synthesize a single patch. A possible reason is that the program of Commons Lang is more complex than that of Commons Math; both jGenProg and jKali cannot handle such a complex search space. Fig. 2 shows the intersections among the three repair approaches as a Venn diagram, where each number is a number of patches for which one system is able to generate a test-suite adequate patch. Nopol can synthesize a patch for 18 bugs that neither jGenProg nor jKali could handle. All the bugs handled by jKali can also be handled by jGenProg and Nopol. For 12 bugs, all three repair systems can generate a patch to pass the test suite. The 2 discarded bugs will be discussed in Section 6.2.

To our knowledge, those results are the very first on automatic repair with the Defects4J benchmark. Recall that they are done with an open-science ethics, all the implementations, experimental code, and results are available on Github (Experimen-talData ( 2016)). Future research in automatic repair may try to synthesize more testsuite adequate patches than our work. Our experimental framework can be used to facilitate future comparisons by other researchers.

Among the incorrect patches identified in RQ2, we identify the bugs for which the primary reason for incorrectness is the test suite itself (and not the repair technique). Answer to RQ1. In Defects4J, at least one of the automatic repair systems under consideration can generate a test-suite adequate patch for 47 out of 224 bugs. Nopol can is the most effective (35 bugs). If jKali finds a test-suite adequate patch, so can jGenProg or Nopol. This experiment sets a record to be beaten by future yet-to-beinvented repair systems.

Patch Correctness

RQ2. Which patches are semantically correct beyond passing the test suite? We manually evaluate the correctness of generated patches by the three repair approaches under study. A generated patch is correct if this patch is the same as the manually-written patch by developers or the patch is semantically equivalent to the manual patch. A generated patch is incorrect if it actually does not repair the bug (beyond test-suite adequacy, beyond making the failing test case pass -a kind of incomplete bug oracle) or if it breaks an expected behavior (beyond keeping the rest of the test suite passing).

Recall the history of automatic repair research. It has been hypothesized that a major pitfall of test-suite based repair is that a test suite cannot completely express the program specifications, so it is hazardous to drive the synthesis of a correct patch with a test suite [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]; [START_REF] Fry | A human study of patch maintainability[END_REF]). Previous works have studied the maintainability of automatic generated patches [START_REF] Fry | A human study of patch maintainability[END_REF]) or their aids for debugging task [START_REF] Tao | Automatically generated patches as debugging aids: a human study[END_REF]). However, only recent work by [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF] has invested resources to manually analyze the correctness previously-generated patches by test-suite based repair. They found that the vast majority of patches by GenProg in the GenProg benchmark of C bugs are incorrect.

To answer the question of patch correctness, we have manually analyzed all the patches generated by Nopol, jGenProg and jKali in our experiment, 84 patches in total. This represents more than ten full days of work. To our knowledge, only [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF] have performed a similar manual assessment of patches synthesized with automatic repair.

Table 3 shows the results of this manual analysis. The "bug id" column refers to the Defects4J identifier, while "Patch id" is an univocal identifier of each patch, for easily identifying the patch on our empirical result page (ExperimentalData (2016)). The three main columns give the correctness, readability and difficulty as explained in Section 3.2.4.

The results in Table 3 may be fallible due to the subjective nature of the assessment. For mitigating this risk, all patches as well as detailed case studies are publicly available on Github (ExperimentalData ( 2016)).

In total, we have analyzed 84 patches. Among these patches, 27, 22, and 35 patches are synthesized by jGenProg, jKali, and Nopol, respectively.

As shown in Table 3, 11 out of 84 analyzed patches are correct and 61 are incorrect. Meanwhile, for the other 12 patches, it is not possible to clearly validate the correctness, due to the lack of domain expertise (labeled as unknown). Figure 3 graphically summarizes this analysis. This figure clearly convey the fact that most test-suite adequate patches are actually incorrect because of overfitting. Section 5 will present three case studies of generated patches via manual analysis.

Among the 11 correct patches, jGenProg, jKali, and Nopol contribute to 5, 1, and 5 patches, respectively. All the correct patches by jGenProg and jKali come from Commons Math; 3 correct patches by Nopol come from Commons lang, one comes from JFreeChart and the other from Commons Math. After the controversy about the effectiveness of GenProg [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]), it is notable to see that there are bugs for which only jGenProg works for real.

Among 84 analyzed patches, 61 patches are identified as easy to read and understand. For the difficulty of patch validation, 21 patches are labeled as hard or expert. This result shows that it is hard and time consuming to conduct a manual assessment of patches.

For the incorrect patches, the main reasons are as follows. First, all three approaches are able to remove some code (pure removal for jKali, replacement for jGen-Prog, precondition addition for Nopol). The corresponding patches simply exploit some under-specification and remove the faulty but otherwise not tested behavior. When the expected behavior seems to be well-specified (according to our understanding of the domain), the incorrect patches tend to overfit the test data. For instance, if a failing test case handles a 2 × 2 matrix, the patch may use such test data to incorrectly force the patch to be suitable only for matrices of size of 2 × 2.

This goes along the line of [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]'s results and Smith et al ( 2015)' findings. Compared to [START_REF] Smith | Is the cure worse than the disease? overfitting in automated program repair[END_REF], we study real bugs in large programs and not small ones in student programs. Compared to [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF], we study a different benchmark in a different programming language. To this extent, the key novelty of our experimental results is that: 1) overfitting also happens for Java software tested with a state-of-the-art testing infrastructure (as opposed to old-school C programs in [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]); 1) overfitting also happens with Nopol, a synthesis based technique (and not only for Genprog and similar techniques). Our results clearly identify overfitting as the next big challenge of test-suite based automatic repair.

Answer to RQ2. Based on the manual assessment of patch correctness, we find out that only 11 out of 84 generated patches are semantically correct (beyond passing the provided test suite). The reason is that the current test suite based repair algorithms tend to overfit the test case data or to exploit the holes left by insufficiently specified functionality.

Under-specified bugs

RQ3. Which bugs in Defects4j are not sufficiently specified by the test suite?

As shown in Section 4.1, the repair system jKali can find test-suite adequate patches for 22 bugs. Among these generated patches, from our manual evaluation, we find out that 18 patches are incorrect (other 3 patches are unknown). In each of those generated patches by jKali, one statement is removed or skipped to eliminate the failing program behavior, instead of making it correct. This kind of patches shows that the corresponding test suite is too weak with respect to the buggy functionality.

The assertions that specify the expected behavior of the removed statement and the surrounding code are inexistent or too weak.

One exception among 22 patches by jKali is the patch of Bug M50. As shown in Section 4.2, the patch of Bug M50 is correct. That is, the statement removal is the correct patch. Another special case is Bug C5 which is patched by jKali (incorrect) and by Nopol (correct). The latter approach produces a patch similar to that done by the developer. A patch (written by developers or automatically generated) that is test-suite adequate for an under-specified bug could introduce new bugs (studied previously by Gu et al ( 2010)) or it could not be completely correct due to a weak test suite used as the bug oracle [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]). Table 4 summarizes this finding and list the under-specified bugs.

This result is important for future research on automatic repair with Defects4J. First, any repair system that claims to correctly repair one of those bugs should be validated with a detailed manual analysis of patch correctness, to check whether the patch is not a variation on the trivial removal solution. Second, those bugs can be considered as the most challenging ones of Defects4J. To repair them, a repair system must somehow reason about the expected functionality beyond what is encoded in the test suite. This is what was actually been done by the human developer. A repair system that is able to produce a correct patch for those bugs would be a great advance for the field. Answer to RQ3. Among the incorrect patches identified in RQ2, there are 21 bugs for which the primary reason for incorrectness is the test suite itself (and not the repair technique), they are under-specified bugs. For them, the test suite does not accurately specify the expected behavior and can be trivially repaired by removing code. To us, they are the most challenging bugs: to automatically repair them, one needs to reason on the expected functionality below what is encoded in the test suite for instance by taking into account a source of information other than test suite execution. Repairing those bugs can be seen as the next grand challenge for automatic repair.

Performance

RQ4. How long is the execution time for each repair approach on one bug?

For real applicability in industry, automatic repair approaches must execute fast enough. By "fast enough", we mean an acceptable time period, which depends on the usage scenario of automatic repair and on the hardware. For instance, if automatic repair is meant to be done in the IDE, repair time should last at most some minutes on a standard desktop machine. On the contrary, if automatic repair is meant to be done on a continuous integration server, it is acceptable to last hours on a more powerful server hardware configuration. The experiments in this paper are run on a grid where most of nodes have comparable characteristics. Typically, we use machines with Intel Xeon X3440 Quad-core processor and 15GB RAM. Table 5 shows the time cost of patch generation in hours for bugs without timeout. As shown in Table 5, the median time for one bug by jGenProg is around one hour. The fastest repair attempt yields a patch in 31 seconds (for Nopol). The median time to synthesize a patch is 6.7 minutes. This means that the execution time of automatic repair approaches is comparable to the time of manual repair by developers. It may be even faster, but we don't know the actual repair time by real developers for the bug of the dataset.

When a test-adequate patch can be synthesized, it is found within minutes. This means that most of the time of the 17.6 days of computation for the experiment is spent on unfixed bugs, which reach the timeout. For jGenProg, it is always the case, because the search space is extremely large. For jKali, we often completely explore the search space, and we only reach the timeout in 20 cases. For Nopol, the timeout is reached in 26 cases, either due to the search space of covered statements or the SMT synthesis that becomes slow in certain cases. One question is whether a larger timeout would improve the effectiveness. According to this experiment, the answer is no. The repairability is quite binary: either a patch is found fast, or the patch cannot be found at all. This preliminary observation calls for future research. Answer to RQ4. Both the median value and the average value of repair execution time are about one hour on machines that represent server machines used nowadays for continuous integration. Performance is not a problem with respect to practical usage of automatic repair.

Other Findings in Defects4J

Our manual analysis of results enables us to uncover two problems in Defects4J. First, we found that bug #8 from project JFreeChart (C8) is flaky, which depends on the machine configuration. Second, bug #99 from Commons Math (M99) is identical to bug M97. Both issues were reported to the authors of Defects4J and will be solved in future releases of Defects4J.

Case Studies

In this section, we present three case studies of generated patches by jGenProg, jKali, and Nopol, respectively. These case studies are pieces of evidence that: State-of-the-art automatic repair is able to find correct patches (Sections 5.1 and 5.3), but also fails with incorrect patches (Section 5.2). It is possible to automatically generate the same patch as the one written by the developer (Section 5.1).

1 double solve(UnivariateRealFunction f, 2 double min, double max, double initial) 3

throws MaxIterationsExceededException, 4

FunctionEvaluationException { 5 // PATCH: return solve(f, min, max); 6 return solve(min, max); 7 } Fig. 4 Code snippet of Bug M70. The manually-written patch and the patch by jGenProg are the same, which is shown in the PATCH comment at Line 5, which adds a parameter to the method call.

Case Study of M70, Bug that is Only Repaired by jGenProg

In this section, we study Bug M70, which is repaired by jGenProg, but cannot be repaired by jKali and Nopol.

Bug M70 in Commons Math is about univariate real function analysis. Fig. 4 presents the buggy method of Bug M70. This buggy method contains only one statement, a method call to an overloaded method. In order to perform the correct calculation, the call has to be done with an additional parameter UnivariateRealFunction f (at Line 1) to the method call. Both the manually-written patch and the patch by jGenProg add the parameter f to the method call (at Line 5). This patch generated by jGenProg is considered correct since the it is the same as that by developers.

To repair Bug M70, jGenProg generates a patch by replacing the method call by another one, which is picked elsewhere in the same class. This bug cannot be repaired by either jKali or Nopol. jKali removes and skips statements; Nopol only handles bugs that are related to if conditions. Indeed, the fact that certain bugs are only repaired by one tool confirms that the fault classes addressed by each approach are not identical. To sum up, Bug M7 shows that the GenProg algorithm, as implemented in jGenProg, is capable of uniquely repairing real Java bugs (only jGenProg succeeds).

Case Study of M8, Bug that is Incorrectly Repaired by jKali and jGenProg

In this section, we present a case study of Bug M8, for which jKali as well as jGenProg find a test-suite adequate patch, but not Nopol. Bug M87 in Commons Math, is about the failure to create an array of a random sample from a discrete distribution. Fig. 5 shows an excerpt of the buggy code and the corresponding manual and synthesized patches (from class DiscreteDistribution<T>). The method sample receives the expected number sampleSize of random values and returns an array of the type

T[].
The bug is due to an exception thrown at Line 11 during the assignment to out[i]. The method Array.newInstance(class, int) requires a class of a data type as the first parameter. The bug occurs when a) the first parameter is of type T1, which is a subclass of T and b) one of the samples is an object which is of type T2, which is a sub-class of T, but not of type T1. Due to the incompatibility of types T1 and T2, an ArrayStoreException is thrown when this object is assigned to the array.

In the manual patch, the developers change the array type in its declaration (from T[] to Object[]) and the way the array is instantiated. The patch generated by jKali simply removes the statement, which assigns sample() to the array. As consequence, method sample never throws an exception but returns an empty array (only containing null values). This patch passes the failing test case and the full test suite as well. The reason of this is that the test case has only one assertion: it asserts that the array size is equal to 1. There is no assertion on the content of the returned array. However, despite passing the test suite, the patch is clearly incorrect. This is an example of a bug that is not well specified by the test suite. For this bug, jGenProg can also generate a patch by replacing the assignment by a side-effect free statement, which is semantically equivalent to removing the code. To sum up, Bug M8 is an archetypal example of under-specified bugs as detected by the jKali system.

Case Study of L55, Bug that is Repaired by Nopol, Equivalent to the Manual Patch

Nopol (DeMarco et al ( 2014)) focuses on condition-related bugs. Nopol collects runtime data to synthesize a condition patch. In this section, we present a case study of Bug L55, which is only repaired by Nopol, but cannot be repaired by jGenProg or jKali.

Bug L55 in Commons Lang relates to a utility class for timing. The bug appears when the user stops a suspended timer: the stop time saved by the suspend action is overwritten by the stop action. Fig. 6 presents the buggy method of Bug L55. In order to solve this problem, the assignment at Line 10 has to be done only if the timer state is running.

As shown in Fig. 6, the manually-written patch by the developer adds a precondition before the assignment at Line 10 and it checks that the current timer state is running (at Line 7). The patch by Nopol is different from the manually-written one. The Nopol patch compares the stop time variable to a integer constant (at Line 9), which is pre-defined in the program class and equals to 1. In fact, when the timer is running, the stop time variable is equals to -1; when it is suspended, the stop time variable contains the stop time in millisecond. Consequently, both preconditions by developers and by Nopol are equivalent and correct. Despite being equivalent, the manual patch remains more understandable. This bug is neither repaired by jGenProg nor jKali. To our knowledge, Nopol is the only approach that contains a strategy of adding preconditions to original statements, which does not exist in jGenProg or jKali. To sum up, Bug L55 shows an example of a repaired bug, 1) that is in a hard-to-repair project (only Nopol succeeds) and 2) whose specification by the test suite is good enough to drive the synthesis of a correct patch. Summary. In this section, we have presented detailed case studies of three patches that are automatically generated for three real-world bugs of Defects4J. Our case studies show that automatic repair approaches are able to repair real bugs. However, different factors, in particular the weakness of some test cases, yield clearly incorrect patches.

Discussion

Threats to Validity

Benchmark

Our results are obtained on four subject programs, from a benchmark in which there was no attempt to provide a representative set of fault classes. Future experiments on other benchmarks are required mitigate this threat to the external validity.

We have taken the bugs of the Defects4J benchmark as they are and we did not modify the test cases. This is very important to decrease the following threats. First, we measure the effectiveness of the repair tool as it would have be at the time when the bug was reported. We do not use other tests of information "from the future". It would introduce a potential experimental bias if we modify the benchmark, which has been set up independently of our experiment.

Implementations of GenProg and Kali

jGenProg and jKali are the re-implementations of the GenProg and Kali algorithms in Java. There exists a threat that the implementations do not produce results that are as good as the original systems would (there is also a risk that the re-implementation produces better results). The authors of jGenProg and jKali carefully and faithfully reimplemented the original systems. Note this threat is absolute. Either one takes the risk or one give up any comparative evaluations between programming languages. To find potential re-implementation issues, the systems are publicly available on Github for peer-review.

Bias of assessing the correctness, readability, and difficulty

In our work, each patch in Table 3 is validated by an analyst, which is one of the authors. An analyst manually identifies the correctness of a patch and labels the related readability and difficulty. Then, he discusses and validates the result of the patch analysis with another author, called a reviewer. Since patch correctness analysis on an unknown code base is a new kind of qualitative analysis, only explored by [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF], we do not know its inherent problems and difficulty. This is why the methodological setup is a pilot one, where no independent validation has been attempted to measure inter-annotator agreement. This results in a threat to the internal validity of the manual assessment.

We share our results online on the experiment Github repository to let readers have a clear idea of the difficulty of our analysis work and the correctness of generated patches (see Section 4.1). For assessing the equivalence, one solution would be to use automatic technique, as done in mutation testing. However, whether the current equivalence detection techniques scale on large real Java code is an open question.

6.1.4 Random nature of jGenProg jGenProg, as the original GenProg implementation, has a random aspect: statements and mutations are randomly chosen during the search. Consequently, it may happen that a new run of jGenProg yields different results. There are two different threats. Our experiment may have over-estimated the repair effectiveness of GenProg (too many patches found) or underestimated the repair effectiveness (too few).

To analyze the overestimation threat, for each of the 27 repaired bug, we have run jGenProg ten times with different seeds. For 18/27 cases, a test-adequate patch was found in all 10 runs, in 6 cases, a test-adequate patch was found in at least 7 runs. In the remaining 3 cases, a patch was found in a minority of runs, and we can consider that our reference run was lucky to find at least one test-adequate patch. Note that this experiment is quite fast because the runs are mostly successful and hence terminate fast, much before the 3-hour time budget limit.

Analyze the underestimation threat is much more challenging, because if a bug is not repaired, it means that the trial lasted at least 3 hours (the time budget used in this experiment). Consequently, if the bug is really unrepairable, trying with n different seeds would require n × 3 hours of computation. For 224 -27 = 197 unrepaired bugs, this sets up a worst case bound of 197 × 10 × 3 = 5910 hours of computation, i.e. 224 days, which is not reasonable, even split into parallel tasks. Consequently, we decided to sample 5 unrepaired bugs and run with 10 different seeds (representing an experimental budget of 5×10×3 = 150 hours). The result was clear cut: no additional patches could be found. This suggests that the "unrepairedness" comes from the essential complexity of the problem and not from the accidental randomness of jGenProg.

Presence of multiple patches

In this experiment, we stop the execution of a repair system once the first patch is found. This is the patch that is manually analyzed. However, as also experienced by others, there are often several if not dozens of different patches per bug. It might happen that a correct patch lies somewhere in this set of generated patches. We did not manually analyze all generated patches because it would require months of manual work. This finding shows that there is a need for research on approaches that order the generated patches so as to reveal the most likely to be correct, this has been preliminarily explored by Long and Rinard (2016b).

Impact of Flaky Tests on Repair

Our experiment uncovered one flaky test in Defects4J (C8). We realized that flaky tests have a direct impact on automatic repair. If the failing test case is flaky, the repair system might conclude that a correct patch has been found while it is actually correct. If one of the passing test cases is flaky, the repair system might conclude that a patch has introduced a regression while it is not the case, this results in an underestimation of the effectiveness of the repair technique.

Reflections on GenProg

The largest evaluations of GenProg are by Le Goues et al (2012a) and [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]. The former reports that 55/105 bugs are repaired (under the definition that the patch passes the test suite), while the latter argued that only 2/105 bugs are correctly repaired (under the definition that the patch passes the test suite and that the patch is correct and acceptable from the viewpoint of the developer). The difference is due to an experimental threat, the presence of under-specified bugs and a potential subjectiveness in the assessment.

In this paper, we find that our re-implementation of GenProg, jGenProg correctly repairs 5/224 (2.2%) bugs. In addition, it uniquely repairs 4 bugs, such as M70 discussed in Section 5.1. We think that the difference in repair rate is probably due to the inclusion criteria of both benchmarks (GenProg and Defects4J). To our opinion, none of them reflect the actual distribution of all bug kinds and difficulty in the field.

However, the key finding is that having correctly and uniquely repaired bugs indicates that the core intuition of GenProg is valid, and that GenProg has to be a component of an integrated repair tool that would mix different repair techniques.

On the Choice of Repair Techniques

Given a new bug, can one choose the most appropriate repair technique? In practice, when a new bug is reported, one does not know in advance the type of its root cause.

The root cause may be an incorrect condition, a missing statement, or something else. It means that given a failure (or a failing test case) there is no reason to choose one or another repair technique (say jGenProg or Nopol). The pragmatic choice is to run all available techniques, whether sequentially or in parallel. However, there may be some information in the failure itself about the type of the root cause. Future results on the automatic identification of the type of root cause given a failure would be useful for repair, in order to guide the choice of repair technique.

7 Related Work

Real-World Datasets of Bugs

The academic community has already developed many methods for finding and analyzing bugs. Researchers employ real-world bug data to evaluate their methods and to analyze their performance in practice. For instance, [START_REF] Do | Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact[END_REF] propose a controlled experimentation platform for testing techniques. Their dataset is included in SIR database, which provides a widely-used testbed in debugging and test suite optimization. [START_REF] Dallmeier | Extraction of Bug Localization Benchmarks from History[END_REF] propose iBugs, a benchmark for bug localization obtained by extracting historical bug data. BugBench by [START_REF] Lu | Mining software repair models for reasoning on the search space of automated program fixing[END_REF] and BegBunch by [START_REF] Cifuentes | Begbunch: Benchmarking for c bug detection tools[END_REF] are two benchmarks that have been built to evaluate bug detection tools. The PROMISE repository [START_REF] Menzies | The promise repository of empirical software engineering data[END_REF]) is a collection of datasets in various fields of software engineering. Le [START_REF] Goues | The manybugs and introclass benchmarks for automated repair of c programs[END_REF] have designed a benchmark of C bugs which is an extension of the GenProg benchmark.

In this experience report, we employ Defects4J by Just et al (2014a) to evaluate software repair. This database includes well-organized programs, bugs, and their test suites. The bug data in Defects4J has been extracted from the recent history of five widely-used Java projects. To us, Defects4J is the best dataset of real world Java bugs, both in terms of size and quality. To our knowledge, our experiment is the first that evaluates automatic repair techniques via Defects4J.

Test-Suite Based Repair Approaches

The idea of applying evolutionary optimization to repair derives from [START_REF] Arcuri | A novel co-evolutionary approach to automatic software bug fixing[END_REF]. Their work applies co-evolutionary computation to automatically generate bug patches. GenProg by Le Goues et al (2012b) applies genetic programming to the AST of a buggy program and generates patches by adding, deleting, or replacing AST nodes. PAR by [START_REF] Kim | Automatic patch generation learned from human-written patches[END_REF] leverages patch patterns learned from human-written patches to find readable patches. RSRepair by [START_REF] Qi | The strength of random search on automated program repair[END_REF] uses random search instead of genetic programming. This work shows that random search is more efficient in finding patches than genetic programming. Their follow-up work [START_REF] Qi | Efficient automated program repair through fault-recorded testing prioritization[END_REF]) uses test case prioritization to reduce the cost of patch generation. [START_REF] Debroy | Using mutation to automatically suggest fixes for faulty programs[END_REF] propose a mutation-based repair method inspired from mutation testing. This work combines fault localization with program mutation to exhaustively explore a space of possible patches. Kali by [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF] has recently been proposed to examine the repair power of simple actions, such as statement removal.

SemFix by [START_REF] Nguyen | Semfix: Program repair via semantic analysis[END_REF] is a notable constraint based repair approach. This approach provides patches for assignments and conditions by combining symbolic execution and code synthesis. Nopol by DeMarco et al ( 2014) is also a constraint based method, which focuses on repairing bugs in if conditions and missing preconditions. DirectFix by [START_REF] Mechtaev | Directfix: Looking for simple program repairs[END_REF] achieves the simplicity of patch generation with a Maximum Satisfiability (MaxSAT) solver to find the most concise patches. Relifix [START_REF] Tan | Relifix: Automated repair of software regressions[END_REF]) focuses on regression bugs. SPR [START_REF] Long | Staged program repair with condition synthesis[END_REF]) defines a set of staged repair operators so as to early discard many candidate repairs that cannot pass the supplied test suite and eventually to exhaustively explore a small and valuable search space.

Besides test-suite based repair, other repair setups have been proposed. [START_REF] Pei | Automated fixing of programs with contracts[END_REF] proposed a contract based method for automatic repair. Other related repair methods include atomicity-violation fixing (e.g. [START_REF] Jin | Automated atomicity-violation fixing[END_REF]), runtime error repair (e.g. [START_REF] Long | Automatic runtime error repair and containment via recovery shepherding[END_REF]), and domain-specific repair (e.g. [START_REF] Samimi | Automated repair of HTML generation errors in PHP applications using string constraint solving[END_REF]; Gopinath et al ( 2014)).

Empirical Investigation of Automatic Repair

Beyond proposing new repair techniques, there is a thread of research on empirically investigating the foundations, impact and applicability of automatic repair.

On the goodness of synthesized patches, [START_REF] Fry | A human study of patch maintainability[END_REF] conducted a study of machine-generated patches based on 150 participants and 32 real-world defects. Their work shows that machine-generated patches are slightly less maintainable than humanwritten ones. [START_REF] Tao | Automatically generated patches as debugging aids: a human study[END_REF] performed a similar study to study whether machinegenerated patches assist human debugging. [START_REF] Monperrus | A critical review of automatic patch generation learned from humanwritten patches: essay on the problem statement and the evaluation of automatic software repair[END_REF] discussed in depth the acceptability criteria of synthesized patches. Martinez and Monperrus (2015) studied thousands of commits to mine repair models from manually-written patches. They later investigated [START_REF] Martinez | Do the fix ingredients already exist? an empirical inquiry into the redundancy assumptions of program repair approaches[END_REF]) the redundancy assumption in automatic repair (whether you can repair bugs by rearranging existing code). [START_REF] Zhong | An empirical study on real bug fixes[END_REF] conducted a case study on over 9,000 real-world patches and found two important facts for automatic repair: for instance, their analysis outlines that some bugs are repaired with changing the configuration files.

A recent study by [START_REF] Kong | Experience report: How do techniques, programs, and tests impact automated program repair?[END_REF] compares four repair systems: including Gen-Prog (Le Goues et al (2012b)), RSRepair [START_REF] Qi | The strength of random search on automated program repair[END_REF]), Brute-force-based repair (Le Goues et al (2012a)), and AE [START_REF] Weimer | Leveraging program equivalence for adaptive program repair: Models and first results[END_REF]). This work reported repair results on 119 seeded bugs and 34 real bugs from the Siemens benchmark and in the SIR repository. Our experiment is on a bigger set of real bugs coming from larger and more complex software applications. Long and Rinard (2016a) have performed an analysis of the search space of their two systems SPR and Prophet on the benchmark of C bugs. They show that not all repair operators are equal with respect to finding the correct patch first. This is clearly different from what we do in this paper: 1) we consider a different benchmark (Defects4J), in a different programming language (Java), while they consider the Manybugs benchmark of C bugs. 2) they do not perform any manual analysis of their results.

Conclusion

We have presented a large scale evaluation of automatic repair on a benchmark of real bugs in Java programs. Our experiment was conducted with three automatic repair systems on 224 bugs in the Defects4J dataset. We find out that the systems under consideration can synthesize a patch for 47 out of 224. Since the dataset only contains real bugs from large-scale Java software, this is a piece of evidence about the applicability of automatic repair in practice.

However, our manual analysis of all generated patches show that most of them are incorrect (beyond passing the test suite), because the overfit on the test data. Our experiment shows that overfitting also happens for Java software tested with a state-ofthe-art testing infrastructure (as opposed to old-school C programs in [START_REF] Qi | An analysis of patch plausibility and correctness for generate-and-validate patch generation systems[END_REF]); and also happens with Nopol, a synthesis based technique (and not only for Genprog and similar techniques).

Our results indicate two grand challenges for this research field: 1) producing systems that do not overfit (RQ2) and 2) producing systems that reason on the expected behavior beyond what is directly encoded in the test suite (RQ3). There is also a third challenge, although not the focus on this experiment: during the exploratory phase, we have observed the presence of multiple patches for the same bug. This indicates a need for research on approaches that order the generated patches so as to reveal the most likely to be correct. Preliminary work is being done on this topic by Long and Rinard (2016b).

Lastly, there is also a need for research on test suites. For instance, any approach that automatically enriches test suites with stronger assertions would have direct impact on repair, by preventing the synthesis of incorrect patches.

  21%) 27 (12%) 22 (9.8%) 35 (15.6%)

Fig. 2

 2 Fig. 2 Venn diagram of bugs for which test-suite adequate patches are found.

Fig. 3

 3 Fig.3Bar chart that summarizes the results from the correctness analysis.

Fig. 5

 5 Fig. 5 Code snippet of Bug M8. The manually-written patch is shown in the MANUAL PATCH comment at Lines 5 and 6 (changing a variable type). The patch by jKali in the PATCH comment removes the loop body at Line 11.

Fig. 6

 6 Fig.6Code snippet of Bug L55. The manually-written patch is shown in the MANUAL PATCH comment at Lines 6 and 7 while the patch by Nopol is shown in the NOPOL PATCH at Lines 8 and 9. The patch by Nopol is equivalent to the manually-written patch by developers.

Table 1

 1 The Main Descriptive Statistics of Considered Bugs in Defects4J. The Number of Lines of Code and the Number of Test Cases are Extracted from the Most Recent Version of Each Project.

	Project	#Bugs	Source KLoC	Test KLoC	#Test cases
	Commons Lang	65	22	6	2,245
	JFreeChart	26	96	50	2,205
	Commons Math	106	85	19	3,602
	Joda-Time	27	28	53	4,130
	Total	224	231	128	12,182
	if the patch is not validated by the test suite, the repair approach goes on with next
	statement and repeats the repair process.
	2.3 Defects4J Dataset

Defects4J by

Just et al (2014a)

is a bug database that consists of 357 real-world bugs from five widely-used open-source Java projects. Bugs in Defects4J are organized in a unified structure that abstracts over programs, test cases, and patches.

Defects4J provides research with reproducible software bugs and enables controlled studies in software testing research (e.g.,

Table 2

 2 Results for 224 Bugs in Defects4J with Three Repair Approaches. In Total, the Three Repair Approaches can Generate Test-suite Adequate Patches for 47 Bugs (21%).

	Project Bug Id jGenProg jKali Nopol

Table 3

 3 Manual Assessment of 84 Patches that are Generated by Three Repair Approaches.

	Project	Bug id	Patch id	Approach	Correctness	Readability	Difficulty	Project	Bug id	Patch id	Approach	Correctness	Readability	Difficulty
	JFreeChart	C1 C1 C3 C3 C5 C5 C5 C7 C13 C13 C13 C15 C15	1 2 3 4 5 6 7 8 9	jGenProg jKali jGenProg Nopol jGenProg jKali Nopol jGenProg jGenProg jKali Nopol jGenProg jKali	Incorrect Incorrect Unknown Incorrect Incorrect Incorrect Correct Incorrect Incorrect Incorrect Incorrect Incorrect Incorrect	Easy Easy Medium Easy Easy Easy Easy Easy Easy Easy Easy Easy Medium	Easy Easy Medium Medium Medium Medium Medium Easy Easy Easy Easy Medium Medium		M40 M42 M49 M49 M49 M50 M50 M50 M53 M57 M58 M69 M70	39 40 41 42 43 44 45 46 47 48 49 50 51	Nopol Nopol jGenProg jKali Nopol jGenProg jKali Nopol jGenProg Nopol Nopol Nopol jGenProg	Unknown Unknown Incorrect Incorrect Incorrect Correct Correct Correct Correct Incorrect Incorrect Unknown Correct	Hard Medium Easy Easy Easy Easy Easy Easy Easy Medium Medium Medium Easy	Expert Expert Medium Medium Medium Easy Easy Medium Easy Medium Hard Expert Easy
		C21		Nopol	Incorrect	Hard	Expert		M71	52	jGenProg	Unknown	Medium	Hard
		C25		jGenProg	Incorrect	Medium	Medium		M71	53	Nopol	Incorrect	Medium	Hard
		C25		jKali	Incorrect	Medium	Medium		M73	54	jGenProg	Correct	Easy	Easy
		C25		Nopol	Incorrect	Easy	Easy		M73	55	Nopol	Incorrect	Easy	Easy
	Commons Lang	C26 C26 L39 L44 L46 L51 L53 L55 L58		jKali Nopol Nopol Nopol Nopol Nopol Nopol Nopol Nopol	Incorrect Incorrect Incorrect Correct Incorrect Incorrect Incorrect Correct Correct	Easy Easy Easy Easy Easy Easy Hard Easy Easy	Medium Medium Medium Medium Medium Easy Expert Medium Medium	Commons Math	M78 M78 M78 M80 M80 M80 M81 M81 M81	56 57 58 59 60 61 62 63 64	jGenProg jKali Nopol jGenProg jKali Nopol jGenProg jKali Nopol	Unknown Unknown Incorrect Incorrect Unknown Unknown Incorrect Incorrect Incorrect	Easy Easy Medium Hard Easy Easy Easy Easy Easy	Hard Hard Hard Medium Medium Medium Medium Medium Medium
		T4		jGenProg	Incorrect	Easy	Medium		M82	65	jGenProg	Incorrect	Easy	Medium
	Time	T4 T11 T11		jKali jGenProg jKali	Incorrect Incorrect Incorrect	Easy Easy Easy	Medium Easy Easy		M82 M82 M84	66 67 68	jKali Nopol jGenProg	Incorrect Incorrect Incorrect	Easy Easy Easy	Medium Medium Easy
		T11		Nopol	Incorrect	Medium	Medium		M84	69	jKali	Incorrect	Easy	Easy
		M2		jGenProg	Incorrect	Easy	Hard		M85	70	jGenProg	Unknown	Easy	Easy
		M2		jKali	Incorrect	Easy	Hard		M85	71	jKali	Unknown	Easy	Easy
	Commons Math	M5 M8 M8 M28 M28 M32 M32		jGenProg jGenProg jKali jGenProg jKali jKali Nopol	Correct Incorrect Incorrect Incorrect Incorrect Incorrect Unknown	Easy Easy Easy Medium Easy Easy Hard	Easy Easy Easy Hard Hard Easy Expert		M85 M87 M88 M95 M95 M97 M104	72 73 74 75 76 77 78	Nopol Nopol Nopol jGenProg jKali Nopol Nopol	Incorrect Incorrect Incorrect Incorrect Incorrect Incorrect Incorrect	Easy Medium Easy Easy Easy Easy Hard	Easy Expert Medium Hard Hard Medium Expert
		M33		Nopol	Incorrect	Medium	Medium		M105	79	Nopol	Incorrect	Medium	Medium
		M40 M40		jGenProg jKali	Incorrect Incorrect	Hard Easy	Hard Medium		84 Patches for 47 bugs 5 patches correct from jGenProg, 1 from jKali and 5 from Nopol 11 Correct 61 Easy 21 Hard/Expert

Table 4

 4 The Most Challenging Bugs of Defects4J Because of Under-specification.

	Project	Bug ID
	Commons Math	M2, M8,M28,M32,M40, M49, M78,
		M80, M81, M82,M84, M85,M95
	JFreeChart	C1,C5, C13, C15, C25,C26
	Time	T4,T11

Table 5

 5 Time Cost of Patch Generation

	Time cost	jGenProg	jKali	Nopol
	Min	40 sec	36 sec	31sec
	Median	1h 01m	18m 45sec	22m 30sec
	Max	1h 16m	1h 27m	1h 54m
	Average	55m 50sec	23m 33sec	30m 53sec
	Total	8 days 12h	3 days 6h	7 days 3h

The dataset and the repair system in[START_REF] Kim | Automatic patch generation learned from human-written patches[END_REF] are not publicly available.

Apache Commons Lang, http://commons.apache.org/lang.

JFreeChart, http://jfree.org/jfreechart/.

Apache Commons Math, http://commons.apache.org/math.

Joda-Time, http://joda.org/joda-time/.

Google Closure Compiler, http://code.google.com/closure/compiler/.

Bug ID in the bug tracking system of Commons Math is Math-942, http://issues. apache.org/jira/browse/MATH-942.