domain: chim phys

text:
Shian Dong

Weihang Gao

Kunming Shi

Qi Kang

Zhenli Xu

Jinkai Yuan

Yingke Zhu

Hongfei Li

Jie Chen

Pingkai Jiang

Guangning Wu

Qiulong Wei

Jieshan Qiu

Xiaoshi Qian
email: xsqian@sjtu.edu.cn

Xingyi Huang
email: xyhuang@sjtu.edu.cn

Dielectric-Electrolyte Supercapacitors

Aqueous supercapacitors (SCs) are highly desirable as the next generation of energy storage devices, holding the promises of being environmentally friendly and free of safety concerns. However, the low water-splitting voltage prevents the practical applications. Here we utilized a dielectric-polymer-coated, hierarchically structured carbon electrode to prepare a dielectric-electrolyte supercapacitor (DESC), which exhibited an ultra-high and stable operating voltage of 3 V. The facile designing of the polymeric superstructure, covalently bonded upon the porous carbon electrode, allowed electrolyte ions to enter and interact with the electrode but effectively blocked water molecules from the entrance. In addition, the dielectric layer could store electrical energy through polarization and provide additional active sites for ions adsorption. With the promises intact, the energy density of the DESC is enhanced by over 1000% compared to the traditionally structured SC. This work paves a new way for producing low-cost and eco-friendly aqueous SCs with enhanced energy density.

Introduction

Supercapacitors (SCs) store energy through ion adsorption or fast surface redox reactions, filling the gap of power density between dielectric capacitors and batteries [START_REF] Simon | Where do batteries end and supercapacitors begin?[END_REF][START_REF] Griffin | In situ NMR and electrochemical quartz crystal microbalance techniques reveal the structure of the electrical double layer in supercapacitors[END_REF][START_REF] Simon | Materials for electrochemical capacitors[END_REF][START_REF] Lin | Materials for supercapacitors: When Li-ion battery power is not enough[END_REF][START_REF] Yu | Thin-Film Electrode-Based Supercapacitors[END_REF] . As a result, SCs are ideal choices for power buffers in applications such as electric transportation systems, automotives, energy recovery/harvesting systems, hybrid battery systems and electric grids [START_REF] Zhong | A review of electrolyte materials and compositions for electrochemical supercapacitors[END_REF][START_REF] Ghidiu | Conductive twodimensional titanium carbide 'clay' with high volumetric capacitance[END_REF][START_REF] Pomerantseva | Energy storage: The future enabled by nanomaterials[END_REF][START_REF] Simon | Perspectives for electrochemical capacitors and related devices[END_REF][10] . Operating the organic solvents and their corresponding inorganic salts, the deployment of the SCs raises safety issues (flammable and toxic), cost, and environmental issues from production to end-of-life phases, which prevent their deeper involvement in our society. Aqueous SCs are highly promising for future large-scale practical applications because of their environmentally friendly natures, fewer requirements for mass production, and high safety throughout the lifetime [11][12][13][14] . However, the inadequate energy density and low output voltage of the aqueous SCs are still the major challenges, owing to the low electrochemical stability window of 1.23 V 15,16 .

The past few decades have witnessed tremendous research efforts on expanding the operating voltage of the aqueous SCs. Significant advancement has been achieved in "water-in-salt (WIS)" electrolyte 15 , which reduces the kinetics of water-splitting reactions, further widening the operating voltage (ca. 2.3 V) in a full device.

However, the use of WIS electrolytes with low ionic conductivity and high viscosity is inevitably accompanied by sacrificed kinetics and power density 17 . In addition, the high concentration and cost of WIS electrolytes (e.g., Lithium bis(trifluoromethane sulfonyl) imide, LiTFSI) may put obstacles to their further implementation in major industrial departments. Other strategies involved designing a pair of suitable pseudocapacitive positive and/or negative electrode materials for asymmetric SCs (which react with electrons preferentially instead of water 11,12,[18][19][20] ) and covering the electrode with a proton exchange membrane to increase the operating voltage of SC (2.4 V) 21 . Besides, the methods that controllably tune P0V (the potential of both electrodes when the SCs operate at 0 V) and the potential variation rate of anodes and cathodes enable the potential practical range, further increasing the operating voltage (1.8 V) 14,22,23 . Despite the potential cost issue (e.g., changing the electrode material from the commonly used carbon materials), these methods failed to expand the operating voltage of the aqueous electrochemical device to near 3 V, where most commercial organic SCs are operating. Therefore, the key is to inhibit the electrochemical interaction between water molecules and electrons without raising issues of cost and/or profound changes in current SC production protocols.

Electrostatic capacitors possess a high operating voltage because of the excellent insulation of the dielectric films. Inspired by this, one may increase the operating voltage of SCs by integrating a dielectric layer with the electrode as the dielectric layer can isolate the water molecules from electrons. In contrast to electrolytic capacitors, the energy storage system possesses a high surface area of the electrodes and an ultra-thin dielectric layer, resulting in a considerably higher capacitance than electrolytic capacitors 24 . As an enabling component, polyvinylidene fluoride-cohexafluoropropylene (PVDF-HFP) was selected because of its nature as a decent dielectric with high polarizability, tuneable crystalline structure, high hydrophobicity, water resistance, and ease of processing [25][26][27][28] . The high hydrophobicity of PVDF-HFP can inhibit the water molecules from crossing the dielectric layer and interacting with the electrons, resulting in increased operating voltage. The high polarizability is favourable for storing electrical energy through polarisation, and the tuneable crystalline structure is beneficial for the electrolyte ions to penetrate the dielectric layer, resulting in an enhanced capacitance. The water resistance makes it possess long-term durable performance at aqueous electrolytes. Moreover, the ease of processing allows better integration of polymers with the electrode, and it is convenient for large-scale preparation as well.

Here, we report a new type of hybrid aqueous SCs named dielectric-electrolyte supercapacitor (DESC) with an ultra-high operating voltage (max 4 V in a full device) and hence a high energy density (an enhancement to 1200% of the conventional SC fabricated under the same device geometry). In DESCs, the electrode surface is coated with a cost-effective dielectric layer that forms hierarchical superstructures to allow ions to go through and interact with electrodes, meanwhile isolating the water molecules from the electrodes, thereby effectively increasing the operating voltage. We observe the entrance of electrolyte ions into the dielectric layer via polymeric chain separation and probe the underlying mechanism that allows the DESCs to work surprisingly well with the enhanced capacitance after the process of charge/discharge activation. Moreover, the dielectric layer may store additional electrical energy through dipolar polarization during the charging process, and the polarized dipoles could provide additional active sites for adsorbed ions, hence further enhancing the capacitance. A full DESC device offers 99.2% coulombic efficiency for up to 11,000 cycles.

Results and Discussion

The DESC is distinguished from the conventional SCs owing to the presence of the hydrophobic, dielectric polymer coating layer on the electrode surface, which serves as a selective pathway for ions while a barrier for water molecules (Figure 1a).

However, the affinity of electrodes and electrolytes and the conductivity of electrodes is positively related to the energy storage performance in past research 29,30 . The polymer featuring the electrochemical inert and hydrophobicity (blocking water molecules) is generally counter-intuitive to functionalize the electrode materials of aqueous SC. Therefore, it is a non-trivial task for DESC to achieve enhanced specific capacitance and improved operating voltage simultaneously.

To ease the dilemma, we designed wavy-like carbon material with a continuous and large outer surface area as a model electrode in order to increase the contact area between the electrode and the dielectric layer and effectively enhance the probability of ions penetration and their interaction with the carbon electrode. The preparation of the multi-scale, hierarchically structured electrode comprised four steps (Figure S1). First, a layer of nano-zinc oxide (ZnO, ca. 220 nm, Figure S2b) was grown vertically on the carbon cloth (CC, Figure S2a) surface via a hydrothermal method to obtain CC@ZnO (Figure S2c,d) 31 . Thereafter, the CC@ZnO, adsorbed by glucose, was carbonised to obtain carbon-coated CC@ZnO (CC@ZnO@C); subsequently, the CC@ZnO@C was treated with an acid solution to remove the ZnO for preparing the CC@C. Figure 1b shows the uniform coating of the CC@C and the presence of radial microchannels in the coating layer. The radial microchannels were formed because of the utilisation of ZnO as the sacrificial template (Figure S2e,f).

No diffraction signals of ZnO were detected in the X-ray diffraction (XRD) curve of CC@C, indicating the successful removal of ZnO during the pickling process (Figure S3a). BET data confirms the specific surface area of CC@C at 7.07 m 2 g -1 (weight of the CC was included, which contributed to the majority of the electrode weight) and the co-existence of micropores, mesopores and macropores (Figure S4a Finally, the CC@C was immersed in a PVDF-HFP solution. After being heated to 200 °C for 10 min and then quenched in ice water, the CC@C@PVDF-HFP composite electrode was obtained (Figure 1c).

The quenched process successfully forms a compact hydrophobic polymer layer (to block the water molecules, Figure 1d) with more disordered phases (e.g., enhanced α and amorphous phases and diminished β phase, see Figures S3b andS6), which holds more defects and exhibits large separated interchain spacing for ion penetration compared to the annealed PVDF-HFP for typical dielectric applications (Figure S7). The uniform and tight coating of the PVDF-HFP layer (ca. 400 nm, Figure S8) on CC@C is attributed to hydrogen bonds forming between O-H in CC@C (Figure S4d) and F-C in PVDF-HFP 34 . Resulting from the coating, the surface water contact angle increases from 87.0° to 137.2°, suggesting the excellent hydrophobicity of CC@C@PVDF-HFP (Figure S9).

The hydrophobic composite electrode would inevitably repel water; therefore, the electrochemical device may be charged to a large voltage of 3 V, but the device cannot work unless there exist channels for ions. The cyclic voltammetry (CV) curve of the as-prepared CC@C@PVDF-HFP was off-balance and featured almost above the X-axis, indicating that the DESC virtually had quite limited energy storage capacity as expected. Fortunately, as the operating voltage increases, we observed that the current increases significantly (Figure S10). The nonlinear rise of current without any redox reaction suggests the directional movement of charges in the DESC. In addition, the discharge current increases as the cycles of CV increase, illustrating an enhanced capability of energy storage. These results indicated that there exist molecular-scaled channels that allow electrolyte ions to enter the dielectric layer when a proper external voltage-bias was applied (Figure S10). On the other hand, the current peak at the high voltage (2.5 V) gradually shifted to the low voltage (1.2 V), suggesting that molecular-scaled channels were being created for ions to enter the dielectric layer. It hence implied that one could develop a proper activation process to prepare an ion-immersed dielectric layer which would further enable the typical electric double layer (EDL) structure on the electrode surface 35 . However, this unidirectional activation would nevertheless induce an imbalance between the charge and discharge currents, as the interaction between the electrolyte anions/cations and the dielectric is quite different. The induced channels for ion transportation can only be found in the dielectric covering one of the two electrodes.

To design a symmetric device, we, therefore, created a bipolar activation method with the operating voltage from -3 to 3 V to intercalate electrolyte ions into the dielectric layer (Figure 1e). As a result, both electrodes can be activated for proper and symmetric ion transportation. After the activation, the voltage range has already tripled the electrochemical window of conventional aqueous SCs in symmetric geometry. Eventually, the aqueous DESC can remain stable from -4 to 4 V for at least 30 cycles (Figure 1f), proving the marked improvement in operating voltage enabled by the rationally activated dielectric layers. XRD and FTIR-ATR show that the activated PVDF-HFP dielectric layer undergoes a phase transition from βand α-phases to the γ-phase (Figures S3b andS6) [36][37][38] . The phase transition indicated that the activation process would further increase the chain separation (the large dspacing of γ-phase PVDF-HFP) and facilitate the entry of electrolyte ions into the already-quenched dielectric layer 39 . Additionally, the activated CC@C@PVDF-HFP exhibits a significantly enhanced hydrophilicity at the interface between the electrolyte and the dielectric layer due to the ion accumulation (Figures S11 andS12) while the polymer layer effectively blocked the water molecule from entering the matrix. Time-of-flight secondary ion mass spectroscopy (TOF-SIMS) shows the abundant presence of F, O and Cl elements on the electrode surface, which confirmed that there were abundant electrolyte ions and water molecules on the electrode surface (Figure S13). Deep inside the PVDF-HFP dielectric layer, however, only the F and Cl elements can be observed (Figure 1g-i and 3D render in Figure S14). The observation indicates that the electrolyte ions can enter the dielectric layer, but the water molecules were prohibited, which is known as the desolvation process and has been previously proved to enhance the specific capacity of the electrode [START_REF] Simon | Materials for electrochemical capacitors[END_REF]40 .

After activation, the galvanostatic charge/discharge (GCD) curves and CV curves of the DESC exhibit the symmetrical triangles and the rectangle shapes, respectively, indicating a typical capacitive behaviour (Figures S15 andS16) 41 . The DESC shows an enhanced specific capacitance at a current density ranging from 0.1 to 1 mA cm -2 , compared with the CC@C SC without the dielectric layer. At low current density (0.1 mA cm -2 ), the specific capacitance was approximately tripled. Additionally, the specific capacitance of the DESC increases with the increasing voltage (Figure 2a), which may attribute to the intercalated ions in the polar polymeric matrix. The increase in voltage enhances the polarisation of PVDF-HFP and the polarised PVDF-HFP provides additional active sites for ion adsorption. Both are advantageous in increasing the specific capacitance of the DESC. When the voltage increases to 2.0 V, the specific capacitance of CC@C@PVDF-HFP reaches 168 mF cm -2 at 0.1 mA cm -2 , 2.8 times the specific capacitance of CC@C. The electrochemical performance of CC SC with 1 M KCl electrolyte is also provided in Figure S17, which exhibits that the capacitance of CC is only ca. 0.3 mF cm -2 , far lower than that of CC@C. In addition, the structure of CC@C is very stable. The capacitance retention rate is still 100% (Figure S18), even after 15,000 charge/discharge cycles, and the structure remains well (Figure S19).

We conducted electrochemical impedance spectroscopy to study further the charge transfer process (Figure 2b and Figure S20). The equivalent circuit models are provided in Figure S21, and the fitted data for circuit elements are shown in Table S1. In the high frequency region, the DESC exhibited a larger equivalent series resistance (RS, 4.45 Ω) than that of the SC (3.7 Ω) without PVDF-HFP coating. In the low frequency region, the ion transfer resistance of the DESC was observably higher than that of the conventionally fabricated SC due to the high resistance of ion transportation in the dielectric layer. In addition, the DESC after activation from -1 to 1 V showed a smaller equivalent series resistance than the freshly assembled DESC. This feature could be attributed to the activation process improving the affinity between the electrode and the electrolyte, reducing the interfacial resistance (RI). Further increasing the bipolar activation voltage from -2 to 2 V enhanced the low-frequency resistance, as more ions were trying to enter the dielectric layer and form conduction paths. At this time, the RI reduces from 387.8 to 310 Ohm, indicating that more interfaces are generated, and ion channels are established. Eventually, after activation from -3 to 3 V, the ion transfer resistance of the DESC reduces significantly from 310 to 4.52 Ohm, suggesting that the stable channels for ions transfer had been established, and the activation process was completed. Owing to the simultaneous increase in the operating voltage and specific capacitance, the energy density of the DESC is approximately 10.5 times that of an SC (Figure 2c). The energy density of the DESC is as high as 0.56 mWh cm -3 based on the device at a power density of 1.4 mW cm -3 , which is comparable to that of commercial organic AC-EC (2.75 V / 44 mF) 42 .

Both CV and GCD methods were used to investigate the electrochemical stability of the DESC. CV curves of the DESC show no obvious change from -3 to 3 V after 1000 cycles at 50 mV s -1 , indicating that the dielectric-coated electrodes remained stable during the charge and discharge processes (Figure 2d). The current peaks at 1.6 and -1.6 V did not appear in the first cycle, but the nonlinearity increased with the increasing cycling number. The reversible peaks may correspond to the field-induced ion intercalations and the polar structural changes 39 . The CV curves of the DESC remain stable (Figure 1f) even from -4 to 4 V for at least 30 cycles. The CV curve area (-4 to 4 V) is significantly larger than that of the CV curve (-3 to 3 V), indicating a significant increase in the specific capacitance, confirming the positive correlation between the specific capacitance of the DESC and the increasing voltage. At a current density of 2 mA cm -2 , the capacitance retention of the DESC increases to 190.8% and coulombic efficiency is still as high as 99.2% after 11,000 cycles (Figure S22). The increased capacitance can be attributed to the channels newly established in the dielectric layer by repeatedly exchanging the positive and negative electrodes. After exchanging the charging direction, we observed that the initial voltage was negative, which can be attributed to the remanent polarisation of the PVDF-HFP matrix. In addition, the charging duration was significantly increased (Figure S23), proving the intercalated ions were wholly removed from the dielectric layer and polarisation reversal of the dielectric layer after the exchange of electrodes, which releases more channels and active sites, further resulting in a substantial increase in capacitance. Besides, the high voltage test (2.5 V for an hour) shows the DESC has almost no change while SC was swollen obviously, indicating that the introduction of dielectric does suppress water splitting effectively (Figure S24).

After 11,000 cycles, the dielectric layer on CC@C@PVDF-HFP remained intact, while the continuous insertion/extraction of ions caused wrinkles on the surface of the dielectric layer (Figure S25). The long-term operation did not deteriorate the structure of the DESC, as the electrode cross-section exhibited a well-defined boundary between different layers. The presence of K + and Cl -ions in the PVDF-HFP layer further demonstrated the high possibility of the transference and storage of electrolyte ions in the dielectric layer (Figure S26). The RS of DESC before and after cycling is near 4.45 and 4.75 Ohm, respectively, proving the excellent stability of DESC as well. In addition, the RI reduces to 3.43 Ohm from 4.52 Ohm (Figure 2b and Table S1), indicating that more developed ion channels have been established after cycling, consistent with the previous analysis (Figures S22 andS23).

To further probe the energy storage mechanism and the operational kinetics of the DESC, we conducted the molecular dynamics (MD) simulation (Figure 3 and Figure S27) 43 . The dipoles, representing the dielectric layer, were set parallel to the electrode surface and did not show a net dipole moment (i.e., non-polarised, Figure 3a). First, the dipoles have rotated under all simulated charge densities after charging, indicating that the dielectric layer can store energy through polarisation (Figure 3b). Besides, the dipoles are in a high polarisation state at the beginning of charging, due to that the voltage is mainly applied to the dielectric layer due to its much lower conductivity than the electrolyte, resulting in a high electric field inside the dielectric layer. The dipolar polarizability starts to decrease as ions enter the dielectric layer, which can be considered as the energy storage transition from dipolar polarisation to ion adsorption. During the activation process, current fluctuations in the voltage range of 2 to 3 V and -2 to -3 V (Figure 1c) correspond to this balance conversion process. To further verify the accuracy of the model, we synthesised an ultra-thin dielectric layer coated on the carbon electrode surface by in-situ polymerisation, thereby observing the voltage fluctuation phenomenon during the charging process before activation (Figure 3c). This fluctuation can be explained by the fact that the capacitance has been enhanced as the electrolyte ions enter the dielectric layer during the charging progress, resulting in a drop in voltage (V = Q / C). As the charging continues, the voltage rises again, and the above process is repeated again to form the fluctuation, which is consistent with the fluctuation of the MD model (Figure 3b).

At the low charge density, electrolyte ions are mainly adsorbed in the outer dipole layer, and few ions are absorbed on the electrode surface, which can be considered as dielectric energy storage. As the charge density increases, the polarisation of the dipole is higher, and the adsorption of ions is closer to the electrode surface (Figure 3d-h). The polarised dipoles provide additional adsorption sites for ions, effectively reducing the force between the ions with the same charge and equivalently extending the length of the Helmholtz layer. The polarised dipoles adsorb a large number of ions, which can be considered dielectric-electrochemical collaborative energy storage, resulting in enhanced capacitance. In addition, we observed salt precipitation in the negative electrode tab while no change could be recognised in the positive electrode during the charging process (Figure 3i), indicating that the K + plays a more significant role in creating channels than the anions during the activation process.

The spatial element distribution in the activated electrodes was analysed through TOF-SIMS to further verify the MD simulation results (Figure 4). The activation was performed from -2 to 2 V (Figure S28) and -3 to 3 V (Figure 1e). The count of Cl signal, representing electrolyte ion, increases significantly in the Z direction as the activation voltage increases to 3 V from 2 V, demonstrating that the increase of voltage made the ions enter the deeper polymer layer (Figure 4b,e). This can explain the greatly enhanced specific capacitance of DESC activated from -3 to 3 V (Figure 2e). The detailed distribution of the Cl element in the polymer layer was plotted in Figures 4c and4f, which is consistent with the MD simulation. The increase of activation voltage greatly enhances the localised electric field in the dielectric layer, promoting the entry of electrolyte ions into the dielectric layer. As the electrolyte ions further penetrated the dielectric layer, the distance between the electrode and the ions was reduced, and the dielectric capacitance of DESC was also enhanced significantly (Figure 4g). The specific capacitance increased significantly (up to 277 mF cm -2 ) as the voltage was nonlinearly increased to 2.3 V (Figure S29), which is 1.6 times the DESC under 2 V (Figure 4h).

It was noted that the aforementioned results were obtained when the CC@C was uniformly coated at an optimised feeding of PVDF-HFP. Decreasing the PVDF-HFP feeding may result in the incomplete coating so that the prohibition of the water split reaction could be easily failed, while the increase of PVDF-HFP feeding does not enhance the performance of DESC significantly (Figures S30 andS31).

To analyse the generosity of the proposed DESC, we also investigated the different polymer dielectric layers such as PMMA, PVDF, PVDF-TrFE and PVDF-TrFE-CFE (Figure S32). These devices worked at an operating voltage of 3 V, suggesting the universality of the method in choosing the polymeric dielectrics.

To reduce the resistance and further enhance the specific capacitance of the device, a high-concentration electrolyte (6 M KOH) with better ionic conductivity was also used to prepare DESCs. Like the DESC with 1 M KCl, the DESC with 6 M KOH exhibited almost no energy storage capacity before activation (Figure S33a). After 20 cycles of activation from -3.5 to 3 V (Figure S33b), the DESC gradually exhibited the ideal capacitive behaviour (Figures S34 andS35). In the case of the DESC with 6 M KOH, the capacitance increases as the operating voltage increases as well; further, the specific capacitance of CC@C@PVDF-HFP can be as high as 333.3 mF cm -2 at 0.4 mA cm -2 with an operating voltage of 2 V (Figure 5a).

Figure 5b shows that the equivalent series resistance of the DESC with the 6 M KOH electrolyte is significantly reduced (3.2 Ω) compared to the DESC with 1 M KCl electrolyte (4.5 Ω). The DESC showed a distinct 45° Warburg impedance, indicating the potential ion diffusion behaviour in the PVDF-HFP layer. The energy density of the DESC with 6 M KOH electrolyte can reach as high as 1.12 mWh cm -3 (5.84 Wh kg -1 , Figure S36) at the power density of 5.4 mW cm -3 (Figure 5c), doubled that of commercial organic SCs 42 . The DESC exhibits superiority in terms of both energy density and power density even compared with the reported SCs prepared with electroactive materials [44][45][46][47] . Furthermore, the DESC-KOH also exhibited excellent stability and 100% capacitance retention even after 5000 cycles (Figure 5d). It is worth noting that the DESC can work even at a high operating voltage of up to 2.5 V, which is the highest record in KOH-based SC so far (Figure S37). With the unique high operating voltage, the light sign "SEITA" (> 5.4 V) can be powered by three DESCs in series, proving many potential applications of DESCs (Figure 5d).

In summary, a novel type of energy storage device, DESC, is proposed, where the SC electrode is innovatively coated with a high-dielectric polymer layer. There are three advantages of the introduced dielectric layer: First, the dielectric layer can effectively prevent direct contact between the electrode and water, resulting in a significantly increased operating voltage. Second, the high-dielectric polymer layer can store electrical energy through polarisation to increase capacitance. was purchased from Sinopharm Chemical Reagent. All of the chemicals were analytically graded and used as received without further purification.

Preparation of CC@ZnO Before the experiment, the CC substrate (3 cm × 5 cm) was ultrasonically washed with acetone, water, and ethanol for 30 min individually and then placed in H2O2 (12 mL), NH3•H2O (12 mL) and distilled water (120 mL) solution

for 12 h at 90 °C. The treated CC was then soaked in 0.008 M Zinc acetate for 30 min and then dried at 110 °C for 4 h to form a seed layer. Besides, an 80 mL precursor solution was prepared with 0.39 g of Zn(NO3)2•6H2O, 0.18 g of HMTA and 2 mL of ammonia under constant magnetic stirring (800 r min -1 ). The obtained solution was then transferred into the 100 mL autoclave liners with treated CC and kept sealing at 90 °C for 4 h. Finally, the CC@ZnO was prepared after washing with distilled water and ethanol repeatedly.

Preparation of CC@C The obtained CC@ZnO was immersed in 1 M glucose solution for 12 h under vacuum conditions, followed by annealing in Ar gas (40 mL min -1 ) at 900 °C for 2 h. The CC@C was prepared after washing with 1 M HCl and distilled water repeatedly.

Preparation of CC@C@PVDF-HFP In a typical experiment, a 20 mL solution was prepared with 0.15 g (0.05 g or 0.25 g) of PVDF-HFP and 20 mL DMF under constant magnetic stirring (800 rpm) at 80 °C. Two CC@C pieces were added to the obtained solution, followed by reduced pressure distillation (80 °C). After drying at 40 °C for 24 h, the CC@C@PVDF-HFP can be obtained by heating to 200 °C for 10 min and quenching in ice water. The mass loading of the PVDF-HFP layer was provided in Table S2.

Preparation of carbon electrode-thin dielectric layer

The carbon electrode was prepared by chemical vapour deposition. In a typical synthesis, carbon cloth (2.5*4 cm 2 , four pieces) in a corundum boat was put into a horizontal tube oven and then heated to 1080 °C at 5 °C min -1 and kept for 12 h in a flowing mixed atmosphere with 6 mL min -1 acetylene, 20 mL min -1 Ar and 70 mL min -1 H2 to make carbon electrode. The prepared carbon electrode was heated in concentrated nitric acid solution at 80 °C for 12 h, then washed with deionised water and dried at 110 °C for 24 h. The above-treated carbon electrode was cut into an electrode (2*2 cm 2 ) with a tab (0.5*2 cm 2 ). A pair of cut carbon electrodes were put in a three-necked flask with 60 mL toluene and 2.5 μL APS, bubbling with N2 and heating to 80 °C for 12 h with flowing N2. The carbon electrode-thin dielectric layer composite was obtained by washing with toluene to remove unreacted APS and dried at 80 °C K for 12 h.

Preparation of CC@C@Polymer

The experiment process is the same as the preparation of CC@C@PVDF-HFP, except that PVDF-HFP is replaced with PMMA, PVDF, PVDF-TrFE and PVDF-TrFE-CFE, respectively.

Fabrication of SC and DESC

The CC@C electrodes (d = 1.2 cm and 2.5 cm × 2 cm, mass loading is ca. 0.7 mg cm -2 ) were assembled in CR2032 stainless steel and pouch SC with porous cellulose membrane and 1 M KCl as separator and electrolyte, respectively. The CC@C@PVDF-HFP electrodes were assembled in pouch DESC with the same separator and electrolyte. The CC@C@Polymer electrodes were assembled in pouch DESC with the same separator and electrolyte. The CC@C electrodes (d = 1.2 cm and 2.5 cm × 1.8 cm, mass loading is ca. 1.0 mg cm -2 ) were assembled in CR2032 stainless steel and pouch SC with porous cellulose membrane and 6 M KOH as separator and electrolyte, respectively. The CC@C@PVDF-HFP electrodes were assembled in pouch DESC with the same separator and electrolyte. which applies a constant force on each ion F = e 2 /4πqlBQSkBT/(e 2 LxLy). In the simulation, the surface charge density σs=-0.25e𝑙 -2 , -el -2 and -2.5el -2 (-e nm -2 , -4e

Characterisation

nm -2 and -10e nm -2 ), respectively.

The dipole is modelled as a hard rod connecting two soft spheres with +-e charges.

The diameter of each dipole sphere is 0.6l, and the length of the rod is 0.3l. Three layers of dipoles (each layer includes 169 dipoles) are located near the two electrodes. The first layer is the nearest dipole layer neighbour with the electrode, the second layer is adjacent to the first layer, and the outermost dipole layer is the third layer. The distance between the first layer and electrode is 1.05l, and the length between every two layers is 1.8l. The forces between ion-ion and ion-dipole are electrostatics force and Lennard-Jones force. The equation of motion is integrated via the Velocity-Verlet algorithm in which the unit time of MD simulation

Δt is set to be 0.002 t*. The simulation was carried out in the canonical ensemble in which a Langevin thermostat controls the temperature. The simulation runs 5*10 4 Δt for equilibrium and another 5*10 4 Δt to calculate the average ensemble quantities.

Electrochemical measurements and calculations

For all capacitors, the electrochemical performances, including cyclic voltammetry (CV), galvanostatic charge/discharge (GCD), and electrochemical impedance spectroscopy (EIS), were tested on an electrochemical workstation (CHI 660e, Shanghai). The cycle stability was tested on a Neware battery cycler.

The specific capacitance of the electrodes, C (mF cm -2 ), is calculated according to the GCD curve according to Eq. ( 1).

C= 2I S ∆V ∆t (1)
Where I is the discharge current (mA), ΔV/Δt is the average slope of the discharge curve after the IR drop (V s -1 ), and S is the area of the electrode (cm 2 ).

The energy density E (mWh cm -3 ) and power density P (mW cm -3 ) are calculated according to Eqs. ( 2) and ( 3).

E= 1000 2×2×3600d CV 2 (2) P= E ∆t ( 3
)
Where d is the thickness of DESC (0.08 cm), V is the usable voltage after IR drop (V), and Δt is the discharge time (s).

Resource availability

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by Xingyi Huang. xyhuang@sjtu.edu.cn

Materials availability

This study did not generate new unique materials. Mater. 27, 2400-2405. 32. Zhao, J. et al. (2015). hydrophilic hierarchical nitrogen-doped carbon nanocages for ultrahigh supercapacitive performance. Adv. Mater. 27, 3541-3545. 33. Yu, S. et al. (2016) under different voltage windows. The CV curves of CC@C@PVDF-HFP DESC still maintain a quasi-rectangular shape even with the operating voltage from -4 to 4 V for 30 cycles. Meanwhile, the capacitance of DESC is enhanced with the increase of operating voltage. (g-i) TOF-SIMS chemical maps of CC@C@PVDF-HFP electrode after CV activation from -3 to 3 V. The distribution of (g) F element, (h) O element, and (i) Cl element in the total depth of the electrode. b) Cl in xz-plane of CC@C@PVDF-HFP electrode after activation from -2 to 2 V, (c) TOF-SIMS cross-section profiles of the CC@C@PVDF-HFP electrode showing the relative intensity for F and Cl elements across the film depth after activation from -2 to 2 V. Mapping of TOF-SIMS signals of (d) F and (e) Cl in xz-plane of CC@C@PVDF-HFP electrode after activation from -3 to 3 V, (f) TOF-SIMS cross-section profiles of the CC@C@PVDF-HFP electrode showing the relative intensity for F and Cl elements after activation across the film depth from -3 to 3 V. (g) Schematic illustration of ion distribution in the dielectric layer of the DESC after activation under a low/high voltage. (h) Specific capacitance values of CC@C and CC@C@PVDF-HFP at 0.1 mA cm -2 under different operating voltages (1.0, 1.5. 2.0 and 2.3 V).

Figure S1

. Schematic for CC@C@PVDF-HFP synthesis preparation. The synthesis procedure of CC@C@PVDF-HFP. First, ZnO nanorods are grown on a carbon cloth (CC) substrate by a facile low-temperature solution deposition method (step 1). Carbon nanosheets are coated on the surface of ZnO to fabricate CC@ZnO@C (step 2). The as-prepared CC@ZnO@C is then washed with 1 M HCl aqueous solution and distilled water repeatedly to prepare CC@C (step 3). Finally, CC@C@PVDF-HFP is prepared by coating with PVDF-HFP on the surface of CC@C by a quenching method (step 4). As a three-dimensional current collector, CC not only provides growth sites for carbon materials to form continuous, binder-free and exposed outer surface but also provides a good electron transmission path, thereby improving the performance of the device. . XRD spectra. (a) CC@ZnO@C and CC@C. The ZnO in CC@ZnO@C has been substantially removed by pickling with 1 M HCl solution. (b) CC@C@PVDF-HFP before and after quenching and after CV activation.

The phase transition of the PVDF-HFP polymer results in a small expansion of the 002 peak (moving from 2θ 20.5° (β-phases) to 20° (γ-phase). This transition is mainly due to the fact that ions enter the dielectric layer under the action of the electric field force during the charging process, thereby inducing the rearrangement of the polymer molecular chains.

Therefore, the transition is irreversible. After this small expansion, the d-spacing value is 0.433 nm according to the Bragg equation, significantly greater than the diameter of unsolvated K + (0.27 nm) and Cl -(0.36 nm) [START_REF] Simon | Where do batteries end and supercapacitors begin?[END_REF] . Therefore, the ions can penetrate the dielectric layer through this spacing. In addition, the polarization of PVDF-HFP can help the ion dissociation and promote the ion-dipole interaction 2 . The area density of carbon cloth (CC) and CC@C are 11.58 mg cm -2 and 12.28 mg cm -2 , respectively. Besides, the SBET of CC and CC@C are 1.03 and 7.07 m 2 g -1 , respectively.

According to the equation of SCC×mCC + SC×mC = SCC@C×mCC@C, the real SBET of loaded carbon is 124 m 2 g -1 . Compared with the reported activated carbon (1000-3000 m 2 g -1 )

electrode, the as-prepared carbon material has a low specific surface area. However, its uniform growth on the CC surface significantly improves complete contact with the dielectric layer, resulting in the full utilization of dielectric polymer for energy storage through polarization. In addition, it is conceivable that when the specific surface area of carbon materials is further increased, the energy density of DESC will also be significantly improved. The melting point of PVDF-HFP is ca. 160 °C. In this work, the CC@C@PVDF-HFP was heated at 200 °C, much higher than the melting point of PVDF-HFP. In this case, one can remove the voids and pores from PVDF-HFP caused by solvent evaporation and ensure a compact dielectric layer. On the other hand, the decreased melting point (from 164.2 to 159.7 °C) and decreased melting enthalpy (from 24.7 to 21.1 J g -1 ) prove that the quenching process results in a lower degree of crystallinity (from 27.8% to 23.7%) and smaller crystal size, facilitating the insertion of electrolyte ions. The electrode cross-sections were mounted on a stud and inserted in the FIB-SEM microscope chamber with the cross-section perpendicular to the electron beam [START_REF] Simon | Materials for electrochemical capacitors[END_REF] . The TOF-SIMS analysis was performed with the sample tilted at a 55° using the TESCAN motorized stage so that the cross-section is perpendicular to the FIB column (which is positioned at a 55° angle relative to the electron column). Since sputtering is a function of the angle of incidence between the incident beam and the sample surface, this step produced good results and facilitated data interpretation. Ga primary ion energy of 30 keV and beam current of 500 pA were used to collect data. A negative potential difference was applied in the TOF-SIMS device in order to collect positive ions, which were the positive-charged elements of interest. The mass-to-charge (m/z) ratios were calibrated using Li + , Mn + and Ga + . The Coulombic efficiency is low at low current densities, mainly because the ions will be trapped and cannot be wholly released after entering the dielectric layer. The efficiency of 1.5 V is lower than that of 2 V because the GCD of 1.5 V is tested before 2 V, so after some traps are filled, ions are no longer trapped. The capacitance of the CC electrode is ca. 0.3 mF cm -2 , far lower than that of CC@C. During the cycling test, the capacitance decreases mainly because the electrolyte ions are trapped in the dielectric layer, trapped in regions similar to deep traps, and cannot be released during the discharge process [START_REF] Yu | Thin-Film Electrode-Based Supercapacitors[END_REF] . In addition, the remanent polarisation of the dielectric layer also affects the cycling performance. However, it is worth noting that the specific capacitance increased approximately 2-5 times after exchanging the positive and negative electrodes. After the electrodes are exchanged, the corresponding ions trapped in the dielectric layer will be fully released under the reverse electric field, exposing more channels and active sites, resulting in a substantial increase in capacitance. The CC@C and CC@C@PVDF-HFP capacitors charge at a constant voltage of 2.5 V for an hour. Compared to CC@C@PVDF-HFP DESC, CC@C SC is swollen obviously, indicating that the introduction of PVDF-HFP does suppress water splitting effectively.

Figure S34

. CV profiles at different scan rates in 6 M KOH. (a) CC@C SC from 0 to 0.8 V, (b) CC@C@PVDF-HFP DESC from 0 to 0.8 V, (c) CC@C@PVDF-HFP DESC from 0 to 1.5 V and (d) CC@C@PVDF-HFP from 0 to 2.0 V. The CC@C@PVDF-HFP DESC can still work normally even if the operating voltage is up to 2.5 V.

  FigureS5)32 . The mesopores and macropores resulted from removing the ZnO template, while the micropores resulted from the chemical activation of ZnO 33 .

  Additionally, the polarised dipoles provide additional active sites for ion adsorption, thereby further enhancing the capacitance. The stable operating voltage of DESC can be possibly increased to a record value of 4 V in 1 M KCl electrolyte, and the energy density of the DESC can reach as high as 1.12 mWh cm -3 in 6 M KOH electrolyte. Overall, we demonstrate the positive role of a dielectric coating on electrodes in electrochemical energy storage and offer sufficient experimental data and MD simulations for understanding polarisation phenomena. It may promote the intersection of the two research directions of dielectric capacitors and SCs, and stimulate more related research. Experimental Procedures Materials Carbon cloth (CC, WOS 1002) was purchased from Ce Tech Co., Ltd. Zinc acetate dihydrate (Zn(CH3COO)2•2H2O, >99% purity), Poly(Vinyl Alcohol) (PVA, degree of hydrolysis 87-89%), Zinc nitrate hexahydrate (Zn(NO3)2•6H2O, 99% purity), hexamethylenetetramine (HMTA, 99% purity), ammonia (NH3•H2O, 25-28 wt%), γ-Aminopropyl triethoxysilane (APS), PMMA and glucose were obtained from Shanghai Titan Scientific Co., Ltd. PVDF and PVDF-HFP with 15 mol% of HFP was kindly mole ratio of 60.5 / 32.5 / 7, PVDF-TrFE-CFE, Terpolymer) was purchased from Piezotech at Arkema. N, N-dimethylformamide (DMF, 99.5%)

  photoelectron spectroscopy (XPS) using a Kratos Analytical Axis UltraDLD UHV spectrometer. N2 adsorption-desorption was performed on a Micromeritics ASAP 2010 instrument at -196 °C. X-ray diffraction (D/max-2200/PC, Rigaku, Japan) with a Cu Kα radiation (λ = 1.5406 Å) and Attenuated Total Reflection Fourier Transform Infrared (PerkinElmer Paragon 1000) were conducted to the crystal structure of PVDF-HFP. Differential scanning calorimetry (DSC, ETZSCH 200 F3, Germany) was

FigureFigure 1 .

 1 Figure Titles and Legends

Figure 2 .

 2 Figure 2. Electrochemical performance of CC@C SC and CC@C@PVDF-HFP DESC in 1 M KCl electrolyte. (a) Specific capacitance values versus current density for CC@C and CC@C@PVDF-HFP under different operating voltages. (b) Nyquist plots for CC@C SC and CC@C@PVDF-HFP DESC. (c) Ragone plots of CC@C SC and CC@C@PVDF-HFP DESC. (d) Capacitance retention test of CC@C@PVDF-HFP DESC at 50 mV s -1 from -3 V to 3 V for 1000 cycles.

Figure 3 .

 3 Figure 3. Molecular dynamics simulations. (a) Schematic illustration of ions and dipole arrangements before and after equilibrium. The distance between the positive and negative charge centers of dipole represents the polarisation degree of the dielectric layer. (b) Polarisation process of the first layer dipoles at different charge densities. (c) The galvanostatic charge curve of carbon electrode-thin dielectric layer DESC before activation. (d-f) Polarization process of different layer dipoles with different charge densities: (d) σs = -e nm -2 , (e) σs = -4e nm -2 and (f) σs = -10e nm -2 . (g) Cation ρN profiles at different charge densities. (h) Anion ρN profiles at different charge densities. (i) The Photograph of carbon electrode-thin dielectric layer DESC at a constant voltage of 2 V for an hour before CV activation.

Figure 4 .

 4 Figure 4. Energy storage mechanism of DESC probed by TOF-SIMS. Mapping of TOF-SIMS signals of (a) F and (b) Cl in xz-plane of CC@C@PVDF-HFP electrode after activation from -2 to 2 V, (c) TOF-SIMS cross-section profiles of the CC@C@PVDF-HFP electrode showing the relative intensity for F and Cl elements across the film depth after activation from -2 to 2 V. Mapping of TOF-SIMS signals of (d) F and (e) Cl in xz-plane of CC@C@PVDF-HFP electrode after activation from -3 to 3 V, (f) TOF-SIMS cross-section profiles of the CC@C@PVDF-HFP electrode showing the relative intensity for F and Cl elements after activation across the film depth from -3 to 3 V. (g) Schematic illustration of ion distribution in the dielectric layer of the DESC after activation under a low/high voltage. (h) Specific capacitance values of CC@C and CC@C@PVDF-HFP at 0.1 mA cm -2 under different operating voltages (1.0, 1.5. 2.0 and 2.3 V).

Figure 5 .

 5 Figure 5. Electrochemical performance of SC and DESC in 6 M KOH electrolyte. (a) Specific capacitance values versus current density for CC@C SC and CC@C@PVDF-HFP DESC under different operating voltages. (b) Nyquist plots for CC@C SC and CC@C@PVDF-HFP DESC. (c) Ragone plot of the fabricated CC@C SC, CC@C@PVDF-HFP DESC and other state-of-the-art works 42,44-47 . (d) Capacitance retention test of CC@C@PVDF-HFP DESC at 20 mA for 5000 cycles. The inset image shows the DESC can power a light sign (> 5.4 V).

Figure S2 .

 S2 Figure S2. Structural characterization for CC@C synthesis preparation. (a) SEM image of CC, (b) TEM image of ZnO nanorod. (c, d) SEM image of CC@ZnO. (e, f) SEM image of CC@C.

  Figure S3. XRD spectra. (a) CC@ZnO@C and CC@C. The ZnO in CC@ZnO@C has been substantially removed by pickling with 1 M HCl solution. (b) CC@C@PVDF-HFP before and after quenching and after CV activation.

Figure S4 .

 S4 Figure S4. Structural characterization of CC@C. (a) Nitrogen adsorption-desorption isotherms at 77 K. The type IV isotherm curve with a distinct hysteresis loop demonstrate the co-existence of micro-, meso-and macropores. (b) Pore size distribution of CC@C calculated by H-K method and BJH method. (c) XPS survey pattern of CC@C. The main elements of CC@C are carbon and oxygen, further proving the complete elimination of ZnO. (d) C 1s and O 1s spectrum of CC@C.

Figure S5 .Figure S6 .Figure S7 .

 S5S6S7 Figure S5. Structural characterization of CC@C. (a) Pore size distribution of CC@C calculated by the NLDFT model. (b) The cumulative pore volume plot of CC@C calculated by the NLDFT model. Those can prove the co-existence of micro-, meso-and macropores.

Figure S9 .Figure S10 .

 S9S10 Figure S9. Contact angle. Water contact angle measurement for the CC@C and CC@C@PVDF-HFP.

Figure S11 .

 S11 Figure S11. Hydrophilic test. (a) The schematic illustration of electrode hydrophilic mechanism. (b, c) Photographs of CC@C@PVDF-HFP electrode before and after electrochemical activation.

Figure S12 .

 S12 Figure S12. Contact angle. Water contact angle measurement for the CC@C@PVDF-HFP after CV activation. The water is absorbed by the CC@C@PVDF-HFP instantly, indicating enhanced hydrophilicity.

Figure S13 .

 S13 Figure S13. TOF-SIMS chemical maps of CC@C@PVDF-HFP electrode surface after CV activation. The lateral distribution of (a) F element, (b) O element, and (c) Cl element elements on the surface of the electrode.

Figure S14 .

 S14 Figure S14. TOF-SIMS analysis. (a) Schematic diagram of the CC@C@PVDF-HFP electrode. The area underlined in red is the test area. (b-d) The 3D render of F, Cl, and O elements in the PVDF-HFP layer. (e) TOF-SIMS cross-section profiles of the CC@C@PVDF-HFP layer showing the intensity for F, Cl, and O across the layer depth.

Figure S15 .

 S15 Figure S15. GCD curves of the capacitors at different current densities in 1 M KCl. (a) CC@C SC from 0 to 1 V, (b) DESC from 0 to 1 V, (c) DESC from 0 to 1.5 V and (d) DESC from 0 to 2 V. (e) The Coulumbic efficiencies of CC@C SC and DESC under different operating voltages.

Figure S16 .Figure S17 .

 S16S17 Figure S16. CV curves at different scan rates in 1 M KCl. (a) CC@C SC from 0 to 1 V, (b) DESC from 0 to 1 V, (c) DESC from 0 to 1.5 V and (d) DESC from 0 to 2 V.

Figure S18 .Figure S20 .

 S18S20 Figure S18. Cycling stability of CC@C SC with 1 M KCl electrolyte at 10 mA for 15,000 cycles.

Figure S21 .Figure S22 .

 S21S22 Figure S21. The equivalent circuit model. (a) The equivalent circuit of Randle's model for the CC@C SC. (b) The equivalent circuit model for the DESC. RS: equivalent series resistance. RCT: charge transfer resistance. W0: Warburg element. RB: bulk resistance. RI: interfacial resistance.

Figure S23 .

 S23 Figure S23. Electrode exchange test. (a) The GCD curve of DESC with electrode exchange after 11000 cycles. (b) D-E hysteresis loops of PVDF-HFP film (thickness of 86 μm) under elevated electric fields. (c) Schematic illustration of ion transfer and polarization reversal in PVDF-HFP layers after exchanging electrodes.

Figure S24 .

 S24 Figure S24. High voltage test.

Figure S26 .

 S26 Figure S26. Structural characterization of the CC@C@PVDF-HFP electrode in 1 M KCl electrolyte after 11000 cycles. Cross-section view HAADF TEM images of CC@C@PVDF-HFP. The following four images show the distribution of the element in the CC@C@PVDF-HFP electrode. EDS element mapping images illustrate that electrolyte ions (K + and Cl -) can pass through the dielectric layer.

Figure S27 .Figure S29 .

 S27S29 Figure S27. Molecular dynamics simulations. Demonstration diagram of dipole polarization and ion transfer process at a charge density of -10e nm -2 .

Figure S30 .Figure S31 .

 S30S31 Figure S30. Different mass of PVDF-HFP. (a, b) The mass of PVDF-HFP is 50 mg; (c, d) the mass of PVDF-HFP is 250 mg.

Figure S32 .Figure S33 .

 S32S33 Figure S32. Electrochemical performance of DESC with different dielectric materials. (a) PMMA, (b) PVDF, (c) PVDF-TrFE and (d) PVDF-TrFE-CFE.

Figure S37 .

 S37 Figure S37. Electrochemical performance of DESC with 6 M KOH electrolyte at 2.5 V.

Acknowledgments

This work was financially supported by the National Natural Science Foundation of China (Nos. 51877132, U19A20105, 52003153, 22005186, 52076127). Program of Shanghai Academic Research Leader (21XD1401600), State Key Laboratory of Electrical Insulation and Power Equipment (EIPE20203, EIPE21206) and China Postdoctoral Science Foundation (2021M690103). X.Q. thanks the National Key R&D Program of China (2020YFA0711500), Natural Science Foundation of Shanghai (20ZR1471700, 22JC1401800) and the Student Innovation Center at Shanghai Jiao Tong University. We thank Z. Bao, Y. Han, X. Li and Y. Li from the Instrumental Analysis Center of Shanghai Jiao Tong University (China) for assistance with SEM measurements, Y. Han, N. Zhang, L. Sun, and J. Ding from the Instrumental Analysis Center of Shanghai Jiao Tong University (China) for assistance with TOF-SIMS measurements.

Data and code availability

Full experimental procedures are provided in supplemental experimental procedures and in the supplemental Information.

Supplemental Information

Supplemental Information can be found online.

Author Contributions

S. D. and X. H. conceived the work. S. D., K. S., and Q. K. fabricated the electrode. S. D. fabricated and characterised the DESC. S. D. and J. Y. repeated the experiment. S. D., W. G., and Z. X. conducted the molecular dynamics simulation. J. Y., G. W., X. Q., Y. Z., H. L., J. C., Q. W. and J. Q. participated in the discussion of the energy storage mechanism. S. D. wrote the first draft of the manuscript. S. D., X. H., Z. X., and X. Q. revised the manuscript. All authors analysed the data and reviewed the manuscript.

Declaration of Interests

The authors declare no competing interests.

Inclusion and Diversity

We support inclusive, diverse, and equitable conduct of research.

SUPPLEMENTAL INFORMATION

Dielectric-Electrolyte Supercapacitors

Shian Dong 1 , Weihang Gao 2 , Kunming Shi 1 , Qi Kang 1 , Zhenli Xu 2 , Jinkai Yuan 3 , Yingke Zhu 1 , Hongfei Li 1 , Jie Chen 1 , Pingkai Jiang 1 , Guangning Wu 4 , Qiulong Wei 5 , Jieshan Qiu 6 , Xiaoshi Qian 7 *, Xingyi Huang CC@C from 0 to 0.8 V, (b) CC@C@PVDF-HFP from 0 to 0.8 V, (c) CC@C@PVDF-HFP from 0 to 1.5 V and (d) CC@C@PVDF-HFP from 0 to 2.0 V. Of which, the current density of CC@C@PVDF DESC is only 2 and 8 mA cm -2 as test voltage from 0 to 0.8 V and 0 to 1.5 V respectively. It does not make sense to test further because the IR drop is too large.


Quentin Le Lidec

Wilson Jallet

Ivan Laptev

Cordelia Schmid

Justin Carpentier

Enforcing the consensus between Trajectory Optimization and Policy Learning for precise robot control

Reinforcement learning (RL) and trajectory optimization (TO) present strong complementary advantages. On one hand, RL approaches are able to learn global control policies directly from data, but generally require large sample sizes to properly converge towards feasible policies. On the other hand, TO methods are able to exploit gradient-based information extracted from simulators to quickly converge towards a locally optimal control trajectory which is only valid within the vicinity of the solution. Over the past decade, several approaches have aimed to adequately combine the two classes of methods in order to obtain the best of both worlds. Following on from this line of research, we propose several improvements on top of these approaches to learn global control policies quicker, notably by leveraging sensitivity information stemming from TO methods via Sobolev learning, and Augmented Lagrangian (AL) techniques to enforce the consensus between TO and policy learning. We evaluate the benefits of these improvements on various classical tasks in robotics through comparison with existing approaches in the literature.

I. INTRODUCTION

B Y leveraging derivative information from the dynamics and costs, optimal control (OC) algorithms [START_REF] Li | Iterative linear quadratic regulator design for nonlinear biological movement systems[END_REF], [START_REF] Mayne | A Second-order Gradient Method for Determining Optimal Trajectories of Non-linear Discrete-time Systems[END_REF], [START_REF] Tassa | Synthesis and stabilization of complex behaviors through online trajectory optimization[END_REF], [START_REF] Mastalli | Crocoddyl: An efficient and versatile framework for multi-contact optimal control[END_REF] efficiently compute local controllers. Model Predictive Control (MPC) [START_REF] Rawlings | Model Predictive Control: Theory, Computation, and Design[END_REF] aims at retrieving a local controller with state feedback through online re-planning. However, current algorithms remain computationally too expensive to be run at high frequencies, are sensible to local minima and integration of information from sensors (e.g. force, vision) is technically difficult. This limits both their real-world capabilities and the practicality of deployment on complex settings requiring various sensor modalities. For these reasons, control policies learned directly from data are appealing as they offer the possibility to perform sensor-fusion and executed at very high frequencies.

Reinforcement learning (RL) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] is the prevailing paradigm when it comes to learning policies directly from observations. In RL, the physical dynamics are usually considered as unknown and classical algorithms [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF] rely on stochastic policies to build 0 th -order gradient estimators. Although they are unbiased, the high variance of such estimators make them very sample-inefficient. Actor critic approaches [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF] combine ideas from policy gradient and approximate dynamic programming by using an estimate of the Q function to reduce variance of policy gradient algorithms. This improves the sample efficiency but often at the cost of less stable training, and more advanced strategies, e.g. Proximal Policy Optimization (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] or Soft Actor-Critic (SAC) [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF], are often focused on regaining more stability. Eventually, even on problems easily solved via standard model-based approaches such as LQR, model-free algorithms exhibit several limitations and simpler rather than more complex methods may perform better [START_REF] Mania | Simple random search of static linear policies is competitive for reinforcement learning[END_REF].

In robotics, RL is never truly model-free, as samples are often generated by simulators which are actually based on models of physics. Therefore, we propose to embrace models when they are available, and, by using gradient information, to move from 0 th -order to 1 st -order methods with superlinear convergence properties. Such an approach is made possible by the recent emergence of differentiable physics engines, relying on derivatives of rigid body algorithms [START_REF] Carpentier | Analytical derivatives of rigid body dynamics algorithms[END_REF] and frictional contacts [START_REF] De Avila Belbute-Peres | End-to-end differentiable physics for learning and control[END_REF], [START_REF] Le Lidec | Differentiable simulation for physical system identification[END_REF]. A second key ingredient for modelbased approaches is access to efficient numerical solvers for trajectory optimization [START_REF] Mastalli | Crocoddyl: An efficient and versatile framework for multi-contact optimal control[END_REF], [START_REF] Jallet | Implicit Differential Dynamic Programming[END_REF]. Indeed, specialized OC algorithms exploit physical models, their derivatives, and the inherent temporal structure of the problem via the Bellman equation (or Pontryagin's principle). In particular, quasi-Newton (e.g., iterative Linear Quadratic Regulator (iLQR) [START_REF] Li | Iterative linear quadratic regulator design for nonlinear biological movement systems[END_REF], [START_REF] Tassa | Synthesis and stabilization of complex behaviors through online trajectory optimization[END_REF]) or second-order methods (e.g., Differential Dynamic Programming (DDP) [START_REF] Mayne | A Second-order Gradient Method for Determining Optimal Trajectories of Non-linear Discrete-time Systems[END_REF]), lead to improved convergence rates.

For complex scenarios, OC is often too slow to be run online, especially when it includes constraints on the system. Yet, it remains attractive to draw benefits from its capabilities offline; typically, for training policies. Following this path, guided policy search (GPS) [START_REF] Levine | Guided policy search[END_REF], [START_REF] Levine | End-to-end training of deep visuomotor policies[END_REF] aims at combining the strengths of OC and policy learning. Closer to our work, [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF] considers known deterministic dynamics and uses an ADMM framework to embed the policy learning problem within the OC formulation. The dual update from ADMM allows to automatically and gradually reach a consensus between the learned policy and the local controllers that OC yields. However, they relax the agreement constraint between RL and OC, which results in shifted, thus degraded, solutions. Additionally, they further exploit the feedback gains from trajectory optimization in order to supervise the first order derivatives of the policy. This technique, also known as Sobolev training [START_REF] Simard | Transformation invariance in pattern recognition-tangent distance and tangent propagation[END_REF], [START_REF] Czarnecki | Sobolev training for neural networks[END_REF], regularizes the training and improves generalization and thus the stability of the control policy. In this work, we extend it by using a stochastic variant [START_REF] Czarnecki | Sobolev training for neural networks[END_REF] to boost computational efficiency. Finally, a more recent work [START_REF] Mora | Pods: Policy optimization via differentiable simulation[END_REF] improves rollouts of the policy by performing approximate Newton steps in the trajectory space.

Here we establish some links between such an approach and OC, which in turn allows us to set the stage for a general formulation incorporating constraints while resulting in more

Precise

Efficient Constraints Sobolev Model-free

RL ✓ DPL [18] ✓ ✓ PODS [21] ✓ ✓ AL (Ours) ✓ ✓ ✓ ✓ PDDP (Ours) ✓ ✓ ✓ ✓
efficient algorithms. Table I summarizes some characteristics of our approach and the related works, i.e. the precision of the learned policy (Precision), the number of samples required to converge (Efficient), the ability to handle physical constraints (Constraints) or to leverage higher order information (Sobolev) and the need for a physical model (Model-free).

In this paper, we first present a general formulation, including physical and geometrical constraints, for policy learning which lays the ground for a better interplay between OC and policy learning (Sec. II). In Sec. III, we propose a first algorithm based on a projected descent and detail how stochastic Sobolev learning can be leveraged to provide a computationally efficient higher-order supervision of policy training. A second variation based on Augmented Lagrangian (AL) techniques is introduced in Sec. IV to automatically and precisely reach a consensus between the policy and the trajectory optimization oracle. To properly account for the lack of consensus at the beginning of the learning process, we propose to add multiple shooting component which allows to sensibly reduce the computational timings of learning a control policy. Finally, in Sec. V, the influence of each component of the approach is evaluated separately and we validate the final algorithms on a set of standard robotics tasks.

II. INTERPLAY BETWEEN POLICY LEARNING AND TRAJECTORY OPTIMIZATION

We consider deterministic and known system dynamics

x t+1 = f (x t , u t ) (1a) x t ∈ X and u t ∈ U (1b)
while other parameters of the optimal control problem (OCP) are randomly distributed. These parameters, denoted by β, include the initial condition of the problem x 0 ∈ X , as well as other parameters (desired terminal state or end-effector position); they follow a random distribution P. The set U = [u min , u max ] contains the admissible control inputs, of dimension n u , while X refers to the set of feasible states also accounting for obstacles. The deterministic assumption on the dynamics is reasonable, as a wide range of systems (robotic arms, manipulators, quadrupeds, drones) can accurately be described without stochasticity. We denote u = (u 0 , . . . , u T -1 ) the sequence of controls, and x = (x 0 , . . . , x T ) the state sequence. The total cost associated with a given control sequence u is:

R(β; u) = T -1 t=0 ℓ t (x t , u t ; β) + ℓ T (x T ; β) (2)
Unlike previous works in the literature [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF], [START_REF] Mora | Pods: Policy optimization via differentiable simulation[END_REF], we also account in this work for state and control constraints in Eq. (1b) directly in the TO problems. Experiments from Sec. V notably demonstrate how this consideration of the constraints substantially improves the precision and the stability of the resulting policy. The goal of this paper is to learn closed-loop control policies π : X → U which approximately minimizes the trajectory cost. The policy π is constrained to lie in a set of parameterized policies Π; typically, we will consider neural network policies with two hidden layers of 256 units each. The policy parameters are denoted by θ and lie in a set Θ. The policy space is thus

Π = {π θ : θ ∈ Θ}.
The activation functions are ReLU except for the final layer which is activated by a hyperbolic tangent so that the output is restricted to [u min , u max ]. We denote by R, the cost function and J(β; π θ ) the policy cost, which is the trajectory cost with controls given by u t = π θ (x t ):

J(β; π θ ) = R (β; π θ (x 0 ), . . . , π θ (x T -1 )) with x 0 = x 0 , x t+1 = f (x t , π θ (x t )) (3)
Our goal is to solve the policy optimization problem:

min θ∈Θ E β [J(β; π θ ))] (4)
where E β [•] is the expectation.

To do so, we will follow the path initiated in [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF], [START_REF] Levine | Learning complex neural network policies with trajectory optimization[END_REF] and adopt a constrained formulation linking together trajectory optimization and policy optimization:

min θ,u E β [R(β; u)] (5a) s.t. ∀t u t = π θ (x t ), a.s. ( 5b
)
where we now optimize over a distribution of control sequences u, and x = (x t ) t is defined by (1a). The constraint enforces that the controls and the policy are consistent on almost all states x t reached by randomization of β.

III. ALTERNATING BETWEEN OC AND SUPERVISED LEARNING: A PROJECTED DDP APPROACH

Projected DDP algorithm. A classical way of solving a constrained optimization problem consists in alternating between making a step on the unconstrained problem and projecting the new iterate on the constraints (e.g., projected gradient algorithms). In this spirit, (5) can be solved by optimizing the control u via trajectory optimization before projecting the new trajectory on the set of trajectories obtainable through Π. Discarding the constraints, the problem (5) can be solved by approximating the expectation with a finite sum R(u) obtained by sampling N instances of the problem β (i) ∼ P:

R(U ) = 1 N N i=1 R(β (i) ; u (i) ). (6)
Algorithm 1: Projected DDP descent (PDDP) Input: Distribution on the parameters β ∼ P, model for the policy:

π θ ∈ Π Output: Optimal policy π ⋆ θ 1 for k = 1 to M do 2 U k+1,0 ← Rollout(π θ k ); // Initial guess 3 U k+1 ← arg min U R(U ); // OC (with guess)
// Supervised learning (can also use d2 )

4 θ k+1 ← arg min θ d(θ, U k+1 ); 5 end
where U = (u (1) , . . . , u (N ) ) is the entire set of control sequences. The objective above is separable, hence each u (i) can be optimized separately in parallel.

The sampled problems are then optimized via discrete-time trajectory optimization methods such as Differential Dynamic Programming (DDP) [START_REF] Mayne | A Second-order Gradient Method for Determining Optimal Trajectories of Non-linear Discrete-time Systems[END_REF] or its variant iLQR [START_REF] Tassa | Synthesis and stabilization of complex behaviors through online trajectory optimization[END_REF], using a rollout originating from the learned policy π θ as a warm-start. Then, the local control trajectories (u (i) ) i are projected onto an element of Π in the least-squares sense. The projection problem is equivalent to the following supervised learning problem:

min θ∈Θ d(θ, U ) = N i=1 T -1 t=0 1 2 u (i) t -π θ x (i) t 2 2 . (7)
which is solved using classical stochastic mini-batch gradient algorithms such as Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. Alternating between these two operations results in Alg. 1 which we call Projected DDP (PDDP). Such an approach encompasses the one proposed in [START_REF] Mora | Pods: Policy optimization via differentiable simulation[END_REF] where Gauss-Newton (GN) steps are successively projected. By establishing an interplay with OC, here we rather perform a DDP step which is just a more efficient way to compute a GN step that also handles the constraints from (1). Previous works [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF], [START_REF] Mora | Pods: Policy optimization via differentiable simulation[END_REF] often ignore these constraints during the trajectory optimization phase. Typically, the box constraint on the control is rather lifted to the policy space via a tanh activation function on the last layer. This discrepancy can be fatal as trajectories obtained from OC might not be reproducible by the neural network and, thus, could cause divergence of the combined approach. Experiments in Sec. V demonstrate how crucial this specificity of our methods is to ensure convergence. Another obvious advantage is the possibility to consider more complex tasks involving geometric constraints or dynamic constraints on the systems to control (e.g., obstacles or joint limits).

Stochastic Sobolev learning. If the constraint from (5) enforces the output of the learned policy to match the local controllers, this should also be true for higher order derivatives. This idea, exploited in [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF], refers to the concept of Sobolev training [START_REF] Czarnecki | Sobolev training for neural networks[END_REF]. Second-order methods such as DDP additionally yield local feedback gains K t around the optimized trajectory. These gains can be used for stabilization of the controlled system around the local optimum, as [START_REF] Dantec | First Order Approximation of Model Predictive Control Solutions for High Frequency Feedback[END_REF] has shown on real-world systems. For policy learning, we can make use of these gains to regularize the policy π θ during the supervised learning phase (Alg. 1, line 4), leading to the following supervised learning problem:

min θ∈Θ d2 (θ, U ) = d(θ, U ) + 1 2 i,t K (i) t - ∂π θ ∂x (x (i) t ) 2 2 (8)
However, getting the second order derivative ∂ xθ π θ , as in [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF], is computationally expensive as it requires to perform multiple backpropagation operations. For this reason, we propose to use a stochastic version of Sobolev learning [START_REF] Czarnecki | Sobolev training for neural networks[END_REF] which is done by matching projections of K and ∂ x π θ on control directions v (i) randomly sampled on the unit sphere. The regularization term from (8) becomes

1 2 i,t ∥(v (i) ) ⊤ K (i) t -∂ x v (i) ⊤ π θ (x (i) t ) ∥ 2
2 and this greatly improves computational efficiency by avoiding backpropagating several times across the neural network.

IV. GRADUALLY ENFORCING A CONSENSUS: AN

AUGMENTED LAGRANGIAN APPROACH The Lagrangian function associated to the constrained formulation ( 5) is given by:

L(θ, u, λ) = E β R(β; u) + T -1 t=0 λ ⊤ t (u t -π θ (x t )) (9)
where λ = (λ t ) 0≤t≤T now designate random variables which are the Lagrange multipliers. We can define the corresponding augmented Lagrangian by:

L µ A (θ, u, λ) = L(θ, u, λ) + E β T -1 t=0 µ t 2 ∥u t -π θ (x t )∥ 2 (10)
ADMM algorithm. We first consider that a set of sample parameters (β (i) ) is fixed during training. Indeed, the Lagrange multipliers are intrinsically linked to the sampled values of β and thus, re-sampling them would require to update λ without losing progress on the optimization, which is a difficult task left as future work. This differs from Alg. 1 which can be naturally run online. Thus, we consider the following sampled version of the constrained problem (5):

min θ,U 1 N N i=1 R(β (i) ; u (i) ) (11a) s.t. u (i) t = π θ (x (i) t ), ∀t, ∀i ∈ 1, N (11b)
where (β (i) ) 1≤i≤N . The classical and augmented Lagrangians for this sampled variant are respectivily given by:

L(θ, U , Λ) = R(U ) + i,t (λ (i) t ) ⊤ (u (i) t -π θ (x (i) t )), (12)
Lµ A (θ, U , Λ) = L(θ, U , Λ) + i,t µ t 2 u (i) t -π θ (x (i) t ) 2 , (13)
where Λ = (λ (1) , . . . , λ (N ) ) are the Lagrange multipliers for each constraint block {u

(i) t = π θ (x (i) t )} t .
Both functions are separable with respect to the entry u (i) .

Algorithm 2: Policy learning via ADMM (PLAL)

Input: Distribution on the parameters β ∼ P, model for the policy:

π θ ∈ Π Output: Optimal policy π θ ⋆ 1 for k = 1 to M do 2 U k+1 ← arg min u Lµ A (θ k , U , Λ k ); // Supervised learning 3 θ k+1 ← arg min θ Lµ A (θ, U k+1 , Λ k ); // Dual update 4 Λ k+1 ← (λ (i) t + µ(u (i) t -π θ (x (i) t ))) t,i ; 5 end
The problem is solved using the alternating direction method of multipliers (ADMM) [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF], as described in Alg. 2 which we call PLAL. The algorithm alternatively minimizes L µ A w.r.t. the primal variables U and θ, before updating the dual variable Λ through a simple dual ascent step. As done in Sec. III, minimization w.r.t. each u (i) is done using a DDPtype algorithm such as FDDP [START_REF] Mastalli | Crocoddyl: An efficient and versatile framework for multi-contact optimal control[END_REF], while the minimization w.r.t. θ is performed using a classical stochastic optimization algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. If [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF] also uses ADMM, the consensus is approximated via a quadratic penalty term, while in this work we enforce it with a hard constraint (11b). This results in more stable and precise solutions as shown in Sec. V. Contrary to Alg. 1, the OC (Alg. 2, line 2) and supervised learning (Alg. 2, line 3) phases are now linked via the Lagrange multipliers Λ. Indeed, the additional terms appearing in [START_REF] De Avila Belbute-Peres | End-to-end differentiable physics for learning and control[END_REF] enforces the demonstrations from OC to be adapted to the current capabilities of the policy, while the dual update (Alg. 2, line 4) gradually leads to an agreement between the local controllers from OC and the policy. The previously introduced stochastic Sobolev term can naturally be integrated in the approach as a regularization of the learning phase (Alg. 2, line 3), and the algorithm is then interpretable as an occurrence of Global Variable Consensus with Regularization [START_REF] Boyd | Distributed optimization and statistical learning via the alternating direction method of multipliers[END_REF].

Multiple shooting formulation. Due to the augmented Lagrangian terms, the trajectory optimization phase (Alg. 2, line 2) requires to optimize the state x through the neural network policy and thus involves the costly computation of the Jacobian ∂ x π θ . Inspired by [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF], we propose to decouple the state variables of the supervised learning and OC problems, in a way similar to what is done in multiple shooting [START_REF] Bock | A multiple shooting algorithm for direct solution of optimal control problems[END_REF]. To do so, two state variables are introduced ( 1 X = ( 1 x (i) ) i and 2 X = ( 2 x (i) ) i ):

min θ,U , 1 X, 2 X 1 N N i=1 R(β (i) ; 1 x (i) , u (i) ) (14a) s.t. ∀t, ∀i, u (i) t = π θ ( 2 x (i) t ) (14b) 2 X = 1 X (14c)
The corresponding Augmented Lagrangian is:

L µ A (θ, 1 X, 2 X, U , Λ, Γ) = 1 N N i=1 R(β (i) ; 1 x (i) , u (i) ) + T -1 t=0 µ (i) t 2 u (i) t + λ (i) t µ (i) t -π θ ( 2 x (i) t ) 2 + µ (i) t 2 1 x (i) t + γ (i) t µ (i) t -2 x (i) t 2 - 1 2µ (i) t λ (i) t 2 - 1 2µ (i) t γ (i) t 2 ( 15
)
and the update rule for γ is:

γ (i) t ← γ (i) t + µ (i) t ( 1 x (i) t -2 x (i) t ) (16)
By decoupling the state variables between the control and learning problems, we avoid the need of computing the Jacobian of π θ when doing optimal control, lifting the burden to a simple dual update [START_REF] Levine | Guided policy search[END_REF].

V. EXPERIMENTS

Our implementation uses Pinocchio [START_REF] Carpentier | The pinocchio c++ library: A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives[END_REF] and Crocoddyl [START_REF] Mastalli | Crocoddyl: An efficient and versatile framework for multi-contact optimal control[END_REF] for defining and solving OCPs, and relies on PyTorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] for learning the neural network. We also propose our own implementations of PODS [START_REF] Mora | Pods: Policy optimization via differentiable simulation[END_REF] and DPL [START_REF] Mordatch | Combining the benefits of function approximation and trajectory optimization[END_REF] as they were not provided along with the original papers. For RL algorithms, i.e. PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and SAC [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF], we use implementations provided by [START_REF] Raffin | Stable-baselines3: Reliable reinforcement learning implementations[END_REF]. Our code is open-source and will be publicly released upon publication acceptance. Every experiment was run on a single laptop without GPUs and took about 5 minutes to train policies, even on complex systems such as UR5.

The first problem we consider is a constrained LQR where the control inputs are forced to stay into a box. This is a typical convex optimization problem already very well-studied in the control community. This control problem already highlights different characteristics and limitations of approaches under consideration. The second set of experiments are related to robotics systems of increasing complexity: an inverted simple pendulum, a double pendulum and an UR5 robotic arm. These problems are challenging as their dynamics are highly non-linear and the considered targets are unstable. In addition, we consider non-linear cost functions of the form:

ℓ(x t , u t ) = 1 2 ∥p(x t ) -p∥ 2 Wp + 1 2 ∥u t ∥ 2 Wu + 1 2 ∥x t ∥ 2 Wx
where p corresponds to the position of the end-effector which should reach a desired position p and W x , W u , W p are given weight matrices. On robot systems, we only penalize the joint velocities in the state penalty term ∥x t ∥ 2 Wx .

A. Ablation study: constrained LQR and inverted pendulum

Model-based vs model-free. proaches are an order of magnitude more efficient than modelfree algorithms. Fig. 2 also demonstrates how our approach is able to reach precise solutions while RL algorithms are limited by the inherent noise from 0 th -order gradients estimates. Among already existing model-based algorithms, PODS seems to dominate DPL both in terms of precision and efficiency. Mainly, DPL suffers from the approximated formulation which shifts the initial problem and hinders progress towards the solution even on the constrained LQR.

Constrained vs unconstrained OC. If including the trajectory constraints (1b) in the trajectory optimization phase allows to slightly improve results on instances of the constrained LQR (Fig. 3), the results appear even more pronounced on the inverted pendulum problem (Fig. 4). In particular, we observe a significant performance drop when removing this component (see PDDP v. PODS or PLAL+S v. PLAL+S-C on Fig. 4). Intuitively, as the policy tries to mimic an unreachable control from the OC phase, it deteriorates the final solution, resulting in a significant gap in performance.

Multiple shooting. As mentioned in Sec. IV, multiple shooting allows to reduce the computational burden of the TO phase without modifying the final solution (Fig. 3). In practice, we observe that the OC phase of PLAL+M is 5 times faster than the one of PLAL. However, the results from Fig. 3,4 also exhibit how multiple shooting can require more simulator calls to learn a good policy (see PLAL+M v. PLAL on Fig. 3, and PLAL+M+S v. PLAL+S on Fig. 4). Sobolev training allows to obtain gains in terms of sample efficiency by exploiting higher-order information made available by the trajectory optimization solver (see PLAL+M v. PLAL+M+S on Fig. 3). In addition, it also yields better generalization properties which significantly improve the stability of the policy rollouts and facilitate the training (see PLAL+M+S v. PLAL+M on Fig. 4). The latter characteristic becomes even more important for tasks on more complex systems with chaotic dynamics (Sec. V-B).

B. Robotics systems

Here, we consider tasks on the double pendulum and UR5 robotic arm. These tasks are made very challenging by the highly non-linear and chaotic dynamics and the instability of the desired target configuration.

As already noticed during experiments on the inverted pendulum, taking into account the physical constraints and exploiting the higher-order information contained in the feedback gains is crucial. In the case of the double pendulum and UR5, the combination of these components even appears to be necessary to get converging algorithms (Fig. 5). Due to the chaotic dynamics, for the PODS algorithm, even slight drifts from the policy w.r.t the TO controller induce very large deviations which, in turn, drive the policy towards saturation We represent the trajectories (right) and the corresponding control (left) on a testing sample of UR5. The 1 st row corresponds to the learned policy while the 2 nd and 3 rd represent the trajectories obtained via TO when the solver is initialized respectively with and without the learned controller. 7. Benefits of policy learning on TO. We record the number of iterations required by the TO solver to reach a given precision on OCPs from the testing set. We compare the cases where the solver is initialized with and without a rollout of the learned policy. and cause the rollouts to diverge. Including the physical constraints in the OC phase and regularizing the learning phase with the feedback gains allow to reduce the gap, and thus stabilize the training process.

Fig. 5 (right) highlights the influence of the augmented Lagrangian parameter µ on the convergence of ADMM algorithms. As described in V-A, multiple shooting already hinders the stability of the learning process, and when using it, we suspect the setting of µ to be even more crucial for the quality of the final solution (Fig. 5,right).

If we already demonstrated how policy learning can take advantage of TO, reciprocally, we observe that TO can also benefit from being warm-started by the learned policy (Fig. 7). In our case, doing so allows to systematically avoid worst case scenarii where the solver needs more than 100 steps.

Eventually, Fig. 6 demonstrates that warm-starting the solver with the policy also impacts the final control as it changes the local minima found by TO algorithms.

VI. DISCUSSION, CONCLUSION AND FUTURE WORK

In this paper, we have introduced a general framework to leverage the interplay between policy learning and trajectory optimization. Our formulation results in two algorithmic variants which both exploit two constrained optimization techniques: a projected DDP method, and the alternating direction method of multipliers for establishing a strong consensus between the learned control policy and TO. Both algorithms proceed in two alternating steps: solving an OCP and solving a supervised learning problem. We have introduced enhancements for either part: constraints handling, stochastic Sobolev regularization and multiple shooting. Although these algorithms achieve faster convergence towards more precise solutions when compared to classical RL algorithms, the experiments also highlight the need for advanced numerical trajectory optimization solvers. Indeed, for challenging robotic tasks, constraint handling and higher order information appear to be necessary to stabilize the training of the policy. The experiments and the ablation study also reveal the interest in our proposed enhancements: stochastic Sobolev learning enhanced sample efficiency and the stability of learned policy rollouts, and multiple shooting eliminates drift in the optimized trajectories.

A possible enhancement is the convergence speed of the method towards a consensus between the OCP and the supervised learning loop. It is known that Augmented Lagrangian methods (which includes ADMM) can reach much higher performance by having a strategy for updating the penalty parameter (µ t ). One such strategy in the literature is that of the bound-constrained Lagrangian (BCL) method of [START_REF] Conn | A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds[END_REF], which could be a direction for further enhancements.

In this work, we assumed the samples (β (i) ) i used for policy optimization are fixed; they are never re-sampled (which is how stochastic algorithms such as SGD work) nor are new samples ever added. This is the domain of constrained stochastic optimization; one of the difficulties here is updating the Lagrange multipliers in a way where optimization progress is not lost at every sampling. Such an extension could allow active sampling of the initial condition x 0 or other parameters so to explore the state space and learn policies more effectively.

We could then explore extensions towards setups closer to GPS [START_REF] Levine | End-to-end training of deep visuomotor policies[END_REF] by considering partially unknown system dynamics, with limited knowledge of some physical parameters (e.g. friction coefficients) -in order to take advantage of physical models once again. This would require embedding a system identification step in the loop which would also benefit from differentiable simulation techniques.

Fig. 3

 3 compares model-based algorithms, i.e. Algs. 1,2, DPL and PODS, to the wellestablished deep RL algorithms PPO and SAC, on a constrained LQR problem. It appears that model-based ap-

Fig. 1 .Fig. 2 .Fig. 3 .

 123 Fig. 1. Learned policy on UR5. After training, a rollout of the policy leads to precise control on a test problem.

Fig. 4 .Fig. 5 .

 45 Fig.[START_REF] Mastalli | Crocoddyl: An efficient and versatile framework for multi-contact optimal control[END_REF]. Sample efficiency: inverted pendulum. Using a constrained formulation[START_REF] Rawlings | Model Predictive Control: Theory, Computation, and Design[END_REF] along with supervision of the derivatives appears essential to reach a precise solution on a non-linear dynamics with unstable target.

Fig. 6 .

 6 Fig.6. Comparisons of controllers from TO and policy learning. We represent the trajectories (right) and the corresponding control (left) on a testing sample of UR5. The 1 st row corresponds to the learned policy while the 2 nd and 3 rd represent the trajectories obtained via TO when the solver is initialized respectively with and without the learned controller.

  Fig.7. Benefits of policy learning on TO. We record the number of iterations required by the TO solver to reach a given precision on OCPs from the testing set. We compare the cases where the solver is initialized with and without a rollout of the learned policy.

TABLE I CHARACTERISTICS

 I OF VARIOUS POLICY LEARNING ALGORITHMS.

ACKNOWLEDGMENTS

This work was supported in part by L'Agence d'Innovation Défense, the HPC resources from GENCI-IDRIS(Grant AD011012215), the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute), the AGIMUS project, funded by the European Union under GA no.101070165 and Louis Vuitton ENS Chair on Artificial Intelligence.