Maria Rodalyn

V Sanchez

Satoru Mishima

Masayuki Fujiwara

Guangyi Ai
email: aiguangyi@nuit.edu.cn

Melanie Jouaiti

Yuliia Kobryn

Sébastien Rimbert
email: sebastien.rimbert@inria.fr

Laurent Bougrain

Patrick Hénaff
email: patrick.henaff@loria.fr

Hiroaki Wagatsuma

Mélanie Jouaiti

Methodological Design for Integration of Human EEG Data with Behavioral Analyses into Human-Human/Robot Interactions in a Real-World Context

Keywords: Electroencephalography, Eye-tracker, Motion capture system, Human-Robot interaction, Synchronization

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

1. Introduction. Studies of human behavior and cognition use questionnaire investigations. However, these contain subjective assessment and need to be completed by physical and physiological measurements of the human behaviors to validate hypotheses such as in psychology [START_REF] Baker | Over half of psychology studies fail reproducibility test[END_REF]. Especially in the interaction between humans or between humans and machines, reproducibility, verifiability, and generality as the foundation of science are crucial not only for the validation of the hypothesis in the scientific question, but also for the development of applications in various fields. In medical field applications, specifically in rehabilitation, brain-computer interface (BCI) for severely disabled persons, assistive devices for helping a physical movement at the right moment, and communication tools to know what is happening and changing in the interaction [START_REF] Ahn | Exploring Neuro-Physiological Correlates of Drivers' Mental Fatigue Caused by Sleep Deprivation Using Simultaneous EEG, ECG, and fNIRS Data[END_REF], obtaining physiological data for analysis is important. Electroencephalography (EEG), which measures the electrical activity of the brain, is suitable for these purposes. It provides a possibility to know what kind of brain processes are on-going in a specific task condition. Measurement and analysis of brain signals are beneficial for providing an insight on how human brain functions work associated with behaviors, actions and intentions.

For the past years, various EEG experiments have been examined such as playing musical instruments [START_REF] Goebl | Temporal Control and Hand Movement Efficiency in Skilled Music Performance[END_REF], performing sports [START_REF] Muraskin | Knowing when not to swing: EEG evidence that enhanced perception-action coupling underlies baseball batter expertise[END_REF][START_REF] Thompson | EEG applications for sport and performance[END_REF], and doing everyday tasks such as walking [START_REF] Peterson | Differentiation in Theta and Beta Electrocortical Activity between Visual and Physical Perturbations to Walking and Standing Balance[END_REF]. Integrating eye-tracking data with EEG recordings have also been done in various studies [START_REF] Noureddin | Online Removal of Eye Movement and Blink EEG Artifacts Using a High-Speed Eye Tracker[END_REF][START_REF] Plöchl | Combining EEG and eye tracking: identification, characterization, and correction of eye movement artifacts in electroencephalographic data[END_REF][START_REF] Ai | Direction and viewing area-sensitive influence of EOG artifacts revealed in the EEG topographic pattern analysis[END_REF], which require effective noise removal methods for EEG data [START_REF] Singh | A Removal of Eye Movement and Blink Artifacts from EEG Data Using Morphological Component Analysis[END_REF][START_REF] Ai | Simultaneous Analysis of EEGs and Movements in Interactive Hand Shaking Required Skills to Synchronize Cooperatively in Game[END_REF]. In the recent years, integrating more types of devices with EEG recording is being explored. However, this poses a challenge of ensuring the synchronization of data gathered from the different devices to guarantee an accurate analysis, since each device has a different internal clock and record data at different sampling rates [START_REF] Delorme | [END_REF]. In a previous study [START_REF] López-Gil | Method for Improving EEG Based Emotion Recognition by Combining It with Synchronized Biometric and Eye Tracking Technologies in a Non-invasive and Low Cost Way[END_REF], data synchronization was done through playing a loud sound that was recorded in the eye-tracker audio data, and appears in the EEG data because of the subject's response to the sound. Another study [START_REF] Reis | Methodological aspects of EEG and body dynamics measurements during motion[END_REF], used a software for data acquisition, called lab streaming layer (LSL) [START_REF] Reis | Methodological aspects of EEG and body dynamics measurements during motion[END_REF]. Various software for data synchronization were introduced [START_REF] Delorme | [END_REF], however, these techniques have the risk for software synchronization delays. To eliminate this risk, hardware implementations for synchronizations through the use of trigger signals [START_REF] Reis | Methodological aspects of EEG and body dynamics measurements during motion[END_REF], are examined, and is what is implemented in this article.

In this article, a method for simultaneously recording EEG with eye-tracker and motion capture system is proposed, together with the corresponding hardware synchronization tool and software treatment for the post-processing of the trigger signals. The three devices chosen are used for the testing and experiment in this article, but implementation is not limited to the specific equipment mentioned, since the methodological design can be applied to different devices in the goal of making reproducible experiments possible.

The objective of this article is to provide a methodological design for integrating EEG with other physiological data recording devices, thus enabling simultaneous recording while ensuring synchronization of the multiple data obtained. In order to achieve this, a generalized system of the methodological design was made to realize the possibility of simultaneously recording EEG together with other devices, including, but not limited to, eye-tracker and motion capture system. Simultaneously recording different devices involves the creation of a data synchronization tool in the form of trigger signals, wherein each device has its own unique trigger signal. In simultaneous recording, all trigger signals are sending at the same time, making the recorded trigger signal the sum of the individual device's unique trigger signals. Results indicate that with the trigger signal data obtained after simultaneous recording, accurate signal reconstruction for the individual device's trigger signal necessary for multiple data synchronization is possible and was demonstrated. The established system for the integration of EEG data with other essential data for the corresponding behavioral analyses is useful in BCI for rehabilitation, human-human and human-robot interaction applications, as presented later in this article.

Problem Statement and Preliminaries. The popularization of EEG experiments in various task conditions presents the same challenge concerning the reproducibility in

psychology studies [START_REF] Baker | Over half of psychology studies fail reproducibility test[END_REF]. Independently of the measurement equipment, for a consistent quality of the specification such as sampling rate, the same result should be obtained in a given experimental condition. Alternatively, how much variability exists in the uncontrollable variables such as condition of the human subject, even in the fully-controlled condition in is a helpful scientific question. To answer this question and realize the reproducible and replaceable experimental scheme, creation of a methodological design and solving technical problems are necessary. 2.1 Methodological design. As illustrated in Figure 1, one subject's EEG, gaze, and motion data can be simultaneously recorded for examining the human brain activity and behaviors. Moreover, the proposed methodological design provides the possibility for a subject to interact with different counterparts: (1) another human, whose EEG, gaze, and motion data can also be obtained, (2) a robot which runs on a computational model, or (3) a screen where the computational model is embedded in the computer. With this, if the experimenter aims to study a natural human-human interaction, the scheme allows for replacing a human counterpart into a computational model embedded in the computer or robot to verify whether the same phenomenon can be observed or not. If it is a reproducible scheme, the minimization of the complexity of embedded functions in the model reveals the core principle to reproduce the same phenomenon, and then the core can be reproduced in the engineering way. 2.2 Technical Challenges. In the aim of integrating EEG with other monitoring devices, such as eye-tracking device and motion capture system, we examined a possible way to synchronize every data in a consistent time management, and proposed the design scheme of how devices can be connected together in an accurate time framework to fit the device with the maximum sampling rate. Since not all equipment have a built-in sync tool, this problem gives rise to designing a system, may it be hardware or software, that will make data synchronization possible. In this article, three devices are used in obtaining data through simultaneous recording and individual device information is discussed below. To synchronize the data obtained from three devices, it is necessary for each one to have a trigger signal. EEG and Eye-tracker devices already have a built-in sync tool, while the motion capture system was not designed to send an accurate trigger signal to a high-sampling rate system. 3.2.1. Trigger Signal for EEG Data. The EEG trigger signal is dependent on the EEG experiment. Trigger behavior can be set per experiment, depending on the occurrence of an event to be observed. In Figure 2, the EEG trigger signal is generated by a photoelectric sensor. The pulse changes whether light or dark-colored surface is shown in the visual stimulus. In this case, the start of the EEG experiment was marked by the long pulse in the beginning and the following pulses mark the presentation of stimuli.

FIGURE 2. Trigger signal of EEG 3.2.2. Trigger Signal for Eye-tracker Data. The eye-tracking device has a built-in trigger for timing. As seen in Figure 3, The trigger signal has a cue for the start of eye-tracker data recording, which is the first three pulses with the frequency of 1 Hz. After that, a pulse is generated every 10 seconds. This behavior of the trigger signal is constant in every eye-tracker recording. FIGURE 3. Trigger signal of eye-tracker 3.2.3. Trigger Signal for Motion Capture System Data. The motion capture system device does not have a built-in trigger to determine the timing, so a circuit that outputs trigger signals for the motion capture system data was created. The time series is from a transistor-transistor logic (TTL) signal output from the created trigger circuit, which can mark the start and end of recording of the motion capture system software, as well as specific events since the created circuit can either output (1) single pulse, or a (2) periodic pulse. In the case seen in Figure 4, the trigger signal for the motion capture system is periodic where one pulse is 100ms long. This gives a frequency of 5 Hz for the motion capture system trigger signal. For the trigger signals to be recorded, the trigger signal circuits of (1) EEG, (2) eye-tracker, and (3) motion capture system should be connected to the digital input of the EEG equipment (g.tec USBamp), being the one having a high-sampling rate system among the other devices. The cable for digital input of the g.tec USBamp is split into ports of different voltages such as 1V, 2V, 4V, and 8V, which means the trigger signal will have the amplitude value of either 1V, 2V, 4V, and 8V, depending on which port it is connected to. In this article, the amplitudes of trigger signals chosen and assigned for each device are seen in Table 1. This approach makes use of the amplitude as the clear differentiating factor between the three trigger signals, which is useful for signal discrimination later on. simultaneously to the digital input channel, the last channel in the EEG data recording, to assure the timing accuracy due to the EEG system having the maximum sampling rate. In the set-up, there are three different trigger signals being sent to only one digital input channel. In the event that two or more signals occur at the same time, a sum of the signals will be obtained. There is an observable shift in amplitude due to this, but the values are still consistent with the individual amplitudes of the three trigger signals as seen in Table 1. 2, where the time increment for each pulse occurrence and the corresponding amplitude were obtained. 2 is then used for amplitude discrimination to distinguish which device the trigger signal is coming from. A MATLAB program was again created and used for the function of separating the trigger signals generated from different devices, from the recorded sum of trigger signals data that came from the last channel of the EEG recording. A sample of a trigger signal reconstruction data for each device is provided in Table 3. From here, the corresponding data of each device can also be synchronized for data matching and analysis, as seen in Figure 7. This methodology of human EEG data analysis involving time series of gaze and motion enables the experimenters to simultaneously record EEG, eye-tracker, and motion capture system in order to provide a significant data analysis.

5.

Applications. This method was partly introduced in the simultaneous analysis of EEG with hand movements in the interactive game [START_REF] Ai | Simultaneous Analysis of EEGs and Movements in Interactive Hand Shaking Required Skills to Synchronize Cooperatively in Game[END_REF] which benefits EEG experiments involving visuomotor tasks to exhibit a specific timing coordination when handshaking rhythms are synchronized together in two persons during the rock-paper-scissors game. Experiments with the proposed method can extend to the analysis of various face-to-face interactions concerned with bodily motion, and motor skill rehabilitation, which is highly affected by the visuomotor processing of the brain.

6. Extended Experimental Scheme. This innovative and consistent approach could be applied in EEG experiments to understand how humans and robots synchronize their own movements, as currently investigated by Jouaiti et al. [START_REF] Jouaiti | Motor Coordination Learning for Rhythmic Movements[END_REF]. They hypothesized that a computational model of a central pattern generator (CPG) based on non-linear relaxation oscillators can express the natural coordination mechanism not only in a coupling of humans, but also in the interaction between a human and a robot embedded with CPG inside. Indeed, they successfully demonstrated the reproduction of the same coupling phenomenon with the robot or a human. They used currently a consistent experimental setup, illustrated in Figure 1 including a TMSI Refa amplifier for EEG (32 channels, 512 Hz), T-Sens motion sensors [START_REF]TEA CAPTIV T-Sens Respiration[END_REF] and the Open Pose system [START_REF] Cao | Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, Computer Vision and Pattern Recognition[END_REF] for the human hand motion estimation as inputs for the model (CPG consists of Rowat-Selverston neurons [START_REF] Jouaiti | Hebbian Plasticity in CPG Controllers Facilitates Self-Synchronization for Human-Robot Handshaking[END_REF] calculated by the Runge-Kutta numerical integration). Instead of the circuit described in Section 3.2.3, a Nano Arduino card controlled by OpenVIBE scenarios [START_REF] Renard | OpenViBE: An Open-Source Software Platform to Design, Test and Use Brain-Computer Interfaces in Real and Virtual Environments[END_REF] was introduced to send the triggers to the T-Sens. This hardware synchronization method could be replaced and compared to the proposed method. 7. Conclusions. In the present study, an effective technique for integrating and data synchronization of EEG, gaze, and motion data involved in the simultaneous recording is presented. Each device had a trigger signal that was used for synchronizing multiple data obtained in the experiment. Trigger signals from three different devices are sent and recorded together with the EEG data. The exact time of signal occurrence of the individual trigger are determined through amplitude discrimination. Moreover, individual signal reconstruction was accomplished and was highly accurate with respect to the original data. This methodology will contribute in providing specific event analysis based on data from multiple devices and in realizing the reproducible experimental scheme where each element is replaceable in the future. Indeed, the scheme is mostly applicable in various experiments including EEG measurements and offer a wide possibility in device developments for rehabilitation and communications.

FIGURE 1 .

 1 FIGURE 1. An overview of the methodological design of simultaneous recording and possible counterparts in examining human-human/robot interactions.

FIGURE 4 . 3 .

 43 FIGURE 4. Trigger signal of Motion Capture System 3.3. Connection of the trigger signals. For the trigger signals to be recorded, the trigger signal circuits of (1) EEG, (2) eye-tracker, and (3) motion capture system should be connected to the digital input of the EEG equipment (g.tec USBamp), being the one having a high-sampling rate system among the other devices. The cable for digital input of the g.tec USBamp is split into ports of different voltages such as 1V, 2V, 4V, and 8V, which means the trigger signal will have the amplitude value of either 1V, 2V, 4V, and 8V,

FIGURE 5 .

 5 FIGURE 5. The last channel in an EEG recording dedicated for digital inputs. 4. Results and Discussion. Simultaneous recording of the three devices makes the trigger signals successfully send to the last channel in the EEG recording designated for digital input, as seen in Figure 5. The trigger signal recorded as seen in Figure 5, was used as an input into a created MATLAB program for finding each peak of the graph. Results are shown in Table2, where the time increment for each pulse occurrence and the corresponding amplitude were obtained. TABLE 2. Recorded trigger channel signal sample data Time (s) Amplitude (V) 54.142578125 14 54.144531250 14 54.14648437512 4.1. Trigger Signal Discrimination. Data from Table 2 is then used for amplitude discrimination to distinguish which device the trigger signal is coming from. A MATLAB program was again created and used for the function of separating the trigger signals generated from different devices, from the recorded sum of trigger signals data that came from the last channel of the EEG recording. A sample of a trigger signal reconstruction data for each device is provided inTable 3. TABLE 3. Sample Trigger Signal Reconstruction Data for Each Device EEG Eye-tracker Motion Capture System Time (s) Amp. (V) Time (s) Amp. (V) Time (s) Amp. (V) 50.972656250 0 63.175781250 0 50.138671875 0 50.974609375 8 63.177734375 2 50.140625000 4 50.976562500 8 63.179687500 2 50.142578125 4 Figures 6B, 6C, and 6D, show the result of each device's trigger signal reconstruction

  FIGURE 5. The last channel in an EEG recording dedicated for digital inputs. 4. Results and Discussion. Simultaneous recording of the three devices makes the trigger signals successfully send to the last channel in the EEG recording designated for digital input, as seen in Figure 5. The trigger signal recorded as seen in Figure 5, was used as an input into a created MATLAB program for finding each peak of the graph. Results are shown in Table2, where the time increment for each pulse occurrence and the corresponding amplitude were obtained. TABLE 2. Recorded trigger channel signal sample data Time (s) Amplitude (V) 54.142578125 14 54.144531250 14 54.14648437512 4.1. Trigger Signal Discrimination. Data from Table 2 is then used for amplitude discrimination to distinguish which device the trigger signal is coming from. A MATLAB program was again created and used for the function of separating the trigger signals generated from different devices, from the recorded sum of trigger signals data that came from the last channel of the EEG recording. A sample of a trigger signal reconstruction data for each device is provided inTable 3. TABLE 3. Sample Trigger Signal Reconstruction Data for Each Device EEG Eye-tracker Motion Capture System Time (s) Amp. (V) Time (s) Amp. (V) Time (s) Amp. (V) 50.972656250 0 63.175781250 0 50.138671875 0 50.974609375 8 63.177734375 2 50.140625000 4 50.976562500 8 63.179687500 2 50.142578125 4 Figures 6B, 6C, and 6D, show the result of each device's trigger signal reconstruction

FIGURE 6 .

 6 FIGURE 6. (A) Trigger signals of the three devices generated at the same time, (B) Trigger signal reconstruction for eye-tracker, (C) Trigger signal reconstruction for motion capture system, and (D) Trigger signal reconstruction for EEG.

FIGURE 7 .

 7 FIGURE 7. Synchronized start time for the trigger signal of the three devices

TABLE 1 .

 1 Trigger Signal Voltage Assignments for Each Device Multi-trigger Recording. All trigger signals are sent

	Device	Amplitude (V)
	EEG	8
	Eye-tracker	2
	Motion Capture System	4
	3.4.

TABLE 2 .

 2 Recorded trigger channel signal sample data

	Time (s)	Amplitude (V)
	54.142578125	14
	54.144531250	14
	54.146484375	12
	4.1.

Trigger Signal Discrimination. Data from Table

TABLE 3 .

 3 Sample

			Trigger Signal Reconstruction Data for Each Device
	EEG		Eye-tracker	Motion Capture System
	Time (s)	Amp. (V)	Time (s)	Amp. (V)	Time (s)	Amp. (V)
	50.972656250	0	63.175781250	0	50.138671875	0
	50.974609375	8	63.177734375	2	50.140625000	4
	50.976562500	8	63.179687500	2	50.142578125	4
	Figures 6B, 6C

Acknowledgment. This work was supported in part by JSPS KAKENHI (15H05874, 16H01616, 17H06383) and the New Energy and Industrial Technology Development Organization (NEDO).