Keywords: Analyse de la fonction cardiaque, Apprentissage profond, Segmentation de l'échocardiographie, Suivi du mouvement en échocardiographie, Décomposition de l'électrocardiogramme, Apprentissage multimodal, Apprentissage profond avec incertitude vi Cardiac function analysis, Deep learning, Echocardiography segmentation, Echocardiography motion tracking, Electrocardiogram decomposition, Multi-modal learning, Deep learning with uncertainty viii

Ensuite, nous proposons un modèle DL explicatif pour la décomposition non supervisée de l'électrocardiogramme. Ce modèle peut extraire des informations explicables liées aux différentes sous-ondes de l'ECG sans annotation manuelle. Nous appliquons ensuite ces paramètres à un classificateur linéaire pour la détection de l'infarctus du myocarde, qui montre une bonne généralisation sur différentes séries de données.

Enfin, nous combinons les données des deux modalités pour une classification multimodale fiable. Notre approche utilise une fusion au niveau de la décision intégrant de v l'incertitude, permettant l'entraînement avec des données multimodales non appariées. Nous évaluons ensuite le modèle entraîné à l'aide de données multimodales appariées, mettant en évidence le potentiel de la détection multimodale de l'IM surpassant celle d'une seule modalité. Dans l'ensemble, nos algorithmes proposés robustes et généralisables pour l'analyse de l'échocardiographie et de l'ECG démontrent un potentiel significatif pour l'analyse de la fonction cardiaque portable. Nous anticipons que notre cadre pourrait être davantage validé à l'aide de dispositifs portables du monde réel.

Résumé

Selon le rapport annuel de la Fédération Mondiale du Coeur de 2023, les maladies cardiovasculaires (MCV) représentaient près d'un tiers de tous les décès mondiaux en 2021. Comparativement aux pays à revenu élevé, plus de 80% des décès par MCV surviennent dans les pays à revenu faible et intermédiaire. La répartition inéquitable des ressources de diagnostic et de traitement des MCV demeure toujours non résolue. Face à ce défi, les dispositifs abordables d'échographie de point de soins (POCUS) ont un potentiel significatif pour améliorer le diagnostic des MCV. Avec l'aide de l'intelligence artificielle (IA), le POCUS permet aux non-experts de contribuer, améliorant ainsi largement l'accès aux soins, en particulier dans les régions moins desservies.

L'objectif de cette thèse est de développer des algorithmes robustes et automatiques pour analyser la fonction cardiaque à l'aide de dispositifs POCUS, en mettant l'accent sur l'échocardiographie et l'électrocardiogramme. Notre premier objectif est d'obtenir des caractéristiques cardiaques explicables à partir de chaque modalité individuelle. Notre deuxième objectif est d'explorer une approche multimodale en combinant les données d'échocardiographie et d'électrocardiogramme.

Nous commençons par présenter deux nouvelles structures d'apprentissage profond (DL) pour la segmentation de l'échocardiographie et l'estimation du mouvement. En incorporant des connaissance a priori de forme et de mouvement dans les modèles DL, nous démontrons, grâce à des expériences approfondies, que de tels a priori contribuent à améliorer la précision et la généralisation sur différentes séries de données non vues. De plus, nous sommes en mesure d'extraire la fraction d'éjection du ventricule gauche (FEVG), la déformation longitudinale globale (GLS) et d'autres indices utiles pour la détection de l'infarctus du myocarde (IM). This thesis explores how AI-based models could help automatic cardiac function analysis for non-expert practitioners, using portable modalities such as echocardiography and electrocardiogram.

1.1 Thesis context 1.1.1 Point-of-care Ultrasound (POCUS)

For the past decades, cardiovascular diseases (CVDs) have stood as a significant global disease burden, witnessing a doubling in prevalent cases from 1990 to 2019 [START_REF] Roth | [END_REF].

Addressing the prevention and management of CVD patients requires effective diagnostic methods, with point-of-care ultrasound (POCUS) proving to enhance the diagnostic landscape [King, 2016]. However, the intricacies of echocardiography analysis demand years of medical training and experience from cardiologists in order to provide accurate diagnostic. Unfortunately, the poorly balanced distribution of cardiovascular medical professionals also hinders the CVD diagnostics and management [Narang, 2016]. To tackle this challenge, automatic cardiac function analysis emerges as a potential solution.

With the help of Artificial Intelligence (AI), POCUS could add more benefits for cardiac diagnosis and decision making. One study showed that medical students using a handheld ultrasound device with the aid of AI, can achieve very good inter-rater reliability when evaluating ejection fraction from 4-chamber view echocardiography [Dadon, 2020].

Another study demonstrated that AI-enabled POCUS movement guidance can help novices to acquire high-quality echocardiography [START_REF] Cheema | [END_REF] and improve trainees' confidence in acquisition [Waldman, 2022]. AI-enabled POCUS not only can improve the workflow of cardiologists but also enables trainees/nurses to help, facilitating the usage of POCUS for patient care at a large scale.

echOpen Project

There are already many market players who are trying to develop POCUS for efficient diagnosis. However, their high cost may be a burden for public use at large scale. echOpen, which aims at developing open-source and low-cost hand-held ultrasound device, will become a strong game changer in the landscape of POCUS. The echOpen device enhanced by AI will simplify and improve health-care in different scenarios, for example, at home, in health centers, elderly houses. It can also be used by non medical doctors especially in under-served areas.

The goal of echOpen is to achieve widespread availability of ultrasound imaging. This objective is pursued by harnessing cost-effective Point-of-Care Ultrasound (POCUS) technology in conjunction with AI tools.

Thesis motivation

This thesis was inspired from our collaboration with echOpen, which started in 2020.

Our objective is to explore AI algorithms for portable cardiac function analysis that may be deployed with portable ultrasound devices such as those from echOpen project. In the meantime, we focus on exploring the possibility of combining portable ultrasound and wearable electrocardiogram devices for more robust diagnosis or cardiac function evaluation from a multi-modal point-of-view.

Clinical context

Why the heart is so important? The heart is the central element of the human's circulation system, the engine that supports body activity. It can be regarded functionally as two blood pumps, with the left heart and the right heart working jointly in order to maintain the systemic circulation and pulmonary circulation, respectively. In particular, the left ventricle (LV) ejects blood into the aorta (Ao), which transports oxygenated blood to all organs involved in the arterial system. Systemic veins receive deoxygenated blood to entry the right atrium (RA). The right ventricle (RV) then pumps blood into pulmonary artery (PA). Oxygen and carbon dioxide get exchanged during pulmonary circulation.

The oxygenated blood passes through the left atrium (LA) and arrives at the left ventricle for another systemic circulation.

Echocardiography

2D echocardiography (ECHO) is a very useful tool to view the heart in real-time. The standard machine to obtain 2D echocardiograhy is the phased array ultrasound transducer.

At the front-end of the transducer, it contains thousands of piezoelectric crystals, which are able to convert electric currents to ultrasound waves and vice versa. Once the reflected waves are received, the vibration will be converted to electric currents and sent for image reconstruction. The standard examination by echocardiography includes different imaging windows and imaging views, whose choice are based on the targeted structure of the heart. For example, as the left ventricle plays an important role in blood bumping, echocardiography is often used to observe left ventricle wall motion and to detect abnormalities by conducting apical 4-chamber, 2-chamber view, long axis view, mid/basal short axis view etc. (Figure 1.

3) The most commonly extracted parameter from echocardiography is the left ventricular ejection fraction (LVEF), an important clinical index. This index is computed from the measurement of the left ventricle volume from 4-chamber and 2-chamber apical view, jointly. Ejection fraction is the percentage of blood ejected by LV during one heart beat, which reflects the effectiveness of the LV for pumping blood into the systemic circulation. Using recent techniques (e.g. speckle tracking echocardiography) one can calculate several myocardial deformation related parameters, such as global longitudinal strain (GLS) which represents the percentage of longitudinal shortening at peak-systole. Studies have shown that GLS is able to detect more subtle changes in myocardial function compared to LVEF [START_REF] Kalam | Prognostic implications of global LV dysfunction: a systematic review and metaanalysis of global longitudinal strain and ejection fraction[END_REF]Morris, 2014]. A healthy heartbeat is controlled by a harmonized series of contractions of the four heart chambers, which depend on the electrical conduction within the heart. The sinoatrial (SA) node (i.e., the primary pacemaker site of the heart) generates an action potential wave that is rapidly propagated across the atria. Active potential waves then arrive at the atrioventricular (AV) node, which is normally the only pathway for them to enter the ventricles. The propagation rate slows down at the AV node so that the atria has enough time for depolarization and contraction, letting atria blood entering the ventricles. Next, the action potentials propagate through the AV node to the ventricle base via the bundle of His, then through left and right bundle branches on the septum. These specialized fibers have high conduction speeds (2 m/s), and branch out into Purkinje fibers, which rapidly transmit impulses (4 m/s) throughout the ventricles, culminating in a direct connection with ventricular myocytes for final conduction at cellular level [START_REF] Klabunde | Cardiovascular physiology concepts[END_REF]. A well-functioning conduction system ensures rapid and near-synchronized depolarization and contraction of cardiac myocytes, which is essential for the normal ejection of ventricles. The electrocardiogram (ECG) is an important tool to detect abnormalities in the electrical conduction within the heart. As the body tissues are able to conduct electrical currents that are generated from the heart, ECG can be recorded by electrodes located on the body's surface. The standard ECG used in clinics usually refers to the 12-lead ECG, and different leads record the cardiac electrical activity from different angles. In particular, the limb leads I, II, III, aVR, aVL and aVF (green points in Figure 1.5) record the electrical currents in the frontal plane of the heart. Precordial chest leads V1-V6 (purple leads in Figure 1.5) measure the electrical activity in the horizontal plane, which is perpendicular to the frontal plane. By analyzing the ECG signals, cardiologists are capable to identify abnormal conduction activity and select patients for further diagnostic or treatment. A single-lead ECG signal (for example those acquired from wearable ECG device), can easily detect common rhythm or conduction related disorders, but are less informative for structural problems, such as myocardial infarction, ventricular hypertrophy and so on [Witvliet, 2021]. A typical ECG trace of one single heartbeat consists of: P wave; QRS complex; and, T wave (Figure 1.6). These sequences represent the atrial depolarization, ventricular depolarization and ventricular repolarization, respectively. The morphology of different ECG components plays an important role in diagnosing cardiac pathologies, such as myocardial ischemia, infarction etc.

Electrocardiogram

Myocardial infarction

Coronary arteries are responsible for blood delivery to the heart muscle, which is crucial for a normal functioning of the heart. According to a national survey in the United States, coronary related heart disease accounts for near 4.9% in adults1 .Myocardial infarction (MI), also known as heart attack, is the most significant contributor to sudden cardiac death. It's reported that there are around 803,000 people encounter a heart attack in the United States every year [Tsao, 2023]. MI occurs due to a gradual accumulation of atherosclerotic plaque within a coronary artery. Eventually, this plaque may suddenly rupture, leading to the rapid formation of a clot (Fig. 1.7) that completely blocks the artery. This blockage, in turn, disrupts the flow of blood to the heart muscle, resulting in severe tissue damage due to lack of oxygen supply. A patient is diagnosed with MI if he/she presents with elevated cardiac troponin values and also falls into at least one of the following conditions: symptoms of myocardial ischaemia; new changes of ST-segment/T-wave in ECG; development of pathological Q waves in ECG; abnormal myocardium motion; and, the presence of coronary thrombus [Thygesen, 2018]. In other terms, ECG and echocardiography could serve as primary tools for detection of MI patients, thus facilitating the classification of MI patients at hospitals.

Methodological context 1.3.1 Automatic echocardiography analysis

2D Echocardiography serves as a widely adopted and economically advantageous method for diagnosing cardiac dysfunction. There is a significant demand for automated solutions that can proficiently and cost-effectively assess cardiac function during clinical evaluations. In this context, the techniques of segmentation and motion tracking play pivotal roles in extracting essential cardiac parameters like the left ventricle ejection fraction (LVEF) and global longitudinal strain (GLS). Nevertheless, these tasks are intricate due to challenges encountered in ultrasound images, such as low signal-to-noise ratios, unclear boundaries, and issues with visibility.

Cardiac segmentation

Recently, deep learning methods have significantly advanced in the medical image segmentation domain, including automatic segmentation in 2D echocardiography (i.e., identifying the structure of left ventricle blood pool, myocardium and left atrium ).

In particular, the UNet architecture and its variants have demonstrated exceptional performance [Leclerc, 2019a;Ling, 2022]. Different regularization techniques have been proposed to constrain the shape regularity for cardiac imaging segmentation [Oktay, 2017;Clough, 2020]. Segmentation output is also a valuable source for both systolic and diastolic function analysis [Puyol-Antón, 2022a]. One crucial application of cardiac segmentation is the estimation of LVEF, a fundamental cardiac measurement in assessing cardiac function [Folse, 1962]. The quantification of LVEF involves using the bi-plane or single-plane Simpson's method [Folland, 1979] for volume assessment, thus is sensitive to image quality and segmentation process. Moreover, automatic identifications of end-diastole (ED) and end-systole (ES) frames [Smistad, 2020;Leclerc, 2019a] are also necessary. An alternative approach for LVEF estimation involves analyzing entire echocardiography sequences. This includes methods employing recurrent neural networks or transformer architectures to predict LVEF [Kazemi Esfeh, 2020;Reynaud, 2021].

Cardiac motion tracking

Traditional motion estimation methods, such as block matching [START_REF] Azarmehr | [END_REF][START_REF] Boukerroui | Velocity estimation in ultrasound images: A block matching approach[END_REF], optical-flow [START_REF] Ahn | Robust myocardial motion tracking for echocardiography: Variational framework integrating local-to-global deformation[END_REF] and non-rigid registration [Vercauteren, 2008;Chakraborty, 2016], usually demand heavy computations. Recent work using deep learning models for echocardiography motion estimation have achieved good trade-off between run time and accuracy [Ahn, 2020;Ta, 2020]. Unsupervised motion estimation relies solely on image pairs (or annotated masks) for training [Ahn, 2020;Ta, 2020;Balakrishnan, 2019;[START_REF] Wang | [END_REF]. In contrast, supervised motion estimation employs dense displacement fields from ground truth data in order to penalize predicted deformation fields [Østvik, 2021]. However, obtaining accurate ground-truth myocardial displacement is challenging, often necessitating the use of synthetic echocardiographic images with known motion [Alessandrini, 2018;Evain, 2022]. However, it remains unclear how synthetic data aligns with in vivo data due to potential domain shifts [Deng, 2022].

Automatic electrocardiography analysis

For automatic ECG analysis, the application of deep learning methods have shown great effectiveness in various tasks, such as beat and rhythm detection [START_REF] Teplitzky | [END_REF], ECG delineation of P, QRS, T segments [Jimenez-Perez, 2021], CVD diagnosis [Jahmunah, 2021;Dai, 2021;Li, 2023], etc. The most common architecture for ECG analysis is the convolutional neural network (CNN) due to its great non-linear modelling potential and recurrent neural network (RNN) owing to its strong processing ability for time series. 12lead ECG signals provide more information about electrical activities, while researchers are also exploring using reduced lead signals, such as single lead, 2 leads etc, for more general and portable application scenarios [START_REF] Nejedly | Classification of ECG using ensemble of Residual CNNs with or without attention mechanism[END_REF]Xu, 2022;Yu, 2022].

Multi-modal learning

Multi-modal learning, which includes multiple modalities such as image, text, audio etc, have shown better performance than using only one modality in different aspects: representation, fusion, co-learning and so on [START_REF] Rahate | Multimodal co-learning: Challenges, applications with datasets, recent advances and future directions[END_REF]. In the field of biomedical data mining, it has also shown great potential [Stahlschmidt, 2022]. For example, some researchers [START_REF] Soto | [END_REF][START_REF] Goto | [END_REF] fused features from echocardiography and electrocardiogram to distinguish hypertrophic cardiomyopathy patients from individuals with similar pathology. They have shown better diagnostic performance using two modalities than using only one. [Puyol-Antón, 2022b] searched for a shared representation space between cardiac magnetic resonance (CMR) and 2D echocardiography, improving the cardiac resynchronisation therapy (CRT) response prediction with joint modalities. In the first chapter, we have presented the thesis background and motivation, as well as clinical and methodological context of this thesis. A more precise literature review will be presented afterwards, in the corresponding chapters.

Manuscript Organisation and Contributions

Since two modalities are involved in this thesis, we design a road map for this objective (Figure 1.8). First, we treat echocardiography (Chapter 2, 3, 4) and electrocardiogram (Chapter 5) separately. Then, in Chapter 6, we explore multi-modal learning combining the two modalities together.

In Chapter 2, we describe three explorations for shape-aware segmentation in 2D echocardiography. In particular, we propose three models that incorporate shape information from global level (landmarks), regional level (AHA segments) and pixel level (structure contours). To improve generalisation, a stack of augmentation strategies tailored for ultrasound images is introduced. By conducting a thorough evaluation on different datasets, we demonstrate that with a pixel-level contour regularisation, we are capable of using a simple U-Net for robust segmentation with reduced anatomy outliers.

In Chapter 3, we focus on weakly-supervised motion tracking using real echocardiography sequences. Inspired from previous regional methods, we propose a novel motion estimation framework: the polyaffine motion model (PAM), which parameterise the dense deformation field by only 60 parameters. Compared to the state-of-art models, our proposed PAM model not only registers image frames with very good accuracy, but also demonstrates more regularity in terms of Jacobian determinant. By applying the trained model on unseen datasets, we obtain comparable results with models trained directly on the same dataset, supporting the conclusion that our PAM model is robust and has strong potential for clinical applications.

In addition, prior to this work, we have designed three neural network structures composing multi-layer perceptrons (MLP) and transformers for patch-based image-toimage registration. The improved regularity of displacement field has inspired us with respect to the potential of regional parameterisation, which has led to the work presented in Chapter 3. As the prior one was a collateral work with a colleague (Zihao Wang), we present the development of this work in Appendix A to keep the integrity of the main thesis.

In Chapter 4, a pipeline for robust echocardiograhy analysis is described. By using global indexes extracted from segmentation and motion tracking results, we achieve generalisable classification of myocardial infarction patients (MI) and non-MIs. This chapter serves as a conclusion for automatic echocardiography analysis.

In Chapter 5, we propose an unsupervised decomposition framework to analyze the morphology of a single heartbeat electrocardiogram. The decomposition network estimates the parameters of underlying subwaves, which provides explainable information of electrical events. Once again, the estimated parameters could serve for generalisable detection of MI patients using 12-lead ECG signals.

In Chapter 6, we adapt an uncertainty-based fusion strategy for multi-modal decision fusion using single modality predictions from echocardiography and electrocardiogram.

Compared with conventional late fusion methods, uncertainty fusion leverages on the most trustworthy modality by measuring the single modality uncertainty, thus improving multi-modal performance by a margin of 7% compared with the single modality classification.

Finally, in Chapter 7, we summarize the main contributions of this thesis and discuss potential directions for future work.

Publications

The contributions during this thesis have resulted in the following peer-reviewed publications:

Journal articles

• [Yang, 2023a] Abstract Automatic segmentation of cardiac structures in 2D echocardiography plays an important step for downstream analysis, such as volume quantification of the left ventricle, ejection fraction calculation etc. However, due to the noisy appearance of ultrasound images, it's not trivial to obtain robust segmentation results with the right anatomy.

In this chapter, we explore how to introduce shape constraints from global, regional and pixel level into a baseline U-Net model, for better segmentation and landmark tracking. Our experiments show that all three propositions perform similarly as a baseline model in terms of geometrical scores, while our pixel-level model (which uses a multi-class contour loss) reduces segmentation outliers and improves the tracking accuracy of 3 landmarks used for GLS computation. With appropriate augmentation techniques, our models also show a good generalisation performance when testing on a larger unseen cohort. This chapter was published in the Proceedings of the 11th biennial International Conference on Functional Imaging and Modeling of the Heart (FIMH) [START_REF] Yang | Shape constraints in deep learning for robust 2D echocardiography analysis[END_REF]. We include extensive evaluation results that will be published in one journal submission [Yang, 2023a] in Appendix section of this chapter.

The main contributions of this chapter are summarized as follows:

• We propose three strategies for anatomy-aware segmentation using deep learning (Section 2.2).

Introduction

Echocardiography, a non-invasive and cost-efficient imaging technique, is widely used by cardiologists to evaluate the cardiac function. Segmentation and motion tracking are two essential tasks that can help cardiologists in clinical decision-making. Segmentation offers important information regarding the shape and volume, while motion tracking provides knowledge on myocardial deformation and function.

Methods leveraging deep learning have consistently demonstrated exceptional performance in the fields of medical segmentation and registration. As for segmentation, the U-Net architecture has proved its overwhelming power when used in large cohort echocardiography segmentation [Leclerc, 2019a]. With appropriate adaptation of U-Net model and data augmentation, the U-Net architecture also demonstrated good generalisation ability in segmenting cardiac magnetic resonance images (CMRI) [Chen, 2020].

To tackle the incorrect anatomy problem in cardiac segmentation, researchers have proposed regularization techniques and refinement models [Oktay, 2017;Leclerc, 2019b;Clough, 2020]. To incorporate temporal information into the segmentation process, Wei et al. [Wei, 2020] proposed two co-learning strategies of parallel segmentation and motion estimation from 2D echocardiography. Painchaud et al. [Painchaud, 2022] introduced a post-processing auto-encoder that corrects the temporal inconsistency of sequence segmentation outputs. Nonetheless, the segmentation of echocardiography remains challenging due to issues like out-of-view structures and suboptimal signal-to-noise ratios, which pose significant hurdles, particularly in the context of myocardium segmentation.

In the field of cardiac motion tracking, unsupervised deep learning are very popular and these schemes reach similar performance or even outperform traditional registration methods. Krebs et al. proposed a conditional variational autoencoder which learned a diffeomorphic transformation from pairwise CMRI in an unsupervised way [Krebs, 2019]. Shawn et al. [Ahn, 2020] designed a U-Net like network for unsupervised pairwise echocardiography motion tracking. However, the displacement field can be unrealistic without a relevant regularisation.

Multiple research investigations have consistently demonstrated that global longitudinal strain (GLS) exhibits higher sensitivity as a metric for assessing systolic function when compared to left ventricular ejection fraction (LVEF). These findings suggest the potential utility of GLS in the clinical identification of left ventricular dysfunction, as reported in studies such as [START_REF] Kraigher-Krainer | [END_REF][START_REF] Hasselberg | [END_REF]. GLS can be approximated through the measurement of left ventricle length change, as indicated in the work by [START_REF] Støylen | Left ventricular global strains by linear measurements in three dimensions: Interrelations and relations to age, gender and body size in the HUNT Study[END_REF]. Hence, there might be no imperative need to estimate a dense displacement.

U-Net like deep learning models depend largely on pixel-level classification, which can generate artefacts which are irregular for the organ shape. Researchers are seeking to combine shape constraints with deep learning methods [Bohlender, 2021]. Having the same intention to improve the segmentation consistency with anatomical shapes in 2D echocardiography, in our work, we explore the introduction of shape constraints from global, regional and pixel level into a baseline U-Net model. From the segmentation results, useful information such as EF and landmark based GLS can be extracted. The detailed model architecture will be explained in Section 2.2. We then present the implementation and experiment results for the segmentation and landmark detection steps in Section 2.3.

Methods

We used a U-Net model as our baseline model. Its encoder consisted of 5 down-sampling (MaxPool + Conv) blocks with ReLU activation after the 3x3 convolution. The corresponding decoder had 5 up-sampling (UpSample + Conv) blocks and is skip-connected with the encoder. Based on this model, we considered to incorporate shape constraints from three levels:

• Global-level: estimate a triangle like landmark map in parallel with segmentation (SEG-LM)

• Regional-level: add a poly-affine myocardium reconstruction network to constrain the shape of myocardium mask (SEG-AFFINE)

• Pixel-level: use a multi-class contour-loss to finely classify the boundary pixel (SEG-CONTOUR)

The three methods will be explained in detail in the following subsections.

SEG-LM: Parallel segmentation and landmark detection

We adapted the baseline U-Net model for simultaneous segmentation and landmark prediction by adding a separate branch of decoder for landmark map estimation. The two decoders processed the encoder information in parallel. The final layer of segmentation branch and landmark detection were followed by SoftMax activation and Sigmoid activation, respectively. In particular, we considered the two end points of mitral valve (basal points) and the apex along the endocardial contour. The basal points were identified as the two end-points of adjacent boundary of both left ventricle and left atrium. The endo-apex point was then calculated as the furthest point to the mid-basal point along the endocardium. The output of landmark detection network was a heatmap of the corresponding target point. From the output heatmap, we extracted the landmark position by either finding the location of maximum or computing the centroid.

As we had different labels in the ground-truth data (myocardium, blood pool, atrium), a multi-class dice L dice was used as the segmentation loss. As for landmark detection, we first penalised on the squared error of landmark heat-map (L2 loss: L l2 ). In order to avoid landmark overlapping on different output layers, we regularised the centre distance loss L CD of different landmark heat-maps as proposed in [Wang, 2019]:

L CD = 1 2 3 i=1,j̸ =i 1/(C(H i ) -C(H j )) 2 (2.1)
with C the operation to obtain the centre position and H i the landmark heat-map. Finally, a mean squared distance loss between the predicted heatmap centre and the ground truth point L point was applied. Thus, the total loss L total for optimisation was given by:

L total = L dice + αL l2 + βL CD + γL point (2.2)

SEG-AFFINE: Poly-affine Regulariser for Myocardium

With the intention to constrain the regularity of predicted myocardium mask, we proposed to model the myocardium mask as a combination of 6 AHA regions [START_REF] Cerqueira | [END_REF].

We first chose a reference myocardium mask R from the training set. All of the N training myocardium masks (M i ) N i=1 were aligned to the reference mask by an affine transform (T i ) N i=1 estimated from the three landmarks (left basal, right basal and endo-apex). Then all aligned masks were averaged to a mean mask R. The mean mask R is threshold-ed ( Rf ) and splitted into 6 AHA regions ( Īj ) 6 j=1 . For every myocardium mask M i , we aimed to first find 1 affine matrix A g that globally transform the reference mask to Mi g . The corresponding reference regions became ( Īg j ) 6 j=1 = A g Īj . We then found 6 affine matrices (A ij ) We used a CNN to estimate the affine parameters and reconstruct the given mask. The proposed network had two sub-networks. The first one aimed to estimate global affine parameters for global alignment which consisted of two hidden convolutional layers with down-sampling. The second sub-network was designed find the 6 regional affine matrix.

It started with an encoder for high level feature extraction. The extracted features were passed through fully connected layers for affine matrix estimation Âij . By using the affine transform, we could reconstruct M 0 from the 6 mean regions M 0 = 6 j=1 Âij Ĩwg j . Two Conv layers were followed to refine the fusion mask M 0 , and thus we obtained the final output M f (detailed information in fig. 3). In order to regularise the value of the affine parameters, we approached the affine parameter estimation problem using Maximum A Posteriori (MAP) with prior probabilities on the parameter values P (A). The MAP aims to optimise: arg max[P (A|M )] ∝ P (M |A)P (A) (i.e. arg min[-log P (M |A) -log P (A)]). We used Gaussian distribution for conditional likelihood and priors. For

P (M |A) ∝ exp(-1 2 (M -M (A)) T Σ -1 (M -M (A))
), the variance is identity. For P (A) ∼ N (μ, Σ), μ and Σ are the maximum likelihood estimate (here we only consider diagonal covariance matrices) from the aligned transformation parameters (T i ) N i=1 . The regularisation for affine parameters was L af f ine = αL l2 + βL prior , where L l2 was the mean square error between the reconstructed image and the input and

L prior = K k=1 6 j=1 δ j (A k j -μj ) 2 Σjj (2.3)
where K = 1 for global affine parameter and K = 6 for regional affine parameters.

Thus, the total loss for the poly-affine reconstruction network was: (2.4) Once the poly-affine regulariser network was trained, the 6+36 affine parameters served as an explicit hidden vector to regularise the shape of myocardium prediction. We trained a U-Net model (same as baseline model) which seeks for the best overlapping of mask as well as the minimum distance between the corresponding affine parameters of predicted myocardium and that of ground truth myocardium. The loss function for this method was loss = dice + αM SE(P A(P ) -P A(M )) (2.5) where MSE represented the mean squared error, P A presented a poly-affine regulariser that outputs the 42 affine parameters.

L total = L dice ( M g ) + β g L g prior global sub-net + L dice ( M 0 ) + L dice ( M f ) + αL l2 + β r L r prior regional sub-net

SEG-CONTOUR: multi-class contour-loss

In order to increase the classification accuracy on the boundary, we chose to use an adapted multi-class contour loss [START_REF] Jia | [END_REF]. Firstly, a distance map D(M ) was calculated from ground truth mask and it illustrated the shortest euclidean distance of each pixel to the closest border. Then the contour loss was calculated as

loss contour = (D(M ) • contour(B(P ))) (2.6)
where • performed element-wise multiplication. P represented the prediction output after SoftMax activation of U-Net for a certain class. B(P ) represented a differentiable thresholded Sigmoid for binarisation

B(P ) = 1 1 + exp -γ(P -T ) (2.7)
where γ = 20 and T = 0.5.

The contour of the binarised mask was obtained by applying a 2D Sobel filter

contour(P ) = |G x * P | + |G y * P | (2.8)
where * denotesd 2D convolution and G x , G y were 2D Sobel kernel in x-,y-dimension:

G x =        1 0 -1 2 0 -2 1 0 -1        , G y =        1 2 1 0 0 0 -1 -2 -1        .

Experiments and Results

Datasets

In this work, we worked on two public data sets: CAMUS1 and ECHONET2 . CAMUS dataset consists of publicly accessible 2D echocardiographies and the corresponding annotations of 450 patients. For each patient, 2D apical 4-chambers (A4C) and 2chambers (A2C) view sequences are available. Manual annotation of cardiac structures (left endocardium, left epicardium and left atrium) were acquired by expert cardiologists for each patient in each view, at end-diastole (ED) and end-systole (ES) [Leclerc, 2019a].

Along with the image and annotation data, the following information are also provided: image quality (good/medium/poor), left ventricle end-diastole volume (LVedv), left ventricle end-systole volume (LVesv) and left ventricle ejection fraction(LVEF). c-d) GLS calculation illustration (background is one echo image from CAMUS). We calculated the GLS from the LV length change by following the approximation method in [START_REF] Støylen | Left ventricular global strains by linear measurements in three dimensions: Interrelations and relations to age, gender and body size in the HUNT Study[END_REF].

ECHONET dataset contains 10 030 apical 4-chambers echocardiography videos as part of routine clinical care at Stanford University Hospital [Ouyang, 2020]. Segmentation measurements (left endocardium) at end-diastole and end-systole are available for all videos. The corresponding LVedv, LVesv, LVEF are also provided for each video.

Experiments

We trained the baseline UNet, SEG-LM, SEG-AFFINE, SEG-CONTOUR models on the CAMUS dataset (both 2-chamber and 4 chamber ED/ES frames) using 10-fold cross validation. The 450 patients were randomly splitted into 10 folds. Each fold had a similar distribution of image quality and LVEF distribution. For every turn we used: 8 fold data for training; 1 fold for validation (for model selection); and, 1 fold for testing.

Data preprocessing

From the segmentation result, we first found the largest connected component for each class and applied a closing operation to fill the potential hole that could exist inside the predicted mask. We then extracted the basal and apex landmark points following protocol described in Section 2.2 for the four models except SEG-LM whose landmark is extracted from the landmark branch. The direction of mid-basal to apex was regarded as the grand axe of LV and used for LV volume estimation by using a modified Simpson's rule [Folland, 1979], and for the calculation of ejection fraction. The distance from midbasal to apex formed the ventricle length and served for GLS calculation [START_REF] Støylen | Left ventricular global strains by linear measurements in three dimensions: Interrelations and relations to age, gender and body size in the HUNT Study[END_REF].

The dataset didn't contain all the 3 apical views for the left ventricle; therefore, the GLS estimation was not be very accurate. However, we still computed the GLS from 2 views (CAMUS) and 1 view (ECHONET) as reference, in order to test the landmark detection accuracy.

Data augmentation

In order to avoid over-fitting and to improve generalisation performance, we applied random data augmentation at training phase for all the networks. Specifically, the implemented augmentation consisted of the following related changes designed for echocardiography (adapted from [Zhang, 2020]):

• Rotation: Rotation is usually caused by different probe orientation and we set the rotation range to [-15,15] degrees.

• Crop and Scaling: Scaling is due to the heart shape variability and myocardium motion. We set the scaling factor to [0.7,1.3] and then cropped (or pad) the resized image to be the same size.

• Brightness adjustment: Image intensity characteristic often varies among device vendors and depends on each image system settings. For brightness adjustment, we shifted the image intensity histogram by a factor of [0.9,1.1].

• Contrast adjustment: Contrast adjustment is a popular tool to improve the visualisation of cardiac structures. We applied a Sigmoid correction

I new = 1 1+exp C * (S-I)
on the input image, with a random range of cutoff S in [0.4,0.6] and a constant multiplier C from [4,10].

• Sharpness adjustment: To modify the sharpness of the image, we applied an unsharp mask

I new = I + A * (I -I σ )
, where A ranged from [1,2] and the σ of the Gaussian filter from [0.25,1.5].

• Blurriness adjuestment: For image blurring, we applied a Gaussian filter with σ in (0.25,1.5).

• Speckle noise: Due to the acoustic interference phenomenon, speckle is the main type of noise present in echocardiography. Speckle is usually modelled as a Rayleigh multiplicative noise. Since our training data is already log-compressed, we added speckle noise following an additive fashion The augmentation was applied in the following order: rotation; crop (pad) and scaling; brightness; contrast; sharpening; blurring; and, speckle noise. For each step, the application of the corresponding change was controlled by a Binominal random variable with ρ = 0.5.

I new = I + √ I * G σ ,

Implementation

All segmentation models were implemented with Pytorch, with a batch size of 8 and input image resized to 256 × 256. CAMUS dataset does not contain the ground truth of desired landmarks. Thus, we generated the 'ground truth' landmark positions from ground truth segmentation masks. The Gaussian heatmap of ground-truth landmarks was computed with σ = 4.

• Baseline model: The baseline U-Net was trained with a multi-class dice loss. An Adam optimiser was applied with a learning rate of 1 e-3 . The training was early stopped when the dice loss on validation data demonstrated no increase for more than 5 epochs.

• SEG-LM model: For SEG-LM model, we set α = 0.05, β = 0.5, γ = 0.5 in Equation 2.

2. An Adam optimiser was applied (lr= 10 -4 ). The output heatmap of landmarks was first processed to keep only one point cluster per layer and then the landmark location was extracted as the centroid.

• SEG-AFFINE model: The poly-affine regulariser network was first trained with the myocardium ground truth for reconstruction. We set α = 0.005, β r = β g = 0.01, σ 6 i=1 = 1 (Equation 2.4) and learning rate at 1e-4. We then trained a baseline U-Net with the polyaffine regulariser using loss function (Equation 2.5) with α = 10. The parameter for global prior was calculated from the training-specific (T i ) N i=1 . The parameter for regional prior was set as µ 1...6 = [1, 0, 0, 0, 1, 0] and [Σ 6i=1 ] = 0.1.

• SEG-CONTOUR model: As for SEG-CONTOUR model, contour loss was optimised along with the dice loss. The contour loss was easy to fall into a local minimum of 0, thus we set a weight of 100, 1e-4 for dice loss and contour loss respectively.

Evaluation metrics

For segmentation results, apart from the most used geometrical metrics: Dice coefficient, Hausdorff distance (HD) and Mean Surface Distance (MSD), we also used two anatomical metrics: Convexity(Cx) and Simplicity(Sp) [Leclerc, 2019b].

Convexity(Cx) = Area(P ) Area(ConvexHull(P ) Simplicity(Sp) = 4π * Area(P ) P erimeter(P ) (2.9) (2.10)
Based on these metrics, we calculated the number of outliers for algorithm/model robustness evaluation. The outlier of segmentation prediction for CAMUS dataset was established from the inter-variability tests with the upper limit values for HD and MSD, and lower limit values for the simplicity and convexity [Leclerc, 2019b]. A prediction mask was considered as a geometrical outlier if its HD > 3.5mm or M SD > 8.2mm at ED, if HD > 4mm or M SD > 8.8mm at ES. The corresponding limit for anatomical outlier was as follows: if Cx < 0.529 or Sp < 0.741 for endocardium, if Cx < 0.694 or Sp < 0.960 for epicardium [Leclerc, 2019b]. baseline model, the SEG-LM, SEG-AFFINE models demonstrated only a slight decrease in terms of segmentation metric, ejection fraction (EF) prediction accuracy, and landmark detection accuracy. The SEG-CONTOUR model shows a similar performance with baseline model in Dice score (Table 2.1), but reduces greatly the number of geometrical and anatomical outliers (Table 2.1). It is reasonable that the SEG-CONTOUR reduced the classification error along the boundary area thus less outlier predictions were observed. This is consistent with the observations that a good Dice score does not always guarantee a good HD [Bernard, 2018], and, in our case, not always leads to anatomically-plausible segmentation. The SEG-CONTOUR model was also capable of tracking more precisely the boundary especially the landmarks (Table 2.2); thus resulting in a smaller bias of GLS prediction. In terms of EF calculation, the baseline U-Net model shows a smaller mean absolute error but a larger bias than the SEG-CONTOUR loss.

Results

Method

We show three CAMUS test examples (see Figure 2.5), where each row has a good/medium/bad performance in terms of HD score. Comparing with the other 3 models, SEG-CONTOUR has a more fluent border and similar to the ground truth annotation. The evaluation results of applying the trained model on a totally different dataset ECHONET (the same test fold of 1277 patients as in [Ouyang, 2020]), showed the same trend of performance for all our four methods. The SEG-CONTOUR method demonstrated a good performance on tasks related to boundary information, for example, lower HD and MSD, better GLS estimation. It is less accurate on area based task (i.e., volume estimation thus ejection fraction prediction). It is noticeable that all of the four models demonstrated acceptable values for Dice coefficient on this different dataset (the Dice coefficient of models trained on ECHONET data was 0.92% [Ouyang, 2020]), which proves the importance of appropriate image augmentation techniques.

Conclusion

In this chapter, we explored methods to introduce shape constraints into 2D Echocardiography segmentation models from three levels: global-level (SEG-LM), regional-level (SEG-AFFINE), pixel-level (SEG-CONTOUR). From the evaluation results on CAMUS dataset and its generalisation result on a unseen dataset ECHONET, we suggest that it is more efficient to introduce pixel-level shape constraint than global or regional level constraints for U-Net based models. With a multi-class contour loss, SEG-CONTOUR model achieved better classification on the boundary pixels with a reduced a Hausdorff distance and more accurate landmark detection result. Overall, our experiments showed the great potential of SEG-CONTOUR for robust segmentation and deformation analysis. We further evaluated the trained SEG-CONTOUR model on the unseen test split of CAMUS dataset and compared the performance with other state-of-art segmentation methods using the same training set. Our proposed method achieved comparable performance with the best performing method either on cardiac structure segmentation or segmentation-based EF estimation. In terms of correct anatomy, comparing with the previous work of [Leclerc, 2019b] which added a refinement network after the U-Net work to improve the shape regularity of predicted LV structures, our U-Net model with a contour prior (SEG-Contour) can directly generate less geometrical and anatomical outliers for the same dataset.

Evaluation on local private dataset

A second external validation dataset contains 2D echocardiography sequences of 76 patients collected from Nice CHU hospital approved by the local ethics guidelines. The final diagnostic as well as EF were given by an expert cardiologist. This dataset served as an evaluation set for real-world monitoring. Since no segmentation ground truth was available for our private dataset, we compared the EF estimated from segmentation model with the value provided by one expert cardiologist. We achieved a MAE of 8.3% and correlation coefficient of 0.74.

Discussion

From segmentation prediction, the most important cardiac index we can extract is EF. The three datasets involved in the segmentation evaluation phase demonstrate different annotation schemes, thus influencing the estimation of EF. For example, CAMUS dataset provided a detailed annotation of LV for ED/ES. EchoNet-Dynamic only provided endocardial traces that were used for volume estimation. The ground truth EF of our private data was obtained by one expert cardiologist's visual estimation. Despite of the diverse discrepancy between different datasets, our model still predict very satisfactory EF results, within the inter-observer's variability (MAE: 10%, Corr. 0.8) reported in [Leclerc, 2019a].

Abstract

The second important task in echocardiography analysis is to track the movement of left ventricle (LV), in order to reveal the motion pattern of the left ventricle wall. However, motion estimation in echocardiography is a challenging task since the ultrasound images suffer largely from low signal-to-noise ratio and out-ofview problems. Current deep learning-based models for cardiac motion estimation in the literature estimate the dense motion field with spatial regularization. However, the underlying spatial regularization can only cover a very small region in the neighboring areas, which is not enough to derive a smooth and realistic motion field for the myocardium in echocardiography.

In this chapter, we introduce a novel framework for cardiac motion tracking. Our method applies polyaffine transformation for motion estimation, which intrinsically regularizes the myocardium motion to be smooth. In order to constrain the motion model to focus on the LV wall, we propose a motion prior that can provide guidance for locating the myocardium. The main advantage of our model is that it provides a compact, smooth and realistic parameterisation of the LV deformation. This chapter is based on our preliminary results published in the Proceedings of the 12th biennial International Conference on Functional Imaging and Modeling of the Heart (FIMH) [Yang, 2023b] and its further improvement included in a journal submission [Yang, 2023a].

Our main contributions in this chapter include:

• the development of a novel polyaffine motion model (PAM) for unsupervised cardiac motion estimation (Section 3.2);

• the design of a comprehensive pipeline for using PAM in a weakly-supervised manner on a large public echocardiography dataset (Section 3.3);

• comprehensive experiments on various real-world datasets, showing that PAM outperforms other popular deep learning-based motion estimation methods and thereby highlighting its potential for clinical applications (Section 3.4).

Introduction

Echocardiography is one of the most widely used modalities for cardiac dysfunction diagnosis. It is a non-invasive radiation-free and real-time imaging modality, very suitable for portable analysis, such as myocardial motion evaluation. However, ultrasound images generally have poorer quality than other cardiac imaging modalities (e.g. MRI and CT), due to their low signal-to-noise ratio. Additionally, limitations in acquisitions and patient variability may cause the myocardium to occasionally be outside of the imaging field-ofview. These critical issues make myocardial motion estimation in echocardiography a particularly challenging task.

Traditional methods for motion estimation in echocardiography, such as block matching [START_REF] Azarmehr | [END_REF]Alessandrini, 2016] or optical flow [Farneb, 2003;Barbosa, 2013;Alessandrini, 2016], are typically time-consuming and not suitable for real-time or portable analysis. However, recent advances in deep learning (DL) have improved both the time efficiency and tracking performance of motion estimation algorithms. DLbased models can generally be classified into two types: optical flow-based models and registration-based models, repsectively. Optical flow-based DL models estimate dense displacement fields between consecutive image frames and typically require ground truth displacement for supervised training [START_REF] Østvik | Myocardial function imaging in echocardiography using deep learning[END_REF]Evain, 2022]. Since obtaining ground truth displacement from real-world echocardiography images can be difficult, synthetic echocardiography datasets are often used for supervised training of these models [Alessandrini, 2018;Evain, 2022]. Registration-based DL models typically register all other frames to end-diastole either in an unsupervised manner or with weak supervision using segmentation masks, enabling them to work on real-world datasets [Ahn, 2020;Ta, 2020]. U-net-like architectures are often used as the core design of such methods [Ahn, 2020;Ta, 2020]. Many other studies focused on cardiac MRI motion estimation have also utilized the registration approach with different temporal smoothness strategies [Qin, 2018;Krebs, 2020]. Recently, researchers have also incorporated bio-mechanical modeling knowledge into deep learning networks with the aim of improving the generalizability of motion estimation performance [START_REF] Qin | [END_REF][START_REF] Zhang | [END_REF].

The motion of the myocardium is not arbitrary, and various spatial regularization techniques have been explored in the literature, such as: the divergence penalty [Mansi, 2011] for enforcing incompressibility; the rigidity penalty for smoothness [START_REF] Staring | A rigidity penalty term for nonrigid registration[END_REF]; and, the elastic strain energy for mechanical correctness [START_REF] Papademetris | Estimation of 3D left ventricular deformation from echocardiography[END_REF], among others. Many deep learning models have incorporated these regularization techniques into their work [Ahn, 2020;De Vos, 2019;[START_REF] Zhang | [END_REF]. However, these regularization techniques are usually based on first-or second-order derivatives of the displacement vector field, which are applied at the pixel level in discrete implementation and are insufficient for studying the myocardium in echocardiography. The high noise-to-signal ratio in echocardiography degrades the quality of the estimated motion field, despite the pixel-wise constraints in deep learning methods. Lastly, other methods tackle the smoothness issue at the transformation level, for instance using regionally affine deformations [McLeod, 2015].

In this work, we introduced a deep learning-based motion model leveraging such approach and tailored for 2D echocardiography motion estimation, called the PolyAffine Motion model (PAM). In particular, based on an encoder-decoder backbone [START_REF] Siarohin | First order motion model for image animation[END_REF], we proposed the following modelling and regularisation strategies aiming to improve motion estimation in a global manner.

• Smoothness ++ : We parameterised the deformation field as polyaffine transformation, which intrinsically incorporated global-level smoothness regularization.

• Right Position ++ : We designed a motion prior and integrate it into loss functions, which helped the model to concentrate on myocardium region.

• Incompressiblity ++ : We proposed an incompressiblity loss term by regularisation through regional transformation, which helped us reduce large unrealistic volume change.

Methodology

Polyaffine motion model

Given a source image I S and target image I T , we aimed to estimate a dense motion field T S←T : R 2 → R 2 from target image I T to source image I S such that:

I S (T S←T (•)) = I T (•). (3.1)
Inspired by the polyaffine motion fusion framework proposed by Arsigny et al. [START_REF] Arsigny | A fast and log-euclidean polyaffine framework for locally linear registration[END_REF], we adapted the motion estimation module from [START_REF] Siarohin | First order motion model for image animation[END_REF] to develop our proposed method for cardiac motion estimation in echocardiography.

The Polyaffine motion model consisted of two steps. First, the motion of the left ventricle was approximated through the sparse motion of several key points. An encoder-decoder network was then used to output the location of key points along with their local affine mapping for both I S and I T separately. Second, from the sparse motion, we obtained the final dense motion field through polyaffine motion fusion. The proposed method is illustrated in Figure 3.1.

Key point and affine transformation estimation

We adopted the encoder-decoder architecture for key point extraction as presented in [START_REF] Siarohin | First order motion model for image animation[END_REF] and provided a brief review of the method from the point of view of affine transformation. In order to process each image separately, we assumed that there exists an abstract reference frame R. Given an image X, the encoder-decoder network generated the estimated key points p k X , k = 1, 2, ...K as well as the corresponding linear mappings A k X←R ∈ R 2×2 , k = 1, 2, ...K. The local affine transformation from target image I T to source image I S was then computed using the following equation:

T k S←T (z) = Āk S←T • z Linear mapping + (p k S -Āk S←T • p k T ) Translation , ( 3.2)
where z ∈ R 2 represents the coordinate in the target image, and

Āk S←T = A k S←R (A k T ←R ) -1 .
In order to capture motion around the myocardium, we introduced a myocardium-related key-point prior (Section 3.3.2) and an associated loss functions (Section 3.2.2.1). This encouraged the network to output key points close to the myocardium, in contrast to [START_REF] Siarohin | First order motion model for image animation[END_REF], which employed a self-supervised approach for learning key-point positions.

Polyaffine motion fusion

Once we obtained the local affine transformation, the dense motion field was computed through direct polyaffine motion fusion. For each local affine motion, a spatial weight W k (p k T , σ 2 ) controlled its influenced region; we achieved this by applying a 2D isotropic Gaussian distribution centered at key point p k T with variance σ 2 . Here, W 0 represented the weight for the background region and the left ventricle cavity area, and was computed as follows:

W 0 = RELU(1 - K k=1 W k (p k T , σ 2 )). (3.3)
We normalised all weights into Wk =

W k K k=0 W k
, where k = 0, 1, 2...K. An example is shown in Figure 3.1. Next, we applied these to compute the polyaffine dense motion

T S←T (z) = W0 z + K k=1 Wk (p k T )T k S←T (z).
(3.4)

Methodology

The original first-order motion model (FOMM) [START_REF] Siarohin | First order motion model for image animation[END_REF] utilised a second encoderdecoder network to estimate the normalised weights for each local affine transformation, which can not guarantee the center of weights close to the corresponding key point, thereby being unstable for myocardial motion estimation.

Sequence motion estimation

Considering a sequence of image frames I 1 , I 2 , ...I N , the assumption of abstract reference frame enabled a fast way to obtain the dense motion field between any arbitrary pair of frames in the sequence. Without loss of generality, we assumed that I ED was the frame at end-diastole and I ES was the frame at end-systole, and that I ED was ahead of I ES in time. The local affine transformation from I ED to I ES could be calculated by composition:

T k ES←ED (z) = T k ED+1←ED • T k ED+2←ED+1 • • • T k ES←ES-1 = Āk ES←ED • z + (p k ES -Āk ES←ED • p k ED ), (3.5)
where

Āk ES←ED = A k ES←R (A k ED←R ) -1
. The final dense motion was computed by combining the local motion, as described in Equation 3.4.

Loss functions

The model was trained end-to-end using a combination of loss functions, which can be grouped into the following subsets for sequence motion estimation.

Keypoint myocardium losses

We proposed two loss functions to enforce the position of key points near the myocardium region. The first loss, denoted by L kp_prior , penalized the L 2 norm between the estimated key-point position and a prior position. We obtained the prior key-point position from the available training masks, as described in Section 3.3.2. The second loss, denoted by L kp_myo_ED , constrained key points at end-diastole (ED) to reside within the myocardium region by penalizing the distance between each key point and the myocardium.

L kp_myo_ED = - K k=1 H(p k , σ 2 H ) * (Mask myo_ED -0.5), (3.6)
where H(p k , σ 2 H ) was the isotropic Gaussian heatmap centered at p k with variance σ 2

H

and Mask myo_ED the binary mask of myocardium at ED.

Keypoint equivalence losses

Another two losses L equi_kp and L equi_affine imposed an equivalence constraint to the detected key points [START_REF] Siarohin | First order motion model for image animation[END_REF]. These forced the model to predict consistent key points and local linear mappings when applying a known transformation to the input image.

Registration losses

The last four losses regularized the final dense motion field through image similarity and key-point similarity. First, for each pair of consecutive frames in the sequence, we constrained

L seq_im = seq N CC(I T , I S (T I S ←I T )), L seq_kp = seq 1 K K k=1 |H(p k T , σ 2 H ) -H(p k S , σ 2 H )(T I S ←I T )|, (3.7) (3.8)
where N CC represented the normalised cross correlation. Second, in order to force the model to learn temporal motion, we applied a second pair of similarity loss between the image at end-diastole and all frames after the end-diastole frame, denoted as L ED2any_im and L ED2any_kp for image similarity and key-point similarity respectively.

Incompressibility penalisation

In order to constrain the compressibility of the myocardium within physiological ranges, we applied an incompressibility penalisation to the deformation field. Different from former work that applied divergence-free regularisation to the dense motion field [Mansi, 2011;Ahn, 2020], here we can efficiently constrain the Jacobian determinant of each local affine transformation to be close to 1:

L incomp = K k=1 (| det(A k X←R )| -1) 2 (3.9)
The total loss for the PAM model was

L total =λ 1 L seq_im + λ 2 L seq_kp + λ 3 L equi_kp + λ 4 L equi_affine + λ 5 L kp_prior + λ 6 L kp_myo + λ 7 L ED2any_im + λ 8 L ED2any_kp + λ 9 L incomp

Datasets

Three public datasets of echocardiography were included in our study.

• EchoNet1 [Ouyang, 2020] contained 10030 echocardiography videos of long-axis 4 chamber view. Left-ventricle tracings at end-diastole and end-systole were available (see Figure 3.2(a)).

• CAMUS dataset2 [Leclerc, 2019a] consisted of apical 4-chamber view videos from 450 patients whose left heart segmentation masks were publicly accessible (see Figure 3.2(e)).

• HMC-QU dataset3 [Degerli, 2021b] All image sequences were cropped around the LV center according to the provided segmentation/tracings (see Figure 3.2(b)). In this study, we followed the given data division of the EchoNet dataset, with 7465 samples for training, 1288 samples for validation and 1277 samples for testing. CAMUS and HMC-QU datasets were used for evaluation during test phase.

Dataset preprocessing 3.3.2.1 Pseudo myocardium mask

There was no ground truth of myocardium mask in EchoNet dataset. To provide guidance for key points, we generated a pseudo myocardium mask for end-diastole (ED) and endsystole (ES) by applying a dilation operation using 13x13 and 17x17 structure element to the left ventricle mask at ED and ES frame respectively. The difference between the dilated mask and the original one was regarded as the pseudo myocardium mask (see Figure 3.2(b)).

Key-point prior at end-diastole

A mean myocardium mask was computed using all the pseudo myocardium masks at ED from the training set. Then, all the pixels of the mean mask were clustered into 10 groups using the KMeans function from the scikit-learn package [Pedregosa, 2011], where the center of each group was considered as one key-point prior (see Figure 3.2(c)).

Implementation

We compared our proposed model with the conditional variational autoencoder (CVAE) method proposed in [Krebs, 2020], which demonstrated effectiveness in sequential motion modelling of cardiac images. Its former registration version [Krebs, 2019] had shown more regular motion field than other deep learning methods, including VoxelMorph [START_REF] Dalca | Unsupervised learning for fast probabilistic diffeomorphic registration[END_REF]. We implemented the CVAE method following its description in [Krebs, 2020] and used the same training hyper-parameters. However, we added a Dice loss between the transformed end-systolic (ES) mask and end-diastolic (ED) mask during CVAE training to keep it consistent with our PAM model, which used segmentation mask information during training (see Section 3.2.2.1). We also integrated a divergence-free term into the final loss function for incompressibility regularisation.

We trained the PAM model using an Adam optimizer with a learning rate of 1e -4. To determine the optimal hyperparameters, we conducted experiments on a randomly selected subset of 1000 training examples from the EchoNet dataset. The hyperparameters for the loss function were set to λ 1 = λ 2 = 100, λ 3 = λ 4 = 50, λ 5 = 20, λ 6 = 0.1, λ 7 = λ 8 = 100, λ 9 = 1 in Equation 3.10. The variance of the Gaussian heatmap was set to σ 2 = 0.05 for affine transformation weights (see Equation 3.3), and σ 2 H = 0.005 for all other scenarios. We trained the model for a maximum of 100 epochs, and applied early stopping if the validation loss did not improve over 10 epochs.

Alation study

The backbone FOMM model [START_REF] Siarohin | First order motion model for image animation[END_REF] optimized the first four terms of Equation 3.10. New strategies tailered for echocardiography proposed in this work are denoted as follows:

• +Prior: integration of prior-related losses (5th and 6th loss terms in Equation 3.10);

• +Polyaffine: replacing the encoder-decoder module for weight estimation by designed polyaffine weights;

• +ED: incorporation of registration losses between ED frame and others (7th and 8th terms in Equation.3.10); and,

• PAM(all): the model that optimized the total loss function of Equation 3.10.

Results

Registration accuracy

We first evaluated the motion estimation accuracy by assessing the registration performance using the available segmentation masks from all test datasets. For the EchoNet test split, we compared the ground truth of the LV mask for end-diastole (ED) with that transformed from end-systole (ES). For the CAMUS dataset, the same evaluation was applied to the myocardium mask for ED/ES. For the HMC-QU dataset, the myocardium masks of one cardiac cycle were all transformed to ED using the estimated motion field.

To enable group-level statistical analysis along the cardiac cycle, the temporal metric was interpolated to the same length.

EchoNet

We compared our proposed PAM model with a state-of-art model CVAE and our backbone model FOMM. The main difference between PAM and FOMM was our proposition of motion prior and explicit polyaffine motion fusion. From Table 3.1 we observed that the introduction of motion prior and polyaffine motion framework improved largely the performance of registration on EchoNet-Dynamic test split. We also achieved a performance comparable performance with that of the state-of-art CVAE model. [Wei, 2020] that was trained on CAMUS training set using 10-fold cross-validation.

HMC-QU

HMC-QU dataset contained 109 4-chamber view samples with myocardium annotated automatically for one cycle [Degerli, 2021a]. The proposed PAM model achieved satisfac-

Deformation regularity

Furthermore, our PAM model excelled in producing motion fields with enhanced regularity as evidenced by Jacobian Determinant closer to 1 and smoother volume change as evidenced by the gradient (see Figure 3.4). The smooth displacement field was advantageous for computing dense strain tensor, which was typically constructed using the first-order derivative of the displacement field. We show examples of Jacobian Determinant of motion field from myocardial infarction patient (MI) and non-MI patient in Figure 3.5.

Besides of the good tracking performance from different evaluation metrics, the proposed PAM model demonstrated good potential for abnormal wall motion detection. We show an example of myocardial infarction (MI) from HMC-QU dataset (Figure 3.6) to visualize how it works. Our prediction shows that the patient has a small radial strain in SEG14 region, which is in accordance with the fact that the patient is diagnosed with MI and in particular the SEG14 is affected.

Discussion and conclusion

In this chapter, we proposed a polyaffine motion model (PAM) for echocardiography motion estimation. The PAM model demonstrated excellent motion estimation performance on real-world echocardiography datasets and showed good generalization to unseen datasets from other centers. Our explicit design of fusion weights enabled efficient learning of local affine transformation, and the intrinsic polyaffine structure improved the smoothness of the motion field, showing potential for abnormal wall motion detection.

In the future, we will focus on integrating temporal regularization for the PAM model and conducting evaluations on synthetic datasets with known ground-truth displacement. Visual results of a myocardial infarction sequence from HMC-QU dataset where SEG14 is annotated as infarcted region.

Appendix

In this appendix, we demonstrate other advantages of using PAM for echocardiography analysis.

Cardiac motion transfer between sequences

One important application is to generate new synthetic datasets that could be used for data augmentation or representation learning and others. PAM model facilitates motion transfer from one sequence to another while preserving the appearance along the total sequence. This is different from other deep learning methods that deform one single frame (such as end-diastole frame) to obtain new sequences [Krebs, 2020].

Given two sequences {X i , i ∈ N } and {Y i , i ∈ N }, our objective was to transfer motion of {X i } to {Y i }. We denoted the new sequence of Y as G. The motion transfer process contained the following steps:

1. Motion estimation: By applying PAM model to input sequences, we obtained a sequence of key points {p k X i }, {p k Y i } and local affine transform from abstract reference frames {A k X i ←R }, {A k Y i ←R } respectively for {X i } and {Y i }.

Align ED and ES frames:

We aligned the key point sequences and local affine transform sequences temporally through linear interpolation such that ED and ES frames of sequences {X i } were aligned to

{Y i } (i.e. t Y ED = t X ED , t Y ES = t X ES , t ∈ N ).
We denoted the interpolated key point and local affine transform as {p k X i } and { Āk X i ←R }.The ED and ES frames of each image sequence were identified either by detection of volume change extremes or from synchronized ECG signal.

Relative motion transfer:

• 1. We deformed all image frames of {Y i } and its corresponding key points

{p k Y i } to ED frame through Ŷi = Y i (T i←ED (•)), pk Y i = arg max H(p k Y i , σ 2 H )(T i←ED (•)) (3.11)
where T i←ED was from the polyaffine dense motion according to Equation 3.4 and Equation 3.2 using its key points and local affine transform matrices. Ŷi represented a sequence of pseudo ED frames of {Y i }.

• 2. We transferred relative motion between X ED and other frames X i to Ŷ framewisely, to obtain the objective sequence G. Specifically, we applied Āk X ED ←X i to the neighbourhood of pk

Y i T k Ŷi ←G i (z) = Āk X ED ←X i • z + (p k Y i + Āk X ED ←X i (p k X i -p k X ED + pk Y i )) (3.12)
where Āk

X ED ←X i = A k X ED ←R (A k X i ←R ) -1
. We obtained the final dense motion through polyaffine motion fusion

T Ŷi ←G i (z) = W0 z + K k=1 Wk (p k X i )T k Ŷi ←G i (z) (3.13)
where we applied directly the key point neighbour weight from sequence X i .

We present two different examples of transferring motion between patients with different ejection fraction (EF) in Figure 3.7 and Figure 3.8. In chapter 2 and chapter 3, we have proposed generalisable solutions for cardiac segmentation and cardiac motion tracking, respectively. It is unknown how these models perform in a down-stream task, such as cardiac pathology detection.

In this chapter, we propose a robust and generalisable pipeline for echocardiography analysis, notably with application in myocardial infarction detection. This chapter is partially included in a journal submission [Yang, 2023a].

The main contributions of this chapter are listed as follows:

• We introduce a robust pipeline for echocardiography analysis using deep learning that make use of shape prior and motion prior (Section 4.2).

• We provide evaluation results across different datasets and compared them with a benchmark study (Section 4.4).

Introduction

As stated in former chapters, automatic algorithms play an important role for cardiac function analysis. Segmentation and motion tracking are two most used approaches to fulfill such objective.

From segmentation output, it is not hard to obtain left ventricle ejection fraction (LVEF) using Simpson's method. Although there exists numerous works that deploy neural networks in order to directly estimate LVEF, we rely on the segmentation mask, this prioritizing its interpretability in our approach.

From deformation field, we can obtain another important index of cardiac function: the global longitudinal strain (GLS). GLS reveals cardiac contractility and is reported to be more sensitive in detecting early cardiac diseases than LVEF [START_REF] Cikes | Beyond ejection fraction: an integrative approach for assessment of cardiac structure and function in heart failure[END_REF]. Computation of GLS can stem from cardiac landmarks [Smistad, 2020;[START_REF] Østvik | Myocardial function imaging in echocardiography using deep learning[END_REF] or dense deformation fields [Morales, 2021]. The strain tensor's sensitivity to the smoothness of the deformation field necessitates a regularization in deep learning models. Common strategies include Gaussian smoothing layers [Krebs, 2020] and rigidity penalties [START_REF] Staring | A rigidity penalty term for nonrigid registration[END_REF]De Vos, 2019]. However, these techniques often apply within limited pixel neighborhoods in the image domain and might not consistently adhere to anatomical constraints. In our approach, we addressed this issue through a poly-affine motion framework, which serves as a well-established motion approximation for myocardial movement [McLeod, 2015]. By integrating a motion prior, we can achieve the capability to track myocardial motion with a small number of parameters.

Myocardial infarction detection in echocardiography

Deep learning and machine learning-based classification methods have demonstrated good performance in myocardial infarction (MI) detection using 2D echocardiography. Convolutional neural networks are commonly employed to extract deep features from 2D echocardiography images for classification purposes [Kusunose, 2020;Huang, 2020;Omar, 2018]. Additionally, handcrafted features, including frequency-related attributes, have been extracted from 2D echocardiography images to aid in MI detection [Liu, 2023;Raghavendra, 2018]. However, much of the existing research relies on training models with local large datasets, posing limitations with respect to external validation by other researchers. A notable exception is the work of Degerli et al. [START_REF] Degerli | Early myocardial infarction detection over multiview echocardiography[END_REF], who introduced a publicly available dataset for MI detection and assessed its performance using segment displacement features. Given the relatively small size of this dataset (130 patients), we opt to extract interpretable features from segmentation and motion estimation outputs. Subsequently, we employ machine learning techniques for MI detection.

Method

The proposed pipeline for 2D echocardiography analysis consisted of three steps. First, a segmentation module estimated the detailed structure of left ventricle, from which the value of ejection fraction was extracted. Second, a motion module predicted the regional myocardium displacement, where strain values can be easily computed. Third, a classification module took the predicted indexes as input and detected patients with myocardial infarction in a multi-view setting.

Shape and motion priors for echocardiography analysis

We used the segmentation model SEG-CONTOUR as introduced in Section 2.

Multi-view myocardial infarction detection

In our study, we chose to formulate a 8-dimension feature vector for multi-view myocardial infarction classification. This vector contained volume and deformation related features, that could be extracted from either segmentation output and motion estimation output. The 8-dimension feature was a concatenation of two 4-dimension vectors from 2-chamber view and 4-chamber view, respectively. Each 4-dimension feature consisted of the following: ejection fraction (EF); global longitudinal strain (GLS) at end-systole; global radial strain (GRS) at end-systole; and, normalised mitral annular plane systolic excursion (MAPSEn). The pipeline is depicted in Figure 4.2.

We choose global features for downstream classification for two primary reasons. Firstly, global features show lower sensitivity to noise and domain shift issues compared to local features. Local features depend on precise segmentation and tracking results, introducing more noise into downstream training when segmentation or tracking is not perfect. In contrast, global features induce less noise due to the denoised average. Secondly, global features provide greater standardization across various views and patients, while local features are contingent on segment split, which may lack consistency across different patients.

Ejection fraction

We applied Simpson's rule [Folse, 1962] to the estimated left ventricle (LV) segmentation output to obtain approximated volume curve along cardiac sequences. End-diastole and end-systole frames were identified as when the volume curve is at its largest and smallest point. EF was calculated following this formula EF = 1 -V ES V ED .

Normalised Mitral annular plane systolic excursion (MAPSEn)

MAPSE measures the displacement of mitral annular plane at systole, which is a helpful cardiac index for the evaluation of LV systolic function. MAPSEn normalizes MAPSE for LV length [START_REF] Støylen | Relation between mitral annular plane systolic excursion and global longitudinal strain in normal subjects: the HUNT study[END_REF].

MAPSEn = MAPSE L ED -1 = L ES -L ED L ED (4.1)
where L t refers to the LV length at time t. We computed the LV length by connecting the apex and mid-basal points from segmentation outputs.

Myocardial strain

We took the end-diastole myocardium segmentation output as the region of interest (ROI) and obtained the pixel displacement within this ROI from motion estimation output across the whole cardiac sequences. For all myocardial pixels at end-diastole, we obtained the 2D Lagrangian finite strain tensor E using

E = 1 2 (F T F -I), (4.2)
where I was the identity matrix and F was the deformation gradient tensor. Radial direction was computed as the tangent field of the correspondence trajectories between endocardial contour and epicardial contour [START_REF] Yezzi | [END_REF]. Longitudinal direction was taken perpendicular to radial direction.

We next identified 8 segments by clustering all the myocardium pixels into 8 clustering using the KMeans method, where the segments 4 and 5 were considered to form together an apex segment. For each segment, one single radial direction and one longitudinal direction were kept, representing the mean of all pixel directions within the chosen segment. Segment strain was the averaged value of pixel strain projected into its segment-wise radial and longitudinal direction. The average strain of all segments were regarded as the global strain. Strains at the end-systolic phase were extracted as features for myocardial infarction detection. CAMUS dataset [Leclerc, 2019a] contains apical 4-chamber (A4C) view and apical 2-chamber (A2C) view of 500 patients. The following annotations are also publicly available: segmentation, volume information and ejection fraction of the left heart at end-diastole (ED) and at end-systole (ES). The dataset contains an official train-test split of 450 patients vs. 50 patients.

Experiments

EchoNet-Dynamic [Ouyang, 2020] is a large open dataset of 10030 A4C echocardiography videos. Left ventricle traces at ED and ES as well as EF values are provided for all videos. The official split of train vs. validation vs. test is 7465/1288/1277.

HMC-QU [START_REF] Degerli | Early myocardial infarction detection over multiview echocardiography[END_REF] is the first public dataset for MI detection. It contains 162 A4C videos, 130 A2C videos, where A2C and A4C views of 130 patients are both available. Notably, this dataset features annotated infarct wall segments across all videos. Moreover, for 109 of the A4C videos, the dataset supplies segmentation masks for the myocardium, generated through a semi-automatic approach [Degerli, 2021a].

CHU dataset contains 2D echocardiography sequences of 76 patients collected retrospectively from Nice CHU hospital. This study protocol was in compliance with the declaration of Helsinki and was approved by the local research Ethics committee. These examinations were conducted and interpreted by four different cardiologists, with only one cardiologist per examination. This clinical dataset contains challenging echocardiography images, with various image quality, coming from different caring scenarios and device vendors. This dataset serves as an evaluation set for the real-world patient monitoring.

A detailed summary of all the datasets are presented in Table .4.1 and Table .4.2.

Experiments and implementation

The segmentation model and the motion tracking model were implemented as described in Chapter 2 and Chapter 3 respectively.

Myocardial infarction detection

We trained the segmentation model and the motion tracking model using CAMUS training data and EchoNet training data respectively (detailed evaluation was presented in Chapter 2 and Chapter 3). In order to perform MI detection, we extracted cardiac features from segmentation and motion estimation outputs in the HMC-QU dataset and CHU dataset, respectively. In particular, we performed 5-fold cross-validation on HMC-QU dataset using multi-view features (130 patients) and evaluated the classifier directly without retraining or finetuning on CHU dataset (64 patients).

Implementation:

We applied two popular machine learning methods from sklearn package [Pedregosa, 2011] for classification: random forest (RF) and support vector machine (SVM), respectively. A resampling strategy was conducted before classification training for the unbalanced HMC-QU dataset.

The best performing method on HMC-QU dataset used normalised segment displacement as features [START_REF] Degerli | Early myocardial infarction detection over multiview echocardiography[END_REF].

f k = max D k (t) min I (k,p) (t) (4.3)
where D k (t) referred to the mean displacement of segment k at time t, and I (k,p) (t) represented the averaged Manhattan distance between segment k and its opposite segment p. Authors in [START_REF] Degerli | Early myocardial infarction detection over multiview echocardiography[END_REF] used active polynomials to extract displacement information. In our case, we obtained the normalised segment displacement using the predicted segmentation and motion estimation results, which resulted in a 6-dimension vector for each apical view. We denoted the method using the proposed global features as 'global', and the method using normalised displacement as 'local'.

Results and Discussion

HMC-QU

First, a significant group difference was observed between MI patients and non-MI patients in the four global parameters on the HMC-QU dataset, as illustrated in Figure 4.3, except for the GRS feature for the apical 4-chamber view.

Second, the results presented in Table 4.3 demonstrate that our method, utilizing global features, outperformed the use of local features, including the state-of-the-art result reported in [START_REF] Degerli | Early myocardial infarction detection over multiview echocardiography[END_REF]. This showcases the effectiveness and generalizability of the proposed pipeline for myocardial infarction detection. Notably, local features extracted using a trained segmentation model and motion tracking model exhibited similar classification accuracy compared to those obtained through the time-consuming active polynomial method.

However, the use of global features for MI detection had its limitations. Further analysis, as illustrated in Figure 4.5, revealed that our proposed method using global features

CHU

As shown in the lower part of Table 4.3, classifiers that utilized global features demonstrated better classification performance on the CHU dataset. This is evident from comparable sensitivity, higher specificity, higher precision, higher F1-score, and higher accuracy. Local features tend to be more sensitive to noisy output from segmentation and motion tracking model due to the domain shift between the test dataset (CHU) and the training dataset (HMC-QU). In contrast, global features, often derived from averaged measurements, exhibit greater stability compared to local features, resulting in superior performance.

Additionally, a significant group difference between MI patients and non-MI patients on the CHU data was observed (Figure 4.6). The evaluation results on the CHU dataset strongly reinforce the robustness and good generalisability of our proposed pipeline for echocardiography analysis.

Discussion

First, all features of the HMC-QU dataset and CHU dataset were extracted from segmentation and motion estimation outputs, whose models were trained on other datasets. We observed significant differences in specific measurements between healthy and MI patients on these two unseen datasets, highlighting the effectiveness and robustness of our segmentation and motion estimation methods in distinguishing between the two groups.

Second, the application of the trained classifier on a real-world unseen dataset (CHU dataset) further supports the robustness of using global features for MI detection. However, we acknowledge that using global features for MI detection faces challenges in identifying patients with only one or two segments containing infarcted myocardium.

Future work will focus on the construction of effective local features and the integration of a second modality, such as an echocardiogram. ECG, revealing the electrical morphology of infarction patients from another perspective, will be explored with echocardiography together in Chapter 6.

Conclusion

In this chapter, we proposed a novel and generalizable pipeline for interpretable 2D echocardiography analysis, incorporating a segmentation model with a shape prior, a motion estimation model with a motion prior, and a classification module using clinical features. Our experiments and comprehensive evaluation demonstrated the robustness of our proposed method compared with other competing methods at different stages.

The validation conducted on diverse public and private datasets, including HMC-QU dataset, serves as benchmark results, highlighting the generalizability of our approach in echocardiography analysis. This chapter was published in the Proceedings of the International Workshop on Statistical Atlases and Computational Models of the Heart (STACOM) [START_REF][END_REF].

Explainable analysis of electrocardiogram

The main contributions of this chapter are listed as follows:

• We propose a time-efficient and automatic pipeline for ECG decomposition and reconstruction by passing through a deep learning model: Cascaded FMMnet, which is capable to reconstruct single beat signals with high quality. The training is unsupervised and make it accessible for all kinds of ECG datasets, with or without annotations. The estimated parameters have explainable meanings for each subwave, such as the amplitude, the position etc. (Section 5.2)

• We have conducted comprehensive experiments using the estimated parameters as features to classify normal and myocardial infarction patients across different datasets. The important features identified are in accordance with the clinical interpretation indexes, such as T wave and Q wave change. (Section 5.3)

Introduction

Myocardial infarction (MI) is a kind of pathology where myocardial cells are necrotic due to the prolonged lack of oxygen (ischaemia). A patient is diagnosed as MI if he/she has elevated cardiac troponin values, and falls into at least one of the following conditions: symptoms of myocardial ischaemia; new changes of ST-segment/T-wave in electrocardiogram (ECG); development of pathological Q waves in ECG; abnormal myocardium motion; and, presence of coronary thrombus [Thygesen, 2018]. Among all diagnostic approaches, ECG is very easy and fast to perform on patients, even by non-experts. However, accurately diagnosing MI patients using ECG recordings is a challenging task.

For example, one study showed that experienced cardiologists only identified 82% of the real ST-segment elevated MI patients [START_REF] Mixon | [END_REF]. Computer-assisted ECG analysis could help cardiologists, non-experts better interpret ECG recordings for MI detection.

There exist many research works exploring automatic MI detection using ECG signals and they can be categorised into feature-based methods and neural network-based methods, respectively. The feature-based methods usually contains three stages: ECG delineation (segmentation); feature extraction; and. classification. Different kinds of features were explored: morphological features (such as ST-elevation value, QRS duration, T wave amplitude, Q wave amplitude, etc.) [Lu, 2000;[START_REF] Arif | [END_REF]Jaleel, 2016]; wavelet transform related features (coefficients) [START_REF] Peimankar | Analysis of features for myocardial infarction and healthy patients based on wavelet[END_REF]Jayachandran, 2010;Bhaskar, 2015]; empirical mode decomposition features [Acharya, 2017a], and so on. Compared with other features, morphological features are explainable but very sensitive to ECG delineation results. ECG delineation methods [START_REF] Graja | Hidden Markov tree model applied to ECG delineation[END_REF][START_REF] Peimankar | Analysis of features for myocardial infarction and healthy patients based on wavelet[END_REF]Jimenez-Perez, 2021], usually depend on annotations for all of the events (P,Q,R,S,T) to train models. They are constrained by the size and type of pathology in the annotated dataset.

The neural network based methods for MI detection have overwhelmed in recent years.

For example, single-beat ECG signals are either directly classified using a 1D convolutional neural network (CNN) [Acharya, 2017b] or are transformed into 12-lead 2D image for input into a 2D CNN [Fang, 2022]. For more detailed reviews of MI detection methods, interested readers are invited to read reviews [Ansari, 2017;Xiong, 2022].

In our work, we tackle this MI detection problem using the feature-based approach by ECG parameterisation. Actually, ECG parameterisation/modelling is not a new idea. For example, Liu et al. [START_REF] Liu | [END_REF] proposed to fit a 20th order polynomial function to a given ECG signal and used the fitted coefficients as ECG features. Second-order ODEs were applied to model the 12-lead ECGs and the estimated time-varying coefficients were used as features [START_REF] Zewdie | [END_REF]. Both methods achieved good accuracy for MI detection; however, the features did not have an explainable meaning for clinicians. In our case, we adapted the explainable Frequency Modulated Moebius (FMM) model as our parameterisation model [Rueda, 2021] for single-beat ECG.

Methods

The pipeline of our explainable ECG analysis consisted of three stages (Figure 5.1(a)). First, the 12-lead ECG recordings were filtered and segmented to obtain single-beat median ECG signals. Second, each lead-wise median signal was passed through an encoder network (Cascaded FMMnet) for subwave decomposition. The encoder network generated 21 estimated parameters crucial for signal reconstruction and interpretation. Third, for each sample (12-lead median ECG signal), myocardial infarction classification was conducted based on the 264(21x12+12) estimated parameters, where 12 additional parameters came from estimated ST-segment voltage value. To explain the classification result, we used the additive weighted features for linear classification models and SHAP value [Lundberg, 2017] for non-linear classification models. Finally, decomposed waveform and feature importance from classifier together gave visual and quantitative explainability of our prediction result.

Data Preprocessing

The preprocessing included 5 steps: resampling; filtering; R-peak detection; ECG segmentation; and, median signal generation.

The original 12-lead ECG recording was resampled to 500Hz if its original sampling rate was not 500Hz. A butterworth high-pass filter with cutoff frequency at 0.5Hz was

Methods

then applied to remove baseline wander. The R-peaks of Lead II were automatically detected and used as reference for all the other leads. For every lead ECG, each single beat segment was set from 35% heart beat duration (s) before the R-peak to 50% heart beat duration (s) after the R-peak. One 1.2-second median beat signal was calculated by aligning the R-peaks of all the single beats (at 0.5-second position) and padded by neighbouring values at the two ends if the medial signal was shorter than 1.2s. Neurokit2 package [Makowski, 2021] was used for filtering, R-peak detection and single beat segment calculation.

Cascaded FMMnet

In order to reinforce explainability in automatic ECG analysis, we utilised the decomposition model proposed by [Rueda, 2021]. The idea was to approximate the single-beat ECG signal by composition of five subwaves (P,Q,R,S,T), each of which was parameterised by a Frequency Modulated Moebius. Assuming X(t i ), t i ∈ [0, 2π], the original signal of a single-beat ECG record, could be decomposed into five subwaves W s , s ∈ {P, Q, R, S, T }. Each subwave was described by a four-dimensional parameter p s = {A s , α s , β s , ω s } respectively,

W s (t, p s ) = A s cos(β s + 2 arctan(ω s tan( t -α s 2 ))) (5.1)
where A, α, β, ω controlled the absolute amplitude, the position, the skewness and the kurtosis of the waveform (see Figure 5.1 (c)(d)). The approximation of original signal X(t) was defined as the addition of the five subwaves W s and an additional baseline

shift M , X(t, θ, M ) = M + s∈{P,Q,R,S,T } W s (t, p s ) (5.2)
where θ = (M, p S , p Q , p R , p S , p T ) and they verified the following ranges

• 1. M ∈ R • 2. p s ∈ R + × [0, 2π] × [0, 2π] × [0, 1], s ∈ {P, Q, R, S, T } • 3. α P ≤ α Q ≤ α R ≤ α S ≤ α T
The aim of decomposition was to estimate the optimal 21 parameters θ that best fit

θ = argmin θ n i=1 [X(t i ) -X(t i )] 2 .
Instead of using the computationally intensive iterative optimisation [Rueda, 2021], we estimated the 21 parameters through a data-driven deep learning model: namely the Cascaded FMMnet. This network consisted of 5 identical cascaded sub-network, each of which was responsible for estimating 5 parameters

(M i , A i , α i , β i , ω i ) of one subwave S i , where S i (t) = M i + A i cos(β i + 2 arctan(ω i tan( t-α i 2 ))).
Assuming the original signal X(t), the input of the ith sub-network X i (t) was the residual of the original signal subtracting former subwaves, i.e. X i (t) = X i-1 (t) -S i-1 (t), where i ∈ [1, 5], X 0 (t) = X(t), S 0 (t) = 0. The encoder block in our network was comprised of: 2 stacks of causal convolution with down-sampled skip-connection; 1 max-pooling layer; and, 2 linear layers. It took an input of 1x600 dimension and outputted a 21-dimension vector which was the estimation of the parameters. The input median ECG signal was resized to be within the range of

Methods

[-1, 1] and the last linear layer had a Sigmoid activation for (A i , α i , β i , ω i ) and a Tanh activation for M i parameter. The final M was the sum of all the M i , i ∈ [1,[START_REF]2 Classification results obtained on different datasets using Cascaded FMM net[END_REF]. The final estimation of θ were obtained by multiplying M , A i with the resize factor and by multiplying α i , β i with 2π.

We penalised the network by minimising the mean square error of reconstructed signal S i (t) and the input signal X i (t). In order to force each subnet to capture a fixed subwave, a regulariser called prior loss was added in the loss function. We randomly chose 100 median ECG samples and estimated the 21 hidden parameters (5 hidden waves) using the FMM R package [Rueda, 2019]. The mean µ α and variance σ 2 α of parameter α were computed and are used to constrain the subwaves' position. We let the Cascaded FMMnet estimate the T,R,S,P,Q subwaves sequentially by regularising the position α to be close to its corresponding pre-calculated distribution. The total loss function is:

loss = loss mse + γloss prior = 5 i=1 |X i (t) -S i (t)| 2 + γ 5 i=1 (α i -µ α ) 2 σ 2 α (5.3) (5.4)
where γ controlled the balance of signal fitting and parameter distribution.

In order to ease the estimation for all leads, we assumed that the P,Q,R,S,T were sequentially positioned in all single beat ECG, which may be different with the conventional names of ECG wave peaks in some leads. For example, in Figure 5.7(a), the decomposed Q wave represented the conventional R wave for lead V1. We included three public ECG datasets and one private dataset in our study. All four datasets contained standard 12-lead ECG recordings and were (re)sampled to 500Hz. The PTB-XL dataset [Wagner, 2020] contained 21837 12-lead ECG recordings and covered multiple ECG diagnostics and morphologies. Our Cascaded FMMnet was trained on this large dataset without considering specific pathology. For classification, as here we were concerned about detecting myocardial infarction (MI) patients from normal (NORM) cases, we included the detailed information of MI and NORM across different datasets in Table 5.1. The private dataset (CHU) was collected from Nice University Hospital in a scanned PDF format. Specific preprocessing was conducted to digitalize the ECG signals [Fortune, 2022].

Experiments and Results

Datasets

Reconstruction

Experiment

The PTB-XL dataset was used to train the Cascaded FMMnet. As this was provided with 10 pre-defined folds, we randomly chose 8 folds as training set, 1 fold as validation set and the rest fold as test set. 100 random samples from training set were picked to compute the mean and variance used in regulariser (equation 4). The encoder network was implemented in Pytorch and trained with batch size of 192, learning rate of 0.0001. γ was initialised from 1 and it was updated to γ = 0.1γ if loss mse ≤ γloss prior .

Results

We evaluated the reconstruction performance by R2 score and mean absolute error

(M AE = 1 600 600 i=1 |X(t i ) -X(t i )|).
The proposed Cascaded FMMnet demonstrated very good reconstruction results and generalised well on three unseen datasets from different centers. First, the FMMnet was trained on 80% of the whole PTB-XL dataset and it demonstrated similar reconstruction result on the train/validation/test set of PTB-XL: they all presented a mean MAE of 0.016mV and a mean R2 score of 0.94. Second, all four datasets showed consistent reconstruction error on NORM/MI patients (Figure 5.3(a-b)) and lead-wisely . Our Cascaded FMMnet took 0.09s/12 leads on a Dell laptop (Intel© Core™ i7-8650U CPU @ 1.90GHz × 4) while the original FMM optimisation [Rueda, 2019] takes more than 10s/1 lead on the same machine. We show an example of 12-lead ECG decomposition in Figure 5.7(a).

Experiments and Results

Classification

Experiment

For each 12-lead ECG sample (median beat), we obtained a 252-dimension feature vector from the Cascaded FMMnet. In addition, we included ST-segment voltage feature for each lead to form a 264-dimension vector. The ST-segment was identified as the short flat platform between peak S and peak T from each lead with the help of positional parameter α T and α S .

Explainable classification:

We explored two approaches to provide explainable classification. The first method began with a partial least squares regression that projects the 264-dimension vector to 3-dimension. A support vector machine (SVM) with linear kernel was applied then to find the best hyperplane that separated the MI/NORM patients. The additive nature of weighted features help to explain the classification results. The second method was to use SHAP value to explain a Logistic regression (LR) model for MI/NORM classification. We trained separate classifiers on PTB-XL, PTB and CPSC(+extra) datasets using 10-fold, 5-fold, 5-fold cross-validation, respectively. Two more specific classifiers for detecting AMI/IMI (vs. non AMI/IMI) were established on PTB-XL using 10-fold cross-validation.

Generalisable classification:

We tested the classifiers trained on PTB-XL to other three datasets: PTB, CPSC(+extra) and our private dataset (CHU), in order to evaluate the generalisation of our proposed classification pipeline.

Results

We present the detailed classification evaluation in Table 5.2. Firstly, it's remarkable that using our classification pipeline, we were able to obtain satisfactory classification performance on different datasets (PTB-XL/PTB/CPSC) compared with other methods. As shown in Figure 5.5, a linear classifier (SVM with linear kernel) was already capable to obtain good separation of MI/NORM patients on both training and test data for PTB-XL, PTB and CPSC datasets, respectively. Using SHAP values, our models (Logistic regression classifiers) were capable to identify important infarction related features such as T wave amplitude change (T_A), T inversion (T_β) etc. They were also able to distinguish the influenced leads for infarction. For example, as shown in Second, models trained on PTB-XL generalised well on other datasets, without much drop of accuracy. However, we observed a drop in the specificity on our small private dataset. Since the signals were extracted from scanned ECG papers, the domain gap could be enlarged. In addition, the manner of annotation may be different. The label of NORM/MI in our private dataset is the final diagnosis decision, given by an experienced cardiologist after an overall examination of the patient, including echocardiography, ECG and sometimes coronary angiography. This may be different from the direct diagnosis based on the ECG alone from PTB-XL dataset.

We further include an example from PTB-XL test data that was correctly detected as MI (MI/NORM classifier) in Figure 5.7. The influenced ECG leads of this patient are I, II, avL, V2, V3, V5 and V6. First, our Cascaded FMMnet successfully identified the subwaves for most of the leads (Figure 5.7a). Second, we grouped the weighted features lead-wisely using the linear SVM model and observed that all leads helped us identify this patient as MI case (that is because they are all positive, as shown in Figure 5.7b). In addition, when examining the top-3 highest weighted features of every lead (Figure 5.7c), we found that most of the important features are in agreement with the clinical diagnostic (see red and blue rectangles in Figure 5.7c). We also observed a similar trend of important features explained by SHAP value from the Logistic regression classifier (Figure 5.7d).

Discussion and Conclusion

In this work, we proposed an automatic decomposition model, Cascaded FMMnet for ECG analysis which facilitated the downstream tasks, such as classification (in our case, the classification of normal vs. myocardial infarction patients). Overall, the Cascaded FMMnet was able to generalise reconstruction across datasets from different clinical centers and the estimated parameters could be used for MI classification with good explainability. It should be noticed that the Cascaded FMMnet was trained on PTB-XL dataset, which contains not only healthy samples but also samples with different pathologies (myocardial infarction, conduction disturbance, hypertrophy and so on). We believe that our novel network is capable to play an important role in other pathology classification. In the future, we will explore to improve the decomposition of different ECG leads with more specification. Our current Cascaded FMMnet assumes P,Q,R,S,T present in all leads; however, in some cases some subwaves could be absent. For example, in Figure 5.7a, we can observe the small S wave (in green) does not represent a meaningful peak and it's superposed onto the large R wave in lead avF, V1-4. This phenomenon is supported by the superior reconstruction accuracy on the leads close to lead II(I, II, V5, V6) than the other leads (Figure 5.3c-d). The original FMM paper [Rueda, 2021] also presented their results on lead II only. Some specific care for leads like V1-V4 should be considered.

Appendix

Paper ECG digitization

Our paper ECG digitization is based on a open-source software [Fortune, 2022] with adequate adaptation. A general pipeline of the ECG digitization is depicted in Figure 5.8. We only show one simple example here, for pipeline description and explanation. Since the CHU dataset contains several types of ECG paper formats (Figure 5.9), the automatic pipeline may fail and need manual interaction, such as signal masking and key point assignment. Abstract Medical devices used in diagnoses capture only one aspect of cardiac function. For example, 2D B-mode echocardiography reveals the anatomy and mechanical change of the heart, while electrocardiogram (ECG) captures electrical activities of the heart from different observation positions. As a routine for cardiac disease diagnosis, those examinations are not conducted synchronously, but sequentially and provide complementary proof for final decision. While most of the current AI-based models focus on a single modality, the combination of multi-modal information in AI research for healthcare has started to gain popularity and to become a hot topic. However, the scarcity of public multi-modal data for cardiac disease diagnosis make this task rather difficult. In this study, we adapted an uncertainty-based deep learning framework using unpaired single modality data, for better diagnosis of MI using echocardiography and eletrocardiogram data jointly. Specifically, we trained two single modality models using public datasets and evaluated the multi-modal classification performance on a small paired dataset collected from a local hospital. Our experiment demonstrated that uncertainty-based multi-modal decision fusion outperforms popular fusion strategies and single modality models. Thus, uncertainty-based models could be a sustainable solution for unpaired multi-modal training.

The contribution of this chapter are listed as follows:

• We proposed to adapt a multi-modal decision fusion strategy with uncertainty using unpaired public single modality datasets (Section 6.2).

• We showed that uncertainty based multi-modal fusion improves the accuracy of detecting MI patients by 7% comparing to the single modality model, while the performance of other traditional fusion models are limited by the bestperforming modality (Section 6.3).

Introduction

In real-world, clinicians combine information from different examinations and measurements to make clinical decisions. Most current AI research for healthcare simply consider one single modality, which does not profit from the complex and heterogeneous information that one can observe from patients using different imaging modalities, sensoring devices, biochemical tests etc. Multi-modal machine learning, which seeks to model the interactions between different modalties, brings opportunities for improving the prevention, diagnosis and therapy in AI-enabled healthcare [Acosta, 2022].

One challenge in biomedical multi-modal learning is to determine how to fuse information from different medical modalities for downstream tasks. Depending on when the fusion occurs, one can distinguish: early fusion and late fusion, respectively. Early fusion combines the raw modality or extracted features at the input level according to certain fusion approaches, such as concatenation, multiplicative interaction [Jayakumar, 2020], polynomial fusion [START_REF] Kefalas | [END_REF], tensor fusion [Zadeh, 2017;Hou, 2019] etc. Late fusion aggregates the prediction outputs of different modalities at the decision level (e.g. using majority voting, weighted voting etc.) to generate a final decision. Early fusion usually demands paired multi-modality data to explore detailed interaction strategies, while late fusion only need single modality outputs, thus being less demanding for paired data.

In our study we were limited by the number of paired multi-modal data (only 56 patients). Therefore, we chose to do late fusion, which can leverage the usage of large single modality datasets. In addition, we adapted the uncertainty based fusion [START_REF] Han | Trusted Multi-View Classification[END_REF], which generated the final decision based on the most trusted modality according to the single modality uncertainty.

6.2 Method

Single modality evidential deep learning

Uncertainty and the Theory of Evidence Evidential deep learning (EDL) quantifies the class probabilities and overall uncertainty in a unified theoretical framework [START_REF] Sensoy | [END_REF]. Based on the Dempster-Shafer Theory of Evidence [Yager, 2008], evidential deep learning adapts the idea from Sujective Logic (SL). SL associates the belief of possible class label assignments with the parameters of Dirichlet Distribution [Jsang, 2018], including the belief that the truth label is equally likely (i.e., "I don not know" for uncertainty quantification).

D(p|α) =      1 B(α) K i=1 p α i -1 i for p ∈ S K , 0
otherwise, (6.2)

Method

where S K represents the K-dimensional unit simplex S K = {p| K i=1 p i = 1 and 0 ≤ p 1 , ..., p k ≤ 1}, and B(α) is the K-dimensional multinomial beta function. According to SL, the class belief assignment is connected to a Dirichlet distribution with α k = e k + 1. The relationship between evidence, belief, uncertainty and Dirichlet parameters is summarized below :

b k = e k S = α k -1 S and u = K S , S = K i=1 (e i + 1) = K i=1 α i . (6.3)
The above relationship reveals that the higher the evidence e k for k th class is observed, the greater the class belief b k and the corresponding Dirichlet parameter α k will be (Figure 6.1 (a)). Similarly, when the total evidence observed from input data is small, i.e. e k closer to 0 and α k , k = 1, ..K closer to 1, the uncertainty of prediction becomes higher (Figure 6.1 (b-c)). We assume that y i is a one-hot vector of ground truth classification label for input data x i . The cross-entropy loss is usually used in conventional neural network classifiers: (6.4) where p ij is the predicted probability for sample x i belonging to class j. Under the theory of evidence and Dirichlet distribution assumption, we can compute the Bayes risk of cross-entropy loss function as

Learning to form opinions

L CE = - N i=1 K j=1 y ij log(p ij ),
L U C i = [ K j=1 -y ij log(p ij )] 1 B(α i ) K j=1 p α ij -1 ij dp i = K j=1 y ij (ψ(S i ) -ψ(α ij )), (6.5)
where ψ(•) represents the digamma function.

The minimization of the above loss does not guarantee that less evidence will be generated when the model predicts incorrect labels. To guide the network learn zero total evidence for uncertain samples, a regularization term is introduced. This term deploys a Kullback-Leibler divergence term to penalize the predictive Dirichlet distribution to be close to D(p|1). 6.6) where Γ(•) represents the gamma function and 1 refers to a K-dim vector of all ones.

KL[D(p

i |α i )||D(p i |1)] = log( Γ( K k=1 αik ) Γ(K) K k=1 Γ(α ik ) ) + K k=1 (α ik -1)[ψ(α ik ) -ψ( K k=1 αij )], (
Thus, the final loss function for evidential deep learning neural networks reads:

L = N i=1 L U C i + λ t N i=1 KL[D(p i |α i )||D(p i |1)], (6.7)
where λ t = min(1, t/10) ∈ [0, 1] is a balancing coefficient for regularization.

Multi-modal fusion with uncertainty

We adapt the multi-modal combination of belief and uncertainty masses presented in [START_REF] Han | Trusted Multi-View Classification[END_REF] in an off-line manner (as illustrated in the lower right part of Figure 6.2).

Considering two independent sets of belief mass values

M 1 = {{b 1 k } K k=1 , u 1 } and M 2 = {{b 2 k } K k=1 , u 2
} that are outputs of two independent single modality neural networks, the fusion of belief and uncertainty mass values is designed as follows: (6.8) where C = i̸ =j b 1 i b 2 j measures the amount of conflict and 1 1-C is applied for normalisation of belief and uncertainty mass values.

b k = 1 1 -C (b 1 k b 2 k + b 1 k u 2 + b 2 k u 1 ), u = 1 1 -C u 1 u 2 ,

Experiments and results

Datasets

In this study, we focused on MI detection using echocardiography and eletrocardiogram data. First, two independant datasets of ECHO and ECG respectively were involved:

• HMC-QU dataset: contains 130 long-axis 2-chamber view sequences (68 with myocardial infarction segments) and 162 long-axis 4-chamber view sequences (93 with MI segments).

• PTB-XL dataset: contains 12-lead ECG data (with 7185 samples of healthy controls and 2955 samples with 100%-certain MI).

In addition, a small number of paired ECHO and ECG data were collected retrospectively from CHU hospital. This dataset contains data from 56 patients, with 56 paired data of ECG and 4-chamber view ECHO, along with 50 paired data of ECG and 2-chamber view ECHO. We trained the models using HMC-QU and PTB-XL for single modalities separately, and evaluated the multi-modal classification performance on CHU dataset.

Experiments

We first obtained input features for ECHO and ECG using the models described in Chapter 3 and Chapter 5 respectively. For ECHO data, we constructed a 40-dimension vector from temporal movement of the myocardium. This vector was composed of mean and standard deviation of each key point movement along the x-and y-axis, respectively, that were estimated from motion estimation deep learning model. For ECG data, we constituted a 21 * 12 dimension vector from ECG decomposition model.

For all experiments, we used a 3-layer fully connected network (FCN) for single modality classification. We trained single modality models with 10-fold cross validation for ECG data and 5-fold cross validation for ECHO data. We then compared the multi-modal classification with uncertainty to several baseline fusion strategies.

We assumed that the output of FCN after Sigmoid function was p k c and the prediction class was ȳk , where c referred to class and k referred to modality, while the MI class was set to label 1. The following fusion strategies were included in our study : Finally, the baseline single modality model was trained using cross-entropy loss (Equation 6.4) and the uncertainty model with Equation 6.7. The multi-modal fusion with uncertainty was performed according to Equation 6.8.

Implementation

The uncertainty model for ECHO and ECG were trained using the following hyperparameters:

• ECG: learning rate 0.01, batch size 512, number of epochs 1000, λ t = 10;

• ECHO: learning rate 0.0001, batch size 8, number of epochs 500, λ t = 50.

We chose the model with best validation loss during training.

6.3

Experiments and results First, we present the cross validation results on HMC-QU and PTB-XL dataset in Table 6.3 and Table 6.2. Although evidential deep learning (EDL) model demonstrated reduced performance compared with model trained using standard cross-entropy loss, its performance was comparable when using mixed 2-chamber and 4-chamber (2CH/4CH mixed) view together (292 samples in total). We made a similar observation on ECG classification using uncertainty-based loss (Table 6.3). Figure 6.3 shows how the test accuracy changes when EDL only keeps predictions under varying uncertainty thresholds. Notably, on both datasets, the accuracy decreased as the uncertainty threshold increased. We present the off-line multi-modal fusion evaluation on CHU dataset in Table 6.4.

Results

The performance of multi-modal conventional fusion (upper part) was limited by the best performing modality, in our case, the ECHO modality. The mean fusion strategy outperformed the other conventional methods, with a slight improvement in sensitivity but significant reduction in specificity due to the erroneous output of ECG prediction.

In the lower part of Table 6.4, we can observe that uncertainty-based fusion improved largely the prediction accuracy compared to single modalities. In addition, this novel approach well combined the advantages of each modality: higher sensitivity than single ECHO output and higher specificity than single ECG output, with only a slight decrease compared with the best value of single modality outputs. Uncertainty-based fusion generated multi-modal prediction according to the most trustworthy modality, therefore improving the final prediction of diagnosis.

Conclusion

In this study, we explored different multi-modal late fusion strategies, demonstrating that uncertainty-based fusion outperformed other conventional fusion methods by improving the single modality accuracy by 7%. The uncertainty-based fusion was based on single modality evidential deep learning and examined the uncertainty of each modality prediction to balance the most trustworthy prediction for final prediction. In addition, the integration of uncertainty did not need any sampling steps and was easy to implement with deep learning methods. The off-line fusion setting made the most use of large public single modality dataset and kept the valuable paired multi-modal data for evaluation.

Conclusion

In this thesis, we proposed a framework for multi-modal cardiac function analysis using portable medical modalities: echocardiography (ECHO) and electrocardiogram (ECG). Specifically, we described in detail our solutions of analysing single modalities and multimodal fusion with a focus on robustness, generalization, explainability and uncertainty. In the following, we summarize the main contributions of this thesis and discuss limitations of current work and potential directions of future research.

Main Contributions

Single-frame segmentation of echocardiography with shape prior In Chapter 2, we introduced shape information from three-levels for 2D echocardiography segmentation and conducted thorough experiments across different datasets. In particular, the strategy from pixel level introduced shape prior in terms of contour loss and outperformed other strategies. With appropriate augmentation techniques, the trained model generalized well on segmentation prediction and left ventricle ejection fraction (LVEF) estimation on different test datasets. LVEF is an important clinical cardiac feature that cardiologists usually measure during examination. AI-enabled robust estimation of LVEF has significant potential to serve as a reference for experts and non-experts.

Poly-affine motion estimation model for echocardiography with motion prior

In Chapter 3, we proposed a weakly-supervised motion estimation framework for 2D echocardiography sequences. Instead of estimating the dense motion field, we parameterized the movement of the left ventricle myocardium through a poly-affine motion framework, using only 60 parameters. Our design of motion prior helped significantly the model to predict key points around the myocardium and the corresponding local transformations. The compact modelling of the LV motion not only improved the registration accuracy but also the motion field regularity in terms of Jacobian determinant. Robust estimation of left ventricle motion help reveal global longitudinal strain (GLS), another sensitive feature for cardiac dysfunction detection. In addition, the poly-affine motion model was capable to transfer motion pattern from one sample to another while keeping the original appearance of ultrasound images. Effective motion transfer is able to enlarge the size of echocardiography dataset, for example for self-supervised learning.

Explainable echocardiography analysis pipeline

In Chapter 4, we introduced a pipeline for echocardiography analysis based on the method proposed in Chapter 2 and Chapter 3. Specifically, we extracted left ventricle ejection fraction (LVEF) and normalized mitral annular plane systolic excursion (MAPSEn) from segmentation prediction, and the global longitudinal strain (GLS) and global radial strain (GRS) from motion estimation prediction. On two test datasets, the extracted features showed significant group difference between healthy controls and myocardial infarction patients. The proposed pipeline demonstrated great potential for explainable cardiac function analysis using portable ultrasound devices.

Unsupervised decomposition of electrocardiogram for explainable analysis

In Chapter 5, we proposed an unsupervised decomposition model for electrocardiogram analysis. The model was able to take any single-beat lead-agnostic ECG signal as input and predict 21 explainable parameters that controlled P, Q, R, S, T subwaves. The ensemble parameters of 12-lead ECG demonstrated great classification performance when detecting myocardial infarction patients from healthy controls across various datasets. Besides, the visualization of ECG decomposition provided an explicit way to examine the analysis result qualitatively. The proposed model exhibits significant promise for application in the classification of other diseases due to its robust and adaptable framework.

Uncertainty guided multi-modal decision fusion

In Chapter 6, we adapted an uncertainty based classification framework for off-line multimodal decision fusion and evaluated its performance on myocardial infarction detection using paired ECG and ECHO data. The proposed method generated final decision leveraging the most trustworthy modality and improved the multi-modal performance by 7% compared with single modality accuracy, while the performance of other conventional fusion strategies was limited by the best performing modality. Our findings illustrated that cardiac dysfunction detection can yield more reliable results compared to a single modality approach, thereby promoting the adoption of a multi-modal setup for real-world portable cardiac diagnosis.

Future research

In this thesis, we presented original contributions to the field of medical imaging and signal analysis, with a focus on employing 2D echocardiography, electrocardiogram and multi-modal classification for better detection of disease (e.g. MI). Although we have achieved promising results, there are several improvement needed which suggest several directions for future work.

Echocardiography analysis

We acknowledge that the segmentation model was applied on separate frames of echocardiography sequences, while temporal information of cardiac sequences were neglected by current model. Enforcing temporal consistency is a good way to reduce segmentation outliers and to engage unlabelled data for training [Wei, 2020;Painchaud, 2022]. In addition, for the motion tracking model, we imposed a strong regional weight for each key point without considering the various thickness of the LV wall across patients and across regions. Thus, it would be interesting to further combine temporal motion tracking model with temporal segmentation model, letting the predicted mask of the myocardium guide the region of effect for each local key point. Furthermore, motion tracking of LV regularizes vice versa the segmentation of cardiac structures for unlabelled temporal frames, accordingly. In addition, it's crucial to estimate the uncertainty of predicted segmentation and motion transformation for clinical applications. Methods proposed in [START_REF] Grzech | A variational Bayesian method for similarity learning in non-rigid image registration[END_REF][START_REF] Judge | Asymmetric Contour Uncertainty Estimation for Medical Image Segmentation[END_REF] are of great interest for the further improvement of our echocardiography analysis pipeline.

Eletrocardiogram analysis

Our proposed ECG decomposition method has two limitations. First of all, it can only analyze R-peak aligned single-beat signal. In the cases where it is difficult to detect the R-peak in the original ECG signal, the utility of the current model will be limited. Second, although the proposed model can process lead-agnostic single-beat ECG signal, the inter-lead relationship is neglected. We vise to analyze continuous time series instead of single-beat signal in the future. One possible approach is to learn contrastive representation of ECG by masking random leads as proposed in [START_REF] Kiyasseh | Clocs: Contrastive learning of cardiac signals across space, time, and patients[END_REF][START_REF] Oh | Lead-agnostic self-supervised learning for local and global representations of electrocardiogram[END_REF]. Specifically, we could integrate the information of physical lead position into the modelling process as in [Chen, 2021], who proposed an ECG synthesis model that takes into consideration the physical lead position encoding. In addition, we could further model the periodic characteristic of ECG time series, for instance, using the dynamic latent trajectory representation proposed in [START_REF] Laumer | [END_REF]Ryser, 2022].

Multi-modal learning for portable analysis

In Chapter 6, we utilised the 12-lead ECG with 2D ECHO data features. While in portable examination, the 12-lead ECG is not always available. It would be more realistic to combine ECHO data with single-lead ECG data. For example, Lead I or Lead II could be easily measured using wearable ECG devices. However, such ambitious objective may be very challenging. That is because single-lead ECG is a useful diagnostic tool for rhythmbased and conduction disorder-related diseases; however, it is not very informative for MI detection [Witvliet, 2021]. Furthermore, it is crucial to model the interaction of full 12 leads and condition the representation on a desired single lead. For example, we could project each lead into a joint latent space using multi-channel variational autoencoder [START_REF] Antelmi | Sparse multi-channel variational autoencoder for the joint analysis of heterogeneous data[END_REF], or generate the 12-lead data using single-lead-based ECG synthesis [START_REF] Lee | [END_REF][START_REF] Golany | SimGANs: Simulator-based generative adversarial networks for ECG synthesis to improve deep ECG classification[END_REF][START_REF] Sofia C Beco | Electrocardiogram lead conversion from single-lead blindly-segmented signals[END_REF]. Moreover, it demands effective modelling between ECHO time series and single-lead ECG signal. Should synchronized ECHO and ECG is obtained, it would be entrancing to examine the temporal correlation between time series from ECHO and ECG. Another possible direction is to explore methods from multi-variate time series modelling, such as LSTM network [Karim, 2019], autoencoders [Audibert, 2020], transformer-based models [START_REF] Zerveas | A transformer-based framework for multivariate time series representation learning[END_REF] and graph neural networks [START_REF] Duan | [END_REF].

analysis. Non-rigid echocardiography image registration is one of the most challenging image registration tasks, as finding the deformation field between noisy images is a highly nonlinear problem in the absence of ground truth deformation. Specifically, various image registration problems require the mapping between moving and fixed images to be folding free [Cao, 2005;Vercauteren, 2008;Arsigny, 2006]. Traditional nonlearning approaches rely on the optimizing similarity metrics to measure the matching quality between image pairs [Oliveira, 2014;[START_REF] Davatzikos | Spatial Transformation and Registration of Brain Images Using Elastically Deformable Models[END_REF]Vercauteren, 2007].

With the rapid promotion of deep learning, various frameworks of convolutional neural networks (CNN) have been introduced in image registration and have shown impressive performance in many research works.

We consider a 2D non-rigid machine learning-based image registration task in this work.

With two given images: I N f ix , N ∈ R and I N mov , N ∈ R, we want to learn a model T ω (I mov , I f ix ) → ϕ(θ) that generates a constrained transformation ϕ(θ) based on a similarity measurement M to warp the moving image by minimising the loss function:

L = arg min θ M(I f ix , I mov • ϕ(θ)) + λC(ϕ(θ)) (A.1)
where the transformation ϕ is parameterised by the parameter θ and constrained by a regularisation term C(ϕ(θ)) to ensure ϕ to be a spatially smooth transformation. However, iterative optimization of Eq. A.1 is very time-consuming, whereas a well-trained CNN does not need any iterative minimization of the loss function at test time. This advantage drives researchers' attention to learning-based registration. Learning-based registration methods can be categorised into supervised and unsupervised registration approaches.

A.1.1 Supervised Registration

The supervised learning registration methods [Wu, 2016;Cao, 2017;[START_REF] Rohé | SVF-Net: Learning Deformable Image Registration Using Shape Matching[END_REF] are primarily trained on a ground-truth training set for which simulated or registered displacement fields are available. The training dataset is usually generated with traditional registration frameworks or by generating artificial deformation fields as ground truth for warping the moving images to get the fixed images. [Sokooti, 2017;[START_REF][END_REF].

One of the limitations of the supervised registration approaches is the registration quality, which is highly influenced by the nature of the training set of deformation map, although the requirement in terms of training set can be partially alleviated by using weaklysupervised learning [Hu, 2018b;[START_REF] Blendowski | Weakly-supervised learning of multimodal features for regularised iterative descent in 3D image registration[END_REF]Balakrishnan, 2019;Hu, 2018a;Ferrante, 2019].

A.1.2 Unsupervised Registration

In unsupervised registration [Krebs, 2019;Dalca, 2019;Balakrishnan, 2019;Hering, 2021;Mansilla, 2020], we rely on a similarity measure and regularisation to optimize the neural network for learning the transformations between the fixed and moving images. Usually, a CNN is used directly for warping the moving images, which is then compared to the fixed-image with the similarity loss. The displacement field can also be obtained from a generative adversarial neural network, which introduces a discriminator neural network for assessing the generated deformation field quality. [Tanner, 2018;Zheng, 2021;[START_REF] Mahapatra | Training data independent image registration using generative adversarial networks and domain adaptation[END_REF]Debayle, 2016].

A.1.3 Multi-layer Perceptron and Transformers

MLP is one of the most classical neural networks and consists of a stack of linear layers along with non-linear activation [Van Der Malsburg, 1986]. For several years, CNN has been widely used due to its performance on vision tasks and its computation efficiency [Krizhevsky, 2012]. Recently, several alternatives to the CNN have been proposed such as Vision Transformer (ViT) [Dosovitskiy, 2021] or MLP-Mixer [Tolstikhin, 2021] which demonstrated comparable or even better performance than CNN on classification or detection tasks. There are currently intense discussions in the community of whether patching, attention or simple MLP play the most important role in such good performance.

In this work, we propose three MLP/Transformer based models for echocardiography image registration and compare them with one representative CNN model in unsupervised echocardiography image registration. There are already works using transformers to register medical images, such as TransMorph [START_REF][END_REF] and Dual Transformer [Zhang, 2021], but they are mostly restricted to high signal-to-noise medical images, such as MRI images and CT images. While ultrasound images (2D) are actually the most popular imaging modality in the real world. Our inspiration not only comes from the trending debate over Transformer and MLP, but also stems from the intuition that patch-based learning methods share similar logic to traditional block matching method for cardiac tracking.

Our contributions are two-folded. First, we show the effectiveness of patch-based MLP/Transformer models in medical image registration compared with a CNN-based registration model. Second, we conduct a thorough ablation study of the influence of different structures (MLP, MLP-Mixer, Transformer) and different scales (single scale or multiple scales). Our results provide empirical support to the observation that the attention mechanism may not be the only key factor in the SOTA performances. [Liu, 2021a;Melas-Kyriazi, 2021], at least in the field of unsupervised image registration.

A.2.3 Multi-scale features

In order to enforce different reception fields for patch-based models, we decide to combine multi-scale models together. This is accomplished by adopting models of different patch sizes together. It is quite similar to how CNN achieves this goal, by applying larger kernel or adding pooling layers. In particular, our multi-scale model consists of several parallel independent single-scale child models. The output of each child model is upsampled and then combined together to form the final estimation of velocity field v(θ)

v(θ) = N C=1 ω C Out C (A.2)
where Out C is the output of child model C. The final velocity field v is then passed to calculate the final transform ϕ as depicted in former subsections.

A.3 Experiments and Results

A.3.1 Dataset

To evaluate the effectiveness of our unsupervised registration models, we use a publicly accessible 2D echocardiography dataset CAMUS1 . This dataset consists of 500 patients, each has 2D apical 4-chamber (A4C) and 2-chamber (A2C) view sequences. Manual annotation of cardiac structures (left endocardium, left epicardium and left atrium) were acquired by expert cardiologists for each patient in each view, at end-diastole (ED) and end-systole (ES) [Leclerc, 2019a]. The structure annotations of 450 patients are public available while that of the other 50 patients unreleased. In total we have 1000 pairs of ED/ES images and we randomly split (still considering age and image quality distribution) the 900 pairs (with annotations) into training (630), validation(90) and test data (180). The 100 pairs (without annotations) are included into the training set (730).

A.3.2 Implementation

We compare our proposed three models with a very popular CNN registration model VoxelMorph [Balakrishnan, 2019]. To be consistent with our setting, we make use of the diffeomorphic version of VoxelMorph model (we use the abbreviation Vxm in the following paper). We train all the models with input images resized to 128x128 pixels and use an Adam optimiser (learning rate = 0.0001). We set training epoch to be 500 and training is early stopped when there is no improvement on validation set over 30 epochs. Our codes are available (https://gitlab.inria.fr/epione/mlp_ transformer_registration).

I f ix ≈ I move • ϕ(θ). For CAMUS test dataset, we report the Dice score, Hausdorff distance (HD) and mean surface distance (MSD) between ground truth ED mask and transformed ES mask and the Jacobian determinant in the area of myocardium region. ). What's more, single-scale models and multi-scale models have similar performance. With single-sized patches, we are already capable to let feature information flow through the whole image area and estimate registration transform efficiently (see time and space complexity in Table .A.1).

A.4 Conclusion

In summary, we propose three novel patch-based registration architectures using only MLPs and Transformers. We show that our single and multi-scale models perform similarly and even better to CNN-based registration frameworks on a large echocardiography dataset. The three proposed models demonstrate similar performance among themselves.

Our experiments show that patch-based models using MLP/Transformer can perform 2D medical image registration. We shared a similar conclusion with previous works [Liu, 2021a;Melas-Kyriazi, 2021] that the success of Transformer in vision tasks cannot be simply attributed to the attention mechanism, at least in image registration task. Future works will concentrate on the application of MLP/Transformer in time-series motion tracking.

List of Figures

(a)

An example of segmentation ground truth of CAMUS dataset. The two basal and apex landmarks were extracted following the procedures described in Section 2.2.1. (b) An example of annotation provided by ECHONET dataset. We generated the ground truth mask of LV by linearly connecting the border points. The grand axe (in red) was considered as the line connecting the apex and mid-basal point. The length of grand axe was considered as the LV length. (c-d) GLS calculation illustration (background is one echo image from CAMUS). We calculated the GLS from the LV length change by following the approximation method in [START_REF] Støylen | Left ventricular global strains by linear measurements in three dimensions: Interrelations and relations to age, gender and body size in the HUNT Study[END_REF]. . . . .

2.4

Randomly selected examples of augmented input images . . . . . . . . . .

  context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1.1 Point-of-care Ultrasound (POCUS) . . . . . . . . . . . . . . . . 3 1.1.2 echOpen Project . . . . . . . . . . . . . . . . . . . . . . .

Fig. 1

 1 Fig. 1.1.: Thesis motivation and aim: explore AI algorithms for portable cardiac function analysis

  Fig. 1.3.: Example views of echocardiography. (Illustration by Patrick J. Lynch and C. Carl Jaffe from Wikipedia commons, CC-BY license.)

Fig. 1 . 4 .:

 14 Fig. 1.4.: An overview of the conduction system of the heart. (Illustration by Open Stax Anatomy & Physiology, CC-BY license.)

Fig. 1 . 5 .

 15 Fig. 1.5.: 12-lead electrocardiogram lead axes. Illustration by Wikipedia Commons, CC-BY-SA license.

Fig. 1 . 6 .

 16 Fig. 1.6.: A typical example of ECG signal. Illustration by Agateller from Wikimedia commons, Public domain.

Fig. 1

 1 Fig. 1.7.: Myocardial infarction. Illustration by Blausen Medical Communications, Inc. from Wikimedia commons, CC-BY license.

  Fig. 1.8.: The road-map of manuscript organisation.

Fig. 2 .

 2 Fig. 2.1.: Detailed information of the 4 explored methods.

Fig. 2 . 2 .:

 22 Fig. 2.2.: Architecture of Proposed Poly-affine Regulariser

Fig. 2 .

 2 Fig. 2.3.: (a)An example of segmentation ground truth of CAMUS dataset. The two basal and apex landmarks were extracted following the procedures described in Section 2.2.1. (b) An example of annotation provided by ECHONET dataset. We generated the ground truth mask of LV by linearly connecting the border points. The grand axe (in red) was considered as the line connecting the apex and mid-basal point. The length of grand axe was considered as the LV length. (c-d) GLS calculation illustration (background is one echo image from CAMUS). We calculated the GLS from the LV length change by following the approximation method in[START_REF] Støylen | Left ventricular global strains by linear measurements in three dimensions: Interrelations and relations to age, gender and body size in the HUNT Study[END_REF].

  where G is a multiplicative Gaussian noise with σ in the range [0.01,0.1].

Fig. 2 .

 2 Fig. 2.4.: Randomly selected examples of augmented input images

Fig. 2 . 5 .

 25 Fig. 2.5.: 3 CAMUS segmentation examples (good/medium/bad in terms of HD). The four columns represent: the baseline model, SEG-LM, SEG-AFFINE, SEG-CONTOUR respectively, from left to right. The red, green, cyan lines represent: the predicted segmentation contours of epicardium, endocardium and left atrium. The transparent green, blue and yellow regions are the ground truth masks of myocardium, left ventricle blood pool and left atrium, respectively.

Fig. 2 . 6 .:

 26 Fig. 2.6.: The baseline U-Net model was trained with only one of the mentioned augmentation methods on CAMUS dataset and then was evaluated on the same test fold of ECHONET segmentation model, whose dice coefficient was 0.92. Contrast adjustment contributed most to the improvement of generalisation result, while with all the techniques we obtained the best Dice score and less variation. No-aug-5: trained model of 5 th epoch without augmentation, No-aug-30: trained model of 30 th epoch without augmentation. At 30 th epoch, the model had already over-fitted the CAMUS data.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.2.1 Polyaffine motion model . . . . . . . . . . . . . . . . . . . . . 38 3.2.2 Loss functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.3.2 Dataset preprocessing . . . . . . . . . . . . . . . . . . . . . . 43 3.3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3.4 Alation study . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.4.1 Registration accuracy . . . . . . . . . . . . . . . . . . . . . . . 44 3.4.2 Deformation regularity . . . . . . . . . . . . . . . . . . . . . . 46 3.5 Discussion and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.6 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.6.1 Cardiac motion transfer between sequences . . . . . . . . . . 50

Fig. 3

 3 Fig. 3.1.: Method overview.

  contained 109 4-chamber view echocardiography recordings with the segmentation of the left myocardium along one cardiac cycle (see Figure 3.2(d)).

Fig. 3 .

 3 Fig. 3.2.: Dataset overview. (a) EchoNet example and the given annotations of LV tracing. (b) Left ventricle cropping with the generated pseudo myocardium contour (blue). (c) The mean mask from the EchoNet training set and the 10 prior key points. (d) HMC-QU example and the given annotation of the myocardium. (e) CAMUS example and the given annotation of different cardiac structures. (MYO: myocardium, LV: left ventricle, LA: left atrium)

Fig. 3 .

 3 Fig. 3.3.: Registration results on HMC-QU dataset (109 A4C samples) using frame-wise myocardium masks. Original: Comparison between the ground truth masks along one cardiac cycle and that of end-diastole. Curves of all samples were interpolated to the same length. MYO: myocardium. HD: Hausdorff distance. MSD: mean surface distance.

Fig. 3 . 4 .:

 34 Fig. 3.4.: Evaluation results of Jacobian Determinant (Jac. Det.) and its gradient in the myocardium regionon on the CAMUS dataset and on the HMC-QU dataset. ES/ED represents the area ratio of the myocardium between end-systole and end-diastole obtained using ground truth masks.

Fig. 3 . 5 .:

 35 Fig. 3.5.: Examples of Jacobian determinant map in the myocardium region from HMC-QU dataset using CVAE and PAM methods.

  Fig. 3.6.:Visual results of a myocardial infarction sequence from HMC-QU dataset where SEG14 is annotated as infarcted region.

Fig. 3 .

 3 Fig. 3.7.: An example of transferring motion from sequences with large EF to sequences with small EF.

Fig. 3 .

 3 Fig. 3.8.: An example of transferring motion from sequences with small EF to sequences with large EF.

  2.3, and the motion tracking model PAM as introduced in Section 3.2. The segmentation model incorporated a shape prior through a contour loss. The motion tracking model integrated a motion prior through weakly-supervised myocardium key point position. A summary of the proposed method is illustrated in Figure 4.1.

Fig. 4 .

 4 Fig. 4.1.: Shape and motion priors for echocardiography analysis.

  Fig. 4.2.: Pipeline for interpretable cardiac features extraction.

4. 4

 4 Results and Discussion

Fig. 4 . 3 .:

 43 Fig. 4.3.: Distribution of global features between healthy (Non-MI) and MI patients from HMC-QU dataset. (Green * : p-value < 0.05 under the one-sided assumption using Wilcoxon rank-sum test.)

Fig. 4 . 4 .:

 44 Fig. 4.4.: Distribution of global features between patients with different number of infarct segments from HMC-QU dataset. Legend 0: non-MI, 12: MI patients with 1-2 infarct segments, 34: MI with 3-4 segments, 56: MI with 5-6 segments. Wilcoxon rank-sum test is performed between non-MI and others. (Green * : p-value < 0.05)

Fig. 4 . 5 .

 45 Fig. 4.5.: 5-fold cross validation result of MI detection using global features and SVM classifier. False: wrong classification, True: correct classification.

Fig. 4 .

 4 Fig. 4.6.: Distribution of global features between healthy (Non-MI) and MI patients from CHU dataset. (Green * : p-value < 0.05 under the one-sided assumption using Wilcoxon rank-sum test.)

Fig. 5 .

 5 Fig. 5.1.: (a) Pipeline of our proposed explainable ECG classification. (b)ω controls the kurtosis of the wave signal. (d) β controls the skewness of the wave signal.

Fig. 5 . 2 .:

 52 Fig. 5.2.: The detailed architecture of Cascaded FMMnet.

Fig. 5 .

 5 Fig. 5.3.: (a-b) The reconstruction metrics of different evaluation datasets on HC/MI separately. (c-d) Lead-wise R2 score of signal reconstruction of NORM patients and MI patients.

Fig. 5 .

 5 Fig. 5.4.:An example of signal decomposition for a 12-lead ECG and the corresponding reconstruction signals.

Fig. 5 . 5 .:

 55 Fig. 5.5.: The classification boundary (hyperplane) of trained linear SVM classifiers and data points (3-dim) projected on one of the 2D plane orthogonal to the corresponding hyperplane.

  Fig. 5.6.: Explanations of feature importance for Myocardial Infarction (MI), Anterior Myocardial Infarction (AMI), Inferior Myocardial Infarction (IMI) and Lateral Myocardial Infarction (LMI) classification respectively using SHAP value on models trained on PTB-XL dataset. Higher shap value helps to augment the chances of detecting positive classes, in our cases, the MI/AMI/IMI/LMI classes.

Figure 5 .

 5 6, the model distinguished the V1,V2,V3 leads for AMI (Figure 5.6b), and the II,III,avF leads for IMI (Figure 5.6c). Since the MI label combined MI of different localisations, the important features for MI/NORM classification were spread widely across different leads (Figure 5.6a).

Fig. 5 .

 5 Fig. 5.7.: An example from PTB-XL test dataset classified as MI by the linear SVM classifier/Logistic regression classifier. According to the PTB-XL description, the patient was diagnosed with old anteroseptal infarction (tiny R waves present in V2,V3 leads), anterolateral ischaemia (inverted T waves and depressed ST-segments in I, avL, flat T waves in V5,V6 leads) and inferior infarction (flat T wave in II lead). (a) Decomposed 12-lead median ECG. (b)(c) Explainability provided by linear SVM model. Red rectangles mark the T-ST change related leads (acute infarction/ischaemia related) and blue rectangles mark the R wave change related leads (old infarction related and are represented here by our parameters prefixed by Q). (d) Explainability provided by Logistic regression model using SHAP value.

  Fig. 5.8.: The general pipeline for ECG digitization.

Fig. 5 .

 5 Fig. 5.9.: Different formats of ECG papers and some signal extraction examples.

  Specifically, let us consider a K classification problem, where SL assigns K belief masses b k to each class and an overall uncertainty mass u to the whole frame using the evidence e k ≥ 0. Evidence e k represents a measure of the amount of support for kth class category collected from data input. The belief and uncertainty mass values are computed as follows: the K + 1 mass values is one, i.e. u + K k=1 b k = 1.

  Fig. 6.1.: Examples of Dirichlet distribution (K=3). The values of α are listed above each figure.

  Fig. 6.2.: Comparison of conventional fusion strategies and uncertainty based fusion.

•

  Max fusion: p c = max{p k c , k = 1, ..., K}, ȳ = arg max c p c ; • Mean fusion: p c = mean{p k c , k = 1, ..., K}, ȳ = arg max c p c ; • Rank fusion: ȳ = ( k ȳk ) ≥ 1; • Multiply fusion: p c = k p k c , ȳ = arg max c p c .

Fig. 6 . 3 .:

 63 Fig. 6.3.: The change of prediction accuracy with respect to uncertainty threshold on PTB-XL ECG dataset and HMC-QU ECHO dataset (2CH/4CH mixed). Bar plots represent the percentage of samples kept under varying uncertainty thresholds.

Fig

  Fig.A.1). The MLP-Mixer block has an identical structure as the MLP block, but with feature map transpose to obtain channel-wise feature fusion (the red cell of Block II in Fig.A.1). We note this model MLPMixer for abbreviation in the following paper.

Fig. A. 1 .:

 1 Fig. A.1.: The detailed composition of proposed three frameworks. Here we only show singlescale models. Please read Section.A.2.2 for more description.

  Fig. A.2.:Comparison of evaluation metrics (Dice score, Hausdorff distance (HD), mean surface distance (MSD) and Jacobin determinant) on test dataset of CAMUS. The Jacobin determinant is only computed in the myocardium region. Except Jacobin determinant figure, the higher the boxplot is in the figure, the better performance it will be.

  examples (good/medium/bad in terms of HD). The four columns represent: the baseline model, SEG-LM, SEG-AFFINE, SEG-CONTOUR respectively, from left to right. The red, green, cyan lines represent: the predicted segmentation contours of epicardium, endocardium and left atrium. The transparent green, blue and yellow regions are the ground truth masks of myocardium, left ventricle blood pool and left atrium, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.6The baseline U-Net model was trained with only one of the mentioned augmentation methods on CAMUS dataset and then was evaluated on the same test fold of ECHONET segmentation model, whose dice coefficient was 0.92. Contrast adjustment contributed most to the improvement of generalisation result, while with all the techniques we obtained the best Dice score and less variation. No-aug-5: trained model of 5 th epoch without augmentation, No-aug-30: trained model of 30 th epoch without augmentation. At 30 th epoch, the model had already over-fitted the CAMUS data. . . . . . . . . 3.1 Method overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Dataset overview. (a) EchoNet example and the given annotations of LV tracing. (b) Left ventricle cropping with the generated pseudo myocardium contour (blue). (c) The mean mask from the EchoNet training set and the 10 prior key points. (d) HMC-QU example and the given annotation of the myocardium. (e) CAMUS example and the given annotation of different cardiac structures. (MYO: myocardium, LV: left ventricle, LA: left atrium) 3.3 Registration results on HMC-QU dataset (109 A4C samples) using framewise myocardium masks. Original: Comparison between the ground truth masks along one cardiac cycle and that of end-diastole. Curves of all samples were interpolated to the same length. MYO: myocardium. HD: Hausdorff distance. MSD: mean surface distance. . . . . . . . . . . . . . . . . . . . . 3.4 Evaluation results of Jacobian Determinant (Jac. Det.) and its gradient in the myocardium regionon on the CAMUS dataset and on the HMC-QU dataset. ES/ED represents the area ratio of the myocardium between end-systole and end-diastole obtained using ground truth masks. . . . . . . . . . . . . 3.5 Examples of Jacobian determinant map in the myocardium region from HMC-QU dataset using CVAE and PAM methods. . . . . . . . . . . . . . . 3.6 Visual results of a myocardial infarction sequence from HMC-QU dataset where SEG14 is annotated as infarcted region. . . . . . . . . . . . . . . . 3.7 An example of transferring motion from sequences with large EF to sequences with small EF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 An example of transferring motion from sequences with small EF to sequences with large EF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4.6 Distribution of global features between healthy (Non-MI) and MI patients from CHU dataset. (Green * : p-value < 0.05 under the one-sided assumption using Wilcoxon rank-sum test.) . . . . . . . . . . . . . . . . . . . . . . . . 5.1 (a) Pipeline of our proposed explainable ECG classification. (b)ω controls the kurtosis of the wave signal. (d) β controls the skewness of the wave signal. 5.2 The detailed architecture of Cascaded FMMnet. . . . . . . . . . . . . . . . . 5.3 (a-b) The reconstruction metrics of different evaluation datasets on HC/MI separately. (c-d) Lead-wise R2 score of signal reconstruction of NORM patients and MI patients. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 An example of signal decomposition for a 12-lead ECG and the corresponding reconstruction signals. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 The classification boundary (hyperplane) of trained linear SVM classifiers and data points (3-dim) projected on one of the 2D plane orthogonal to the corresponding hyperplane. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Explanations of feature importance for Myocardial Infarction (MI), Anterior Myocardial Infarction (AMI), Inferior Myocardial Infarction (IMI) and Lateral Myocardial Infarction (LMI) classification respectively using SHAP value on models trained on PTB-XL dataset. Higher shap value helps to augment the chances of detecting positive classes, in our cases, the MI/AMI/IMI/LMI classes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 An example from PTB-XL test dataset classified as MI by the linear SVM classifier/Logistic regression classifier. According to the PTB-XL description, the patient was diagnosed with old anteroseptal infarction (tiny R waves present in V2,V3 leads), anterolateral ischaemia (inverted T waves and depressed ST-segments in I, avL, flat T waves in V5,V6 leads) and inferior infarction (flat T wave in II lead). (a) Decomposed 12-lead median ECG. (b)(c) Explainability provided by linear SVM model. Red rectangles mark the T-ST change related leads (acute infarction/ischaemia related) and blue rectangles mark the R wave change related leads (old infarction related and are represented here by our parameters prefixed by Q). (d) Explainability provided by Logistic regression model using SHAP value. . . . . . . . . . . 5.8 The general pipeline for ECG digitization. . . . . . . . . . . . . . . . . . . 5.9 Different formats of ECG papers and some signal extraction examples. . . 6.1 Examples of Dirichlet distribution (K=3). The values of α are listed above each figure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . prediction accuracy with respect to uncertainty threshold on PTB-XL ECG dataset and HMC-QU ECHO dataset (2CH/4CH mixed). Bar plots represent the percentage of samples kept under varying uncertainty thresholds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 The detailed composition of proposed three frameworks. Here we only show single-scale models. Please read Section.A.2.2 for more description. . . . .

  3 Thesis motivation . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Clinical context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 1.2.1 Echocardiography . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 Electrocardiogram . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.3 Myocardial infarction . . . . . . . . . . . . . . . . . . . . . . . 9 1.3 Methodological context . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.3.1 Automatic echocardiography analysis . . . . . . . . . . . . . . 10 1.3.2 Automatic electrocardiography analysis . . . . . . . . . . . . . 11 1.3.3 Multi-modal learning . . . . . . . . . . . . . . . . . . . . . . . 12 1.4 Manuscript Organisation and Contributions . . . . . . . . . . . . . . . 12 1.5 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

  Yingyu Yang † , Maxime Sermesant, and Hervé Delingette. "Unsupervised Echocardiography Registrationthrough Patch-based MLPs and Transformers". In: International Workshop on Statistical Atlases and Computational Models of the Heart. Springer. 2022, pp. 168-178 † These authors contributed equally to this work. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

	Automatic segmentation of	2
	echocardiography images
	Contents
	2.1 2.2.1 SEG-LM: Parallel segmentation and landmark detection . . . . 21
	2.2.2 SEG-AFFINE: Poly-affine Regulariser for Myocardium . . . . . 21
		1.5 Publications

Yingyu Yang, Rocher Marie, Moceri Pamela, and Maxime Sermesant. "Shape and Motion Priors for Generalisable Echocardiography Analysis using Deep Learning". 2023. Paper in preparation for submission.

Conference papers

•

[START_REF] Yang | Shape constraints in deep learning for robust 2D echocardiography analysis[END_REF]

Yingyu Yang and Maxime Sermesant. "Shape constraints in deep learning for robust 2D echocardiography analysis". In: International Conference on Functional Imaging and Modeling of the Heart. Springer. 2021, pp. 22-34 • [Yang, 2023b] Yingyu Yang and Maxime Sermesant. "Unsupervised PolyaffineTransformation Learning for Echocardiography Motion Estimation". In: International Conference on Functional Imaging and Modeling of the Heart. Springer. 2023, pp. 384-393 • [Yang, 2022] Yingyu Yang, Marie Rocher, Pamela Moceri, and Maxime Sermesant. "Explainable Electrocardiogram Analysis with Wave Decomposition: Application to Myocardial Infarction Detection". In: InternationalWorkshop on Statistical Atlases and Computational Models of the Heart. Springer. 2022, pp. 221-232 • [Wang, 2022] Zihao Wang † , 2.2.3 SEG-CONTOUR: multi-class contour-loss . . . . . . . . . . . . 23 2.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.3.3 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . 27 2.3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.5 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2.5.1 Evaluation on CAMUS test data . . . . . . . . . . . . . . . . . 32 2.5.2 Evaluation on local private dataset . . . . . . . . . . . . . . . 33 2.5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

.2.: Landmark/GLS and EF Prediction on CAMUS training data (450 patients, 10-fold cross validation

  Baseline SEG-LM SEG-AFFINE SEG-CONTOUR

		Dice	0.931 ± 0.040	0.928 ± 0.042	0.930 ± 0.042	0.931 ± 0.041
	Endo	HD	5.04	5.47	5.14	4.99
		(mm)	± 3.00	± 3.04	± 3.00	± 2.95
		MSD	1.51	1.59	1.53	1.50
		(mm)	± 0.83	± 0.84	± 0.88	± 0.73
		Dice	0.951 ± 0.025	0.950 ± 0.025	0.951 ± 0.027	0.952 ± 0.026
	Epi	HD	5.75	6.22	5.84	5.63
		(mm)	± 3.61	± 3.74	± 3.75	± 3.32
		MSD	1.71	1.79	1.72	1.67
		(mm)	± 0.91	± 0.93	± 0.95	± 0.87
		Geo.	15%	20%	14.3%	13.8%
	Outlier	Ana.	2.8%	7.6%	5.1%	1.5%
		Both	2.5%	5.3%	3.7%	1.2%
	Basal1	MAE (mm)	2.36	2.82	2.35	2.25
	Basal2	MAE (mm)	3.06	3.45	2.97	2.80
	Apex	MAE (mm)	4.06	4.48	4.22	3.97
		MAE (%)	3.97	4.51	4.03	3.96
	GLS(%)	Corr.	0.74	0.70	0.74	0.75
		Bias(%)	-0.73	-1.74	-0.9	-0.36
		± std	± 5.20	± 5.27	± 5.15	± 5.34
		MAE (%)	4.76	5.26	4.96	5.06
	EF(%)	Corr.	0.86	0.84	0.85	0.84
		Bias(%)	0.93	2.51	1.16	0.7
		± std	± 7.07	± 8.04	± 7.31	± 7.50
	Tab. 2

Tab. 2.1.: Segmentation Metric on CAMUS training data (450 patients, 10-fold cross validation) Endo.: endocardium, Epi: epicardium, HD: Hausdorff distance, MSD: mean surface distance, Geo.: Geometrical outlier, Ana.: Anatomical outlier. Values in bold represent the best score. Method Baseline SEG-LM SEG-AFFINE SEG-CONTOUR ) Basal1: the left mitral valve end point, Basal2: the right mitral valve end point, EF: ejection fraction, GLS: global longitudinal strain. Values in bold represent the best score.

Table 2

 2

.1 and 2.2 include the results computed from 450 patients of CAMUS dataset using 10 fold cross-validation for the four methods detailed in Section 2.2. Compared with the

Tab. 2.3.: ECHONET Prediction

	2.5 Appendix
	2.5.1 Evaluation on CAMUS test data
		ED			Dice	Endo HD	MSD	Dice	Epi HD	MSD
	U-Net[Leclerc, 2019a] 0.939 5.3	1.6	0.954	6.0	1.7
	ACNNs[Oktay, 2017] 0.936 5.6	1.7	0.953	5.9	1.9
	CLAS[Wei, 2020]		0.947 4.6	1.4	0.961	4.8	1.5
	nnUNet[Ling, 2022]	0.952 4.3	1.4	0.963	4.6	1.5
		Ours			0.949 4.8	1.4	0.961	5.0	1.6
	Method Baseline SEG-LM SEG-AFFINE SEG-CONTOUR 0.892 0.886 0.887 0.895 Endo Epi ES Dice HD MSD Dice HD MSD Dice ± 0.069 ± 0.063 ± 0.068 ± 0.057 HD 12.99 13.54 13.12 12.56 U-Net[Leclerc, 2019a] 0.916 5.5 LV 1.6 0.945 6.1 1.9
	(pxls) ACNNs[Oktay, 2017] 0.913 5.6 ± 7.96 ± 6.86	± 7.25 1.7 0.945	± 6.34 5.9 2.0
	MSD CLAS[Wei, 2020]	3.74	3.87 0.929 4.6	3.87 1.4	0.955	3.58 4.9	1.6
	(pxls) nnUNet[Ling, 2022] ± 3.42 0.935 4.2 ± 2.63 MAE (%) 7.97 8.79 Ours 0.930 4.7	± 2.97 1.3 0.959 9.35 1.4 0.955	± 2.16 4.4 1.5 8.39 5.0 1.6
	EF	Corr.		0.69	0.72 EDV	0.67 ESV	0.69 EF
		Bias(%)		4.46	6.21 Corr. MAE Corr. 7.11	MAE	5.03 Corr. MAE
	± std U-Net[Leclerc, 2019a] 0.926 11.2 0.960 ± 12.80 ± 13.39 ± 14.41 7.5 MAE (%) 4.96 5.61 5.88 ACNNs[Oktay, 2017] 0.928 9.7 0.954 6.9	± 13.21 0.845 5.0 4.39 0.807 5.5
	GLS CLAS[Wei, 2020] Corr.	0.38	0.36 0.958 7.7 0.979 0.29	4.4	0.51 0.926 4.0
	Bias(%) nnUNet[Ling, 2022] 0.36	-1.9 0.977 5.9 0.987 -0.75	4.0	0.59 0.857 4.7
		± std Ours	± 7.94 0.965 6.4 0.983 ± 7.07 ± 9.29	4.6	± 6.52 0.912 4.4
		Method				Geometrical Anatomical
		R-Unet [Leclerc, 2019b]		16%	1.2%
		U-Net [Leclerc, 2019a]		18%	3.75%
		Ours				15.4%	1.15%
								2.4 Conclusion

Tab. 2.4.: Segmentation evaluation on CAMUS test split data (50 patients).

Tab. 2.5.: CAMUS Segmentation shape outliers evaluated on 500 patients (2,000 images in total) with geometrical and anatomical outliers.

  Registration evaluation on EchoNet-Dynamic test split. Endo: endocardium. HD: Hausdorff distance. MSD: mean surface distance. Without alternative training or fine-tuning, we applied directly the trained motion model on unseen dataset CAMUS training set. Table 3.2 summarizes the generalisation performance of our proposed PAM model and CVAE model. Our PAM model not only generalized well on CAMUS dataset when compared with the backbone FOMM and the state-of-art CVAE, but also achieved the closest registration performance with the best tracking model (OTA+OTS)

	Method	Endo Dice HD(pixels) MSD(pixels)
	CVAE [Krebs, 2020]	0.91	7.97		2.30
	FOMM [Siarohin, 2019]	0.75	22.99		6.07
	FOMM+Polyaffine	0.82	11.50		4.61
	FOMM+Prior	0.77	18.36		5.46
	FOMM+Prior+Polyaffine	0.91	7.53		2.36
	FOMM+Prior+Polyaffine+ED 0.92	7.34		2.23
	PAM(all)	0.92	7.17		2.14
	Tab. 3.1.: Epi	Endo
	Method	Dice	MSD (mm)	Dice	MSD (mm)
	OTA+OTS [Wei, 2020]	0.933	2.0	0.898	1.8
	CVAE [Krebs, 2020]	0.917	3.0	0.859	3.3
	FOMM [Siarohin, 2019]	0.883	4.3	0.778	5.2
	FOMM+Polyaffine	0.881	4.3	0.796	4.8
	FOMM+Prior	0.893	3.8	0.804	4.3
	FOMM+Prior+Polyaffine	0.932	2.5	0.885	2.6
	FOMM+Prior+Polyaffine+ED 0.933	2.4	0.894	2.4
	PAM(all)	0.927	2.6	0.883	2.6

Tab. 3.2.: Registration evaluation on CAMUS training set (450 patients with 2CH and 4CH samples). Endo: endocardium. Epi: Epicardium. MSD: mean surface distance. 3.4.1.2 CAMUS

  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4.1.1 Myocardial infarction detection in echocardiography . . . . . 56 4.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.2.1 Shape and motion priors for echocardiography analysis . . . . 57 4.2.2 Multi-view myocardial infarction detection . . . . . . . . . . . 58

	Echocardiography analysis pipeline	4
	Contents
	4.1

4.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3.2 Experiments and implementation . . . . . . . . . . . . . . . . 61 4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.4.1 HMC-QU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4.4.2 CHU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

  Three public datasets (i.e., CAMUS, EchoNet, HMCQU) and one private dataset (i.e., CHU) with 2D echocardiography data were included in our study.

	4.3.1 Datasets
	Tab. 4.1.: Datasets inncluded in this study
	Dataset	Training Task		Pathology	A2C	View A4C	2+4
	CAMUS	Segmentation		-	500	500	500
	EchoNet motion estimation	-	-	10030	-
	HMC-QU	Classification	MI/Non-MI 130	162	130
	CHU	-	MI/Non-MI 66	76	64
	Tab. 4.2.: Datasets with pathology diagnostics
		Dataset	MI Non-MI Total
		HMC-QU 88	42	130
		CHU	42	22	64
						4.3 Experiments

  Contents 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 5.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.2.1 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . 69 5.2.2 Cascaded FMMnet . . . . . . . . . . . . . . . . . . . . . . . . 70

	Abstract Automatic analysis of electrocardiograms with adequate explainability is
	a challenging task. Many deep learning based methods have been proposed for
	automatic classification of electrocardiograms. However, very few of them provide
	detailed explainable classification evidence.
	In this chapter, we explore explainable ECG classification through explicit decompo-
	sition of single-beat (median-beat) ECG signal. In particular, every single-beat ECG
	sample is decomposed into five subwaves and each subwave is parameterised by a
	Frequency Modulated Moebius. Those parameters have explicit meanings for ECG
	interpretation. In stead of solving the optimisation problem iteratively which is time-
	consuming, we design a Cascaded CNN network to estimate the parameters for each
	single-beat ECG signal. Our preliminary results show that with appropriate position
	regularisation strategy, our neural network is able to estimate the subwave for P, Q,
	R, S, T events and maintain a good reconstruction accuracy (with R2 score 0.94 on
	test dataset of PTB-XL) in a unsupervised manner. Using the estimated parameters,
	we achieved very good classification and generalisation performance on myocardial
	infarction detection on four different datasets. Features of high importance are in
	accordance with clinical interpretations.

5.3 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.3.2 Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . 73 5.3.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5.4 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . 81 5.5 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.5.1 Paper ECG digitization . . . . . . . . . . . . . . . . . . . . . . 82

  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 6.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 6.2.1 Single modality evidential deep learning . . . . . . . . . . . . 87 6.2.2 Multi-modal fusion with uncertainty . . . . . . . . . . . . . . 89

	Multi-modal detection of myocardial	6
	infarction: uncertainty-based fusion
	using echocardiography and
	eletrocardiogram
	Contents
	6.1
		5.5 Appendix

6.3 Experiments and results . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.3.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 6.3.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

Single) MLPMixer Model (Single) SwinTrans Model (Single)

  . The MLP-Mixer block has an identical structure as the MLP block, but with feature map transpose to obtain channel-wise feature fusion (the red cell of Block II in Fig.A.1). We note this model MLPMixer for abbreviation in the following paper. This model uses the MLP block (Block I in Fig.A.1) to first extract patch based features for both I f ix and I mov . For cross feature block, we adapt the recent Swin Transformer[Liu, 2021b] to do the cross-patch attention locally (Swin Block in Fig.A.1). Our Swin block accepts feature input from both images (I f ix and I mov ), where key K and value V are normalised I f ix features while query Q comes from normalised I mov features. Swin block calculates the cross-attention within a pre-defined window region. We perform normal window partition for I f ix features and one normal partition, one shifted partition for I mov features. The cross-attention under the two types of window partition configurations are summed together before feeding to the final linear layers. Due to the page limit, we invite interested readers to refer to[Liu, 2021b] for detailed description of Swin transformer mechanism. We note this model SwinTrans for abbreviation in the following paper.

	A.2.2.3 Swin-Transformer registration framework
	I_fix Patches ... General framework I_mov Patches ...	Linear Linear GELU MLP Block	MLP Block + LayerNorm Block I	Block I PureMLP Model (Feature extractor	Block I Cross Feature Block +
	Feature	Feature				LayerNorm
	extractor	extractor				MLP Block
			Swin Block I_fix	I_mov	+		Feature extractor	Cross Feature Block +
	Cross Block Feature		V LayerNorm K	Q LayerNorm	LayerNorm Block II	Block II	Block II
	Linear Tanh		(Shift) Window Multi-head Cross Attention +	+	MLP Block	Transpose	Feature extractor	Cross Feature Block
	Linear			LayerNorm	LayerNorm
			+	MLP Block	MLP Block +	Block I	Swin Block + Shifted Swin Block

  Thesis motivation and aim: explore AI algorithms for portable cardiac function analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 An overview of cardiovascular circulation system. (Illustration by Open Stax Anatomy & Physiology, CC-BY license.) . . . . . . . . . . . . . . . . . . . . 1.3 Example views of echocardiography. (Illustration by Patrick J. Lynch and C. Carl Jaffe from Wikipedia commons, CC-BY license.) . . . . . . . . . . . . 1.4 An overview of the conduction system of the heart. (Illustration by Open Stax Anatomy & Physiology, CC-BY license.) . . . . . . . . . . . . . . . . . 1.5 12-lead electrocardiogram lead axes. Illustration by Wikipedia Commons, CC-BY-SA license. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 A typical example of ECG signal. Illustration by Agateller from Wikimedia commons, Public domain. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Myocardial infarction. Illustration by Blausen Medical Communications, Inc. from Wikimedia commons, CC-BY license. . . . . . . . . . . . . . . . . . . 1.8 The road-map of manuscript organisation. . . . . . . . . . . . . . . . . . . 2.1 Detailed information of the 4 explored methods. . . . . . . . . . . . . . . . 2.2 Architecture of Proposed Poly-affine Regulariser . . . . . . . . . . . . . . .

	1.1

1.2 Clinical context

https://wwwn.cdc.gov/NHISDataQueryTool/SHS_adult/index.html 1.2 Clinical context

1.3 Methodological context

1.4 Manuscript Organisation and Contributions

Chapter 1 Introduction

2.2 Methods

https://www.creatis.insa-lyon.fr/Challenge/camus/databases.html

https://echonet.github.io/dynamic/

2.3 Experiments and Results

Chapter 2 Automatic segmentation of echocardiography images

2.5 Appendix

3.1 Introduction

https://echonet.github.io/dynamic/

https://www.creatis.insa-lyon.fr/Challenge/camus/

https://www.kaggle.com/datasets/aysendegerli/hmcqu-dataset

3.3 Experiments

3.4 Results

3.5 Discussion and conclusion

3.6 Appendix

4.4 Results and Discussion

4.5 Conclusion

5.3 Experiments and Results

5.4 Discussion and Conclusion

Chapter 6 Multi-modal detection of myocardial infarction: uncertainty-based fusion using

6.2 Method

6.4 Conclusion

7.2 Future research

https://www.creatis.insa-lyon.fr/Challenge/camus/

A.4 Conclusion

Bibliography

Acknowledgements

First and foremost, I would like to express my gratitude to my supervisor, Maxime Sermesant. Since my first internship with you, your trust and encouragement have been invaluable in shaping my confidence and interest in research. Your positive attitude towards research has left a lasting impact. I sincerely appreciate your support and guidance for each of my research projects during my thesis.

Financial Support

The research leading to these results has been supported by the Inria PHD funding and the National Research Agency (ANR) 3IA Côte d'Azur (ANR-19-P3IA-0002). The project was also supported by the Inria Sophia Antipolis -Méditerranée, "NEF" computation cluster. can bring superior performance in pattern recognition tasks. In contrast, whether the superior performance of the Transformer comes from the long-winded architecture or is attributed to the use of patches for dividing the inputs is unclear yet. This work introduces three patch-based frameworks for image registration using MLPs and transformers. We provide experiments on 2D-echocardiography registration to answer the former question partially and provide a benchmark solution. Our results on a large public 2D-echocardiography dataset show that the patch-based MLP/Transformer model can be effectively used for unsupervised echocardiography registration. They demonstrate comparable and even better registration performance than a popular CNN registration model. In particular, patch-based models better preserve volume changes in terms of Jacobian determinants, thus generating robust registration fields with less unrealistic deformation. Our results demonstrate that patch-based learning methods, whether with attention or not, can perform high-performance unsupervised registration tasks with adequate time and space complexity. This chapter was a joint work with Zihao Wang and was published in the International Workshop on Statistical Atlases and Computational Models of the Heart (STACOM) [START_REF] Wang | [END_REF].

Appendix

A.1 Introduction

Image registration is essential for clinical usage; for example, the registration of cardiac images between end-diastole and end-systole is meaningful in myocardium deformation

A.2 Methodology

A.2.1 Diffeomorphic Registration

We estimate a diffeomorphic transform between images, which preserves topology and is folding-free. Our model generates stationary velocity field v(θ) [Ashburner, 2000] instead of generating displacement maps, thanks to an integration layer applied to the velocity field leading to diffeomorphism ϕ(θ). Formally, the diffeomorphic transformation ϕ is solution to a differential equation related to the predicted (stationary) velocity field V [Dalca, 2019]: ∂ϕt ∂t = v(ϕ t ); ϕ t=0 = Id. In a stationary velocity field setting, the transformation ϕ is defined as the exponential of the velocity field ϕ = exp(v) [Arsigny, 2006]. The integration (exponential) layer applies the scaling and squaring method to approximate the diffeomorphic transform [Krebs, 2018]. The obtained transformation ϕ is then used by a spatial transform layer to deform the image.

A.2.2 Proposed frameworks

Given two images, I f ix and I mov , we would to estimate the transformation ϕ(θ) that transforms the moving image to the fixed image so that I f ix ≈ I mov •ϕ(θ). We approximate the ideal ϕ(θ) by the following proposed frameworks. The following three propositions are all based on patch-wise manipulations and share a similar general architecture. As shown in Fig. A.1, I f ix and I mov are both processed by an identical feature extractor (green block) separately. The two feature maps are then passed through the cross feature block (blue block). After two linear layers, we obtain their corresponding velocity field. The velocity field passes through an integration layer and we obtain the final displacement field by upsampling it to the original image size.

A.2.2.1 Pure MLP registration framework

The same MLP block (Block I in Fig. A.1) are used for feature extractor and the cross feature block in this model. The outputs from two separate feature extractor (shared weights) are added together before feed into the cross feature block. We note this model PureMLP for abbreviation in the following paper.

A.2.2.2 MLP-Mixer registration framework

The MLP-Mixer registration framework is very similar to the former Pure MLP framework. The only difference is that the three MLP blocks used for separate feature extraction and cross feature processing are replaced by MLP-Mixer blocks [mlpmixer] (Block II in

Loss function

In order to enforce the diffeomorphic property of our registration model, we apply a symmetric loss function for all the unsupervised models:

where φ-1 is the inverse of φ and L dif f is a diffusion regularizer for smoothness L dif f = ||∇ x ϕ + ∇ y ϕ|| 2 and set λ = 0.01 according to [Balakrishnan, 2019].

A.3.2.2 Data augmentation

In order to improve model generalisation and avoid outfitting, we apply the same random data augmentation tricks for each image pair during training phase. (abbreviation: model name + _S) we run single-scale models for PureMLP, MLPMixer and SwinTrans three proposed frameworks (using patch size of 4x4 pixels). The same configuration is set as for child model with patch-size 4x4 in multi-scale models.

A.3.4 Results

Since our SwinTrans model relies mostly on features of I f ix (with skip-connection of I f ix features), we report only the metrics related to the transformation ϕ(θ) that

A.3 Experiments and Results

List of Tables