BRIE, and Saïd MOUSSAOUI, for their very valuable comments on this thesis. Especially, I would like to thank Prof. Charles SOUSSEN who his invaluable and numerous comments had a significant impact

1.7 Forward problem is solved when the head (scalp, skull, and brain layers), source (K source positions), and sensor (m sensors) models are given. The lead-field matrix Φ of dimension m by 3K is resulted from concatenation of K three-column matrices corresponding to K source positions in source space. . . 12 1.8 (a) MEG measures vectorial magnetic field, whereas EEG measures scalar potential differences, (b) MEG sensors are located outside the head, whereas EEG sensors are placed on the surface of the head, (c) MEG is mostly sensitive to primary currents, whereas EEG is sensitive to secondary currents, (d) MEG has very low sensitivity to radial dipoles, (e) MEG has low sensitivity to deep sources, (f) EEG is highly sensitive to the geometry and conductivity of media, and 3.12 Brain source space segmentation, when clustering the coherent blocks of lead-field, repeated for two modalities of EEG and MEG, and three head models with spherical, realistic inflated and realistic highly-folded cortical sheets. . . .

3.13

The proposed subject-specific brain source space segmentation, using (a) EEG, and (b) MEG lead-fields, compared to (c) the conventional anatomical and functional brain lobes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1

The rows of (a) EEG lead-field Φ EEG and (b) MEG lead-field Φ MEG can be combined together to form (c) the new multi-modal lead-field Φ EMEG . . . . . .

4.2

The role of (a) ε inter and ε intra in generation of random dictionary Φ 1 , and (b) ε inter-dictionary in generation of Φ 2 to produce the multi-modal dictionary Φ 12 .

4.3 Block-SL 2,2 (Φ) [%] for each level of clustering is computed for the complete method, Block-MCC 2,2 , d=2, N ={4, 5}, ε inter-dictionary =1, and different values of ε inter and ε intra for simulating dictionary Φ∈R 10×80 . . . . . . . . . . . . . . .

x List of figures 4.4 (a) spherical, (b) realistic inflated and (c) realistic highly-folded cortical sheets for the head models used in EEG and MEG multi-modality experiment. . . . . 4.5 EEG and MEG complementarity in brain source segmentation using Block-MCC 2,2 and complete method for some clustering levels and for (a) spherical, (b) realistic inflated, and (c) realistic highly-folded cortical sheets. . .

Mathematical notations and nomenclature

Mathematical notations Throughout the manuscript, we represent matrices, vectors and scalars by boldface uppercase characters (e.g., A), boldface lowercase italic characters (e.g., a), and lowercase italic characters (e.g., a), respectively.

The mathematical notations used in the manuscript are shown in table 1. To better formulate the block-wise indexing, let P be a partitioning of indices of elements in vector a or indices of columns in matrix A. Then suppose P i indicates the i th partition and P i (j) indexes the j th element in P i .

As an example of block-wise indexing, take vector b∈R 5 , matrix A∈R 2×5 , and the partitioning P={{1, 2}, {3, 4}, {5}}. Then, b[2] is concatenation of the elements b 3 and b 4 , whereas A[2] is concatenation of the columns a 3 and a 4 , and a 1 [2] is the column a 3 .

Operators

A T

Transpose of matrix A A † Moore-Penrose pseudo-inverse of matrix A I d

Identity matrix in R d×d 1 d 1 ×d 2 Matrix of ones in R d 1 ×d 2 Element-wise indexing a i i th element in vector a a i i th column in matrix A a i,j

The element in the i th row and j th column in the matrix A Block-wise indexing

P

Partitioning of indices a[i]

i th block of elements in vector a, i.e.,

a[i]=[a k | k∈P i ] A[i]
i th block of columns in matrix A, i.e.,

A[i]=[a k | k∈P i ] a j [i] j th column in block A[i], i.e., a j [i]=[a k | k∈P i (j)]

Introduction

Many problems in different areas of engineering and science, such as inverse problems, can be reformulated as an underdetermined system of linear equations, i.e., the number of equations is less than the number of unknowns. In order to retain an appropriate unique solution from infinitely many solutions to a such system, problem-related constraints need to be applied. According to Tropp, for more than one century sparsity constraints have been studied and applied to numerous applications [START_REF] Aaron | Topics in sparse approximation[END_REF].

However, if a comprehensive measurement is not provided, attaining a desired solution will not be feasible, even considering optimal constraints. This is mainly due to the rich characteristics of the phenomenon of interest that makes it impossible for a single modality to project all aspects of the phenomenon. Therefore, the main research question of this dissertation is: What is the added value of multi-modality, when solving inverse problems?

To answer the above-mentioned question, three main challenges will raise. First of all, it should be noticed that the desirable sparse solution can be extracted from a set of possible solutions through a proper constrained optimisation problem.

In order to ensure the uniqueness of the sparsest solution of the optimisation problem, some theoretical recovery conditions are proposed in literature. These conditions are based on the amount of similarity or coherence between the columns of coefficient matrix of the inverse problem. More precisely, the less coherence, the more relaxed are the conditions under which recovery is guaranteed to be successful.

For vastly underdetermined systems of linear equations, the columns of coefficient matrix are more likely to be coherent. This gives rise to the following first challenge: Challenge 1. Many real-world inverse problems are vastly underdetermined, and classical sparse estimation techniques do no longer give acceptable recovery conditions. How can high-dimensional problems be adapted in favour of the coherence-based notion of conventional conditions? conditions does no longer hold true. How can appropriate recovery conditions be developed for block-structured problems? Finally, assuming that the two mentioned challenges are addressed successfully, and returning to the main research question, the last challenge is related to the impact of multi-modality: Challenge 3. Is joining multiple modalities always beneficial, knowing that each modality provides us with different properties of the same phenomenon? How can the added value of multi-modality be demonstrated?

In order to explore and address the three aforementioned challenges, this dissertation is organized in three main parts. Since Challenge 1 uses the results of Challenge 2, first Challenge 2 is presented in the dissertation, then Challenge 1, and ultimately in the last part a partial answer to Challenge 3 is provided through experiments.

The first part of this thesis offers general theoretical recovery conditions based on block-sparsity constraint, which ensure the uniqueness of the block-sparse solution of corresponding weighted (pseudo-)mixed-norm optimisation problem in an underdetermined system of linear equations. This part responds to Challenge 2.

The second part of the thesis suggests the clustering idea to improve the general theoretical recovery conditions proposed in the first part, which responds to Challenge 1.

Consequently, the third part of the thesis investigates Challenge 3, while considering the constraint in Challenge 2, i.e., block-sparsity. Our main application problem is distributed EEG/MEG source reconstruction.

Given a linear operator relating the current density to the observed potential/magnetic fields, which is called lead-field, the inverse problem in EEG/MEG source reconstruction solves a vastly underdetermined system of linear equations.

In EEG/MEG source reconstruction problem, two concepts are integrated: block-sparsity, and multi-modality. In the following, we elucidate the existence of the two mentioned concepts. There is the notion of block activity, because the activity of each brain source is related to a current dipole which can be represented by three magnitudes in x, y, and z directions, respectively, and can be modelled by a vector of dimension three. Hence, the source vector will be a concatenation of groups or blocks of length three. Moreover, usually for a given brain task, a few regions of brain are activated, then it is consistent with the notion of block-sparsity. be used as a leading example.

Considering the properties of the real-world distributed EEG/MEG source reconstruction problem, which require few number of active blocks in the source vector, in Chapter 2 a general framework for block-sparse recovery problem is proposed. In other words, this chapter gives a partial answer to "In an underdetermined system of linear equations, under which conditions a unique block-sparse solution can be recovered?".

In fact, in a wide range of problems, the value of a single coefficient inside a block of coefficients, whether zero or non-zero, is not important, but the status of the whole block which is determined by all of its coefficients is the atomic meaningful entity. Therefore, instead of the sparsity constraint which penalizes the non-zero coefficients, a block of coefficients should be penalized.

Moreover, in contrast to the conventional point of view, where, the number of non-zero values of coefficients determines the unique solution, in the block-structured problems the status of block of coefficients (active or inactive) has an effective role. In addition, depending on the definition of a block, an active block can also have zero coefficients.

The mentioned generality of the framework of Chapter 2 is in terms of the properties of the underdetermined system of linear equations, extracted characterisations, optimisation problems, and ultimately the recovery conditions. This chapter offers four main groups of recovery conditions based on:

1. Block-Spark, 2. block null space property, 3. block mutual coherence constant, and 4. cumulative coherence constant.

It is worth mentioning that utilising the block structure, besides the conformity with circumstances of some of the real-world problems, has the advantage of weakening the conventional sparse recovery conditions. Then, the benefit of block-sparsity assumption over conventional sparsity is proved theoretically, while it is shown that due to the natural generalisation, all the proposed materials reduce to the conventional findings for the unit block length. Finally, we prove the improvement of the proposed conditions over other recently proposed block-sparse recovery conditions.

As mentioned implicitly, the lead-field is a fat matrix. For instance, in an ordinary EEG/MEG source reconstruction problem with 30 sensors and 3000 sources, the lead-field would be a matrix of dimensions 30 by 9000 (3000 sources times 3 values in x, y, and z directions).

Since the mentioned block mutual coherence constant introduced in Chapter 2 is a coherence characterisation extracted from the lead-field matrix, and due to the fact that the low coherence is favourable, the idea of clustering coherent parts of lead-field is studied in Chapter 3. The clustering algorithm is applied in a hierarchical manner which enables us to estimate the number of clusters according to the resulted clustering tree.

To construct the similarity matrix in hierarchical clustering algorithm, the proposed coherence characterisation introduced in Chapter 2 is used. By applying clustering on the lead-field matrix we will have two main achievements:

1. improved block-sparse recovery conditions, and 2. brain source space segmentation.

To explain more about the brain source space segmentation, it should be mentioned that each consecutive block of three columns of the lead-field matrix corresponds to a single brain source, so by clustering the blocks of the lead-field matrix we are actually clustering brain sources.

It should be highlighted that in contrast to other existing brain source space segmentation scenarios, in this study this segmentation is done in the most general case, i.e., the clustering algorithm is blind to any information about the brain sources activity and EEG/MEG signals.

In other words, the segmentation is not restricted to a special brain activity.

In Chapter 4, the goal is to show the effect of combining the information of two complementary modalities of EEG and MEG. We show that in multi-modal case more refined and precise brain regions appear. Hence, EEG and MEG multi-modality advantage will be proved.

Extensive experiments on synthetic and real data indicate a significant improvement over the state-of-the-art findings in our three main research orientations.

Background material

Introduction

This chapter presents a background on the methods to face two challenges of multi-modality, in Section 1.2, and block-sparsity. In order to review the recent studies on block-sparsity, we start from the conventional sparsity constraint in finding a unique solution of an USLE1 in Section 1.3. Then, a review of recent studies on block-sparsity is presented in Section 1.4. At last, the role of block-sparsity in multi-modality is discussed in Section 1.5.

6

Chapter 1. Background material 1.2 Multi-modality

Introduction

Many natural phenomena consist of very complex structural and functional properties. For instance, considering the sophisticated example of human brain as the phenomenon, the electric, magnetic, functional and anatomical properties can be taken into account.

On the other hand, each modality or measurement instrument measures only a specific property of the phenomenon. Then, due to the rich characteristics of a natural multi-property phenomenon, a comprehensive information cannot be obtained by only a single modality. Therefore, the research domain of multi-modality revealed to jointly analyse the data collected from different modalities which are somehow complementary. The concept of multi-modality is shown in figure 1.1 in a simple language. As shown in figure 1.1, a natural multi-property phenomenon is represented by a multi-coloured object, where, each colour represents a specific property of the object. In figure 1.1, modalities are been represented by eyes that are observing the multispectral object. Then, the fact of shortcoming of a single modality to observe all the Figure 1.1: Mono-modal dataset cannot describe the multispectral phenomenon of interest, due to the monospectral detectability of each modality. While by combining mono-modal datasets, an acceptable description of the phenomenon can be recovered.

properties of the phenomenon is represented by the blindness of each of the eyes to a range of colours. Finally, the data observed from each of the eyes can be combined to build a correct image of the multi-coloured object of interest, which is the ultimate goal in multi-modality, i.e., compensating each modality's relative shortcomings by integrating other modality's dataset, in order to extract the maximum available information. In multi-modality, the combined dataset is called multi-modal dataset in contrast to the dataset of a single modality which is called mono-modal or uni-modal dataset. The interest in the multi-modality originates from the idea that the information in the multi-modal dataset is more than the sum of the information in each of individual mono-modal datasets.

In our study, the multi-property phenomenon of interest is the human brain activity and we are interested in its electromagnetic properties as represented in figure 1.2. In neurophysiology, a wide variety of modalities are used for bioelectric measurements, among which the EEG is the most interesting from the clinical applications point of view, thanks to the following four main advantages: (1) non-invasive, (2) high temporal resolution (in order of milliseconds), (3) relatively cost-effective, and (4) easy to use and portable.

In addition, for biomagnetic measurements the MEG is commonly used. As represented graphically in figure 1.2, fusion of EEG and MEG can be used to improve the quality of source estimation, due to the fact that multi-modality can also reduce the ill-posedness of problems and make them better determined [START_REF] Velmurugan | Combined MEG-EEG Source Localisation in Patients with Sub-Acute Sclerosing Pan-Encephalitis[END_REF]. In the Section 1.2.2, we review the electrophysiological properties of neuronal populations. In Section 1.2.3, the required steps to solve the EEG/MEG source reconstruction problem is discussed. Finally, Section 1.2.6 elucidates some complementarities of EEG and MEG.

Physiological basis of EEG and MEG

The main direct neuronal source of EEG and MEG signals is pyramidal neuron. Task of a pyramidal neuron is to take synaptic inputs and produce patterned output of action potential [START_REF] Okada | Empirical Bases for Constraints in Current-Imaging Algorithms[END_REF]. As can be seen in figure 1.3, the postsynaptic dendrite part is polarized due to the concentration discrepancy of ions. When the action current reaches the presynaptic axon terminal, the glands of neurotransmitters are released into the synapse and bind to the receptors on the postsynaptic dendrite, and this leads to a flow of ions of N a + from outside of cell to inside, and producing post synaptic current as shown in figure 1.3. As shown in figure 1.3, by spreading this ions, intracellular current (primary current) and extracellular current (secondary current) are produced. A small patch of brain cortex which comprises thousands of these simultaneously activated parallel pyramidal neurons can be represented by a current dipole [START_REF] Cohen | Magnetoencephalography (Neuromagnetism)[END_REF].This relationship between the macroscopic representation and the corresponding microscopic neuronal activity is important [START_REF] Yoshio | Genesis of MEG Signals in a Mammalian CNS Structure[END_REF]. These pyramidal neurons are oriented perpendicular to the brain cortex, but since the cortex is folded, 'radial', 'tangential', and 'oblique' dipole orientations defined with respect to the local curvature of the skull, must be considered.

EEG/MEG forward and inverse problems

The goal in EEG/MEG source analysis is finding the appropriate sources based on the electric/magnetic measurements, which is also called the inverse problem. It is called an inverse problem because it exploits the information at the results stage and then leads to the causes.

Inverse problems are not specific to the EEG/MEG source analysis research field and are considered as one of the most important mathematical problems in science. The importance of the inverse problems is due to the fact that by solving the problems inversely, it would be possible to discover the latent parameters, which are not measurable directly.

In order to solve inverse problem, one must first solve the so-called forward problem, i.e., what would be the electromagnetic fields of a known source? This is the inverse of a inverse problem, which exploits the information of the causes and then leads to the results.

In figure 1.4, the forward problem is to determine EEG/MEG measurements in sensor space resulted from three active dipoles in source space, whereas, the inverse problem is to estimate the activity of dipoles in source space given EEG/MEG measurements in sensor space.

Figure 1.4: The forward and inverse problems in EEG/MEG source analysis.

Volume conduction head model

In bioelectromagnetism, the transmission of electromagnetic fields from a current dipole through the head tissues towards measurement sensors is called volume conduction head model, which contains the electrical conduction properties of the head. In frequencies below 1 kHz, the quasi-static approximation of Maxwell's equations can be used for modelling the volume conduction head model [START_REF] Plonsey | Considerations of Quasi-Stationarity in Electrophysiological Systems[END_REF]. Therefore, the level of realism of the volume conduction head model determines the quality of EEG/MEG source analysis [START_REF] Haueisen | Influence of Volume Conductor Modeling on Source Reconstruction in Magnetoencephalography and Electroencephalography[END_REF].

The volume conduction head model can be modelled analytically, or numerically. In other words, for the EEG/MEG, the head can be modelled analytically by a single sphere or three or even four concentric spheres corresponding to brain, cerebrospinal fluid, skull, and scalp [START_REF] Hosek | The Contributions of Intracerebral Currents to the EEG and Evoked Potentials[END_REF]. The boundary element methods and finite element methods have been developed to numerically model the head and to better represent the realistic shape of the head [START_REF] Yan | Finite-Element Model of the Human Head: Scalp Potentials Due to Dipole Sources[END_REF]; [START_REF] Akalin-Acar | An Advanced Boundary Element Method (BEM) Implementation for the Forward Problem of Electromagnetic Source Imaging[END_REF]. Although utilising numerically-modelled head models lead to increased accuracy in the source localization problem, it is computationally heavy. Therefore, based on the requirements of the problem a trade-off between computational burden and source localization accuracy should be taken into account.

In figure 1.5, the three-layer analytical and numerical volume conduction head models are shown. The three layers correspond to scalp, skull, and brain, respectively. The head model in figure 1.5(a) is computed analytically, whereas the ones in (b) and (c) are computed numerically. The realistic brain layer in figure 1.5(b) is an inflated cortical sheet, whereas in figure 1.5(c) it is a highly-folded cortical sheet.

Naturally, in order to build realistic head models we need to structural information of head, which is provided by MRI2 . Figure 1.5: Three-layer (scalp, skull, and brain) head models modelled (a) analytically, and (b,c) numerically. In (b), the cortical sheet is inflated, whereas in (c) it is highly-folded.

Lead-field

The transmission of electromagnetic fields from sources space through head tissues towards sensors space can be modelled with quasi-static approximations of Maxwell's equations [START_REF] Sarvas | Basic Mathematical and Electromagnetic Concepts of the Biomagnetic Inverse Problem[END_REF].

The quasi-static approximations of Maxwell's equations, which allows to ignore the time derivatives, can be divided into two groups of:

• quasi-electrostatics, which are describing electric field E, and • quasi-magnetostatics, which are describing magnetic field B.

The mentioned quasi-static approximations with electric potential V and magnetic field B in distance r in an infinite homogeneous medium with permittivity ε 0 and permeability µ 0 for a dipole p with electric charge density ρ and electric current density J are shown in figure 1.6, and table 1.1.

The quasi-static approximations of Maxwell's equations in addition to the assumed discrete positions of the brain sources lead to a discretised and linearised relationship between electromagnetic signals and the source activities. The mentioned relationship is realised through the so-called lead-field matrix which contains the electromagnetic and geometrical properties of the head.

Quasi-electrostatics

Quasi-magnetostatics Gauss's law

∇ • E = ρ ε 0 Gauss's law ∇ • B = 0 Faraday's law ∇ × E = 0 Ampère's law ∇ × B = µ 0 J V = 1 4πε 0 p • r |r| 3 B = µ 0 4π p × r |r| 3
where, ∇•, ∇×, •, and × are divergence, curl, dot and cross product operators, respectively.

Table 1.1: The quasi-static approximations of Maxwell's equations.

The lead-field matrix, which is also considered as the solution of the forward problem, is computed when the volume conduction head model, source model, and sensor model is given.

As shown in figure 1.7, knowing the volume conduction head model including scalp, skull, and brain models with K source positions in source space and m sensors in sensor space makes possible the computation of the lead-field matrix of dimension m by 3K.

Each block of three columns in lead-field matrix, consists of the response of all sensors to a specific probing dipole in the source space. Actually, it is a block of three canonical probing vectors.

Figure 1.7: Forward problem is solved when the head (scalp, skull, and brain layers), source (K source positions), and sensor (m sensors) models are given. The lead-field matrix Φ of dimension m by 3K is resulted from concatenation of K three-column matrices corresponding to K source positions in source space.

Multi-modality 13

EEG and MEG complementarities

In recent years, the partial independence of bioelectric and biomagnetic measurements has been studied and the added value of their combination under a multi-modal signal processing framework has been the subject of some researches. In this section, some relative merits of cerebral bioelectromagnetic measurements, i.e., EEG and MEG, are reviewed.

• Dimension of measurement. MEG measurement is vectorial, whereas EEG measurement is scalar.

• Reference measurement. MEG measures magnetic field, whereas EEG measures potential differences. In other words, the existence of reference electrode is essential in EEG contrary to MEG (figure 1.8(a)).

• Sensor position. MEG sensors are located outside but near the head, whereas EEG sensors are placed on the surface of the head. Therefore, MEG makes a contact-less recording indicating that it has the advantage of very low pre-recording preparation time and the disadvantage of necessity of existence of helmet for localization of the head position (figure 1.8(b)).

• Sensitivity to cerebral currents. MEG is mostly sensitive to primary currents, whereas EEG is sensitive to secondary currents (figure 1.8(c)). The cerebral currents were briefly discussed in Section 1.2.2.

• Sensitivity to orientation of dipole. MEG has very low sensitivity to radial dipoles and is zero for spherical head model [START_REF] Seppo | Sensitivity of MEG and EEG to Source Orientation[END_REF] (figure 1.8(d)).

• Sensitivity to depth of dipole. MEG has low sensitivity to deep sources, because deeper sources become more quasi-radial (figure 1.8(e)).

• Sensitivity to head tissues. In contrast to MEG, EEG is highly sensitive to the geometry and conductivity of media [START_REF] Acar | Sensitivity of EEG and MEG to conductivity perturbations[END_REF]; [START_REF] Nevzat | Sensitivity of EEG and MEG Measurements to Tissue Conductivity[END_REF]. Therefore, EEG is distorted while passing through the brain tissues, especially the skull because of its low conductivity, i.e., high resistivity.

On the other hand, MEG is far less sensitive to internal heterogeneities [START_REF] Hari | MEG-EEG Primer[END_REF]. For instance in figure 1.8(f), for a same source model, which is realistic highly-folded cortical sheet, MEG is quite equal for the two head models, while EEG is significantly different.

Because of the transparency of the skull to magnetic fields, MEG is able to measure the cerebral activity of smaller brain regions [START_REF] Malmivuo | Comparison of the Properties of EEG and MEG in Detecting the Electric Activity of the Brain[END_REF].

• Topography. EEG and MEG topographies for a same source activity are almost orthogonal to each other (figure 1.8(g)). Because of these infinitely many solutions, the problem is said ill-posed [START_REF] Hadamard | Sur les problèmes aux dérivés partielles et leur signification physique[END_REF]. According to some a priori knowledge about the nature of the data of interest, and consequently the solution, this eligible infinite number of solutions, which results in ambiguity in solution space, could be restricted to a smaller class of solutions or, pragmatically, to a unique solution, which is a good approximation to a true solution.

Commonly, the amount of sparsity of the true solution has been exploited as additional constraint to disappear the mentioned ill-posedness of the USLE Sparsity assumption is a generic constraint that its application in different domains can lead to solutions with different properties, e.g., smoothness of the solution. In this work, the sparsity assumption is utilised. In fact, the sparsity assumption is a very relevant constraint due to the practical observation indicating that many real world signals have approximately sparse representations.

If the coefficient matrix of USLE is union of orthonormal bases, the solution is efficient, where efficiency in solution means requiring very few significant coefficients in comparison to the dimension of the solution, so sparsity [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF]; [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]. But practically it has been observed that because of the rich characteristics of the natural phenomena, if the coefficient matrix is a single orthonormal basis, usually the natural images or sounds do not necessarily have sparse solution [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]. Therefore, the more generalised concept of coefficient matrix is introduced which is the so-called dictionary [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF]. Each column of a dictionary is called atom. For explanation of the atom-dictionary terminology, the interested reader is referred to [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF] and [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF]. The coefficient matrix appears in different names of design matrix, measurement matrix, coding matrix, codebook, or dictionary, based on its nature and the research community.

In the community of overcomplete signal representations, finding the sparse solution of a vastly USLE is translated as recovering a high-dimensional ideally sparse vector, which can be a digital signal or image. In addition, it can be translated as an efficient representation of a signal in an overcomplete dictionary. Overcompleteness of the dictionary implies that the linear system of equations is underdetermined, because the number of atoms in the dictionary is more than the number of entries in each of the atoms. However, despite of the fact that the problem is underdetermined, there are some heuristic and theoretical arguments on benefits of overcompleteness in theoretical neuroscience [START_REF] Olshausen | Sparse coding with an overcomplete basis set: A strategy employed by V1?[END_REF], approximation theory [START_REF] Candès | New Tight Frames of Curvelets and Optimal Representations of Objects with C2 Singularities[END_REF], signal processing [START_REF] Debrunner | Lapped multiple bases algorithms for still image compression without blocking effect[END_REF]; [START_REF] Berg | A survey of mixed transform techniques for speech and image coding[END_REF]; [START_REF] Cotter | Sparse channel estimation via matching pursuit with application to equalization[END_REF], and image processing [START_REF] Huo | Sparse Image Representation Via Combined Transforms[END_REF]; [START_REF] Starck | The curvelet transform for image denoising[END_REF]; [START_REF] Starck | Image Decomposition: Separation of Texture from Piecewise Smooth Content[END_REF].

Section 1.3.2 defines the sparse representation theory, next in Section 1.3.3, recovery conditions are reviewed.

Sparse representation theory

To elucidate the theory, this section is divided into three parts: (1) structure and model, (2) optimisation problem, and (3) characterisations:

1) Structure and model In general, the USLE can be formulated as [START_REF] Candès | Decoding by linear programming[END_REF]; [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF]:

y = Φβ 0 + e, (1.1)
where, y∈R m is the measurement vector, Φ∈R m×n , m<n, is the dictionary, β 0 ∈R n is the true representation vector, or input variable, and e∈R m is noise, bounded by a known noise level, e.g., e 2 < . if e=0, then the model is noiseless [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]. In this study, we will concentrate on the following noiseless model:

y = Φβ 0 . (1.2)
As mentioned before, the goal is to extract a representation vector β which is the sparsest among all solutions, i.e., a representation with the fewest non-zero elements relative to its dimension. For example, suppose the following estimated representation vector β:

β = β1 , • • • , βk , • • • , βn T ,
and following dictionary:

Φ = [ϕ 1 , • • • , ϕ k , • • • , ϕ n ] ,
with ϕ j ∈R m and without loss of generality, it is assumed that atoms have unit Euclidean norm, i.e., ∀j, ϕ j 2 =1. Then, the sparsity structure of the noiseless USLE is graphically represented in figure 1.9. Commonly, this problem is referred to as atomic decomposition, since the measurement vector y is decomposed into its fewest active atoms of the dictionary [CDS01]; [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF].

Figure 1.9: Sparsity structure of the noiseless USLE.

Sparsity

2) Optimisation problem The problem of finding a sparse solution β in the noiseless case is called exact (unique) signal recovery whereas for the problem in the noisy case, it is called stable (robust) signal recovery.

In general, exact signal recovery provides deep insight into the problem of stable signal recovery [START_REF] Candès | Decoding by linear programming[END_REF]; [START_REF] Candès | The Dantzig selector: statistical estimation when p is much larger than n[END_REF]; [START_REF] David | For Most Large Underdetermined Systems of Linear Equations the Minimal 1 -norm Solution is also the Sparsest Solution[END_REF]; [START_REF] David | For most large underdetermined systems of equations, the minimal 1 -norm near-solution approximates the sparsest near-solution[END_REF]. Exact signal recovery remains invariant by scaling of the dictionary, Φ→kΦ, whereas stable signal recovery has a direct relation with scaling of the dictionary [START_REF] Davies | Restricted Isometry Constants where p sparse recovery can fail for 0 < p ≤ 1[END_REF]; [START_REF] Foucart | Sparsest solutions of underdetermined linear systems via q -minimization for 0 < q ≤ 1[END_REF].

Stablity in stable signal recovery means that bounded changes in the measurement vector y result in bounded changes in the estimated representation vector β [START_REF] Candès | Stable signal recovery from incomplete and inaccurate measurements[END_REF]. Based on the amount of the bounded change either small or arbitrary, there are two main types of stability named local stability and global stability, respectively [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF].

The best case in stable signal recovery is obtained when the size of the representation error is guaranteed to be bounded by the noise level, i.e., β-β 0 2 < . In a real-world situation, usually the measurement is severely incomplete and also inaccurate and when the measurement vector is noisy and the true representation vector is not exactly sparse, the measurement is said to be imperfect [START_REF] Candès | Stable signal recovery from incomplete and inaccurate measurements[END_REF].

Norms play an essential role in quantifying distances between two vectors or amplitude of a single vector in the definition of cost functions for optimisation problems [START_REF] Walter | Numerical Methods and Optimization -A Consumer Guide[END_REF]. Exact and stable signal recovery problems can be realised through a proper constrained optimisation problem. The straightforward optimisation problem for stable signal recovery is:

min β β 0 s.t. y -Φβ 2 ≤ ε, (1.3)
where, for ε=0, the optimisation problem (1.3) solves an exact signal recovery problem. In a general framework in order to cover all the previous types of optimisation problems, consider the following constrained p norm minimisation problem:

P p,q,ε (a, b) : min a a p s.t. b q ≤ ε, (1.4)
where, a and b are vectors, and p norm of vector a is defined:

a p def =                  i I (a i ) , for p = 0 i |a i | p 1 p , for 0 < p < +∞ max i {|a i |}, for p = +∞,
where, I(•) is the indicator function, i.e., I(a

) def = 1, if |a| >0 0, if a=0
. The noise level , which upper-bounds the Euclidean norm of noise e in the general model of (1.1) is bounded such that 0≤ ≤ε. Therefore, according to (1.4) the problem (1.3) can be represented as P 0,2,ε (β, y-Φβ).

By defining r(β)=y-Φβ as a remainder vector, it is assumed in this work as a consequence that r(β) q =0 is equivalent to y=Φβ. For the sake of simplicity we refer to r(β) as r. The estimated solution of the problem P p,q,ε (a, b) is called β. In the introduced general optimisation problem, the widely used case is when the minimisation is over β and the problem is an exact signal recovery, i.e., P p,q,0 (β, r), which for the sake of simplicity we refer to it as P p . Because regardless of choice of q, the r q =0 directly indicates to the noiseless model, so the only parameter to be determined is p.

A norm function f has three following properties [START_REF] Golub | Matrix Computations[END_REF]; [START_REF] Walter | Numerical Methods and Optimization -A Consumer Guide[END_REF]:

1. f (a) ≥ 0 (nonnegativity), 2. f (αa) = |α|f (a) (positive homogeneity or scalability), 3. f (a ± b) ≤ f (a) + f (b) (triangle inequality).
Some of the most important norms that their minimum induces sparsity in exact and stable signal recovery problems are:

• 0 pseudo-norm (or semi-norm, or Hamming norm [START_REF] Hurley | Comparing Measures of Sparsity[END_REF]),

• 1 norm (or taxicab norm, or Manhattan norm, or grid norm [START_REF] Walter | Numerical Methods and Optimization -A Consumer Guide[END_REF]),

• 2 norm (or Euclidean norm), and • ∞ norm (or maximum norm, or Chebyshev norm, or uniform norm).

Remark 1.1. Notice that, a p for p=0 does not satisfy the required positive homogeneity property of a norm function, i.e., αa 0 =|α| a 0 . Therefore a 0 is commonly referred to as a 0 pseudo-norm instead of 0 norm. a 0 simply counts the number of non-zero elements in a.

Remark 1.2. It should be noticed that, a p for 0<p<1 does not satisfy the required triangle inequality property of a norm function. Therefore a p for 0<p<1 is commonly referred to as a p pseudo-norm instead of p norm.

For a vector β, the basic sparsity measure is the 0 pseudo-norm [START_REF] Hurley | Comparing Measures of Sparsity[END_REF], which is the number of non-zero entries of the vector β. Therefore, P 0 which is an equality-constrained 0 pseudo-norm minimisation problem can be used for the exact signal recovery problem [START_REF] Candès | Decoding by linear programming[END_REF]. Similarly, P 0,2,ε (β, r) which is an 2 -constrained 0 pseudo-norm minimisation problem can be used for the stable signal recovery problem [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF], where, both P 0 and P 0,2,ε (β, r) are in general NP-hard, i.e., they cannot be solved by a polynomial time algorithm [START_REF] Natarajan | Sparse Approximate Solutions To Linear Systems[END_REF]; [START_REF] Amaldi | On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems[END_REF].

In finding the unique and sparse representations, some researchers focused on a more general cases of P p , 0<p≤1 [Cha07]; [START_REF] Davies | Restricted Isometry Constants where p sparse recovery can fail for 0 < p ≤ 1[END_REF]; [START_REF] Foucart | Sparsest solutions of underdetermined linear systems via q -minimization for 0 < q ≤ 1[END_REF]; [START_REF] Davies | The Restricted Isometry Property and p sparse recovery failure[END_REF], and P p , 0≤p≤1 [GN03b]; [START_REF] Gribonval | On the Strong Uniqueness of Highly Sparse Representations from Redundant Dictionaries[END_REF]; [START_REF] Gribonval | Highly sparse representations from dictionaries are unique and independent of the sparseness measure[END_REF] for the exact signal recovery problem and P p,2,ε (β, r), 0<p≤1 [START_REF] Foucart | Sparsest solutions of underdetermined linear systems via q -minimization for 0 < q ≤ 1[END_REF] for the stable signal recovery problem.

For P p,q,ε (β, r), 0≤p<1, the optimisation problem is non-convex, combinatorial, and generally computationally intractable [START_REF] Davis | Adaptive greedy approximations[END_REF], i.e., all possible combinations of the columns of the dictionary should be tested for finding the smallest subset of columns satisfying the optimisation constraint, and its complexity grows exponentially with n the length of β. But there is still hope, either by approximating the solution of P p,q,ε (β, r), 0≤p<1, and relaxing the optimisation problem to P 1,q,ε (β, r), which can be viewed as a convexification of P p,q,ε (β, r), 0≤p<1, or using numerical bypasses (shortcuts), such as greedy algorithms, e.g., matching pursuit [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF]; [START_REF] Mallat | A Wavelet Tour of Signal Processing[END_REF], orthogonal matching pursuit [START_REF] Pati | Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition[END_REF]; [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF]; [START_REF] Davis | Adaptive greedy approximations[END_REF], and basis pursuit [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF], or iterative thresholding algorithms, e.g., iterative hard thresholding [START_REF] Figueiredo | An EM Algorithm for Wavelet-Based Image Restoration[END_REF]; [START_REF] Daubechies | An Iterative Thresholding Algorithm For Linear Inverse Problems With A Sparsity Constraint[END_REF]; [START_REF] Blumensath | Iterative Thresholding for Sparse Approximations[END_REF]; [START_REF] Blumensath | Iterative hard thresholding for compressed sensing[END_REF], iterative soft thresholding, and iterative thresholding with inversion [START_REF] Maleki | Coherence Analysis of Iterative Thresholding Algorithms[END_REF]; [START_REF] Maleki | Optimally Tuned Iterative Reconstruction Algorithms for Compressed Sensing[END_REF].

Matching pursuit as an iterative algorithm can be used for approximating the P 0 , whereas basis pursuit leads to the exact solution to the P 0 . At first, in practice it had been observed that P 1 and P 0 often lead to the same results [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF], later it turned out that under sufficient conditions, i.e., if β is sufficiently sparse, the solutions of the mentioned problems are indeed the same and unique [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Elad | On sparse signal representations[END_REF]. Regarding the conditions for the relationship between the solutions of P 1 and P 0 , Donoho and Elad have addressed the following two main questions [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF]: Under which conditions, Q1: the solution to P 1 is necessarily the sparsest representation (solution to P 0 ) (Uniqueness)? Q2: the solution to P 0 is necessarily equal to the solution to P 1 (Equivalence)?

In general, P 1,q,ε (a, b) is convex, implying that there is no local minima problems in its numerical solution, and based on the optimisation theory, it can be recast as a linear programming problem [START_REF] Gill | Numerical Linear Algebra and Optimization[END_REF]; [START_REF] Bertsekas | Nonlinear Programming[END_REF]; [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF]. Therefore, it can be efficiently solved by search problems based on either the classical simplex method or recently popular interior point methods [START_REF] Wyner | An analog scrambling scheme which does not expand bandwidth, Part II: Continuous time[END_REF]; [START_REF] Bertsekas | Convex Analysis and Optimization[END_REF]; [START_REF] Boyd | Convex Optimization[END_REF]. There are also some other fast algorithms [START_REF] Cormode | Combinatorial Algorithms for Compressed Sensing[END_REF]; [START_REF] Gilbert | Algorithmic linear dimension reduction in the 1 norm for sparse vectors[END_REF]; [START_REF] Gilbert | One Sketch for All: Fast Algorithms for Compressed Sensing[END_REF]; [START_REF] Xu | Efficient Compressive Sensing with Deterministic Guarantees Using Expander Graphs[END_REF]. Although P 1,q,ε (a, b) is convex, it is not strictly convex, so the range of unicity of the solution should be determined [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]. Because of the mentioned convexity, researchers started using P 1 or equality-constrained 1 norm minimisation problem or basis pursuit [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF]; [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Candès | Decoding by linear programming[END_REF] for exact signal recovery problem, and in parallel P 1,2,ε (β, r) or 2 -constrained 1 norm minimisation problem or basis pursuit denoising [START_REF] Scott Shaobing Chen | Atomic Decomposition by Basis Pursuit[END_REF]; [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF]; [START_REF] Tropp | Just relax: convex programming methods for identifying sparse signals in noise[END_REF], for stable signal recovery problem.

There are two other widely used optimisation problems used for stable signal recovery: P 2,1,ε (r, β), and P 1,∞,ε (β, Φ T r). The former, 1 -constrained 2 norm minimisation problem is LASSO3 , and is also known as 1 -regularized least squares [START_REF] Tibshirani | Regression Shrinkage and Selection Via the Lasso[END_REF]; [START_REF] Tropp | Just relax: convex programming methods for identifying sparse signals in noise[END_REF]; [START_REF] Koh | An Efficient Method for Large-Scale l1-Regularized Convex Loss Minimization[END_REF]; [START_REF] Meinshausen | Lasso-type recovery of sparse representations for high-dimensional data[END_REF] and the latter, ∞ -constrained 1 norm minimisation problem is known as Dantzig selector [START_REF] Candès | The Dantzig selector: statistical estimation when p is much larger than n[END_REF]; [START_REF] Cai | On Recovery of Sparse Signals Via 1 Minimization[END_REF]; [START_REF] Cai | Shifting Inequality and Recovery of Sparse Signals[END_REF].

3) Characterisations To define a framework for recovery or identifiability conditions, some properties and characterisations of the dictionary are introduced in literature, among which the widely studied are gathered in table 1.2. Most of the mentioned properties and characterisations of the dictionary are computationally unrealistic or intractable. Only the property of mutual coherence property or the characterisation of mutual coherence constant have the great advantage to be simple and tractable, but with the expense of making the recovery conditions more restrictive. Next, we recall some basic notations and characterisations which will be used in our work.

Support and Cardinality: The Support (or active set) and Cardinality of a vector β are defined as:

Support: S (β) def = {k : β k = 0} , Cardinality: |S (β)| def = β 0 .
Indeed, the non-convex pseudo-norm β 0 , simply counts the non-zero elements of vector β.

Dictionary Kernel: The Kernel (or null space) of a dictionary Φ is defined as:

Ker(Φ) def = {x ∈ R n , Φx = 0} .
The Kernel of a dictionary plays an important role in establishing two different classes of recovery conditions, which will be investigated later. Spark has a very important role in guaranteeing the uniqueness of the sparse representation. This importance has been already demonstrated by Gorodnitsky and Rao [START_REF] Gorodnitsky | Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm[END_REF].

The Spark of a dictionary Φ is defined mathematically as [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]; [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF]; [START_REF] Bruckstein | From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images[END_REF]:

Spark (Φ) def = min x∈Ker(Φ)\{0}
x 0 .

(1.5)

By the definition of the Spark(Φ), any vector x in the null space of the dictionary, i.e., Φx=0, must satisfy x 0 ≥Spark(Φ).

As it can be seen, Spark characterises the null space of a dictionary using the 0 pseudo-norm.

Despite of the superficial similarity between Rank and Spark, they are entirely different in concept and computational complexity.

The Rank of a dictionary is defined as the largest number of the columns of Φ which are linearly independent, and its computation is sequential (easy to compute), whereas Spark is the size of the smallest number of linearly dependent columns of Φ, and its computation is of combinatorial search with exponential complexity.

In general, without zero columns, the Spark of a dictionary Φ∈R m×n is bounded by [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]:

2 ≤ Spark(Φ) ≤ min {Rank(Φ) + 1, n} ≤ min {m + 1, n} = m + 1.
(1.6) A dictionary with maximal Spark, i.e., Spark(Φ)=m+1, is called full-Spark.

For a dictionary with linearly dependent columns, at least two columns are linearly dependent, i.e., 2≤Spark(Φ), and for many randomly generated dictionaries, Spark(Φ) is equal to Rank(Φ)+1.

On the other hand, we have Rank(Φ)≤ min{m, n}, and since the problem is underdetermined, i.e., m<n, min{m, n} is m and also min{m + 1, n}=m+1.

As it can be seen in (1.5), the dictionary characterization Spark(Φ) is obtained by minimisation of the intractable non-convex 0 pseudo-norm, which is not easy to compute. More precisely, given a matrix Φ, computing Spark(Φ) is NP-hard4 . The computational complexity of Spark(Φ) is investigated in [START_REF] Tillmann | The Computational Complexity of Spark, RIP, and NSP[END_REF].

22

Chapter 1. Background material Mutual coherence constant: The maximum pairwise absolute correlation between the normalized atoms of a dictionary can be used as a characterization of the dictionary, which is called Mutual Coherence Constant (MCC) [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]. From computational point of view, MCC can be considered as the maximum element of off-diagonal absolute Gram (Gramian) matrix

G(Φ) def = Φ T Φ [DE03b]; [DE03a], i.e., M (Φ) def = max k,k =k | ϕ k , ϕ k | = max k,k =k ϕ T k ϕ k = max k,k =k G k,k (Φ) ,
where, a, b computes the inner product of the vectors a and b. Obviously, the main diagonal of G(Φ) is composed of 1s, due to the 2 -normalization of the atoms of the dictionary. Since it is assumed that the atoms of the dictionary have unit 2 norm, MCC satisfies M (Φ)≤1, because of the Cauchy-Schwarz inequality

| ϕ k , ϕ k |≤ ϕ k 2 ϕ k 2 .
For a dictionary as an orthonormal basis, M (Φ)=0, whereas for a dictionary consisting two or m orthonormal bases, M (Φ) is bounded by [START_REF] Heath | On Quasi-Orthogonal Signatures for CDMA Systems[END_REF]:

1 √ m ≤ M (Φ) ≤ 1. (1.7)
For an equiangular tight frame deterministic dictionary, which satisfies the following three conditions [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF]:

ϕ k 2 = 1, for k = 1, • • • , n, | ϕ k , ϕ k | = c, ∀k = k and constant c, m n n k=1 ϕ, ϕ k ϕ k = ϕ, ∀ϕ ∈ R m .
the lower-bound of M (Φ) or the Welch bound is (n-m)/(m(n-1)) [START_REF] Welch | Lower Bounds on the Maximum Cross Correlation of Signals (Corresp.)[END_REF]; [START_REF] Strohmer | Grassmannian frames with applications to coding and communication[END_REF]. The Welch bound defined for equiangular tight frame deterministic dictionaries is even less than the lower-bound of M (Φ) for a general random dictionary in (1.7), unless m=1, for which the two lower-bounds are equal.

MCC was first computed for two orthonormal bases Φ 1 and Φ 2 , which is called basic MCC :

M (Φ 1 , Φ 2 ) = max k,k ϕ T 1 k ϕ 2 k .
Notice that for orthonormal bases Φ 1 and Φ 2 and their concatenation

[Φ 1 , Φ 2 ], we have M (Φ 1 , Φ 2 ) =M ([Φ 1 , Φ 2 ]). Supposing the dimension of two orthonormal bases Φ 1 and Φ 2 is m by m, it is proved that 1/ √ m≤M (Φ 1 , Φ 2 )≤1 [DH01]; [EB01]; [EB02].
The lower-bound is achieved when the pair of orthonormal bases are spikes and sines [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF] or Identity and Hadamard [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF] matrices or any other orthonormal matrices corresponding to the orthonormal bases, whereas the upper-bound is achieved when at least one of the columns in each of two orthonormal bases is common.

Sparsity

Cumulative mutual coherence constant: MCC uses the maximum absolute off-diagonal element of the Gram matrix G(Φ) as the characterization of the dictionary Φ, while by summing over any k elements of G(Φ), we would better characterize the dictionary. This kind of dictionary characterization is called cumulative MCC.

The conventional cumulative MCC, (or Babel function [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF], or 1 -coherence function [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF]) of a dictionary Φ∈R m×n is defined as:

M (Φ, k) def = max |Λ|=k max j / ∈Λ i∈Λ | ϕ i , ϕ j | , (1.8)
where,

Λ represents k different indices from {1, • • • , n}.
Although cumulative MCC is computationally more difficult than MCC characterization, it leads to more weakened or relaxed recovery conditions, i.e., the solution of the corresponding optimisation problem is ensured to be unique even for less sparse representation vectors.

A straightforward extension of 1 -coherence function would be p -coherence function, defined for any p>0 [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF]:

M p (Φ, k) def = max |Λ|=k max j / ∈Λ i∈Λ | ϕ i , ϕ j | p 1 p .
The p parameter in M p (Φ, k) controls the p norm of k off-diagonal elements of G(Φ), which makes the M p (Φ, k) to have the following properties:

• For p=1, M p (Φ, k) reduces to the conventional cumulative MCC, i.e., M 1 (Φ, k)≡M (Φ, k).

• For p=∞, for any value of k, M p (Φ, k) reduces to the conventional MCC, i.e., M ∞ (Φ, k)≡M (Φ), because ∞ norm of any vector is equal to the maximum absolute value of the vector.

The Welch bound mentioned for the MCC can be extended to the cumulative MCC, i.e., for k< √ n-1, cumulative MCC is lower-bounded by k (n-m)/(m(n-1)), where, again the lower-bound is achieved when the dictionary is an equiangular tight frame [START_REF] Schnass | Dictionary Preconditioning for Greedy Algorithms[END_REF].

In a similar work for extracting the cumulative coherence of the dictionary, Donoho and Elad introduced µ 1/2 and µ 1 of the Gram matrix, which is the smallest m off-diagonal entries in a single row or column of the Gram matrix G, which sums at least to 1/2 and 1, respectively.

There are the relationships M -1 (Φ)≤µ 1 (G)<Spark(Φ) and µ 1/2 (G)≤(1/2)µ 1 (G) [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF].

Chapter 1. Background material

In another study, based on the manner of identifying the Support set of a sparse signal in the OMP5 algorithm, union cumulative coherence is proposed and denoted by M U (Φ, k) [START_REF]Estimation de fonctions géométriques et déconvolution[END_REF]; [START_REF] Zhao | New Conditions for Uniformly Recovering Sparse Signals via Orthogonal Matching Pursuit[END_REF]:

M U (Φ, k) def = max |Λ|=k    max j / ∈Λ i∈Λ | ϕ i , ϕ j | + max l∈Λ i∈Λ\{l} | ϕ i , ϕ l |    , where, Λ represents k different indices from {1, • • • , n}.
M U (Φ, k) even better characterizes the dictionary Φ in comparison to conventional cumulative MCC, i.e., M (Φ, k). Because, a part from the first common term in M U (Φ, k), which seeks the maximum of sum of pairwise absolute inter-set correlations between the columns i∈Λ and j / ∈Λ, the second term in M U (Φ, k) seeks the same characterization for intra-set columns i∈Λ and l∈Λ.

Although M U (Φ, k) is computationally more complicated than M (Φ, k), it leads to more accurate analysis of the reconstruction capacity of the orthogonal matching pursuit [START_REF] Zhao | New Conditions for Uniformly Recovering Sparse Signals via Orthogonal Matching Pursuit[END_REF].

Tropp has shown the following properties for cumulative MCC [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF]:

• M (Φ, 0)=0 (by convention), • M (Φ, 1)=M (Φ), • M (Φ, k)≤k M (Φ), • M (Φ, k+1)-M (Φ, k)≥0,
• M (Φ, k+2)-2M (Φ, k+1) + M (Φ, k)≤0, ∀k≥0, and • For orthonormal basis, M (Φ, k)=0, ∀k≥0.

In a more general case, for 1≤k 1 , k 2 ≤n-1 with k 1 +k 2 ≤n-1, we have [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF]:

max {M (Φ, k 1 ), M (Φ, k 2 )} ≤ M (Φ, k 1 + k 2 ) ≤ M (Φ, k 1 ) + M (Φ, k 2 ).
As it can be learned from the above-mentioned different formula of cumulative version of the ordinary coherence measure, this type of dictionary characterisation is more informative and general. Since instead of the first maximum off-diagonal absolute value of the Gram matrix G(Φ), k of them characterise the dictionary Φ.

Sparsity 25

Exact recovery condition

One of the fundamental contributions of the overcomplete signal representations community is theoretically-proved necessary and sufficient conditions for exact or stable signal recovery problems. In other words, if based on some regularity conditions for sparsity assumption the representation vector is sufficiently sparse, then it can be recovered exactly or stably [START_REF] Vaiter | Low Complexity Regularization of Linear Inverse Problems[END_REF]. Therefore, ill-posedness issue of the USLE can be solved [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF].

The importance of the Exact Recovery Condition(s) (ERC) or stable recovery condition is in guaranteeing the uniqueness or faithful approximation of the solution of the model through the sparse optimisation problem, otherwise, different optimisation algorithms may return different or significantly different solutions.

The theoretical ERC and stable recovery conditions guarantee that the solution can be found independent of the algorithm used, i.e., supposing the existence of a sparse enough solution, it is possible to derive necessary and sufficient conditions for the recovery of the desired solution, regardless of the recovery algorithm used.

As mentioned earlier, in this study we concentrate on the ERC, i.e., the noiseless model in (1.2), i.e., y=Φβ 0 .

The aforementioned sufficient sparsity condition, which sometimes can be applied explicitly on the representation vector, is determined by a so-called Sparsity Level (SL) or sparsity bound, which is the upper-bound for the number of non-zero entries of the representation vector, and is derived from the dictionary. The sparsity level of a dictionary is represented by SL(Φ).

In other words, when the representation vector is very sparse, the sparsity level is low and vice versa [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Candès | Decoding by linear programming[END_REF]; [START_REF] Chartrand | Exact Reconstruction of Sparse Signals via Nonconvex Minimization[END_REF]; [START_REF] Cai | Shifting Inequality and Recovery of Sparse Signals[END_REF].

A representation is said to be k-sparse if it has at most k non-zero entries, which can be arbitrarily placed anywhere in the representation. In other words, k is less than sparsity level, so for a k-sparse representation β 0 , we have β 0 0 ≤k<SL(Φ).

In this section, we briefly explain the main following ERC:

• ERC based on Spark,

• ERC based on null space property,

• ERC based on mutual coherence constant, and • ERC based on cumulative coherence constant.

ERC based on Spark

In literature, to approach the problem of determining the sufficient conditions for unique sparse recovery, i.e., ERC, a different problem inspired by the concept of uncertainty principle is considered [START_REF] Gorodnitsky | Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm[END_REF]; [START_REF] Donoho | Uncertainty Principles and Signal Recovery[END_REF].

Consider the problem P 0 and suppose β 0 and β 1 are two distinct representations of the non-zero signal y, in the dictionary Φ, i.e., y=Φβ 0 and y=Φβ 1 . The uncertainty principle of redundant solutions states that a non-zero signal cannot have multiple highly sparse representations. In other words, in a given dictionary Φ there is a limit on the sparsity level of the representations β 0 and β 1 :

β 0 0 + β 1 0 ≥ Spark (Φ) .
(1.9)

The mentioned uncertainty principle has been proved based on the definition of Spark in (1.5), i.e., Spark(Φ) def = min x∈Ker(Φ)\{0} x 0 , indicating that for β 0 -β 1 in the Kernel of the dictionary Φ, i.e., Φ(β 0 -β 1 )=0, we have β 0 -β 1 0 ≥Spark(Φ). Then, triangle inequality induces (1.9).

The uncertainty principle in (1.9) has been stated and demonstrated for different cases of dictionaries. At first, the dictionary was considered as a concatenation of two orthonormal bases [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Elad | On sparse signal representations[END_REF]; [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF]. Then, this uncertainty principle was generalised to dictionaries which arise from the union of more than two orthonormal bases [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]. Finally, it was generalised to dictionaries which can be the concatenation of less structured blocks [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF]; [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF].

Using the aforementioned uncertainty principle in different cases of dictionary, and the simple criterion of Spark, the uniqueness of the sparse solution can be demonstrated. If β 0 is a candidate solution of the P 0 problem for a general dictionary Φ, and meets

β 0 0 < Spark (Φ) 2 ,
(1.10) then according to the mentioned uncertainty principle in (1.9), any other solution must be denser. Therefore the solution which is sufficiently sparse according to (1.10), is unique and the sparsest.

For a dictionary Φ∈R m×n , considering the upper bound of Spark in (1.6), i.e., m+1, and the above-mentioned conventional Spark-based condition in (1.10), the admitted sparsity level is at most (1+m)/2.

Although among different ERC, ERC based on Spark is the most relaxed condition, i.e., the highest sparsity level, the computation of the Spark characterisation is not tractable [START_REF] Tillmann | The Computational Complexity of Spark, RIP, and NSP[END_REF].

Sparsity

27

ERC based on null space property Conventional Null Space Property (NSP) provides necessary and sufficient conditions for the exact recovery of k-sparse representation via P 1 , in other words the equivalence of P 0 and P 1 , i.e., the solution to P 0 is necessarily equal to the solution to P 1 .

For a k-sparse representation β∈R n , i.e., |S(β)|≤k, assuming that the Support S of the true solution lies within S(β)⊂{1, • • • , n}, i.e., S(β 0 )⊂S(β), the mentioned traditional condition states that if

Q p (S (β) , Φ) def = max x∈Ker(Φ)\{0} i∈S(β) |x i | p i |x i | p < 1 2 ,
then β 0 is the unique solution to the problem P p .

NSP has been demonstrated by starting to prove that for all x∈Ker(Φ), we have β 0 p p < β 0 +x p p . Then using a variant of quasi-triangle inequality, i.e., |a+b| p -|a| p ≥-|b| p , and calculating p norm over on-Support (∈S(β)) and off-Support ( / ∈S(β)) parts, Q p (S(β), Φ) is achieved.

NSP was first stated for a dictionary which is concatenation of two orthonormal matrices corresponding to orthonormal bases and for p equal to one [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Elad | On sparse signal representations[END_REF]; [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF]; [START_REF] Feuer | On sparse representation in pairs of bases[END_REF], then was proved for arbitrary nonorthogonal dictionaries [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]; [START_REF] Zhang | A Simple Proof for Recoverability of 1-Minimization: Go Over or Under? Tech. rep[END_REF]; [START_REF] Stojnic | Compressed sensing -probabilistic analysis of a null-space characterization[END_REF]; [START_REF] Cohen | Compressed sensing and best k -term approximation[END_REF]. Later, it was more generalised to 0≤p≤1 and for dictionaries being a union of orthonormal bases [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]; [START_REF] Gribonval | Sparse representations in unions of bases[END_REF] and for general arbitrary dictionaries [START_REF] Gribonval | On the Strong Uniqueness of Highly Sparse Representations from Redundant Dictionaries[END_REF]; [START_REF] Gribonval | Highly sparse representations from dictionaries are unique and independent of the sparseness measure[END_REF].

In order to investigate the relationship between the ERC based on NSP and the previously mentioned ERC based on Spark in (1.10), let p=0 in Q p (S(β), Φ):

Q 0 (S (β) , Φ) def = max x∈Ker(Φ)\{0} |S (β)| x 0 = |S (β)| min x∈Ker(Φ)\{0} x 0 = |S (β)| Spark (Φ) < 1 2 .
Then, |S(β)|<Spark(Φ)/2.

On the other hand, due to the assumption S(β 0 )⊂S(β), we have |S(β 0 )|≤|S(β)|, which leads to the ERC based on Spark. Therefore, the ERC based on NSP is a general property, which in a special case of p=0 reduces to the ERC based on Spark in (1.10).

Another stable variant of NSP is also introduced in literature, which is called robust NSP [START_REF] Davies | lp minimization and sparse approximation failure for compressible signals[END_REF]; [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF]. From algorithmic point of view, robust NSP is used for stable signal recovery via basis pursuit [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF].

Chapter 1. Background material ERC based on mutual coherence constant In general, Spark and NSP are computationally unrealistic, in other words, it is computationally intractable in polynomial time to check the identifiability of the model through the recovery conditions, specially when the number of atoms in the dictionary is high. For detailed information about the computational complexity of Spark and NSP, the interested reader is referred to [START_REF] Tillmann | The Computational Complexity of Spark, RIP, and NSP[END_REF] and [START_REF] Tillmann | The Computational Complexity of the Restricted Isometry Property, the Nullspace Property, and Related Concepts in Compressed Sensing[END_REF].

To overcome this shortcoming, another characterisation of the dictionary called Mutual Coherence Constant (MCC) was exploited in literature with the expense of making the recovery conditions more restrictive, i.e., lowering the sparsity level.

MCC, which is defined on page 22, i.e., M (Φ)

def = max k,k =k | ϕ k , ϕ k |,
is a simple approach for characterising the proximity or similarity between the atoms of the dictionary. MCC was first introduced by Mallat and Zhang to heuristically evaluate the performance of the MP 6 algorithm [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF].

Like Spark, to approach the problem of exact signal recovery based on mutual coherence, uncertainty principle was used in literature.

Suppose β 1 and β 2 are two distinct representations of the non-zero signal y in two orthonormal bases Φ 1 and Φ 2 , respectively, i.e., y=Φ 1 β 1 =Φ 2 β 2 . The basic or classic uncertainty principle states that a non-zero signal cannot have multiple sparse representations in two distinct orthonormal bases, if both bases are mutually incoherent [START_REF] Donoho | Uncertainty Principles and Signal Recovery[END_REF]; [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Elad | On sparse signal representations[END_REF]; [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF]. Therefore, there is a limit on the sparsity level of the representations β 1 and β 2 :

β 1 0 + β 2 0 ≥ 2 M (Φ 1 , Φ 2 ) , (1.11) where, M (Φ 1 , Φ 2 )= max k,k |ϕ T 1 k ϕ 2 k | is the basic MCC.
In an attempt to extend the basic uncertainty principle to non-orthonormal bases but still square and non-singular matrices Φ 1 and Φ 2 , β 1 0 + β 2 0 ≤(1/2) M -1 (Φ 1 , Φ 2 ) is developed as recovery condition for the uniqueness of the solutions of P 0 and P 1 , and their equivalence, where,

M (Φ 1 , Φ 2 )= max{max i,j |Φ -1 1 Φ 2 | i,j , max i,j |Φ -1 2 Φ 1 | i,j } [DH01].
In fact, the definition of coherence in M (Φ 1 , Φ 2 ), which computes the ∞ norm (maximum entry in a vector) of 1,∞ norm (maximum absolute entry in a matrix) of matrices Φ -1 1 Φ 2 and Φ -1 2 Φ 1 , implicitly indicates to the definition of coherence of blocks in a special case, which will be explained in Chapter 2.

Returning back from basic two orthornomal bases Φ 1 and Φ 2 to the dictionaries Φ, and in order to shift the recovery conditions based on Spark to MCC, first we need to find their relationship.

MCC for an orhtogonal matrix is zero, i.e., no constraint. It should be mentioned that, Spark is lower bounded by a function of inverse of MCC, i.e., Spark(Φ)≥f (M -1 (Φ)).

Sparsity 29

Therefore, we can approximate the intractable characterisation Spark with a computationally tractable characterisation MCC.

For a general dictionary, the smallest value of the MCC is of interest, in other words the tightest lower bound of the Spark, because as mentioned before in (1.10), the sparsity level is defined as half of the Spark, i.e., β 0 0 <Spark(Φ)/2.

In addition to theoretic results, from algorithmic point of view, generally the smaller the MCC, the better the performance of recovery algorithms. [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF] has justified the claim for OMP7 , BP8 , and basic thresholding algorithms.

If the dictionary is the concatenation of two orthonormal bases, [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF] proved that for guaranteeing the uniqueness of the solution of P 0 and equivalence of P 1 , it is sufficient for

f (M -1 (Φ))=1+M -1 (Φ).
Later, Elad and Bruckstein improved the condition by getting f (M -1 (Φ))=2M -1 (Φ) (also proved in [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF] and [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF] as a special case) and f (M -1 (Φ))=(2 √ 2-1)M -1 (Φ) for guaranteeing the uniqueness of the solution of P 0 and equivalence of P 1 , respectively [START_REF] Elad | On sparse signal representations[END_REF]. Feuer and Nemirovski proved that the latter is the maximum lower bound for Spark that can be achieved in the P 1 problem, i.e., the bound is tight [START_REF] Feuer | On sparse representation in pairs of bases[END_REF].

Supposing that the dictionary arises from the union of L orthonormal bases (L≥2), Gribonval and Nielsen proved f (M -1 (Φ))=L/(L-1)M -1 (Φ) and

f (M -1 (Φ))=(2 √ 2-2+1/(L-1))M -1 (Φ)
for guaranteeing the uniqueness of the solution of P 0 and equivalence of P 1 , respectively [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]; [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]. Tropp also proved the recovery condition resulted from the latter f (M -1 (Φ)), for the equivalence of OMP and BP algorithms [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF].

Later, Donoho, Elad, Gribonval, Nielsen, and Bruckstein demonstrated that the previous results of Donoho and Huo, i.e., f (M -1 (Φ))=1+M -1 (Φ), can be generalised from a union of two orthonormal bases to a dictionary in a general case, which it can be the concatenation of less structured blocks (in addition to orthonormal bases) [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Donoho | Maximal Sparsity Representation via l1 Minimization[END_REF]; [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]; [START_REF] Bruckstein | From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images[END_REF].

For β 0 and β 1 as two distinct representations of the non-zero signal y, in the dictionary Φ, i.e., y=Φβ 0 and y=Φβ 1 , and by combining f (M -1 (Φ))=1+M -1 (Φ) with (1.9), we have

β 0 0 + β 1 0 ≥ Spark (Φ) ≥ 1 + M -1 (Φ) .
(1.12) Therefore, for any general dictionary, the k-sparse representation vector β 0 is the unique solution of the P 0 and P 1 problems, if

β 0 0 ≤ k < 1 + M -1 (Φ) 2 .
(1.13)

According to the lower bound of MCC for a general random dictionary Φ in (1.7), i.e., 1/ √ m, and the above-mentioned conventional MCC-based condition in (1.13), the sparsity Chapter 1. Background material level is at most (1+ √ m)/2, although for equiangular tight frame deterministic dictionaries it can goes up until (1+ (m(n-1))/(n-m))/2.

Gribonval and Nielsen demonstrated that under the condition in (1.13), and for an arbitrary dictionary, the problems P p with 0≤p<1 and P 1 are equivalent [START_REF] Gribonval | Highly sparse representations from dictionaries are unique and independent of the sparseness measure[END_REF].

From algorithmic point of view, Tropp proved the same condition of (1.13) for representation recovery through some greedy recovery algorithms, i.e., equivalence of OMP and BP algorithms [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF].

In another algorithmic study, Maleki demonstrated slightly stronger sufficient recovery conditions than (1.13) via iterative thresholding algorithms of iterative thresholding with inversion, namely IHT9 and IST 10 . He showed that, supposing β 0 is sorted in descending order of its absolute values [START_REF] Maleki | Coherence Analysis of Iterative Thresholding Algorithms[END_REF]:

1. if k<(1/3)M -1 (Φ), then iterative thresholding with inversion recovers the true solution, i.e., β 0 , in at most k iterations.

2. if k<(1/3.1)M -1 (Φ) and |β 0 i |/|β 0 i+1 |<3 i -4 , 1≤i<k, then iterative hard thresholding will recover Support of the true solution, i.e., S(β 0 ), in at most k i=1 i +k iterations, and after this number of iterations, without changing the Support, the error of β 0 recovery will be eliminated exponentially. i is the step after which β 0 i+1 will get into the Support.

3. if k<(1/4.1)M -1 (Φ) and |β 0 i |/|β 0 i+1 |<3 i -5 , 1≤i<k, then iterative soft thresholding will recover Support of the true solution, i.e., S(β 0 ), in at most k i=1 i +k iterations, and after this number of iterations, without changing the Support, the error of β 0 recovery will be eliminated exponentially. i is the step after which β 0 i+1 will get into the Support.

In another algorithm-based condition, it has been shown that for k<M -1 (Φ)/4 every k-sparse representation can be exactly recovered via iterative hard thresholding [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF].

In addition, Donoho et al. demonstrated the same condition of (1.13) for a stable solution in a stable signal recovery problem of P 0,2,ε (β, r) [START_REF] Donoho | Stable recovery of sparse overcomplete representations in the presence of noise[END_REF].

Sparsity

ERC based on cumulative coherence constant The coherence characterisation of MCC [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF] represents only the most extreme correlation between the atoms of the dictionary and does not offer a comprehensive description of the dictionary. In order to extract more information and to better characterise the dictionary, researchers focused on cumulative coherence characterisations. Although it is computationally more difficult than the conventional coherence characterisation, it leads to weakened recovery conditions; i.e., it provides sharper results for the equivalence of P 0 and P 1 optimisation problems.

The characterization µ 1/2 (G) introduced by Donoho and Elad as a cumulative coherence characterisation is the smallest m off-diagonal entries in a single row or column of the Gram matrix G, which sums at least to 1/2. Based on µ 1/2 (G), if β 0 0 <µ 1/2 (G), then β 0 is the unique solution of the P 1 optimisation problem [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF].

From algorithmic point of view, considering cumulative MCC defined in (1.8), i.e., M (Φ, k)

def = max |Λ|=k max j / ∈Λ i∈Λ | ϕ i , ϕ j |, Tropp demonstrated that, if M (Φ, k) + M (Φ, k -1) < 1, (1.14)
then k-sparse representation vector β 0 can be recovered correctly from orthogonal matching pursuit and basis pursuit algorithms, which is a sufficient condition [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF].

The ERC based on cumulative MCC, i.e., (1.14), is weaker than the ERC based on MCC, i.e., the inequality of (1.13). Because by rearranging (1.13) we have 2 k M (Φ)-M (Φ)<1, which can be reformulated such as: kM (Φ)+(k-1)M (Φ)<1. Now we can compare the reformulated condition of (1.13), i.e., kM (Φ)+(k-1)M (Φ)<1, with condition (1.14). Considering the M (Φ, k)≤k M (Φ) and M (Φ, k-1)≤(k-1) M (Φ) properties, we conclude that the left-hand side of the just obtained reformulated inequality is greater than or equal to the left-hand side of the inequality in (1.14), i.e., M (Φ, k)+M (Φ, k-1)≤kM (Φ)+(k-1)M (Φ). Then, higher values of k could meet the condition in (1.14) compared to the reformulated inequality of (1.13). Hence, the condition based on cumulative MCC is weaker than the condition based on MCC.

The results were improved more by a characterisation called union cumulative coherence:

M U (Φ, k) def = max |Λ|=k {max j / ∈Λ i∈Λ | ϕ i , ϕ j |+ max l∈Λ i∈Λ\{l} | ϕ i , ϕ l |} [Dos05]; [Zha+15]. If M U (Φ, k)<1
, then the k-sparse representation vector can be exactly recovered by the orthogonal matching pursuit algorithm. The ERC using M U (Φ, k) is weaker than (1.14), since M U (Φ, k)<M (Φ, k)+M (Φ, k-1), and so higher values of k can be selected in M U (Φ, k)-based condition. There is also a stable recovery condition via orthogonal matching pursuit using the characterisation

M U (Φ, k) [Zha+15]. In another work, it is been demonstrated that if M (Φ, k)+M (Φ, k-1)< min i∈T |β 0 i |/ max i∈T |β 0 i |,
where, |T |=k, then the k-sparse representation vector β 0 can be recovered exactly via basic thresholding algorithm [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF]. Moreover, it has been shown that if 2M (Φ, k)+M (Φ, k-1)<1 and M (Φ, 2k)<1/2, then the k-sparse representation is exactly recovered via hard thresholding pursuit and iterative hard thresholding algorithms, respectively [START_REF] Foucart | A Mathematical Introduction to Compressive Sensing[END_REF].

Block-sparsity

Introduction

According to Tropp, for more than a century, sparsity constraint has been utilised to recover the sparsest unique solution of a proper optimisation problem and to derive the corresponding recovery conditions [START_REF] Aaron | Topics in sparse approximation[END_REF]. As described in Section 1.3, sparsity constraint penalizes the non-zero coefficients; however, in some real-world engineering problems and applications, which some of them are gathered in table 1.3, applying penalty on non-zero coefficients does not lead to a proper solution compatible to the real-world conditions.

In these kinds of problems, in addition to sparsity, structural, geometrical, or categorical constraints or a priori knowledge may be available that leads to block sparsity [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], group sparsity [START_REF] Rao | Universal Measurement Bounds for Structured Sparse Signal Recovery[END_REF] and joint sparsity [START_REF] Fornasier | Recovery Algorithms for Vector-Valued Data with Joint Sparsity Constraints[END_REF]. Therefore, a block of coefficients need to be penalized and recovery conditions determine the maximum number of active blocks, and not coefficients, to ensure the uniqueness of the block-sparse solution of the corresponding optimisation problem.

On demonstrating the benefits of the block structure constraint, Kwon and Rao showed that for exact Support recovery in the presence of noise, block-sparsity leads to the reduced number of measurements by the factor of at least the inverse of block length, compared to the conventional element-wise sparse recovery [START_REF] Kwon | On the benefits of the block-sparsity structure in sparse signal recovery[END_REF]. If each block of columns in the dictionary corresponding to each block in the representation vector is composed of linearly dependent columns, the blocks are said to be redundant blocks, whereas if the columns are not linearly dependent, the blocks are referred to as non-redundant blocks [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF]. Section 1.4.2 defines the block-sparse representation theory, next in Section 1.4.3, recovery conditions are reviewed.

Applications

References

Radar [MZ06] Sensor arrays [MCW05] Signal sampling [CK04]; [LD08]; [BD09a]; [ME09]; [Eld09]; [ME10]; [GE10] Face recognition [Wri+09]; [EV11]; [EV12]; [PL15] MRI pulse design [Zel+08b]; [Zel+08a] Speech recognition [GC08a]; [GC08b] Music segmentation [Krs+05]; [Dau06]; [KOP13] Gene expression levels [ES05]; [Par+08] Multiple measurement vector [Cot+05]; [CH06]; [ME08]; [FR08]; [BF10]; [ER10]; [LL11]; [LWY13] Sparse communication channels [CR02] Finding datasets' representatives [ESV12] Multi-band signal reconstruction [Lan67]; [ME09]; [Mis+11]; [ME10] EEG/MEG source reconstruction [GGR95] Data clustering on multiple subspaces [EV09]; [EV10]; [EV13]
Table 1.3: Some engineering problems that have structural constraints.

Block-sparsity

Block-sparse representation theory

To elucidate the theory, this section is divided into the following three parts: (1) structure and model, (2) optimisation problem, and (3) characterisations:

1) Structure and model As mentioned before, the goal is to extract a representation vector β which is the block-sparsest among all solutions, i.e., a representation with the fewest active blocks relative to its dimension. For instance, suppose the following estimated representation vector β which consists of nonoverlapping equally-sized blocks of length d, i.e., ∀k, β[k]∈R d :

β = βT [1] , • • • , βT [k] , • • • , βT [K] T ,
where, Kd=n, and the k th block is:

β [k] = β1 [k] , • • • , βd [k] T .
Then, the dictionary is represented by:

Φ = [Φ [1] , • • • , Φ [k] , • • • , Φ [K]] ,
where, the k th blocks of atoms of the dictionary is:

Φ [k] = [ϕ 1 [k] , • • • , ϕ d [k]] ,
with ϕ j [k]∈R m and without loss of generality, it is assumed that atoms have unit Euclidean norm, i.e., ∀j, k, ϕ j [k] 2 =1. Then, assuming that the true solution β 0 is exactly recovered, i.e., β=β 0 , the block-sparsity structure of the estimated representation vector β in the noiseless linear model is represented graphically in figure 1.10. The model can be either noiseless, i.e., y=Φβ 0 , or in general case, noisy, i.e., y=Φβ 0 +e, which are defined in (1.2) and (1.1), respectively. Chapter 1. Background material 2) Optimisation problem Similar to the conventional element-wise sparse recovery, the problem of finding a solution β, in the noiseless case of (1.2), i.e., y=Φβ 0 , is called block-sparse exact (unique) signal recovery whereas in the other case of (1.1), i.e., y=Φβ 0 +e, is called block-sparse stable (robust) signal recovery.

Block-sparse exact and stable signal recovery problems can be realised through a proper constrained optimisation problem. The straightforward optimisation problem for block-sparse stable signal recovery is:

min β β 2,0 s.t. y -Φβ 2 ≤ ε, (1.15)
where, for ε=0, the optimisation problem (1.15) solves a block-sparse exact signal recovery problem.

In a general framework, consider the following constrained p 1 ,p 2 (pseudo-)mixed-norm minimisation problem:

P (p 1 ,p 2 ),q,ε (a, b) : min a a p 1 ,p 2 s.t. b q ≤ ε, (1.16)
where, a and b are vectors, ε bounds the noise level, i.e., 0≤ ≤ε, and p 1 ,p 2 (pseudo-)mixed-norm of block-structured vector a= a

T [1] , • • • , a T [k] , • • • , a T [K]
T , where,

a [k] = [a 1 [k] , • • • , a d [k]] T , is defined: a p 1 ,p 2 def =                  k I a [k] p 1 , for p 2 = 0 k a [k] p 2 p 1 1 p 2 , for 0 < p 2 < +∞ max k a [k] p 1 , for p 2 = ∞,
where, I(a) is the indicator function, i.e., I(a

) def = 1, if |a| >0 0, if a=0 .
According to (1.16) the problem (1.15) can be represented as P (2,0),2,ε (β, y-Φβ).

Although the notation P (p 1 ,p 2 ),q,ε (a, b) can cover all the previous types of optimisation problems, the widely used case is when the minimisation is over β and the problem is a block-sparse exact signal recovery, i.e., P (p 1 ,p 2 ),q,0 (β, r), which for the sake of simplicity we refer to as P p 1 ,p 2 . Because regardless of the choice of q, the r q =0 directly indicates to the noiseless model, i.e., y=Φβ 0 , so the only parameters to be determined are p 1 and p 2 .

Since the optimisation problem in block-sparsity domain is a generalisation of the conventional optimisation problem in sparsity domain, for d=1, the two optimisation problems are equivalent, i.e., P p 1 ,p 2 ≡P p 2 .

The straightforward configuration of the mentioned optimisation problem in a block-sparse exact or stable signal recovery problem is P (p,0),q,ε (β, r), which simply minimise the number of active blocks, i.e., blocks with non-zero p norm. However, the P (p,0),q,ε (β, r) optimisation problem is in general non-convex and finding its solution is NP-hard11 [START_REF] Natarajan | Sparse Approximate Solutions To Linear Systems[END_REF]; [START_REF] Amaldi | On the approximability of minimizing nonzero variables or unsatisfied relations in linear systems[END_REF]. Therefore, there is a need for convexification, relaxation or approximation of P (p,0),q,ε (β, r). Similar to sparsity domain, in block-sparsity domain the following two main questions are addressed: Q1: Is the solution to P 2,1 necessarily the block-sparsest representation (solution to P 2,0 ) (Uniqueness)?

Q2: Is the solution to P 2,0 necessarily equal to the solution to P 2,1 (Equivalence)?

It has been shown that under sufficient conditions, P 2,1 and P 2,0 are equivalent [EB09]; [EM09];

[SPH09]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], and in a more general case the equivalence of P p,1 and P p,0 for p>0 and for both non-redundant and redundant blocks is investigated [START_REF] Elhamifar | Robust classification using structured sparse representation[END_REF]; [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF].

Subspace matching pursuit is a greedy algorithm, which is used for a block-sparse exact and stable signal recovery problems of P (2,0),q,0 (β Φ , r) and P (2,0),2,ε (β Φ , r), respectively, where,

β Φ = (Φ[1]β[1]) T , • • • , (Φ[k]β[k]) T , • • • , (Φ[K]β[K]) T T ,
and the Φ[k]'s are assumed to be full rank matrices [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF].

In addition, subspace base pursuit and its modified version is used for a block-sparse exact and stable signal recovery problems of P (2,1),q,0 (β Φ , r) and P (2,1),2,ε (β Φ , r), respectively, and under sufficient conditions, P (2,1),q,0 (β Φ , r) and P (2,0),q,0 (β Φ , r) are equivalent [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF].

Moreover, there are theoretical conditions for ensuring the equivalence of P (p,1),q,0 (β Φ , r) and P (p,0),q,0 (β Φ , r) for p>0, while the blocks could be non-redundant or redundant [START_REF] Elhamifar | Robust classification using structured sparse representation[END_REF]; [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF].

In fact, unlike P p,0 that extracts a representation with the fewest active blocks β[k], the P (p,0),q,0 (β Φ , r) extracts a representation with the fewest active reconstructed vectors

Φ[k]β[k] [Elh12]
. When the blocks Φ[k]'s are non-redundant, the solution to P (p,0),q,0 (β Φ , r) has also the fewest active blocks [START_REF] Elhamifar | Sparse modeling for high-dimensional multi-manifold data analysis[END_REF].

There are also some recovery conditions of P 2,1 via the mixed 2 / 1 -optimisation program [START_REF] Eldar | Robust Recovery of Signals From a Structured Union of Subspaces[END_REF]; [START_REF] Stojnic | On the Reconstruction of Block-Sparse Signals With an Optimal Number of Measurements[END_REF], also known in the statistics literature as group LASSO12 [YL06]; [START_REF] Lv | The Group Lasso for Stable Recovery of Block-Sparse Signal Representations[END_REF], BMP (Block Matching Pursuit) and BOMP (Block Orthogonal Matching Pursuit) algorithms [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF].

In addition, there are block-sparse stable recovery conditions for block orthogonal matching pursuit and block-thresholding algorithms [START_REF] Ben-Haim | Near-Oracle Performance of Greedy Block-Sparse Estimation Techniques From Noisy Measurements[END_REF]. Also 2 / 1 -optimisation program [START_REF] Eldar | Robust Recovery of Signals From a Structured Union of Subspaces[END_REF], block versions of CoSaMP13 and IHT 14 [Bar+10] algorithms (which are the extensions of iterative sparse coding algorithms to model-based sparse representation) lead to robust recoveries in the presence of noise.

36

Chapter 1. Background material 3) Characterisations Similar to the conventional sparse representation theory, in order to define block-sparse recovery conditions, some characterisations of the dictionary are required.

In the following, the main characterisations extended from the conventional notions of Spark and mutual coherence are provided:

• Subspace Spark,

• Block-coherence and sub-coherence,

• Extended block-coherence,

• Fusion coherence,

• Mutual subspace coherence, and • Cumulative mutual subspace coherence constant.

Subspace Spark: In a related work for defining block-sparse recovery conditions, Spark of the set of subspaces S i =span(Φ[i])∈R d , denoted by Spark S (Φ), is defined as the smallest integer r such that r i=1 s i =0, where,

s i =0, s i ∈S i and i∈{1, • • • , K} [GZM09].
Block-coherence and sub-coherence: Eldar and her colleagues introduced two different measures of the block-wise coherence of the dictionary, named block-coherence and sub-coherence [EB09]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF].

Block-coherence measures the global coherence properties of the dictionary, in other words, quantifies the inter-block coherence through:

M Eldar Inter (Φ) def = max k,k =k 1 d Φ T [k] Φ k 2→2 , (1.17)
where, ∀k, Φ[k]∈R m×d , and d is the size of all equally-sized blocks, i.e.,

d 1 = • • • =d K =d. For a typical matrix A, the 2→2 operator-norm, i.e., A 2→2 def = max a 2 ≤1
Aa 2 , is equal to its maximum singular value. For d=1, the block-coherence is equivalent to the conventional MCC, i.e., M Eldar Inter (Φ)≡M (Φ).

On the other hand, sub-coherence measures the local coherence properties of the dictionary by computing the intra-block coherence through:

M Eldar Intra (Φ) def = max i,j =i k ϕ T i [k] ϕ j [k] . (1.18)
The sub-coherence characterisation for dictionaries with intra-block orthonormality is zero, i.e., for 1≤k≤K,

Φ T [k]Φ[k]=I d , we have M Eldar Intra (Φ)=0.
Extended block-coherence: As an extension of the studies of Eldar et al., Ziaei and his colleagues study the same problem with different block sizes [START_REF] Ziaei | Compressed sensing of different size block-sparse signals: Efficient recovery[END_REF]. For establishing their findings, they introduced a new version of the block-coherence of Eldar et al., i.e., M Eldar Inter (Φ) defined in (1.17), replacing d by max i,j =i d i d j , where, d i is the length of block i [START_REF] Ziaei | Compressed sensing of different size block-sparse signals: Efficient recovery[END_REF].

Fusion coherence: Fusion coherence in the context of fusion frames is defined as

M F (Φ)= max i =j {| ϕ i , ϕ j | • ||P i P j || 2→2 }
, where, P i denotes the orthogonal projection onto the fusion frame subspace

W i [BKR11]; [Aya14]; [ADR16].
Fusion coherence in the special setting, where the projection matrices are equal to one or all the subspaces have dimension one, reduces to MCC, i.e., M F (Φ)≡M (Φ) [START_REF] Boufounos | Sparse Recovery From Combined Fusion Frame Measurements[END_REF], whereas for the equally-sized subspaces, reduces to the block-coherence defined by Eldar et al., i.e., M F (Φ)≡M Eldar Inter (Φ) [START_REF] Boufounos | Sparse Recovery From Combined Fusion Frame Measurements[END_REF].

Mutual subspace coherence: In a related work, mutual subspace coherence has been introduced by Ganesh et al. as a measure of inter-block coherence, which measures the smallest angle between any two subspaces that do not intersect with each other, i.e., disjoint subspaces [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF]:

M S (Φ) def = max i,j =i max x∈S i y∈S j x T y x 2 y 2 ,
where, S i =span(Φ[i])∈R d , under the assumption of linearly independent columns of each block, i.e., non-redundant blocks.

In another work, the concept of mutual subspace coherence has been used for establishing recovery conditions with more relaxed assumption of redundant blocks, where, the columns of each block can be linearly dependent [START_REF] Elhamifar | Robust classification using structured sparse representation[END_REF]; [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF]. When the blocks are non-redundant and orthonormal, the inter-block coherence proposed by Eldar et al. is equivalent to mutual subspace coherence, i.e., M Eldar Inter (Φ)≡M S (Φ) [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF]. For d equal to 1, the mutual subspace coherence is equivalent to the conventional MCC, i.e., M S (Φ)≡M (Φ) [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF].

Cumulative mutual subspace coherence constant: In literature on block sparsity, there is another cumulative characteristic based on subspaces, named cumulative mutual subspace coherence, which is defined as [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF]:

M S (Φ, k) def = max |Λ|=k max j / ∈Λ i∈Λ max x∈S i y∈S j x T y x 2 y 2 ,
where, Λ represents k different indices from {1, • • • , K}. For d equal to 1, the cumulative mutual subspace coherence is equivalent to the conventional cumulative coherence, i.e., M S (Φ, k)≡M (Φ, k) [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF]. Similar properties of conventional cumulative coherence hold true for cumulative mutual subspace coherence, i.e., M S (Φ, 1) is equal to M S (Φ), and M S (Φ, k)≤kM S (Φ) [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF].

Block-sparse exact recovery condition

Recently, there has been a huge surge of interest in developing recovery conditions, which ensure the uniqueness or robustness of the block-sparse representation of the USLE. In addition, block-sparsity can be used in dictionary learning, where there is a joint block-sparse representation of signals [START_REF] Zelnik-Manor | Dictionary Optimization for Block-Sparse Representations[END_REF].

In addition to the aforementioned practical interest of the block-sparse representation, i.e., compatibility with some real world problems, from a mathematical point of view, assuming the block-wise structure for the representation leads to weakened recovery conditions [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF]; [START_REF] Ziaei | Compressed sensing of different size block-sparse signals: Efficient recovery[END_REF]; [START_REF] Ben-Haim | Near-Oracle Performance of Greedy Block-Sparse Estimation Techniques From Noisy Measurements[END_REF]. By weakened recovery conditions we mean that for the same number of non-zero elements in the representation, assuming the block structure guarantees the uniqueness of the representation with a higher sparsity level.

Similarly, conditions for guaranteeing the uniqueness or faithful approximation of the solution with the models of (1.2), i.e., y=Φβ 0 , and (1.1), i.e., y=Φβ 0 +e, are called Block-sparse Exact Recovery Condition (Block-ERC) and block-sparse stable recovery condition.

Sparsity level in the block-wise world is called Block-Sparsity Level (Block-SL) or block-sparsity bound, which is represented by Block-SL(Φ). As it can be easily derived, supposing equally-sized blocks of length d, i.e., d 1 = • • • =d K =d, to improve the conventional ERC and stable recovery conditions, the d times block-sparsity level have to be greater than the sparsity level, i.e., d×Block-SL(Φ)>SL(Φ).

Such sparse representations whose non-zero entries appear in a few blocks are referred to as block-sparse or block k-sparse representation [EB09]; [SPH09]; [START_REF] Ben-Haim | Near-Oracle Performance of Greedy Block-Sparse Estimation Techniques From Noisy Measurements[END_REF]; [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF] , which k is the maximum number of active blocks.

Similarly,

for a block k-sparse representation, for all p≥0 we have β 0 p,0 ≤k<Block-SL(Φ). From conventional element-wise sparsity point of view, for equally-sized blocks, block k-sparse representation is equivalent to kd-sparse representation, i.e., β 0 0 ≤kd.

In this section, we review the following main block-sparse exact recovery conditions:

• Block-ERC based on Spark,

• Block-ERC based on null space property,

• Block-ERC based on mutual coherence constant, and • Block-ERC based on cumulative mutual coherence constant.

1.4. Block-sparsity 39 1) Block-ERC based on Spark From algorithmic point of view, Ganesh and his colleagues proved that supposing a k-subspace sparse measurement vector y, if k<Spark S (Φ)/2 then the sparse decomposition is necessarily unique in their proposed algorithms of subspace matching pursuit and subspace base pursuit [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF].

2) Block-ERC based on null space property In literature, this type of Block-ERC can be divided into the following groups:

Based on block null space property: [START_REF] Stojnic | On the Reconstruction of Block-Sparse Signals With an Optimal Number of Measurements[END_REF] demonstrated that the conventional NSP can be generalised to block-sparse representation. By proposing the following condition, they showed the equivalence of the optimisation problems P 2,1 and P 2,0 :

Q 2,1 (S b (β) , Φ) def = max x∈Ker(Φ)\{0} k∈S b (β) d j=1 |x j [k]| 2 1 2 K k=1 d j=1 |x j [k]| 2 1 2 < 1 2 ,
where, S b (β) is subset of a set with all subset of size k of {1, • • • , K}, i.e., block k-sparse representation, and d is the length of equally-sized blocks. Also, he mentioned that his result can be generalised to P 2,p , 0<p≤1.

Based on fusion null space property: In addition, the fusion NSP defined in the fusion frame framework proposed by Boufounos et al., in a special case in which all the subspaces share the same dimension, becomes similar to the NSP proposed by Stojnic et al. [START_REF] Boufounos | Sparse Recovery From Combined Fusion Frame Measurements[END_REF].

Moreover, the notions of exact, stable and robust NSP for fusion frames are presented in [START_REF] Ayaz | Sparse Recovery with Fusion Frames[END_REF]; [START_REF] Ayaz | Uniform Recovery of Fusion Frame Structured Sparse Signals[END_REF].

3) Block-ERC based on mutual coherence constant In literature, this type of Block-ERC can be divided into the following groups:

Based on block-coherence: Eldar and her colleagues extended the basic uncertainty principle to the block-sparse representations [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF]. In other words, there is a limit on the block-sparsity level of the representations β 1 and β 2 :

β 1 2,0 + β 2 2,0 ≥ 2 d M Eldar Inter (Φ 1 , Φ 2 ) , (1.19)
where, the basic block-coherence of Eldar et al., i.e., M Eldar Inter (Φ 1 , Φ 2 ), is the maximal coherence between the blocks of the orthonormal bases Φ 1 and Φ 2 , i.e.,:

M Eldar Inter (Φ 1 , Φ 2 ) = max k,k 1 d Φ T 1 [k] Φ 2 k 2→2 ,
where, d is the length of the equally-sized blocks.

Notice that for orthonormal bases Φ 1 , Φ 2 and their concatenation [Φ 1 , Φ 2 ], we have

M Eldar Inter (Φ 1 , Φ 2 )=M Eldar Inter ([Φ 1 , Φ 2 ]).
Again, supposing the dimension of two orthonormal bases Φ 1 and Φ 2 is m by m, and

m=R d (R is an integer), it is proved that M Eldar Inter (Φ 1 , Φ 2 )≥1/ √ d m [EB09]; [EKB10b]; [EKB10a].
Notice that for d=1, all the formulation reduces to its conventional one described in Section 1.3.3.

Then, they proposed Block-ERC for ensuring the uniqueness of the solution of their proposed recovery algorithms of 2 / 1 -optimisation program [START_REF] Eldar | Robust Recovery of Signals From a Structured Union of Subspaces[END_REF], block matching pursuit and block orthogonal matching pursuit [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF] based on their proposed block version of the conventional MCC.

Their Block-ERC for recovering a block k-sparse representation through their proposed recovery algorithms and for equally-sized blocks of length d is [EB09]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF]: In the presence of noise, Ben-Haim and Eldar proposed block-sparse stable recovery conditions for block orthogonal matching pursuit and block-thresholding algorithms [START_REF] Ben-Haim | Near-Oracle Performance of Greedy Block-Sparse Estimation Techniques From Noisy Measurements[END_REF].

β 0 2,0 < 1 + d M Eldar Inter (Φ) -1 1 -(d -1) M Eldar Intra (Φ) 2 . ( 1
Based on fusion coherence: In addition, in the context of fusion frame theory, Boufounos et al. have proved that if k<(1+M -1 F (Φ))/2, then the unique recovery of his 0 and 1 norm optimisation problems is guaranteed [START_REF] Boufounos | Sparse Recovery From Combined Fusion Frame Measurements[END_REF].

Later, Ayaz et al. improves some of the results of Boufounos et al. in the fusion frame setup [Aya14]; [ADR16].

The recovery condition based on the fusion coherence in the special setting, where, all the subspaces share the same dimension, reduces to (1.20) [START_REF] Boufounos | Sparse Recovery From Combined Fusion Frame Measurements[END_REF].

Based on mutual subspace coherence:

There is a relationship between subspace Spark and mutual subspace coherence, i.e., Spark S (Φ)≥1+M -1 S (Φ), which is similar to its conventional relationship [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF].

From algorithmic point of view, Ganesh and his colleagues proved similar conditions in recovering the representation. Supposing that the measurement vector y is k-subspace sparse, they demonstrated that if k<(1+M -1 S (Φ))/2, then subspace matching pursuit and subspace base pursuit algorithms are guaranteed to find the k subspaces exactly [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF].

In a related work based on M S (Φ), Elhamifar and Vidal proved that if the measurement vector y has a unique block k-sparse representation vector β 0 , and the subspaces spanned by the columns of the blocks are disjoint, then under the condition

k < 1 + 1+ p 1-p M S (Φ) -1 1 + 1+σp 1+ p
, the optimisation problems P p,1 and P p,0 are equivalent, and also under the condition

k < 1 + 1+ p 1-p M S (Φ) -1 2 ,
the optimisation problems P (p,1),q,0 (β Φ , r) and P (p,0),q,0 (β Φ , r) are equivalent, where, σ p , p , and p are some intra-block p-restricted isometry constants which are defined based on the restricted isometry properties introduced in the paper [START_REF] Elhamifar | Robust classification using structured sparse representation[END_REF]; [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF].

4) Block-ERC based on cumulative mutual coherence

The mutual subspace coherence of Ganesh et al. [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF] represents only the most extreme correlation between the subspaces of the dictionary and does not offer a comprehensive description of the dictionary.

In order to extract more information and to better characterise the dictionary, Elhamifar and Vidal focused on the cumulative version of mutual subspace coherence.

In the block-wise world, Elhamifar and Vidal proved that if the measurement vector y has a unique block k-sparse representation vector β 0 , and the subspaces spanned by the columns of the blocks are disjoint, then under

1 + σ p 1 + p M S (Φ, k) + M S (Φ, k -1) < 1 -p 1 + p ,
the optimisation problems P p,1 and P p,0 are equivalent, and also under

M S (Φ, k) + M S (Φ, k -1) < 1 -p 1 + p ,
the optimisation problems P (p,1),q,0 (β Φ , r) and P (p,0),q,0 (β Φ , r) are equivalent, where, σ p , p , and p are some intra-block p-restricted isometry constants which are defined based on the restricted isometry properties introduced in the paper [START_REF] Elhamifar | Block-Sparse Recovery via Convex Optimization[END_REF].

1.5 Block-sparsity and multi-modality

1.5.

Introduction

In Section 1.2, first the concept of multi-modality is discussed in general. Then, we showed that due to the fact that a brain neuronal activity have electric and magnetic properties, the brain imaging modality of EEG or MEG individually, cannot completely describe the electromagnetic behaviour of the brain neurons. On the other hand, EEG and MEG measure the electrical and magnetic activities of the same cerebral currents, respectively. Since they have complementary information, they can be an appropriate choice as brain imaging modalities in a multi-modality framework. Therefore, we took the distributed EEG and MEG source reconstruction problem as a real-world example of multi-modality.

However, the distributed EEG or/and MEG source reconstruction problems are vastly underdetermined, i.e., the number of EEG or/and MEG sensors is significantly less than the number of brain sources, e.g., in a typical problem there could be 30 sensors against 3000 sources.

Hence, in order to recover a unique solution from the USLE of distributed EEG or/and MEG source reconstruction problems, appropriate constraint(s) need(s) to be applied on the corresponding optimisation problem .

In Section 1.4, we pointed out that, in certain USLE a block of coefficients need to be penalized and not necessarily a single coefficient. Hence, the notion of block-sparsity was extended from the conventional notion of sparsity, which was reviewed in Section 1.3.

As a real-world example, we again took the distributed EEG or/and MEG source reconstruction problem. Since each brain source, which is a dipole, can be represented as a block of size d=3, and the source reconstruction problem is consistent with the constraint of block-sparsity.

Although the two mentioned notions of multi-modality and block-sparse representation theory are fundamentally independent, they meet each other in our main real-world problem, i.e., distributed EEG and MEG source reconstruction problem.

In order to recover a unique solution, in addition to imposing the block-sparsity constraint on the corresponding optimisation problem, the number of active blocks should be less than a threshold. The mentioned restricted number of active blocks is explained through recovery conditions, which are explained in Chapter 2.

Next, in Chapter 3 by following a framework to merge the lead-field matrices of each source position, we show that at level of grouping corresponding to maximum inter-group distance and minimum number of groups, the system satisfies the existing conditions. Finally, in Chapter 4, we demonstrate the impact of EEG and MEG multi-modality and merging framework on brain source space segmentation.

1.5. Block-sparsity and multi-modality

Distributed EEG and MEG source reconstruction problem

Regardless of the chosen strategy to combine the EEG and MEG measurements and lead-field matrices, consider the combined measurement vector as y EMEG , the lead-field matrix as Φ EMEG , and the true source vector as β 0 . Then, the distributed EEG and MEG source reconstruction problem can be translated in mathematical form as an USLE model in an ideally noiseless case, i.e., y EMEG =Φ EMEG β 0 .

On the other hand, each current dipole in the source space, which can be represented as a vector, has three values in three dimensions of x, y and z in the Cartesian coordination. So, each current dipole can be represented as a block of length three, i.e., β 0 [k]∈R 3 , ∀k. Then, each successive three elements in the source vector β 0 represents the activity of a current dipole.

From physiological a priori knowledge, it is known that for a given brain task only a few regions of brain will be activated, i.e., sparse regions. Furthermore, for a given active current dipole, the activity of each of values in three dimensions of x, y and z is not constrained, then sparsity inside each active block β 0 [k] is not important.

Therefore, among the infinitely many solutions of the above-mentioned USLE or source activity vectors β, solutions with the fewest active blocks of dimension three would be of interest and not necessarily solutions with the fewest non-zero scalar entries.

Ultimately, our real-world distributed EEG and MEG source reconstruction problem perfectly complies with the notions of multi-modality and block-sparse representation theory.

As shown in figure 1.11, without imposing appropriate constraints on the optimisation problem corresponding to the EEG and MEG multi-modal USLE in the source reconstruction problem, there would be infinitely many solutions β 1 , • • • , β ∞ (left branch), but by applying block-sparsity constraints, the desired block-sparse solution β 0 (right branch), can be recovered.

Notice, the left branch in figure 1.11, in contrary to the right branch, is obtained without exploiting any anatomical information from MRI15 , i.e., the volume conduction head model is spherical and is computed analytically. By utilising anatomical constraint, the position of sources is restricted to be placed on a specific points, hence, the resulted lead-field matrix is more precise.

In addition, in order to uniquely recover the desired block-sparse solution β 0 , we need to consider block-sparse exact recovery conditions, which are going to be discussed in the next chapter.

Introduction

This chapter contains the main contribution of the Ph.D. thesis. In this chapter, a general framework for block-sparse exact signal recovery in an USLE1 problem, i.e., Block-ERC2 , is proposed. The framework consists of four theoretical and one algorithmic-dependent Block-ERC.

Chapter 2. Block-ERC in general dictionaries

The mentioned generality is in terms of

• structure of the dictionary, i.e., blocks could be orthonormal bases or there may exist intra-block orthonormality, however, in one of the proposed Block-ERC the computation of the Moore-Penrose pseudo-inverse of each block is involved, which requires the blocks to be full column rank,

• norm in the corresponding optimisation problem,

• length of blocks,

• parameters of the proposed block-sparsity measures, e.g., we are not constrained to only Euclidean norm criterion for determining the active blocks, and

• there is no constraint on the relationship between the dimensions of size of the dictionary Φ∈R m×n , except m<n.

As mentioned, the recovery conditions are proposed for the exact solution of the optimisation problem, therefore the uniqueness of the representation is the center of our focus. In addition, the exact recovery conditions ensure the uniqueness of the Support of the solution, i.e., the problem is Support recovery, and not coefficient recovery.

Through the aforementioned generalisations, we can relax some constraints as mentioned before, and improve the results of Eldar et al. [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], and extend some findings of Donoho et al. [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF], Elad and and Bruckstein [START_REF] Elad | On sparse signal representations[END_REF]; [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF], and Gribonval and Nielsen [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]; [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF] from element-wise (scalar-wise) to block-wise (vector-wise) framework [START_REF] Afdideh | Recovery Guarantees for Mixed Norm p 1 ,p 2 Block Sparse Representations[END_REF]. This chapter is a major extended version of [START_REF] Afdideh | Recovery Guarantees for Mixed Norm p 1 ,p 2 Block Sparse Representations[END_REF]. Section 2.2 presents the required terminology and tools to establish the main block-sparse recovery conditions which are introduced in Section 2.3. At last, to demonstrate the supremacy of the proposed theoretical recovery conditions in comparison to the state-of-the-art results, some numerical experiments are implemented in Section 2.4, and finally conclusions in Section 2.5 terminate this chapter.

Block-sparsity

Firstly, we need to redefine the structure of the dictionary and the representation vector and consequently the model. Then, an appropriate optimisation problem yielding block-sparse solutions should be investigated. Finally, suitable block-wise characterisations of the dictionary, which would be the base material for proposing the solution's uniqueness will be required. All the above mentioned requirements for establishing Block-ERC are investigated in this section through the following parts of structure and model, optimisation problem, and characterisations.

Structure and model

Consider the representation vector β can be viewed as a concatenation of K individual blocks:

β = βT [1] , • • • , βT [k] , • • • , βT [K] T ,
where, the k th block of β can be represented as:

β [k] = β1 [k] , • • • , βd k [k] T , where, βd k [k]=[ βi | i= k-1 j=1 d j +d k ],
and the length of k th block is d k , whereas the vector of the blocks' length is:

d = [d 1 , • • • , d K ] ,
where, K k=1 d k =n, i.e., the block structure is non-overlapped, and there is no constraining relationship between any d k and m. Then, the block structure of the representation vector β, which is consecutive d k -length vectors, is assumed to be known a priori. Similarly, the following block-wise structure is assumed for the dictionary Φ, which can be viewed as a concatenation of all K individual blocks:

Φ = [Φ [1] , • • • , Φ [k] , • • • , Φ [K]] ,
where, Φ[k]∈R m×d k , and as it is mentioned there is not any imposed relationship between m and n, other than m<n. The k th block is defined as the d k columns of matrix Φ:

Φ [k] = [ϕ 1 [k] , • • • , ϕ d k [k]] ,

Optimisation problem

As mentioned before, in the conventional element-wise sparse recovery problem, the goal is to extract a representation vector among the set of eligible solutions which is the sparsest one, i.e., a representation with the fewest non-zero elements. But in the framework of block-sparsity, the fewest active blocks are of interest, and not fewest non-zero elements. Any active block k has at least one non-zero element in β[k] which results in its non-zero p norm (for any p), which can be used for determining the active blocks. Usually, p is assigned to two, because the 2 norm is a rotational invariant measure. Here we use the p norm in more general cases of 0≤p and 1≤p.

As we said, a representation β is called block k-sparse if it has at most k active blocks, i.e., β p,0 ≤k. The p,0 pseudo-mixed-norm of β, i.e., k I( β[k] p ), which can be used for counting the number of active blocks, measures the activity of each block in p norm sense and the sparsity of the active blocks in 0 pseudo-norm sense. Therefore, block's p norm for any p could be a suitable criterion for determining the activity of blocks and then using another sparsity inducing norm applied on the resulted vector containing the p norm of each block could implement the concept of block-sparsity.

The previously-introduced (equation (1.4), page 17) general constrained p norm optimisation problem P p,q,ε (a, b) : min a a p s.t. b q ≤ε, is defined to recover sparse solution, i.e., solution with the fewest non-zero elements, with the widely used problem P p : min β β p s.t. y=Φβ. Then, the general constrained p 1 ,p 2 (pseudo-)mixed-norm optimisation problem P (p 1 ,p 2 ),q,ε (a, b) : min a a p 1 ,p 2 s.t. b q ≤ε, introduced (equation (1.16), page 34) to recover equally-sized block-sparse solution, i.e., solution with the fewest active blocks, where, ∀k, d k =d, with the widely used problem P p 1 ,p 2 : min β β p 1 ,p 2 s.t. y=Φβ. In this chapter, we introduce the following constrained w p 1 ,p 2 weighted (pseudo-)mixed-norm optimisation problem P (w;p 1 ,p 2 ),q,ε (a, b) in a more general case to recover differently-sized block-sparse solution, and cover all previously introduced optimisation problems: Definition 2.1 (Constrained w p 1 ,p 2 weighted (pseudo-)mixed-norm optimisation problem).

P (w;p 1 ,p 2 ),q,ε (a, b) : min a a w;p 1 ,p 2 s.t. b q ≤ ε,
w p 1 ,p 2 weighted (pseudo-)mixed-norm of a vector a, i.e., a w;p 1 ,p 2 , is defined by following definition:

Definition 2.2 (Weighted (pseudo-)mixed-norm). The w p 1 ,p 2 weighted (pseudo-)mixed-norm of a block-structured vector a=[a T [1], • • • , a T [k], • • • , a T [K]] T , where, a[k]=[a 1 [k], • • • , a d k [k]] T , is defined as: a w;p 1 ,p 2 def =                  k I w k a [k] p 1 , for p 2 = 0 k w p 2 k a [k] p 2 p 1 1 p 2 , for 0 < p 2 < +∞ max k w k a [k] p 1 , for p 2 = ∞,
where, I(a) is the indicator function, i.e., I(a

) def = 1, if |a| >0 0, if a=0
, and w=[w 1 , • • • , w K ] is the weight vector.

The w p 1 ,p 2 for either 0≤p 1 <1 or 0≤p 2 <1 is called weighted pseudo-mixed-norm, consequently, the corresponding problem P (w;p 1 ,p 2 ),q,ε (a, b) is called constrained w p 1 ,p 2 weighted pseudo-mixed-norm optimisation problem. w p 1 ,p 2 weighted (pseudo-)mixed-norm when all the weights are 1, i.e., w=1 1×K , is equal to the p 1 ,p 2 (pseudo-)mixed-norm, i.e., 1 p 1 ,p 2 ≡ p 1 ,p 2 . In addition, for p 1 >0 and p 2 =0, the w p 1 ,0 weighted pseudo-mixed-norm is independent of weight vector w, so it equals to ordinary pseudo-mixed-norm, i.e., w p,0 ≡ p,0 for p>0.

Although P (w;p 1 ,p 2 ),q,ε (a, b) with carefully selected values for p 1 and p 2 , is defined generally and can cover lots of scenarios, but the proposed conditions in this study concentrate on the following special cases:

• a: a is chosen to be the representation vector, i.e., a=β.

• b: b is selected as r, i.e., b=r=y-Φβ.

• ε (and q): The representation vector β is minimised in a block-sparse exact signal recovery problem, hence, ε should be zero, i.e., P (w;p 1 ,p 2 ),q,0 (β, r). For the sake of simplicity we refer to P (w;p 1 ,p 2 ),q,0 (β, r) as P w;p 1 ,p 2 , because regardless of choice of q, the r q =0 directly indicates to the noiseless model, so the only parameters to be determined are w, p 1 and p 2 . Therefore, the utilised optimisation problem is P w;p 1 ,p 2 : min β β w;p 1 ,p 2 s.t. y=Φβ.

• w: Throughout this thesis, only two cases for w is considered. In a block-structured vector a=[a

T [1], • • • , a T [K]] T with length vector d=[d 1 , • • • , d K ], w k in w k a[k] p 1 is either d -1/p 1 k
or 1. Therefore, although in general the weight vector w can consist of any arbitrary values, in this work it is considered as d -1/p 1 and the corresponding problem is represented as P w;p 1 ,p 2 , except when mentioned explicitly that it is a vector of ones, i.e., 1 1×K , and it can be represented as P 1;p 1 ,p 2 . But P 1;p 1 ,p 2 is equal to P p 1 ,p 2 , so we refer this case to P p 1 ,p 2 . Therefore, we have two cases of P p 1 ,p 2 and P w;p 1 ,p 2 , where the weight vector w is:

∀k ∈ {1, • • • , K} , w k = d -1 p 1 k .
• p 1 and p 2 : The main proposed Block-ERC include the cases 0=p 2 ≤p 1 , and the more general one of 0≤p 2 ≤1≤p 1 .

It is obvious that, if the size of all the blocks is chosen to be 1, i.e., ∀k, d k =1, then the block-sparse exact signal recovery reduces to the conventional exact signal recovery, i.e., P w;p 1 ,p 2 ≡P p 1 ,p 2 ≡P p 2 . In addition, the problem P w;p 1 ,p 2 for 0=p 2 <p 1 is independent of the weight vector w and equals to ordinary pseudo-mixed-norm problem, i.e., P w;p,0 ≡P p,0 for p>0.

As it can be seen in Section 1.4.2, most of the previous works are based on Euclidean norm as a measure of block activity, i.e., P 2,p . However, it has been shown that by using norms other than Euclidean, the performance of the selection of group of variables increases [START_REF] Zhao | The Composite Absolute Penalties Family for Grouped and Hierarchical Variable Selection[END_REF]. So, we study the impact of other norms.

In this chapter, we first introduce Block-ERC based on P p,0 in two different cases of p≥0 and p≥1. Then, we generalise the results to P p 1 ,p 2 and P w;p 1 ,p 2 , where, the activity of blocks is measured by p 1 norm, p 1 ≥1, and the sparsity of the blocks by p 2 (pseudo-)norm, 0≤p 2 ≤1.

Remark 2.1 (P w;p 1 ,p 2 v.s. P p 1 ,p 2 ). Notice that in general: • As mentioned before, if 0=p 2 <p 1 then the w p 1 ,p 2 weighted pseudo-mixed-norm optimisation problem is equivalent to the p 1 ,p 2 pseudo-mixed-norm optimisation problem, i.e., ∀p>0, P w;p,0 ≡P p,0 .

The selection of the values of the (pseudo-)mixed-norm coefficients p 1 and p 2 in the corresponding optimisation problems P p 1 ,p 2 and P w;p 1 ,p 2 is very important. Since the optimisation problems P p 1 ,p 2 and P w;p 1 ,p 2 are presented in a general case, by selecting a certain values of the (pseudo-)mixed-norm coefficients p 1 and p 2 , one can switch the problem between the conventional sparsity and block-sparsity problem. Map of the values of the (pseudo-)mixed-norm coefficients p 1 and p 2 , which leads to sparsity and block-sparsity problems is shown in figure 2.2. Figure 2.2 represents schematically our regions of interest in orange colour for the (pseudo-)mixed-norm coefficients p 1 and p 2 in a block sparsity problems P p 1 ,p 2 or P w;p 1 ,p 2 , compared to the traditional sparsity problem, shown in blue colour region. As it can be seen in figure 2.2, considering additional constraint of proposed block sparsity (orange-coloured regions) on the representation vector, gives rise to a wider region of the (pseudo-)mixed-norm coefficients of the corresponding optimisation problem compared to the existing block sparsity (green-coloured regions), and conventional sparsity (blue-coloured regions). In figure 2.2, the most frequently studied cases are represented by solid circles. For instance, P 0,1 (≡P 0 ) and P 1,1 (≡P 1 ), which are represented by blue circles (left and right one, respectively) are the widely used cases in sparse exact recovery problems, whereas P 2,0 and P 2,1 , which are represented by green circles (down and top one, respectively) are the widely used cases in block-sparse exact recovery problems. Remark 2.2 (Sparsity regions in figure 2.2). For 0<p 1 =p 2 =p≤1, the block-sparse exact signal recovery optimisation problems P p,p and P w;p,p reduce to the traditional exact signal recovery problem, i.e., P p,p ≡P p and P w;p,p ≡d -1/p P p (∀k, d k =d). In other words, in the common region of sparsity and block-sparsity, i.e., p 1 =p 2 =1 (the right blue circle in figure 2.2), and under certain conditions, the conventional exact signal recovery problem can recover the block-sparse representation. In addition, for p 1 =0 and p 2 =1 (the left blue circle in figure 2.2), the P 0,1 problem reduces to the classic exact signal recovery problem, i.e., P 0,1 ≡P 0 , but it is still non-convex.

Remark 2.3 (Block-sparsity regions in figure 2.2). As shown in figure 2.2 the proposed block-sparsity region (orange region) covers the existing most frequently studied cases in block sparsity problem (green circles) and the widely used tractable case in the traditional sparsity problem (the right blue circle) in literature.

Characterisations

To define the Block-ERC, it is necessary to introduce some notations. Except when mentioned explicitly, (1) the block length d k ∈N, where, K k=1 d k =n, and d max and d min are the maximum and minimum block length, respectively, and (2) the characterisations coincide with the results known in the conventional element-wise case if all blocks are of size 1, i.e.,

d 1 = • • • =d K =1.
We first give some general definitions, which then will be used to prove uniqueness results.

Definition 2.3 (Block-Support and Block-Cardinality). The Block-Support and Block-Cardinality of a representation vector β∈R n are defined as:

Block-Support: S b (β) def = k : β [k] p = 0, 1 ≤ k ≤ K ∀p ≥ 0, Block-Cardinality: |S b (β)| def = β p,0 ∀p ≥ 0.
β p,0 simply counts the number of active blocks in the sense of their non-zero p norm.

In fact, considering the definition of β p,0 , each block is first represented by its p norm. So, non-zero representatives indicate active blocks. Then, by applying the 0 pseudo-norm on the blocks' representatives, the active blocks are counted. When

d 1 = • • • =d K =1, the
Block-Support and Block-Cardinality will be equivalent to the conventional Support and Cardinality (see Section 1.3.2), respectively. Definition 2.4 (Dictionary Kernel). The Block-Kernel or block null space of a dictionary Φ∈R m×n , is equivalent to its usual Kernel:

Block-Ker(Φ) def = x ∈ R n , K k=1 Φ [k] x [k] = Φx = 0 = Ker(Φ).
Actually, the Block-Kernel is the block-wise definition of the conventional Kernel of a dictionary. Because of their equivalency and for the sake of simplicity, we use the notation of conventional Kernel.

Regarding the uniqueness of the solution of P p,0 problem, an interesting question is "What is the minimal value of x p,0 for x∈Ker(Φ)\{0}, and p>0?". To elucidate the significance of the mentioned question, assume that β 0 and β 1 are both solutions to problem P p,0 , i.e., y=Φβ 0 =Φβ 1 . Then Φ(β 0 -β 1 )=0, so x=β 0 -β 1 ∈Ker(Φ). Clearly, x p,0 = β 0 -β 1 p,0 ≤ β 0 p,0 + β 1 p,0 =2 β 0 p,0 since both β 0 p,0 and β 1 p,0 attain the minimum p,0 pseudo-mixed-norm. Consequently, assuming two solutions we reach to x p,0 /2≤ β 0 p,0 .

Hence, if β 0 p,0 < min x∈Ker(Φ)\{0} x p,0 /2, then no other solution β 1 exists for which β 0 p,0 = β 1 p,0 . We stress the importance of the latter through the following definition which is a straightforward generalisation of the conventional Spark [GN03b]; [DE03b]: 2.2. Block-sparsity 53 Definition 2.5 (Block-Spark). The Block-Spark of a dictionary can be defined based on p,0 pseudo-mixed-norm:

Block-Spark(Φ) def = min x∈Ker(Φ)\{0}
x p,0 ∀p ≥ 0.

Block-Spark characterises the block null space of a dictionary using the p,0 pseudo-mixed-norm. By definition, any vector x in the block null space of the dictionary, i.e., K k=1 Φ[k]x[k]=0, must satisfy ∀p≥0, x p,0 ≥Block-Spark(Φ). This important block-wise characterisation of the dictionary is the extension of the conventional Spark defined in equation (1.5) page 21.

Because of the fact that the proposed characterisation is a block-wise generalisation of the conventional element-wise case, in a unit block size setting, i.e., ∀k, d k =1, our proposed Block-Spark is equivalent to the conventional Spark, i.e., Block-Spark(Φ)≡Spark(Φ).

The following property shows the relationship between the proposed Block-Spark and the conventional Spark, which will be used in Section 2.3.1 to demonstrate the improvement of the recovery conditions based on Block-Spark compared to the ones based on the conventional Spark: Due to non-convexity of the pseudo-mixed-norm • p,0 , Block-Spark is not computationally tractable. Therefore, the following block coherence measure, which is the block-wise extension of the conventional MCC 3 is proposed: 54 Chapter 2. Block-ERC in general dictionaries Definition 2.6 (Block coherence). The (q, p)-Block Mutual Coherence Constant (Block-MCC q,p ) of a dictionary is defined ∀(q, p)∈R 2 >0 as:

M q,p (Φ) def = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q→p ,
where, Φ † [k] is Moore-Penrose pseudo-inverse of the full column rank block Φ[k]∈R m×d k , and Φ q→p is the q→p operator-norm of dictionary Φ, which can be computed through any of the following expressions [START_REF] Aaron | Topics in sparse approximation[END_REF]; [START_REF] Golub | Matrix Computations[END_REF]:

A q→p def = max a =0 Aa p a q = max a q =1
Aa p = max

a q ≤1
Aa p .

For a unit block size setting,

d 1 = • • • =d K =1
, and 2 -normalized columns of Φ, as expected, Block-MCC q,p is equivalent to conventional MCC (page 22), i.e., M q,p (Φ)≡M (Φ), because each block of d k columns Φ[k] will be a single column, i.e., Φ[k]=ϕ k , and for the vector ϕ k , we have ϕ † k =ϕ T k and for the scalar

ϕ T k ϕ k , we have ϕ T k ϕ k q→p =|ϕ T k ϕ k | for ∀(q, p)∈R 2 ≥0 , so M q,p (Φ)= max k,k =k |ϕ T k ϕ k |=M (Φ).
Although Block-MCC q,p is valid for ∀ (q, p) ∈R 2 >0 , in practice, because of the computational complexity, only some basic operator-norms can be calculated [START_REF] Aaron | Topics in sparse approximation[END_REF]. Table 2.1 explains these basic operator-norms.

In order to investigate the bounds and relationship of the proposed Block-MCC q,p to the existing dictionary characterisations, we need to study the properties of the q→p operator-norm of a matrix, which in turn requires to establish the bounds of division of norm of two vectors: Property 2.2 (Bounds of two vector norms division). ∀(q, p)∈R 2 >0 , and ∀a∈R d , we have:

min 1, d 1 p -1 q ≤ a p a q ≤ max 1, d 1 p -1 q .
The proof of Property 2.2 is provided in Section A.1 (page 119). The general bounds of

p = 1 p = 2 p = ∞ q = 1 Maximum 1 norm of a column
1 ≤ a 1 a 2 ≤ √ d, 1 ≤ a 2 a ∞ ≤ √ d, 1 ≤ a 1 a ∞ ≤ d.
Property 2.3 (Bounds of two (pseudo-)mixed-norms division). ∀(q, p)∈R 2 >0 , for a= [a [1] , . . . , a [K]] and ∀k, a [k] ∈R d k , and for weighted (pseudo-)mixed-norms we have:

a w;p,1 a w;q,1 ≥ min k min 1, d 1 q -1 p k = 1, if 0 < q ≤ p d 1 q -1 p max , if 0 < p < q , a w;p,1 a w;q,1 ≤ max k max 1, d 1 q -1 p k = d 1 q -1 p max , if 0 < q ≤ p 1, if 0 < p < q .
Similarly, for non-weighted (pseudo-)mixed-norms we have:

a p,1 a q,1 ≥ min k min 1, d 1 p -1 q k = d 1 p -1 q max , if 0 < q ≤ p 1, if 0 < p < q , a p,1 a q,1 ≤ max k max 1, d 1 p -1 q k = 1, if 0 < q ≤ p d 1 p -1 q max , if 0 < p < q .
Proof. For any real fractions x 1 /y 1 , . . . , x K /y K with positive denominators, we have [pah05]:

min x 1 y 1 , . . . , x K y K ≤ x 1 + . . . + x K y 1 + . . . + y K ≤ max x 1 y 1 , . . . , x K y K ,
where, equality happens if and only if all fractions x 1 /y 1 , . . . , x K /y K are equal.

Then, for any k, considering

x k = a[k] p /d 1/p k and y k = a[k] q /d 1/q k , we have x k /y k =d 1/q-1/p k a[k] p / a[k] q .
On the other hand, from Property 2.2 (bounds of two vector norms division), we have

∀(q, p)∈R 2 >0 , ∀b∈R d : min{1, d 1/p-1/q }≤ b p / b q ≤ max{1, d 1/p-1/q }. Consequently, min{1, d 1/q-1/p k }≤x k /y k ≤ max{1, d 1/q-1/p k
}, which by considering the above-mentioned inequality, we have:

min k min 1, d 1 q -1 p k ≤ k x k k y k ≤ max k max 1, d 1 q -1 p k .
On the other hand, from Definition 2.2 (weighted (pseudo-)mixed-norm) we have k x k = a w;p,1 and k y k = a w;q,1 , which proves the bounds. Similarly, for any k, considering x k = a[k] p and y k = a[k] q , the bounds for division of two non-weighted (pseudo-)mixed-norms can be obtained.

Chapter 2. Block-ERC in general dictionaries Property 2.4 ( q→p operator-norm properties). Assuming A∈R m×n , B∈R m×n , and C∈R n×l , the q→p operator-norm of a matrix satisfies the following properties:

• Nonnegativity: ∀(q, p)∈R 2 ≥0 : A q→p ≥0.

• Positivity: ∀(q, p)∈R 2 ≥0 : A q→p =0 if and only if A=0.

• Homogeneity: ∀q∈R ≥0 , ∀p∈R >0 , ∀α∈R : αA q→p = |α| A q→p .

• Triangle inequality: ∀q∈R ≥0 , ∀p∈R ≥1 , for p=0 :

A+B q→p ≤ A q→p + B q→p .
Remark 2.4 (Generalised matrix norm). Any matrix norm definition that satisfies the above four properties is called generalised matrix norm [START_REF] Johnson | Matrix Analysis[END_REF]. All the above-mentioned properties hold true ∀q∈R ≥0 , in contrast to p. Table 2.2 summarises the ranges of p in which different properties hold true. As it can be seen in table 2.2, for ∀q∈R ≥0 and ∀p∈R ≥1 , the q→p operator-norm satisfies all the above four properties, hence it is a generalised matrix norm in the mentioned range for q and p.

• Submultiplicativity: In general, we have:

∀(q, p) ∈ R 2 >0 , A C q→p ≤ A q→p C q→p max 1, n 1 q -1 p .
For q≥p we have max{1, n 1/q-1/p }=1, and then there exists submultiplicativity property, i.e., for q≥p>0, we have A C q→p ≤ A q→p C q→p .

Remark 2.5 (Matrix norm). Any norm definition that satisfies the above five properties is called matrix norm [START_REF] Johnson | Matrix Analysis[END_REF]. The q→p operator-norm ∀q∈R ≥0 and ∀p∈R ≥1 satisfies the first four properties (generalised matrix norm), whereas for q≥p>0 satisfies the fifth property. Then, q→p operator-norm for q≥p≥1 satisfies all the above five properties, hence it is a matrix norm.

• Bounds: We define the following four types of bounds and inequalities for the q→p operator-norm: 1) ∀ (q, p, q , p ) ∈R 4 >0 , we have: which the bounds are based on the operator norm having one of the original domains, either q, or p.

A q→p ≥ max min 1, m 1 p -1 p A q→p , min 1, n 1 q -1 q A q →p , A q→p ≤ min max 1, m 1 p -1 p A q→p , max 1, n 1 q -1 q A q →p , ∀q∈R ≥0 p = 0 0 < p < 1 p ≥

Block-sparsity 57

1 ≤ A 1→1 A 1→2 ≤ m 1 2 , n -1 ≤ A 1→1 A ∞→∞ ≤ m, n -1 2 ≤ A 2→2 A ∞→∞ ≤ m 1 2 , 1 ≤ A 1→2 A 1→∞ ≤ m 1 2 , 1 ≤ A 1→1 A 1→∞ ≤ m, 1 ≤ A 2→2 A 1→2 ≤ n 1 2 , m -1 2 ≤ A ∞→∞ A 1→2 ≤ n, n -1 2 ≤ A 1→2 A 2→∞ ≤ m 1 2 , n -1 2 ≤ A 1→1 A 2→2 ≤ m 1 2 , 1 ≤ A 2→2 A 1→∞ ≤ (m n) 1 2 , 1 ≤ A ∞→∞ A 1→∞ ≤ n, n -1 2 ≤ A 1→∞ A 2→∞ ≤ 1. n -1 2 ≤ A 1→1 A 2→∞ ≤ m, 1 ≤ A 2→2 A 2→∞ ≤ m 1 2 , 1 ≤ A ∞→∞ A 2→∞ ≤ n 1 2 , n -1 ≤ A 1→2 A ∞→∞ ≤ m 1 2 ,
2) In addition, the bounds can be based on the operator norm having totally new domains, e.g., q and p , i.e., ∀ (q, p, q , p ) ∈R 4 >0 we have:

A q→p ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q A q →p , A q→p ≤ max 1, m 1 p -1 p max 1, n 1 q -1 q A q →p .
Remark 2.6 ( q→p operator-norm inequalities). The above-mentioned general inequalities for basic tractable q→p operator-norms based on table 2.1 (page 54) is shown in table 2.3, which includes the following standard inequalities [START_REF] Golub | Matrix Computations[END_REF]:

A 1→∞ ≤ A 2→2 ≤ √ mn A 1→∞ , 1 √ n A ∞→∞ ≤ A 2→2 ≤ √ m A ∞→∞ , 1 √ m A 1→1 ≤ A 2→2 ≤ √ n A 1→1 .
In addition, in general (tractable and intractable), the q→p operator-norm inequalities for a fixed p or q is shown schematically in figure 2.3.

Figure 2.3: q→p operator-norm inequalities for common q→p operator-norms according to table 2.1 (page 54).

3) Another useful lower-and upper-bound ∀ (q, p, q , p ) ∈R 4 >0 are:

A q→p ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q min 1, m 1 p -1 2 min 1, n 1 2 -1 q min {m, n} A F , A q→p ≤ max 1, m 1 p -1 p max 1, n 1 q -1 q max 1, m 1 p -1 2 max 1, n 1 2 -1 q A F ,
where, the Frobenius norm is defined as

A F = m i=1 n j=1 |a i,j | 2 . 4)
In addition, the q→p operator-norm of any matrix A is less than or equal to the q→p operator-norm of another matrix B, in which all the elements are the maximum absolute value of the elements of first matrix. It also holds true, when all the on-diagonal entries of A and B are set to zero:

∀i, j, ∀ (q, p) ∈ R 2 >0 , if |a i,j | ≤ b i,j = max i,j |a i,j | ⇒ A q→p ≤ B q→p , and
∀i, j, ∀ (q, p) ∈ R 2 >0 , if |a i,j | ≤ b i,j = max i,j |a i,j | , i = j a i,j = b i,j = 0, i = j ⇒ A q→p ≤ B q→p .
See Section A.2 (page 121) for the proof of Property 2.4. Now, based on the above-mentioned properties of the q→p operator-norm introduced in Property 2.4 ( q→p operator-norm properties), in the following property we investigate the possible relationship between different Block-MCC q,p characterisations (Definition 2.6, page 54) with basic (q, p) pairs according to table 2.1 (page 54):

Property 2.5 (Block-MCC q,p inequalities). The different Block-MCC q,p characterisations calculated for basic tractable q→p operator-norms of table 2.1 (page 54) have the following relationships:

M 1,1 (Φ) ≤ M 1,2 (Φ) ≤ M 1,∞ (Φ) , M 2,2 (Φ) ≤ M 1,2 (Φ) ≤ M 1,∞ (Φ) , M 2,2 (Φ) ≤ M 2,∞ (Φ) ≤ M 1,∞ (Φ) , M ∞,∞ (Φ) ≤ M 2,∞ (Φ) ≤ M 1,∞ (Φ) ,
while their general relationship is represented schematically in figure 2.4.

The proof of Property 2.5 is provided in Section A.3 (page 126).

Since the recovery conditions started with orthonormal bases, the following property shows Definition 2.6 (block coherence) in special case of intra-block orthonormality: Property 2.6 (Block-MCC q,p for intra-block orthonormality). If the dictionary Φ has intra-block orthonormality, i.e., for 1≤k≤K,

Φ T [k]Φ[k]=I d k , then ∀(q, p)∈R 2
>0 we have:

M q,p (Φ) = max k,k =k d -1 p k d 1 q k d max Φ T [k] Φ k q→p .
Figure 2.4: Block-MCC q,p inequalities for common q→p operator-norms according to table 2.1 (page 54).

Proof. Taking into account that, for an orthonormal matrix we have A † =A T , the result is immediately obtained from the Definition 2.6 (block coherence).

Block-MCC q,p is a measure of similarity between different blocks of the columns of the dictionary, which is the block-wise extension of the conventional MCC (page 22). Since the dictionary coherence has an inverse effect on the theoretical recovery conditions, the lower bounds for coherence is desired. When a dictionary has intra-block orthonormality, Block-MCC q,p can be bounded, as shown in the following property: Property 2.7 (Block-MCC q,p upper-bound with intra-block orthonormality I). For a dictionary Φ with intra-block orthonormality, i.e., for 1≤k≤K,

Φ T [k]Φ[k]=I d k , the upper-bound of Block-MCC q,p is shown in table 2.4.
The proof of Property 2.7 is provided in Section A.4 (page 129). Later, in Property 2.9, we will show another upper-bounds for Block-MCC q,p based on the relationship with conventional MCC. Property 2.8 investigates the bounds of Block-MCC q,p in terms of the conventional MCC (page 22).

Property 2.8 (Block-MCC q,p bounds). Block-MCC q,p is bounded based on the conventional MCC, i.e., M (Φ)

def = max k,k =k | ϕ k , ϕ k |.
1) Suppose for a dictionary with full column rank blocks, we have M (Φ)<d

1/q-2 max (d max -1) -1/2 (q, p) (1, 1) (1, 2) (1, ∞) (2, 2) (2, ∞) (∞, ∞) max M q,p (Φ) d -1 2 min m 1 2 d -1 2 min m 1 2 m d -1 2 min d -1 2 max d -1 2 max m 1 2 d -1 2 max m 1 2
Table 2.4: Upper-bound of Block-MCC q,p obtained based on the relationship with unit 2→2 operator-norm, for different basic (q, p) pairs and for a dictionary Φ∈R m×n with intra-block orthonormality.

(q = q , p = p ) (1, 1) (1, 2) (1, ∞) (2, 2) (2, ∞) (∞, ∞) max M q,p (Φ) d 1 2 max d 1 2 max d max 1 d 1 2 max d 1 2 max Table 2
.5: Upper-bound of Block-MCC q,p obtained based on the relationship with MCC, for different basic values of (q, p) pairs and for a dictionary with intra-block orthonormality. and q≥p≥1, then:

0 ≤ M q,p (Φ) ≤ d 3 2 -1 p max M (Φ) 1 -d 2-1 q max (d max -1) 1 2 M (Φ)
.

2) For a dictionary with intra-block orthonormality, ∀(q, p, q , p )∈R 4 >0 we have:

0 ≤ M q,p (Φ) ≤ M (Φ) d max max k,k =k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k × max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k .
The proof of Property 2.8 is provided in Section A.5 (page 131). In addition, we have:

Property 2.9 (Block-MCC q,p upper-bound with intra-block orthonormality II).

For a dictionary with orthonormal blocks, Block-MCC q,p is upper-bounded based on its relationship with MCC, i.e., ∀(q, p, q , p )∈R 4 >0 , we have:

0 ≤ M q,p (Φ) ≤ d -1 max max k,k =k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k × max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k .
The mentioned upper-bound of Block-MCC q,p for basic tractable (q, p) pairs is shown in table 2.5.

Proof. Since MCC is upper-bounded by one, from Property 2.8 the proof is done.

Notice that in Property 2.7 we introduced the bounds of Block-MCC q,p based on the bounds of the operator-norms in terms of the unit 2→2 operator-norm, whereas now in Property 2.9 we introduced another upper-bound based on its relationship with MCC and considering the upper-bound of MCC. By minimising upper-bounds of Block-MCC q,p in properties 2.7 (table 2.4) and 2.9 (table 2.5), the final resulted upper-bounds are shown in table 2.6.

Remark 2.7. By comparing the bounds of conventional MCC, i.e., 1/ √ m≤M (Φ)≤1 (equation (1.7), page 22), with the upper-bounds of the proposed dictionary coherence in table 2.6, where, there is intra-block orthonormality and considering d min =d max =m (orthonormal complete bases), it can be seen that, although the obtained upper-bound for Block-MCC 1,∞ is

(q, p) (1, 1) and (1, 2) (1, ∞) (2, 2) (2, ∞) and (∞, ∞) max M q,p (Φ) min d -1 2 min m 1 2 , d 1 2 max min {m, d max } d -1 2 min d -1 2 max min d -1 2 max m 1 2 , d 1 2 max Table 2
.6: Upper-bound of Block-MCC q,p obtained based on properties 2.7 (table 2.4) and 2.9 (table 2.5), for different basic values of (q, p) pairs and for a dictionary Φ∈R m×n with intra-block orthonormality.

greater than that of conventional MCC (m v.s. 1), but both characterisations have the same upper-bound for (q, p) pairs of (1, 1), (1, 2), (2, ∞), and (∞, ∞), while the upper-bound for (2, 2) is 1/m, which is much less than the conventional unit upper-bound, and even less than the lower-bound of conventional MCC (1/m v.s.

1/ √ m).
If MCC is small enough, the dictionary is said to be incoherent, similarly if Block-MCC q,p is small enough, the dictionary will be block-incoherent. The term block-incoherent is previously used in some studies, of course with a different definition for the characterisation of the coherence of the dictionary which is called block-coherence [PV07]; [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF].

Next,

we demonstrate the relationship between the proposed general Block-MCC q,p and the conventional characterisation of Donoho and Huo, i.e.,

M (Φ 1 , Φ 2 ) def = max{max i,j |Φ -1 1 Φ 2 | i,j , max i,j |Φ -1 2 Φ 1 | i,j }
def = max k,k =k Φ T [k]Φ[k ] 2→2
/d explained on page 36 equation (1.17) [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], in block-sparsity domain.

Property 2.10 (Block-MCC q,p v.s. coherence of Donoho and Huo). For a dictionary Φ=[Φ 1 , Φ 2 ] with two invertible square blocks of same size, d 1 =d 2 =m, we have:

M 1,∞ (Φ) = M (Φ 1 , Φ 2 ) , and {M 1,1 (Φ) , M 1,2 (Φ) , M 2,2 (Φ) , M 2,∞ (Φ) , M ∞,∞ (Φ)} ≤ M (Φ 1 , Φ 2 ) ,
where, M (Φ 1 , Φ 2 ) is the dictionary characterisation of Donoho and Huo, i.e.,

M (Φ 1 , Φ 2 ) def = max{max i,j |Φ -1 1 Φ 2 | i,j , max i,j |Φ -1 2 Φ 1 | i,j
}, and M q,p (Φ) is the proposed Block-MCC q,p , i.e., M q,p (Φ)

def =d -1 max max k,k =k d -1/p k d 1/q k Φ † [k]Φ[k ] q→p .
Proof. Considering that 1→∞ operator-norm represents the maximum absolute value of matrix, i.e., A 1→∞ = max i,j |a i,j | (table 2.1, page 54), and the fact that here

Φ[k]=Φ k , the definition M (Φ 1 , Φ 2 ) def = max{max i,j |Φ -1 1 Φ 2 | i,j , max i,j |Φ -1 2 Φ 1 | i,j } (page 28) can be rewritten as M (Φ 1 , Φ 2 )= max k,k =k Φ -1 [k]Φ[k ] 1→∞ . On the other hand for the invertible block Φ[k], we have Φ -1 [k]=Φ † [k], so M (Φ 1 , Φ 2 )= max k,k =k Φ † [k]Φ[k ] 1→∞ . On the other hand, M 1,∞ (Φ)=d -1 max max k,k =k d k Φ † [k]Φ[k ] 1→∞ = max k,k =k Φ † [k]Φ[k ] 1→∞ , because ∀k, d k =d max (d 1 =d 2 =m). Then, M 1,∞ (Φ)= M (Φ 1 , Φ 2 ).
In addition from Property 2.5 (Block-MCC q,p inequalities), we know that M 1,∞ (Φ) upper-bounds the other basic tractable (q, p) pairs of table 2.1 (page 54), hence, the proof for the second part is done.

Property 2.11 (Block-MCC q,p v.s. block-coherence of Eldar et al.). For a dictionary with equally-sized blocks of length d and intra-block orthonormality, ∀(q, p)∈R 2 >0 we have:

M q,p (Φ) ≥ d 1 q -1 p min 1, d 1 p -1 2 min 1, d 1 2 -1 q M Eldar Inter (Φ) , M q,p (Φ) ≤ d 1 q -1 p max 1, d 1 p -1 2 max 1, d 1 2 -1 q M Eldar Inter (Φ) ,
where,

M
def = max k,k =k Φ T [k]Φ[k ] 2→2 /d
, and M q,p (Φ) is the proposed Block-MCC q,p , i.e., M q,p (Φ)

def =d -1 max max k,k =k d -1/p k d 1/q k Φ † [k]Φ[k ] q→p .
Then, for q=p=2, we have M 2,2 (Φ)=M Eldar Inter (Φ).

Proof. From Property 2.6 (Block-MCC q,p for intra-block orthonormality; page 58) in a special setting of

d 1 = • • • =d K =d, we have ∀(q, p)∈R 2 >0 : M q,p (Φ)=d 1/q-1/p-1 max k,k =k Φ T [k]Φ[k ] q→p .
Then, from Property 2.4 ( q→p operator-norm properties; second set of bounds), we can determine the bounds of q→p operator-norm based on q →p operator-norm, i.e., ∀A∈R m×n , ∀ (q, p, q , p ) ∈R 4 >0 : A q→p ≥(≤) min(max){1, m 1/p-1/p } min(max){1, n 1/q -1/q } A q →p , and by selecting q =p =2, we have:

M q,p (Φ) ≥ d 1 q -1 p -1 min 1, d 1 p -1 2 min 1, d 1 2 -1 q max k,k =k Φ T [k] Φ k 2→2 , M q,p (Φ) ≤ d 1 q -1 p -1 max 1, d 1 p -1 2 max 1, d 1 2 -1 q max k,k =k Φ T [k] Φ k 2→2 .

Next, considering the definition M Eldar

Inter (Φ)

def = max k,k =k Φ T [k]Φ[k ]
2→2 /d presented on page 36 equation (1.17), the proof is done.

The above introduced coherence constant Block-MCC q,p characterises the dictionary by means of the maximum coherence, but with the expense of heavier computations it is possible to better characterise it using a cumulative coherence.

Definition 2.7 (Cumulative Block-MCC q,p ). The cumulative Block-MCC q,p of a dictionary with full column rank blocks, is defined for all integers 1≤k≤K-1 and ∀(q, p)∈R 2 >0 as:

M q,p (Φ, k) def = max |Λ|=k max j / ∈Λ i∈Λ d -1 p i d 1 q j d max Φ † [i] Φ [j] q→p ,
where, Λ⊂{1, • • • , K} of Cardinality k.

Cumulative Block-MCC q,p measures the maximum total Block-MCC q,p between a fixed block and a collection of other blocks, which is the block-wise generalisation of the conventional cumulative MCC defined in (1.8), i.e., M (Φ, k)

def = max |Λ|=k max j / ∈Λ i∈Λ | ϕ i , ϕ j |.
As expected, for a unit block size scenario, i.e., d 1 = • • • =d K =1, cumulative Block-MCC q,p in Definition 2.7 is equivalent to the conventional cumulative MCC in (1.8) presented on page 23. The following property shows the cumulative Block-MCC q,p properties, which are similar to the cumulative MCC's one presented on page 24: Property 2.12 (Cumulative Block-MCC q,p properties). For any dictionary Φ with full column rank blocks, and Block-MCC q,p M q,p (Φ), ∀(q, p)∈R 2 >0 we have:

M q,p (Φ, 1) = M q,p (Φ) , and
M q,p (Φ, k) ≤ k M q,p (Φ) .
Proof. An investigation of the formula of cumulative Block-MCC q,p , shows that ∀(q, p)∈R 2

>0

and for k=1, it reduces to Block-MCC q,p , i.e.,:

M q,p (Φ, 1) = max |Λ|=1 max j / ∈Λ i∈Λ d -1 p i d 1 q j d max Φ † [i] Φ [j] q→p = max i max j =i d -1 p i d 1 q j d max Φ † [i] Φ [j] q→p = M q,p (Φ) .
To prove the second part, similar to the proof of Proposition 2.1 in [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF], ∀(q, p)∈R 2 >0 we have:

M q,p (Φ, k) = max |Λ|=k max j / ∈Λ i∈Λ d -1 p i d 1 q j d max Φ † [i] Φ [j] q→p ≤ max |Λ|=k i∈Λ M q,p (Φ) = k M q,p (Φ) .
According to the inter-block coherence defined by Eldar et al. in (1.17) page 36, i.e., M Eldar Inter (Φ)

def = max k,k =k Φ T [k]Φ[k ] 2→2 /d, [EB09]; [EKB10b]; [EKB10a]
, and the concept of cumulative coherence introduced by Tropp [Tro04a] (page 24), we extend the inter-block coherence and introduce the following definition: Definition 2.8 (Eldar's cumulative coherence). The cumulative inter-block coherence constant of a general dictionary Φ is defined for all integers 1≤k≤K-1 as:

M Eldar Inter (Φ, k) def = max |Λ|=k max j / ∈Λ i∈Λ 1 d Φ T [i] Φ [j] 2→2 , where, Λ⊂{1, • • • , K} is of Cardinality k and d 1 = • • • =d K =d.
As expected, for d=1, cumulative inter-block coherence constant in Definition 2.8 is equivalent to the conventional cumulative MCC defined in (1.8) presented on page 23. The following property shows the properties of cumulative inter-block coherence constant, which are similar to the properties of cumulative MCC presented on page 24:

Chapter 2. Block-ERC in general dictionaries Property 2.13 (Eldar's cumulative coherence properties). For any general dictionary Φ with equally-sized block structure, i.e., d 1 = • • • =d K =d, we have:

M Eldar Inter (Φ, 1) = M Eldar Inter (Φ) , and
M Eldar Inter (Φ, k) ≤ k M Eldar Inter (Φ) .
Proof. It is similar to the proof of Property 2.12 (cumulative Block-MCC q,p properties).

In the following property, the relationship between our two proposed cumulative coherences, i.e., cumulative Block-MCC q,p (Definition 2.7) and Eldar's cumulative coherence (Definition 2.8), is investigated: Property 2.14 (Cumulative Block-MCC q,p v.s. Eldar's cumulative coherence). For a dictionary Φ with orthonormal equally-sized blocks, we have:

M 2,2 (Φ, k) = M Eldar Inter (Φ, k)
Proof. From Definition 2.7 (cumulative Block-MCC q,p ), we have ∀(q, p)∈R 2

>0 : M q,p (Φ, k) def =d -1 max max |Λ|=k max j / ∈Λ i∈Λ d -1/p i d 1/q j Φ † [i]Φ[j] q→p .
On the other hand, for orthonormal blocks we have Φ † [i]=Φ T [i], and by selecting q=p=2 for an equally-sized block structure, i.e.,

d 1 = • • • =d K =d, we get M q,p (Φ, k)=d -1 max |Λ|=k max j / ∈Λ i∈Λ Φ T [i]Φ[j] 2→2 =M Eldar
Inter (Φ, k).

2.3. Block-sparse exact recovery condition 65

Block-sparse exact recovery condition

In this section Block-ERC based on above-mentioned different characterisations and properties of a dictionary will be studied, which can be considered as the main theoretical contribution of this thesis.

Block-ERC based on Block-Spark

To introduce the Block-ERC we need a block-wise version of the conventional uncertainty principle defined in (1.9) page 26. Therefore, generalising the ideas from [START_REF] Gorodnitsky | Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm[END_REF]

β 0 p,0 + β 1 p,0 ≥ Block-Spark(Φ).
The proof of Lemma 2.1 is provided in Section A.6 (page 136).

As expected, for conventional element-wise sparse recovery case, i.e., d 1 = • • • =d K =1, block-sparse uncertainty principle based on Block-Spark is equivalent to the conventional Spark-based uncertainty principle of (1.9) on page 26, i.e., β 0 0 + β 1 0 ≥Spark(Φ).

Using the proposed Lemma 2.1 (Block-UP based on Block-Spark), we introduce the following Block-ERC based on Block-Spark : Theorem 2.1 (Block-ERC based on Block-Spark). For any general dictionary Φ, such that y=Φβ 0 , ∀p≥0, if

β 0 p,0 < Block-Spark(Φ) 2 ,
then β 0 is the unique solution to the P p,0 optimisation problem.

Proof. Considering Lemma 2.1 (Block-UP based on Block-Spark), suppose that in addition to β 0 there is another solution β 1 , that satisfies the same linear model, i.e., y=Φβ 0 =Φβ 1 .

Since it is assumed that the number of active blocks of the candidate solution β 0 is less than Block-Spark(Φ)/2, from Lemma 2.1 it can be deduced that any alternative solution such as β 1 necessarily is denser and has more than Block-Spark(Φ)/2 active blocks.

Chapter 2. Block-ERC in general dictionaries

As expected, for d 1 = • • • =d K =1, Theorem 2.1 (Block-ERC based on Block-Spark) is equivalent to the conventional Spark-based ERC of (1.10) on page 26, i.e., β 0 0 <Spark(Φ)/2.

As mentioned earlier, from a mathematical point of view, exploiting the block structure information of the representation leads to improved recovery conditions, i.e., conditions with higher sparsity levels, i.e., more non-zero coefficients.

For investigating this claim, consider a block-sparse representation β 0 , which satisfies the condition of the Theorem 2.1 (Block-ERC based on Block-Spark), i.e., β 0 p,0 <Block-Spark(Φ)/2. The number of non-zero elements is at most d max times greater than the number of active blocks, i.e., ∀p≥0, β 0 0 ≤d max β 0 p,0 , where, equality occurs when all of the elements in each of the blocks are non-zero in an equally-sized blocks case. Therefore, using Theorem 2.1 (Block-ERC based on Block-Spark) we have:

β 0 0 < d max Block-Spark(Φ) 2 . (2.1)
Now, if we treat the block-sparse representation β 0 as a conventional sparse representation, i.e., without exploiting its block structure, the sufficient recovery condition would be β 0 0 <Spark(Φ)/2 as explained in (1.10) on page 26, and by comparing with (2.1), it is clear that for showing the benefit of the assumption of the block structure on the recovery condition, the relation between d max Block-Spark(Φ) and Spark(Φ) should be investigated.

According to the relationship in Property 2.1 (Block-Spark v.s. Spark, page 53), i.e., d Block-Spark(Φ)≥Spark(Φ), it is clear that the right-hand side of the inequality (2.1) is greater than or equal to the right-hand side of the inequality (1.10), i.e., Spark(Φ)/2, because we have d max ≥d.

Therefore, exploiting the block structure information of the representation and using the proposed characterisation of the dictionary, named Block-Spark (Definition 2.5, page 52), improves the conventional Spark-based ERC presented in (1.10) by increasing the sparsity level, hence, weakening the corresponding conditions. Gribonval and Nielsen [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]; [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF], and Bruckstein et al. [START_REF] Bruckstein | From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images[END_REF], when it comes to practical computations both conditions would become unrealistic, because both the characterisation Block-Spark (and corresponding Spark) and the related optimisation problem P p,0 (and corresponding P 0 ) are non-convex.

Therefore, the next step would focus on at least the related optimisation problems which are tractable. In the next section we propose conditions for more general optimisation problems of P w;p 1 ,p 2 and P p 1 ,p 2 , where, 0≤p 2 ≤1≤p 1 . Block-NSP includes more general case of differently-sized blocks, i.e., d 1 = • • • =d K , arbitrary dictionary and integrating the block-structure information into the proposed characterisation, e.g., Q w;p 1 ,p 2 (S b (β), Φ), and problems, e.g., P w;p 1 ,p 2 and P p 1 ,p 2 , where, 0≤p 2 ≤1≤p 1 .

Block-ERC based on block null space property

Theorem 2.2 (Block-NSP). Let Φ be a general dictionary in y=Φβ 0 and S b (β)⊂{1, • • • , K} a set of block indices. For 0≤p 2 ≤1≤p 1 and Block-Support S b (β 0 ), define:

Q w;p 1 ,p 2 (S b (β) , Φ) def = max x∈Ker(Φ)\{0} k∈S b (β) w k p 2 d k j=1 |x j [k]| p 1 p 2 p 1 K k=1 w p 2 k d k j=1 |x j [k]| p 1 p 2 p 1
, with the convention of x 0 = lim

i→0 + x i = 1 x =0 0 x=0 and w k =d -1/p 1 k
, ∀k are the elements of the weight vector w. If Q w;p 1 ,p 2 (S b (β), Φ)<1/2, then for all β 0 whose Block-Support lies within S b (β), i.e., S b (β 0 )⊂S b (β), β 0 is the unique solution to the P w;p 1 ,p 2 .

Notice that for the case of equally-sized blocks, i.e.,

d 1 = • • • =d K =d, if Q p 1 ,p 2 (S b (β), Φ)<1/2 and S b (β 0 )⊂S b (β)
, then β 0 is the unique solution to the P p 1 ,p 2 , where,

Q p 1 ,p 2 (S b (β), Φ) is the unweighted variant of Q w;p 1 ,p 2 (S b (β), Φ): Q p 1 ,p 2 (S b (β) , Φ) def = max x∈Ker(Φ)\{0} k∈S b (β) d j=1 |x j [k]| p 1 p 2 p 1 K k=1 d j=1 |x j [k]| p 1 p 2 p 1
.

Chapter 2. Block-ERC in general dictionaries

The proof of Theorem 2.2 is provided in Section A.7 (page 137).

As expected, for d 1 = • • • =d K equal to 1, p 2 equal to p, and for all p 1 ≥1, Block-NSP (Theorem 2.2) is equivalent to the conventional NSP (page 27). Theorem 2.2 (Block-NSP) determines sufficient conditions on S b (β) by determining a 50% upper threshold on the concentration of the p 1 norm of blocks of Kernel x in Block-Support S b (β), such that guarantee the uniqueness of the solution to the P w;p 1 ,p 2 minimisation problem.

Corollary 2.1. β is the unique solution of P p,0 (or

P w;p,0 ) minimisation problem, if ∀p≥1 |S b (β)| < Block-Spark(Φ) 2 ,
where, |S b (β)| is the Block-Cardinality of β defined in Definition 2.3 (page 52).

Proof. In Theorem 2.2 (Block-NSP) for a special case of p 2 =0, ∀p 1 =p≥1 we have

∀S b (β) , Q w;p,0 (S b (β) , Φ) ≤ max x∈Ker(Φ)\{0} |S b (β)| x w;p,0 .
Maximum value of the right-hand side is obtained by minimising x w;p,0 , which the minimum value is defined according to Definition 2.5 as Block-Spark, so

∀S b (β) , Q w;p,0 (S b (β) , Φ) ≤ |S b (β)| Block-Spark(Φ)
.

But from Theorem 2.2 (Block-NSP), it is known that for unique solution of the problem P w;p,0 , it is required that Q w;p,0 (S b (β), Φ)<1/2. On the other hand, ∀p≥1, P w;p,0 is equal to P p,0 , which proves the corollary.

Following corollary is a double proof for Theorem 2.1 (Block-ERC based on Block-Spark, page 65) for p≥1 instead of p≥0.

Corollary 2.2. For any general dictionary, ∀p≥1 if

β 0 p,0 < Block-Spark(Φ) 2 ,
then β 0 is the unique solution to the P p,0 (or P w;p,0 ) optimisation problem.

Proof. From Theorem 2.2 (Block-NSP), we have S b (β 0 )⊂S b (β), so β 0 p,0 =|S b (β 0 )|≤|S b (β)| and using Corollary 2.1 the proof is done.

Remark 2.9. Although the proposed Block-NSP (Theorem 2.2) is a generalisation of all the previous null space properties, in general the characterisation Q w;p 1 ,p 2 (S b (β) , Φ) is not easy to calculate. Therefore, in the next section the tractable characterisation of coherence will be taken into consider.

Block-ERC based on block mutual coherence constant

y = K k=1 Φ 1 [k] β 1 [k] = K k=1 Φ 2 [k] β 2 [k] ,
then ∀(q, p)∈R 2 ≥1 , and ∀r∈R ≥0 :

β 1 r,0 + β 2 r,0 ≥ 2 d -1 2 min d 3 2 max M q, p p-1 (Φ 1 , Φ 2 )
, where, M q,p/(p-1) (Φ 1 , Φ 2 ) is defined based on the parameter changing of p→p/(p-1) in M q,p (Φ 1 , Φ 2 ), which is called basic Block-MCC q,p . Basic Block-MCC q,p is derived from the special case of Property 2.6 (Block-MCC q,p for intra-block orthonormality, page 58), when the characterisation is extracted from two separated matrices instead of one matrix, and is defined as follows:

∀(q, p) ∈ R 2 >0 , M q,p (Φ 1 , Φ 2 ) = max k,k d -1 p k d 1 q k d max Φ T 1 [k] Φ 2 k q→p .
The proof of Lemma 2.2 is provided in Section A.8 (page 140).

From the notation point of view, notice that the basic Block-MCC q,p of two orthonormal matrices is equal to the Block-MCC q,p of the dictionary built from the concatenation of those previous orthonormal matrices, i.e., M q,p/(p-1)

(Φ 1 , Φ 2 )=M q,p/(p-1) ([Φ 1 , Φ 2 ]).
As expected, for d 1 = • • • =d K equal to 1, Lemma 2.2 reduces to the basic uncertainty principle explained in (1.11), i.e., (

β 1 0 + β 2 0 )≥2/M (Φ 1 , Φ 2 ).
In comparison of different uncertainty principles, it should be taken into account that a typical uncertainty principle I is weaker than uncertainty principle II, when minimal sum of 0 pseudo-norm or r,0 pseudo-mixed-norm of two candidate solutions β 1 and β 2 in I is higher than II. To see the effect of block-sparsity on the uncertainty principle, using β 1 0 ≤d max β 1 r,0 and β 2 0 ≤d max β 2 r,0 , ∀r≥0, in (1.11), we get ( β 1 r,0 + β 2 r,0 )≥2/(d max M (Φ 1 , Φ 2 )). Then, comparing with Lemma 2.2, we need to demonstrate the relationship between d max M (Φ 1 , Φ 2 ) and d

-1/2 min d 3/2 max M q,p/(p-1) (Φ 1 , Φ 2 ).

Chapter 2. Block-ERC in general dictionaries

Property 2.15 (Basic Block-MCC q,p v.s. basic MCC). Supposing Φ 1 and Φ 2 are two orthonormal matrices, ∀(q, p)∈R 2 >0 we have:

0 ≤ d -1 2 min d 3 2 max M q, p p-1 (Φ 1 , Φ 2 ) d max M (Φ 1 , Φ 2 ) ≤ d -1 2 min d -1 2 max max k,k d 1 p -1 2 k d 1 q + 1 2 k max 1, d 1 2 -1 p k max 1, d 1 2 -1 q k .
The above upper-bound for different values of q and p are shown in table 2.7.

Proof. From second part of Property 2.8 (Block-MCC q,p bounds, page 59), which the dictionary has intra-block orthonormality, in a special case of two distinct orthonormal bases instead of one dictionary, ∀(q, p, q , p )∈R 4 >0 we have:

0 ≤ M q,p (Φ 1 , Φ 2 ) ≤ M (Φ 1 , Φ 2 ) d max max k,k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k × max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k .
Then, by parameter changing of p→p/(p-1), and then selecting q =p =2, we get:

0 ≤ M q, p p-1 (Φ 1 , Φ 2 ) ≤ M (Φ 1 , Φ 2 ) d max max k,k d 1 p -1 2 k d 1 q + 1 2 k max 1, d 1-1 p -1 p k max 1, d 1 q -1 q k × max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k = M (Φ 1 , Φ 2 ) d max max k,k d 1 p -1 2 k d 1 q + 1 2 k max 1, d 1 2 -1 p k max 1, d 1 2 -1 q k .
Next, by multiplying by d

-1/2 min d 1/2
max >0, the proof is done.

Remark 2.10. As it is mentioned before, in order to compare the proposed basic block-sparse uncertainty principle in Lemma 2.2 (Basic Block-UP, page 69), i.e., (

β 1 r,0 + β 2 r,0 )≥2/(d -1/2 min d 3/2 max M q,p/(p-1) (Φ 1 , Φ 2 ))
, with the conventional basic uncertainty principle explained in (1.11) on page 28, i.e., (

β 1 0 + β 2 0 )≥2/M (Φ 1 , Φ 2 ), the relationship between d -1/2 min d 3/2 max M q,p/(p-1) (Φ 1 , Φ 2 ) and d max M (Φ 1 , Φ 2
) should be investigated. The mentioned comparison is shown in table 2.7. Since the two terms are in the denominator of the right-hand side of the uncertainty principle, the smaller values lead to more weakened uncertainty principles, hence, improve the conditions. Therefore, the values in table 2.7, which are less than or equal to one, correspond to weakened uncertainty principles due to

0 < q & p ≤ 2 q & p ≥ 2 0 < q ≤ 2 & p ≥ 2 q ≥ 2 & 0 < p ≤ 2 d -1 2 min d 3 2 max M q, p p-1 (Φ 1 ,Φ 2 ) dmax M (Φ 1 ,Φ 2 ) ≤ d -1 2 min d 1 p + 1 q -1 2 max d -1 2 min d 1 2 max d -1 2 min d 1 q max d -1 2 min d 1 p max Table 2.7: Upper-bound of d -1/2 min d 3/2 max M q,p/(p-1) (Φ 1 , Φ 2 )/d max M (Φ 1 , Φ 2 )
for different values of q and p. using block-sparsity. From table 2.7, it can be seen that for ∀(q, p)∈R 2 ≥2 and d min =d max , the proposed basic block-sparse uncertainty principle in Lemma 2.2 (Basic Block-UP) improves the conventional basic uncertainty principle explained in (1.11).

To compare the proposed basic block-sparse uncertainty principle in Lemma 2.2 (Basic Block-UP, page 69, with d min =d max =d) with the block-sparse uncertainty principle proposed by Eldar et al. in (1.19) on page 39, i.e.,

β 1 2,0 + β 2 2,0 ≥2/(d M Eldar Inter (Φ 1 , Φ 2 )), where M Eldar Inter (Φ 1 , Φ 2 )= max k,k Φ T 1 [k]Φ 2 [k ] 2→2 /d, we need to investigate the relationship between M q,p/(p-1) (Φ 1 , Φ 2 ) and M Eldar Inter (Φ 1 , Φ 2 ).
Property 2.16 (Basic Block-MCC q,p v.s. basic block-coherence of Eldar et al.).

Supposing Φ 1 and Φ 2 are two orthonormal matrices with equally-sized blocks of length d, ∀(q, p)∈R 2 >0 we have:

M q, p p-1 (Φ 1 , Φ 2 ) M Eldar Inter (Φ 1 , Φ 2 ) ≥ d 1 q + 1 p -1 min 1, d 1 2 -1 p min 1, d 1 2 -1 q , M q, p p-1 (Φ 1 , Φ 2 ) M Eldar Inter (Φ 1 , Φ 2 ) ≤ d 1 q + 1 p -1 max 1, d 1 2 -1 p max 1, d 1 2 -1 q .
Then, for q=p=2 the lower-and upper-bound are both equal to one, then

M 2,2 (Φ 1 , Φ 2 )=M Eldar Inter (Φ 1 , Φ 2 ).
The above bounds for different values of q and p are shown in table 2.8.

Proof. It follows from Property 2.11 (Block-MCC q,p v.s. block-coherence of Eldar et al., page 62) in a special case of two orthonormal matrices ∀(q, p)∈R 2 >0 :

M q,p (Φ 1 , Φ 2 ) ≥ d 1 q -1 p min 1, d 1 p -1 2 min 1, d 1 2 -1 q M Eldar Inter (Φ 1 , Φ 2 ) , M q,p (Φ 1 , Φ 2 ) ≤ d 1 q -1 p max 1, d 1 p -1 2 max 1, d 1 2 -1 q M Eldar Inter (Φ 1 , Φ 2 ) .
Then, by parameter changing of p→p/(p-1), the proof is done.

Remark 2.11. To compare the proposed basic block-sparse uncertainty principle in Lemma 2.2 (Basic Block-UP, page 69, with d min =d max =d), i.e., ( β 1 r,0 + β 2 r,0 )≥2/(dM q,p/(p-1) (Φ 1 , Φ 2 )), with the block-sparse uncertainty principle proposed by Eldar et

al. in (1.19), i.e., β 1 2,0 + β 2 2,0 ≥2/(d M Eldar Inter (Φ 1 , Φ 2 )), the 0 < q & p ≤ 2 q & p ≥ 2 0 < q ≤ 2 & p ≥ 2 q ≥ 2 & 0 < p ≤ 2 M q, p p-1 (Φ 1 ,Φ 2 ) M Eldar Inter (Φ 1 ,Φ 2 ) ≥ 1 d 1 q + 1 p -1 d 1 p -1 2 d 1 q -1 2 M q, p p-1 (Φ 1 ,Φ 2 ) M Eldar Inter (Φ 1 ,Φ 2 ) ≤ d 1 q + 1 p -1 1 d 1 q -1 2 d 1 p -1 2 Table 2.8: Bounds of M q,p/(p-1) (Φ 1 , Φ 2 )/M Eldar Inter (Φ 1 , Φ 2 )
for different values of q and p.

relationship between M q,p/(p-1) (Φ 1 , Φ 2 ) and M Eldar Inter (Φ 1 , Φ 2 ) should be investigated. With the same reasoning as the one used in Remark 2.10 (page 70), the upper-bound values in table 2.8, which are less than or equal to one, correspond to weakened uncertainty principles. From table 2.8, it can be seen that for ∀(q, p)∈R 2 ≥2 , the proposed basic block-sparse uncertainty principle in Lemma 2.2 improves the basic block-sparse uncertainty principle proposed by Eldar et . Next, we show the corresponding lower-bound for the proposed basic Block-MCC q,p in a special case of q=p=2, i.e., M 2,2 (Φ 1 , Φ 2 ).

Property 2.17 (Basic Block-MCC 2,2 lower-bound). Supposing Φ 1 and Φ 2 are two m×m orthonormal matrices, we have:

M 2,2 (Φ 1 , Φ 2 ) ≥ d min d -3 2 max √ m . (2.2)
The proof of Property 2.17 is provided in Section A.9 (page 144).

Remark 2.12 (Block-incoherency). It can be seen that for special settings of max ≤1, the lower-bound in Property 2.17 is less than or equal to the lower-bound in conventional case, which means that in the proposed block-structured scenario, dictionaries can be more block-incoherent compared to the conventional case. Since there is a direct relationship between the sparsity level of recovery condition and block-incoherency, hence improved recovery conditions are obtained in block-structured scenario. In addition, notice that in the proof of all above basic two orthonormal bases we used the relationships of a dictionary with intra-block orthonormality, because for an orthonormal base Φ∈R m×m , with Φ T Φ=I m , we have

d 1 = • • • =d K =1 and equally-sized blocks, i.e., d 1 = • • • =d K =d,
Φ T [k]Φ[k]=I d k .
Returning back from basic two orthonormal bases Φ 1 and Φ 2 to the dictionaries Φ, the corresponding relations based on Block-MCC q,p (Definition 2.6, page 54) should be established. Thanks to Block-MCC q,p , we can overcome the issue of intractability of the Block-Spark (Definition 2.5, page 52). Now, we introduce the following Block-ERC based on Block-MCC q,p : Theorem 2.3 (Block-ERC based on Block-MCC q,p ). For any dictionary Φ with full column rank blocks, let y=Φβ 0 , then supposing Block-MCC q,p M q,p (Φ), Block-Support relation S b (β 0 )⊂S b (β), ∀(q, p)∈R 2 ≥1 , and ∀r∈R ≥0 , if

β 0 r,0 < 1 + (d max M q,p (Φ)) -1 min k min 1, d 1 q -1 p k 2 ,
then β 0 is the unique solution to the P w;q,1 problem.

The proof of Theorem 2.3 is provided in Section A.10 (page 145).

For

d 1 = • • • =d K =1, Theorem 2.
3 converges to its conventional element-wise counterpart explained in (1.13) on page 29, i.e., β 0 0 <(1+M -1 (Φ))/2.

As mentioned before (page 38), the right-hand side of the equation in Theorem 2.3 is called Block-SL and is represented as Block-SL(Φ). In order to make a comparison of Block-SL of Theorem 2.3, for different basic tractable (q, p) pairs of table 2.1 (page 54), let us represent it as Block-SL q,p (Φ) and call it (q, p)-Block-Sparsity Level (Block-SL q,p ). In the following property we investigate the possible relationship between different Block-SL q,p : Property 2.18 (Block-SL q,p inequalities). The different Block-SL q,p calculated for basic tractable q→p operator-norms of table 2.1 (page 54) have the following relationships:

Block-SL 1,∞ (Φ) ≤ Block-SL 1,2 (Φ) ≤ Block-SL 1,1 (Φ) , Block-SL 1,∞ (Φ) ≤ Block-SL 1,2 (Φ) ≤ Block-SL 2,2 (Φ) , Block-SL 1,∞ (Φ) ≤ Block-SL 2,∞ (Φ) ≤ Block-SL 2,2 (Φ) , Block-SL 1,∞ (Φ) ≤ Block-SL 2,∞ (Φ) ≤ Block-SL ∞,∞ (Φ) ,
which is shown in figure 2.5(a), while their general relationship for equally-sized case, i.e., The proof of Property 2.18 is provided in Section A.11 (page 148). Now we need to prove the claim of improved recovery conditions in the block-wise domain through comparing sparsity level with block-sparsity level. To this end, supposing that β 0 has a block-sparse structure and satisfies the condition of Theorem 2.3 (Block-ERC based on Block-MCC q,p ), and considering the fact that ∀r≥0, β 0 0 ≤d max β 0 r,0 , we have:

d 1 = • • • =d K =d, is
∀(q, p) ∈ R 2 ≥1 , β 0 0 < d max + M -1 q,p (Φ) min k min 1, d 1 q -1 p k 2 .
Comparing with β 0 0 <(1+M -1 (Φ))/2 in (1.13) on page 29, we see that the relationship between d max +M -1 q,p (Φ) min k min{1, d

1/q-1/p k } and 1+M -1 (Φ) should be investigated. We show this relationship in the following property, which the proof is provided in Section A.12 (page 149):

Property 2.19 (SL6 v.s.

Block-SL q,p ). For a dictionary Φ with intra-block orthonormality, and Block-MCC q,p M q,p (Φ), ∀(q, p, q , p )∈R 4 >0 if the MCC M (Φ) is small enough, i.e.,

M (Φ) ≤ 1 - dmax min k min 1, d 1 q -1 p k max k,k =k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k d max -1 ,
then the proposed sparsity level in Theorem 2.3 (block-ERC based on Block-MCC q,p , page 73) is higher than the conventional sparsity level, i.e.,

d max + M -1 q,p (Φ) min k min 1, d 1 q -1 p k ≥ 1 + M -1 (Φ) .
Remark 2.13. Therefore, exploiting the block structure information of the representation and using the proposed characterisation of the dictionary, named Block-MCC q,p (Definition 2.6, page 54), improves the conventional MCC-based ERC of Donoho et al. [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF];

[DET06], Gribonval and Nielsen [START_REF] Gribonval | Sparse decompositions in "incoherent" dictionaries[END_REF]; [GN07], Tropp [START_REF] Tropp | Greed is good: algorithmic results for sparse approximation[END_REF], and Bruckstein et al. [START_REF] Bruckstein | From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images[END_REF] by increasing the sparsity level, hence weakening the corresponding conditions. The mentioned improvement is under condition on conventional MCC, which ensures that the dictionary is sufficiently incoherent, i.e., MCC is small enough.

In order to have a clear sense of the upper-bound on the conventional MCC in Property 2.19 (SL v.s. Block-SL q,p ), this value is calculated in table 2.9 for the basic tractable (q, p) pairs of table 2.1 (page 54). Although, for the MCC values less than the upper-bounds in table 2.9, the supremacy of the proposed sparsity level is ensured, but these bounds are very pessimistic and in practice for even higher values of MCC the supremacy of the proposed sparsity level can be observed.

(q = q , p = p ) (1, 1) (1, 2) (1, ∞) (2, 2) (2, ∞) (∞, ∞) M (Φ) ≤ 1 d 1 2 max d 1 2 max +1 1 d 1 2 max d 1 2 max +1 1 dmax 0 1 d 1 2 max d 1 2 max +1 1 d 1 2 max d 1 2 max +1
Table 2.9: Upper-bound of MCC ensuring the supremacy of the proposed sparsity level, for different basic values of (q, p) pairs and for a dictionary with intra-block orthonormality.

Remark 2.14. Table 2.9 demonstrates that our proposed sparsity level for the 2→2 operator-norm, which is equal to the conventional 2 matrix norm, requires the most strict upper-bound on M (Φ) in order to be higher than the conventional sparsity level, while the dictionaries with characterisations that use operator-norms other than 2→2 can be less incoherent (more coherent, i.e., higher M (Φ) is allowed), and especially for 1→∞ operator-norm the most relaxed upper-bound on M (Φ) is achieved. This is one of the advantages of utilising operator-norms, which enable us to have more relaxed conditions by exploiting norms other than the conventional 2 . Now Block-ERC of Eldar et al., i.e., β 0 2,0 <(1+(dM Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)))/2, explained in (1.20) on page 40, can be compared to our Block-ERC proposed in Theorem 2.3 (Block-ERC based on Block-MCC q,p , page 73). Supposing that all blocks sharing the same block length d, it is clear that the relationship between (M Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)) and M -1 q,p (Φ) min{1, d 1/q-1/p } should be investigated. Lemma 2.3 (Eldar et al.'s v.s. proposed Block-SL q,p ). For a dictionary with intra-block orthonormality, Block-ERC proposed in Theorem 2.3 (Block-ERC based on Block-MCC q,p , page 73) for q=p=2 as the best case is equal to Block-ERC of Eldar et

(Φ)) -1 (1-(d-1)M Eldar Intra (Φ)))/2=(1+(dM Eldar Inter (Φ)) -1 )/2.
On the other hand, based on lemma 2.3, the proposed Block-ERC for q=p=2 is equivalent to the condition of Eldar and her co-workers. Therefore, our proposed Block-ERC in Theorem 2.3 (page 73) in the following special setting, reduces to Block-ERC of Eldar et al.:

• q=p=2,

• equally-sized blocks, i.e., d 1 = • • • =d K =d, and • intra-block orthonormality of dictionary, i.e., for 1≤k≤K,

Φ T [k]Φ[k]=I d .
In other words, in the mentioned special setting, theoretically and independent of the recovery algorithm, the same Block-ERC can be achieved.

Block-ERC based on cumulative coherence constant

All the coherence characterisations such as conventional MCC (page 22) [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF], block-coherence of Eldar et al. (page 36) [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], mutual subspace coherence of Ganesh et al. (page 37) [START_REF] Ganesh | Separation of a subspace-sparse signal: Algorithms and conditions[END_REF], or our Block-MCC q,p (page 54) regardless of their definition in the block-wise or element-wise domain represent only the most extreme correlation between the atoms, blocks or subspaces of the dictionary and do not offer a comprehensive description of the dictionary. Based on the proposed Eldar's cumulative coherence (Definition 2.8, page 63) for equally-sized blocks of length d, we introduce the following condition named Block-ERC based on cumulative inter-block coherence constant: Theorem 2.4 (Block-ERC based on Eldar's cumulative coherence). For any general dictionary Φ with equally-sized blocks of length d, and for

(d-1)M Eldar Intra (Φ)+d M Eldar Inter (Φ, k-1)<1, if M Eldar Inter (Φ, k) + M Eldar Inter (Φ, k -1) < 1 -(d -1) M Eldar Intra (Φ) d ,
where, M Eldar Inter (Φ, k)

def =d -1 max |Λ|=k max j / ∈Λ i∈Λ Φ T [i]Φ[j] 2→2 (Eldar's cumulative coherence, Definition 2.8, page 63), and M Eldar Intra (Φ) def = max i,j =i k |ϕ T i [k]ϕ j [k]| (page 36
), then block k-sparse representation vector β 0 can be recovered correctly from block orthogonal matching pursuit and 2 / 1 -optimisation program algorithms.

The proof of Theorem 2.4 is provided in Section A.14 (page 154).

For d equal to 1, the Block-ERC based on cumulative inter-block coherence constant which is proposed in Theorem 2.4 is equivalent to its element-wise counterpart defined in (1.14) on page 31, i.e., the ERC based on cumulative MCC, i.e., M (Φ, k)+M (Φ, k-1)<1. The last proposed Block-ERC is the only algorithm-dependent condition, which the algorithms are explained in [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF].

Numerical experiments

In this section, the block-sparsity levels resulted from the proposed Block-ERC based on Block-MCC q,p (Theorem 2.3, page 73), i.e., Block-SL q,p (Φ)=(1+(d max M q,p (Φ)) -1 min k min{1, d 1/q-1/p k })/2, are going to be calculated. Block-MCC q,p is selected, because it is computationally tractable. In all of the numerical simulations performed below, we have:

1. Dictionaries are generated from independent and identically distributed (i.i.d.) random variables, with 40 rows and 400 columns.

2. All columns of the dictionaries are normalized to have unit 2 norm.

3. The results are shown in terms of average and standard deviation over 100 repetitions of random dictionary generation.

4. All blocks share the same length d, i.e.,

d 1 = • • • =d K =d.
5. Only the basic tractable operator-norms of table 2.1 (page 54) are considered for the calculation of the Block-MCC q,p .

In Definition 2.6 (page 54), we proposed the characterisation Block-MCC q,p in a general format, which has three main free parameters of d k , Φ, and q→p operator-norm, i.e.,

M q,p (Φ)= max k,k =k (d -1/p k d 1/q k )/d max Φ † [k]Φ[k ] q→p .
Therefore, following parameters can be considered in the computation of sparsity level:

• Block length d k : Block-MCC q,p is a function of the length of each block, i.e. d k , ∀k.

• Type of dictionary Φ: Although Block-MCC q,p is defined for a dictionary in general, but in Property 2.6 (Block-MCC q,p for intra-block orthonormality, page 58), we investigated a special case, where, there is intra-block orthonormality, i.e., Φ T [k]Φ[k]=I d , ∀k.

• q→p operator-norm in Block-MCC q,p : Based on the table 2.1 (page 54), only six tractable operator-norms of 1→1 , 1→2 , 1→∞ , 2→2 , 2→∞ , and ∞→∞ are considered in the numerical experiments.

In order to investigate the behaviour of sparsity level in Block-ERC based on Block-MCC q,p as a function of the three above-mentioned parameters, in each of the following three experiments, one parameter out of three is kept fixed. (d-1)M Eldar Intra (Φ))/2, are provided. Notice, in order to compare to the conventional sparsity level of Donoho et al., the block-sparsity level resulted in the Block-ERC, whether ours or the results of Eldar et al., are multiplied by d to be transformed to the sparsity level, because the dictionary consists of equally-sized blocks. At last, the higher the sparsity level, the more improved the recovery condition.

Effect of dictionary and operator-norm type

In this experiment, assume that the block length d is fixed. The goal is to investigate the sparsity level for two types of random dictionaries without and with intra-block orthonormality, and for the six pairs of q and p with tractable operator-norms.

Finally, the sparsity levels for two types of random dictionaries each with six different pairs of q and p are compared to the conventional sparsity levels of Eldar et

Effect of block length and operator-norm type

In this experiment, assume that the dictionary is in general with full column rank blocks. The goal is to investigate the sparsity levels for the six pairs of q and p with tractable operator-norms and for different dictionaries with equally-sized blocks d={1, 2, 4, 5, 8, 10}. Finally, the sparsity levels for six Block-MCC q,p computed for different values of d, are compared to the conventional sparsity levels of Donoho et al. and Eldar and her co-workers. As it can be seen in figure 2.7, for a general dictionary, for d>1 all the proposed sparsity levels based on Block-MCC q,p are higher than the conventional sparsity levels introduced by Donoho et al. and Eldar and her co-workers. By increasing the block length d, the difference between the proposed and conventional sparsity levels becomes more pronounced.

Assuming that the sparsity level corresponding to Block-MCC q,p is represented by SL q,p (Φ), the following SL q,p inequalities was obtained from simulation experiments for all values of d:

SL 1,∞ (Φ)<SL 1,2 (Φ)<SL 2,∞ (Φ)<SL 1,1 (Φ)<SL ∞,∞ (Φ)<SL 2,2 (Φ), whereas SL 1,2 (Φ), SL 2,∞ (Φ), SL 1,1 (Φ)
, and SL ∞,∞ (Φ) are closer to each other compared to the two other sparsity levels. Part of the sparsity level inequalities obtained from simulation experiments is proved theoretically in Property 2.18 (Block-SL q,p inequalities, page 73).

Effect of block length and dictionary type

In this experiment, assume that the dictionary characterisation of Block-MCC 2,2 is used. The goal is to investigate the sparsity levels for two types of random dictionaries without and with intra-block orthonormality, and for different dictionaries with equally-sized blocks d={1, 2, 4, 5, 8, 10}. Finally, the sparsity levels for two types of random dictionaries computed for different values of d, are compared to the conventional sparsity levels of Donoho et al. and Eldar and her co-workers.

As it can be seen in figure 2.8, for Block-MCC 2,2 as dictionary characterisation, the proposed sparsity levels for random dictionaries without intra-block orthonormality and with full column rank blocks (Φ, hollow downward orange triangles) are higher than the conventional sparsity levels introduced by Donoho

Conclusion

In this chapter, the sufficient conditions for unique representation of block-sparse recovery of an arbitrary signal y in a general arbitrary dictionary Φ using general weighted (pseudo-)mixed-norm P w;p 1 ,p 2 (and also P p 1 ,p 2 as explained in Remark 2.1, P w;p 1 ,p 2 v.s. P p 1 ,p 2 , page 50) optimisation problem, are proposed.

The dictionary is general (with full column rank blocks) and not restricted to be a union of two or more orthonormal bases, although we have some results in the special setting of the dictionary with intra-bock orthonormality. For theoretical conditions, weighted optimisation problems of P w;p 1 ,p 2 (and also P p 1 ,p 2 for 0≤p 2 ≤1≤p 1 ) and P w;p,0 (and also P p,0 for p>0) are defined in their general form.

In addition, the proposed characterisations of Block-Spark, Block-NSP, Block-MCC q,p , cumulative Block-MCC q,p , and the properties are introduced in the general case.

Then, we demonstrated the relationship between the proposed Block-MCC q,p , and conventional MCC and block-coherence proposed by Eldar et al. [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF]; [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF].

The proposed various properties and block-sparse uncertainty principles are defined in the general case and introduced to deduce recovery conditions for block-sparse representations.

We defined basic block-sparse uncertainty principle based on the proposed basic Block-MCC q,p , upper-bounded it in terms of its conventional basic MCC, then lower-bounded in terms of m and length of blocks, then showed its relationship with basic block-coherence of Eldar et al., and in the end we showed in which conditions the dictionary can be more block-incoherent relative to the conventional and existing block-wise cases.

We demonstrated that the proposed Block-ERC based on Block-Spark improves the conventional ERC based on Spark [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF]; [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]. In addition, the proposed Block-NSP generalises the conventional NSP [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF]; [START_REF] Elad | On sparse signal representations[END_REF]; [START_REF] Gribonval | Sparse representations in unions of bases[END_REF]; [START_REF] Feuer | On sparse representation in pairs of bases[END_REF]; [START_REF] Zhang | A Simple Proof for Recoverability of 1-Minimization: Go Over or Under? Tech. rep[END_REF]; [START_REF] Stojnic | Compressed sensing -probabilistic analysis of a null-space characterization[END_REF]; [START_REF] Cohen | Compressed sensing and best k -term approximation[END_REF] and another existing Block-wise NSP [START_REF] Stojnic | On the Reconstruction of Block-Sparse Signals With an Optimal Number of Measurements[END_REF], and Fusion NSP in special setting [START_REF] Boufounos | Sparse Recovery From Combined Fusion Frame Measurements[END_REF].

Further, we showed that the proposed Block-ERC based on Block-MCC q,p improves the conventional ERC based on MCC [START_REF] Donoho | Uncertainty principles and ideal atomic decomposition[END_REF].

Moreover, we improved the Block-ERC of Eldar et al. and demonstrated that our results algorithm-independently weaken the condition. In addition in special setting of q=p=2, intra-block orthonormality and equally-sized blocks our proposed Block-ERC and the results of Eldar et al. are equivalent [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF].

Next, we proposed Block-MCC q,p -based recovery conditions for convex problem P w;p,1 (and P p,1 for equally-sized blocks), where, p≥1.

Finally, we introduced the cumulative version of the block-coherence of Eldar et al. [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF], named it cumulative inter-block coherence constant, demonstrated its relationship with cumulative Block-MCC q,p , and the inter-block coherence defined by Eldar et al., and proposed Block-ERC based on cumulative inter-block coherence constant.

All the contributions of this work are the natural generalisation of the existing conventional concepts, thus, they all reduce to the conventional ones for the unit block size, i.e., ∀k, d k =1. Therefore, all the results are consistent with the previous findings.

In numerical experiments in Section 2.4, we showed that in computation of sparsity level, there are three types of parameters: block length d k , existence of intra-block orthonormality, and q→p operator-norm in Block-MCC q,p .

Then, by assuming that only two types of parameters are variable, we investigated the behaviour of sparsity level in three different experiments.

In this part, we are going to investigate the sparsity levels while all the three mentioned parameters are variable. In figure 2.9, the first twelve bars for each block size d, indicate (d+M -1 q,p (Φ))/2.

The proposed sparsity levels are calculated for the six tractable basic operator-norms of 1→1 , 1→2 , 1→∞ , 2→2 , 2→∞ , and ∞→∞ , based on the table 2.1 (page 54). Then, repeated for dictionaries without (Φ) and with (Φ ort ) intra-block orthonormality.

The last four bars are the sparsity levels of the Block-ERC proposed by Eldar et al., i.e., d times the right-hand side of the (1.20), and of the ERC proposed by Donoho et al., i.e., (1+M -1 (Φ))/2 defined in (1.13), for two types of dictionary, i.e., Φ and Φ ort .

As it can be seen in figure 2.9, all sparsity levels for d=1 converge to the conventional sparsity level of Donoho et al., whereas by increasing d, the sparsity level related to the Block-ERC, whether ours or the condition of Eldar et al., are significantly increased, which practically demonstrates the claim of weakened recovery conditions in the block-wise domain, which has been proved theoretically in Property 2.19 (SL v.s. Block-SL q,p , page 74).

In addition, it can be seen in figure 2.9 that for q=p=2, our proposed sparsity level is equal to that of Eldar et al., which demonstrates the correctness of the claim in Lemma 2.3 and consequently Remark 2.15. As discussed in Remark 2.15 and can be seen in figure 2.9, for q=p=2 and intra-block orthonormality, the proposed sparsity level reduces to the sparsity level of Eldar and her co-workers.

Furthermore, for all variable parameters, we have

SL 1,∞ (Φ)<SL 1,2 (Φ)<SL 1,1 (Φ), SL 1,∞ (Φ)<SL 1,2 (Φ)<SL 2,2 (Φ), SL 1,∞ (Φ)<SL 2,∞ (Φ)<SL 2,2 (Φ), and SL 1,∞ (Φ) < SL 2,∞ (Φ) < SL ∞,∞ (Φ)
, which is in agreement with Property 2.18 (Block-SL q,p inequalities).

In addition, it can be seen that for a given q and p pairs, the sparsity level of dictionaries Φ ort with intra-block orthonormality is higher than the sparsity level of dictionaries Φ without intra-block orthonormality, i.e., conditions based on Φ ort are weaker. Figure 2.9: Sparsity level as a function of d for different values of q and p, averaged over 100 realisations of random dictionaries, with m=40 and n=400.

Future research should focus on:

• Introducing block-sparse recovery conditions based on the proposed cumulative Block-MCC q,p defined in Definition 2.7.

• Generalising the conventional dictionary characterisation of µ defined in [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF], to establish block-sparse recovery conditions.

• Transforming all the previously mentioned block-sparse exact recovery conditions to block-sparse stable recovery conditions. In stable or robust recovery conditions we have, y-Φ β 2 <e, where, e is the noise level.

• Study on block-sparse optimisation algorithms, and the relationship between the theoretical and algorithmic block-sparse recovery conditions.

Chapter 3

A general framework for block structure identification

Introduction

The structure of block-sparsity, which is defined in Section 2.2.1 had been assumed to be known. In other words, the block structure of the representation vector β and dictionary Φ is assumed to be stored in

d=[d 1 , • • • , d K ].
However, in some problems the block structure prior knowledge might be not available. Therefore, the problem of block structure identification needs to be taken into account [START_REF] Eksioglu | A clustering based framework for dictionary block structure identification[END_REF].

In this chapter, the proposed block structure identification method is explained in Section 3.2. Next, the advantages of the proposed method including the improved recovery conditions and brain source space segmentation are investigated in sections 3.3 and 3.4, respectively. Then, some experiments on synthetic dictionary and real lead-field in Section 3.5 demonstrate the advantages of the proposed block structure identification method.

86

Chapter 3. A general framework for block structure identification

Block structure identification

As mentioned earlier, many real-world inverse problems are vastly underdetermined. In a fat coefficient matrix of a vastly USLE 1 , the columns are more likely to be coherent. Therefore, computing any kind of dictionary characterisation based on inter-columns coherence leads to high values. As discussed in Chapter 2, because of the inverse relationship between the coherence characterisation of a general dictionary and the corresponding sparsity level in Block-ERC 2 , the resulted Block-ERC would be tighter, which is not desired.

One possible solution to the issue of high coherence characterisation of dictionary or coefficient matrix in vastly USLE, is grouping the coherent columns of the dictionary to form some groups of columns, where, the columns inside each group or block are highly coherent. Then, computing the inter-blocks coherence characterisation leads to lower coherence characterisation of dictionary in comparison to the standard inter-columns coherence characterisation, because the resulted block of columns are more incoherent compared to the columns of dictionary. The proposed Block-MCC q,p 3 is a suitable coherence characterisation of dictionary, due to its generality and flexibility in computing the coherence between differently-sized blocks of columns.

By clustering the coherent columns or blocks of columns of a given dictionary utilising general characterisation of Block-MCC q,p as the similarity measure, the block structure of the dictionary can be identified. Because each cluster of columns represents a block of the dictionary. In other words, clusters resulted from the clustering algorithm are equivalent to the blocks of the dictionary. After applying clustering algorithm on the dictionary, each cluster, e.g., cluster k, contains some columns, where, the number of columns determines the size of corresponding block in the dictionary, i.e., d k , therefore, the block structure d=

[d 1 , • • • , d K ] is identified.
Each column in the dictionary Φ∈R m×n is multiplied to a corresponding element in the representation vector β∈R n , i.e., y= n i=1 β i ϕ i . So, each column ϕ i in dictionary is linked to a corresponding element in β, i.e., β i . Therefore, by determining the block structure of dictionary, the block structure of the representation vector is also identified.

By utilising the proposed clustering-based framework, we attain two main goals, which are represented graphically in figure 3.1 for a real-world problem of EEG 4 /MEG 5 source reconstruction USLE:

• improved Block-ERC, and • support segmentation.

1 Underdetermined System(s) of Linear Equations 2 Block-sparse Exact Recovery Condition(s) 3 (q, p)-Block Mutual Coherence Constant 4 ElectroEncephaloGraphy 5 MagnetoEncephaloGraphy Assuming the EEG/MEG source reconstruction problem in figure 3.1, there are four brain sources of s 1 , • • • , s 4 , as a simple example. The clustering tree is resulted from clustering the lead-fields of four sources. As shown graphically, the inter-cluster distance in clustering tree at the clustering level equivalent to three clusters is maximum. Then, the estimated number of clusters in the lead-field is three. As represented graphically in figure 3.1 (right part), the SL[%] 6 for three clusters is higher than the SL[%] for four clusters, where, the clustering is not applied. So there is improvement in Block-ERC based on Block-MCC q,p . The computation of different sparsity levels will be explained in Section 3.3.

Clustering coherent blocks of a general dictionary

Whether the block structure of a given dictionary is assumed to be known beforehand but it is not the optimal structure or it is unknown, by clustering coherent blocks in a bigger block, there would be the possibility of enhancing the structure of the dictionary as a preprocessing step.

In this chapter, we made use of standard agglomerative hierarchical clustering method, where, the proposed Block-MCC q,p is used as the measure of similarity or coherence. Hierarchical clustering provides a set of clustering structures in multilevel hierarchy manner, which allows to decide the level of clustering that is most suitable to the application of interest. The set of clustering structures is called clustering tree or dendrogram. The dendrogram consists of some nodes indicating the clusters, and usually the clustering level corresponding to the maximum inter-node distance is selected as the desired clustering level.

In figure 3.2, it has been assumed that before blocks clustering all of the blocks of the dictionary Φ∈R 8×12 share the same length of two, i.e., d 1 = • • • =d 6 =2. Then, by applying hierarchical clustering on initial blocks of dictionary and choosing the clustering level corresponding to the maximum inter-node distance in the clustering tree, i.e., three clusters, the clustered representation appears.

As it can be seen in figure 3

Block-ERC in clustered representation

To investigate Block-ERC in clustered representation, we need to calculate Block-SL q,p7 , which has inverse relationship with Block-MCC q,p . Since blocks growing due to clustering can lead to a case where a big block covers whole space, we apply PCA 8 to clustered blocks before computing Block-MCC q,p .

In order to compare the resulted Block-SL q,p to conventional sparsity level, we need to state the number of blocks of columns in terms of the number of columns. In a dictionary with equally-sized blocks, i.e., d 1 = • • • =d K =d, the Block-SL q,p can be simply converted to sparsity level through multiplying Block-SL q,p by the length of each of the blocks, i.e., d×Block-SL q,p (Φ) or Block-SLq,p(Φ) k=1

d k .
Next, we explain how to compute sparsity levels in a dictionary with differently-sized blocks.

Sparsity levels in differently-sized block structure of a dictionary

In a dictionary with differently-sized blocks, i.e., In some cases like figure 3.2, a block in clustered representation is in fact a concatenated version of a set of other blocks with the same size d, and not columns necessarily, i.e., d=1. This is the case in our real-world EEG and/or MEG source reconstruction problem, where, the Table 3.1: Sparsity levels for the dictionary in figure 3.2, if Block-SL q,p (Φ)=1.

d 1 = • • • =d K , it
initial blocks are lead-fields with the same size of three. In this type of problem, the number of blocks of size d or the blocks themselves have meaningful information. For instance, in our problem, each block of size d=3 represents a brain source.

Therefore, we can simply divide the number of elements to the length d to provide another means of representing the results, which is the number of initial blocks. Then, considering maximum sparsity level in the most pessimistic case, i.e., SL max , we can also represent the results in terms of maximum block-sparsity level, i.e., Block-SL max , which is SL max (Φ)/d. Therefore, we can consider the following basic types of sparsity level and Block-SL q,p for the maximum range:

• Block-SL q,p : Number of differently-sized blocks resulted from

(1+(d max M q,p (Φ)) -1 min k min{1, d 1/q-1/p k })/2, (Theorem 2.3, page 73).
-SL min : Number of columns resulted from As an example, consider the dictionary in figure 3.2 with Block-SL q,p equal to one. Then, the mentioned different sparsity levels are shown in table 3.1.

EEG/MEG source reconstruction problem and USLE

It was explained in Section 1.2.5 that the solution of the EEG/MEG forward problem is called lead-field, denoted by Φ EEG/MEG ∈R m×n , which linearly relates the activities in source, denoted by β, and EEG/MEG sensor space, denoted by y EEG/MEG . In other words, we have y EEG/MEG =Φ EEG/MEG β.

As mentioned earlier in Section 1.2.3, the inverse problem involves computing the parameters of the brain sources. Since the number of the sensors, i.e., m, is much less than the number of the brain sources, i.e., K, there would be infinitely many states of the sources that lead to the same electromagnetic fields, i.e., y EEG/MEG =Φ EEG/MEG β i , where i∈{1, • • • , ∞}. Hence, the EEG/MEG source reconstruction problem is ill-posed.

To make the problem solvable with a unique solution, a dipole source model can be applied to the brain current sources. In other words, a population of cerebral neurons can be represented by a single dipole. A dipole can be represented mathematically as a vector. Considering the 3D Cartesian coordination, for each dipole we have six parameters to be determined.

• Three parameters are related to the dipole position in i, j, and k direction, which by meshing the brain, these positions can be also limited to some predefined discrete values placed in the vertices of brain mesh.

• Three parameters characterise the power of the dipole in each of three directions, i.e., the magnitudes of the dipole moment in i, j, and k direction.

For instance in figure 3.4, the dipole located in s 4 is related to three consecutive entries in the source activity vector β∈R n , where K=n/3.

Assuming only four brain source positions s 1 , • • • , s 4 and three electric or magnetic sensors in a simple case, figure 3.4 represents EEG and MEG linear models.

As it can be seen in figure 3.4, regardless of the head electromagnetic properties, and in terms of the parameters of the dipole source model, each element of EEG lead-field is a function f of direction vectors i, j, or k and the distance between the position of corresponding source and sensor, e.g., the EEG lead-field for source s 1 is represented in figure 3.4. Each consecutive three elements in a row belongs to a certain sensor and source. Similarly, in terms of the parameters of the dipole source model, each element of MEG lead-field is a function g of direction vectors i, j, or k and the distance between the position of corresponding source and sensor, e.g., the MEG lead-field for source s 2 is represented in figure 3.4. Therefore, due to discretising and linearising effect of lead-field and having much less sensors than sources, an EEG/MEG source reconstruction problem can be stated in a form of USLE.

EEG/MEG source space segmentation

As mentioned earlier, the EEG/MEG source reconstruction problem can be modelled as an USLE. In our real-world problem, the dictionary in USLE is called the lead-field matrix. the whole lead-field matrix, is built by horizontal concatenation of lead-fields of all sources in the source space. As described before, the lead-field matrix of a single source is a m by 3 matrix, where, m is the number of sensors in the sensor space. Hence, a whole lead-field matrix is composed of consecutive three-column individual lead-field matrices.

On the other hands, as mentioned before, in the EEG/MEG linear model each of three-column individual lead-field matrices Φ[k], ∀k, are multiplied to the activity of corresponding dipole in the source vector β to construct the activity of the sensor space, i.e., y=

K k=1 Φ[k]β[k].
Then, each block of dictionary, Φ[k], ∀k, is the direct coefficient of its corresponding source activity. Therefore, by clustering the coherent lead-field matrices Φ[k] of the whole lead-field matrix Φ, their corresponding blocks in the source activity vector β will also be clustered. Hence, the sources will be grouped and will form some segments in the source space.

To realise the mentioned idea, we propose the general Block-MCC q,p dictionary characterisation, which measures the similarity between the blocks of the dictionary. Then, Block-MCC q,p can be used in the clustering step to make similarity matrix.

Results

Assuming that the number of clusters is not known, a strategy will be needed to appropriately estimate it, which leads us to hierarchical clustering analysis. Hierarchical clustering can be visualized by the so-called clustering tree or dendrogram. The clustering tree is composed of nodes (clusters) and branches, which the length of each branch shows the distance between the two nodes. A scale with the largest distance between consecutive nodes or longest branch can be considered as an appropriate scale for clustering, which gives the estimated number of clusters. The agglomerative hierarchical clustering algorithm is used. In agglomerative as opposed to divisive method, each single data is considered as a cluster, then at each clustering level the clusters are successively merged until all the data are clustered into a single cluster. Suppose that in figure 3.6(a), each block Φ[k] of dictionary is represented by a single circle, and there are 100 blocks, i.e., k∈{1, • • • , 100}. The relative distances between circles are determined by the proposed coherence measure Block-MCC 2,2 . The random dictionary is generated in a way to have five equally-sized clusters of blocks of length 20. Then,

Φ[1], • • • , Φ[20],
construct cluster I and so on. The process of generating desired random dictionaries is explained in Section 3.5.1. A method called complete has been used to compute the distance between two clusters, which is equal to the longest distance between two points in the two clusters as illustrated in figure 3.6(a). In figure 3.6(b) the dendrogram resulted from clustering the random dictionary is shown. As expected, the magenta dotted horizontal line in the scale with the largest distance between consecutive nodes intersects the dendrogram five times, so the estimated number of clusters is five, which is true.

Results

95

Synthetic dictionary

Dictionaries Φ∈R m×n , with equally-sized blocks of length d 1 = • • • =d K =d, are generated from independent and identically distributed (i.i.d.) random variables and all columns of the dictionaries are normalized to have unit 2 norm. The idea to generate the whole dictionary is first to start from an initial block, next some representative blocks of clusters are generated, then from each representative block of a cluster, the blocks belonging to the same cluster are produced. To construct a second block from a first one, the random matrix

exp((C-C T ) √ 2ε/ C-C T F
) is multiplied to the first block, where, C is a square random matrix of dimension m. Depending on the role of ε whether to generate representative blocks of clusters or blocks of dictionary, it is named ε inter or ε intra , respectively.

In figure 3.7, the process of generating a random dictionary Φ with N clusters and K blocks is illustrated. First, from a random initial block µ 0 ∈R m×d , N representatives of clusters µ 1 , • • • , µ N are created, using the mentioned multiplier and parameter ε inter . Then, from each representative µ i , ∀i, the blocks Φ i [j], ∀j, belonging to the same cluster i are generated, using the mentioned multiplier and parameter ε intra . The parameter ε in the multiplier controls the amount of overlap and similarity between blocks. In fact, ε inter defines the overlap between the representative blocks of clusters, whereas ε intra controls the overlap between the blocks belonging to the same cluster. Finally, the results are shown in terms of average and standard deviation over 100 repetitions of random dictionary Φ∈R 10×80 generation with d j =2, ∀j. Block-ERC in clustered representation and the number of clusters In this experiment, the agglomerative hierarchical clustering algorithm is applied on the blocks of a random dictionary with certain number of clusters, whereas the Block-MCC 2,2 and complete method are used to measure the inter-blocks coherence and inter-clusters distance, respectively. The mentioned certain number of clusters is once set to four and once set to eight.

As it can be seen in figure 3.8, by applying hierarchical clustering on the blocks of the dictionary, i.e., by decreasing the number of clusters, there exists at least one clustering level in which the relative Block-SL 2,2 in the most pessimistic case, i.e., Block-SL 2,2 (Φ)[%], increases in comparison to when the clustering is not applied on the dictionary, i.e., the rightmost part of each diagram corresponding to 40 clusters. Therefore, clustering coherent blocks of the dictionary improves the Block-ERC through increasing the Block-SL 2,2 .

In addition, for ε inter >1 and ε intra <0.1, the Block-SL 2,2 (Φ)[%] has a peak in a clustering level equal to the number of the clusters in the simulated dictionary. In fact, in figure 3.8 when there is four clusters in the simulated dictionary (blue curve, square markers), the maximum is at the fourth clustering level and also the consistent results are obtained for eight clusters in the dictionary (red curve, circle markers). For the clustering level lower than the optimal level, the space is under-sampled and there would be a block spanning the whole space, whereas for the clustering level higher than the optimal level, the space is over-sampled and the over-partitioning leads to high coherence measure. Figure 3.8: Block-SL 2,2 (Φ)[%] for each level of clustering computed for complete method, Block-MCC 2,2 , d=2, N ={4, 8}, and different values of ε inter and ε intra for simulating dictionary Φ∈R 10×80 .

Results

97

Block-ERC in clustered representation and the conventional ERC In this experiment, the goal is to compare the sparsity level obtained in the proposed Block-ERC to the conventional sparsity level which is equal to the half of the number of rows of Φ.

First, we need to express the block-sparsity level in terms of the conventional sparsity level as described in Section 3.3, e.g., SL min and SL max . Then, consider one of the cases in figure 3.8, e.g., ε inter =3.5, ε intra =0.1. So, the complete method is used to calculate inter-clusters distance, whereas the Block-MCC 2,2 measures the inter-blocks coherence of a dictionary Φ∈R 10×80 with equally-sized blocks, i.e.,

d 1 = • • • =d 40 =d=2.
In figure 3.9, it can be seen that by applying hierarchical clustering on the blocks of the dictionary, the increase in the sparsity levels even surpasses the conventional sparsity level which is marked with a black dotted line. It can be seen in figure 3.9 that close to the clustering level equal to the number of clusters in the simulated dictionary, even the proposed SL min can surpass the conventional sparsity level. Therefore, clustering the coherent blocks of a dictionary in addition to enhancing the Block-ERC through increasing the Block-SL 2,2 , improves the conventional sparsity level. Hence, improves the conventional ERC.

At last, in order to make sure that the clustering structure in a clustering level, where, there is a peak in sparsity level is done properly, the average clustering accuracy over 100 repetitions is computed for fourth (for the case of four clusters in the dictionary) and eighth (for the case of eight clusters in the dictionary) clustering levels, which are equal to 99.65% and 100%, respectively. Therefore, the peak in the sparsity level diagram, which is more striking in figure 3.8 in addition to giving an estimation of the number of clusters in the dictionary, also with high probability (at least in this example) corresponds to the correct clustering structure of the dictionary. Figure 3.9: SL min [%] and SL max [%] for each level of clustering computed for complete method, Block-MCC 2,2 , d=2, ε inter =3.5 and ε intra =0.1 for (a) N =4 and (b) N =8 in the dictionary Φ∈R 10×80 .

Real EEG/MEG lead-field

In this experiment the hierarchical clustering is used because of its characteristic in estimation of the number of clusters, as described in section 3.5. In order to simulate or import real-world sensor, source and volume conduction head model required to generate the real lead-field we used an open source software package MATLAB-based toolbox named FieldTrip [START_REF] Oostenveld | FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data[END_REF].

EEG lead-field is obtained using 32 standard electrodes and utilising three different volume conduction head models of three-layer concentric spheres, realistic three-layer with inflated cortical sheet obtained by boundary element methods and realistic three-layer with highly-folded cortical sheet obtained by SPM8 software [Spm]. Realistic head models are obtained by imposing anatomical constraint, which is provided by MRI 9 .

Similarly, for MEG lead-field generating, 32 standard sensors closest to the previously selected 32 EEG standard electrodes are selected and single sphere, realistic single layer with inflated and highly-folded cortical sheets are used as volume conduction head models. The sensor model, three-compartment (brain, skull, and scalp) volume conduction head model (analytical and realistic), and source model (spherical and realistic) are shown in figure 3.10.

Results

99

Block-ERC in clustered representation As explained in Section 3.3, generally, in transforming the Block-SL q,p , which is determined by Block-ERC, into the conventional sparsity level, different types of sparsity levels can be appeared, e.g., minimum sparsity level SL min , and maximum sparsity level SL max . On the other hand, as described earlier in the Section 3.5, number of branches in a clustering level with maximum inter-node distance in a clustering tree resulted from a hierarchical clustering algorithm can be used as an estimation of the number of clusters of a dictionary. Assume the sparsity level computed in the clustering level corresponding to the maximum inter-node distance in the clustering tree is shown by SL 2 , whereas the sparsity level computed for the lead-field without clustering is called SL 1 . In order to investigate the effect of clustering the coherent blocks of the lead-field on sparsity levels, we compute the relative change quantity, i.e., (SL 2 (Φ)-SL 1 (Φ))/SL 1 (Φ).

In figure 3.11, the relative change of sparsity levels SL min , and SL max is shown. In addition, the experiment is repeated for two modalities of EEG and MEG, and three head models with spherical, realistic inflated and realistic highly-folded cortical sheets.

As it can be seen in figure 3.11, by clustering coherent blocks of the lead-field (whether EEG/MEG or spherical/realistic cortical sheet) using the proposed Block-MCC 2,2 , all sparsity levels significantly increase, especially SL max . Hence, clustering coherent blocks of lead-field using the proposed Block-MCC q,p coherence measure leads to improved Block-ERC. Figure 3.11: Relative increase of sparsity levels SL min , SL max , when clustering the coherent blocks of lead-field, repeated for two modalities of EEG and MEG, and three head models with spherical, realistic inflated and realistic highly-folded cortical sheets.

EEG/MEG source space segmentation As described in Section 3.4.1, clustering the coherent blocks of the lead-field matrix results in some brain regions, because each block of the lead-field correspond to a single source position in the source space. Then, coherent source positions form brain regions, where, coherency is defined by the Block-MCC q,p coherence measure applied on the bocks of the lead-field.

The results of clustering coherent source positions using EEG and MEG lead-field, and for three head models with spherical, realistic inflated and realistic highly-folded cortical sheets are shown in figure 3.12. Although, in this experiment the number of clusters or brain regions is estimated from the clustering tree, but since there is a series of clustering structures in the hierarchical clustering analysis, by introducing extra information to the problem about the number of brain regions, there would be the possibility of having different brain regions. In other words, the resulted brain regions in figure 3.12 are not fixed and can be adapted based on the number of regions. In addition, the resulted brain regions are a function of Block-MCC q,p and inter-cluster distance method. Figure 3.12: Brain source space segmentation, when clustering the coherent blocks of lead-field, repeated for two modalities of EEG and MEG, and three head models with spherical, realistic inflated and realistic highly-folded cortical sheets.

Results

101

Comparison to the conventional lobes of the brain In order to compare the resulted brain source space segmentation in a specific clustering level to the conventional lobes of the brain, the closest boundaries to the conventional boundaries are shown in figure 3.13(a) and (b).

The brain segmentation and boundaries resulted from EEG, and MEG lead-fields are shown in figure 3.13(a), and (b), respectively. The anatomical lobes and the functional areas of the brain are shown in figure 3.13(c).

Therefore, using the proposed framework, i.e., clustering coherent blocks of the lead-field with the help of the proposed Block-MCC q,p coherence measure, the conventional structural and functional brain lobes can be divided into subject-specific refined regions.

Conclusion

In this chapter, we investigated the effect of clustering the coherent blocks of the dictionary on the Block-ERC and brain source space segmentation. The coherency is defined based on the proposed general characterisation of Block-MCC q,p . To this aim, different experiments were done on synthetic dictionary and real EEG/MEG lead-field.

In experiments with synthetic dictionary, we demonstrated in figure 3.8 the positive effect of clustering the coherent blocks of the dictionary on the sparsity levels. Then, block-sparse exact recovery condition based on the block mutual coherence constant improves by clustering the coherent blocks. Furthermore, in figure 3.9 we showed that this improvement in the proposed block-sparse exact recovery conditions due to hierarchical clustering, even holds to the conventional exact recovery condition.

Finally, in experiments with real EEG/MEG lead-field in figure 3.11, we showed that the sparsity levels in different cases increase by clustering coherent lead-fields, hence, improves the corresponding block-sparse exact recovery condition. In figure 3.12, we showed that even by the mentioned clustering method, some consistent brain regions appear. By utilising the MRI of a person in calculating the realistic volume conduction head model, subject-specific brain regions appear. In fact, here we proposed a segmented structure of brain sources, making use of just the lead-field matrix in a general way, without being restricted to a special activity of dipoles β 0 or EEG/MEG signals y.

Ultimately, in this chapter for identifying the block structure of the dictionary or lead-field, a general framework based on the proposed block mutual coherence constant Block-MCC q,p is proposed, which at the same time improves the block-sparse exact recovery condition and provides a brain source space segmentation strategy.

Chapter 4

EEG and MEG multi-modality

Introduction

As mentioned in Section 1.2, EEG1 and MEG2 source reconstruction problems solve a same brain source activity problem when acquired simultaneously [START_REF] Molins | Quantification of the benefit from integrating MEG and EEG data in minimum l2-norm estimation[END_REF]. In other words, for EEG or MEG signal y EEG/MEG ∈R m , EEG or MEG lead-field matrix Φ EEG/MEG ∈R m×n , and brain source activity β 0 ∈R n , we have the following EEG and MEG USLE3 :

y EEG = Φ EEG β 0 ,
and

y MEG = Φ MEG β 0 .
The key aim in this chapter is to integrate the complementary information of EEG and MEG modalities within the block structure identification framework introduced in Chapter 3 with application to brain source space segmentation.

In contrast to the brain source space segmentation studies in literature, which are integrating the information of EEG or MEG signals y EEG/MEG and lead-fields Φ EEG/MEG into the problem, the proposed multi-modality strategy uses the block structure identification framework, which requires only the lead-field matrices, in order to demonstrate the benefit of merging EEG and MEG modalities at the same complexity or number of sensors.

104

Chapter 4. EEG and MEG multi-modality 4.2 Multi-modal lead-field

Structure and model

In order to investigate the impact of multi-modality, i.e., the integration of the complementary information of EEG and MEG modalities, we need to think of a strategy to combine the mono-modalities.

Suppose that the odd rows of USLE related to EEG generative model is represented as y O EEG =Φ O EEG β 0 , and similarly the even rows of USLE related to MEG model as

y E MEG =Φ E MEG β 0 .
Then, by concatenating the odd rows and even rows of EEG and MEG generative models, respectively, the EEG and MEG multi-modal model is as follows:

y O EEG y E MEG = Φ O EEG Φ E MEG β 0 . (4.1)
There are three reasons behind the mentioned strategy of integration of mono-modalities.

1) Fixed number of sensors Assuming that the number of sensors in both modalities is even, then each signal y O EEG ∈R m/2 or y E MEG ∈R m/2 has as half of observations as the original signal y EEG/MEG ∈R m , i.e., each modality contributes the same in terms of the number of sensors, so the comparison would be more fair compared to the odd number of one of original modalities' sensors. Then, by concatenating them, the number of equations in the resulted USLE in multi-modality would be equal to the the original USLE in mono-modality.

By keeping fixed the number of sensors in mono-modality and multi-modality, we assure that the probable added value is because of the multi-modality itself and not in reason of the increased observation. Because, in case of utilising all the equations in each of mono-modal USLE corresponding to EEG and MEG, the equations in final USLE would be more than the original USLE, i.e., y EMEG ∈R 2m , where, y EMEG is the multi-modal observation. Therefore, we would have an increased input information in multi-modality in comparison to mono-modality, which would not lead to a fair comparison.

2) Minimal change in sensors position By drawing for example the odd rows of one modality and even rows of the other, i.e., without shared index for sensors position, we again ensure that the possible enhancement is due to the multi-modality itself and not because of major change in sensors position. Otherwise, if for example we select odd rows from both mono-modalities, the sensors layout in mono-modality would be significantly different from multi-modal layout. Because we are assuming that each MEG sensor is selected among the whole standard sensors, based on the proximity to an EEG sensor, to minimise the distance between each pair of EEG and MEG sensors. So, if the index set of selected sensors in one mono-modality does not have any shared sensor's index with the other mono-modality, the difference in sensors position in mono-modality and multi-modality will be minimised.

Multi-modal lead-field 105

3) Uniform sub-sampling in sensor space By selecting even or odd rows of USLE, we are assuring that the sensors are uniformly distributed all over the head. Otherwise, by selecting for example the first half of one mono-modal sensors and the second half of the other mono-modal sensors, although the multi-modal sensors cover whole parts of the head, we would lose the information of one part of brain in mono-modality. Therefore, considering the three mentioned reasons in multi-modality strategy, we are eliminating or reducing the side effects of increased observation, and optimum sensor position, so we can investigate the impact of multi-modality itself.

As mentioned before, the purpose of this chapter is to investigate the impact of multi-modal lead-field matrix within the block structure identification framework introduced in Chapter 3 with application to brain source space segmentation. Therefore, from the multi-modal USLE in (4.1), we only need to lead-field matrices Φ EEG and Φ MEG , or precisely Φ O EEG and Φ E MEG . In figure 4.1(c), the multi-modal lead-field Φ EMEG is built by concatenating the odd rows of Φ EEG (blue lead-field) and the even rows of Φ MEG (green lead-field).

As represented graphically in figure 4

Results

In this section, the multi-modality strategy will be experimented on synthetic dictionaries and real EEG/MEG lead-fields. In all of the experiments, the agglomerative hierarchical clustering analysis based on the block structure identification framework proposed in Chapter 3 will be applied on the dictionaries.

The proposed inter-block coherence measure in a special case, Block-MCC 2,24 , is used to compute the similarity matrix in the hierarchical clustering algorithm, i.e.,

M 2,2 (Φ)= max k,k =k Φ † [k]Φ[k ] 2→2 /d max .
The standard complete method is used to compute the inter-cluster distance required in the hierarchical clustering algorithm.

Synthetic dictionary

In this part, the set of two complementary random dictionaries Φ i ∈R 10×80 , i={1, 2}, with equally-sized blocks of length d j =d=2, ∀j, is generated from independent and identically distributed (i.i.d.) random variables and all columns of the dictionaries are normalized to have unit 2 norm.

In order to generate two complementary random dictionaries, firstly, as described in Section 3.5.1, one of the dictionaries is generated by setting the related parameters ε inter and ε intra . Then, utilising the same strategy and by setting the parameter ε inter-dictionary , which controls the overlap between the two dictionaries, the second dictionary is produced. In figure 4.2(a) and (b), the role of different types of ε in generation of random dictionaries Φ 1 and Φ 2 is shown. Then, as shown in figure 4.2(c) by drawing half of the rows of each of the dictionaries and finally by making a sufficient shift on one of the set of rows belonging to one dictionary, here Φ 2 , the combined dictionary Φ 12 is obtained. All the results are shown in average and standard deviation over 100 repetitions of generation of complementary random dictionaries pairs. Figure 4.2: The role of (a) ε inter and ε intra in generation of random dictionary Φ 1 , and (b) ε inter-dictionary in generation of Φ 2 to produce the multi-modal dictionary Φ 12 .

Results

107

Block-ERC in clustered representation for a combined dictionary In Section 3.5.1, we demonstrated that in a random dictionary with certain values of ε inter and ε intra , the Block-SL 2,25 computed in each clustering level of the hierarchical clustering algorithm have a peak value at a clustering level equal to the number of clusters in the generated dictionary. Therefore, under certain values of ε inter and ε intra , the argument of the peak in Block-SL 2,2 (Φ)[%] computed at each level of the hierarchical clustering algorithm is equal to the number of clusters in the dictionary, i.e., N .

In this part, in an experiment similar to Section 3.5.1, the block sparsity-level in the pessimistic case, i.e., Block-SL 2,2 (Φ)[%], is computed in each clustering level of the hierarchical clustering analysis, but here the dictionary is a combination of two complementary dictionaries instead of being a simple mono-modal random matrix. In Section 4.2.1, the procedure of combining two complementary dictionaries is described. Also, the complementarity of two dictionaries is defined based on their overlap, determined by ε inter-dictionary .

In generating two complementary dictionaries, one of them is produced as explained in Section 3.5.1. Then the other dictionary is generated by multiplying the first dictionary to a random matrix exp((C-C T ) √ 2ε inter-dictionary / C-C T F ), where, C is a square random matrix of dimension m.

As it can be seen in figure 4.3, by applying hierarchical clustering on the blocks of the combined dictionary for ε inter-dictionary =1, ε inter ≥3.5 and ε intra ≤0.1, a peak in Block-SL 2,2 (Φ)[%] appears in an argument equal to the sum of the clusters in each of the two original dictionaries.

For instance, when there are four clusters in each of the two complementary dictionaries i.e., N =4 (blue squares), which construct the combined one, the maximum Block-SL 2,2 (Φ)[%] for combined dictionary is at the eighth clustering level.

Similarly the consistent results are obtained for five clusters in each of the two complementary dictionaries, i.e., N =5 (red circles). In other words, there is a peak in Block-SL 2,2 (Φ)[%] at the tenth clustering level, when the multi-modal dictionary is composed of two mono-modal dictionaries, each has five clusters. For the clustering level lower than the optimal level, the space is under-sampled and there would be a block spanning the whole space, whereas for the clustering level higher than the optimal level, the space is over-sampled and the over-partitioning leads to high coherence measure.

On the other hand, when the average blocks of each cluster of blocks are not different enough, i.e., ε inter <3.5, but still the blocks in each cluster of blocks are similar enough, i.e., ε intra ≤0.1, we can observe that the number of clusters in the combined dictionary or the argument of the peak in Block-SL 2,2 (Φ)[%] is less than the sum of clusters in each of initial dictionaries but greater than the number of clusters in each of initial dictionaries.

Chapter 4. EEG and MEG multi-modality

Therefore, the number of clusters in the combined dictionary depending on the overlap between the blocks, which is determined by ε, is greater than the number of clusters in one of the initial dictionaries and at most is equal to the sum of the number of clusters in the initial dictionaries.

In addition, the amplitude of peak of Block-SL 2,2 (Φ)[%] increases when the number of clusters decreases. For instance, in figure 4.3, the amplitude of average peak of Block-SL 2,2 (Φ)[%] for multi-modal dictionaries composed of two four-clusters dictionaries (blue square) is higher than the corresponding average amplitude for five clusters in each of mono-modal dictionaries (red circle).

Therefore, by reducing the number of clusters in the dictionary, Block-SL 2,2 (Φ)[%] increases, hence, more weakened Block-ERC6 are obtained.

Real EEG and MEG lead-field

In this part, the original lead-fields, which build the multi-modal lead-field are the ones explained in Section 3.5.2. Therefore, there are three head models with spherical, realistic inflated, and realistic highly-folded cortical sheets as shown in figure 4.4.

The 32 sensors shown in figure 4.4 are in multi-modal case, where, there are equal number of sensors from both modalities. By selecting the odd and even rows of EEG and MEG USLE, respectively, we can see that the sensor distribution is almost uniform, i.e., the sensors are not concentrated in a region of brain.

EEG and MEG complementarity in brain source segmentation In Section 1.2.6, some basic complementarities of EEG and MEG were discussed. In this section we show that in the brain source segmentation using the proposed strategy of clustering coherent lead-fields corresponding to brain sources utilising Block-MCC q,p , we can observe some complementarities between brain source segmentation employing EEG or MEG lead-field.

In figure 4.5, the brain clusters or regions based on clustering of the EEG or MEG lead-field for clustering levels 3, 10, 15, 17, and 20 and three types of cortical sheets are shown.

It can be seen that for the all shown clustering levels, the previously mentioned complementarity also appears in our brain source segmentation strategy, which is more visible in low clustering levels in figure 4.5.

Results

111

EEG and MEG multi-modality and the clustering tree The agglomerative hierarchical cluster analysis, using the standard inter-cluster distance and our proposed inter-block similarity methods builds a hierarchy of clustering structures, which is represented in the clustering tree, as shown in figure 4.6.

The clustering trees built using EEG, MEG, and multi-modal EEG+MEG lead-fields for spherical, realistic inflated, and realistic highly-folded cortical sheets are shown in figure 4.6(a), (b), and (c), respectively.

As mentioned before, the advantage of using the hierarchical clustering algorithm is in estimating the number of clusters and in general the block structure, by selecting a block structure corresponding to the longest inter-node distance. Also, one can impose other constraints such as minimum number of clusters.

In figure 4.6, for each source model, the block structure is opted based on a minimum number of clusters and maximum inter-node distance. To make clusters distinguishable, each cluster is colour-coded. EEG and MEG multi-modality and the number of regions In order to estimate the number of clusters from the dendrogram as described in Section 3.5, in addition to the constraint of the maximum distance between consecutive nodes or clusters in dendrogram, the number of clusters is also set to be greater than a threshold.

As it can be seen in figure 4.7(a), the estimated number of clusters or brain regions resulted from the EEG and MEG multi-modal lead-field for spherical cortical sheet is 10, which is more than the estimated number of brain regions using EEG or MEG lead-field alone, which both are 8 clusters. The similar results are obtained for realistic head models as shown in figure 4.7(b) and (c).

Therefore, brain source space segmentation using the proposed strategy of clustering the coherent lead-fields based on the introduced Block-MCC 2,2 is more refined when utilising EEG and MEG multi-modal lead-field compared to EEG or MEG mono-modal lead-fields.

Results

113

EEG and MEG multi-modality and the area of regions In this part, for a fixed number of regions or clusters, the area of brain regions is investigated for EEG/MEG mono-modal lead-fields and multi-modal EEG and MEG lead-field clustering.

In this experiment, to better demonstrate the impact of multi-modality on the area of brain regions, only the source model with spherical cortical sheet is used. In addition the standard average method is used to compute the inter-cluster distance. In this method, the distance between two clusters is defined as the average distance between each point in one cluster to every point in the other cluster. Furthermore, the Block-MCC 1,∞ is used as coherence measure.

In the top row of figure 4.8, it can be seen that in each certain number of clusters, the biggest area (maximum value in bar chart) belongs to multi-modal EEG and MEG lead-field clustering and nearly for all the other regions the area for multi-modal EEG and MEG is less than EEG or MEG lead-field alone.

In addition, the area of brain regions are represented in pie charts. The largest area is coloured as blue in pie charts, which is not very obvious in brain source space in the current angle of view. Because, the region with maximum area represents brain sources at the bottom of the head, where, there is not any sensor on it, whereas the other regions represent brain sources under the sensors.

Therefore, for a fixed number of clusters, brain regions under sensors in multi-modal lead-field clustering are smaller than mono-modal lead-field clustering. Hence, EEG and MEG multi-modality leads to increased brain source space resolution.

Conclusion

In this chapter, first we showed the effectiveness of multi-modality on synthetic data through demonstrating that the number of clusters in a combined random dictionary is equal to the sum of the number of clusters in its two original random dictionaries.

Then, passing from synthetic data to real one, we showed that by clustering coherent brain sources through clustering their corresponding EEG or MEG lead-fields, some regions appear. The brain regions resulted from two mono-modal lead-fields are somehow complement each other.

Then, by applying the block structure identification framework introduced in Chapter 3, i.e., assuming that the number of clusters can be estimated by the largest distance between the adjacent nodes in dendrogram, we demonstrated that by combining EEG and MEG modalities, the number of clusters existing in the resulted multi-modal lead-field increases.

In addition, in another experiment we demonstrated that for a fixed number of clusters, the under-sensor brain regions are smaller in multi-modal lead-field compared to the mono-modal one. Therefore, more refined segmentation can be achieved in EEG and MEG multi-modality clustering using the proposed coherency measure. In all the above experiments the number of sensors for mono-modal or multi-modal cases was kept to be equal.

Future research should focus on:

• Study on electromagnetic properties in 2D/3D mediums with different number of boundaries as a general case.

• Designing more scenarios (other than brain segmentation) to investigate the impact of multi-modality.

• Designing more methods to combine modalities.

• The impact of combining more modalities (greater than two).

• The optimum ratio of the number of measurements from each modality (not necessarily 50% in two modalities case).

High-dimensional problem

Challenge 1. Many real-world inverse problems are vastly underdetermined, and classical sparse estimation techniques do no longer give acceptable recovery conditions. How can high-dimensional problems be adapted in favour of the coherence-based notion of conventional conditions?

For vastly underdetermined systems of linear equations, which the columns of the coefficient matrix are great in number, the classical dictionary characterisations such as mutual coherence constant are more likely to be high. Because the classical dictionary characterisations measure the coherency of the columns of the dictionary. Naturally, increasing in the number of columns of the dictionary would be equivalent to increasing the probability of coherency between the columns of the dictionary. On the other hands, due to the inverse relationship between the sparsity level defined in the exact recovery conditions and the classical dictionary characterisation, high coherency leads to low sparsity level. Therefore, being vastly underdetermined has a negative effect on the exact recovery conditions.

Assuming that the atomic entity in classical dictionary characterisations is columns of the dictionary, one possible solution would be to decrease the number of atomic entities by changing the concept of atomic entity from columns to block of columns of the dictionary. Therefore, in Chapter 3 we proposed to cluster the coherent entities (columns or block of columns) of the dictionary. By the idea of clustering the coherent entities to build new entities, we are improving the exact recovery conditions by gaining two advantages at the same time:

(1) reducing the number of entities, (2) building more incoherent entities.

In the EEG/MEG source reconstruction problem, the initial entity is already a fixed-length block of columns (three columns), then we clustered the coherent blocks of columns. In order to determine the coherency between block of columns we utilised the dictionary characterisation block mutual coherence constant proposed in Chapter 2.

Then, for EEG/MEG source reconstruction problem we profited the idea of clustering the coherent blocks of columns, i.e., brain source lead-fields, to segment the brain source space. The proposed strategy for brain source space segmentation is in a general case, without being restricted to know any information about the sources activity and sensors measurement, only utilises EEG/MEG lead-field matrix. In addition, at the same time by clustering the coherent sources and forming the brain regions, the number of brain regions in which it is ensured to have a unique solution in the EEG or MEG source reconstruction problem is improved.

As perspective in this domain, different standard or customized clustering strategies can be investigated to find an optimum clustering strategy, which leads to improvement in the exact recovery conditions. One possible strategy in hierarchical clustering algorithm would be to update the similarity matrix at each clustering level, i.e., recompute the new distance of clusters and update the similarity matrix based on new clustering structure.

Block-structured problem

Challenge 2. In classical assumption, the recovery conditions are generated from the columns of coefficient matrix, but clustered coefficient matrix consists of some differently-sized blocks, and not necessarily columns. Therefore, the initial assumption of classical recovery conditions does no longer hold true. How can appropriate recovery conditions be developed for block-structured problems?

In Chapter 2, we defined a general framework to cover all types of atomic entity including columns, and equally/differently-sized blocks of columns, in order to propose general theoretical exact recovery conditions. The proposed theoretical recovery conditions are based on block-sparsity constraint, which ensure the uniqueness of the block-sparse solution of corresponding weighted (pseudo-)mixed-norm optimisation problem in an underdetermined system of linear equations. The mentioned generality of the framework is in terms of the properties of the underdetermined system of linear equations, extracted characterisations, optimisation problems, and ultimately the recovery conditions. The mentioned theoretical exact recovery conditions are categorized in four different groups based on the utilised characterisations and properties, i.e., conditions based on (1) Block-Spark, (2) block null space property, (3) block mutual coherence constant, and (4) cumulative coherence constant.

On the other hand, the proposed framework is consistent with the base findings, since all the materials in the proposed infrastructure are a generalisation of the existing references Indeed, we investigated the theoretical relationship between the proposed infrastructure and the classical one, and showed that all the new materials reduce to the conventional ones in specific cases. We redemonstrated the benefit of block-sparsity assumption compared to conventional sparsity in the improvement of the recovery conditions. In addition, we theoretically proved the supremacy of the theoretical exact recovery conditions defined in the proposed general infrastructure over existing conditions, which are assuming the same block-sparsity constraint.

As perspective in this research direction, we could mention to the following subjects:

• Introducing block-sparse recovery conditions based on the proposed cumulative Block-MCC q,p defined in Definition 2.7.

• Generalising the conventional dictionary characterisation of µ defined in [START_REF] Donoho | Optimally sparse representation in general (nonorthogonal) dictionaries via 1 minimization[END_REF], to establish block-sparse recovery conditions.

• Transforming all the previously mentioned block-sparse exact recovery conditions to block-sparse stable recovery conditions. In stable or robust recovery conditions we have, y-Φ β 2 <e, where, e is a bounded noise.

• Study on block-sparse optimisation algorithms, and the relationship between the theoretical and algorithmic block-sparse recovery conditions.

Proof. In order to demonstrate the bounds of a p / a q , first we prove it using the derivative method for ∀(q, p)∈R >1 , next utilising the Hölder's inequality we show that the same bounds hold true for the wider range of q and p, i.e., ∀(q, p)∈R >0 .

1) To compute the critical point of a p / a q using the derivative method, we need to compute its derivative with respect to the coordinates, knowing that (f /g) =(f g-g f )/g 2 , and (|f |) =f f /|f |, where f is derivative of f with respect to x, i.e. d f (x)/d x, we have:

∀(q, p) ∈ R >1 , ∂ ∂a i a p a q = ∂ a p ∂a i a q - ∂ a q ∂a i a p a 2 q (A.1)
Then, we need to compute ∂ a p /∂a i and ∂ a q /∂a i , so:

∂ a p ∂a i = ∂ ∂a i i |a i | p 1 p = 1 p i |a i | p 1 p -1 ∂ ∂a i i |a i | p = a 1-p p |a i | p-2 a i .
Therefore, substituting ∂ a p /∂a i and ∂ a q /∂a i in (A.1) and make it equal to zero, we have:

∂ ∂a i a p a q = a q a 1-p p |a i | p-2 a i -a p a 1-q q |a i | q-2 a i a 2 q = a i a p a q |a i | p-2 a p p - |a i | q-2 a q q = 0 ⇒ |a i | ∈    0, a p p a q q 1 p-q    .
Hence, the derivative cancels for all a with 1≤m≤d non-zero elements that all are equal, i.e., |a i |=( a p p / a q q ) 1/(p-q) =C∈R >0 , whereas all other elements are identically zero. Then, the fraction in the critical point evaluates to

∀(q, p) ∈ R >1 , a p a q = (mC p ) 1 p (mC q ) 1 q = m 1 p -1 q ,
119 which is minimised for m= d 1/p-1/q , if q≤p 1, if q>p = min{1, d 1/p-1/q }, and maximised for

m= 1, if q≤p d 1/p-1/q , if q>p = max{1, d 1/p-1/q }.
2) Knowing that the p norm is a decreasing function in p, the following lower-bound is resulted for p≤q:

∀a ∈ R d , ∀(q, p) ∈ R 2 >0 , if p ≤ q ⇒ a q ≤ a p ⇒ 1 ≤ a p a q .
Now to demonstrate the upper-bound, the following Hölder's inequality is used [START_REF] Golub | Matrix Computations[END_REF]:

∀r ∈ R ≥1 , d i=1 |x i y i | ≤ x r y r r-1 = d i=1 |x i | r 1 r d i=1 |y i | r r-1 r-1 r .
Next, assuming x i =a i p , y i =1, and r=q/p (the above condition on r is met, because p≤q), we have

d i=1 |x i y i | = d i=1 |a i p | = d i=1 |a i | p ,
and the above equation turns into:

∀a ∈ R d , ∀(q, p) ∈ R 2 >0 , p ≤ q, d i=1 |a i | p ≤ d i=1 (|a i | p ) q p p q d i=1 1 q q-p q-p q = d i=1 |a i | q p q d q-p q .
Then, taking the both sides to the power of 1/p, we get:

∀a ∈ R d , ∀(q, p) ∈ R 2 >0 , p ≤ q, d i=1 |a i | p 1 p ≤ d i=1 |a i | q 1 q d 1 p -1 q ⇒ a p a q ≤ d 1 p -1 q .
Therefore, for p≤q, we have 1≤ a p / a q ≤d 1/p-1/q . Similarly, for q≤p, we get d 1/p-1/q ≤ a p / a q ≤1, which proves the property.

A.2 Proof of Property 2.4 ( q→p operator-norm properties, page 56)

Proof. Based on the definition of the A q→p , i.e., max a =0 Aa p / a q or max a q ≤1 Aa p , we prove the following properties, for A∈R m×n , B∈R m×n , and C∈R n×l :

• Nonnegativity: In definition A q→p = max a q ≤1 Aa p , Aa p is greater than or equal to zero, then the nonnegativity of A q→p is obvious ∀(q, p)∈R 2 ≥0 . • Positivity: ∀(q, p)∈R 2 ≥0 if A q→p =0, then Aa p =0 for each a∈R n , i.e., each column in A is zero. Hence, A=0.

• Homogeneity: ∀q≥0 and ∀p>0, we have

αA q→p = max a =0 αAa p a q = max a =0 |α| Aa p a q = |α| A q→p .
The above second equality in the first line is resulted from the homogeneity property of norm of vectors, which holds true only for p>0 [START_REF] Elad | Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing[END_REF]; [START_REF] Golub | Matrix Computations[END_REF].

• Triangle inequality: ∀q≥0, p=0 or ∀p≥1, it is obtained as follows

A + B q→p = max a =0 (A + B) a p a q ≤ max a =0 Aa p + Ba p a q ≤ max a =0 Aa p a q + max a =0 Ba p a q = A q→p + B q→p .
The above inequality in the first line is resulted from the triangle inequality property of norm of vectors, which holds true only for p=0 and p≥1 [START_REF] Elad | Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing[END_REF]; [START_REF] Golub | Matrix Computations[END_REF].

• Submultiplicativity: It is obtained as follows:

AC q→p = max a =0 ACa p a q = max a =0 ACa p a q Ca q Ca q Ca p Ca p ≤ max Ca =0 ACa p Ca q max a =0 Ca p a q max a =0 Ca q Ca p = A q→p C q→p max 1, n 1 q -1 p . (A.2)
The last term of (A.2) is obtained using the upper-bound in Property 2.2 (bounds of two vector norms division), i.e.,:

∀ (q, p) ∈ R 2 >0 , ∀a ∈ R d , min 1, d 1 q -1 p ≤ a q a p ≤ max 1, d 1 q -1 p .
Therefore, from (A.2) for q≥p>0, we see that the submultiplicativity holds true, i.e., AC q→p ≤ A q→p C q→p .

• Bounds: 1) Considering the definition of q→p operator-norm, i.e., max a =0 Aa p / a q , and the lower-bound of division of two vector norms introduced in Property 2.2 (bounds of two vector norms division), i.e., ∀(q, p)∈R 2 >0 , ∀a∈R d : a q / a p ≥ min 1, d 1/q-1/p , we conclude the following lower-bounds for A∈R m×n , a∈R n , and ∀ (q, p, q , p ) ∈R 4 >0 :

∀a = 0, Aa p a q ≥ min 1, m 1 p -1 p Aa p a q , ∀a = 0, Aa p a q ≥ min 1, n 1 q -1 q Aa p a q .
Therefore, we have

∀a = 0, Aa p a q ≥ max min 1, m 1 p -1 p Aa p a q , min 1, n 1 q -1 q Aa p a q .
Then, taking into account that the max operator is order preserving, i.e., ∀x, if f (x)≤g(x), then max x f (x)≤ max x g(x), the lower-bound of A q→p is achieved.

Similarly, utilising the upper-bound of division of two vector norms introduced in Property 2.2 (bounds of two vector norms division), i.e., ∀(q, p)∈R 2 >0 , ∀a∈R d : a q / a p ≤ max{1, d 1/q-1/p }, results in the following inequalities, which conclude the upper-bound of A q→p for A∈R m×n , a∈R n , and ∀ (q, p, q , p ) ∈R 4 >0 :

∀a = 0, Aa p a q ≤ max 1, m 1 p -1 p Aa p a q , ∀a = 0, Aa p a q ≤ max 1, n 1 q -1 q Aa p a q .
Therefore,

∀a = 0, Aa p a q ≤ min max 1, m 1 p -1 p Aa p a q , max 1, n 1 q -1 q Aa p a q .
2) To prove the second set of bounds, from Property 2.2 (bounds of two vector norms division), i.e., ∀(q, p)∈R 2 >0 , ∀a∈R d : a q / a p ≥ min{1, d 1/q-1/p }, we have the following inequalities for A∈R m×n , a∈R n , and ∀ (q, p, q , p ) ∈R 4 >0 :

∀a = 0, Aa p ≥ min 1, m 1 p -1 p Aa p , ∀a = 0, 1 a q ≥ min 1, n 1 q -1 q 1 a q .
Considering that both sides of the above inequalities are positive, by multiplying the same sides we get:

∀a = 0, Aa p a q ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q Aa p a q .
Then taking into account that the max function is order preserving, the proof is done for lower-bound.

Similarly, for the upper-bound, from Property 2.2 (bounds of two vector norms division), i.e., ∀(q, p)∈R 2 >0 , ∀a∈R d : a q / a p ≤ max{1, d 1/q-1/p }, we have the following inequalities for A∈R m×n , a∈R n , and ∀ (q, p, q , p ) ∈R 4 >0 :

∀a = 0, Aa p ≤ max 1, m 1 p -1 p Aa p , ∀a = 0, 1 a q ≤ max 1, n 1 q -1 q 1 a q .
Considering that both sides of the above inequalities are positive, by multiplying the same sides we get:

∀a = 0, Aa p a q ≤ max 1, m 1 p -1 p max 1, n 1 q -1 q Aa p a q .
Then taking into account that the max function is order preserving, the proof is done for upper-bound.

In order to prove the q→p operator-norm inequalities shown schematically in figure 2.3, utilising the lower-bound of second set of bounds in Property 2.4 ( q→p operator-norm properties), i.e., ∀ (q, p, q , p ) ∈R 4 >0 , ∀A∈R m×n : A q→p ≥ min{1, m 1/p-1/p } min{1, n 1/q -1/q } A q →p , it is straightforward that for a fixed p=p if q ≤q (or similarly, for a fixed q=q if p≤p ) we have A q→p ≥ A q →p .

3) In order to prove the lower-bound of the third set of bounds, using the lower-bound of second set of bounds in Property 2.4 ( q→p operator-norm properties), i.e., ∀ (q, p, q , p ) ∈R 4 >0 , ∀A∈R m×n : A q→p ≥ min{1, m 1/p-1/p } min{1, n 1/q -1/q } A q →p , we will have the following first line of inequalities. Then, by utilising the same inequality once again, this time for q =p =2, we get the following second line of inequalities. The reason that first the relation with A q →p is stated and then the relation with A 2→2 , is that if in a problem the q and p values are bounded based on some thresholds such as q and p , respectively, e.g., q≤/≥q and p≤/≥p , then there could be a unique solution to the related min and max operators. Whereas, if directly the relation with A 2→2 was stated, and supposing that p≥1, then form instance min{1, m 1/p-1/2 } would have two values based on the value of p compared to 1. Then from [GL13], we have A 2→2 ≥ A F / Rank(A), where the Frobenius norm is defined as ( m i=1 n j=1 |a i,j | 2 ) 1/2 , which produces the following third line. But Rank(A) is upper-bounded by min{m, n} [GL13], then 1/ Rank(A) is lower-bounded by 1/ min{m, n}, which produces the following last line:

A q→p ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q A q →p ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q min 1, m 1 p -1 2 min 1, n 1 2 -1 q A 2→2 ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q min 1, m 1 p -1 2 min 1, n 1 2 -1 q Rank(A) A F ≥ min 1, m 1 p -1 p min 1, n 1 q -1 q min 1, m 1 p -1 2 min 1, n 1 2 -1 q min {m, n} A F .
In order to prove the upper-bound of the last set of bounds, using the upper-bound of second set of bounds in Property 2.4 ( q→p operator-norm properties), i.e., ∀ (q, p, q , p ) ∈R 4 >0 , ∀A∈R m×n : A q→p ≤ max{1, m 1/p-1/p } max{1, n 1/q -1/q } A q →p , we will have the following first line of inequalities. Then, by utilising the same inequality once again, this time for q =p =2, we get the following second line of inequalities. But from [GL13], we have A 2→2 ≤ A F , which produces the following last line:

A q→p ≤ max 1, m 1 p -1 p max 1, n 1 q -1 q A q →p ≤ max 1, m 1 p -1 p max 1, n 1 q -1 q max 1, m 1 p -1 2 max 1, n 1 2 -1 q A 2→2 ≤ max 1, m 1 p -1 p max 1, n 1 q -1 q max 1, m 1 p -1 2 max 1, n 1 2 -1 q A F .
4) In order to prove the relation between the q→p operator-norm of two matrices A∈R m×n and B∈R m×n , having the condition ∀i, j, |a i,j |≤b i,j = max i,j |a i,j |, or even when the on-diagonal entries are set to zero, we can utilise the second bounds proposed in the current property. First, using the lower-bound of the mentioned set of bounds, i.e., ∀ (q, p, q , p ) ∈R 4 >0 , ∀A∈R m×n : A q→p ≥ min{1, m 1/p-1/p } min{1, n 1/q -1/q } A q →p , for q =1 and p =∞, the following first line is produced.

Then considering min{1, m 1/p }=1 and also A 1→∞ = max i,j |a i,j |= max i,j |b i,j |= B 1→∞ , we can substitute A 1→∞ by B 1→∞ to have the following second line. Again, using the same lower-bound, this time for q=1, p=∞, q =q and p =p, we reach the following third line:

A q→p ≥ min 1, m 1 p min 1, n 1-1 q A 1→∞ = min 1, n 1-1 q B 1→∞ ≥ min 1, n 1-1 q min 1, m -1 p min 1, n 1 q -1 B q→p = m -1 p n -1-1 q B q→p . Therefore, A q→p B q→p ≥ m -1 p n -1-1 q m -1 p n -| 1-1 q | ≤1 = ========= =⇒ A q→p ≤ B q→p .
Similarly, using the upper-bound of the mentioned set of bounds, i.e., ∀ (q, p, q , p ) ∈R 4 >0 , ∀A∈R m×n : A q→p ≤ max{1, m 1/p-1/p } max{1, n 1/q -1/q } A q →p , and following above steps, we obtain:

B q→p A q→p ≤ m 1 p n 1-1 q m 1 p n | 1-1 q | ≥1 = ======= =⇒ A q→p ≤ B q→p .
A.3 Proof of Property 2.5 (Block-MCC q,p inequalities, page 58)

Proof. From Block-MCC q,p (Definition 2.6, page 54) we have:

∀(q, p) ∈ R 2 >0 , M q,p (Φ) def = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q→p .
Next, considering the relationship between the 1→1 and 1→2 , and also 1→2 and 1→∞ operator-norms, which are presented in general in Property 2.4 ( q→p operator-norm properties) and for certain (q, p) pairs in table 2.3, the proof is straightforward, as follows:

M 1,1 (Φ) def = max k,k =k d -1 k d k d max Φ † [k] Φ k 1→1 ≤ max k,k =k d -1 k d k d max d 1 2 k Φ † [k] Φ k 1→2 = max k,k =k d -1 2 k d k d max Φ † [k] Φ k 1→2 = M 1,2 (Φ) ≤ max k,k =k d -1 2 k d k d max d 1 2 k Φ † [k] Φ k 1→∞ = max k,k =k d k d max Φ † [k] Φ k 1→∞ = M 1,∞ (Φ) .
Similarly, for the second inequality taking into account the relationship between the 2→2 and 1→2 operator-norms, we have:

M 2,2 (Φ) def = max k,k =k d -1 2 k d 1 2 k d max Φ † [k] Φ k 2→2 ≤ max k,k =k d -1 2 k d 1 2 k d max d 1 2 k Φ † [k] Φ k 1→2 = max k,k =k d -1 2 k d k d max Φ † [k] Φ k 1→2 = M 1,2 (Φ) .
Then, for the third inequality considering the relationship between the 2→2 and 2→∞ , A.3. Proof of Property 2.5 (Block-MCC q,p inequalities, page 58) 127 and also 2→∞ and 1→∞ operator-norms, we have:

M 2,2 (Φ) def = max k,k =k d -1 2 k d 1 2 k d max Φ † [k] Φ k 2→2 ≤ max k,k =k d -1 2 k d 1 2 k d max d 1 2 k Φ † [k] Φ k 2→∞ = max k,k =k d 1 2 k d max Φ † [k] Φ k 2→∞ = M 2,∞ (Φ) ≤ max k,k =k d 1 2 k d max d 1 2 k Φ † [k] Φ k 1→∞ = max k,k =k d k d max Φ † [k] Φ k 1→∞ = M 1,∞ (Φ) .
Finally, for the last inequality we need to know the relationship between the ∞→∞ and 2→∞ operator-norms:

M ∞,∞ (Φ) def = max k,k =k 1 d max Φ † [k] Φ k ∞→∞ ≤ max k,k =k 1 d max d 1 2 k Φ † [k] Φ k 2→∞ = max k,k =k d 1 2 k d max Φ † [k] Φ k 2→∞ = M 2,∞ (Φ) .
For the proof of general relationships of figure 2.4, utilising the upper-bound of second set of bounds in Property 2.4 ( q→p operator-norm properties), i.e., ∀ (q, p, q , p ) ∈R 4 >0 , ∀A∈R m×n : A q→p ≤ max{1, m 1/p-1/p } max{1, n 1/q -1/q } A q →p , and for a fixed p=p , and q ≤q, which yields A q→p ≤n 1/q -1/q A q →p , we get the following inequalities:

∀(q, p, q ) ∈ R 3 >0 , q ≤ q, M q,p (Φ) def = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q→p ≤ max k,k =k d -1 p k d 1 q k d max d 1 q -1 q k Φ † [k] Φ k q →p = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q →p = M q ,p (Φ) .
Similarly, using the same upper-bound of Property 2.4, this time for a fixed q=q , and p≤p , which yields A q→p ≤m 1/p-1/p A q→p , we get the following inequalities:

∀(q, p, p ) ∈ R 3 >0 , p ≤ p , M q,p (Φ) def = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q→p ≤ max k,k =k d -1 p k d 1 q k d max d 1 p -1 p k Φ † [k] Φ k q→p = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q→p = M q,p (Φ) .
A.4. Proof of Property 2.7 (Block-MCC q,p bounds with intra-block orthonormality I, page 59) 129

A.4 Proof of Property 2.7 (Block-MCC q,p bounds with intra-block orthonormality I, page 59)

Proof. Using the definition of M q,p (Φ) in Property 2.6, and submultiplicativity property of operator-norm introduced in Property 2.4 ( q→p operator-norm properties), we have:

∀ (q, p) ∈ R 2 >0 , M q,p (Φ) = max k,k =k d -1 p k d 1 q k d max Φ T [k] Φ k q→p ≤ max k,k =k d -1 p k d 1 q k d max Φ T [k] q→p Φ[k ] q→p max 1, m 1 q -1 p . (A.3)
On the other hand, from the following property, • q→p can be bounded in terms of • 2→2 , which is often called the spectral norm:

Property A.1 (Bounds of q→p operator-norm in terms of 2→2 ). The bounds of the q→p operator-norm of a matrix A∈R m×n in terms of its 2→2 operator-norm based on the second set of bounds in Property 2.4 ( q→p operator-norm properties) for q =p =2 is:

A q→p ≥ min 1, m 1 p -1 2 min 1, n 1 2 -1 q A 2→2 , A q→p ≤ max 1, m 1 p -1 2 max 1, n 1 2 -1 q A 2→2 .
These bounds for different values of q and p are shown in table A.1.

Therefore, based on Property A.1, we upper-bound the q→p operator-norms of the upper-bound of Block-MCC in equation (A.3):

M q,p (Φ) ≤ max k,k =k d -1 p k d 1 q k d max max 1, d 1 p -1 2 k max 1, m 1 2 -1 q Φ T [k] 2→2 × max 1, m 1 p -1 2 max 1, d 1 2 -1 q k Φ[k ] 2→2 max 1, m 1 q -1 p . (A.4) But ∀k, Φ T [k] 2→2 = Φ[k] 2→2 =1
, because for a typical matrix A∈R m×n we have A 2→2 =σ max (A)= λ max (A T A), where, σ and λ are singular value and eigenvalue, respectively, but here A is an orthonormal matrix, then A T A=I n , and then λ max (I n )=1.

q & p ≤ 2 q & p ≥ 2 q ≤ 2 & p ≥ 2 q ≥ 2 & p ≤ 2 A q→p A 2→2 ≤ m 1 p -1 2 n -1 q + 1 2 1 m 1 p -1 2 n -1 q + 1 2 A q→p A 2→2 ≥ n -1 q + 1 2 m 1 p -1 2 m 1 p -1 2 n -1 q + 1 2 1 Table A.1: Bounds of A q→p / A 2→2
for different values of q and p, where, A is a m by n matrix.

max M q,p (Φ) max M q,p (Φ)

0 < q & p ≤ 2 max k,k =k d -1 2 k d 1 q k d -1 max m 1 p -1 2 max 1, m 1 q -1 p ≤ d -1 2 min d 1 q -1 max m 1 p -1 2 max 1, m 1 q -1 p q & p ≥ 2 max k,k =k d -1 p k d 1 2 k d -1 max m 1 2 -1 q max 1, m 1 q -1 p ≤ d -1 p min d -1 2 max m 1 2 -1 q max 1, m 1 q -1 p 0 < q ≤ 2 & p ≥ 2 max k,k =k d -1 p k d 1 q k d -1 max m 1 q -1 p ≤ d -1 p min d 1 q -1 max m 1 q -1 p q ≥ 2 & 0 < p ≤ 2 max k,k =k d -1 2 k d 1 2 k d -1 max m 1 p -1 q ≤ d -1 2 min d -1 2 max m 1 p -1 q Table A.2:
Upper-bound of Block-MCC q,p for different ranges of q and p and for a dictionary with intra-block orthonormality.

Therefore, the inequality (A.4) is developed into table A.2. In each case in the last column of the table A.2, the maximum value is achieved for minimum block length d min with negative power, and maximum block length d max with positive power. The table 2.4 shows the upper-bound of M q,p (Φ) based on table A.2, for the basic tractable values of q and p based on table 2.1.

A.5 Proof of Property 2.8 (Block-MCC q,p bounds, page 59)

Proof. 1) Since the operator-norm is nonnegative (Property 2.4), it can be seen that ∀k, k =k, ∀(q, p)∈R 2 ≥0 : Φ

† [k]Φ[k ] q→p ≥0 ⇒ ∀(q, p)∈R 2 >0 : M q,p (Φ)≥0.
Considering that for all k the blocks Φ[k] are full column rank, using the Moore-Penrose pseudo-inverse property of matrices, ∀(q, p)∈R 2 >0 we have:

M q,p (Φ) = max k,k =k d -1 p k d 1 q k d max Φ † [k] Φ k q→p = max k,k =k d -1 p k d 1 q k d max Φ T [k] Φ [k] -1 Φ T [k] Φ k q→p .
Applying the general format of submultiplicativity property of operator-norm of a matrix, which is introduced in the Property 2.4 ( q→p operator-norm properties) ∀(q, p)∈R 2 >0 , on the above last equality, ∀(q, p)∈R 2 >0 we have:

M q,p (Φ) ≤ max k,k =k d -1 p k d 1 q k d max Φ T [k] Φ [k] -1 q→p Φ T [k] Φ k q→p max 1, d 1 q -1 p k . (A.5) Since each element of the Φ T [k]Φ[k ]
is the pairwise correlation between the columns of the k and k =k blocks, for any k and k =k the Φ T [k]Φ[k ] can be upper bounded by M (Φ) 1 d k ×d k , so it is true for their operator-norms, because using Property 2.4 ( q→p operator-norm properties) for comparison of q→p operator-norm of two matrices (the forth property of the part Bounds), we have the following first line (based on table 2.1, the maximum absolute entry of a matrix is represented by its 1→∞ operator-norm), whereas by the homogeneity property of q→p operator-norm defined in Property 2.4 ∀q≥0 and ∀p>0, the following second line is obtained ∀(q, p)∈R 2 >0 :

∀k, k = k, Φ T [k] Φ k q→p ≤ Φ T [k] Φ k 1→∞ 1 d k ×d k q→p = Φ T [k] Φ k 1→∞ 1 d k ×d k q→p ≤ M (Φ) 1 d k ×d k q→p .
The above last line comes from the fact that ∀k, k =k, the maximum absolute value of multiplication of blocks k and k is less than or equal to M (Φ), which is by definition the maximum off-diagonal absolute value of multiplication of the whole dictionary to itself, i.e., M

Φ) def = max k,k =k |G k,k (Φ)|, where G(Φ) def =Φ T Φ. (
Then, using the upper-bound of the third set of bounds in Property 2.4, we will have the following second line. Then, by computing the Frobenius norm in the second line, ∀k, k =k and ∀(q, p, q , p )∈R 4 >0 we get the following third line:

Φ T [k] Φ k q→p ≤ M (Φ) 1 d k ×d k q→p ≤ M (Φ) max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k 1 d k ×d k F = (d k d k ) 1 2 M (Φ) max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k . (A.6) Therefore, substituting Φ T [k]Φ[k ]
q→p with its upper-bound in the (A.5), ∀(q, p, q , p )∈R 4

>0

we get:

M q,p (Φ) ≤ M (Φ) max k,k =k d 1 2 -1 p k d 1 q + 1 2 k d max Φ T [k] Φ [k] -1 q→p × max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k max 1, d 1 q -1 p k . (A.7) Now an upper bound for (Φ T [k]Φ[k]) -1 q→p should be developed.
As part of the required tools to reach this aim we will utilise the following property:

Property A.2. For q≥p≥1, if A q→p <1, then [HR12] (I + A) -1 = ∞ i=0 (-A) i .
Proof. It follows from Corollary 5.6.16 of [START_REF] Johnson | Matrix Analysis[END_REF]. Based on the Corollary 5.6.16 of [START_REF] Johnson | Matrix Analysis[END_REF], the matrix norm of A should be less than one. Therefore, we imposed the constraint q≥p≥1, because according to Remark 2.5 (matrix norm) on page 56 the q→p operator-norm for q≥p≥1 satisfies all the required properties to be conceived as a matrix norm. Therefore, according to Property A.2, we need to decompose the Φ T [k]Φ[k] into a sum of two matrices, which one of them is an identity matrix. For now we continue with the current constraints on q and p, i.e., ∀(q, p)∈R 2 >0 , but as soon as we use the above property the constraints will change to q≥p≥1.

Φ T [k]Φ[k] is a d k ×d k matrix
which all of its diagonal elements are one, and its off-diagonal elements measure pairwise correlation between the columns of the same block k. Therefore,

Φ T [k]Φ[k] can be decomposed as Φ T [k]Φ[k]=I d k +F [k], where: ∀k, f i,j [k] = 0, if i = j ϕ T i [k]ϕ j [k], if i = j.
But we are interested in the upper-bound of (Φ T [k]Φ[k]) -1 q→p . By the mentioned decomposition we have the following first line of inequalities. Then, utilising the Property A.2, assuming q≥p≥1 and F [k] q→p <1 (its veracity will be investigated) the following second line is obtained. Next utilising the generalisation (from two matrices to more than two matrices) of the triangle inequality property of the q→p operator-norm introduced in Property 2.4 ( q→p operator-norm properties) ∀q∈R ≥0 , ∀p∈R ≥1 , or for p=0, the following third line is obtained. Then utilising the generalisation (from two matrices to more than two matrices) of the submultiplicativity property of the q→p operator-norm introduced in Property 2.4 for q≥p>0, the following forth line is obtained. Finally, utilising the sum of an infinite geometric series, i.e., ∞ i=0 r i =1/(1 -r), where |r|<1, the following last line is obtained, considering the assumption F [k] q→p <1 (to be investigated).

∀k, q ≥ p ≥ 1, Φ T [k] Φ [k] -1 q→p = (I d k + F [k]) -1 q→p = ∞ i=0 (-F [k]) i q→p ≤ ∞ i=0 (-F [k]) i q→p ≤ ∞ i=0 F [k] i q→p = 1 1 -F [k] q→p . (A.8)
Now we investigate the veracity of the assumption F [k] q→p <1. Since each off-diagonal entry of F [k] is the pairwise correlation between the columns of the k th block and the on-diagonal values are all zero, it can be upper bounded by M (Φ) (1 d k -I d k ), so it is true for their operator-norms, because using Property 2.4 ( q→p operator-norm properties) for comparison of q→p operator-norm of two matrices ∀(q, p)∈R 2 >0 (the forth property of the part Bounds), we have:

∀k, q ≥ p ≥ 1, F [k] q→p ≤ F [k] 1→∞ (1 d k -I d k ) q→p = F [k] 1→∞ (1 d k -I d k ) q→p ≤ M (Φ) (1 d k -I d k ) q→p .
The 1→∞ operator-norm of a matrix computes the maximum absolute entry of the matrix (table 2.1). The above second line uses the homogeneity property of q→p operator-norm defined in Property 2.4 ( q→p operator-norm properties) ∀q≥0 and ∀p>0. The above last line comes from the fact that ∀k, the maximum off-diagonal absolute value of multiplication of block k to itself is less than or equal to M (Φ), because the off-diagonal values of F [k] are a subset of off-diagonal values of the Gram matrix G(Φ)

def =Φ T Φ, which the M (Φ) is derived from, i.e., M (Φ) def = max k,k =k |G k,k (Φ)|= max k,k =k |ϕ T k ϕ k |, hence F [k] 1→∞ ≤M (Φ).
Then, using the upper-bound of the third set of bounds in Property 2.4 ( q→p operator-norm properties), we will have the following second line. By computing the Frobenius norm in the second line, the third line is obtained. Considering the obtained constraint on q and p, i.e., q≥p≥1, we choose q =p =1 in the following third line, because both q and p are lower-bounded by one.

F [k] q→p ≤ M (Φ) (1 d k -I d k ) q→p ≤ M (Φ) max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k (1 d k -I d k ) F = d 2 k -d k 1 2 M (Φ) max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k = d 2 k -d k 1 2 d 1-1 q k d 1 2 k M (Φ) = d 2-1 q k (d k -1) 1 2 M (Φ) .
Therefore, for all k we have F [k] q→p ≤d 2-1/q max (d max -1) 1/2 M (Φ). On the other hand, in the Property 2.8, it is assumed that d 2-1/q max (d max -1) 1/2 M (Φ)<1, so the required condition to use the Property A.2 in (A.8), i.e., F [k] q→p <1, is satisfied. Substituting the obtained upper-bound of F [k] q→p in (A.8), we get:

∀k, q ≥ p ≥ 1, Φ T [k] Φ [k] -1 q→p ≤ 1 1 -F [k] q→p ≤ 1 1 -d 2-1 q k (d k -1) 1 2 M (Φ)
.

On the other hand, the previously obtained main equation (A.7) for new constraints q≥p≥1 (max{1, d 1/q-1/p k }=1) and q =p =1 becomes:

M q,p (Φ) ≤ M (Φ) max k,k =k d 1 2 -1 p k d 1 q + 1 2 k d max Φ T [k] Φ [k] -1 q→p × max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k max 1, d 1 q -1 p k = M (Φ) max k,k =k d 1 2 -1 p k d 1 q + 1 2 k d max Φ T [k] Φ [k] -1 q→p × max 1, d 1 p -1 k max 1, d 1-1 q k max 1, d 1 2 k max 1, d -1 2 k = M (Φ) max k,k =k d 1-1 p k d 3 2 k d max Φ T [k] Φ [k] -1 q→p .
(A.9)

Substituting the upper-bound of (Φ T [k]Φ[k]) -1 q→p obtained in equation (A.5) into the updated main equation (A.9), we get:

q ≥ p ≥ 1, M q,p (Φ) ≤ M (Φ) d max max k,k =k d 1-1 p k d 3 2 k 1 -d 2-1 q k (d k -1) 1 2 M (Φ) ≤ M (Φ) d max max k d 1-1 p k max k =k d 3 2 k max k 1 1 -d 2-1 q k (d k -1) 1 2 M (Φ) = d 3 2 -1 p max M (Φ) 1 -d 2-1 q max (d max -1) 1 2 M (Φ) .
2) Starting from the definition of coherence in Property 2.6 (block-MCC q,p for intra-block orthonormality) we have the following first line for ∀(q, p)∈R 2 >0 . Then considering the upper-bound of Φ T [k]Φ[k ] q→p for any k and k =k, obtained in (A.6), we will have the following second line ∀(q, p, q , p )∈R 4 >0 :

M q,p (Φ) = max k,k =k d -1 p k d 1 q k d max Φ T [k] Φ k q→p ≤ M (Φ) d max max k,k =k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k .
A.7 Proof of Theorem 2.2 (Block-NSP, page 67)

Proof. Under the assumption of Q w;p 1 ,p 2 (S b (β), Φ)<1/2 and S b (β 0 )⊂S b (β), to show that β 0 is the unique solution to the P w;p 1 ,p 2 minimisation problem, we need to prove that:

∀x ∈ Ker(Φ), β 0 p 2 w;p 1 ,p 2 < β 0 + x p 2 w;p 1 ,p 2 , where as defined in Definition 2.2 (weighted (pseudo-)mixed-norm, page 48), the w p 1 ,p 2 weighted (pseudo-)mixed-norm of a vector β 0 is:

β 0 w;p 1 ,p 2 def =                              k I   β 0 [k] p 1 d 1 p 1 k   , for p 2 = 0   k β 0 [k] p 2 p 1 d p 2 p 1 k   1 p 2 , for 0 < p 2 < +∞ max k    β 0 [k] p 1 d 1 p 1 k    , for p 2 = ∞.
Then, dividing the whole blocks to on-Block-Support (∈S b (β)) and off-Block-Support ( / ∈S b (β)), we have

0 < k∈S b (β)         d k j=1 β 0 j [k] + x j [k] p 1 d k p 2 p 1 - d k j=1 β 0 j [k] p 1 d k p 2 p 1         + k / ∈S b (β) d k j=1 |x j [k]| p 1 d k p 2 p 1 .
It can be rewritten in terms of the norm operator, so

0 < k∈S b (β) w p 2 k β 0 [k] + x [k] p 2 p 1 -β 0 [k] p 2 p 1 + k / ∈S b (β) w p 2 k x [k] p 2 p 1 .
where, in a block-structured vector a=

[a T [1], • • • , a T [K]] T with length vector d=[d 1 , • • • , d K ], w k in w k a[k] p is equal to d -1/p k .
From quasi-triangle inequality for scalars, we generalised it to vector space and derived the corresponding triangle inequality a+b p 2 p 1 -a p 2 p 1 ≥-b p 2 p 1 for 0≤p 2 ≤1≤p 1 . Therefore, it is sufficient to prove:

0 < k / ∈S b (β) w p 2 k x [k] p 2 p 1 - k∈S b (β) w p 2 k x [k] p 2 p 1 . Adding 2 k∈S b (β) w p 2 k x[k] p 2 p 1 to both sides, we have 2 k∈S b (β) w p 2 k x [k] p 2 p 1 < k / ∈S b (β) w p 2 k x [k] p 2 p 1 + k∈S b (β) w p 2 k x [k] p 2 p 1 = k w p 2 k x [k] p 2 p 1 , or equivalently, k∈S b (β) w p 2 k x [k] p 2 p 1 k w p 2 k x [k] p 2 p 1 < 1 2 .
But the above left-hand side expression is exactly Q w;p 1 ,p 2 (S b (β), Φ) and the inequality is the initial assumption of the proof, i.e., Q w;p 1 ,p 2 (S b (β), Φ)<1/2.

Similarly, for the case of equally-sized blocks, i.e., d 1 = • • • =d K =d, under the assumption of Q p 1 ,p 2 (S b (β), Φ)<1/2 and S b (β 0 )⊂S b (β), to show that β 0 is the unique solution to the P p 1 ,p 2 minimisation problem, we need to prove that:

∀x ∈ Ker(Φ), β 0 p 2 p 1 ,p 2 < β 0 + x p 2 p 1 ,p 2 ,
where as defined on page 34, the p 1 ,p 2 (pseudo-)mixed-norm of a vector β 0 is:

β 0 p 1 ,p 2 def =                  k I β 0 [k] p 1 , for p 2 = 0 k β 0 [k] p 2 p 1 1 p 2 , for 0 < p 2 < +∞ max k β 0 [k] p 1 , for p 2 = ∞,
Then, the above-mentioned inequality is equivalent to showing

K k=1 d k j=1 β 0 j [k] p 1 p 2 p 1 < K k=1 d k j=1 β 0 j [k] + x j [k] p 1 p 2 p 1
.

Then, dividing the whole blocks to on-Block-Support (∈S b (β)) and off-Block-Support ( / ∈S b (β)), we have

0 < k∈S b (β)    d k j=1 β 0 j [k] + x j [k] p 1 p 2 p 1 - d k j=1 β 0 j [k] p 1 p 2 p 1    + k / ∈S b (β) d k j=1 |x j [k]|
β 0 [k] + x [k] p 2 p 1 -β 0 [k] p 2 p 1 + k / ∈S b (β) x [k] p 2 p 1 .
Utilising a+b p 2 p 1 -a p 2 p 1 ≥-b p 2 p 1 for 0≤p 2 ≤1≤p 1 , it is sufficient to prove:

0 < k / ∈S b (β) x [k] p 2 p 1 - k∈S b (β) x [k] p 2 p 1 .
Adding 2 k∈S b (β) x[k] p 2 p 1 to both sides, we have

2 k∈S b (β) x [k] p 2 p 1 < k / ∈S b (β) x [k] p 2 p 1 + k∈S b (β) x [k] p 2 p 1 = k x [k] p 2 p 1 , or equivalently, k∈S b (β) x [k] p 2 p 1 k x [k] p 2 p 1 < 1 2 .
But the above left-hand side expression is exactly Q p 1 ,p 2 (S b (β), Φ) and the inequality is the initial assumption of the proof, i.e., Q p 1 ,p 2 (S b (β), Φ)<1/2.

β T 1 [k] Φ T 1 [k] Φ 2 k β 2 k ≤ K k,k =1 β 1 [k] p Φ T 1 [k] Φ 2 k β 2 k p ≤ d max M q,p (Φ 1 , Φ 2 ) K k,k =1 d 1 p k β 1 [k] p d -1 q k β 2 k q = d max M q,p (Φ 1 , Φ 2 ) K k=1 d 1 p k β 1 [k] p K k =1 d -1 q k β 2 k q = d max M q,p (Φ 1 , Φ 2 ) β1 r,0 k=1 d 1 p k β 1 [k] p β2 r,0 k =1 d -1 q k β 2 k q .
(A.10)

For vectors a and b we have a T b≤ i |a i b i |, and using Hölder's inequality [START_REF] Golub | Matrix Computations[END_REF], i.e., i |a i b i | ≤ a p b p , where, ∀(p, p )∈R 2 ≥1 : 1/p+1/p =1, we have a T b≤ a p b p , which results the above first inequality. Considering the definition of basic Block-MCC q,p , i.e., M q,p (Φ 1 , Φ

2 )= max k,k d -1/p k d 1/q k /d max Φ T 1 [k]Φ 2 [k ] q→p
, where p =p/(p-1), we have

Φ T 1 [k]Φ 2 [k ]x p ≤M q,p (Φ 1 , Φ 2 )d max d 1/p k d -1/q k
x q , which produces the above third line with second inequality. The above last equality follows from summation only over non-zero blocks for any r≥0.

Before finding the upper-bound for the above last inequality (A.10), it should be taken into account that according to the Parseval's theorem, we have y 2 2 = β 1 2 2 = β 2 2 2 =1, which will be used in the following optimisation problem. Therefore, in order to upper-bound the inequality (A.10), it is sufficient to solve the following optimisation problem: Although the equation (A.10) holds true for ∀q∈R >0 , we are going to upper-bound the (A.10) for ∀q∈R ≥1 , as it is mentioned in (A.11). On the other hand, since during the procedure of finding the upper-bound, the derivative of the q and p norms of a vector should be computed, we solve (A.11) in two parts: 1) ∀(q, p)∈R 2 >1 (differentiable q and p norms), and 2) q=p=1 (following the related proof in [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF], page 3).

∀(q, p, p ) ∈ R 3 ≥1 , ∀r ∈ R ≥0 , max k,k β 1 r,0 k=1 d 1 p k   d k j=1 β 1 j [k] p   1 p β 2 r,0 k =1 d -1 q k   d k j=1 β 2 j k q   1 q s.t.
Above optimisation problem is separable, so we need to maximise ∀(q, p)∈R 2 >1 ,

β
∂L ∂β 1 j [k] = d 1 p k β 1 j [k] β 1 j [k] p-2   d k j=1 β 1 j [k] p   1 p -1 -2λβ 1 j [k] = 0 ⇒ β 1 j [k] ∈        0,    2λ d 1 p k β 1 [k] 1-p p    1 p-2        .
From the above last equality it can be derived that all the absolute value of the coefficients in a block k has the same value. On the other hand, all the identical elements cannot be zero, because it leads to β 1 [k] r,0 =0, but from the last line of (A.10), only non-zero blocks are selected for optimisation. Therefore, it reduces to:

β 1 j [k] = 1 2λ .
Next, applying the unit-energy constraint of the coefficients, we get:

1 = β 1 r,0 k=1 d k j=1 β 2 1 j [k] = d k β 1 r,0 4λ 2 ⇒ λ = d 1 2 k β 1 1 2 r,0 2 ⇒ β 1 j [k] = d k β 1 r,0 -1 2 .
Therefore, max M q,p (Φ 1 , Φ 2 ) β 1 r,0 β 2 r,0

β
1 2 ≤ d -1 2 min d 3 2
max M q,p (Φ 1 , Φ 2 )

β 1 r,0 + β 2 r,0
2 .

The proof is completed in the above equation by replacing p by p/(p-1), according to the condition of the Hölder's inequality [START_REF] Golub | Matrix Computations[END_REF], and using the inequality of arithmetic-geometric means, i.e., √ ab≤(a+b)/2. Now, consider the case where q=p=1. Then, the optimisation problem in (A.11) becomes: The above optimisation problem is separable, so we need to maximise But, in order to solve these problems (following the related proof in [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF], page 3), let us consider the following Lagrangian function, in which the positivity constraint is not enforced explicitly:

∀r ∈ R ≥0 , max k,k β 1 r,0 k=1 d k j=1 β 1 j [k]
L (β 1 , λ) = β 1 r,0 k=1 d k j=1 β 1 j [k] + λ   1 - β 1 r,0 k=1 d k j=1 β 2 1 j [k]   ∂L ∂β 1 j [k] = 1 -2λβ 1 j [k] = 0 ⇒ β 1 j [k] = 1 2λ .
Next, applying the unit-energy constraint of the coefficients, we get:

1 = β 1 r,0 k=1 d k j=1 β 2 1 j [k] = d k β 1 r,0 4λ 2 ⇒ λ = d 1 2 k β 1 1 2 r,0 2 ⇒ β 1 j [k] = d k β 1 r,0 -1 2 .
It can be seen that, although the constraint of β 1 j [k]>0 is not explicitly included in the Lagrangian function, but from the final closed form solution this constraint can be verified. Therefore, min . Substituting the recently mentioned upper-bounds into (A.10), we get the same results as for the previous case of ∀(q, p)∈R 2 >1 . Hence, the Lemma holds true for ∀(q, p)∈R 2 ≥1 .

β 1 r,0 k=1 d k j=1 β 1 j [k]
A.9 Proof of Property 2.17 (Basic Block-MCC 2,2 lower-bound, page 72)

Proof. From Property 2.6 (Block-MCC q,p for intra-block orthonormality, page 58), in the special case of two matrices and parameter changing of p→p/(p-1), we have:

∀ (q, p) ∈ R 2 >0 , M q, p p-1 (Φ 1 , Φ 2 ) = max k,k d -p-1 p k d 1 q k d max Φ T 1 [k] Φ 2 k q→ p p-1
.

Then, squaring both sides and for q=p=2, we get:

M 2 2,2 (Φ 1 , Φ 2 ) = max k,k d -1 k d k d 2 max Φ T 1 [k] Φ 2 k 2 2→2 = max k,k d -1 k d k d 2 max Φ T 2 k Φ 1 [k] Φ T 1 [k] Φ 2 k 2→2 .
The above second equality follows from a property of operator-norms, i.e., A 2 2→2 = A T A 2→2 . Summing over k and k , we have:

K 2 M 2 2,2 (Φ 1 , Φ 2 ) ≥ K k=1 K k =1 d -1 k d k d 2 max Φ T 2 k Φ 1 [k] Φ T 1 [k] Φ 2 k 2→2 ≥ 1 d 2 max K k =1 d k Φ T 2 k K k=1 d -1 k Φ 1 [k] Φ T 1 [k] Φ 2 k 2→2 ≥ d min d 3 max K k =1 Φ T 2 k K k=1 Φ 1 [k] Φ T 1 [k] Φ 2 k 2→2 = d min d 3 max KI d k 2→2 = d min d 3 max K.
The above second inequality follows from the triangle inequality, whereas the third one results from considering the minimum values for variable block length coefficients in the sum operator. In fact, in this problem, having some finite integer values d k ≥1, ∀k, and the corresponding orthonormal bases Φ[k], we are utilising the lower-bound

k d k Φ[k]≥ min k d k k Φ[k]=d min k Φ[k].
The above first equality results from the fact that for orthonormal matrices Φ 1 and Φ 2 we have,

K k=1 Φ 1 [k]Φ T 1 [k]=Φ 1 Φ T 1 =I m and ∀k , Φ T 2 [k ]Φ 2 [k ]=I d k .
The last equality results from the homogeneity property (Property 2.4, page 56) and unity of operator-norm of the identity matrix. The proof is then completed by taking square roots from both sides and noticing that for each matrix Φ 1 and Φ 2 , we have m= • p ≥ • p , we have

k =k Φ † [k]Φ[k ]x[k ] p ≥ x[k] p .
On the other hand, from the definition of mutual coherence constant (Definition 2.6, page 54), i.e., ∀(q, p)∈R 2 >0 : M q,p (Φ)= max k,k =k

x[k ] =0 d -1/p k d 1/q k d -1 max Φ † [k]Φ[k ]x[k ] p / x[k ] q , we can
see that in order to compute M q,p (Φ), the value d

-1/p k d 1/q k d -1 max Φ † [k]Φ[k ]x[k ] p / x[
k ] q is calculated for each k and k =k and finally the maximum calculated value is considered as M q,p (Φ). Then for any k and k =k, the d

-1/p k d 1/q k d -1 max Φ † [k]Φ[k ]x[k ] p / x[
k ] q is upper-bounded by M q,p (Φ), and since d

1/p k d -1/q k d max x[k ] q is positive, we have Φ † [k]Φ[k ]x[k ] p ≤d 1/p k d -1/q k d max x[k ] q M q
,p (Φ). Hence, returning back to the proof, we have

k =k d 1/p k d -1/q k d max x[k ] q M q,p (Φ)≥ k =k Φ † [k]Φ[k ]x[k ] p ≥ x[k] p or d max M q,p (Φ) k =k d 1/p k d -1/q k x[k ] q ≥ x[k] p .
Then by rearranging the inequality we get

d max M q,p (Φ) k =k x[k ] q /d 1/q k ≥ x[k] p /d 1/p k . Adding d max M q,p (Φ) x[k] q /d 1/q k
to both sides, we have:

∀q ∈ R >0 , ∀p ∈ R ≥1 , d max M q,p (Φ) k x [k ] q d 1 q k ≥ d max M q,p (Φ) x [k] q d 1 q k + x [k] p d 1 p k ,
which using the definition of weighted (pseudo-)mixed-norm w p 1 ,p 2 (Definition 2.2, page 48), with w k =d -1/q k , p 1 =q, and p 2 =1, it is equivalent to ∀q ∈ R >0 , ∀p ∈ R ≥1 , d max M q,p (Φ) x w;q,1 ≥ d max M q,p (Φ)

x [k] q d 1 q k + x [k] p d 1 p k .
Now take any vector β with Block-Support S b (β) such that S b (β 0 )⊂S b (β), then summing over blocks k∈S b (β), and knowing that for any constants α and ∀r≥0, we have k∈S b (β) α=α β r,0 , we get:

∀q ∈ R >0 , ∀p ∈ R ≥1 , d max M q,p (Φ) x w;q,1 β r,0 ≥ k∈S b (β)   d max M q,p (Φ) x [k] q d 1 q k + x [k] p d 1 p k   = d max M q,p (Φ) k∈S b (β) x [k] q d 1 q k + k∈S b (β)
x

: ( x[k] p /d 1/p k )/( x[k] q /d 1/q k )≥ min{1, d 1/q-1/p k }.
Again using the lower-bound on the fraction of two sums of values explained in Property 2.3, we have ∀(q, p)∈R 2 >0 :

( k∈S b (β) x[k] p /d 1/p k )/( k∈S b (β) x[k] q /d 1/q k )≥ min k∈S b (β) min{1, d 1/q-1/p k },
which is a special case of Property 2.3. Hence, returning back to the proof, ∀q∈R >0 , ∀p∈R ≥1 , and ∀r∈R ≥0 we get:

d max M q,p (Φ) x w;q,1 β r,0 ≥ d max M q,p (Φ) The above last line results from min k∈S f (k)≥ min k f (k). Then, by dividing the both sides by x w;q,1 , ∀q∈R >0 , ∀p∈R ≥1 , and ∀r∈R ≥0 we have:

∀x ∈ Ker(Φ), d max M q,p (Φ) β r,0 ≥ d max M q,p (Φ) + min

k min 1, d 1 q -1 p k k∈S b (β) x [k] q d 1 q k
x w;q,1 , or ∀x ∈ Ker(Φ), d max M q,p (Φ) β r,0 d max M q,p (Φ) + min

k min 1, d 1 q -1 p k -1 ≥ k∈S b (β) x [k] q d 1 q k
x w;q,1 .

Since the above inequality holds true for ∀x∈Ker(Φ), it also holds for the maximiser of the right-hand side (because the left-hand side does not depend on x). But that maximiser gives us exactly Q w;p 1 ,p 2 (S b (β), Φ) by its definition: x w;q,1 = Q w;q,1 (S b (β) , Φ) .

d max M q,p (
where, ∀(q, p)∈R 2 ≥1 (constraint on q was imposed by the Block-NSP condition), ∀r∈R ≥0 , and w k =d -1/q k . The above last line is obtained considering the definition of Q w;p 1 ,p 2 (S b (β), Φ) in Block-NSP (Theorem 2.2, page 67). On the other hand, from the Block-NSP condition in Theorem 2.2, we have, if Q w;q,1 (S b (β), Φ)<1/2 then a solution β 0 to the problem P w;q,1 is the unique solution whenever S b (β 0 )⊂S b (β). Hence, we have that if: ∀(q, p) ∈ R 2 ≥1 , ∀r ∈ R ≥0 , d max M q,p (Φ) β r,0 d max M q,p (Φ) + min

k min 1, d 1 q -1 p k -1 < 1 2 ,
Then, the Block-NSP condition of Theorem 2.2 is met, hence any β 0 solution to y=Φβ is unique if S b (β 0 )⊂S b (β). Now, by rearranging the above inequality, we have ∀(q, p) ∈ R 2 ≥1 , ∀r ∈ R ≥0 , β r,0 < 1 + (d max M q,p (Φ)) -1 min k min 1,

d 1 q -1 p k 2 ,
and since S b (β 0 )⊂S b (β), then β 0 r,0 ≤ β r,0 , so ∀(q, p) ∈ R 2 ≥1 , ∀r ∈ R ≥0 , β 0 r,0 < 1 + (d max M q,p (Φ)) -1 min

k min 1, d 1 q -1 p k 2 ,
which is exactly the condition of the theorem. Therefore, the Block-NSP theory under the mentioned condition is actually met and the unique solution of the problem P w;q,1 is ensured.

1 q -1 p 2 = 1 + d 1+ 1 p -1 q M q,p (Φ) -1 2 = 1 + d 1+ 1 p -1 q max k,k =k d -1-1 p + 1 q Φ † [k] Φ k q→p -1 2 = 1 + max k,k =k Φ † [k] Φ k q→p -1
2 .

(A.12)

On the other hand, for q=p, and d 1 = • • • =d K =d, we get the following Block-SL q,p : ∀q ∈ R ≥1 , Block-SL q,q (Φ) = 1 + (d M q,q (Φ))

-1 2 = 1 + d max k,k =k d -1 Φ † [k] Φ k q→q -1 2 = 1 + max k,k =k Φ † [k] Φ k q→q -1
2 .

(A.13) By comparing the Block-SL q,p in (A.12) for p>q, with (A.13) for q=p, we see that the only difference between the equations is the q→p operator-norm. On the other hand, from figure 2.3, we know that for a fixed p by increasing q, and for a fixed q by decreasing p, the q→p operator-norm increases, hence, Block-SL q,p decreases, which results in the figure 2.5(b).

A.12. Proof of Property 2.19 (SL v.s. Block-SL q,p , page 74) 149

A.12 Proof of Property 2.19 (SL v.s. Block-SL q,p , page 74)

Proof. Starting from the relation between M (Φ) and M q,p (Φ) proposed in Property 2.8 (Block-MCC q,p bounds, page 59), the aim is to build the d max times of the numerator of the condition of the Block-ERC based on Block-MCC q,p in Theorem 2.3, i.e., d max +M -1 q,p (Φ) min k min{1, d 1/q-1/p k }, which is comparable with the conventional sparsity level. First, we investigate the required condition on M (Φ) for a dictionary with full column rank blocks. Using the first part of Property 2.8 (Block-MCC q,p bounds, page 59) with conditions M (Φ)<d 1/q-2 max (d max -1) -1/2 and q≥p≥1, and then multiplying by the positive coefficient min k min{1, d 1/q-1/p k } and summing to d max , for q≥p≥1 we get: But, ∀q≥p, we have min k min{1, d 1/q-1/p k }=d 1/q-1/p max , which leads to the following inequality: max (d max -1)

M -1 q,p (Φ) ≥ 1 -d 2-1 q max (d max -1) 1 2 M (Φ)
q ≥ p ≥ 1, d max + d 1 q -1 p max M -1 q,p (
1 2 + d 1 q -3 2 max M -1 (Φ) .
The above right-hand side expression under the following condition would be greater than or equal to 1 + M -1 (Φ):

d max -d 1 2
max (d max -1)

1 2 + d 1 q -3 2 max M -1 (Φ) ≥ 1 + M -1 (Φ) , ⇒ M (Φ) ≤ 1 -d 1 q -3 2 max (d max -1) 1 2 (d max -1) 1 2 -d 1 2 max .
Therefore, considering another condition of the utilised property, i.e., M (Φ)<d 1/q-2 max (d max -1) -1/2 in Property 2.8 (Block-MCC q,p bounds, page 59), the following upper-bound condition on M (Φ) ensures that the proposed Block-SL q,p would be greater than or equal to the conventional SL, i.e., d max +M -1 q,p (Φ) min k min{1, d 1/q-1/p k }≥1+M -1 (Φ):

q ≥ p ≥ 1, M (Φ) ≤ min        d 1 q -2 max (d max -1) 1 2 , 1 -d 1 q -3 2 max (d max -1) 1 2 (d max -1) 1 2 -d 1 2 max        .
But, the above second argument of the min operator ∀q, d max is negative. Therefore, although in practice the numerical simulations approve that the proposed sparsity level for general dictionaries with full column rank blocks is higher than the conventional one, in theory due to utilising the most pessimistic bounds, its supremacy cannot be proved. Now, suppose that the dictionary Φ has intra-block orthonormality, which is the case in the Property 2.19 (SL v.s. Block-SL q,p , page 74). Using the second part of Property 2.8 (Block-MCC q,p bounds, page 59) as the following first line, multiplying by min k min{1, d 1/q-1/p k } and summing to d max , ∀(q, p, q , p )∈R 4 >0 we get: The above right-hand side expression under the following condition and ∀(q, p, q , p )∈R 4 >0 would be greater than or equal to 1 + M -1 (Φ): A.13 Proof of Lemma 2.3 (Eldar et al.'s v.s. proposed Block-SL q,p , page 75)

M -1 q,p (Φ) ≥ dmax M -1 (Φ) max k,k =k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k , M -1 q,p (
d max + d max M -1 (Φ) max k,k =k d 1 2 -1 p k d 1 q + 1 2 k max 1, d 1 p -1 p k max 1, d 1 q -1 q k max 1, d 1 p -1 2 k max 1, d 1 2 -1 q k min k min 1, d 1 q -1 p k ≥ 1 + M -1 (Φ) ,
Proof. The proof is similar to the proof of Property 2.8 (Block-MCC q,p bounds). First, we investigate the required condition on M Eldar Intra (Φ) for a dictionary with full column rank blocks. Utilising the Block-MCC q,p proposed in Definition 2.6 (54), we have the following first line in (A.14). Using the pseudo-inverse property of full column rank matrices and submultiplicativity property of operator-norms introduced in Property 2.4 ( q→p operator-norm properties), and for d 1 = • • • =d K =d, ∀(q, p)∈R 2 >0 we have: Intra (Φ)<d 1/q-2 (d -1) -1/2 to meet the related condition), we have:

M q,p (Φ) = d 1 q -1 p -1 max k,k =k Φ † [k] Φ k q→p = d 1 q -1 p -1 max k,k =k Φ T [k] Φ [k] -1 Φ T [k] Φ k q→p ≤ d 1 q -1 p -1 max k,k =k Φ T [k] Φ [k]
∀k, q ≥ p ≥ 1, Φ T [k] Φ [k] -1 q→p ≤ 1 1 -d 2-1 q (d -1) 1 2 M Eldar Intra (Φ)
.

For q≥p≥1 (condition imposed by Property A.2), we have max{1, d 1/q-1/p }=1 in (A.14). Therefore, by substituting the upper-bound of (Φ T [k]Φ[k]) -1 q→p and the value of max{1, d 1/q-1/p } in (A.14), for q≥p≥1 we get:

M q,p (Φ) ≤ d 1 q -1 p -1 1 -d 2-1 q (d -1) 1 2 M Eldar Intra (Φ) max k,k =k Φ T [k] Φ k q→p ≤ d 1 q -1 p -1 max 1, d 1 p -1 2 max 1, d 1 2 -1 q 1 -d 2-1 q (d -1) 1 2 M Eldar Intra (Φ) max k,k =k Φ T [k] Φ k 2→2 .
The above last inequality is achieved based on the upper-bound of q→p operator-norm based on 2→2 operator-norm explained in Property A.1. Then by definition, substituting

max k,k =k Φ T [k]Φ[k ] 2→2 by d M Eldar
Inter (Φ) in the above inequality, for q≥p≥1 we have:

M q,p (Φ) ≤ d 1 q -1 p M Eldar Inter (Φ) max 1, d 1 p -1 2 max 1, d 1 2 -1 q 1 -d 2-1 q (d -1) 1 2 M Eldar Intra (Φ)
, or, A.13. Proof of Lemma 2.3 (Eldar et al.'s v.s. proposed Block-SL q,p , page 75)153

M -1 q,p (Φ) ≥ 1 -d 2-1 q (d -1)
(q, p) (1, 1) (1, 2) (1, ∞) (2, 2) (2, ∞) (∞, ∞)
M -1 q,p (Φ) min 1,d

1 q -1 p (M Eldar Inter (Φ)) -1 ≥ d -1 2 d -1 2 d -1 1 d -1 2 d -1 2
Table A.3: Lower-bound of M -1 q,p (Φ) min{1, d 1/q-1/p }/(M Eldar Inter (Φ)) -1 ensuring the supremacy of the proposed sparsity level when it is greater than one, for different basic values of (q, p) pairs and for a dictionary with intra-block orthonormality. relationship between ∀(q, p)∈R 2 ≥1 : M -1 q,p (Φ) min{1, d 1/q-1/p } and (M Eldar Inter (Φ)) -1 should be investigated. Starting from the relationship between M q,p (Φ) and M Eldar Inter (Φ) explained in Property 2.11, we build the required relationship ∀(q, p)∈R 2 >0 :

M -1 q,p (Φ) ≥ d 1 p -1 q M Eldar Inter (Φ) -1 max 1, d 1 p -1 2 max 1, d 1 2 -1 q , M -1 q,p (Φ) min 1, d 1 q -1 p ≥ d 1 p -1 q M Eldar Inter (Φ) -1 min 1, d 1 q -1 p max 1, d 1 p -1 2 max 1, d 1 2 -1 q , M -1 q,p (Φ) min 1, d 1 q -1 p M Eldar Inter (Φ) -1 ≥ min 1, d 1 p -1 q max 1, d 1 p -1 2 max 1, d 1 2 -1 q .
The numerator and denominator of the above left-hand side are the values that we need to compare with each other. To enhance Eldar's Block-ERC the above right-hand side should be greater than one. Those right-hand side values are shown in table A.3 for different basic values of (q, p) pairs. From table A.3, it can be seen that for q=p=2, M -1 q,p (Φ) min{1, d 1/q-1/p } is greater than or equal to (M Eldar Inter (Φ)) -1 , which is in fact just equal (not greater), because according to Property 2.11 for dictionaries with intra-block orthonormality, we have M 2,2 (Φ)=M Eldar Inter (Φ). Therefore, the proposed Block-ERC for q=p=2 as the best case equals to Eldar's Block-ERC.

Résumé -De nombreux phénomènes naturels sont trop complexes pour être pleinement reconnus par un seul instrument de mesure ou par une seule modalité. Par conséquent, le domaine de recherche de la multi-modalité a émergé pour mieux identifier les caractéristiques riches du phénomène naturel de la multi-propriété naturelle, en analysant conjointement les données collectées à partir d'uniques modalités, qui sont en quelque sorte complémentaires. Dans notre étude, le phénomène d'intérêt multi-propriétés est l'activité du cerveau humain et nous nous intéressons à mieux la localiser au moyen de ses propriétés électromagnétiques, mesurables de manière non invasive. En neurophysiologie, l'électroencéphalographie (EEG) et la magnétoencéphalographie (MEG) constituent un moyen courant de mesurer les propriétés électriques et magnétiques de l'activité cérébrale. Notre application dans le monde réel, à savoir le problème de reconstruction de source EEG / MEG, est un problème fondamental en neurosciences, allant des sciences cognitives à la neuropathologie en passant par la planification chirurgicale. Considérant que le problème de reconstruction de source EEG / MEG peut être reformulé en un système d'équations linéaires sous-déterminé, la solution (l'activité estimée de la source cérébrale) doit être suffisamment parcimonieuse pour pouvoir être récupérée de manière unique. La quantité de parcimonie est déterminée par les conditions dites de récupération. Cependant, dans les problèmes de grande dimension, les conditions de récupération conventionnelles sont extrêmement strictes. En regroupant les colonnes cohérentes d'un dictionnaire, on pourrait obtenir une structure plus incohérente. Cette stratégie a été proposée en tant que cadre d'identification de structure de bloc, ce qui aboutit à la segmentation automatique de l'espace source du cerveau, sans utiliser aucune information sur l'activité des sources du cerveau et les signaux EEG / MEG. En dépit du dictionnaire structuré en blocs moins cohérent qui en a résulté, la condition de récupération conventionnelle n'est plus en mesure de calculer la caractérisation de la cohérence. Afin de relever le défi mentionné, le cadre général des conditions de récupération exactes par bloc-parcimonie, comprenant trois conditions théoriques et une condition dépendante de l'algorithme, a été proposé. Enfin, nous avons étudié la multi-modalité EEG et MEG et montré qu'en combinant les deux modalités, des régions cérébrales plus raffinées sont apparues.

Mots clés : (q,p)-constante cohérence mutuelle par bloc, propriété null space par bloc, bloc-spark, conditions de reconstruction exacte par bloc-parcimonieux, Principe d'incertitude par bloc-parcimonieux, bloc-parcimonieux, identification de la structure de bloc, segmentation spatiale de l'activité cérébrale, (q,p)-constante cohérence mutuelle cumulée par bloc, multimodalité en EEG et MEG.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Multi-modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.3 Sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4 Block-sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 1.5 Block-sparsity and multi-modality . . . . . . . . . . . . . . . . . . . . . . . . . 42 2 Block-sparse exact recovery conditions in general dictionaries 45 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.2 Block-sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.3 Block-sparse exact recovery condition . . . . . . . . . . . . . . . . . . . . . . . . 65 2.4 Numerical experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3 A general framework for block structure identification 85 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.2 Block structure identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 iii iv Contents 3.3 Block-ERC in clustered representation . . . . . . . . . . . . . . . . . . . . . . . 89 3.4 EEG/MEG source reconstruction problem and USLE . . . . . . . . . . . . . . . 91 3.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4 EEG and MEG multi-modality 103 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.2 Multi-modal lead-field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Contributions and perspectives 115 High-dimensional problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 Block-structured problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Multi-modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 List of Figures 1.1 Mono-modal dataset cannot describe the multispectral phenomenon of interest, due to the monospectral detectability of each modality. While by combining mono-modal datasets, an acceptable description of the phenomenon can be recovered. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2 Electromagnetic properties of brain activity is measured by EEG and MEG modalities and can be used to reconstruct the brain activity. Source reconstruction using EEG and MEG multi-modality leads to improved results. 7 1.3 Action current leads to releasing neurotransmitters into the synapse and producing post synaptic current inside the cell. A current dipole is a representation of thousands of simultaneously firing neighbouring neurons and can have different orientations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.4 The forward and inverse problems in EEG/MEG source analysis. . . . . . . . . 9 1.5 Three-layer (scalp, skull, and brain) head models modelled(a) analytically, and (b,c) numerically. In (b), the cortical sheet is inflated, whereas in (c) it is highly-folded. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.6 Electromagnetic fields (E, B) at distance r of dipole p. . . . . . . . . . . . . . 11

  =3.5 and ε intra =0.1 for (a) N =4 and (b) N =8 in the dictionary Φ∈R 10×80 . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10 Different geometrical models of the EEG/MEG experiment: 1-Spherical (a,d), realistic inflated (b,e), and realistic highly-folded (c,f) source models; 2-EEG (a,b,c), and MEG (d,e,f) sensor models; 3-Three-layer concentric spheres (a), realistic three-layer (b,c), single sphere (d), and realistic single layer (e,f) volume conduction head models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11 Relative increase of sparsity levels SL min , SL max , when clustering the coherent blocks of lead-field, repeated for two modalities of EEG and MEG, and three head models with spherical, realistic inflated and realistic highly-folded cortical sheets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4. 6

 6 Clustering tree using EEG or MEG mono-modal lead-field and EEG+MEG multi-modal lead-field for (a) spherical, (b) realistic inflated, and (c) realistic highly-folded cortical sheets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Number of clusters using EEG or MEG mono-modal lead-field and EEG+MEG multi-modal lead-field for (a) spherical, (b) realistic inflated, and (c) realistic highly-folded cortical sheets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.8 At each level of clustering, the area of brain regions are shown in pie and bar charts for EEG, MEG and their combination. At the top row, the descendingly sorted brain regions are shown for each one of three types of lead-fields. The hierarchical clustering method is average, and BMIC 1,∞ is used as coherency measure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 1 . 2 :

 12 Figure 1.2: Electromagnetic properties of brain activity is measured by EEG and MEG modalities and can be used to reconstruct the brain activity. Source reconstruction using EEG and MEG multi-modality leads to improved results.

Figure 1 . 3 :

 13 Figure 1.3: Action current leads to releasing neurotransmitters into the synapse and producing post synaptic current inside the cell. A current dipole is a representation of thousands of simultaneously firing neighbouring neurons and can have different orientations.

Figure 1 . 6 :

 16 Figure 1.6: Electromagnetic fields (E, B) at distance r of dipole p.

Figure 1

 1 Figure 1.8: (a) MEG measures vectorial magnetic field, whereas EEG measures scalar potential differences, (b) MEG sensors are located outside the head, whereas EEG sensors are placed on the surface of the head, (c) MEG is mostly sensitive to primary currents, whereas EEG is sensitive to secondary currents, (d) MEG has very low sensitivity to radial dipoles, (e) MEG has low sensitivity to deep sources, (f) EEG is highly sensitive to the geometry and conductivity of media, and (g) EEG and MEG topographies for a same source activity are almost orthogonal to each other. .

Figure 1 .

 1 Figure 1.10: Block-sparsity structure of the exactly recovered representation vector β in the noiseless linear model of y=Φβ 0 .

  .20) In a similar work, Ziaei et al. proved nearly the same Block-ERC of (1.20), for the different block size setting, where, all the d-s in (1.20) and M Eldar Inter (Φ) have been replaced by max i,j =i d i d j [Zia+10]. Ziaei et al. proved that their proposed modification of the 2 / 1 -optimisation program for different-size blocks outperforms the 2 / 1 -optimisation program [EM09] and basis pursuit [CDS01], in terms of the error rate [Zia+10].

Figure 1 .

 1 Figure 1.11: EEG and MEG source reconstruction problem using the measurements stored in y EEG and y MEG vectors, and their corresponding lead-field matrices, leads to infinitely many solutions of β 1 , • • • , β ∞ , while by imposing anatomical and block-sparsity constraints under recovery conditions, a unique solution β 0 can be recovered.

  with ϕ j [k]∈R m , and without loss of generality, it is assumed that ϕ j [k] has unit Euclidean norm, i.e., ∀j, k : ϕ j [k] 2 =1. The block structure of Φ and β is shown schematically in figure 2.1. The matrix multiplication can be decomposed as sum of multiplication of blocks to reconstruct the measurement vector y∈span{Φ}∈R m . Therefore, as the model we use the noiseless USLE explained in (1.2), which can be viewed as the block-wise structure, i.e., y=Φ β= K k=1 Φ[k] β[k].

Figure 2

 2 Figure 2.1: Block structure of the dictionary Φ and the representation vector β in the noiseless linear model, i.e., y=Φ β= K k=1 Φ[k] β[k].

Figure 2

 2 Figure 2.2: Sparsity and block sparsity regions in terms of β p 1 ,p 2 , and β w;p 1 ,p 2 , assuming ∀k, d k =d. The most commonly studied cases are indicated with solid circles.

  Property 2.1 (Block-Spark v.s. Spark). Let d k be the block length of the k th block in Φ, and denoting d= k∈S b (x b ) d k / |S b (x b )| the average block length, with x b = arg min x∈Ker(Φ)\{0} x p,0 , ∀p≥0, then we have: d Block-Spark(Φ) ≥ Spark(Φ). Proof. Let x ∈ arg min x∈Ker(Φ)\{0} x 0 and x b ∈ arg min x∈Ker(Φ)\{0} x p,0 . Obviously, x 0 ≤ x b 0 , and x b 0 ≤ k∈S b (x b ) d k , indeed the whole block is activated even if only a single element is needed. Now letting d= |S b (x b )| -1 k∈S b (x b ) d k = x b -1 p,0 k∈S b (x b ) d k , the transitivity of the inequalities yields x 0 ≤d x b p,0 , which is exactly d Block-Spark(Φ)≥Spark(Φ).

Remark 2. 8 .

 8 Although the Block-ERC based on Block-Spark in Theorem 2.1 (Block-ERC based on Block-Spark) improves the conventional condition of Gorodnitsky and Rao [GR97], Donoho et al. [DS89]; [DH01]; [DE03b]; [DE03a], Elad and Bruckstein [EB01]; [EB02],

  By refining and generalising the ideas from Donoho et al. [DH01]; [DE03b], Elad and Bruckstein [EB01]; [EB02], Gribonval and Nielsen [GN03a]; [GN03b]; [GN04]; [GN07], Feuer and Nemirovski [FN03a], Zhang [Zha05], Cohen et al. [CDD09], Stojnic et al. [SXH08]; [SPH09] and Boufounos et al. [BKR11], we propose the following theorem, called Block Null Space Property (Block-NSP).

  represented schematically in figure 2.5(b).

Figure 2 . 5 :

 25 Figure 2.5: Block-SL q,p inequalities for (a) differently-sized, and (b) equally-sized block structure, for common q→p operator-norms according to table 2.1 (page 54).

  Finally, for comparison with the existing results, the sparsity level in the sparsity domain introduced by Donoho et al. (equation (1.13), page 29), i.e., SL Donoho =(1+M -1 (Φ))/2, and in the block-sparsity domain introduced by Eldar et al. (equation (1.20), page 40), i.e., SL Eldar =(d+(M Eldar Inter (Φ)) -1 (1-

  al. and Donoho and his co-workers. The result of the experiment is shown in figure 2.6 for (a) d=2 and (b) d=4. The first six bars in figure 2.6(a) and (b) correspond to the proposed sparsity levels for random dictionaries without (SL(Φ), orange colour) and with (SL(Φ ort ), blue colour) intra-block orthonormality. Sparsity levels of Donoho et al. for the mentioned two types of dictionary are equal. As it can be seen in figure 2.6(a) and (b), all the proposed sparsity levels based on Block-MCC q,p for random dictionaries Φ without intra-block orthonormality are higher than the conventional sparsity levels introduced by Eldar et al. and Donoho et al. (orange colour), whereas for random dictionaries Φ ort with intra-block orthonormality (blue colour), the highest value corresponds to Block-MCC 2,2 , which is equal to the sparsity level of Eldar et al., as proved theoretically in Lemma 2.3 (page 75), and explained in Remark 2.15. At last, all the proposed sparsity levels for two types of dictionaries are higher than the conventional sparsity level of Donoho et al. as proved theoretically in Property 2.19 (page 74), and explained in Remark 2.13.

Figure 2 . 6 :

 26 Figure 2.6: Sparsity levels for six tractable Block-MCC q,p , for dictionaries without (SL(Φ)) and with (SL(Φ ort )) intra-block orthonormality compared to two conventional sparsity levels for (a) d=2, and (b) d=4.

For a general

  dictionary with full column rank blocks, sparsity level of Eldar et al. is not computable for all values of d, because from Block-ERC of Eldar et al., i.e., β 0 2,0 <(1+(dM Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)))/2, (equation (1.20), page 40), the inequality 1-(d-1)M Eldar Intra (Φ)+d M Eldar Inter (Φ)>0 should hold true in order to have a positive sparsity levels. At last, for d=1, the SL in block domain are equal to the scalar domain.

Figure 2 . 7 :

 27 Figure 2.7: Sparsity levels for different values of block length d using six Block-MCC q,p , compared to two conventional sparsity levels proposed by Donoho et al. and Eldar and her co-workers.

  et al. (Φ, hollow green circles) and Eldar et al. (Φ, hollow upward blue triangles) for all values of d.Except for the results of Donoho et al., the sparsity levels for random dictionaries with intra-block orthonormality are higher than for random dictionaries without intra-block orthonormality (with full column rank blocks). Sparsity level of Donoho et al. is invariable to two types of dictionaries Φ and Φ ort , and also to changes in d (the corresponding two set of markers are superimposed). At last, for d=1, the sparsity levels in block domain are equal to the scalar domain.

Figure 2 . 8 :

 28 Figure 2.8: Sparsity levels for different values of block length d for two types of random dictionaries without (Φ) and with (Φ ort ) intra-block orthonormality, compared to two conventional sparsity levels.

Contents 3. 1

 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.2 Block structure identification . . . . . . . . . . . . . . . . . . . . . . . . 86 3.2.1 Clustering coherent blocks of a general dictionary . . . . . . . . . . . . . . 88 3.3 Block-ERC in clustered representation . . . . . . . . . . . . . . . . . . 89 3.3.1 Sparsity levels in differently-sized block structure of a dictionary . . . . . 89 3.4 EEG/MEG source reconstruction problem and USLE . . . . . . . . . 91 3.4.1 EEG/MEG source space segmentation . . . . . . . . . . . . . . . . . . . . 93 3.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.5.1 Synthetic dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.5.2 Real EEG/MEG lead-field . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

  The clustering structure corresponding to three clusters is P={s 1 , s 2 , [s 3 , s 4 ]}, where, s 3 and s 4 are clustered to form a new cluster, while s 1 and s 2 are single-element clusters. Therefore, d=[3, 3, 6], because in EEG/MEG source reconstruction problem each source's lead-field corresponds to three columns in the whole lead-field matrix. The partitioning P of sources s 1 , • • • , s 4 , indicates to the brain source space segmentation, as represented graphically in figure3.1 (left part), and will be explained in detail in Section 3.4.

Figure 3 . 1 :

 31 Figure 3.1: Clustering the coherent blocks using Block-MCC q,p determines the block structure, while improves Block-ERC and segments brain sources.

  .2, after blocks clustering, blocks Φ [1], Φ [4], and Φ [6] are concatenated to form a new block Φ cls [1] in new representation, whereas Φ [2] and Φ [3] are concatenated to form Φ cls [2]. Hence, the identified block structure is d=[6, 4, 2].Consequently, their corresponding blocks in the representation vector β will be clustered. The clustered representation of the new model, i.e., y=Φ cls β cls , leads to more relaxed Block-ERC, i.e., higher sparsity levels, which will be demonstrated in Section 3.5.

Figure 3

 3 Figure 3.2: Clustered representation of general dictionary Φ and representation vector β.

3. 3 .

 3 Block-ERC in clustered representation 89

  can be assumed a range of sparsity levels between SL min (Φ)= Block-SLq,p(Φ) k=1 d (k) and SL max (Φ)= K k=K-Block-SLq,p(Φ)+1 d (k) , where, d (k) are elements of the ascendingly-sorted d, i.e., d (1) ≤ • • • ≤d (K) . In other words, SL min and SL max are sum of the Block-SL q,p minimum and maximum elements of d, respectively, as shown in figure 3.3. Remark 3.1. For d 1 = • • • =d K , the upper-bound and lower-bound of the range of sparsity level are equal, i.e., SL max (Φ)≡SL min (Φ).

Figure 3 . 3 :

 33 Figure 3.3: The computation of SL min and SL max from ascendingly-sorted d.

  Block-SLq,p(Φ) k=1 d (k) . * Block-SL min : Number of equally-sized blocks of size d resulted from SL min (Φ)/d. -SL max : Number of columns resulted from K k=K-Block-SLq,p(Φ)+1 d (k) . * Block-SL max : Number of equally-sized blocks of size d resulted from SL max (Φ)/d.

Figure 3

 3 Figure 3.4: A simple EEG (left) and MEG (right) linear models for four brain sources s 1 , • • • , s 4 , and three electric and magnetic sensors, respectively. A 3D dipole located in s 4 is shown and represented mathematically.

3. 4 .

 4 EEG/MEG source reconstruction problem and USLE 93

  As represented graphically in figure 3.5, the six sources in the brain are clustered in different three segments based on the clustering of their corresponding lead-fields in the lead-field matrix. In other words, clustering of the coherent lead-field matrices Φ[1], Φ[4], and Φ[6] leads to clustering of their corresponding sources β[1], β[2], and β[3], and so on.

Figure 3

 3 Figure 3.5: Segmenting the source space of brain using clustering the corresponding lead-fields.

Figure 3

 3 Figure 3.6: (a) The longest distance between two points in the two clusters determines inter-cluster distance; (b) Number of vertical branches at a clustering level with longest branch approximates the number of clusters.

Figure 3 . 7 :

 37 Figure 3.7: From initial block µ 0 , N representative blocks of clusters are computed. Then, from each representative block µ i , all blocks Φ i [j] belonging to the same cluster i are generated.

Figure 3 .

 3 Figure 3.10: Different geometrical models of the EEG/MEG experiment: 1-Spherical (a,d), realistic inflated (b,e), and realistic highly-folded (c,f) source models; 2-EEG (a,b,c), and MEG (d,e,f) sensor models; 3-Three-layer concentric spheres (a), realistic three-layer (b,c), single sphere (d), and realistic single layer (e,f) volume conduction head models.

Figure 3 .

 3 Figure 3.13: The proposed subject-specific brain source space segmentation, using (a) EEG, and (b) MEG lead-fields, compared to (c) the conventional anatomical and functional brain lobes.

Contents 4 . 1

 41 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.2 Multi-modal lead-field . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.2.1 Structure and model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.3.1 Synthetic dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.3.2 Real EEG and MEG lead-field . . . . . . . . . . . . . . . . . . . . . . . . 109 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

  .1, number of multi-modal EMEG sensors in figure 4.1(c) is equal to the number of mono-modal EEG and MEG sensors in figure 4.1(a) and (b), which is equal to the number of rows of the lead-field matrices, i.e. m. As mentioned before, the MEG sensors in figure 4.1 are the closest sensors to EEG sensors, e.g., MEG sensor #1 is the closest MEG sensor to EEG sensor #1, and so on. The minimal change in sensors position in figure 4.1(c) in comparison to figure 4.1(a) and (b), also the uniform distribution of sensors, would be more evident by increasing the density of sensors.

Figure 4

 4 Figure 4.1: The rows of (a) EEG lead-field Φ EEG and (b) MEG lead-field Φ MEG can be combined together to form (c) the new multi-modal lead-field Φ EMEG .

Figure 4 . 3 :

 43 Figure 4.3: Block-SL 2,2 (Φ)[%] for each level of clustering is computed for the complete method, Block-MCC 2,2 , d=2, N ={4, 5}, ε inter-dictionary =1, and different values of ε inter and ε intra for simulating dictionary Φ∈R 10×80 .

Figure 4

 4 Figure 4.4: (a) spherical, (b) realistic inflated and (c) realistic highly-folded cortical sheets for the head models used in EEG and MEG multi-modality experiment.

Figure 4

 4 Figure 4.5: EEG and MEG complementarity in brain source segmentation using Block-MCC 2,2 and complete method for some clustering levels and for (a) spherical, (b) realistic inflated, and (c) realistic highly-folded cortical sheets.

Figure 4 . 6 :

 46 Figure 4.6: Clustering tree using EEG or MEG mono-modal lead-field and EEG+MEG multi-modal lead-field for (a) spherical, (b) realistic inflated, and (c) realistic highly-folded cortical sheets.

Figure 4 . 7 :

 47 Figure 4.7: Number of clusters using EEG or MEG mono-modal lead-field and EEG+MEG multi-modal lead-field for (a) spherical, (b) realistic inflated, and (c) realistic highly-folded cortical sheets.

Figure 4 . 8 :

 48 Figure 4.8: At each level of clustering, the area of brain regions are shown in pie and bar charts for EEG, MEG and their combination. At the top row, the descendingly sorted brain regions are shown for each one of three types of lead-fields. The hierarchical clustering method is average, and BMIC 1,∞ is used as coherency measure.

k

  ( d k j=1 |β 1 j [k]| p ) 1/p subject to |β 2 j [k ]| q ) 1/q subject to β 2 r,0 k =1 d k j=1 β 2 2 j [k ]=1, separately. In order to solve two mentioned problems, first we need to form the Lagrangian function. Then, for the first problem we have:L (β 1 , λ) = to compute its critical point. Considering that (|f |) =f f /|f |,where f is derivative of f with respect to x, i.e. d f (x)/d x, we have:

k--

  ( d k j=1 |β 1 j [k]| p ) 1/p is upper-bounded by β 1 |β 2 j [k ]| q ) 1/qis upper-bounded by β 2 the recently mentioned upper-bounds into (A.10), we get:∀(q, p) ∈ R 2 >1 , ∀r ∈ R ≥0 ,

  j [k] , β 2 j k > 0.

  k ]=1, β 2 j [k ]>0, separately.

-

  is upper-bounded by β 1 -1/2 r,0β 1 r,0 k=1 d 1/2 k ,which is again upper-bounded by β 1 1/2 r,0 d 1/2 max . Similarly, for the second problem it can be proved that β 2 j [k ] is upper-bounded by β 2

  K k=1 d k ≥ min k d k K k=1 1=K d min , then K≤m/d min , i.e.,:M 2,2 (Φ 1 , Φ 2 ) . Proof of Theorem 2.3 (Block-ERC based on Block-MCC q,p , page 73) 145A.10 Proof of Theorem 2.3 (Block-ERC based onBlock-MCC q,p , page 73)Proof. Because x∈Ker(Φ), we have k Φ[k ]x[k ]=0.Hence, for all k,k =k Φ[k ]x[k ]=Φ[k]x[k]. Therefore, since for all k the block Φ[k] is full column rank, we havek =k Φ † [k]Φ[k ]x[k ]=x[k].Applying • p to both sides and using the triangular inequality for p≥1 and p=0,

=

  d max M q,p (Φ) + min k∈S b (β)

  Φ) β r,0 d max M q,p (Φ) + min k

Φ

  T [k] Φ k q→p max 1, d 1 q -1 p . (A.14)Similar to the proof of the Property 2.8 in finding upper-bound of(Φ T [k]Φ[k]) F i,j [k]≤M Eldar Intra (Φ) def = max i,j =i k |ϕ T i [k]ϕ j [k]|≤M (Φ)), and using Property A.2 (q≥p≥1, MEldar

  g) EEG and MEG topographies for a same source activity are almost orthogonal to each other. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.9 Sparsity structure of the noiseless USLE. . . . . . . . . . . . . . . . . . . . . . . 16 EEG and MEG source reconstruction problem using the measurements stored in y EEG and y MEG vectors, and their corresponding lead-field matrices, leads to infinitely many solutions of β 1 , • • • , β ∞ , while by imposing anatomical and block-sparsity constraints under recovery conditions, a unique solution β 0 can be recovered. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 2.1 Block structure of the dictionary Φ and the representation vector β in the noiseless linear model, i.e., y=Φ β= K k=1 Φ[k] β[k]. . . . . . . . . . . . . . . . . 47 2.2 Sparsity and block sparsity regions in terms of β p 1 ,p 2 , and β w;p 1 ,p 2 , assuming ∀k, d k =d. The most commonly studied cases are indicated with solid circles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Sparsity level as a function of d for different values of q and p, averaged over 100 realisations of random dictionaries, with m=40 and n=400. . . . . . . . . . 83 3.1 Clustering the coherent blocks using Block-MCC q,p determines the block structure, while improves Block-ERC and segments brain sources. . . . . . . . . 87 3.2 Clustered representation of general dictionary Φ and representation vector β. . 88 3.3 The computation of SL min and SL max from ascendingly-sorted d. . . . . . . . . 89 3.4 A simple EEG (left) and MEG (right) linear models for four brain sources s Segmenting the source space of brain using clustering the corresponding lead-fields. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 (a) The longest distance between two points in the two clusters determines inter-cluster distance; (b) Number of vertical branches at a clustering level with longest branch approximates the number of clusters. . . . . . . . . . . . . 3.7 From initial block µ 0 , N representative blocks of clusters are computed. Then, from each representative block µ i , all blocks Φ i [j] belonging to the same cluster i are generated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Block-SL 2,2 (Φ)[%] for each level of clustering computed for complete method, Block-MCC 2,2 , d=2, N ={4, 8}, and different values of ε inter and ε intra for simulating dictionary Φ∈R 10×80 . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 SL min [%] and SL max [%] for each level of clustering computed for complete method, Block-MCC 2,2 , d=2, ε inter

	List of Figures	ix
	3.5

1.10 Block-sparsity structure of the exactly recovered representation vector β in the noiseless linear model of y=Φβ 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 vii viii List of Figures 1.11 2.3 q→p operator-norm inequalities for common q→p operator-norms according to table 2.1 (page 54). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 2.4 Block-MCC q,p inequalities for common q→p operator-norms according to table 2.1 (page 54). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 2.5 Block-SL q,p inequalities for (a) differently-sized, and (b) equally-sized block structure, for common q→p operator-norms according to table 2.1 (page 54). . 73 2.6 Sparsity levels for six tractable Block-MCC q,p , for dictionaries without (SL(Φ)) and with (SL(Φ ort )) intra-block orthonormality compared to two conventional sparsity levels for (a) d=2, and (b) d=4. . . . . . . . . . . . . . . . . . . . . . . 78 2.7 Sparsity levels for different values of block length d using six Block-MCC q,p , compared to two conventional sparsity levels proposed by Donoho et al. and Eldar and her co-workers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 2.8 Sparsity levels for different values of block length d for two types of random dictionaries without (Φ) and with (Φ ort ) intra-block orthonormality, compared to two conventional sparsity levels. . . . . . . . . . . . . . . . . . . . . . . . . . 80 2.9 1 , • • • , s 4 , and three electric and magnetic sensors, respectively. A 3D dipole located in s 4 is shown and represented mathematically. . . . . . . . . . . . . . . 92

Table 1 :

 1 The mathematical notations used throughout the manuscript. Inner product of vectors a and b, i.e., a T b P p,q,ε (a, b) min a a p s.t. b q ≤ε P p min β β p s.t. y=Φβ a p 1 ,p 2 p 1 ,p 2 (pseudo-)mixed-norm of vector a P (p 1 ,p 2 ),q,ε (a, b) min a a p 1 ,p 2 s.t. b q ≤ε P p 1 ,p 2 min β β p 1 ,p 2 s.t. y=Φβ a w;p 1 ,p 2

	Nomenclature The nomenclature used throughout the manuscript can be divided into three
	categories as shown in table 2.
	xv

w p 1 ,p 2 weighted (pseudo-)mixed-norm of vector a P (w;p 1 ,p 2 ),q,ε (a, b) min a a w;p 1 ,p 2 s.t. b q ≤ε P w;p 1 ,p 2

  Contents 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2 Multi-modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6 1.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 Physiological basis of EEG and MEG . . . . . . . . . . . . . . . . . . . . 8 1.2.3 EEG/MEG forward and inverse problems . . . . . . . . . . . . . . . . . . 9 1.2.4 Volume conduction head model . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.5 Lead-field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.2.6 EEG and MEG complementarities . . . . . . . . . . . . . . . . . . . . . . 13 1.3 Sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.2 Sparse representation theory . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.3 Exact recovery condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4 Block-sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 1.4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 1.4.2 Block-sparse representation theory . . . . . . . . . . . . . . . . . . . . . . 33 1.4.3 Block-sparse exact recovery condition . . . . . . . . . . . . . . . . . . . . 38 1.5 Block-sparsity and multi-modality . . . . . . . . . . . . . . . . . . . . . 42 1.5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 1.5.2 Distributed EEG and MEG source reconstruction problem . . . . . . . . . 43

  , scientists and engineers end up with vastly USLE, which have an infinite number of solutions, if any. For instance, the research domain of inverse problem mainly deals with USLE.

	1.3. Sparsity
	1.3 Sparsity
	1.3.1 Introduction
	In science and technology, e.g., statistics [CT07], or biomedical engineering [GH04]; [ES05];
	[LDP07]

•

  For equally-sized blocks, i.e., ∀k, d k =d, we propose Block-ERC based on P p 1 ,p 2 , where, p 1 can have zero value, while for a more general case of differently-sized blocks we propose Block-ERC based on P w;p 1 ,p 2 for p 1 =0, to have finite values for the elements of the weight vector w, because w k =d

-1/p 1 k .

  Maximum 2 norm of a column Maximum absolute entry of matrix

	q = 2	NP-hard	Maximum singular value	Maximum 2 norm of a row
	q = ∞	NP-hard	NP-hard	Maximum 1 norm of a row

Table 2.1: Computational complexity of q→p operator-norm for different basic (q, p) pairs [Tro04b]. Property 2.2 include the following special cases ∀a∈R d [GL13]:

Table 2 .

 2 1 2: Properties of q→p operator-norm for different ranges of p, while ∀q∈R ≥0 , where

	q→p properties N, P, T	N, P, H	N, P, H, T

N, P, H, and T stand for the existence of nonnegativity, positivity, homogeneity, and triangle inequality properties, respectively.

Table 2 .

 2 3: Inequalities of basic tractable q→p operator-norms based on table 2.1 (page 54) for a matrix A∈R m×n .

  explained on page 28 [DH01], in sparsity, and block-coherence of Eldar et al., i.e., M Eldar Inter (Φ)

  Lemma 2.1 (Block-UP 4 based on Block-Spark). For any arbitrary non-zero signal y with two distinct representations β 0 and β 1 in any general dictionary Φ, i.e., y=Φβ 0 =Φβ 1 , we have: ∀p ≥ 0,

	; [DS89];
	[DH01]; [EB01]; [EB02]; [DE03b]; [DE03a]; [GN03b]; [GN03a]; [BDE09], we propose the
	following lemma which is called block-sparse uncertainty principle based on Block-Spark :

  Lemma 2.2 (Basic Block-UP 5 ). Supposing β 1 and β 2 are two distinct block-structured representations of the non-zero signal y in two orthonormal m×m matrices Φ 1 and Φ 2 with similar block structure (same number of blocks and blocks' length vector d), respectively, i.e.,:

	Eldar et al. extended the basic uncertainty principle defined in (1.11) on page 28 to
	block-sparse domain defined in (1.19) on page 39 [EB09]; [EKB10b]; [EKB10a]. Now we
	generalise the results of Eldar et al., which will be also considered as a generalisation of the
	basic uncertainty principle [DS89]; [DH01]; [EB01]; [EB02] and name it basic block-sparse
	uncertainty principle.

  al. in (1.19) on page 39. As mentioned before, we had lower-bound for the basic MCC, i.e., M (Φ 1 , Φ 2 )≥1/ √ m (page 22), and for basic block-coherence of Eldar et al., i.e., M Eldar Inter (Φ 1 , Φ 2 )≥1/

	√	d m (page
	40)

  [k] p

													1	.
												d	p k
	On	the	other	hand	from	the	proof	of	Property	2.3	(bounds	of
	two	(pseudo-)mixed-norms	division,	page	55),	we	have	∀(q, p)∈R 2 >0

Underdetermined System(s) of Linear Equations

Magnetic Resonance Imaging1.2. Multi-modality

Least Absolute Shrinkage and Selection Operator

Non-deterministic Polynomial-hard

Orthogonal Matching Pursuit

Orthogonal Matching Pursuit

Basis Pursuit

Iterative Hard Thresholding

Non-deterministic Polynomial-hard

Least Absolute Shrinkage and Selection Operator

Compressive Sampling Matched Pursuit

Magnetic Resonance Imaging

Underdetermined System(s) of Linear Equations

Block-sparse Exact Recovery Condition(s)

Block-sparse Uncertainty Principle

Block-sparse Uncertainty Principle

Sparsity Level

(q, p)-Block-Sparsity Level

ElectroEncephaloGraphy

MagnetoEncephaloGraphy

Underdetermined System(s) of Linear Equations

(2, 2)-Block Mutual Coherence Constant

(2, 2)-Block-Sparsity Level

Block-sparse Exact Recovery Condition(s)

Now let us consider the assumption of the lemma, i.e., intra-block orthonormality. By definition, M Eldar Intra (Φ) is equal to zero in this case. Therefore, according to Theorem 2.3, the

Acknowledgements I would like to express my sincere gratitude to my advisors, Prof. Christian JUTTEN and Ronald PHLYPO, for their continuous support of my Ph.D research, who their door of office was always open whenever I had a question about my research, in addition to their guidances in our regular meetings, which steered me in the right research direction. This work would have been impossible without the financial support of Prof. Christian JUTTEN as the principal investigator of the granted fund of the European Research Council (ERC) for the research project of the Challenges in the Extraction and Separation of Sources (CHESS).

I am also grateful to each of the jury members, Prof. Maureen CLERC, Charles SOUSSEN,

Contents

The contributions of this thesis can be categorized into three main research domains. The three classes of research contributions address the three challenges that have been mentioned and discussed in Section Introduction of the dissertation.

In order to explore and address the three mentioned challenges in Introduction, this dissertation is organized in three main parts. Since the first challenge uses the results of the second challenge, the second challenge is presented in the dissertation before the first challenge, and ultimately in the last part a partial answer to the third challenge is provided through experiments.

First challenge is about the ineffectiveness of some classic methods in high-dimensional problems. Second challenge raises when classic recovery conditions cannot be established for a new concept of constraint of corresponding optimisation problem, because of materials shortage in new framework. Finally, the combination of different information of a same phenomenon is the subject of the third challenge.

In this section, first we briefly recall the challenges and then mention the corresponding contributions to deal with them.

As mentioned in Section Introduction, all the three independent research orientations meet each other in our real-world application, which is distributed EEG/MEG source reconstruction problem, to partially respond to the main research question of the thesis, which is: What is the added value of multi-modality, when solving inverse problems?

In the following the contributions and perspectives are organized into three sections: 115

Contributions and perspectives

Multi-modality

Challenge 3. Is joining multiple modalities always beneficial, knowing that each modality provides us with different properties of the same phenomenon? How can the added value of multi-modality be demonstrated?

In order to approach the above-mentioned important and huge challenge, we simplify and limit the challenge to the conditions of our main problem, i.e., distributed EEG and MEG source reconstruction problem. In other words, we restrict the mentioned general challenge to have only two modalities, and try to partially address the challenge.

In Chapter 4, we propose a multi-modality framework based on the block structure identification framework proposed in Chapter 3 and block mutual coherence constant proposed in Chapter 2.

To this aim, we applied the block structure identification framework on multi-modal lead-field instead of mono-modal one, to segment brain source space. To investigate the impact of multi-modality, we defined a lead-field combining strategy, which reduces the impact of other factors such as change in the position and number of sensors.

First, we showed that brain regions resulted from clustering the coherent sources of EEG and MEG lead-field matrices separately, are complementary. Then, we made use of complementarities of EEG and MEG lead-field matrices to generated a combined EEG and MEG multi-modal lead-field matrix, and it turned out that in multi-modality case the number of clusters determined by the largest distance between adjacent nodes in dendrogram is higher than mono-modal cases. In addition, for a fixed number of clusters, the under-sensor brain regions in multi-modal lead-field is smaller than mono-modal lead-field clustering.

Therefore, it can be deduced that in multi-modality case, more refined and precise regions appear, hence, the resolution of identifying the active regions increases in comparison to the mono-modal cases.

As perspective in this research area, we could mention to the following points:

• Study on electromagnetic properties in 2D/3D mediums with different number of boundaries as a general case.

• Designing more scenarios (other than brain segmentation) to investigate the impact of multi-modality.

• Designing more methods to combine modalities.

• The impact of combining more modalities (greater than two).

• The optimum ratio of the number of measurements from each modality (not necessarily 50% in two modalities case).

Appendix A

Proofs

A.1 Proof of Property 2.2 (Bounds of division of two vector norms, page 54)

A.6 Proof of Lemma 2.1 (Block-UP based on Block-Spark, page 65)

Proof. Here, we used the following triangle inequality:

which is the the block-structured generalisation of one of the properties of the 0 pseudo-norm operator for vector space. The above relation simply indicates that, if two representations have no overlapped supports, the sum of their number of active blocks are equal to the number of active blocks of their subtraction (equality), while in case of overlapped supports, the sum of their number of active blocks is greater than the number of active blocks of their subtraction (inequality).

On the other hand, we have Φ(β 0 -β 1 )=0. Therefore (β 0 -β 1 ) is in the Kernel of the dictionary, and based on the definition of Block-Spark in Definition 2.5 (page 52), we have:

which proves the lemma.

A.8 Proof of Lemma 2.2 (Basic Block-UP, page 69)

Proof. The following proof generalises the corresponding proofs in [START_REF] Elad | On sparse signal representations[END_REF]; [START_REF] Elad | A generalized uncertainty principle and sparse representation in pairs of bases[END_REF]; [START_REF] Eldar | Block-sparsity: Coherence and efficient recovery[END_REF]; [START_REF] Eldar | Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery[END_REF] and [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF]. Without loss of generality, assume that the non-zero signal y is normalized to have unit squared Euclidean norm, i.e., y 2 2 =y T y=1, and suppose that p is Hölder conjugate to p , i.e., 1/p+1/p =1 [START_REF] Golub | Matrix Computations[END_REF]. Then,

A.11 Proof of Property 2.18 (Block-SL q,p inequalities, page 73)

Proof. First, considering that for all basic tractable (q, p) pairs of table 2.1 (page 54), we have q≤p, then the term min k min{1, d 1/q-1/p k } in Block-SL q,p of Theorem 2.3 (page 73) would be equal to one. Hence, Block-SL q,p has directly an inverse relationship with Block-MCC q,p (Definition 2.6, page 54), i.e., for 1≤q≤p, Block-SL q,p (Φ)=(1+(d max M q,p (Φ)) -1 )/2. Thus, according to the previously-mentioned relationship between different Block-MCC q,p characterisations with basic (q, p) pairs according to table 2.1 (page 54) demonstrated in Property 2.5 (Block-MCC q,p inequalities, page 58), the Block-SL q,p inequalities would be in the opposite direction, which proves the inequalities of the Property. Therefore, for q≤p the directions in figure 2.5(a) (page 73) is in the opposite of the ones in figure 2.4 (page 59).

For the proof of general relationships (tractable and intractable) of figure 2.5(b), we have the following Block-SL q,p for q>p, and

By comparing Eldar's Block-ERC, i.e., β 0 2,0 <(1+(dM Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)))/2, explained in (1.20) and the proposed equally-sized Block-ERC proposed in Theorem 2.3, i.e., ∀(q, p)∈R 2 ≥1 , ∀r∈R ≥0 : β 0 r,0 <(1 + (dM q,p (Φ)) -1 min{1, d 1/q-1/p })/2, it is clear that for q≥p≥1 (the constraint imposed up to this step) the relationship between d 1/q-1/p M -1 q,p (Φ) and (M Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)) should be investigated. By multiplying both sides of the above inequality by d 1/q-1/p >0, for q≥p≥1 we have:

Assuming that the above right-hand side is greater than or equal to the expression that we are interested in to its relation with d 1/q-1/p M -1 q,p (Φ), i.e., (M Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)), we extract the required corresponding condition. In other words, assuming that for q≥p≥1, we have:

, we conclude that for

the assumption will hold true.

On the other hand, we had another condition on the upper-bound of M Eldar Intra (Φ) to meet the condition of Property A.2. Therefore, the minimum of the obtained upper-bounds on the M Eldar Intra (Φ) ensures that ∀q≥p≥1 the Block-ERC proposed in Theorem 2.3 improves Eldar's Block-ERC, i.e., d 1/q-1/p M -1 q,p (Φ)≥(M Eldar Inter (Φ)) -1 (1-(d-1)M Eldar Intra (Φ)):

The above last inequality yields from the fact that a fraction with smaller numerator and denominator is smaller. But, the resulted upper-bound is negative. Therefore, although in practice the numerical simulations approve that the proposed block-sparsity level for general dictionaries with full column rank blocks is higher than the block-sparsity level proposed by Eldar et al., in theory due to utilising the most pessimistic bounds, its supremacy cannot be proved.

A.14 Proof of Theorem 2.4 (Block-ERC based on Eldar's cumulative coherence, page 76)

Proof. The proof is similar to the proof of Theorem 3 in [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF]. First, divide the whole matrix Φ into two complementary matrices Φ opt and Φ opt . Suppose Φ opt is a full column rank m by kd matrix whose blocks correspond to non-zero blocks of β 0 , and let Φ opt be its complementary matrix. From Theorem 2 in [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], a sufficient condition for block orthogonal matching pursuit and 2 / 1 -optimisation program algorithms to correctly recover the block k-sparse β 0 , is that ρ c (Φ † opt Φ opt )<1, where, ρ c (A)= max j i A[i, j] 2→2 and A[i, j] is the (i, j) th d×d block of A. Using Moore-Penrose pseudo-inverse property of matrices and the submultiplicativity property of ρ c (•) been proved in Lemma 2 in [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF], we have:

On the other hand, we have:

where, Λ is the set of indices of blocks of Φ which are in Φ opt , and by Definition 2.8

. Therefore, we have:

Now it remains to upper-bound ρ c ((Φ T opt Φ opt ) -1 ). To this aim, we decompose Φ T opt Φ opt as Φ T opt Φ opt =I kd +F , where, F is a kd by kd matrix with blocks F [i, j] of size d×d, that

All the main diagonal entries of F are zero. Then, we have: where, the first term is obtained by applying Geršgorin's disc theorem [START_REF] Johnson | Matrix Analysis[END_REF], and using the definition of sub-coherence proposed by Eldar [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF]. Precisely, from the definition of F [k, k]∈R d×d , we have ∀k, F i,j =i [k, k]≤M Eldar Intra (Φ), while F i,i [k, k]=0. On the other hand, from Corollary 6.1.5 in [START_REF] Johnson | Matrix Analysis[END_REF] (Geršgorin's disc theorem), we have ∀A∈R m×m , A 2→2 ≤ min{ A 1→1 , A ∞→∞ }. Therefore, max j F [j, j] 2→2 ≤(d-1)M Eldar Intra (Φ). The second term follows from the definition of the cumulative inter-block coherence constant, defined in Definition 2.8. Using Lemma 4 in [START_REF] Eldar | Block-sparse signals: Uncertainty relations and efficient recovery[END_REF] and considering the assumption of Theorem 2.4 which indicates that ρ c (F )<1, we have:

, where, the first inequality follows from the triangle inequality and submultiplicativity properties. Then we have the following inequality, which is a simple rearrangement of the equation in Theorem 2.4:

Abstract -Many natural phenomena are too complex to be fully recognised by only a single measurement instrument or mono-modality.

Therefore, the research domain of multi-modality has emerged to better identify the rich characteristics of the natural multi-property phenomenon, through jointly analysing the data collected from mono-modalities, which are somehow complementary. In our study, the multi-property phenomenon of interest is the human brain activity and we are interested in better localising it by means of its electromagnetic properties which are measurable non-invasively. In neurophysiology, a common way to measure the electric and magnetic properties of the brain activity is ElectroEncephaloGraphy (EEG), and MagnetoEncephaloGraphy (MEG), respectively. Our real-world application, i.e., EEG/MEG source reconstruction problem, is a fundamental problem in neuroscience ranging from cognitive science to neuropathology to surgical planning. Considering that the EEG/MEG source reconstruction problem can be reformulated as an underdetermined system of linear equations, the solution (estimated brain source activity) should be sufficiently sparse in order to be recovered uniquely. The amount of sparsity is determined by the so-called recovery conditions. However, in high-dimensional problems, the conventional recovery conditions are extremely strict. By regrouping the coherent columns of a dictionary, the more incoherent structure could be achieved. This strategy was proposed as a block structure identification framework, which results in the automatic segmentation of the brain source space, without using any information about the brain sources activity and EEG/MEG signals. Despite the resulted less coherent block-structured dictionary, the conventional recovery condition is no longer capable of computing the coherence characterisation. To address the mentioned challenge, the general framework of block-sparse exact recovery conditions including three theoretical and one algorithmic-dependent conditions was proposed. Finally, we investigated the EEG and MEG multi-modality and demonstrated that by combining the two modalities, more refined brain regions appeared.

Keywords: (q, p)-Block Mutual Coherence Constant (Block-MCC q,p ), Block Null Space Property (Block-NSP), Block-Spark, Block-sparse Exact Recovery Conditions (Block-ERC), block-sparse uncertainty principle, Block-sparsity, block structure identification, brain source space segmentation, cumulative Block-MCC q,p , EEG and MEG multi-modality.