Calabi-Yau Manifolds

Harold Erbin
email: erbin@mit.edu

Riccardo Finotello
email: riccardo.finotello@cea.fr

Mohamed Tamaazousti
email: mohamed.tamaazousti@cea.fr

Machine learning for complete intersection

We describe the recent developments in using machine learning techniques to compute Hodge numbers of complete intersection Calabi-Yau (CICY) 3-and 4folds. The main motivation is to understand how to study data from algebraic geometry and solve problems relevant for string theory with machine learning. We describe the state-of-the art methods which reach near-perfect accuracy for several Hodge numbers, and discuss extrapolating from low to high Hodge numbers, and conversely.

Introduction

Calabi-Yau (CY) manifolds are a very active field of research in both mathematics (algebraic geometry) and theoretical physics (string theory). They correspond to Kähler manifolds (complex manifolds with a closed 2-form) whose holonomy group is a subgroup of SU(N ) [1]. These manifolds have a special role in string theory because they are used to describe compactifications: indeed, superstring theory predicts that spacetime must be 10-dimensional, in apparent contradiction with the observed 4-dimensional universe. The solution is to compactify 6 dimensions into a tiny volume at each spacetime point, such that they cannot be observed at the energy scales from the current experiments; consistency of string theory requires the manifold describing these six dimensions to be CY [START_REF] Graña | Flux compactifications in string theory: A comprehensive review[END_REF][START_REF] Ibáñez | String Theory and Particle Physics: An Introduction to String Phenomenology[END_REF]. The shape of the manifold used for the compactification characterizes the low-energy effective action, which should match the existing Standard Model of particle physics. For this reason, classifying CY manifolds by computing their topological and geometrical properties is of the utmost importance in string theory.

String theory is a theory of quantum gravity which also unifies all force and matter fields. Hence, finding the appropriate compactification would allow, in principle, to completely determine the physics of our universe. However, there are some important challenges for pushing this program to its term [START_REF] Ruehle | Data science applications to string theory[END_REF]. First, there is an extremely large number of compactifications, which makes an exhaustive scan impossible. Second, a complete expression for the low-energy effective action is lacking because it requires knowing the CY metric. While existence has been proven by Yau, no metric has been constructed for compact CY until today (though recent progresses with machine learning have been achieved [START_REF] Ashmore | Machine learning Calabi-Yau metrics[END_REF][START_REF] Michael | Numerical Calabi-Yau metrics from holomorphic networks[END_REF][START_REF] Jejjala | Neural Network Approximations for Calabi-Yau Metrics[END_REF][START_REF] Anderson | Moduli-dependent Calabi-Yau and SU(3)-structure metrics from Machine Learning[END_REF][START_REF] Larfors | Learning Size and Shape of Calabi-Yau Spaces[END_REF][START_REF] Larfors | Numerical Metrics for Complete Intersection and Kreuzer-Skarke Calabi-Yau Manifolds[END_REF]). Third, topological properties (which largely constrain the general form of the Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

effective theory) can be difficult to compute with standard techniques of algebraic geometry, which lead to complicated algorithms, without closed-form solutions in general. For instance, computing cohomology group dimensions often requiring summing and subtracting huge integers at intermediate steps, to only get relatively small integers at the end.

Progress in string theory goes in hand with understanding better mathematical structures, and machine learning (ML) has been called to the rescue for all three problems [START_REF] Ruehle | Data science applications to string theory[END_REF][START_REF] He | The Calabi-Yau Landscape: From Geometry, to Physics, to Machine Learning[END_REF]. In this paper, we focus on computing Hodge numbers for complete intersection CY (CICY) manifolds [START_REF] He | Machine-learning the string landscape[END_REF][START_REF] Bull | Machine Learning CICY Threefolds[END_REF][START_REF] Bull | Getting CICY High[END_REF][START_REF] He | Machine Learning Calabi-Yau Four-folds[END_REF][START_REF] Erbin | Inception Neural Network for Complete Intersection Calabi-Yau 3-folds[END_REF][START_REF] Erbin | Machine learning for complete intersection Calabi-Yau manifolds: A methodological study[END_REF][START_REF] Erbin | Deep multi-task mining Calabi-Yau four-folds[END_REF][START_REF] Erbin | Deep learning complete intersection Calabi-Yau manifolds[END_REF]. This topic is interesting for fundamental mathematics and physics, as argued above, but also for the machine learning community as it showcases a new type of data which has not been properly studied until now. CICYs form a particularly appropriate set with which to begin our investigation since they correspond to the simplest category of CY, they have been completely classified, and their topological properties fully computed. As a consequence, this allows us to design and test new methods in a controlled environment before generalizing to new problems. Moreover, CICYs are often used in string theory model building due to their simple properties.

2 Complete intersection Calabi-Yau manifolds CY N -folds are N -dimensional complex manifolds with SU(N ) holonomy, or, equivalently, with a vanishing first Chern class [1]. They are characterized by their topological properties, such as the Hodge numbers. These features directly translate into properties of the 4-dimensional effective action, such as the number of chiral multiplets in heterotic compactifications, and the number of hyper and vector multiplets in type II compactifications. Ultimately, they are connected to the number of fermion generations, which could be used to test the effectiveness of the models.

The simplest CYs are constructed as complete intersections of hypersurfaces in a product of complex projective spaces P n1 × • • • × P nm . They are defined by systems of homogeneous polynomial equations, whose solutions identify CY manifolds. When considering topological properties, it is sufficient to keep track only of the dimensions of the projective spaces and the degree of the equations. In the general case of m projective spaces and k equations, a CICY X is represented by a configuration matrix, which will be the input to the network:

X =    P n1 α 1 1 • • • α 1 k . . . . . . . . . . . . P nm α m 1 • • • α m k    , n i + 1 = k r=1 α i r , dim C X = m i=1 n i -k = N, α i r ∈ N + . (1)
The Hodge numbers h (p,q) are topological invariants which count the dimensions of the Dolbeault cohomology groups, whose elements are complex (p, q)-forms.

We will investigate the CICY 3-and 4-folds. The numbers, matrix size, and statistics (written as ⟨•⟩ = average max min ) of their Hodge numbers are:

• N = 3: 7890 CICYs with 15 × 18 matrices, ⟨h (1,1) ⟩ = 7.4 19 1 , ⟨h (2,1) ⟩ = 29 101 15 [START_REF] Candelas | Complete intersection Calabi-Yau manifolds[END_REF][START_REF] Green | All the Hodge numbers for all Calabi-Yau complete intersections[END_REF]. • N = 4: 921 497 CICYs with 16×20 matrices, ⟨h (1,1) ⟩ = 10 24 1 , ⟨h (2,1) ⟩ = 0.8 33 0 , ⟨h (3,1) ⟩ = 40 426 20 , ⟨h (2,2) ⟩ = 240 1752 204 [START_REF] Gray | All Complete Intersection Calabi-Yau Four-Folds[END_REF][START_REF] Gray | Topological Invariants and Fibration Structure of Complete Intersection Calabi-Yau Four-Folds[END_REF].

Methodology

In this section, we present briefly the CICYMiner architecture and explain its main characteristics.

The paper [START_REF] Erbin | Machine learning for complete intersection Calabi-Yau manifolds: A methodological study[END_REF] performed a complete data analysis for the 3-folds dataset, including feature engineering and selection. It was found that no derived feature helped in computing the Hodge numbers, and that neural networks outperform any other tested algorithm. As a consequence, we will focus on the latter, and review the architecture with the highest performance.

CICYMiner [START_REF] Erbin | Deep multi-task mining Calabi-Yau four-folds[END_REF] is built from a series of Inception modules [START_REF] Erbin | Inception Neural Network for Complete Intersection Calabi-Yau 3-folds[END_REF][START_REF] Erbin | Machine learning for complete intersection Calabi-Yau manifolds: A methodological study[END_REF], starting with a common trunk which splits in branches, such that it computes the different Hodge numbers at the same time given a configuration matrix m × k. Inception modules are inspired by GoogleNet [START_REF] Szegedy | Going deeper with convolutions[END_REF] and perform h (1,1) aux h (1,1) Figure 1: CICYMiner architecture, with dropout (D) with rate 0.2, and batch normalization (BN). The auxiliary branches replicate the output predictions and help to stabilize the learning process. two parallel convolutions with 1d kernels m × 1 and 1 × k, followed by concatenation and batch normalization. This structure is important 1) to put on an equal footing all the equations and projective spaces defining the configuration matrix (thanks to weight sharing), while also 2) taking into account the symmetries of the system. The ablation study performed in [START_REF] Erbin | Inception Neural Network for Complete Intersection Calabi-Yau 3-folds[END_REF] displayed strikingly how both these ingredients are necessary to reach the highest accuracy. A neural network made of 3 Inception modules [START_REF] Erbin | Inception Neural Network for Complete Intersection Calabi-Yau 3-folds[END_REF][START_REF] Erbin | Machine learning for complete intersection Calabi-Yau manifolds: A methodological study[END_REF] also achieved a near-perfect accuracy for the 3-folds h (1,1) while requiring much fewer parameters compared to earlier results [START_REF] He | Machine-learning the string landscape[END_REF][START_REF] Bull | Machine Learning CICY Threefolds[END_REF]. The architecture, inspired from [START_REF] Benzine | Deep Miner: A Deep and Multibranch Network which Mines Rich and Diverse Features for Person Re-identification[END_REF], is shown in Figure 1 for the 4-folds; the network contains approximately 10 7 parameters. The same architecture can be used for the 3-folds which is achieved by removing the legs for h (2,2) and h (3,1) (3.3 × 10 6 parameters). Hyperparameter optimization is performed on the 4-folds dataset using a grid search over a reasonable portion of the hyperparameter space defined using insights from other architectures which used Bayesian or BOHB optimizations.

We preprocess the input data by simply rescaling the entries of the configuration matrices in the training set in the [0, 1] range, but we don't rescale the labels. The computation is performed as a regression:1 outputs are forced to be positive by adding a ReLU after each output layer, and predictions are rounded to the closest integer before computing the accuracy of the network. We train with the Huber loss with hyperparameter δ = 1.5, and loss weights of 0.05, 0.3, 0.25 and 0.35 for h (1,1) , h (2,1) , h (3,1) and h (2,2) , respectively, in the 4-folds case (determined using hyperparameter optimization as described above). In order to maintain a similar ratio, we use 0.17 and 0.83 for h (1,1) and h (2,1) for the CICY 3-folds. Performance of the network is assessed by computing the accuracy (agreement between the predicted and real integer values). Training has been performed on a single NVIDIA V100 GPU over a fixed amount of 300 epochs for CICY 4-folds, while we use 1500 epochs in the 3-folds case, due to limited cluster time. We use the Adam optimizer with an initial learning rate of 10 -3 and a mini-batch size of 64 configuration matrices. The learning rate is reduced by a factor 0.3 after 75 epochs without improvement in the validation loss.

The code is available at: https://github.com/thesfinox/ml-cicy-4folds, https:// github.com/melsophos/cicy.

Results

We provide three applications of the CICYMiner network: 1) Hodge numbers predictions, extrapolation after training only with 2) low and 3) high Hodge numbers. Except for the 4-folds predictions in 1), the results are new.

Predicting Hodge numbers We are interested in predicting the Hodge numbers, using the configuration matrix as input of the neural network. We will train with x% of the data selected by stratifying on h (2,1) in order to preserve the distribution of the samples, and varying x how the accuracy improves with the size of the training set. The validation set is chosen totally at random,

WUDLQLQJUDWLR DFFXUDF\ h 1, 1 h 2, 1
(a) 3-folds. Table 1: Accuracy when training with an upper bound on h (1,1) or lower bound on h (2,2) (4-folds). As a reminder, extremal values for both Hodge numbers are h (1,1) ≤ 24, h (2,2) ≥ 204. (2,2) > 400 1.6 % 0.76 0.70 0.03 0.0 using 10 % of the samples. The remaining samples are included in the test set. The results for the 3and 4-folds are given in Figures 2a and2b.

h (1, 1) h (2, 1) h (3, 1) h (2, 2) (b) 4-folds.
ratio h (1,1) h (2,1) h (3,1) h (2,2) h (1,
For the 4-folds and h (1,1) of the 3-folds, the CICYMiner network is capable of learning accurately the discreteness of the Hodge numbers: for most of them, the regression metrics show values which indicate confidently integer numbers (MAE ≪ 0.50 and MSE ≪ 0.25). Note that the network is still underfitting the training set: longer training or different choices of the learning rate may be needed to investigate this aspect.

Extrapolation from low Hodge numbers We focus on the CICY 4-folds dataset. Following [START_REF] Bull | Getting CICY High[END_REF], we test the ability to predict the entire range of Hodge numbers when training only with samples whose h (1,1) is below a fixed value. Then, we test how well the network performs in predicting Hodge numbers outside the training range. The motivation is that, for other datasets, CYs with low h (1,1) are easier to understand, such that one can be interested in training for those instances and extrapolate the predictions. We reuse the same hyperparameters as before, but train CICYMiner on the reduced training set. Results are summarized in Table 1, where the ratio of the data refers to the size of the training set. As we can see, the performance increases quickly for the first two Hodge numbers, but remain lower than in the previous study for the last two.

Extrapolation from high Hodge numbers The same experiment can be run by imposing a lower bound on h (2,2) (largest number of outliers and largest interval of variation). As previously, no hyperparameter optimization is run in this case. Results are shown in Table 1. As in the previous case, good results on h (1,1) and h (2,1) can be recovered starting from approximately 50 % of the training samples, but h (2,2) performance stays low in any case. In this and previous cases, it is not clear why the network behaves this way.

Conclusion

This paper described the state-of-the-art in predicting CICY Hodge numbers with neural networks. This shows that machine learning techniques, and especially deep learning, are completely adapted to manipulate data in algebraic geometry. While the other topological quantities of the CICY are relatively easy to compute, it would still be interesting to explore if neural networks can predict them accurately. These different studies open the stage for attacking more difficult problems in algebraic geometry, such as studying CY built from the Kreuzer-Skarke polytopes [START_REF] Kreuzer | Complete classification of reflexive polyhedra in four dimensions[END_REF][START_REF] Demirtas | Bounding the Kreuzer-Skarke Landscape[END_REF].

Computing accurately h (2,1) for the 3-folds is still an open problem. While the CICYMiner architecture performs slightly better than the pure Inception model from [START_REF] Erbin | Inception Neural Network for Complete Intersection Calabi-Yau 3-folds[END_REF][START_REF] Erbin | Machine learning for complete intersection Calabi-Yau manifolds: A methodological study[END_REF], the drawback is a much slower training time and many more parameters. Moreover, while results are more precise for the 4-folds, it would be desirable to increase the accuracy of h (2,2) . Work in progress explores if the graph representation of the data [1,[START_REF] Green | Polynomial deformations and cohomology of Calabi-Yau manifolds[END_REF][START_REF] Krippendorf | Detecting Symmetries with Neural Networks[END_REF] can help.

Finally, the last step is extracting analytical formulas in order to interpret the neural network and improve our knowledge. The simplest approach is to perform symbolic regression on the neural network following [START_REF] Miles | Learning Symbolic Physics with Graph Networks[END_REF][START_REF] Cranmer | Discovering Symbolic Models from Deep Learning with Inductive Biases[END_REF]. It can be expected that the result contains piecewise functions, given the structure of the analytic expressions found for line bundle cohomologies [START_REF] Constantin | Formulae for Line Bundle Cohomology on Calabi-Yau Threefolds[END_REF][START_REF] Larfors | Line bundle cohomologies on CICYs with Picard number two[END_REF][START_REF] Klaewer | Machine Learning Line Bundle Cohomologies of Hypersurfaces in Toric Varieties[END_REF][START_REF] Callum | Index Formulae for Line Bundle Cohomology on Complex Surfaces[END_REF][START_REF] Callum | Machine Learning Line Bundle Cohomology[END_REF].

Broader impacts

Historically, codes used to compute properties of CY for string theory have not been made open source, which made the barrier to enter the field high. One goal of developing ML algorithms and make them open source is to help people get involved in the field.

Figure 2 :

 2 Figure 2: Accuracy for CICYMiner as a function of percentage of training data.

( b )

 b Did you mention the license of the assets? [N/A] The data we used was not licensed by the authors. (c) Did you include any new assets either in the supplemental material or as a URL? [Yes] (d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]

Approaching the problem as a classification task does not make sense because it would assume an a priori knowledge of the value ranges, which is not desirable.

Acknowledgments and Disclosure of Funding

This project has received funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 891169. This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/). This publication was made possible by the use of the FactoryIA supercomputer, financially supported by the Île-De-France Regional Council.


Harold Erbin
email: erbin@mit.edu

Vincent Lahoche
email: vincent.lahoche@cea.fr

Dine Ousmane Samary
email: dine.ousmanesamary@cea.fr

Renormalization in the neural network-quantum field theory correspondence

A statistical ensemble of neural networks can be described in terms of a quantum field theory (NN-QFT correspondence). The infinite-width limit is mapped to a free field theory, while finite N corrections are mapped to interactions. After reviewing the correspondence, we will describe how to implement renormalization in this context and discuss preliminary numerical results for translation-invariant kernels. A major outcome is that changing the standard deviation of the neural network weight distribution corresponds to a renormalization flow in the space of networks.

Introduction

While neural networks (NN) perform extremely well on several tasks, they generally behave as black boxes which are hard to interpret [START_REF] Weld | The Challenge of Crafting Intelligible Intelligence[END_REF][START_REF] Zhang | A Survey on Neural Network Interpretability[END_REF]. This is a problem for applications where safety can be put in jeopardy [START_REF]Ethics Guidelines for Trustworthy AI[END_REF], but also if concrete explanations are needed, as in sciences [START_REF] Roscher | Explainable Machine Learning for Scientific Insights and Discoveries[END_REF][START_REF] Cranmer | Discovering Symbolic Models from Deep Learning with Inductive Biases[END_REF][START_REF] Raghu | A Survey of Deep Learning for Scientific Discovery[END_REF]. Training is another concern because it is computationally expensive and has possible convergence issues. Indeed, the loss function is typically non-convex such that it can be hard to find the global minimum [START_REF] Choromanska | The Loss Surfaces of Multilayer Networks[END_REF][START_REF] Li | Visualizing the Loss Landscape of Neural Nets[END_REF]. There is also no systematic hyperparameter tuning procedure and one has to rely on random scans, possibly improved with Bayesian and bandit methods [START_REF] Bergstra | Random Search for Hyper-Parameter Optimization[END_REF][START_REF] Snoek | Practical Bayesian Optimization of Machine Learning Algorithms[END_REF][START_REF] Li | Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization[END_REF][START_REF] Falkner | BOHB: Robust and Efficient Hyperparameter Optimization at Scale[END_REF] which results in very high financial [START_REF] Sharir | The Cost of Training NLP Models: A Concise Overview[END_REF] and environmental costs [START_REF] Strubell | Energy and Policy Considerations for Deep Learning in NLP[END_REF][START_REF] Lacoste | Quantifying the Carbon Emissions of Machine Learning[END_REF][START_REF] Henderson | Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning[END_REF]. Finally, the question of knowing which functions can be expressed by a given NN remains open [START_REF] Lu | The Expressive Power of Neural Networks: A View from the Width[END_REF][START_REF] Raghu | On the Expressive Power of Deep Neural Networks[END_REF]: while universal approximation theorems guarantee existence [START_REF] Cybenko | Approximation by Superpositions of a Sigmoidal Function[END_REF][START_REF] Hornik | Multilayer Feedforward Networks Are Universal Approximators[END_REF][START_REF] Hornik | Approximation Capabilities of Multilayer Feedforward Networks[END_REF][START_REF] Leshno | Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function[END_REF][START_REF] Pinkus | Approximation Theory of the MLP Model in Neural Networks[END_REF][START_REF] Csáji | Approximation with Artificial Neural Networks[END_REF], finding the appropriate architecture for a new task often boils down to trials and errors. Improving our theoretical understanding of NN is primordial for addressing these issues.

Physics provides a natural starting point for designing a theory of NN [START_REF] Lin | Why Does Deep and Cheap Learning Work so Well?[END_REF][START_REF] Zdeborová | Understanding Deep Learning Is Also a Job for Physicists[END_REF][START_REF] Agliari | Machine Learning and Statistical Physics: Preface[END_REF]. First, thanks to its effective descriptions, it is not necessary to know the fundamental theory. Second, efficient representations of statistical models have been developed (path integrals, Feynman diagrams, statistical mechanics. . . ). Third, it allows characterizing the collective dynamics of degrees of freedom and organizing a phenomenon by scales. Applications of physics to machine learning include statistical physics [START_REF] Choromanska | The Loss Surfaces of Multilayer Networks[END_REF][START_REF] Amit | Statistical Mechanics of Neural Networks near Saturation[END_REF][START_REF] Gardner | The Space of Interactions in Neural Network Models[END_REF][START_REF] Gardner | Optimal Storage Properties of Neural Network Models[END_REF][START_REF] Krauth | Basins of Attraction in a Perceptron-like Neural Network[END_REF][START_REF] Mézard | Learning in Feedforward Layered Networks: The Tiling Algorithm[END_REF][START_REF] Saitta | Phase Transitions in Machine Learning[END_REF][START_REF] Bahri | Statistical Mechanics of Deep Learning[END_REF], renormalization [START_REF] Bény | Deep Learning and the Renormalization Group[END_REF][START_REF] Mehta | An Exact Mapping between the Variational Renormalization Group and Deep Learning[END_REF][START_REF] Bény | Inferring Relevant Features: From QFT to PCA[END_REF][START_REF] De Mello Koch | Is Deep Learning a Renormalization Group Flow?[END_REF], and QFT [START_REF] Schoenholz | A Correspondence Between Random Neural Networks and Statistical Field Theory[END_REF][START_REF] Helias | Statistical Field Theory for Neural Networks[END_REF][START_REF] Halverson | Neural Networks and Quantum Field Theory[END_REF][START_REF] Maiti | Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators[END_REF][START_REF] Halverson | Building Quantum Field Theories Out of Neurons[END_REF][START_REF] Erbin | Nonperturbative Renormalization for the Neural Network-QFT Correspondence[END_REF][START_REF] Grosvenor | The Edge of Chaos: Quantum Field Theory and Deep Neural Networks[END_REF].

In this paper, we will review the neural network-quantum field theory (NN-QFT) correspondence developed in [START_REF] Halverson | Neural Networks and Quantum Field Theory[END_REF][START_REF] Erbin | Nonperturbative Renormalization for the Neural Network-QFT Correspondence[END_REF] since it provides concrete and testable tools to improve our analytical understanding of neural network building and training. This correspondence states that, for a very general class of architectures, it is possible to associate a quantum field theory (QFT) with a statistical ensemble of NN. We focus on a fully connected NN with a single hidden layer and setup non-perturbative renormalization group equations (valid for any finite width). The main result is that varying the standard deviation of the weight distribution induces a renormalization group (RG) flow in the space of NN. Code is available at: https://github.com/melsophos/nnqft.

NN-QFT correspondence

Take a fully connected neural network f θ,N : R din → R dout with one hidden layer of width N :

f θ,N (x) = W 1 g(W 0 x + b 0 ) + b 1 , (2.1)
where g is the non-linear activation function, and the parameters θ = (W 0 , b 0 , W 1 , b 1 ) (weights and biases) have Gaussian distributions:

W 0 ∼ N (0, σ 2 W /d in ), W 1 ∼ N (0, σ 2 W /N ), b 0 , b 1 ∼ N (0, σ 2 b ) . (2.2)
Consider next a statistical ensemble of neural networks, such that a given neural network is sampled from the distribution in parameter space: f θ,N ∼ P [θ]. Then, there is a dual description in terms of another distribution in function space, which is induced by the parameter distribution plus the architecture: f θ,N ∼ p[f ] [START_REF] Halverson | Neural Networks and Quantum Field Theory[END_REF]. Changing the parameter distribution by training corresponds to flowing in the function space.

In the large N limit (infinite width), the function distribution becomes a Gaussian process with kernel K (as a consequence of the central limit theorem) [START_REF] Neal | Bayesian Learning for Neural Networks[END_REF]:

f ∼ N (0, K) . (2.3)
This statement generalizes to most architecture and training [START_REF] Yang | Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture Are Gaussian Processes[END_REF]. We denote as S 0 [f ] the (Gaussian) log-probability:

S 0 [f ] := 1 2 d din xd din x f (x)Ξ(x, x )f (x ), Ξ := K -1 , ( 2.4)
and as

G (n) 0 (x 1 , . . . , x n ) := E 0 [f (x 1 ) • • • f (x n )] ≡ df e -S0[f ] f (x 1 ) • • • f (x n ) (2.5)
the Gaussian expectation value (GEV) for a product of n fields f (x i ). The measure df is suitably normalized such that df exp(-S 0 [f ]) = 1. In physics, this setting corresponds to a free QFT, K to the free propagator and G

(n) 0

to the free n-point correlation functions (also called Green functions). At finite N , the distribution is not a Gaussian process, and we denote as

∆G (n) := G (n) -G (n) 0 (2.6)
the difference between the full expectation value (FEV)

G (n) (x 1 , . . . , x n ) := E[f (x 1 ) • • • f (x n )] ≡ df e -S[f ] f (x 1 ) • • • f (x n ) (2.7)
and the GEV. The main message of the NN-QFT correspondence is that even at finite N , the log-probability S[f ] can be designed with non-Gaussian contributions to reproduce the FEVs with arbitrary precision up to the numerical uncertainties in the simulations. We denote as

S int [f ] the non-Gaussian contributions in S[f ]: S[f ] = S 0 [f ] + S int [f ], (2.8)
where y) is N -independent and fixed by the NN, such that the Gaussian part must be different and such that:

S 0 [f ] = S 0 [f ] is some new Gaussian action. Indeed, the 2-point FEV G (2) (x,
G (2) (x, y) = G (2) 0 (x, y) ≡ K(x, y) .
(2.9)

A complete dictionary between NN and QFT is given by Table 1. This formulation is promising because correlation functions between outputs give a measure of learning; e.g., the 1-point function E[f (x)] corresponds to the average prediction for input x (which is related to the idea of symmetry breaking in QFT [START_REF] Maiti | Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators[END_REF]). Hence, having a QFT may allow performing (semi-)analytic predictions in advance of the outcome of the learning process.

Kernels in data-space are typically bi-local [START_REF] Halverson | Neural Networks and Quantum Field Theory[END_REF] such that one can expect non-local interactions. Moreover, it is not clear what are the symmetries of the inputs and outputs (in the QFT sense) for general data. With these observations, we follow an approach which can be called NN phenomenology: 1) make assumptions dictated by numerical evidence, 2) write a QFT model to match observations, 3) use the model to check theoretical facts.

Constructing the QFT

The expectation values G (n) can be computed analytically using QFT tools ("theory") or computed from a statistical ensemble of neural networks ("measurements"). Hence, we can make an ansatz for S int [f ] and match the parameters by computing enough correlation functions. The choice of this ansatz especially regarding symmetries and the way the fields are coupled depend on the Gaussian kernel K. In this paper, we set d in = d out = 1 and focus on a translation-invariant activation function:

g(W 0 x + b 0 ) = exp(W 0 x + b 0 ) exp 2 σ 2 b + σ 2 W din x 2 (3.1)
such that the Gaussian kernel is [START_REF] Halverson | Neural Networks and Quantum Field Theory[END_REF]:

K(x, y) := σ 2 b + K W (x, y), K W (x, y) = σ 2 W e - σ 2 W 2d in |x-y| 2 . (3.2)
In order to compute the "experimental" Green functions for a given N , we create n bags distinct statistical ensembles of n nets networks each [START_REF] Halverson | Neural Networks and Quantum Field Theory[END_REF][START_REF] Erbin | Nonperturbative Renormalization for the Neural Network-QFT Correspondence[END_REF], and compute Ḡ(n) exp as the average of the (empirical) FEV:

G (n) exp = 1 n nets nnets α=1 f α (x 1 ) • • • f α (x n ) , ( 3.3)
computed in a given bag. We furthermore define

∆G (n) exp := Ḡ(n) exp -G (n) 0 , ( 3.4)
and the normalized deviation

m n := ∆G (n) exp /G (n) 0 .
For the numerical investigations, we consider the points x (1) , . . . , x (6) ∈ {-0.01, -0.006, -0.002, 0.002, 0.006, 0.01} and evaluate the Green functions for all inequivalent combinations. Moreover, all numerical tests are performed with σ b = 1, σ W = 1, n bags = 20, n nets = 30000, and N ∈ {2, 3, 4, 5, 10, 20, 50, 100, 500, 1000}. Computations ran during one week on the internal cluster of one of our institute. Empirically, we find that m 2 ≈ 0 (the second momentum is almost independent of N ) and m 2n = O 1 N for n > 1, the last result meaning that the empirical 2n-cumulant of the distribution G (2n) c,exp must be of order 1/N n-1 . The histogram of values for m 2 and m 4 are given in Figure 1.

The translation invariance of the Gaussian kernel is reminiscent of standard QFTs, where S int can be expanded in powers of f coupled at the same point, namely: for some n 0 ∈ N. We can check the validity of this ansatz experimentally. Indeed, in that expression, ūn is nothing but the magnitude of the lowest order deviation from the GEV, and is called bare coupling. It is different from the effective coupling u n which is measured by the simulations and which includes quantum corrections; in perturbation theory (assuming ūn small enough), and u n = ūn + O(ū 2 n ) (schematically). At higher order, this deviation receives many contributions which can be formally resumed. For the lowest order, the full (normalized) deviation from the GEV u 4 (x 1 , x 2 , x 3 , x 4 ) reads:

S int [f ] := n0 n=2 ūn (2n)! d din x f (x) 2n , ( 3
u 4 (x 1 , x 2 , x 3 , x 4 ) = - ∆G (4) exp (x 1 , x 2 , x 3 , x 4 ) d din x K W (x, x 1 )K W (x, x 2 )K W (x, x 3 )K W (x, x 4 ) . ( 3.6)
Empirically, focusing on the truncation n 0 = 3, we find that u 4 is negative but almost constant and u 6 remains small but positive as required for stability. Results for different σ W and N are given below in Figure 2.

Renormalization group

In the previous section, we considered an effective field theory able to reproduce FEV corresponding to a NN ensemble. The RG is a set of techniques allowing to understand the dependency of the effective theory on a typical observation scale. The machine precision provides an example of such an observation scale, and we could consider the dependency of the parameters defining the QFT regarding the machine precision. In this paper, we consider another kind of scaling, induced by the NN itself and called active RG. The motivation stems from the observation that the propagator (3.2) in momentum space looks like the usual Gaussian kernel in QFT at low momentum:

K W (p) = (σ 2 W ) 1-d in 2 d in 2π d in 2 exp - d in 2σ 2 W p 2 ≈ Z -1 0 Λ 2 + p 2 + O(p 2 ) . ( 4.1)
In the QFT terminology, Λ defines the mass scale, and the large momenta p 2 Λ 2 are exponentially suppressed, blinding the physics beyond scale p 2 ∼ Λ 2 . Hence, in the active RG, Λ (or equivalently the standard deviation σ W ) define the typical momentum scale.

Having defined the notion of scale, we are aiming to construct a smooth interpolation between a large cut-off regime (called ultraviolet regime) and a small cut-off regime (called infrared regime). In the large cut-off regime, fluctuations are essentially frozen and the behavior of the network is mainly fixed by the saddle point of the log-probability S[f ]. On the contrary, in the infrared regime, fluctuations are integrated out and look as a novel effective physics. The Wetterich equation describes how the effective description changes with the observation scale and leads to:

Λ d dΛ Γ (2)
Λ (p, -p) = -

1 2 d din q (2π) din Λ dr Λ dΛ (q 2 ) Γ (4)
Λ (p, -p, q, -q) G 2 Λ (q 2 ), (

where Γ

(n)

Λ is the n-th derivative of Γ Λ with respect to f cl , which is defined such that:

Γ Λ [f cl ] := j • f cl -W Λ [j] - 1 2 f cl • r Λ • f cl , f cl (x) := δW Λ δj , ( 4.3)
where

W Λ [j] := E[e -1 2 f •r Λ •f +j•f ] , ( 4
Λ (p 1 , p 2 ) and r Λ (p 2 ) δ (din) (p 1 + p 2 ) can be arbitrary functions of the momentum p 2 , their sum is constrained to be Λ 2 exp

p 2 1 Λ 2 δ (din) (p 1 + p 2 )
for any Λ. Because Λ dr Λ dΛ (q 2 ) has to select only a short window of momenta in the vicinity of the scale Λ, the smooth function Γ

Λ (p, -p, q, -q) can be expanded in power of q for Λ small enough. At zero order and using the Litim's regulator:

r Λ (p 2 ) := α (Λ 2 -p 2 )θ(Λ 2 -p 2 ) , (4.5)
we predict a purely scaling behavior with respect to the standard deviation σ W for the zero momenta function Γ

Λ (0, 0, 0, 0) =: u 4 (Λ)δ(0):

σ W du 4 dσ W = (4 -d in )u 4 =⇒ log u 4 = (4 -d in ) log σ W + cst. ( 4.6)
This equation can be verified numerically (Figure 2). A similar equation can be derived for u 6 : log u 6 = (6 -2d in ) log σ W + cst.

Conclusion

In this paper, we have reviewed the NN-QFT correspondence and described several checks. Our main result is about the derivation of exact renormalization equations where the standard deviation σ W looks like a RG flow parameter, and the nice agreement between theoretical predictions and numerical experiments. Future directions include: increasing d in , d out and N expansion, studying the large d in limit (large data), increasing number of hidden layers and extending to non-translation invariant kernels (ReLU. . . ) using the 2PI formalism [START_REF] Blaizot | Functional Renormalization Group and 2PI Effective Action Formalism[END_REF], and finally studying the evolution of the QFT under training.

Figure 1 :

 1 Figure 1: Normalized deviations with respect to the free theory. For m 2 , values centered around 0 and independent of N . For m 4 , the values decrease as N increases.

NN

  = 100, log10|u4,0| = 0.349 log10|u4| = 3.09log10 W + 0.19 theory: log10|u4| = 3.00log10 W + 0.35 exp fit (c) N = 100. = 1000, log10|u4,0| = 0.828 log10|u4| = 3.08log10 W 0.83 theory: log10|u4| = 3.00log10 W 0.83 exp fit (d) N = 1000.

Figure 2 :

 2 Figure 2: Dependence of u 4 in terms of σ W , computed numerically and with the flow equation (4.6). Parameters: σ b = 0, σ W ∈ {1.0, 1.5, . . . , 10, 20}, n bags = 30, n nets = 30000.

Table 1 :

 1 NN-QFT dictionary.

		QFT	NN / GP
	x	spacetime points	data-space inputs
	p	momentum space	dual data-space
	f	field	neural network
	K(x, y)	propagator	Gaussian kernel
	S	action	negative log-probability
	S 0	free action	Gaussian log probability
	S int	interactions	non-Gaussian corrections
	• ≡ E[•] expectation value, Green function	correlation function

  .[START_REF] Roscher | Explainable Machine Learning for Scientific Insights and Discoveries[END_REF] the dot denoting the inner-product defined by integrating over the data space. Once again, let us note that in the power field expansion of Γ Λ the weights are effective rather than bare couplings. The regulator r Λ depends on p 2 and is designed such that Γ Λ→∞ → S (large cut-off regime) and Γ Λ→0 ≡ Γ (vanishing cut-off regime), Γ being the full effective action, i.e. the Legendre transform of the characteristic function E[e j•f ]. The expectation value K W (p) being fixed by the NN, although both Γ

Acknowledgments

This project has received funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 891169. This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/).


Dayong Wang
email: wangdayong@cqupt.edu.cn

Yu Sun
email: yusun@uca.edu

Weisheng Li

Lele Xie

Xin Lu
email: xin.lu@dmu.ac.uk

Frédéric Dufaux

Ce Zhu

Frederic Dufaux
email: frederic.dufaux@l2s.centralesupelec.fr

A Novel Mode Selection-Based Fast Intra Prediction Algorithm for Spatial SHVC

Keywords: SHVC, ILR mode, rate distortion costs, GMM-EM

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

VIDEO applications, such as digital TV broadcasting, video conferencing, wireless video streaming, and smartphone communications, are more and more widely used in our daily life. At the same time, more and more new terminal devices with different spatial resolutions are emerging. This requires that video streaming must be adaptive to different screen resolutions. Scalable High-Efficiency Video Coding (SHVC) is an efficient solution to this requirement. In order to adapt to different screen resolutions, Spatial SHVC (SSHVC) encodes different layers with different spatial resolutions. By selecting an appropriate layer, SSHVC can adapt to different screen resolutions.

SSHVC consists of a base layer (BL) and one and more enhancement layers (ELs). A BL only has Intra-layer prediction, while an EL further includes Inter-layer prediction. The coding process of Intra-layer prediction is the same as that of HEVC. Since the contents between BL and EL are the same except for their different resolutions, a coding unit (CU) in BL can be up-sampled to predict the co-located CU in EL. This prediction process is denoted as Inter-layer prediction, and its prediction mode is called Inter-layer Reference (ILR) mode. The SSHVC encoding process is very complex [START_REF] Boyce Y. Yan | Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard[END_REF], which definitely restricts its wide applications, especially for wireless and real-time applications. Therefore, it is crucial to accelerate its encoding speed.

For this purpose, in this paper, we propose a novel fast intra prediction algorithm based on Mode Selection for SSHVC.

RELATED WORK

In SHVC [START_REF] Boyce Y. Yan | Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard[END_REF], each CU contains both ILR mode and Intra mode which includes 35 DMs. By checking both modes to select the best one, SHVC can obtain the best coding efficiency. However, doing so can result in a very complicated coding process. In order to improve coding speed, a number of algorithms have been developed, which are reviewed and discussed below.

Generally speaking, the current CU and its relevant CUs are very similar, so relevant CUs can be used to predict candidate coding modes. Tohidypour et al. [START_REF] Tohidypour | Content adaptive complexity reduction scheme for quality/fidelity scalable HEVC[END_REF][START_REF] Tohidypour | Fast mode assignment for quality scalable extension of the high efficiency video coding (HEVC) standard: A Bayesian approach[END_REF][START_REF] Tohidypour | An encoder complexity reduction scheme for Quality/Fidelity scalable HEVC[END_REF] use relevant CUs to predict likelihood modes and skip unlikely modes in EL. Wang et al. [START_REF] Sun | Fast Inter Mode Predictions for SHVC[END_REF], [START_REF] Sun | Fast Depth and Inter Mode Prediction for Quality Scalable High Efficiency Video Coding[END_REF] and [START_REF] Sun | Fast Depth and Mode Decision in Intra Prediction for Quality SHVC[END_REF] first check ILR mode and merge mode, and then compare the difference of their RD costs to early terminate mode selection. The above algorithms are proposed based on correlations only. Lu et al. [START_REF] Yu | A fast Intra coding algorithm for spatial scalability in SHVC[END_REF] and [START_REF] Yu | Fast intra-and intercoding algorithms for the spatially scalable extension of H.265/HEVC[END_REF] jointly use texture complexity and spatio-temporal correlation to predict candidate modes.

If a mode is predicted very well, its residual coefficients should be very small and follow the Gaussian distribution [START_REF] Pao | Modeling DCT coefficients for fast video encoding[END_REF] or the Laplacian distribution [START_REF] Hu | Fast motion estimation based on confidence interval[END_REF]. The research in [START_REF] Sun | Fast Depth and Mode Decision in Intra Prediction for Quality SHVC[END_REF] also first checks ILR mode and then calculates its part-zero block based on the distribution of its residual coefficients to early terminate mode selection. Wang et al. [START_REF] Zhu | Efficient multi-strategy intra prediction for quality scalable high efficiency video coding[END_REF] first check ILR mode and then decide whether its residual coefficients follow Gaussian distribution so as to early skip Intra mode. Pan et al. [START_REF] Kwong | Early MERGE mode decision based on motion estimation and hierarchical depth correlation for HEVC[END_REF] combine depth correlation and all-zero block to early terminate mode selection. Obviously, a mode's probability of being selected as the best mode is strongly related to mode selection. Wang et al. [START_REF] Sun | Probability-Based Fast Intra Prediction Algorithm for Spatial SHVC[END_REF] combine probabilities of ILR mode with its residual coefficients to early skip Intra mode.

Although all the above algorithms can improve coding speed, the underlying mechanisms of mode selection have not yet been investigated, which hinders the further improvement of SSHVC coding speed. Therefore, we develop a novel mode-distribution-based fast Intra prediction algorithm for SSHVC.

JUSTIFICATION OF THE PROPOSED MODE SELECTION ALGORITHM

In order to improve the coding speed, we have conducted extensive experiments to investigate Intra coding in SSHVC. In order to ensure the generality of features and rules, different types of sequences, including motion and texture from simple to complex, are selected. More specifically, we use Blue-sky, Ducks, Park Joy, Pedestrian, Tractor, Town and Station2 in our experiments. According to common SHM test conditions (CSTC) [START_REF]Common SHM Test Conditions and Software Reference Configurations[END_REF], QPs set in BL and In EL are set as (22, 26, 30, 34) and (24, 28, 32, 36) in conducting experiments, respectively.

We can obtain the RD costs of ILR mode and Intra mode, which are listed in Table 1.

From Table 1, we can observe that the average values of the RD costs of ILR mode at depths from 0 to 2 are significantly larger than those of Intra mode. In contrast, for depth 3, the average value of the RD costs of ILR mode is smaller than that of Intra mode, but not very significantly.

In addition to the RD cost relationships of ILR mode and Intra mode, we further investigate their RD cost distribution. We have conducted extensive experiments on the RD cost distribution of ILR mode and Intra mode. In Fig. 1, the horizontal axis represents RD cost, and the vertical axis represents the histograms, i.e., the corresponding number of CUs in each bin.

GMM-EM-BASED MODE SELECTION

It is observed from Fig. 2 that, despite the different average RD costs, the distribution for the ILR mode overlaps with that for the Intra mode. Therefore, it is impossible to directly decide which is the better choice. GMM-EM is very suitable for classifying the data that follow the Gaussian distribution but have significantly different average values. GMM-EM is a widely used machine learning algorithm for clustering. It uses mixed Gaussian distribution as the parametric model and utilizes the maximum Expectation (EM) algorithm for training. Since the average value of RD costs of both ILR mode and Intra mode follow Gaussian distribution, we add them into a mixed Gaussian distribution model. Based on the model, the probability of sample i belonging to each part can be derived based on the currently available mixture parameters. Then, the mixture parameters are refined. Repeating the process until converge, then the probabilities of all samples belonging to each part can be obtained.

Since ILR mode is more likely to be selected as the best mode [START_REF] Sun | Fast Depth and Mode Decision in Intra Prediction for Quality SHVC[END_REF] [12], we encode the current CU using ILR mode first and then use GMM-EM to determine whether ILR mode is the best mode based on its RD cost. In the affirmative, we can early terminate mode selection; otherwise, we need to further check Intra mode. As shown in Fig. 3, U 0 represents the current CU, U 1 , U 2 , U 3 and U 4 are the neighbouring CUs of the current CU, U 5 , U 6 , U 7 , U 8 and U 9 are the CUs of U 0 , U 1 , U 2 , U 3 and U 4 in the previous frame. Since there are only two parts, i.e., ILR mode and Intra mode, let part one represent ILR mode, and part two indicates Intra mode. The corresponding Gaussian Mixture Model can be written as:

p (rd|π, µ, Σ) = π 1 N (rd|µ 1 , Σ 1 ) + π 2 N (rd|µ 2 , Σ 2 ) , (1)
where rd is the RD cost of a CU, π 1 is the probability of relevant CUs using ILR mode, µ 1 and Σ 1 are respectively the expected value and the variance of the RD costs of these CUs selecting ILR mode as the best mode; and π 2 is the probability of relevant CUs using Intra mode, µ 2 and Σ 2 are respectively the expected value and the variance of RD costs of these CUs selecting Intra mode as the best mode.

In order to obtain these six parameters, based on Eq. ( 1), the corresponding maximum likelihood estimation function is:

f = M i=1 p (rd|π, µ, Σ) = M i=1 (π1N (rd|µ1, Σ1) + π2N (rd|µ2, Σ2)), ( 2
)
where M is the number of the current CU and its relative CUs, and it is equal to 10.

The logarithm of the maximum likelihood function is:

log (f ) = M i=1 log (π 1 N (rd i |µ 1 , Σ 1 ) + π 2 N (rd i |µ 2 , Σ 2 )), (3)
where π k , µ k and Σ k (k=1 or 2) can be calculated by:

∂ log (f ) ∂π k = 0, ∂ log (f ) ∂µ k = 0, ∂ log (f ) ∂Σ k = 0. ( 4
)
We can derive:

µ k = 1 N k N i=1 γ (i, k)rdi, Σ k = 1 N k N i=1 γ (i, k) (rdi -µ k ) (rdi -µ k ) , (5)
where

N k = N i=1
γ (i, k), then we have:

π k = N k N . (6)
γ (i, k) is the probability that the i-th CU, denoted as U i , belongs to the k-th part, and it can be obtained by:

γ (i, k) = π k N (x i |µ k , Σ k ) π 1 N (x i |µ 1 , Σ 1 ) + π 2 N (x i |µ 2 , Σ 2 ) . (7)
In order to start iterations, we must obtain their initial values first. For relative CUs selecting ILR mode as the best mode, we calculate the expected value and variance of their RD costs as initial µ 1 and Σ 1 , and set the ratio between the number of these CUs and that of all the relevant CUs as initial π 1 . Using the same way, we can obtain initial µ 2 , Σ 2 and π 2 . After these six values are obtained, we use Eq. ( 7) to derive initial γ (i, k). Repeat iterations ( 5), ( 6) and ( 7) until γ (i, k) converges.

As mentioned above, the current CU is U 0 and ILR mode is part one, hence γ (0, 1) denotes the probability of the current CU selecting ILR mode as the best one. We denote the i-th iteration of γ (0, 1) as γ i (0, 1). In order to avoid unnecessary iterations, if the absolute difference between γ i-1 (0, 1) and γ i (0, 1) is small enough, we can terminate the iteration. We empirically select 0.01 as the threshold, then we obtain the early termination condition below:

|γ i (0, k) -γ i-1 (0, k) | ≤ 0.01. (8)
If condition ( 8) is met, we can terminate the iteration and obtain the probability of the current CU selecting ILR mode as the best mode. Since this probability is obtained based on RD cost, we define it as the RD-based ILR probability.

Since relevant CUs are usually very similar, we can use them to predict the probability of the current CU using ILR mode. Obviously, if more relevant CUs use ILR mode, the current CU is more likely to use this mode, and vice versa. In other words, the probability of the current CU using ILR mode is proportional to the number of relevant CUs using ILR mode. As shown in Fig. 3, there are 9 relevant CUs, so we simply set the possibility of the current CU using ILR mode as k 9 , where k is the number of relevant CUs using ILR mode. Since this probability is obtained based on the number of relevant CUs using ILR mode, we define it as the number-based ILR probability.

Since both the RD-based ILR probability and the numberbased ILR probability strongly relate to ILR mode selection, we combine them to further predict the probability of the current CU using ILR mode. Let p(A) and p(B) denote the RDbased ILR probability and the number-based ILR probability, respectively. Since both of them are independent, we further derive the probability of the current CU selecting ILR mode as the best mode, p r , by:

p r = p (A + B) = p(A) + p(B) -p(A)p(B). (9)
Theoretically, if p r ≥ 0.95, the current CU is very possible to use ILR mode. Therefore, we select 0.95 as the threshold value for p r .

EXPERIMENTAL RESULTS

In order to assess the performance of the proposed Mode Selection-based fast Intra prediction algorithm for SSHVC, we use the reference software (SHM 11.0) and test the proposed algorithm on a server with Intel (R) 2.0 GHz CPU and 30 GB memory. The performances of algorithms are evaluated by coding efficiency and coding speed. Coding efficiency is measured by bitrate and visual quality together, which is indicated by BDBR [START_REF] Bjontegaard | Calculation of Average PSNR Differences between RD-curves[END_REF]. It refers to the bitrate difference at an equal PSNR compared with the reference software in EL. Coding speed improvement is denoted by TS, which is the percentage of encoding run-time savings only in EL.

In order to demonstrate the performance of the proposed algorithm, we conduct a performance evaluation of our algorithm with FIICA [START_REF] Yu | Fast intra-and intercoding algorithms for the spatially scalable extension of H.265/HEVC[END_REF] and PBFIP [START_REF] Sun | Probability-Based Fast Intra Prediction Algorithm for Spatial SHVC[END_REF]. To the best of our knowledge, these two algorithms are the best and most recent relative algorithms for SSHVC. For fair comparisons, all algorithms are tested on the same computing platform. According to the CSTC [START_REF]Common SHM Test Conditions and Software Reference Configurations[END_REF], the QPs in the BL are set to (22, 26, 30, 34), and the corresponding QPs in the ELs are set to (22, 26, 30, 34) and (24, 28, 32, 36), respectively. We denote In Table 2 (case 1), the average BDBRs of the proposed algorithm, FIICA and PBFIP are 0.36%, 0.38 and 0.58%, respectively. While the average TS of the proposed algorithm, FIICA and PBFIP are 53.65%, 42.22% and 41.27% correspondingly. In Table 3 (case 2), the average BDBRs of the proposed algorithm, FIICA and PBFIP are 0.31%, 0.40% and 0.43%, respectively. While the average TS of the proposed algorithm, FIICA and PBFIP are 59.39%, 43.25% and 47.81% correspondingly. Compared with the other two algorithms, we can observe that the average BDBRs of the proposed algorithm are smaller, meanwhile, the average TSs of the proposed algorithm are significantly faster in both cases.

CONCLUSION

In this paper, we fully investigate the special features of SSHVC and propose a new Mode Selection-based Intra prediction algorithm for SSHVC. Experimental results show that the proposed algorithm can improve the coding speed significantly with negligible coding efficiency losses.

  This work was supported in part by the National Natural Science Foundation of China under Grant U19A2052, Grant 62020106011; in part by the Natural Science Foundation of Chongqing under Grant cstc2020jcyj-msxmX0766; in part by the Science and Technology Research Program of Chongqing Municipal Education Commission under Grant KJZD-K202100604; and in part by the Shangrao Basic Research Program under Grant 2021F003.

  Fig. 1 (a) and Fig. 1 (b) show the RD cost distribution of ILR mode for sequence "Blue sky" in depth 2 and depth 3, respectively. From Fig. 1, we can observe that both the RD costs of ILR in depths 2 and 3 follow the Gaussian distribution. Our extensive experimental results have demonstrated that the RD cost distribution of both ILR mode and Intra mode in all sequences follows Gaussian distribution.

Fig. 1 .

 1 Fig. 1. The RD cost distribution of ILR mode for sequence "Blue sky"

3 Fig. 2 .

 32 Fig. 2. RD cost distributions of ILR and Intra modes at different depth levels.

Fig. 3 .

 3 Fig. 3. The current CU and its relevant CUs

Table 1 .

 1 The RD Costs of ILR Mode and Intra Mode in All Depths

	Sequence	Depth 0 ILR Intra ILR Intra ILR Intra ILR Intra Depth 1 Depth 2 Depth 3
	Blue sky 49801 11074 12172 3198 3039 1523 778 946
	Ducks 140109 49873 34478 13686 8701 5732 2247 2444
	Park Joy 166607 21382 41191 5616 10308 3066 1888 2621
	Pedestrian 37458 26252 9333 7063 2351 2189 609 763
	Tractor 57344 22816 14445 5734 3719 2433 988 1189
	Town 108752 63737 27495 16467 6875 4908 1721 1981
	Station2 44750 12625 11201 4611 2839 2300 742 901
	Average 86403 29680 21474 8054 5405 3164 1282 1549

Table 2 .

 2 Overall performance comparisons with case 1

	Sequence	Proposed (%) FIICA(%) [9] PBFIP(%) [14] BDBR TS BDBR TS BDBR TS
	Traffic	-0.04 58.21 0.41 36.37 0.08 44.61
	PeopleOnStreet 0.23 55.44 0.10 39.43 0.03 40.30
	Kimono	-0.23 65.42 -0.13 60.27 -0.22 53.30
	ParkScene	-0.17 62.51 0.22 36.49 -0.16 32.32
	Cactus	0.59 53.23 0.89 37.92 0.91 39.41
	BasketballDrive 0.96 46.76 0.71 41.48 2.09 45.61
	BQTerrace	1.18 33.95 0.50 43.56 1.30 33.30
	Average	0.36 53.65 0.38 42.22 0.58 41.27
	QP sets of (22, 26, 30, 34) and (24, 28, 32, 36) for EL as case
	1 and case 2, respectively.

Table 3 .

 3 Overall performance comparisons with case 2

	Sequence	Proposed (%) FIICA(%) [9] PBFIP(%) [14] BDBR TS BDBR TS BDBR TS
	Traffic	-0.18 64.02 -0.30 37.89 0.01 45.82
	PeopleOnStreet -0.29 62.66 -0.23 40.15 -0.23 42.65
	Kimono	-0.30 66.29 0.19 60.18 -0.21 59.21
	ParkScene	-0.15 65.04 0.11 38.13 0.11 41.81
	Cactus	0.41 60.33 0.70 39.29 0.52 48.09
	BasketballDrive 0.96 54.42 1.72 42.74 1.72 50.45
	BQTerrace	1.69 42.95 0.61 44.37 1.09 46.62
	Average	0.31 59.39 0.40 43.25 0.43 47.81


Rodrigo B Pinheiro

Jean-Eudes Marvie

Giuseppe Valenzise

Frédéric Dufaux

NF-PCAC: NORMALIZING FLOW BASED POINT CLOUD ATTRIBUTE COMPRESSION

Keywords: Point clouds, Learning-Based, Compression, Attributes, Normalizing Flow

Learning-based point cloud (PC) compression is a promising research avenue to reduce the transmission and storage costs for PC applications. Existing learning-based methods to compress PCs have mainly focused on geometry and employ variational autoencoders to learn compact signal representations. However, autoencoders leverage low-dimensional bottlenecks that limit the maximum reconstruction quality, even at high bitrates. In this paper, we propose a different and novel approach to compress PC attributes by using normalizing flows. Since normalizing flows model invertible transforms, the proposed approach can achieve better reconstruction quality than variational autoencoders over a large range of bitrates. Our Normalizing Flow-based Point Cloud Attribute Compression (NF-PCAC) outperforms previous learning-based methods for attribute compression, and has comparable performance as G-PCC v.14, showing the potential of this scheme for PC compression.

INTRODUCTION

In the past few years, video content consumption has been evolving towards immersive formats [START_REF] Valenzise | Immersive Video Technologies[END_REF], in particular for entertainment. For example, the most recent Olympic Games in Tokyo (2021) featured a 3D replay that brought the spectator closer to the action. Video games new trend is using virtual reality goggles and devices to reproduce 3D worlds and make the experience as immersive as possible. In this context, Point Clouds (PCs) are one of the most popular volumetric representations. PCs are a set of unordered points in space that contain the geometry information, the spatial location of the points in the x, y and z axes, and the associated attribute information, in most cases, color. However, PCs can contain millions of points and are thus expensive to store and transmit. For this reason, to make PCs a viable option to diffuse 3D content, compression is necessary.

The field of PC compression has significantly advanced in the past few years, notably thanks to the standardization efforts in MPEG [START_REF] Graziosi | An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)[END_REF]. In addition to conventional coding techniques, recently a number of learning-based techniques have been proposed for PCs [START_REF] Quach | Survey on Deep Learning-based Point Cloud Compression[END_REF]. However, the majority of these techniques focus on the compression of the PC geometry, but not the attributes. In addition, existing methods use variational autoencoders to learn low-dimensional representations for compression. While this is very effective at low bitrates, the intrisically lossy nature of autoencoders limits the maximum reconstruction quality at higher bitrates.

In this paper, we propose the first end-to-end learning based approach that makes use of a normalizing flow architecture to encode the PC attributes. We call the proposed approach NF-PCAC for Normalizing Flow Point Cloud Attribute Compression. Normalizing flows are neural networks that model invertible transforms. In contrast to variational autoencoders (VAE), these architectures do not have a lowdimensional bottleneck and can, in principle, achieve lossless reconstructions. We adapt the 2D normalizing flow architecture to take into consideration the 3D nature and the sparsity of PCs and we add some approximations to obtain a better trade off between quality and bitrate. Thus our architecture is no longer fully invertible, but produces state-of-the-art performance and provides higher coding gains than existing learning based attribute compression approaches. It is also the first learning based approach that achieves comparable and in some cases, even better results than G-PCC for attributes.

RELATED WORK

The baseline for today's research on PC static compression is the Geometry-based PC Compression (G-PCC) algorithm standardized by MPEG [START_REF] Graziosi | An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)[END_REF]. G-PCC encodes the geometry and attributes separately. Geometry coding uses an octree approach to code the PC, whereas attribute coding is performed by a Region Adaptive Hierarchical Transform (RAHT) [START_REF] De Queiroz | Compression of 3d point clouds using a region-adaptive hierarchical transform[END_REF].

Following the recent success of learning-based coding for 2D images (e.g., [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF]), variational autoencoders have been also applied to the compression of PCs, in particular for geometry. In [START_REF] Quach | Learning convolutional transforms for lossy point cloud geometry compression[END_REF], the authors use 3D voxel convolutions, and cast the decoding problem as one of classifying which voxels are nonempty. This initial architecture led to some extensions and improvements, [START_REF] Quach | Improved deep point cloud geometry compression[END_REF][START_REF] Wang | Multiscale point cloud geometry compression[END_REF]. In particular, [START_REF] Wang | Sparse tensor-based multiscale representation for point cloud geometry compression[END_REF] introduces the use of sparse convolutions for PC compression, allowing the input to be the entire PC instead of just a partition.

Learning-based coding of PC attributes has been less ex- plored. In [START_REF] Quach | Folding-based compression of point cloud attributes[END_REF], attributes are interpreted as a signal over a 2D manifold and are coded using a conventional image codec. However, this mapping fails when the underlying PC geometry is complex, limiting the coding performance. Deep-PCAC [START_REF] Sheng | Deep-PCAC: An end-to-end deep lossy compression framework for point cloud attributes[END_REF] employs second-order point convolutions instead of voxel-based convolution, inspired by [START_REF] Qi | Pointnet++: Deep hierarchical feature learning on point sets in a metric space[END_REF], and a similar architecture to [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF] to code the color attributes of PCs. A limitation of point-based convolution is the inability to capture spatial dependencies, which leads to relatively poor coding performance. A recent method proposed in [START_REF] Wang | Sparse tensor-based point cloud attribute compression[END_REF] consists of an extended version of [10] using a similar architecture and sparse voxel convolutions to compress the attributes of PCs. The use of sparse convolutions is interesting for PCs learning based tasks because it can help overcome the limitations of available memory as well as avoid the dilation of the features to empty spaces [START_REF] Graham | 3d semantic segmentation with submanifold sparse convolutional networks[END_REF]. This work is currently achieving state-of-the-art results for learning-based attribute coding. Normalizing flow architectures have initially been proposed as generative models [START_REF] Dinh | Density estimation using real NVP[END_REF] and later applied to image compression [START_REF] Helminger | Lossy image compression with normalizing flows[END_REF]. An interesting property of normalizing flows is that they produce invertible representations, similar to the orthogonal transforms used in conventional coding. To obtain competitive performance at low bitrates, the authors of [START_REF] Xie | Enhanced invertible encoding for learned image compression[END_REF] introduce attentive squeezing layers to reduce the dimensionality of the latent representations and increase the compression ratio of images. In this paper, we adapt this architecture to the case of 3D point cloud attributes.

PROPOSED METHOD

Normalizing Flow PC Attribute Compression

We propose a novel method to compress PC attributes using a normalizing flow based architecture. Normalizing flow architectures transform the input signal using a diffeomorphic and orientation preserving function [START_REF] Ruthotto | An introduction to deep generative modeling[END_REF]. A function is diffeomorphic if it is completely invertible and differentiable. In other words, the transformation necessary to decode the original data from the latent space consists in inverting the op-erations of the encoder, with the same trained weights. To obtain a bijective mapping, the latent space must have the same dimension as the original data space. This is in contrast with variational autoencoders, where an information bottleneck forces the latent space to be low-dimensional. Initially proposed for image generation [START_REF] Dinh | Density estimation using real NVP[END_REF], normalizing flows have been less popular for lossy image compression [START_REF] Helminger | Lossy image compression with normalizing flows[END_REF][START_REF] Xie | Enhanced invertible encoding for learned image compression[END_REF], where the main paradigm is still VAE [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF]. Nevertheless, normalizing flows have the potential to provide better performance than VAE at higher bitrates, avoiding or reducing the typical quality saturation in autoencoders.

To achieve good compression performance over a large range of bitrates, we propose the NF-PCAC scheme depicted in Figure 1, which extends the architecture presented in [START_REF] Xie | Enhanced invertible encoding for learned image compression[END_REF] to point cloud attributes. It contains a feature enhancement block, an invertible neural network (INN), a channel average block and an attentive layer, followed by a hyperprior model [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF] for the entropy coding of the coefficients in the latent space.

The feature enhancement block, composed of dense blocks as defined in [START_REF] Huang | Densely connected convolutional networks[END_REF], has the goal to help extract more non-linear features from the original PC. The use of dense blocks with skip connections helps preserve the original features in the output of the block, and also transforms the signal so that it can be better encoded. We extend this block to use sparse 3D convolutions instead of 2D regular convolutions to increase memory efficiency and avoid noise in feature computation due to convolutions with empty voxels.

The INN is the core of the architecture as it computes the latent features. It is composed by completely invertible operations, meaning that the output of this block when processed by the reverse operations will result in the original input in a lossless fashion. It is constituted of 3 repetitions of the sequence: 3D voxel squeeze layer, sparse 1 × 1 convolution and sparse coupling layers. The coupling layers consist on a sequence of operations as defined in [START_REF] Dinh | Density estimation using real NVP[END_REF], designed to obtain a bijective function that is flexible and easy to invert.

Following the INN and the average channel layer (see be-low), we have an attentive layer [START_REF] Cheng | Learned image compression with discretized Gaussian mixture likelihoods and attention modules[END_REF] that consists of convolutions followed by a sigmoid function with the goal of helping the architecture focus on more important areas in the PC. For the entropy modeling, we use a hyperprior to generate a mean and a scale for the distribution such as [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF][START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF].

3D squeeze layer

The normalizing flow architecture needs to be adapted to take into account the sparse nature of 3D PCs. A key module in the architecture is the squeeze layer to effectively reduce the spatial dimension and increase the number of channels, without changing the overall dimensionality of the feature space. This step is essential to increase the receptive field for the convolutions, and the squeeze operation serves this purpose in the normalizing flow, converting spatial information into channel information and using it for subsequent convolutions.

The squeezing operation trades spatial size for channels by implementing a masking scheme for 2D images [START_REF] Dinh | Density estimation using real NVP[END_REF], where the features in a 2 × 2 neighborhood are all aggregated as channels of the first spatial location, reducing the number of pixels by a factor of 4 and increasing the number of features also by a factor of 4. The extension for 3D data comes naturally by just increasing the neighborhood to a 2 × 2 × 2 region, reducing the number of pixels by a factor of 8 and increasing the number of channels by the same factor (see Figure 2). However, this naive extension in a sparse PC would generate empty channels, impacting subsequent convolutions, and increasing the number of coefficients, since we would add new voxels to the original data. To reduce the number of coefficients, a channel averaging layer is typically used after the normalizing flow blocks. This layer reduces the number of channels by a factor of 2 by taking the average values of these channels. However, by using this strategy alone in the architecture we worsen the problems caused by the empty channels produced in the squeezing layers. These empty channels cause the learning process to suffer a big slowdown, since they do not contribute any information.

To solve this problem we develop a 3D voxel squeeze layer, derived from the squeezing operation presented in [START_REF] Dinh | Density estimation using real NVP[END_REF]. The newly proposed layer uses artificially filled points in the PC to replace the zeros and improve the convergence of the algorithm, reducing at the same time the artifacts due to filtering across empty voxels. Specifically, we propose to fill the empty voxels using the average attribute values of all the occupied voxel neighbors in the region to be squeezed. An example illustration of empty voxel filling is given in Figure 2b, where we have an average value of (1+4+6+7)/4 = 4.5 and the output of the squeeze layer is displayed on the right side of the arrow in Figure 2b.

EXPERIMENTS AND RESULTS

In this section we detail the training procedure and present a comparison between our method and other learning-based attribute compression methods, as well as the last available version of G-PCC (TMC13v14). For all the cases we assume the geometry is losslessly coded in a different bitstream.

Training

The training set is a mix of different PCs that represent real characters from 8i [START_REF] D'eon | 8i voxelized full bodies -a voxelized point cloud dataset[END_REF], owlii [START_REF] Xu | Owlii dynamic human mesh sequence dataset[END_REF], volucap and xdprod [START_REF] Schaefer | Volucap and XD Productions Datasets[END_REF], as well as PCs generated from axyz [START_REF]Axyz design 3d people models and character animation software[END_REF] textured mesh bundles, sampled to produce PCs of roughly 1,000,000 points, a comparable density to the other PCs in our dataset. We retain for tests and validation the PCs Soldier [START_REF] D'eon | 8i voxelized full bodies -a voxelized point cloud dataset[END_REF], Basketball Player [START_REF] Xu | Owlii dynamic human mesh sequence dataset[END_REF], Thomas [START_REF] Schaefer | Volucap and XD Productions Datasets[END_REF], and Facade [START_REF] Graziosi | An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)[END_REF] while all the other sequences are used for training. We repeat the same protocol using Longdress and Redandblack [START_REF] D'eon | 8i voxelized full bodies -a voxelized point cloud dataset[END_REF], as test set for a broader validation. We choose a percentage of the frames in each sequence to be used in our framework. To compensate for the lack of available data for training, an octree partition of the chosen frames was performed, generating blocks of size 128 × 128 × 128. With this block size we manage to produce a dataset with enough variety of data and the training process becomes less memory consuming, allowing for larger batches. We obtained 32809 training blocks and 5413 validation blocks. We trained our architecture with no data augmentation for 20 epochs with a learning rate of 1e-4 and a batch size of 8. The training was performed in a Tesla V100 with 32 GB of memory.

Validation

We trained a network with the same architecture in the same conditions for one bitrate point, using a naive squeeze layer instead of our solution to validate our contribution. Our 3D squeeze layer converges to a smaller loss, meaning we obtain better PSNR results with fewer bits per occupied voxel, as seen in Table 1 During inference, to avoid blocking artifacts that occur when coding the divided PC, and since we have a network that is completely convolutional, the input of the network is the complete PC, instead of the blocks as used for training.

We compare the performance of the proposed method with those obtained with a VAE-type architecture similar to the one presented in [START_REF] Wang | Sparse tensor-based point cloud attribute compression[END_REF]. For a fair comparison, we employ the same hyperprior as used in our architecture. We call this method Sparse Tensor PCAC. This baseline was trained with the same dataset, learning rate and batch size. We also present the results of Deep-PCAC, the architecture used in [START_REF] Sheng | Deep-PCAC: An end-to-end deep lossy compression framework for point cloud attributes[END_REF] with the available weights in their official repository. The PSNR results were generated using the MPEG metric software [START_REF] Tian | Updates and integration of evaluation metric software for PCC[END_REF].

Rate-distortion results are presented in Figure 3 for 4 PCs, with the Y-PSNR as quality metric. As observed, our method outperforms all others learning-based methods for similar bitrates. However, when it comes to higher bitrates, G-PCC is still the best performing approach. We also present BD-Rate and BD-PSNR in Table 2, some qualitative results in Figure 4 and the inference time on CPU and GPU in Table 3. G-PCC manages to better represent higher frequencies in smaller areas, such as in the soldier's helmet, which might explain the weaker performance on point clouds Facade and Longdress. This behavior might be due to the average calculations that are performed in our architecture and the channel average layer that is used to reduce the number of channels. However, our method obtains better visual results in the more accentuated contours such as the fingers or the end of the sleeves, whereas in G-PCC these areas are more noisy. 3: Average inference time for test models on CPU (Intel ® . i9-9900K -3.6GHz, 64GB RAM) and on GPU (Nvidia ® . Tesla ® . V100, 32 GB)

CONCLUSION AND FUTURE WORKS

We explore the use of normalizing flows as a tool for compressing point cloud attributes. We first adapt a 2D image coding architecture based on normalizing flows to the case of 3D PCs. Then, we propose a simple yet effective 3D squeeze layer that avoids filtering over empty voxels by filling the nonoccupied space with local attribute averages. More sophisticated designs for the squeeze layer are currently under study.

Despite being a first attempt to replace the widely used VAE with normalizing flows in PC compression, the proposed approach obtains good coding gains compared to previous learning-based compression methods for PC attributes. These preliminary results support the conclusion that invertible transforms such as normalizing flows have a significant potential for PC coding and represent a promising research direction for future work.

Fig. 1 :

 1 Fig.1: Proposed PC attribute coding architecture based on normalizing flows. Compared to a 2D normalizing flow architecture for image coding[START_REF] Xie | Enhanced invertible encoding for learned image compression[END_REF], our scheme is adapted to compress PC attributes: all the convolutions are replaced by sparse convolutions, the squeeze layer is replaced by our novel 3D squeeze layer. The PC goes through a feature enhancement block, a normalizing flow module, a channel average and the attention layers before being coded.

Fig. 2 :

 2 Fig. 2: Squeeze Layer implementation; (a) The original squeeze layer implemented in 2D. (b) The 3D extension of the squeeze layer on sparse data. Here, the voxels in black are empty and are replaced by the average of the non-empty neighbors after the squeezing.

Fig. 3 :Fig. 4 :

 34 Fig. 3: PSNR curves for four different PCs.

  -Y BD-PSNR-Y BD-R-Y BD-PSNR-Y BD-R-Y BD-PSNR-Y Soldier 10.05 -0.

Table 1 :

 1 . Ablation study: 3D squeeze layer vs. naive approach

		Soldier	Basketball	Facade	Thomas
		Naive Ours Naive Ours Naive Ours Naive Ours
	PSNR	35.20 37.90 37.26 40.56 33.00 33.65 32.62 34.38
	Bits per input point 0.59	0.56	0.30	0.18	0.69	0.59	0.43	0.39

Table 2 :

 2 BD-Rate (%), BD-PSNR (dB) -other methods against ours

		G-PCCv14		Sparse Tensor		NF-PCAC (Ours)
	PC size	CPU	CPU	GPU	CPU	GPU
		Enc	Dec	Enc	Dec	Enc Dec	Enc	Dec	Enc Dec
	10-bits	3,23	3,15	5,08	4,06 0,40 0,26 49,19	48,75	1,36 1,25
	11-bits 14,19 11,67 20,47 16,73 1,13 0,76 172,89 171,08 4,08 3,69

Table