Guichem Antoine Simon

Anne Guishaume

Chloé Val Sam

arbitraire et les poses de leurs caméras.

Dans l'ensemble, cette thèse soutient le principe que, dans des régimes où les données sont rares, exploiter la structure du problème permet de développer des architectures à la fois efficaces et performantes. vii challenge my work for the sake of science. I feel very lucky to evolve in an environment where open science occupies such a central place. I would probably be knapping flint right now without the amazing, freely-available work of developers and maintainers of Linux, Python, Conda, CUDA, PyTorch, Py-Torch Lightning, Torch-Points3D, PyTorch Geometric, Git, Latex, Overleaf, Plotly, Weights & Biases. I must also pay my respect to Yannick Kilcher, Tim Scarfe, Kurzgesagt, and Veritasium for their awesome YouTube channels which have been comforting shelters for my darkest procrastination times.

Obviously, I would not be where I am today without the support of people I hold dear. I thank my parents and sisters for supporting me in the foggy endeavor that this thesis

Abstract

Over the past decade, deep learning has advanced the analysis of text, image, audio, and video. More recently, transformers and self-supervised learning have triggered a global competition to train gigantic models on Internet-scale datasets, with massive computational resources. This thesis deals with large-scale 3D point cloud analysis and adopts a different approach focused on efficiency. We introduce methods which improve several aspects of the state-of-the-art: faster training, fewer parameters, smaller compute or memory footprint, and better utilization of realistically-available data. In doing so, we strive to devise solutions towards a more frugal and accessible Artificial Intelligence (AI).

We first introduce a 3D semantic segmentation model that combines the efficiency of superpoint-based methods with the expressivity of transformers. We build a hierarchical data representation which allows us to drastically accelerate the parsing of large 3D point clouds. Our network proves to match or even surpass state-of-the-art approaches on a range of sensors and acquisition environments, while boasting orders of magnitude fewer parameters, with faster training and inference.

We then build on this framework to tackle panoptic segmentation of large-scale 3D point clouds. Existing instance and panoptic segmentation methods do not scale well to large scene with numerous objects because the computation of their loss function implies a costly matching step between true and predicted instances. Instead, we frame this task as a scalable graph clustering problem, which a small network is trained to address from local objectives only, without computing the actual object instances at train time. Our lightweight model can process ten-million-point scenes at once on a single GPU in a few seconds, opening the door to 3D panoptic segmentation at unprecedented scales.

Finally, we propose to exploit the complementarity of image and point cloud modalities to enhance 3D scene understanding. We place ourselves in a realistic acquisition setting where multiple arbitrarily-located images observe the same scene, with potential occlusions. Unlike previous 2D-3D fusion approaches, we learn to select information from various views of the same object based on their respective observation conditions: camera-to-object distance, occlusion rate, optical distortion, etc. Our efficient implementation achieves state-of-the-art results both in indoor and outdoor settings, with minimal requirements: raw point clouds, arbitrarily-positioned images, and their cameras poses.

Overall, this thesis upholds the principle that for settings with limited data availability, exploiting the structure of the problem unlocks both efficient and performant architectures.

Résumé

Au cours de la dernière décennie, l'apprentissage profond a fait progresser l'analyse de texte, d'image, d'audio et de vidéo. Plus récemment, les transformers et l'apprentissage auto-supervisé ont déclenché une compétition généralisée visant à entraîner des modèles gigantesques sur d'immenses jeux de données, au moyen d'énormes ressources de calcul. Cette thèse porte sur l'analyse de nuages de points 3D à grande échelle et adopte une approche différente centrée sur l'efficacité. Nous introduisons des méthodes qui améliorent plusieurs aspects de l'état de l'art : entrainement plus rapide, moins de paramètres, coût de calcul plus faible, plus économe en mémoire et meilleure utilisation des données disponibles de manière réaliste. Ce faisant, nous nous efforçons de concevoir des solutions en vue d'une Intelligence Artificielle (IA) plus sobre et plus accessible.

Nous introduisons d'abord un modèle de segmentation sémantique 3D qui combine l'efficacité des méthodes basées superpoints avec l'expressivité des transformers. Nous construisons une représentation hiérarchique des données qui nous permet d'accélérer considérablement l'analyse de grands nuages de points 3D. Notre réseau se révèle égaler, voire surpasser, les approches de pointe sur une gamme de capteurs et d'environnements d'acquisition, tout en réduisant le nombre de paramètres et le temps d'entrainement de un à deux ordres de grandeur.

Nous étendons ensuite ce cadre à la segmentation panoptique de nuages de points à grande échelle. Les méthodes existantes de segmentation d'instance et de segmentation panoptique ne sont pas adaptées aux grandes scènes comportant de nombreux objets, car le calcul de leur fonction de coût implique une étape fastidieuse d'appariement entre les instances réelles et prédites. Au lieu de cela, nous formulons cette tâche comme un problème de clustering de graphe, qu'un petit réseau est entrainé pour résoudre à partir d'objectifs locaux uniquement, sans nécessiter le calcul d'instances durant l'entraînement. Notre modèle peut traiter des scènes de dix millions de points à la fois sur un seul GPU en quelques secondes, ouvrant la voie à la segmentation panoptique 3D à des échelles sans précédent.

Enfin, nous proposons d'exploiter la complémentarité des modalités image et nuage de points pour améliorer l'analyse de scènes 3D. Nous nous plaçons dans un cadre d'acquisition réaliste, où plusieurs images arbitrairement positionnées observent la même scène, avec de potentielles occultations. Contrairement aux approches existantes de fusion 2D-3D, nous apprenons à sélectionner des informations à partir de différentes vues du même objet en fonction de leurs conditions d'observation respectives : distance caméra-objet, taux d'occultation, distorsion optique, etc. Notre implémentation efficace atteint l'état de l'art tant pour des scènes d'intérieur que d'extérieur, avec des exigences minimales : nuages de points bruts, images positionnées de manière I am very grateful to my advisor Loïc Landrieu for making a researcher out of me. I have seen myself grow as the past 3 years flew by, and I have no one to thank more for this than him.

I would like to thank people whom I have worked closely with. Bruno Vallet for his trust, guidance, and support, always keeping in mind the potential applications of our work. Hugo Raguet for making half my thesis possible with awesome ideas and open code.

I am thankful to Clément Mallet and Ana-Maria Raimond from the LASTIG, and Philippe Calvez and Fabrice Boudaud from the CRIGEN, for welcoming me in their respective labs. This PhD was made possible thanks to funding from the ENGIE Lab CRIGEN, the IGN LASTIG lab, and the ANR project READY3D ANR-19-CE23-0007. A large portion of my work was carried out using HPC resources of IDRIS under the allocation AD011013388R1 made by GENCI.

My stay at IGN would never have been the same without all the great people I met there. I must thank Romain and Yanis for making our office the liveliest during the pandemic. Emile for his generosity, animated chats, and honey waffles. Vivien and Raphael for letting me join their arkose-sandwich sessions and all ensuing fun. Ewelina and Valerio for their constant warmth and kindness. The IGN restaurant's lunches would not have been as tasty without the company of all my labmates, whom will recognize themselves here and forgive my non-exhaustive list.

My occasional visits at CRIGEN were always a pleasure. Thanks to Marcos for his swift help with GPU servers and open-hearted discussions, and thanks Dmitriy and Irene for their enthusiasm for my projects. Doing a PhD has been a lot of fun to me, I heard myself claim in multiple occasions: "I wish it would never end... only with a little salary raise each year !" I am grateful to the many people who have influenced the meandering path that led me here today. MLF for being the first to ever mention "machine learning" and "neural networks" to me. Andrew Ng and Sebastian Thrun for introducing me to all the fun stuff. Simon, Antoine, Hugo L., Alexis F., and Alex G. for helping me see how much I wanted to be a researcher and that there is no age limit to do a PhD. None of the things I have accomplished in this thesis would have been possible without the rest of the scientific community. I want to thank all the (doubleblind) reviewers and the members of my jury for taking the time to assess and Chapter 1 3D point cloud processing finds applications in disciplines as diverse as environment monitoring [START_REF] Smeeckaert | Large-scale classiőcation of water areas using airborne topographic LiDAR data ż[END_REF][START_REF] Xu | LiDAR applications to estimate forest biomass at individual tree scale: Opportunities, challenges and future perspectives ż[END_REF][START_REF] Hauglin | Large scale mapping of forest attributes using heterogeneous sets of airborne laser scanning and national forest inventory data ż[END_REF], city planning [START_REF] Yeung | ń Urban land cover classiőcation using airborne LiDAR data: A review ż[END_REF][START_REF] Wang | ń LiDAR point clouds to 3D urban models : A review ż[END_REF][START_REF] Jiang | ń Industrial applications of digital twins ż[END_REF], and autonomous navigation [START_REF] Li | Deep learning for LiDAR point clouds in autonomous driving: A review ż[END_REF][START_REF] Simegnew | ń A survey on deep-learning-based LiDAR 3D object detection for autonomous driving ż[END_REF]. These domains benefit from the acquisition of large amounts of 3D data and its automated analysis. As both acquisition technologies and computer vision have seen recent advances, deep learning on 3D point clouds has become a promising research direction.

Of late, the prevailing recipe for deep learning on text or images consists in training gigantic models on extensive datasets using ever-growing computational resources. This practice is motivated by the observation that increasing model and dataset sizes leads to gains in performance, albeit with diminishing returns. However, unlike its image processing counterpart, the 3D community does not benefit from web-scale open annotated datasets to train on, which currently hampers the emergence of very large 3D models. To illustrate, the largest open 3D dataset to date Objaverse-XL [START_REF] Deitke | Objaverse: A universe of annotated 3D objects ż[END_REF][START_REF] Deitke | Objaverse-XL: A universe of 10M+ 3D objects ż[END_REF] contains 10 million 3D objects collected from the Web, while its image counterpart LAION-5B [START_REF] Schuhmann | LAION-5B: An open large-scale dataset for training next generation image-text models ż[END_REF] encompasses 5 billion images. In addition, this "bigger is better" trend comes at a high energy cost and progressively excludes practitioners with limited hardware and data resources.

We propose a different stance and call for more sober approaches in the context of 3D point cloud analysis. Specifically, we seek efficient methods accessible to researchers and practitioners alike. These methods should scale to large 3D scenes, without sacrificing performance or efficiency. They will preferably involve small models, with fast training and reasonable hardware requirements. Besides, we favor solutions evaluated on publicly available datasets, providing open-source code for easy reproducibility. In this thesis, we present 3 works that follow these principles. We propose scalable methods for 3D point cloud analysis which are compact and resource efficient, and a multimodal approach capable of leveraging arbitrarily localized images to improve 3D scene parsing.

In this introductory chapter, we first situate this thesis in the context of a recent 3D deep learning history in Section 1.1. From there, we present in Section 1.2 our motivations for working on the efficient analysis of large 3D point clouds, and the corresponding challenges that stand up against our goals in Section 1. 3. Next, we present in Section 1.4 the main contributions of this thesis and summarize our scientific publications and research activities in Section 1.5. Finally, Section 1.6 summarizes the outline of the present document.

Prerequisites. This thesis assumes that the reader is familiar with deep learning, and deep learning for computer vision, in particular. For an introduction to these concepts, we suggest referring to Goodfellow et al . [START_REF] Goodfellow | Deep learning[END_REF].

A Glance at 3D Deep Learning

First, we provide contextual background on the past 10 years of deep learning research and how they have shaped the field of 3D deep learning; from the recent supremacy of artificial neural networks to the wide adoption of Transformer architectures and the revolution of self-supervised learning.

The Rise of Large Neural Networks

Deep learning is a subfield of Machine Learning, which focuses on training deep artificial neural networks. Contrary to other algorithms in the machine learning toolbox [START_REF] Verhulst | ń Mathematical researches into the law of population growth increase ż[END_REF][START_REF] Boser | ń A training algorithm for optimal margin classiőers ż[END_REF][START_REF] Tin | ń Random decision forests ż[END_REF], artificial neural networks do not need carefully engineered input features, but learn to construct useful feature representations directly from raw data [START_REF] Lecun | Gradient-based learning applied to document recognition ż[END_REF]. Inspired first by the study of biological neural networks [START_REF] Warren | ń A logical calculus of the ideas immanent in nervous activity ż[END_REF], early work on artificial neural networks dates back to the mid-20th century [START_REF] Rosenblatt | ń The Perceptron: A probabilistic model for information storage and organization in the brain ż[END_REF]. However, as shown in Figure 1.2, deep learning only gained its current popularity in the early 2010s, when advances in hardware and the availability of large public datasets allowed deep neural network training that outperformed all previous approaches in speech recognition [START_REF] Dahl | Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition ż[END_REF], text processing [START_REF] Collobert | ń Natural language processing (almost) from scratch ż[END_REF], and image classification [START_REF] Krizhevsky | ń Imagenet classiőcation with deep convolutional neural networks ż[END_REF].

From then on, deep learning helped push the boundaries in research fields as diverse as speech-to-text [START_REF] Graves | ń Speech recognition with deep recurrent neural networks ż[END_REF] and text-to-speech [START_REF] Van Den Oord | WaveNet: A generative model for raw audio ż[END_REF] translation, video classification [START_REF] Karpathy | Large-scale video classiőcation with convolutional neural networks ż[END_REF], image generation [START_REF] Goodfellow | Generative adversarial nets ż[END_REF], playing Go [START_REF] Silver | Mastering the game of go with deep neural networks and tree search ż[END_REF], or protein folding prediction [START_REF] Jumper | Highly accurate protein structure prediction with alphafold ż[END_REF]. Important breakthroughs came from methods that enabled the training of deeper networks, such as Rectified Linear Unit (ReLU) [5], batch normalization [START_REF] Ioffe | ń Batch normalization: Accelerating deep network training by reducing internal covariate shift ż[END_REF], or residual connections [START_REF] He | Deep residual learning for image recognition ż[END_REF].

Although artificial neural networks do not account for all the complexity and mechanisms of biological neural networks, deep learning has long been influenced by neurosciences [START_REF] Zhao | When brain-inspired AI meets AGI ż[END_REF]. In particular, the study of neuroplasticity shows that, when a person loses one sense during childhood, brain cells that used to be dedicated to processing the associated input signals are progressively taken over by other senses, indiscriminately [START_REF] Olivier Collignon | Cross-modal plasticity for the spatial processing of sounds in visually deprived subjects ż[END_REF][START_REF] Sathian | ń Cross-modal plasticity of tactile perception in blindness ż[END_REF]. Going even further, the human brain can learn to process and interpret new sensory inputs such as electrotactile signals encoding a camera input sent to the tongue or the skin [START_REF] Eagleman | Can we create new senses for humans?[END_REF]. These findings suggest the possibility of a learning algorithm for training neural networks to process any input sensory signal, without priors on its structure. This underlying idea has been encouraging deep learning research towards a unified, input-agnostic learning architecture [START_REF] Jaegle | Perceiver: General perception with iterative attention ż[END_REF].

For a while, deep learning models mostly relied on modality-and taskspecific building blocks with relatively strong inductive priors. Inductive priors [START_REF] Goyal | ń Inductive biases for deep learning of higherlevel cognition ż[END_REF] (or inductive biases) are assumptions introduced in a learning algorithm to predict outputs for previously unseen inputs. Simply put, these are design choices which force the algorithm to learn one specific pattern over another. There are many ways to encode such biases in an artificial neural network, among which regularization strategies [START_REF] Christopher | Neural networks for pattern recognition[END_REF][START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overőtting ż[END_REF][START_REF] Kukačka | ń Regularization for deep learning: A taxonomy ż[END_REF], architectural restrictions [START_REF] He | Deep residual learning for image recognition ż[END_REF][START_REF] Long | ń Fully convolutional networks for semantic segmentation ż[END_REF][START_REF] Yu | ń Multi-scale context aggregation by dilated convolutions ż[END_REF], parameter sharing [START_REF] Hochreiter | ń Long short-term memory ż[END_REF][START_REF] Pham | Efficient neural architecture search via parameters sharing ż[END_REF], training recipes [START_REF] Hinton | ń A fast learning algorithm for deep belief nets ż[END_REF][START_REF] Erhan | Why does unsupervised pre-training help deep learning? ż[END_REF][START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding ż[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations ż[END_REF], or invariance or equivariance to known transformations [START_REF] Lecun | Convolutional networks for images, speech, and time series ż[END_REF][START_REF] Ravanbakhsh | ń Equivariance through parameter-sharing ż[END_REF][START_REF] Zhang | ń mixup: Beyond empirical risk minimization ż[END_REF][START_REF] Krizhevsky | ń Imagenet classiőcation with deep convolutional neural networks ż[END_REF][START_REF] Finzi | Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data ż[END_REF]. While beneficial for practical use cases, the reliance on inductive biases tailored to specific tasks and modalities does not align with the quest for a unified deep learning algorithm.

Transformers Take Over

In 2017, the introduction of the transformer architecture [START_REF] Vaswani | Attention is all you need ż[END_REF] for neural machine translation significantly impacted the Natural Language Processing (NLP) community. Unlike a Recurrent Neural Network (RNN), which iteratively processes elements of a sequence, the transformer constructs a representation for each element based on its surrounding context, in parallel. This approach outperforms previous methods, while making fewer assumptions, hence having fewer inductive priors. Soon, transformer-based architectures permeated other fields, such as image [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale ż[END_REF], audio [START_REF] Gong | ń AST: Audio spectrogram transformer ż[END_REF], speech [START_REF] Dong | ń Speech-Transformer: A no-recurrence sequence-to-sequence model for speech recognition ż[END_REF], or video [START_REF] Bertasius | ń Is space-time attention all you need for video understanding? ż[END_REF] processing, replacing domain-specific building blocks with more versatile selfattention modules. Pushing this idea further, Perceiver [START_REF] Jaegle | Perceiver: General perception with iterative attention ż[END_REF] proposed a modality-agnostic transformer architecture to process image, audio, video, and point clouds.

While this may have been a bitter lesson for researchers attempting to instill domain expertise into their models [START_REF] Sutton | The bitter lesson[END_REF], today's deep learning community seems to largely share the belief that larger models with as little human-engineered inductive priors are always better, if trained end-to-end on sufficiently large datasets. This strategy is not suitable for 3D computer vision, whose publicly available datasets are smaller, as will be discussed in Section 1.2.2. In this thesis, we uphold the principle that in data-scarce regimes, exploiting the structure of the problem to design strong inductive priors is key to achieving good performance.

The Revolution of Self-Supervised Learning

Humans and animals are able to learn representations about the world from multiple sensory inputs at once without explicit supervision and can generalize concepts from very few examples [START_REF] Carey | ń Acquiring a single new word ż[END_REF][START_REF] Landau | ń The importance of shape in early lexical learning ż[END_REF][START_REF] Markman | Categorization and naming in children: Problems of induction[END_REF]. However, deep learning models trained with supervised learning require large amounts of annotated data to address a single task and tend to poorly generalize to other domains or tasks. For instance, training an image classification model requires a collection of images, each tagged with a class label. The vast majority of available data are gathered from the Internet in the form of text, images, and videos. Yet, these do not come with explicit task-specific annotations for models to learn from, and producing such annotations or labels is costly and labor-intensive. Thus, there is a need for methods capable of learning generalizable representations from readily accessible unlabeled data.

Arguably different from evolutionary and natural learning mechanisms, self-supervised learning [START_REF] Lecun | Self-supervised learning[END_REF] has recently shown promising results for building generalizable concepts from large amounts of raw data. In self-supervised learning, the model is trained on unlabeled data to address a pretext task designed to provide a supervision signal, in the hope of learning generic representations expressive enough to be useful for downstream tasks. The learned representations can then be used as a starting point for fine-tuning a model on a task for which few annotations are available. Pretext tasks for self-supervised learning usually rely on a priori knowledge of the input data structure and are designed to encourage the model to learn semantic representations of the data. For example, in image processing, relevant pretext tasks may consist of predicting the relative position of two image patches [START_REF] Doersch | ń Unsupervised visual representation learning by context prediction ż[END_REF], the order of a sequence of images [START_REF] Misra | ń Shuffle and learn: Unsupervised learning using temporal order veriőcation ż[END_REF], or the rotation angle of an image [START_REF] Gidaris | ń Unsupervised representation learning by predicting image rotations ż[END_REF] as presented in Figure 1. 3. In text processing, a common pretext task is to predict masked words from a sentence [START_REF] Radford | Improving language understanding by generative pre-training ż[END_REF][START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding ż[END_REF]. Of the many self-supervised learning pretext tasks in the literature, two predominant families emerge: masked modeling, in which models aim to reconstruct hidden portions of an input signal, and contrastive learning, in which models are trained to map two distinct views of the same input to similar representations. [START_REF] Gidaris | ń Unsupervised representation learning by predicting image rotations ż[END_REF] is the following. Given an image rotated by a random multiple of 90°, the model is tasked with learning to recover the rotation angle. The authors show that addressing this task requires learning meaningful semantic representations of the image content: the sky is up, the ground is down, eyes are above the nose which is above the mouth, etc. Source: [START_REF] Gidaris | ń Unsupervised representation learning by predicting image rotations ż[END_REF] research since 2020, covering domains as diverse as language [START_REF] Radford | Improving language understanding by generative pre-training ż[END_REF][START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding ż[END_REF][START_REF] Radford | ń Language models are unsupervised multitask learners ż[END_REF][START_REF] Brown | ń Language models are few-shot learners ż[END_REF][START_REF] Touvron | LLaMA: Open and efficient foundation language models ż[END_REF], image [START_REF] Chen | A simple framework for contrastive learning of visual representations ż[END_REF][START_REF] Goyal | Self-supervised pretraining of visual features in the wild ż[END_REF][START_REF] Grill | Bootstrap your own latent-a new approach to selfsupervised learning ż[END_REF][START_REF] Caron | Emerging properties in self-supervised vision transformers ż[END_REF][START_REF] Bao | BEiT: Bert pre-training of image transformers ż[END_REF][START_REF] He | Masked autoencoders are scalable vision learners ż[END_REF], audio [START_REF] Liu | ń Audio self-supervised learning: A survey ż[END_REF], video [START_REF] Schiappa | ń Self-supervised learning for videos: A survey ż[END_REF], and time series [START_REF] Wickstrøm | ń Mixing up contrastive learning: Self-supervised representation learning for time series ż[END_REF] processing. In computer vision, self-supervised representations have been able to match and even surpass representations learned in a fullysupervised fashion on image classification [START_REF] Deng | ń ImageNet: A large-scale hierarchical image database ż[END_REF], provided that model and pretraining datasets are sufficiently large [START_REF] He | Momentum contrast for unsupervised visual representation learning ż[END_REF][START_REF] Caron | Emerging properties in self-supervised vision transformers ż[END_REF][START_REF] He | Masked autoencoders are scalable vision learners ż[END_REF]. Furthermore, the contrastive framework has proven valuable for aligning representations across multiple modalities without explicit supervision, particularly exemplified in the domain of image-text alignment [START_REF] Radford | Learning transferable visual models from natural language supervision ż[END_REF][START_REF] Ramesh | Zero-shot text-to-image generation ż[END_REF][START_REF] Nichol | GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models ż[END_REF][START_REF] Alayrac | ń Flamingo: A visual language model for few-shot learning ż[END_REF][START_REF] Wang | ń Image as a foreign language: Beit pretraining for vision and vision-language tasks ż[END_REF]. These notable advances can be attributed to the recent combination of transformer-based models, vast web-scale datasets, and massive computational resources. In this context, the AI community has seen the appearance of loosely defined foundation models [START_REF] Bommasani | On the opportunities and risks of foundation models ż[END_REF]: large networks pretrained on extensive datasets and serving as the basis for various downstream tasks with little to no supervision.

Self-supervised learning has been central to several breakthroughs in AI

While crucial in text and image processing tasks, training such models demands colossal computational resources and extensive datasets that only a handful of private actors such as OpenAI, Meta, or Google currently have. As an example, to train their GPT-3 [START_REF] Brown | ń Language models are few-shot learners ż[END_REF] language model, OpenAI used 1024 V100 GPUs for 95 GPU-hours and spent $4.6M in compute alone [START_REF] Narayanan | Efficient large-scale language model training on gpu clusters using megatron-lm ż[END_REF]. Nevertheless, the open-source community has been actively pushing to share these essential tools, contributing to their widespread availability and usability.

In this work, we explore 3D computer vision problems where dataset size and compute power tend to be more modest, hindering the emergence of foundation models for 3D understanding. For this reason, we choose to focus on efficient approaches trained with supervised learning.

The 3D Data Boom

In recent years, the field of 3D computer vision has evolved significantly under the influence of 2D deep learning and the growing availability of 3D data. Indeed, advances in 3D sensing techniques like structure-frommotion (SFM) [START_REF] Tomasi | ń Shape and motion from image streams under orthography: A factorization method ż[END_REF][START_REF] Faugeras | The geometry of multiple images: The laws that govern the formation of multiple images of a scene and some of their applications[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF], time-of-flight cameras [START_REF] Oggier | An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (swissranger) ż[END_REF], structured-light cameras [START_REF] Zhang | ń Rapid shape acquisition using color structured light and multi-pass dynamic programming ż[END_REF][START_REF] Freedman | ń Depth mapping using projected patterns ż[END_REF], and LiDAR [START_REF] Albert V Jelalian | Laser radar systems[END_REF][START_REF] Wehr | ń Airborne laser scanningÐan introduction and overview ż[END_REF] have led to the proliferation of affordable 3D acquisition devices capable of capturing the geometry of large scenes, as represented in Figure 1.4. Consequently, the increasing quantity of 3D data calls for scalable and efficient processing methods.

Various 3D deep learning architectures have been proposed for processing point clouds, which may be categorized by the data structure on which they operate. Point clouds can be treated as sets [START_REF] Charles R Qi | PointNet: Deep learning on point sets for 3D classiőcation and segmentation ż[END_REF][START_REF] Charles R Qi | PointNet++: Deep hierarchical feature learning on point sets in a metric space ż[END_REF], rendered images [START_REF] Boulch | SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks ż[END_REF], converted to voxel grids [START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF], or graphs of groups of points [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF]. For a detailed review of the field, the reader is referred to Chapter 2. Like other deep learning fields, the "transformer craze" has permeated the 3D community [START_REF] Zhao | Point Transformer ż[END_REF][START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF].

More recently, self-supervised learning also made inroads into 3D computer vision, with promising avenues including pretraining for downstream 3D reconstruction and registration tasks [START_REF] Xie | PointContrast: Unsupervised pre-training for 3D point cloud understanding ż[END_REF][START_REF] Zhang | Self-supervised pretraining of 3D features on any point-cloud ż[END_REF], 3D masked modeling [START_REF] Qian | ń Pix4Point: Image pretrained transformers for 3D point cloud understanding ż[END_REF], 2D-3D alignment [START_REF] Qian | ń Pix4Point: Image pretrained transformers for 3D point cloud understanding ż[END_REF][START_REF] Hou | ń Mask3D: Pre-training 2D vision transformers by learning masked 3D priors ż[END_REF], and text-3D alignment [START_REF] Nichol | Point-E: A system for generating 3D point clouds from complex prompts ż[END_REF][START_REF] Peng | ń OpenScene: 3D scene understanding with open vocabularies ż[END_REF]1] leveraging 2D as a pivot modality. However, self-supervised learning has yet to impact 3D semantic understanding as it did image processing. Overall, progress in 3D deep learning has been partly driven by insights from 2D deep learning. However, its smaller community, limited datasets, and the challenges of its data structures have hindered the development of foundation models for 3D computer vision (more details in Section 1.3).

The past decade has been eventful for deep learning research, and methods for processing data as diverse as text, image, video, and 3D point clouds have been converging towards more unified transformer-based architectures pretrained in a self-supervised manner. However, the success of these models generally hinges on the availability of enormous datasets and computational resources. In the following Section 1.2, we will discuss the limitations of these trends and our motivation for more efficient 3D deep learning methods.

Motivations

This thesis focuses on efficient methods for analyzing large 3D point clouds.

In the section at hand, we first present domains of application for 3D point cloud processing. Next, we justify the need for efficient approaches, in light of the current trends in 3D deep learning, previously outlined in Section 1.1.

We then introduce a simple typology for algorithmic efficiency which will help us navigate the rest of this work.

Automated 3D Scene Understanding

The semantic analysis of large 3D scenes finds applications in diverse domains, where both private and public actors resort to point clouds for distinct purposes. In the field of autonomous driving, car manufacturers employ 3D scene analysis to empower their automation systems [START_REF] Li | Deep learning for LiDAR point clouds in autonomous driving: A review ż[END_REF], while other industrial companies use it for facility management [4,[START_REF] Anjanappa | ń Deep learning on 3D point clouds for safety-related asset management in buildings ż[END_REF]. Meanwhile, public institutions rely on 3D point cloud processing for various objectives such as urban planning with digital twins [START_REF] Lafarge | ń Creating large-scale city models from 3Dpoint clouds: A robust approach with hybrid representation ż[END_REF][START_REF] Wang | ń LiDAR point clouds to 3D urban models : A review ż[END_REF], natural disaster prevention [START_REF] Yu | ń Big data in natural disaster management: A review ż[END_REF][START_REF] Jaboyedoff | Use of LiDAR in landslide investigations: A review ż[END_REF], or forest inventory [START_REF] Michael | LiDAR sampling for large-area forest characterization: A review ż[END_REF][START_REF] Kalinicheva | Multi-layer modeling of dense vegetation from aerial LiDAR scans ż[END_REF]. Figure 1.5 shows point cloud acquisitions with their potential application domains. These real-world applications typically deal with data scales that exceed the processing capabilities of conventional academic methods. Our aim is to engineer techniques capable of effectively handling vast 3D scenes and extensive objects, such as long pipes in an industrial plant or entire buildings in a city [START_REF] Hu | Towards semantic segmentation of urban-scale 3D point clouds: A dataset, benchmarks and challenges ż[END_REF][START_REF] Nina | ń DALES Objects: A large scale benchmark dataset for instance segmentation in aerial LiDAR ż[END_REF][START_REF] Lin | Capturing, reconstructing, and simulating: The Urbanscene3D dataset ż[END_REF]. In this process, we rely on deep learning, which, as mentioned in Section 1.1, has proven to be a powerful tool for computer vision.

Bigger Is Not Always Better

As presented in Section 1.1, the rapid progress of deep learning in the past decade has been marked by an increase in model size, dataset scale, and computational resources (see Figure 1.6). Although the performance of the model correlates with these three factors [START_REF] Kaplan | Scaling laws for neural language models ż[END_REF], the constant competition for more powerful models sometimes seems reduced to "the bigger the better" [START_REF] Dehghani | ń Scaling vision transformers to 22 billion parameters ż[END_REF], with little consideration of the impacts of this ideology. Yet, the race for training enormous transformer models with self-supervision on massive datasets is not without inconvenience, as we explain in this section.

Laborious Training. Despite their wide adoption, training a transformer-based model is notoriously difficult. First, transformers need more training data to match the performance of their CNN counterparts [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale ż[END_REF]. One explanation could be that their reduced inductive priors make transformers more "datahungry". This observation suggests that models with inductive priors may be necessary to achieve good performance with limited data. Second, optimizing numerous attention layers is challenging, requiring careful architecture design and hyperparameter tuning [START_REF] Toan | ń Transformers without tears: Improving the normalization of self-attention ż[END_REF][START_REF] Shi | Improving transformer optimization through better initialization ż[END_REF][START_REF] Liu | Understanding the difficulty of training transformers ż[END_REF].

Likewise, in spite of its current popularity and exciting results, selfsupervised learning remains challenging. To such extend that training a model to learn representations with self-supervision even calls for a "cookbook" [START_REF] Balestriero | A cookbook of self-supervised learning ż[END_REF]. In particular, designing a relevant pretext task in the hope that the learned features will be useful for other tasks of interest is not trivial. For example, pretext tasks suitable for 3D reconstruction or registration do not necessarily produce good representations for semantic understanding [START_REF] Xie | PointContrast: Unsupervised pre-training for 3D point cloud understanding ż[END_REF][START_REF] Zhang | Self-supervised pretraining of 3D features on any point-cloud ż[END_REF].

Environmental Impact. Deep learning requires considerable computational resources in the form of GPU or TPU clusters. The construction and operation of these systems incur substantial energy costs, carbon emissions, and nonrenewable resource consumption. Keeping with our previously used example, the training of GPT-3 [START_REF] Brown | ń Language models are few-shot learners ż[END_REF] on 1024 V100 GPUs emitted 552 tCO 2 e [START_REF] Patterson | Carbon emissions and large neural network training ż[END_REF],

which matches the annual CO 2 emissions of 62 French citizens in 2021 [START_REF]Ministère de la Transition Ecologique et de la Cohésion des Territoires. L'empreinte carbone de la france de 1995 à[END_REF].

For this reason, the rapid growth of deep learning has been characterized by a negative environmental impact, which is slow to be taken into account by the community [START_REF] Ligozat | Unraveling the hidden environmental impacts of AI solutions for environment ż[END_REF][START_REF] Wu | Sustainable AI: Environmental implications, challenges and opportunities ż[END_REF][START_REF] Savazzi | An energy and carbon footprint analysis of distributed and federated learning ż[END_REF]. In this context, the general trend to blindly increase model size and compute budget for (often marginally) improved performance is not sustainable and calls for more sober methods.

Social Impact. The growing disparity in access to computational resources widens the gap between the research capabilities of a select few private entities (e.g. OpenAI, Google, Meta) and the broader scientific community.

For illustration, reproducing a single training experiment of the "efficient foundation language model" LLaMA [START_REF] Touvron | LLaMA: Open and efficient foundation language models ż[END_REF], would take more than two centuries on a single Nvidia V100 GPU. In addition, the achievements of models such as ChatGPT [242] and DALL-E [START_REF] Ramesh | Zero-shot text-to-image generation ż[END_REF] have sparked enthusiasm within society for their widespread distribution. However, the deployment of very large models is, again, challenged by the resources they demand. Therefore, the development of powerful yet efficient models is of prime importance for protecting open academic research and to permit their adoption by laboratories with limited compute or knowledge of deep learning, and by society at large.

Scaling 3D Models. Contrary to image and text, "large" academic point cloud datasets are relatively small. For comparison purposes, the ImageNet dataset [START_REF] Deng | ń ImageNet: A large-scale hierarchical image database ż[END_REF], which is now considered relatively "small" within the realm of 2D computer vision, comprises roughly 10 4 times the number of pixels compared to the popular S3DIS [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF] and ScanNet [START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF] point cloud datasets.

For this reason, the model scaling strategies [START_REF] Kaplan | Scaling laws for neural language models ż[END_REF] used in other fields cannot be directly implemented on 3D deep learning, for lack of comparably large open datasets. Instead, we favor approaches designing smaller architectures with more inductive priors, which are better suited for "low-data" regimes.

To summarize, far from dismissing the power of large transformers and selfsupervised learning on extensive datasets, we propose to focus on developing efficient 3D point cloud analysis methods matching the performance of larger The number of parameters of language models has been experiencing exponential growth over the past years. Although scaling model, dataset, and compute has proven to consistently improve performance [START_REF] Kaplan | Scaling laws for neural language models ż[END_REF], these large language models also have negative environmental and social repercussions. Figure taken from a 2020 Microsoft blog post [START_REF] Rosset | Turing-NLG: A 17-billion-parameter language model by microsoft[END_REF] models, due to environmental, social, and practical considerations.

Five Shades of Efficiency

We propose a simple typology of five properties characterizing the efficiency of a machine learning algorithm. This loose categorization encompasses five key dimensions of efficiency: compute, memory, hardware, data, and human labor.

Memory Efficiency. This aspect refers to the memory footprint of an algorithm. Unless specified otherwise, in the context of deep learning, memory efficiency relates to GPU or CPU memory usage, but it could also refer to disk occupancy. In the general deep learning setup, the GPU is used to perform the bulk of neural network computations, while CPU processes are tasked with asynchronously preparing data batches to be fed to the GPU.

Optimizing memory use may serve several purposes: running on a smaller less-expensive hardware, expanding the model size for higher expressivity, or increasing the batch size for faster training and better generalization. For instance, mixed precision training [START_REF] Micikevicius | Mixed precision training ż[END_REF] reduces the memory footprint of a model by performing less precision sensitive operations on float16 rather than float32. Investigating memory efficiency is key to processing batches of large 3D point clouds whose size can easily exceed the memory capacity of a single GPU.

Compute Efficiency. We refer to the number of operations required to run an algorithm as its computational efficiency. In the context of deep learning, minimizing CPU and GPU operations will usually permit running the model faster, which can benefit time-sensitive applications, or simply reduce experimentation time. However, a more subtle relationship links compute and memory when training neural networks. In fact, the dominant method for training neural networks is the backpropagation algorithm [START_REF] Linnainmaa | ń The representation of the cumulative rounding error of an algorithm as a taylor expansion of the local rounding errors ż[END_REF], which requires storing the output of each operation in memory for the gradient calculation. For this reason, minimizing the number of operations in a neural network will also reduce memory usage during training. Similarly to memory efficiency, compute efficiency is pivotal to extracting meaningful local and contextual information from large 3D point clouds within reasonable time. These categories are not mutually exclusive. Often, improving one aspect of the efficiency of a method will also impact another. For example, a technique to train with fewer data while maintaining performance will be both data efficient and human efficient. In this thesis, our proposed approaches explicitly focus on memory, compute, and data efficiency.

In a nutshell, this work is motivated by the development of 3D deep learning techniques capable of processing real-world large-scale point clouds, while aiming for efficiency in terms of memory, compute, hardware, data, or human labor. Next, Section 1.3 will detail the challenges relative to this goal. Processing 3D point clouds comes with a set of challenges that we detail in this section. Point clouds imperfectly characterize 3D scenes.

Challenges

Unlike meshes, which explicitly describe a surface, point clouds only represent a (potentially noisy) sampling of a real surface. Sparsely sampled regions may discard fine-grained geometric details [START_REF] Shannon | ń Communication in the presence of noise ż[END_REF]. The variety of acquisition techniques results in point clouds with distinct characteristics.

For example, photogrammetric point clouds [START_REF] Tomasi | ń Shape and motion from image streams under orthography: A factorization method ż[END_REF] exhibit distance-dependent precision and present artifacts resulting from erroneous matching. Meanwhile, LiDAR sensors [START_REF] Albert V Jelalian | Laser radar systems[END_REF] are subject to reflections producing outliers. Common to all 3D acquisition methods, the constant angular resolution of the sensor induces a radial distribution of points; objects farther away from the sensor are sampled with fewer points. This phenomenon can be observed in Figure 1.7.

Consequently, extracting meaningful information from point clouds calls for methods robust to uneven density, noisy point coordinates, occlusions, and acquisition artifacts.

The above "generic" challenges are inherent to 3D point clouds and are shared by all point cloud processing methods. In this thesis, we tackle three important 3D computer vision challenges, specific to our ambition to efficiently process large 3D scenes: extracting information at multiple scales, enriching point cloud representations with additional modalities, and 3D processing implementation. In image processing, the extraction of features at multiple scales provides rich representations of the content of an image [START_REF] Peter | ń Fast őlter transform for image processing ż[END_REF][START_REF] Peter | ń The laplacian pyramid as a compact image code ż[END_REF][START_REF] Mallat | ń Group invariant scattering ż[END_REF]. High-resolution (high-frequency) features capture local details, while lower-resolution (low-frequency) features capture contextual, long-range interactions in the image.

Efficient Multiscale Reasoning

Similarly, point cloud processing methods rely on features captured at different scales to characterize 3D shapes [START_REF] Charles R Qi | PointNet++: Deep hierarchical feature learning on point sets in a metric space ż[END_REF][START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF][START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF]. state-of-the-art 3D deep learning methods either rely on a hierarchy of arbitrary point [START_REF] Qian | PointNeXt: Revisiting PointNet++ with improved training and scaling strategies ż[END_REF][START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF] or voxel [START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF] samplings of decreasing resolutions to capture local and contextual information. However, these methods are compute-and memory-intensive and do not scale to large scenes. For instance, KPConv [START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF] can only consume 2 m-radius crops of 3D scenes at once, and Stratified Transformer [START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF] requires four GPUs to process a single room of the S3DIS dataset [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF]. Such approaches do not scale to scenarios that require both local geometric details and longrange interactions to be captured. For example, in an urban mapping [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF] scenario, objects of interest may span a wide range of scales: from the traffic sign characterized by small geometric patterns and its global location on the street, to the large multi-story building, characterized by the aggregation of numerous smaller shapes grouped into one large concept.

This challenge has led us to develop in this thesis a hierarchical data representation, shown in Figure 1.8, that adapts to the geometric complexity of the scene, allowing for compute-and memory-efficient multiscale reasoning.

Efficient Multimodal Fusion

Fusing information from additional modalities such as images can improve the performance of 3D point cloud analysis. Indeed, these two modalities carry complementary information, as point clouds capture the geometry, while images capture the texture and context. Figure 1.9 illustrates this synergy between 3D point clouds and images.

One way of exploiting this complementarity is to colorize point clouds.

Yet, such colorization is often a blackbox preprocessing [188, [START_REF]Faro Freestyle 2 handheld scanner[END_REF]306] which discards a large portion of the dense textural and contextual information carried by the images. Methods capable of directly processing point clouds and images should have access to more information about the scene.

Recent multimodal 2D-3D approaches [START_REF] Dai | ń 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation ż[END_REF][START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF][START_REF] Hu | Bidirectional projection network for cross dimension scene understanding ż[END_REF] propose using a 3D network to process point clouds, and a 2D network to extract image features that are projected onto the point cloud, outperforming methods operating on colorized point clouds. However, we identify two limitations of these approaches. First, their reliance on specific depth cameras or computeintensive meshing operations to map points to pixels does not align with our search for hardware and compute-efficient solutions. Second, they aggregate features from all views of the same object, without taking their observation conditions into account.

In this thesis, we propose a method for efficiently extracting and fusing features from point clouds and an arbitrary number of images in the wild.

Our method does not require any specific sensor or meshing, only raw point clouds, images, and their poses.

Contributions

This thesis introduces two main contributions for the efficient analysis of large-scale 3D scenes.

Lightweight 3D Analysis Our first main contribution is an efficient framework for point cloud semantic analysis, which can match and even surpass state-of- and inference times on a single consumer-grade GPU, our algorithm makes experimentation on 3D point clouds fast and accessible, proving to be both hardware and human-efficient. We validate our approach for semantic [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF] and panoptic [START_REF] Robert | ń Scalable 3D panoptic segmentation as superpoint graph clustering ż[END_REF] segmentation, across multiple datasets, sensor technologies, and acquisition environments. Our method proves to be suitable for capturing long-range dependencies and scales to very large scenes without performance loss. In particular, our semantic segmentation architecture matches or outperforms competing methods on all benchmarks, with up to 200 times fewer parameters and 70 times faster training. For panoptic segmentation, we set a new state-of-the-art on all benchmarks and process of unprecedented size in a few seconds scenes on a single GPU.

Multimodal 2D-3D Analysis Our second main contribution is an architecture capable of jointly extracting information from both 3D point clouds and 2D images. In particular, our method learns to aggregate information from an arbitrary number of views of the same point, based on their viewing conditions, such as angle of view or distance to the sensor.

Unlike similar methods which rely on mesh reconstruction or depth cameras to map points with pixels, we only use raw point clouds and images with poses. Our sparse, parallelized implementation ensures that we can efficiently scale to large 3D scenes with an arbitrary number of images. In addition, by leveraging the knowledge learned from larger image datasets to improve 3D scene understanding without additional 3D annotations, our approach is data and human efficient.

To summarize, this thesis exploits structures in 3D point clouds and multimodal data to devise deep learning architectures achieving high performance, while being efficient and scalable.

Publications and Research Activities

International Conferences. All the works presented in this thesis were published in three international computer vision conferences.

-

Outline

This thesis is organized as follows.

Chapter 1: Introduction. We start by placing this thesis in the larger context of deep learning for 2D and 3D computer vision. Then we discuss our motivations behind this work and the challenges that stand in our way to achieve our goals. Finally, we present the contributions of this thesis and outline the structure of the document. refer the reader to [START_REF] Guo | Deep learning for 3D point clouds: A survey ż[END_REF].

Formally, in our setting, a 3D point cloud refers to a collection of points defined by their (generally Cartesian) coordinates in three-dimensional space.

These points are assumed to form a discrete, unordered, and potentially noisy sampling of a real 3D surface. Optionally, such points may be endowed with additional information such as color, normal vector, or intensity.

Image-Based Methods. Image-based or view-based approaches project 3D point clouds into multiple 2D views and analyze the resulting images with 2D neural networks. First introduced for shape classification by MVCNN [START_REF] Su | Multi-view convolutional neural networks for 3D shape recognition ż[END_REF],

this strategy was extended to 3D object detection [START_REF] Chen | ń Multi-view 3D object detection network for autonomous driving ż[END_REF] and semantic segmentation of large scenes [START_REF] Boulch | SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks ż[END_REF]. Image-based methods conveniently leverage readily-available architectures and easier-to-annotate datasets from the 2D computer vision community. However, some limitations hinder the wide adoption of these methods: rendering occlusions when projecting points necessitates a costly surface reconstruction [START_REF] Boulch | SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks ż[END_REF], aggregating information from multiple views of the same object is nontrivial [START_REF] Johannes | ń Pixelwise view selection for unstructured multi-view stereo ż[END_REF][START_REF] Waechter | ń Let there be color! largescale texturing of 3D reconstructions ż[END_REF], and the projection operation inevitably discards information.

Voxel-Based Methods. Voxel-based (or volumetric) methods convert unstructured 3D point clouds to 3D voxel grids. This straightforward representation adapts successful 2D computer vision methods operating on pixel grids to 3D. VoxNet [START_REF] Maturana | ń VoxNet: A 3D convolutional neural network for real-time object recognition ż[END_REF] introduces a 3D convolutional neural network for processing dense 3D grids, and OctNet [START_REF] Riegler | ń OctNet: Learning deep 3D representations at high resolutions ż[END_REF] uses an octree to mitigate the memory footprint of dense voxel representations. SEGCloud [START_REF] Tchapmi | SEGCloud: Semantic segmentation of 3D point clouds ż[END_REF] circumvents the memory problem by using convolutions on large voxels and a graphical model for subvoxel segmentation. Motivated by the sparsity of voxelized point clouds, MinkowskiNet [START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF] designs a library for sparse 3D convolutions on GPU, making it possible to process large scenes with volumetric methods. Recently, hybrid approaches [START_REF] Liu | Point-voxel CNN for efficient 3D deep learning ż[END_REF][START_REF] Tang | Searching efficient 3D architectures with sparse point-voxel convolution ż[END_REF][START_REF] Zhang | PVT: Point-voxel transformer for point cloud learning ż[END_REF] address the small-scale information loss inherent to voxelization. Although conceptually simple, voxel-based methods tend to be compute-and memory-intensive, and their three-dimensional kernels require more parameters than their 2D counterparts. [START_REF] Li | PointCNN: Convolution on χ-transformed points ż[END_REF][START_REF] Boulch | ConvPoint: Continuous convolutions for point cloud processing ż[END_REF]. Contrary to pixels or voxels, where convolution kernels align with grid-based neighborhoods of constant size and shape, defining a convolution operator for points localized in a continuous space is not straightforward [START_REF] Michael M Bronstein | Geometric deep learning: Grids, groups, graphs, geodesics, and gauges ż[END_REF], due to the varying size and distribution of the neighborhood of points. KPConv [START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF] and ConvPoint [START_REF] Boulch | ConvPoint: Continuous convolutions for point cloud processing ż[END_REF],

Point

for instance, learn convolution kernels which can be placed at any position in a continuous space. Although mathematically elegant, convolution-based methods involve sensitive parameterization, and their computational cost hinders their scalability.

Graph-Based Methods. Graph-based methods treat individual points as graph vertices, connected based on their spatial proximity. Graph processing models are then used to extract features in the spatial [START_REF] Simonovsky | ń Dynamic edge-conditioned őlters in convolutional neural networks on graphs ż[END_REF][START_REF] Wang | ń Dynamic graph CNN for learning on point clouds ż[END_REF] or spectral [START_REF] Te | RGCNN: Regularized graph CNN for point cloud segmentation ż[END_REF] domain. The adjacency graph structure presents a sensible representation for analyzing point clouds and their underlying surface geometry. Nevertheless, graph-based methods are challenged by their restricted connectivity, sequential treatment, limited expressivity, and small receptive fields.

Transformer-Based Methods. The success of the Transformer [START_REF] Vaswani | Attention is all you need ż[END_REF] architecture in natural language processing and its adaptation to image analysis [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale ż[END_REF][START_REF] Liu | Swin Transformer: Hierarchical vision transformer using shifted windows ż[END_REF] have inspired the development of transformer-based methods for point clouds. These models can be seen as a generalization of graph-based approaches to dynamic graphs, where edge weights evolve depending on the node features, allowing for the adaptive aggregation of neighborhood information. 3D vision transformers have demonstrated strong performance on various point cloud analysis tasks [START_REF] Zhao | Point Transformer ż[END_REF] and the ability to capture long-range interactions [START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF]. Yet, the quadratic complexity of their self-attention scheme leads to high memory consumption when applied on dense point clouds, which limits their ability to process large scenes.

Efficient Point Cloud Analysis

As 3D scans of real-world scenes can contain hundreds of millions of points, the efficiency of 3D analysis is critical. In this section, we first provide a high-level overview of research directions for efficient deep learning. Then, we introduce some works specific to efficient 3D point cloud analysis.

Efficient Deep Learning

Inspired by Menghani [START_REF] Menghani | Efficient deep learning: A survey on making deep learning models smaller, faster, and better ż[END_REF], we broadly categorize research directions pursuing efficient deep learning as follows: compression, training, AutoML, implementation, and architecture. Compression. Compression techniques aim to reduce the size of a neural network, with a potential trade-off in performance. As illustrated in Figure 2.2, pruning methods search for non-essential neurons to sparsify the network [START_REF] Lecun | ń Optimal brain damage ż[END_REF][START_REF] Hassibi | ń Optimal brain surgeon and general network pruning ż[END_REF]. Quantization methods reduce the memory cost of neural networks by encoding their activations and weights with low-precision datatypes. This can be done at inference time, by mapping high-precision values to finite sets of discrete values [START_REF] Krishnamoorthi | ń Quantizing deep convolutional networks for efficient inference: A whitepaper ż[END_REF][START_REF] Benoit | Quantization and training of neural networks for efficient integer-arithmetic-only inference ż[END_REF][START_REF] Choukroun | Low-bit quantization of neural networks for efficient inference ż[END_REF]. Alternatively, quantized networks can be trained directly with low precision [START_REF] Hubara | Quantized neural networks: Training neural networks with low precision weights and activations ż[END_REF][START_REF] Li | ń Training quantized nets: A deeper understanding ż[END_REF]. Meanwhile, low-rank compression [START_REF] Idelbayev | Low-rank compression of neural nets: Learning the rank of each layer ż[END_REF] approximates the weight matrix of each layer by the product of two smaller matrices.

Training. Efficiency can be gained by improving the training process itself.

Data augmentation can be seen as an efficient strategy to artificially increase the size of the data set [START_REF] Krizhevsky | ń Imagenet classiőcation with deep convolutional neural networks ż[END_REF][START_REF] Sun | Revisiting unreasonable effectiveness of data in deep learning era ż[END_REF]. Distillation techniques [START_REF] Hinton | ń Distilling the knowledge in a neural network ż[END_REF] use a large, pretrained teacher model to supervise a smaller student network to be deployed in downstream applications. Low-rank adaptation [START_REF] Edward | LoRA: Low-rank adaptation of large language models ż[END_REF] makes the finetuning of large models easier by using a low-rank approximation of the network weights. Departing from traditional supervised learning, weakly supervised [START_REF] Zhou | ń A brief introduction to weakly supervised learning ż[END_REF] makes use of inexpensive or partial annotations. For instance, for the subfield of semi-supervised learning [START_REF] Yang | A survey on deep semi-supervised learning ż[END_REF] only a fraction of data points have annotations, while multi-instance learning [START_REF] Carbonneau | Multiple instance learning: A survey of problem characteristics and applications ż[END_REF] only provides labels for bags of data points. Pushing even further, self-supervised learning [START_REF] Gidaris | ń Unsupervised representation learning by predicting image rotations ż[END_REF][START_REF] Chen | A simple framework for contrastive learning of visual representations ż[END_REF][START_REF] He | Momentum contrast for unsupervised visual representation learning ż[END_REF][START_REF] He | Masked autoencoders are scalable vision learners ż[END_REF] relies on pretext tasks to learn useful representations from data without labels. Typical pretext tasks involve reconstructing the input from a corrupted version of itself [START_REF] He | Masked autoencoders are scalable vision learners ż[END_REF], or pairing transformed versions of the same input [START_REF] Chen | A simple framework for contrastive learning of visual representations ż[END_REF].

AutoML. AutoML techniques propose to minimize human intervention in the model search process. Hyperparameter optimization [START_REF] Bergstra | ń Random search for hyper-parameter optimization ż[END_REF][START_REF] Močkus | ń On bayesian methods for seeking the extremum ż[END_REF][START_REF] Agnihotri | ń Exploring bayesian optimization ż[END_REF] automates the exploration of hyperparameters for a given neural network architecture, while neural architecture search [START_REF] Elsken | ń Neural architecture search: A survey ż[END_REF][START_REF] Tan | ń EfficientNet: Rethinking model scaling for convolutional neural networks ż[END_REF] explores the design space of neural network architectures to find the most efficient one for a given task.

Implementation. The implementation of deep learning algorithms may lead to savings in computation or memory. For example, mixed precision training [START_REF] Micikevicius | Mixed precision training ż[END_REF] only uses high-precision datatypes for sensitive operations, thus reducing the memory footprint of the network. By accumulating the gradients computed on smaller batches, microbatching [START_REF] Huang | GPipe: Efficient training of giant neural networks using pipeline parallelism ż[END_REF] can train large models on limited memory.

Differently, gradient checkpointing [START_REF] Chen | Training deep nets with sublinear memory cost ż[END_REF][START_REF] Bulatov | Fitting larger networks into memory[END_REF] By separating the traditional convolution operation into a per-channel convolution followed by a 1x1 channel-mixing operation, depth-separable convolution [START_REF] Chollet | ń Xception: Deep learning with depthwise separable convolutions ż[END_REF] reduce the number of parameters per convolutional layer.

Source: [START_REF] Tsang | Review: Xception -with depthwise separable convolution, better than Inception-V3 (image classification)[END_REF] Architecture. Designing new building blocks for neural networks can improve model efficiency. Often, introducing inductive priors in the architecture may reduce the number of parameters or operations. For example, by sharing parameters to enforce translation equivariance, convolutional neural networks [START_REF] Lecun | Gradient-based learning applied to document recognition ż[END_REF] are more efficient than fully-connected networks for image processing. Similarly, atrous convolutions [START_REF] Chen | DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs ż[END_REF] assume some local regularity in the features to increase the receptive field of convolutions without increasing the number of parameters. As shown in Figure 2.3, the depth-separable convolutional layers [START_REF] Chollet | ń Xception: Deep learning with depthwise separable convolutions ż[END_REF][START_REF] Sandler | ń MobileNetV2: Inverted residuals and linear bottlenecks ż[END_REF] disentangle spatial and channel-wise correlations by factorizing standard convolutions, with fewer parameters. With the recent popularity of transformer architecture, multiple works have proposed efficient variants [START_REF] Tay | Efficient transformers: A survey ż[END_REF][START_REF] Kitaev | ń Reformer: The efficient transformer ż[END_REF][START_REF] Wang | Linformer: Self-attention with linear complexity ż[END_REF], while ConvNeXt [START_REF] Liu | ń A convnet for the 2020s ż[END_REF] has shown that convolutional networks can still outperform their less efficient transformer counterparts when trained with similar recipes.

While the combination of several of these strategies certainly holds potential, our contribution mainly lies in proposing new architectures for efficient 3D deep learning.

Efficient Deep Learning on Point Clouds

Various architectures for efficiently processing 3D point clouds can be found in the literature. The seminal PointNet [START_REF] Charles R Qi | PointNet: Deep learning on point sets for 3D classiőcation and segmentation ż[END_REF] and PointNet++ [START_REF] Charles R Qi | PointNet++: Deep hierarchical feature learning on point sets in a metric space ż[END_REF] models provide fast, parameter-efficient 3D analysis baselines. RandLANet [START_REF] Hu | RandLA-Net: Efficient semantic segmentation of large-scale point clouds ż[END_REF] demonstrates that efficient sampling strategies can yield excellent results, by replacing the costly Farthest Point Sampling of PointNet++ with simple random sampling. Akin to ConvNeXt [START_REF] Liu | ń A convnet for the 2020s ż[END_REF] mentioned above for images, PointNeXt [START_REF] Qian | PointNeXt: Revisiting PointNet++ with improved training and scaling strategies ż[END_REF] achieves state-of-the-art performance by updating PointNet++ [START_REF] Charles R Qi | PointNet++: Deep hierarchical feature learning on point sets in a metric space ż[END_REF] with modern training recipes, outperforming less efficient transformer-based architectures [START_REF] Zhao | Point Transformer ż[END_REF].

Other works explore efficient data representations for point cloud analysis.

To circumvent the prohibitive memory and compute footprint of 3D convolutions on dense voxel grids, SparseConvNet [START_REF] Graham | ń 3D semantic segmentation with submanifold sparse convolutional networks ż[END_REF] and MinkowskiNet [START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF] operate on sparse representations, conducing voxel-based models capable of processing large scenes while maintaining high resolution. More recently, hybrid point cloud representations [START_REF] Liu | Point-voxel CNN for efficient 3D deep learning ż[END_REF][START_REF] Loiseau | ń Online segmentation of LiDAR sequences: Dataset and algorithm ż[END_REF] have also helped reduce memory cost, while capturing both fine-grained local details and contextual information.

However, by leveraging the local similarity of dense point clouds to construct geometry-informed data structures, superpoint-based methods can achieve an input reduction of several orders of magnitude, resulting in unparalleled efficiency. The next section provides a more in-depth introduction to these methods.

Superpoint-Based Learning

Part of our work draws inspiration from superpoint-based methods to develop efficient and scalable approaches for 3D semantic segmentation and panoptic segmentation. This section provides an overview of image partition methods and these more recently inspired 3D point cloud partition. We then focus on existing superpoint-based approaches for semantic segmentation, which serve as the basis for our methods in Chapter 3 and Chapter 4.

Superpixel Partitioning

Partitioning images into superpixels has been extensively studied to simplify image analysis, both before [START_REF] Ren | ń Learning a classiőcation model for segmentation ż[END_REF]3] and after [START_REF] Tu | Learning superpixels with segmentation-aware affinity loss ż[END_REF][START_REF] Jampani | ń Superpixel sampling networks ż[END_REF] the widespread use of deep learning. By decomposing images into meaningful parts, superpixels offer a computationally efficient representation, which has been applied to various tasks such as object detection [START_REF] Shu | ń Improving an object detector and extracting regions using superpixels ż[END_REF][START_REF] Yan | ń Object detection by labeling superpixels ż[END_REF], semantic segmentation [START_REF] Gadde | ń Superpixel convolutional networks using bilateral inceptions ż[END_REF][START_REF] Sharma | ń Recursive context propagation network for semantic scene labeling ż[END_REF], and depth estimation [START_REF] Van Den Bergh | ń Depth seeds: Recovering incomplete depth data using superpixels ż[END_REF].

Superpixel algorithms can be categorized into graph-based and clusteringbased approaches. Graph-based methods formulate superpixel segmentation as a graph partitioning problem, with nodes corresponding to pixels and edges representing the connectivity between adjacent pixels. Popular algorithms Here, two images are partitioned using SLIC with different resolutions. Source: [3] in this category include normalized cuts [START_REF] Ren | ń Learning a classiőcation model for segmentation ż[END_REF], Felzenszwalb and Huttenlocher [START_REF] Pedro | ń Efficient graph-based image segmentation ż[END_REF], energy minimization [START_REF] Boykov | ń An experimental comparison of mincut/max-ŕow algorithms for energy minimization in vision ż[END_REF], and entropy rate superpixels [START_REF] Liu | Entropy rate superpixel segmentation ż[END_REF].

Clustering-based methods embed pixels into a space where clustering techniques can be leveraged to produce a superpixel partition. Central to this category, SLIC [3] embeds pixels with their XY position and Lab color features, and uses a k-means variant to mitigate the computational cost. See [START_REF] Li | ń Superpixel segmentation using linear spectral clustering ż[END_REF][START_REF] Liu | Manifold SLIC: A fast method to compute content-sensitive superpixels ż[END_REF], make SLIC faster [2], or differentiable [START_REF] Jampani | ń Superpixel sampling networks ż[END_REF].

Superpoint Partitioning

Partitioning large 3D point clouds into groups of adjacent and homogeneous points, called superpoints, is also an active area of research. Superpoint partitioning has been used successfully applied for point cloud oversegmen-tation [START_REF] Papon | Voxel cloud connectivity segmentation-supervoxels for point clouds ż[END_REF][START_REF] Lin | Toward better boundary preserved supervoxel segmentation for 3D point clouds ż[END_REF][START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF], semantic segmentation [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF][START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF], and object detection [START_REF] Han | ń OccuSeg: Occupancy-aware 3D instance segmentation ż[END_REF][START_REF] Engelmann | ń 3D-MPA: Multi-proposal aggregation for 3D semantic instance segmentation ż[END_REF].

Akin to superpixels, superpoint methods may be clustering-based or graphbased. Clustering approaches such as VCCS [START_REF] Papon | Voxel cloud connectivity segmentation-supervoxels for point clouds ż[END_REF] draw inspiration from SLIC [3] and use k-means on point features, under local adjacency constraints.

However, k-means-based methods rely on a fixed number of randomly initialized clusters, proscribing the processing of point clouds of arbitrary size and geometric complexity. On the other hand, Landrieu et al . [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF] cast point cloud oversegmentation as a structured optimization problem and uses the cut-pursuit [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions ż[END_REF] algorithm to generate superpoints. This method does not make any assumption on the number of superpoints and produces a partition whose granularity adapts to the 3D geometry.

Superpoints for Semantic Segmentation

Superpoint Graph (SPG) [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF] proposes learning the relationship between superpoints using graph convolutions [START_REF] Simonovsky | ń Dynamic edge-conditioned őlters in convolutional neural networks on graphs ż[END_REF] for semantic segmentation, as detailed in Figure 2.5. While this method trains fast, its preprocessing is slow and its expressivity and range are limited, as it operates on a single partition.

Recent works have proposed ways of learning superpoints themselves [START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF][START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF][START_REF] Thyagharajan | Segment-Fusion: Hierarchical context fusion for robust 3D semantic segmentation ż[END_REF], which yields improved results but at the cost of an extra training step or a large point-based backbone [START_REF] Kang | ń Region-enhanced feature learning for scene semantic segmentation ż[END_REF].

Hierarchical partitions are used for image processing [START_REF] Arbelaez | ń Boundary extraction in natural images using ultrametric contour maps ż[END_REF][START_REF] Xu | ń Hierarchical image simpliőcation and segmentation based on mumfordśshah-salient level line selection ż[END_REF][START_REF] Zhang | Nested hierarchical transformer: Towards accurate, dataefficient and interpretable visual understanding ż[END_REF] and 3D

analysis tasks, such as point cloud compression [START_REF] Fan | ń Point cloud compression based on hierarchical point clustering ż[END_REF] and object detection [START_REF] Chen | Hierarchical aggregation for 3D instance segmentation ż[END_REF][START_REF] Liang | ń Instance segmentation in 3D scenes using semantic superpoint tree networks ż[END_REF]. Hierarchical approaches for semantic segmentation use octrees with fixed grids [START_REF] Narasimhamurthy | Hierarchical-based semantic segmentation of 3D point cloud using deep learning ż[END_REF][START_REF] Riegler | ń OctNet: Learning deep 3D representations at high resolutions ż[END_REF]. On the contrary, our model Superpoint Transformer

Superpoints for Panoptic Segmentation

The panoptic segmentation task is related to semantic and instance segmentation [START_REF] Kirillov | ń Panoptic segmentation ż[END_REF]. Like semantic segmentation, all points must be assigned a single semantic label. But unlike instance segmentation, each point must also belong to exactly one predicted instance. This setting makes partition-based methods naturally suitable for panoptic segmentation. We further elaborate on this idea and the related literature in Section 4.2.

The superpoint paradigm is central to our work. In Chapter 3, we get inspiration from SPG [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF] to develop an efficient 3D semantic segmentation architecture that relies on expressive transformer blocks to reason on a hierarchical partition of the scene. In Chapter 4, we propose a fast and scalable superpoint graph clustering approach to address panoptic segmentation of large 3D point clouds.

Leveraging Images for 3D Understanding

Point cloud understanding may be enriched by information from other modalities, such as images. In doing so, data-efficient methods can leverage readily-available knowledge from 2D models pretrained on large 2D datasets.

In Chapter 5, we propose a method that learns to aggregate information from an arbitrarily-posed views of the same 3D object, based on their viewing conditions. This section contextualizes our approach. A deeper review of the literature on this topic can be found in Section 5.2.

Point Cloud Colorization

One way to exploit images to improve 3D understanding is to colorize point clouds. Unlike photogrammetry-based [START_REF] Tomasi | ń Shape and motion from image streams under orthography: A factorization method ż[END_REF] acquisition techniques which naturally produce colorized points, active sensors such as LiDAR [START_REF] Albert V Jelalian | Laser radar systems[END_REF] or time-of-flight cameras [START_REF] Oggier | An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (swissranger) ż[END_REF] do not. In practice, these clouds can be colorized through a nontrivial heuristics-based preprocessing that requires localized RGB images and their camera parameters [START_REF] Julin | Evaluating the quality of TLS point cloud colorization ż[END_REF]. Colorized point cloud datasets [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF][START_REF] Hackel | Semantic3D.Net: A new large-scale point cloud classiőcation benchmark ż[END_REF][START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF][START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF] are frequently used to compare 3D deep learning methods [START_REF] Charles R Qi | PointNet: Deep learning on point sets for 3D classiőcation and segmentation ż[END_REF][START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF][START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF], which consistently perform better when radiometric information is available [START_REF] Qian | PointNeXt: Revisiting PointNet++ with improved training and scaling strategies ż[END_REF]. In short, point cloud colorization assumes either a specific sensor or heuristics-based preprocessing, and discards dense, contextual, multi-view information carried by images. Hence, 3D analysis methods capable of directly processing raw point clouds and localized images would be less hardware-dependent and more data-efficient.

Learning to Fuse Points and Images

Advances in image and point cloud analysis using deep neural networks naturally invite to devise architectures capable of jointly extracting features from both modalities. We further present this active field of research in Section 5.2.

The work presented in Chapter 5 is a novel method for learning to extract and fuse information from arbitrarily-posed images and large point clouds in an end-to-end fashion.

Chapter 3

Efficient and Scalable 3D Semantic As the expressivity of deep learning models increases rapidly, so do their complexity and resource requirements [START_REF] Giattino | ń Artiőcial intelligence ż[END_REF]. In particular, vision transformers have demonstrated remarkable results for 3D point cloud semantic segmentation [START_REF] Zhao | Point Transformer ż[END_REF][START_REF] Park | ń Fast point transformer ż[END_REF][START_REF] Guo | PCT: Point Cloud Transformer ż[END_REF][START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF][START_REF] Loiseau | ń Online segmentation of LiDAR sequences: Dataset and algorithm ż[END_REF], but their high computational requirements make them challenging to train effectively. Additionally, these models rely on regular grids or point samplings, which do not adapt to the varying complexity of 3D data: the same computational effort is allocated everywhere, regardless of the local geometry or radiometry of the point cloud. This issue leads to needlessly high memory consumption, limits the number of points that can be processed simultaneously, and hinders the modeling of long-range interactions.

Superpoint-based methods [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF][START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF][START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF][START_REF] Quana | Hierarchical semantic segmentation of urban scene point clouds via group proposal and graph attention network ż[END_REF] address the limitation of regular grids by partitioning large point clouds into sets of points-superpointswhich adapt to the local complexity. By directly learning the interaction between superpoints instead of individual points, these methods enable the analysis of large scenes with compact and parsimonious models that can be trained faster than standard approaches. However, superpoint-based methods often require a costly preprocessing, and their range and expressivity are limited by their use of local graph-convolution schemes [START_REF] Simonovsky | ń Dynamic edge-conditioned őlters in convolutional neural networks on graphs ż[END_REF].

In this chapter, we propose a novel superpoint-based transformer architecture that overcomes the limitations of both approaches, see -Efficient Superpoint Computation: We propose a new method to compute a hierarchical superpoint structure for large point clouds, which is more than 7 times faster than existing superpoint-based methods. Our preprocessing time is also comparable or faster than standard approaches, addressing a significant drawback of superpoint methods.

-State-of-the-Art Performance: Our model reaches performance at or close to the state-of-the-art for three open benchmarks with distinct settings: S3DIS for indoor scanning [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF], KITTI-360 for outdoor mobile acquisitions [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF], and DALES for city-scale aerial LiDAR [START_REF] Varney | ń DALES: A large-scale aerial LiDAR data set for semantic segmentation ż[END_REF][START_REF] Nina | ń DALES Objects: A large scale benchmark dataset for instance segmentation in aerial LiDAR ż[END_REF].

-Resource-Efficient Models: SPT is particularly resource-efficient as it only has 212k parameters for S3DIS and DALES, a 200-fold reduction compared to other state-of-the-art models such as PointNeXt [START_REF] Qian | PointNeXt: Revisiting PointNet++ with improved training and scaling strategies ż[END_REF] and takes 70 times fewer GPU-h to train than Stratified Transformer [START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF]. The even more compact SPT-nano reaches 70.8% 6-Fold mIoU on S3DIS with only 26k parameters, making it the smallest model to reach above 70% by a factor of almost 300.

Method

Our method has two key components. First, we use an efficient algorithm to segment an input point cloud into a compact multiscale hierarchical structure.

Second, a transformer-based network leverages this structure to classify the elements of the finest scale.

Efficient Hierarchical Superpoint Partition

We consider a point cloud C with positional and radiometric information.

To learn multiscale interactions, we compute a hierarchical partition of C into geometrically-homogeneous superpoints of increasing coarseness; see subsets.

P := [P 0 , • • • , P I ] is a hierarchical partition of X if P 0 = X , and
P i+1 is a partition of P i for i ∈ [0, I -1].
Throughout this chapter, all functions or tensors related to a specific partition level i are denoted with an exponent i.

Hierarchical Superpoint Partitions. We propose an efficient approach for constructing hierarchical partitions of large point clouds. First, we associate each point c of C with features f c representing its local geometric and radiometric information. These features can be handcrafted [START_REF] Guinard | ń Weakly supervised segmentation-aided classiőcation of urban scenes from 3D LiDAR point clouds ż[END_REF] or learned [START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF][START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF].

See Section 3.2.4 for more details on point features. We also define a graph G encoding the adjacency between points usually based on spatial proximity, e.g. k-nearest neighbors.

We view the features f c for all c of C as a signal f defined on the nodes of the graph G. Following the ideas of SuperPoint Graph [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF], we compute an approximation of f into constant components by solving an energy minimization problem penalized with a graph-based notion of simplicity. The resulting constant components form a partition whose granularity is determined by a regularization strength λ > 0: higher values yield fewer and coarser components.

For each component of the partition, we can compute the mean position (centroid) and feature of its elements, defining a coarser point cloud on which we can repeat the partitioning process. We can now compute a hierarchical

partition P := [P 0 , • • • , P I ] of C from a list of regularization strengths λ 1 , • • • , λ I .
First, we set P 0 as the point cloud C and f 0 as the point features f . Then, for i = 1 to I, we compute (i) a partition P i of f i-1 penalized with λ i ; (ii) the mean signal f i for all components of P i . The coarseness of the resulting partitions [P 0 , • • • , P I ] is thus strictly increasing. See Section B-4 for a more detailed description of this process, and Section B-5 for our parameterization recipe.

Hierarchical Graph Structure. A hierarchical partition defines a polytree structure across the different levels. Let p be an element of

P i . If i ∈ [0, I -1], parent(p) is the component of P i+1 which contains p. If i ∈ [1, I], children(p)
is the set of components of P i-1 whose parent is p.

Superpoints also share adjacency relationships with superpoints of the same partition level. For each level i ≥ 1, we build a superpoint-graph G i by connecting adjacent components of P i , i.e. superpoints whose closest points are within a distance gap ϵ i > 0. For p ∈ P i , we denote N (p) ⊂ P i the set of neighbors of p in the graph G i . See Section 3.2.5 for more details on the superpoint-graph construction.

Hierarchical Parallel ℓ 0 -Cut Pursuit. Computing the hierarchical components involves solving a recursive sequence of non-convex, non-differentiable optimization problems on large graphs. We propose an adaptation of the ℓ 0 -cut pursuit algorithm [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs ż[END_REF] to solve this problem. To improve efficiency, we adapt the graph-cut parallelization strategy initially introduced by Raguet et al . [START_REF] Raguet | ń Parallel cut pursuit for minimization of the graph total variation ż[END_REF] in the convex setting.

Superpoint Transformer

Our proposed SPT architecture draws inspiration from the popular U-Net [START_REF] Ronneberger | ń U-Net: Convolutional networks for biomedical image segmentation ż[END_REF][START_REF] Gao | ń Graph U-Nets ż[END_REF]. However, instead of using grid, point, or graph subsampling, our approach derives its different resolution levels from the hierarchical partition P.

General Architecture. As represented in Figure 3.3, SPT comprises an encoder with I stages and a decoder with I -1 stages: the prediction takes place at the level P 1 and not on individual points. We start by computing the relative positions x of all points and superpoints with respect to their parent. For a superpoint p ∈ P i , we define x i p as the position of the centroid of p relative to its parent's. The coarsest superpoints of P I have no parent and use the center of the scene as a reference centroid. We then normalize these values so that the sets {x i p |p ∈ children(q)} have a radius of 1 for all q ∈ P i+1 . We compute features for each 3D point by using a multi-layer perceptron (MLP) to mix their relative positions and handcrafted features:

g 0 := ϕ 0 enc ([x 0 , f 0 ]), with [•,
•] the channelwise concatenation operator. Each level i ≥ 1 of the encoder maxpools the features of the finer partition level i -1, adds relative positions x i and propagates information between neighboring superpoints in G i . For a superpoint p in P i , this translates as:

g i p = T i enc • ϕ i enc x i p , max q∈children(p) g i-1 q (3.1)
with ϕ i enc an MLP and T i enc a transformer module explained below. By avoiding communication between the 3D points of P 0 , we bypass a potential computational bottleneck.

The decoder passes information from the coarser partition level i + 1 to the finer level i. It uses the relative positions x i and the encoder features g i to improve the spatial resolution of its feature maps h i [START_REF] Ronneberger | ń U-Net: Convolutional networks for biomedical image segmentation ż[END_REF]. For a superpoint p in partition P i with 1 ≤ i < I -1, this can be expressed as:

h i p = T i dec • ϕ i dec x i p , g i p , h i+1 parent(p) (3.2)
with h I = g I , ϕ i dec an MLP, and T i dec an attention-based module similar to T i enc .

Self-Attention Between Superpoints. We propose a variation of graph-attention networks [START_REF] Veličković | Graph attention networks ż[END_REF] to propagate information between neighboring superpoints of the same partition level. For each level of the encoder and decoder, we associate to superpoint p ∈ P i a triplet of key, query, value vectors K p , Q p , V p of size D key , D key and D val . These values are obtained by applying a linear layer to the corresponding feature map m after GraphNorm normalization [START_REF] Cai | GraphNorm: A principled approach to accelerating graph neural network training ż[END_REF].

We then characterize the relationship between two superpoints p, q of P i adjacent in G i by a triplet of features a key p,q , a que p,q , a val p,q of dimensions D key , D key and D val , and whose computation is detailed in the next section. Given a superpoint p, we stack the vectors a key p,q , a que p,q , a val p,q for q ∈ N (p) in matrices

A key p , A que p , A val p of dimensions | N (p)| × D key or | N (p)| × D val . The modules T i
enc and T i dec gather contextual information as follows:

[T (m)] p + = att(Q ⊺ p ⊕A que p , K N (p) +A key p , V N (p) +A val p ) , (3.3)
with + = a residual connection [START_REF] He | Deep residual learning for image recognition ż[END_REF], ⊕ the addition operator with broadcasting on the first dimension, and K N (p) the matrix of stacked vectors K q for q ∈ N (p). The attention mechanism writes as follows:

att(Q, K, V ) := V ⊺ softmax Q ⊙ K1 | N (p)| , (3.4)
with ⊙ the Hadamard termwise product and 1 a column-vector with D key ones. Our proposed scheme is similar to classic attention schemes with two differences: (i) the queries adapt to each neighbor, and (ii) we normalize the softmax with the neighborhood size instead of the key dimension. In practice, we use multiple independent attention modules in parallel (multihead attention) and several consecutive attention blocks.

Leveraging the Hierarchical Graph Structure

The hierarchical superpoint partition P can be used for more than guidance for graph pooling operations. Indeed, we can learn expressive adjacency encodings capturing the complex adjacency relationships between superpoints and employ powerful supervision and augmentation strategies based on the hierarchical partitions.

Adjacency Encoding. While the adjacency between two 3D points is entirely defined by their distance vector, the relationships between superpoints are governed by additional factors such as their alignment, proximity, and difference in sizes or shapes. We characterize the adjacency of pairs of adjacent superpoints of the same partition level using a set of handcrafted features whose description is provided in Section 3.2.4.

For each pair of superpoints (p, q) adjacent in G i , we jointly compute the concatenated a key p,q , a que p,q , a val p,q by applying an MLP ϕ i adj to the handcrafted adjacency features defined above. Further details on the superpoint-graph construction are provided in Section 3.2.5.

Hierarchical Supervision. We propose to take advantage of the nested structure of the hierarchical partition P into the supervision of our model. We can naturally associate the superpoints of any level i ≥ 1 with a set of 3D points in P 0 . The superpoints at the finest level i = 1 are almost semantically pure (see Figure 3.7), while the superpoints at coarser levels i > 1 typically encompass multiple objects. Therefore, we use a dual learning objective:

(i) we predict the most frequent label within the superpoints of P 1 , and (ii) we predict the label distribution for the superpoints of P i with i > 1. We supervise both predictions with the cross-entropy loss.

Let y i p denote the true label distribution of the 3D points within a superpoint p ∈ P i , and ŷi p a one-hot-encoding of its most frequent label. We use a dedicated linear layer at each partition level to map the decoder feature g i p to a predicted label distribution z i p . Our objective function can be formulated as follows:

L = p∈P 1 -N 1 p | C | H(ŷ 1 p , z 1 p )+ I i=2 p∈P i µ i N i p | C | H(y i p , z i p ) , (3.5)
where µ 2 , • • • , µ I are positive weights, N i p represents the number of points within a superpoint p ∈ P i , and |C| is the total number of points in the point cloud, and H(y, z) = -k∈K y k log(z k ) and K the class set.

Superpoint-Based Augmentations. Although our approach classifies superpoints rather than individual 3D points, we still need to load the points of P 0 in memory to embed the superpoints from P 1 . However, since superpoints are designed to be geometrically simple, only a subset of their points is needed to characterize their shape. Therefore, when computing the feature g 1 p of a superpoint p of P 1 containing n points with Equation 3.1, we sample only a portion tanh(n/n max ) of its points, with a minimum of n min . This sampling strategy reduces the memory load and acts as a powerful data augmentation.

The lightweight version of our model SPT-nano goes even further. It ignores the points entirely and only use handcrafted features to embed the superpoints of P 1 , thus avoiding entirely the complexity associated with the size of the input point cloud P 0 .

To further augment the data, we exploit the geometric consistency of superpoints and their hierarchical arrangement. During the batch construction, we randomly drop each superpoint with a given probability at all levels.

Dropping superpoints at the fine levels removes random objects or object parts, while dropping superpoints at the coarser levels removes entire structures such as walls, buildings, or portions of roads, for example. Similar to SPG [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF], our method relies on simple handcrafted features to build the hierarchical partition and learn meaningful points and adjacency relationships. In this section, we provide further details on the definition of these features and how to compute them. It is important to note that these features are only computed once during preprocessing, and thanks to our optimized implementation, this step only takes a few minutes.

Handcrafted Features

Point Features. We can associate each 3D point with a set of 8 easy-tocompute handcrafted features, described below.

-Radiometric features (3 or 1): RGB colors are available for S3DIS and KITTI-360, and intensity values for DALES. These radiometric features are normalized to [0, 1] at preprocessing time. For KITTI-360, we find that using the HSV color model yields better results.

-Geometric features (5): We use PCA-based features: linearity, planarity, scattering, [START_REF] Demantké | Dimensionality based scale selection in 3D LiDAR point clouds ż[END_REF] and verticality [START_REF] Guinard | ń Weakly supervised segmentation-aided classiőcation of urban scenes from 3D LiDAR point clouds ż[END_REF], computed on the set of 50-nearest neighbors of each point. This neighbor search is only computed once during preprocessing and is also necessary to build the graph G. We also define elevation as the distance between a point and the ground below it. Since the ground is neither necessarily flat nor horizontal, we use the RANSAC algorithm [START_REF] Martin | ń Random sample consensus: A paradigm for model őtting with applications to image analysis and automated cartography ż[END_REF] on a coarse subsampling of the scene to find a ground plane. We normalize the elevation by dividing it by 4 for S3DIS and 20 for DALES and KITTI-360.

At preprocessing time, we only use radiometric and geometric features to compute the hierarchical partition. At training time, as mentionned in Section 3.2.2, SPT computes point embeddings by mapping all available point features, along with the normalized point position, to a vector of size D point with a dedicated MLP ϕ 0 enc . We provide an illustration of the geometric point features in Figure 3.4, to help the reader apprehend these simple geometric descriptors.

Adjacency Features. The relationship between adjacent superpoints provides crucial information to leverage their context. For each edge of the superpointgraph, we compute the 18 following features:

-Interface features [START_REF] Simegnew | ń A survey on deep-learning-based LiDAR 3D object detection for autonomous driving ż[END_REF]: All adjacent superpoints share an interface, i.e. pairs of points from each superpoint that are close and share a line of sight. SPG [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF] uses the Delaunay triangulation of the entire point cloud to compute such interfaces, while we propose a faster heuristic approach in Section 3.2.5 called the Approximate Superpoint Gap algorithm. Each pair of points of an interface defines an offset, i.e. a vector pointing from one superpoint to its neighbor. We compute the mean offset (dim 3), the mean offset length (dim 1), and the standard deviation of the offset in each canonical direction (dim 3).

-Ratio features (4): As defined in [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF], we characterize each pair of adjacent superpoints with the ratio of their lengths, surfaces, volumes, and point counts.

-Pose features [START_REF] Simegnew | ń A survey on deep-learning-based LiDAR 3D object detection for autonomous driving ż[END_REF]: For each superpoint, we define a normal vector as its principal component with the smallest eigenvalue. We then characterize the relative position between two superpoints with the cosine of the angle between the superpoint normal vectors (dim: 1) and between each of the two superpoints' normal and the mean offset direction (dim:

2). Additionally, the offset between the centroids of the superpoints is used to compute the centroid distance (dim: 1) and the unit-normalized centroid offset direction (dim: 3).

Note that the mean offset and the ratio features are not symmetric and imply that the edges of the superpoint-graphs are oriented.

As mentioned in Section Section 3.2.3, a network ϕ i adj maps these handcrafted features to a vector of size D key + D que + D val , for each level i ≥ 1 of the encoder and the decoder.

Superpoint-Graphs Computation

The Superpoint Graph method by Landrieu and Simonovsky [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF] builds a superpoint-graph from a point cloud using Delaunay triangulation, which can take a long time for large point clouds. In contrast, our approach connects two superpoints in P i , where i ≥ 1 if their closest points are within a distance gap ϵ i > 0. However, computing pairwise distances for all points is computationally expensive. We propose a heuristic to approximately find the closest pair of points for two superpoints, see Algorithm 1. We also accelerate the computation of adjacent superpoints by approximating only for superpoints with centroids closer than the sum of their radii plus the gap distance. This approximation helps to reduce the number of computations required for adjacency computation, which leads to faster processing times.

All steps involved in the computation of our superpoint-graph are implemented on the GPU to further enhance computational efficiency.

Algorithm 1 Approximate Superpoint Gap

Input: superpoints p 1 and p 2 , num_steps

c 1 ← centroid(p 1 ) c 2 ← centroid(p 2 ) for s ∈ num_steps do c 2 ← arg min p∈p2 ∥c 1 -p∥ c 1 ← arg min p∈p1 ∥c 2 -p∥ end for return ∥c 1 -c 2 ∥
Recovering the interface between two adjacent superpoints as evoked in Section 3.2.4 involves a notion of visibility: we connect points from each superpoint which are facing each other. This can be a challenging and ambiguous problem, which SuperPoint Graph [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions ż[END_REF] tackles using a Delaunay triangulation of the points. However, this method is impractical for large point clouds. To address this issue, we propose a heuristic approach with the following steps: (i) first, we use the Approximate Superpoint Gap algorithm to compute the approximate nearest points for each superpoint. Then, we restrict the search to only consider points within a certain distance of the nearest points. Finally, we match the points by sorting them along the principal component of the selected points.

Experiments

We evaluate our model on three diverse datasets described in Section 3.3.1.

In Section 3.3.2, we evaluate our approach in terms of precision, but also quantify the gains in terms of preprocessing, training, and inference times.

Finally, we propose an extensive ablation study in Section 3.3.3.

Datasets and Models

Datasets. To demonstrate its versatility, we evaluate SPT on three large-scale datasets of different natures.

S3DIS [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF]. This indoor dataset of office buildings contains over 274 million points across 6 building floors-or areas. The dataset is organized by individual rooms, but can also be processed by considering entire areas at once. KITTI-360 [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF]. This outdoor dataset contains more than 100 k laser scans acquired in various urban settings on a mobile platform. We use the accumulated point clouds format, which consists of large scenes with around 3 million points. There are 239 training scenes and 61 for validation.

DALES [START_REF] Varney | ń DALES: A large-scale aerial LiDAR data set for semantic segmentation ż[END_REF][START_REF] Nina | ń DALES Objects: A large scale benchmark dataset for instance segmentation in aerial LiDAR ż[END_REF]. This 10 km 2 aerial LiDAR dataset contains 500 millions of points across 40 urban and rural scenes, including 12 for evaluation.

We subsample the datasets using a 3 cm grid for S3DIS, and 10 cm for KITTI-360 and DALES. All accuracy metrics are reported for the full, unsampled point clouds. We use a two-level partition (I = 2) with µ 2 = 50 for all datasets and select the partition parameters to obtain a 30-fold reduction between P 1 and P 0 and a further 5-fold reduction for P 2 . See Table 3.1 for more details.

Models. We use the same model configuration for all three datasets with minimal adaptations. All transformer modules have a shared width D val , Optimization. We use the ADAMW optimizer [START_REF] Loshchilov | ń Decoupled weight decay regularization ż[END_REF] with default parameters, a weight decay of 10 -4 , a learning rate of 10 -2 for DALES and KITTI-360 on and 10 -1 for S3DIS. The learning rate for the attention modules is 10 times smaller than for other weights. Learning rates are warmed up from 10 -6 for 20 epochs and progressively reduced to 10 -6 with cosine annealing [START_REF] Loshchilov | ń SGDR: Stochastic gradient descent with warm restarts ż[END_REF]. Training Speed. We trained several state-of-the-art methods from scratch and report in Figure 3.6 the evolution of test performance as a function of training time. We used the official training logs for the multi-GPU Point Transformer and Stratified Transformer. SPT can train much faster than all methods not based on superpoints while attaining similar performance. Although Superpoint Graph trains even faster, its performance saturates earlier, 6.0 mIoU points below SPT. We also report the inference time of our method SPT performs on par or better than complex models with up to two orders of magnitude more parameters and significantly longer training times.

Quantitative Evaluation

Such efficiency and compactness are beneficial for real-world scenarios where hardware, time, or energy may be limited.

Ablation Study

We evaluate the impact of several design choices in Table 3.4 and reports here our observations. a) Handcrafted features. Without handcrafted point features, our model performs worse on all datasets. This observation is in line with other works which also remarked the positive impact of well-designed handcrafted features on the performance of smaller models [START_REF] Hsu | ń Incorporating handcrafted features into deep learning for point cloud classiőcation ż[END_REF][START_REF] Haoxi Ran | ń Surface representation for point clouds ż[END_REF].

A more in-depth investigation of the influence of the handcrafted features can be found in Section B-3.

b) Influence of Edges. Removing the adjacency encoding between superpoints leads to a significant drop of 6.3 points on S3DIS; characterizing the relative position and relationship between superpoints appears crucial to exploiting their context. We also find that pruning the 50% longest edges of each superpoint results in a systematic performance drop, highlighting the importance of modeling long relationships. We assess the impact of several improvements made possible by using hierarchical superpoints. First, we find that using all available points when embedding the superpoints of P 1 instead of random sampling resulted in a small performance drop. Second, setting the superpoint dropout rate to 0 worsens the performance by over 2.5 points on S3DIS and KITTI-360.

While we did not observe better results with three or more partition levels, only using one level leads to a significant loss of performance for all datasets.

d) Influence of Partition Purity. In Figure 3.7, we plot the performance of the "oracle" model which associates each superpoint of P 1 with its most frequent true label. This model acts as an upper bound on the achievable performance with a given partition. Our proposed partition has significantly higher semantic purity than a regular voxel grid with as many nonempty voxels as superpoints. This implies that our superpoints adhere better to semantic boundaries between objects.

We also report the performance of our model for different partitions of varying coarseness, measured as the number of superpoints in P 1 . Using, respectively, ×1.5 (×3) fewer superpoints leads to a performance drop of 2.2 (4.7) mIoU points, but reduce the training time to 2.4 (1.6) hours. The performance of SPT is more than 20 points below the oracle, suggesting that the partition does not strongly limit its performance.

Model Scalability

We study the scalability of SPT by comparing models with different parameter counts on each dataset. It is important to note that the superpoint approach drastically compresses the training set, which can lead to overfitting, see Section 3.3.6. For example, as illustrated in

Hierarchical Supervision

We explore, in Table 3.8, alternatives to our hierarchical supervision introduced in Section Section 3.2.3 : predicting the most frequent label for P 1

and the distribution for P 2 . We use "freq-P i " to refer to the prediction of the most frequent label applied the P i partition. Similarly, "dist-P i " denotes the prediction of the distribution of labels within each superpoint of the partition

P i .
We observe a consistent improvement across all datasets by adding the dist-P i supervision. This illustrates the benefits of supervising higher-level partitions, despite their lower purity. Moreover, supervising P 1 with the distribution rather than the most frequent label leads to a further performance drop. This validates our choice to consider P 1 superpoints as sufficiently pure to be supervised using their dominant label.

Limitations

Our model provides significant advantages in terms of speed and compactness but also comes with its own set of limitations. Errors in the Partition. Object boundaries lacking obvious discontinuities, such as curbs vs. roads or whiteboards vs. walls, are not well recovered by our partition. As partition errors cannot be corrected with our approach, this may lead to classification errors. To improve this, we could replace our handcrafted point descriptors (Section 3.2.4) with features directly learned for partitioning [START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF][START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF]. However, such methods significantly increase the preprocessing time, contradicting our current focus on efficiency. In line with [START_REF] Hsu | ń Incorporating handcrafted features into deep learning for point cloud classiőcation ż[END_REF][START_REF] Haoxi Ran | ń Surface representation for point clouds ż[END_REF], we use easy-to-compute yet expressive handcrafted features. Predictions. Finally, our method predicts labels at the superpoint level P 1

and not individual 3D points. Since this may limit the maximum performance achievable by our approach, we could consider adding an upsampling layer to make point-level predictions. However, this does not appear to us as the most profitable research direction. Indeed, this may negate some of the efficiency of our method. Furthermore, as shown in the ablation study Section 3.3.3 of the present chapter, the "oracle" model outperforms ours by a large margin. This may indicate that performance improvements should primarily be searched in superpoint classification rather than in improving the partition.

Our model also learns features for superpoints and not individual 3D points.

This may limit downstream tasks requiring 3D point features, such as surface reconstruction or panoptic segmentation. However, we argue that specific adaptations could be explored to perform these tasks at the superpoint level.

Conclusion

We have introduced the Superpoint Transformer approach for semantic segmentation of large point clouds, combining superpoints and transformers to achieve state-of-the-art results with significantly reduced training time, inference time, and model size. This approach particularly benefits large-scale applications and computing with limited resources. More broadly, we argue that small, tailored models can offer a more flexible and sustainable alternative to large, generic models for 3D learning. With training times of a few hours on a single GPU, our approach allows practitioners to easily customize the models to their specific needs, enhancing the overall usability and accessibility of 3D learning. We plot the highest achievable łoraclež prediction for our partitions and a regular voxel grid. We also show the performance of SPT for 4 partitions with a coarseness ratio from ×1 to ×10. The signiőcant performance gap between SPT and the łoraclež suggests model design is more limiting than partition purity.

Chapter 4

Efficient and Scalable 3D Panoptic Segmentation

Introduction

Understanding large-scale 3D environments is pivotal for numerous highimpact applications such as the creation of "digital twins" of extensive industrial facilities [START_REF] Quattrocchi | ń Panoptic segmentation in industrial environments using synthetic and real data ż[END_REF][START_REF] Steven | Scaling digital twins from the artisanal to the industrial ż[END_REF][START_REF] Jiang | ń Industrial applications of digital twins ż[END_REF] or even the digitization of entire cities [START_REF] Lafarge | ń Creating large-scale city models from 3Dpoint clouds: A robust approach with hybrid representation ż[END_REF][START_REF] Xue | From LiDAR point cloud towards digital twin city: Clustering city objects based on gestalt principles ż[END_REF][START_REF] Nochta | A socio-technical perspective on urban analytics: The case of city-scale digital twins ż[END_REF]. Extensive and comprehensive 3D analysis methods also benefit large-scale geospatial analysis, e.g. for land [START_REF] Yeung | ń Urban land cover classiőcation using airborne LiDAR data: A review ż[END_REF][START_REF] Smeeckaert | Large-scale classiőcation of water areas using airborne topographic LiDAR data ż[END_REF] or forest surveys [START_REF] Xu | LiDAR applications to estimate forest biomass at individual tree scale: Opportunities, challenges and future perspectives ż[END_REF][START_REF] Hauglin | Large scale mapping of forest attributes using heterogeneous sets of airborne laser scanning and national forest inventory data ż[END_REF], as well as building inventory [START_REF] Wang | ń Urban 3D modeling with mobile laser scanning: A review ż[END_REF] for country-scale mapping. These problems call for scalable models that can process large point clouds with millions of 3D points, accurately predict the semantics of each point, and recover all instances of specific objects, a task referred to as 3D panoptic segmentation [START_REF] Kirillov | ń Panoptic segmentation ż[END_REF].

Most existing 3D panoptic segmentation methods focus on sparse LiDAR scans for autonomous navigation [START_REF] Aygun | ń 4D panoptic LiDAR segmentation ż[END_REF][START_REF] Whye | Panoptic nuScenes: A large-scale benchmark for LiDAR panoptic segmentation and tracking ż[END_REF][START_REF] Zhou | ń Panoptic-PolarNet: Proposal-free LiDAR point cloud panoptic segmentation ż[END_REF]. Given the relevance of largescale analysis for industry and practitioners, there is surprisingly little work on large-scale 3D panoptic segmentation [START_REF] Xiang | ń A review of panoptic segmentation for mobile mapping point clouds ż[END_REF]. Although they contain nonoverlapping instance labels, S3DIS [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF] and ScanNet [START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF] only have a few panoptic segmentation entries, and KITTI-360 [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF] and DALES [START_REF] Varney | ń DALES: A large-scale aerial LiDAR data set for semantic segmentation ż[END_REF][START_REF] Nina | ń DALES Objects: A large scale benchmark dataset for instance segmentation in aerial LiDAR ż[END_REF] currently have none.

Large-scale 3D panoptic segmentation is particularly challenging due to the sheer scale of the scenes, often featuring millions of 3D points, and the diversity in objects-ranging from a few to thousands and with extreme size variability.

Current methods typically rely on large backbone networks with millions of parameters, restricting their analysis to small scenes or portions of scenes due to their high memory consumption. Furthermore, training these models requires resource-intensive procedures, such as non-maximum suppression and instance matching. These costly operations prevent the analysis of large scenes with many points or objects. Most methods also require a pre-set limit on the number of detectable objects, introducing unnecessary complexity and the risk of missing objects in large scenes. Although recent mask-based intance segmentation methods [START_REF] Schult | ń Mask3D: Mask transformer for 3D semantic instance segmentation ż[END_REF] have demonstrated high performance and versatility, they fail to scale effectively to large scenes, as they predict a binary mask that covers the entire scene for each proposed instance.

To address these limitations, we present Super-Cluster, a novel approach for large-scale and efficient 3D panoptic segmentation. Our model differs from existing methods in three main ways:

-Scalable graph clustering: We view the panoptic segmentation task as a scalable graph clustering problem, which can be resolved efficiently at a large scale without setting the number of predicted objects in advance.

-Local supervision: We use a neural network to predict the parameters of the graph clustering problem and supervise with auxiliary losses that do not require an actual segmentation. This allows us to avoid resourceintensive non-maximum suppression or instance-matching steps.

-Superpoint-only segmentation: Our approach can easily be adapted to a superpoint-based approach. Feature computation, supervision, and prediction are entirely conducted at the superpoint level and never individual points, starkly decreasing their complexity.

These features make SuperCluster particularly resource-efficient, fast, and scalable, while ensuring high precision, as shown in Figure 4.1. Our primary contributions are:

-Large-scale panoptic segmentation: SuperCluster significantly improves the panoptic segmentation state-of-the-art for two indoor scanning datasets: 50.1 PQ (+7.8) on S3DIS Fold5 [START_REF] Armeni | ń 3D Scene Graph: A structure for uniőed semantics, 3D space, and camera ż[END_REF], and 58.7 PQ (+25.2) on ScanNetV2 [START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF]. We also set the first panoptic state-of-the-art for S3DIS 6-fold and two large-scale benchmarks (KITTI-360 [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF] and DALES [START_REF] Varney | ń DALES: A large-scale aerial LiDAR data set for semantic segmentation ż[END_REF][START_REF] Nina | ń DALES Objects: A large scale benchmark dataset for instance segmentation in aerial LiDAR ż[END_REF]).

-Fast and scalable segmentation: SuperCluster contains only 209k trainable parameters (205k in the backbone), yet outperforms networks over 30 times larger. SuperCluster inference is also as fast as the fastest instance segmentation methods and trains up to 15-times faster: 4 h for one S3DIS fold and 6 h for ScanNet.

Related Work

The panoptic segmentation of point clouds with millions of points has received little attention from the 3D computer vision community. In this chapter, we aim to address this gap.

Over the last few years, deep learning approaches for 3D point clouds have garnered considerable interest [START_REF] Guo | Deep learning for 3D point clouds: A survey ż[END_REF]. Autonomous driving, in particular, has been the focus of numerous studies, resulting in multiple proposed approaches for object detection [START_REF] Zamanakos | A comprehensive survey of LiDAR-Based 3D Object detection methods with deep learning for autonomous driving ż[END_REF][START_REF] Simegnew | ń A survey on deep-learning-based LiDAR 3D object detection for autonomous driving ż[END_REF], as well as semantic [START_REF] Zhu | ń Cylindrical and asymmetrical 3D convolution networks for LiDAR segmentation ż[END_REF][START_REF] Loiseau | ń Online segmentation of LiDAR sequences: Dataset and algorithm ż[END_REF][START_REF] Zhang | PolarNet: An improved grid representation for online LiDAR point clouds semantic segmentation ż[END_REF], instance [START_REF] Zhao | ń A technical survey and evaluation of traditional point cloud clustering methods for LiDAR panoptic segmentation ż[END_REF][START_REF] Zhou | Joint 3D instance segmentation and object detection for autonomous driving ż[END_REF], and panoptic segmentation [START_REF] Aygun | ń 4D panoptic LiDAR segmentation ż[END_REF][START_REF] Whye | Panoptic nuScenes: A large-scale benchmark for LiDAR panoptic segmentation and tracking ż[END_REF][START_REF] Zhou | ń Panoptic-PolarNet: Proposal-free LiDAR point cloud panoptic segmentation ż[END_REF][START_REF] Mohan | ń EfficientPS: Efficient panoptic segmentation ż[END_REF]. However, these methods consider sequences of sparse LiDAR acquisition, and focus on a small set of classes (pedestrians, cars).

For the panoptic segmentation of dense LiDAR point clouds, the volume of research is surprisingly small [START_REF] Xiang | ń A review of panoptic segmentation for mobile mapping point clouds ż[END_REF]. A limited number of studies have addressed the panoptic segmentation of indoor spaces using RGB-D images [START_REF] Wu | SceneGraphFusion: Incremental 3D scene graph prediction from RGB-D sequences ż[END_REF][START_REF] Narita | PanopticFusion: Online volumetric semantic mapping at the level of stuff and things ż[END_REF]. Dense scans have primarily been used in the context of instance segmentation [START_REF] Schult | ń Mask3D: Mask transformer for 3D semantic instance segmentation ż[END_REF][START_REF] Ngo | ń ISNBET: A 3D point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution ż[END_REF][START_REF] He | ń DyCo3D: Robust instance segmentation of 3D point clouds through dynamic convolution ż[END_REF][START_REF] Yang | Learning object bounding boxes for 3D instance segmentation on point clouds ż[END_REF][START_REF] Jiang | PointGroup: Dual-set point grouping for 3D instance segmentation ż[END_REF][START_REF] Vu | ń SoftGroup for 3D instance segmentation on point clouds ż[END_REF]. However, while this task is related to panoptic segmentation, these methods often adopt specific strategies to maximize instance segmentation metrics [START_REF] Xiang | ń A review of panoptic segmentation for mobile mapping point clouds ż[END_REF][START_REF] Cheng | Masked-attention mask transformer for universal image segmentation ż[END_REF]. Moreover, many methods require specifying the maximum number of predicted instances in advance, a constraint that proves inefficient for small scenes and results in missing objects in large scenes. Additionally, when implementing a sliding-window strategy, the predicted instances must be stitched together using either heuristic techniques or resource-intensive post-processing. Lastly, the best-performing methods [START_REF] Schult | ń Mask3D: Mask transformer for 3D semantic instance segmentation ż[END_REF][START_REF] Ngo | ń ISNBET: A 3D point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution ż[END_REF] rely on a computationally expensive matching step between the predicted and true instances [START_REF] Carion | End-to-end object detection with transformers ż[END_REF][START_REF] Yang | Learning object bounding boxes for 3D instance segmentation on point clouds ż[END_REF][START_REF] Jia | DETRs with hybrid matching ż[END_REF]. This process often depends on the Hungarian algorithm, which has cubic complexity in the number of objects and, therefore, cannot scale to large scenes.

The strategy of partitioning large 3D point clouds into groups of adjacent and homogeneous points, called superpoints, has been used successfully for point cloud oversegmentation [START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF][START_REF] Papon | Voxel cloud connectivity segmentation-supervoxels for point clouds ż[END_REF][START_REF] Lin | Toward better boundary preserved supervoxel segmentation for 3D point clouds ż[END_REF], semantic segmentation [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF][START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF][START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF], and object detection [START_REF] Han | ń OccuSeg: Occupancy-aware 3D instance segmentation ż[END_REF][START_REF] Engelmann | ń 3D-MPA: Multi-proposal aggregation for 3D semantic instance segmentation ż[END_REF]. Our approach shares similarities with some superpoint-based approaches for 3D instance segmentation [START_REF] Sun | ń Superpoint transformer for 3D scene instance segmentation ż[END_REF][START_REF] Liang | ń Instance segmentation in 3D scenes using semantic superpoint tree networks ż[END_REF]. However, these methods are limited in scalability due to their reliance on point-wise encoders. Furthermore, the work by Sun et al . [START_REF] Sun | ń Superpoint transformer for 3D scene instance segmentation ż[END_REF] employs a Hungarian-type instance matching scheme and allocates a binary mask to each predicted instance, covering the entire scene and drastically limiting the number of detected instances. Liang et al . [START_REF] Liang | ń Instance segmentation in 3D scenes using semantic superpoint tree networks ż[END_REF] resort to quadratic-complexity agglomerative clustering to merge superpoints, and heavy post-processing for refining and scoring superpoints. In contrast, our method employs a fast graph clustering approach [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs ż[END_REF][START_REF] Kolmogorov | ń What energy functions can be minimized via graph cuts? ż[END_REF], which does not require any instance matching or post-processing steps. Our objective is to perform panoptic segmentation of a large 3D point cloud P with potentially numerous and broad objects. For clarity, we first present our graph clustering formulation at the point level. We then explain how our approach can be supervised purely with local objectives, making its training particularly efficient. Finally, we detail how our method can be easily generalized to superpoints to further increase its scalability. Our final pipeline is illustrated in Figure 4.2.

Method

Problem Statement. Consistently with the image panoptic segmentation setup [START_REF] Kirillov | ń Panoptic segmentation ż[END_REF], each point p ∈ P is associated with its position, a semantic label cls(p) ∈ [1, C] with C the total number of classes, and an object index obj(p) ∈ N. Points identified with a "thing" label (e.g. chair, car) are given an index uniquely identifying this object. Conversely, points with a "stuff" label (e.g. road, wall) are assigned an index shared by all points with the same class within P. Our goal is to recover the class and object index of all points in P.

Panoptic Segmentation as Graph Clustering

We propose viewing the panoptic segmentation task as grouping adjacent points with compatible class and object predictions. We formulate this task as an optimization problem structured by a graph. Specifically, we connect the points of P to their K nearest neighbors, forming a graph G = (P, E)

where E ⊂ P × P denotes these connections.

Spatial-Semantic Regularization. We use a neural network to associate each point p with a probabilistic class prediction x class p ∈ [0, 1] C . The architecture and supervision of this network are detailed in Section 4.3.2. A simple way to obtain a panoptic segmentation would be to group spatially adjacent points with the same class prediction arg max c x class p,c . However, this approach ignores the structure of objects, and thus would lead to two types of issues: adjacent but distinct objects of the same class might be erroneously merged, and the probabilistic nature of the prediction x class may lead to unwanted object fragmentation.

To tackle this last issue, we aim to enforce the spatial consistency of the object prediction. We introduce the signal x, defined for each point p as the channelwise concatenation of its position x pos p and its semantic prediction:

x p = [x class p , x pos p ].
We propose to compute a piecewise-constant approximation y ⋆ of x with an energy minimization problem regularized by the graph cut [START_REF] Boykov | ń Fast approximate energy minimization via graph cuts ż[END_REF] between its constant components [START_REF] Landrieu | A structured regularization framework for spatially smoothing semantic labelings of 3D point clouds ż[END_REF]. This approach aligns with well-established practices in 2D [START_REF] Yvan | ń Constructing simple stable descriptions for image partitioning ż[END_REF][START_REF] Bryant | ń Optimal approximations by piecewise smooth functions and associated variational problems ż[END_REF] and 3D [START_REF] Labatut | ń Robust and efficient surface reconstruction from range data ż[END_REF] analyses, and leads to the following optimization problem:

y ⋆ =arg min y∈R (C+3)×|P| p∈P d(x p , y p ) + λ (p,q)∈E w p,q [y p ̸ = y q ] , (4.1)
where [a ̸ = b] := 0 if a = b and 1 otherwise, λ > 0 is a parameter controlling the regularization strength, and w p,q is a nonnegative weight associated with edge (p, q), see below. Object-Guided Edge Weights. The edge weight w p,q determines the cost of predicting an object transition between p and q. Designing appropriate edge weights is critical to differentiate between objects of the same class that are spatially adjacent, such as rows of chairs or cars in traffic. Edge weights should encourage cuts along probable object transitions and prevent cuts within objects.

To facilitate this, we propose to train a neural network to predict an object agreement a p,q ∈ [0, 1] for each edge (p, q) in E. This value represents the probability that both points belong to the same object. We then determine the edge weight w p,q ∈ [0, ∞] as follows:

w p,q = a p,q /(1 -a p,q + ϵ) , (4.3)
with ϵ > 0 a fixed parameter. High values of w p,q discourage cuts between points p and q that are confidently predicted to belong to the same object.

Conversely, a smaller w p,q means that cuts between edges with a probable transition a p,q are not heavily penalized.

Graph Clustering. The constant components of the solution y ⋆ of Equation 4.1 define a clustering K of P. The clusters K contain spatially adjacent points with compatible semantics, and their contours should follow predicted object transitions.

Converting to a Panoptic Segmentation. We can derive a panoptic segmentation from the clusters K. For each cluster, we calculate the average point distribution of its constituent points and select the class with the highest probability. We then associate a unique object index to each cluster k predicted as a "thing" class. Likewise, we assign to each cluster classified as "stuff"

an index shared by all clusters predicted as the same class. Finally, each individual point is labeled with the class and object index of its respective cluster.

Optimization. The optimization problem expressed in Equation 4.1 is widely explored within graph optimization literature. Referred to as the generalized minimal partition problem [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs ż[END_REF], this problem is related to the Potts models [START_REF] Burnard | ń Some generalized order-disorder transformations ż[END_REF] and image partitioning techniques [START_REF] Yvan | ń Constructing simple stable descriptions for image partitioning ż[END_REF][START_REF] Bryant | ń Optimal approximations by piecewise smooth functions and associated variational problems ż[END_REF]. We adapt the parallel ℓ 0 -cut pursuit algorithm [START_REF] Raguet | ń Parallel cut pursuit for minimization of the graph total variation ż[END_REF][START_REF] Landrieu | A structured regularization framework for spatially smoothing semantic labelings of 3D point clouds ż[END_REF] to the dual spatial-semantic nature of the regularized signal. The resulting algorithm is particularly scalable and can handle graphs with hundreds of millions of edges on a standard workstation.

This allows us to process large point clouds in one inference without the need for tiling and instance stitching post-processing.

Local Supervision

A major benefit of our approach is that it can be entirely supervised with local auxiliary tasks: all losses described in this section are sums of simple functions depending on one or two points at the time. In particular, we bypass the computationally expensive step of matching true instances with their predicted counterparts.

Recall from Section 4.3.1 that we can obtain a panoptic segmentation by predicting the parameters of a graph clustering problem: the semantic predictions x class p and the object agreements a p,q . This quantities are both derived from a common pointwise embeddings {e p } p∈P , computed by a neural network.

Predicting Semantics. We predict the class distribution x class p = softmax(ϕ class (e p )) with ϕ class a Multi-Layer Perceptron (MLP). This distribution is supervised by its cross-entropy against the true class cls(p):

L class p = H(x class p , 1(cls(p))) , (4.4)
with 1(c) ∈ {0, 1} C the one-hot embedding of class c.

Predicting Object Agreement. To predict the object agreement a p,q between two adjacent points (p, q) ∈ E, we employ an MLP ϕ object whose input is a symmetric combination of the points' embedding vectors:

a p,q = sigmoid ϕ object 1 2 (e p + e q ), | e p -e q | , (4.5)
where | • | refers to the termwise absolute value. The true object agreement âp,q is assigned the value of 1 if obj(p) = obj(q) and 0 otherwise. The prediction of a s,t can be seen as a binary edge classification problem as interand intra-object edges [START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF], and is supervised with the cross-entropy between true and predicted object agreements:

L object p,q
= H(Bern(a p,q ), Bern(â p,q )) ,

where Bern(a) denote the Bernoulli distribution parametrized by a ∈ [0, 1].

Loss Function. We combine the two losses above into a single objective L:

L = 1 | P | p∈P L class p + 1 | E | (p,q)∈E L object p,q , (4.7)
with | E | and | P | the total number of edges and 3D points, respectively.

Extension to Superpoints

In this section, we discuss the extension of our method to a superpointbased approach for enhanced scalability.

Motivation. We aim to design a panoptic segmentation method that can scale to large 3D point clouds. While the formulation presented in the previous section is advantageous, it still requires computing embeddings and predictions for each individual point, which can be memory-intensive and limits the volume of data that can be processed simultaneously. We propose to group adjacent points with similar local geometry and color into superpoints, and to only compute embeddings and predictions for superpoints and not individual points. By doing so, we drastically reduce the computational and memory requirements of our method, enabling it to handle larger 3D point clouds at once. Our pipeline is illustrated in Figure 4.2.

Computing Superpoints. We partition the point cloud P into a set of nonoverlapping superpoints S. We use the superpoint partition method implemented by Robert et al . in SPT [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF], which defines the superpoints as the constant components of a low-surface piecewise constant approximation of geometric and radiometric point features.

Although the superpoints S form a high-purity oversegmentation of P, some superpoints can span multiple objects. To account for this, we associate each superpoint s with its majority-object obj(s) defined as the most common object index within its points: obj(s) = mode{obj(p) | p ∈ s}. Likewise, we define cls(s) = mode{cls(p) | p ∈ s}.

Adapting Graph Clustering. Our clustering step can be directly adapted by substituting the point set P with the superpoint set S, and defining the graph G by connecting superpoints with adjacent points following the approach of SPT [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF]. We replace the point position x pos p by the coordinates of the superpoints' centroids x pos s . All other steps are unchanged.

Superpoint Embedding. We use a superpoint-embedding network to assign compute the superpoint features e s for s ∈ S. We employ the SuperPoint

Transformer model [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF] for its efficiency and ability to leverage large spatial context.

Superpoint Semantic Supervision. We supervise the semantic superpoint prediction x class s with Equation 4.4 where we replace cls(p) with cls(s). We compute for each pair of adjacent superpoint (s, t) an object agreement score âs,t . This value is deőned by the average overlap ratio between s and t and their majority-objects obj(t) and obj(s), see Equation 4.8.

Superpoint Object Agreement Supervision. While the true object agreement âp,q between two points is binary, the agreement between superpoints spans a continuum. As illustrated in Figure 4.3, we quantify this agreement as:

âs,t = 1 2 | s ∩ P |obj(t) | | s | + | t ∩ P |obj(s) | | t | , (4.8)
where P |o := {p ∈ P | obj(p) = o} is the set of points of P with the object index o, and | s | is the count of 3D points in s. We can now supervise the predicted object agreement a s,t with Equation 4.6 unchanged.

Experiments

We first present the datasets and metrics used for evaluation in Section 4.4.1, then our main results and their analysis in Section 4.4.2, and finally an ablation study in Section 4.4.3.

Datasets and Metrics

Datasets. We present the four datasets used in this chapter.

-S3DIS [14]: This indoor scanning dataset consists of 274 million points distributed across 271 rooms in 6 building floors-or areas. We do not use the provided room partition, as they require significant manual processing and may not translate well to other environments such as open offices, industrial sites, or mobile mapping. Instead, we merge all rooms in the same area and treat each floor as one single large-scale acquisition [START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF][START_REF] Chaton | Torch-Points3D: A modular multi-task framework for reproducible deep learning on 3D point clouds ż[END_REF].

We follow the standard evaluation protocol, using the area 5 as a test set and implementing 6-fold cross-validation. In line with Xiang et al .'s [START_REF] Xiang | ń A review of panoptic segmentation for mobile mapping point clouds ż[END_REF] proposal, we treat all 13 classes as "thing". However, certain classes, such as walls, ceiling, and floors, are susceptible to arbitrary division due to room splitting, making their evaluation somewhat inconsistent.

As a result, we also present panoptic metrics where these three classes are considered as "stuff".

-ScanNet. [START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF] This dataset consists of 237M 3D points organized in 1501 medium-scale indoor scenes. We evaluate SuperCluster on Scan-Net's open test set, as the hidden test set is not evaluated for panoptic segmentation. We use for "things" the class evaluated in the instance segmentation setting: bathtub, bed, bookshelf, cabinet, chair, counter, curtain, desk, door, other furniture, picture, refrigerator, shower curtain, sink, sofa, table, toilet, and window. The walls and floor class are designated as "stuff".

-KITTI-360 [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF]: Containing over 100k mobile mapping laser scans from an outdoor urban environment, we utilize the accumulated point clouds format, which aggregates multiple sensor rotations to form 300 extensive scenes with an average of more than 3 million points. We train on 239 scenes and evaluate it on the remaining 61. Building and cars classes are treated as "thing" while the remaining 13 are classified as "stuff".

-DALES [START_REF] Varney | ń DALES: A large-scale aerial LiDAR data set for semantic segmentation ż[END_REF][START_REF] Nina | ń DALES Objects: A large scale benchmark dataset for instance segmentation in aerial LiDAR ż[END_REF]. This large-scale aerial scan data set spans 10 km 2

and contains 500 millions of 3D points organized along 40 urban and rural scenes, of which we use 12 for evaluation. The "thing" classes are buildings, cars, trucks, power lines, fences, and poles. Ground and vegetation evaluated as "stuff".

Evaluation Metrics. Recognition Quality (RQ) assesses a model's capacity to identify and classify objects. Segmentation Quality (SQ) evaluates the alignment between an object's true and predicted segmentation. Panoptic Quality (PQ) combines both measures by computing their product. We also compute the semantic segmentation performance of our method by associating each point with the class of the superpoint to which it belongs and measuring the class-averaged Intersection over Union (mIoU).

Model Parameterization. Our backbone for the S3DIS and DALES datasets is a small SPT-64 model [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF]

Results and Analysis

We compare our method quantitatively with state-of-the-art models in Table 4.2 to 4.6. We also report a runtime analysis in Table 4.7 and qualitative illustrations in Figure 4.4.

S3DIS. We report in Table 4.2 the performance of our algorithm evaluated for Area 5 of the S3DIS dataset. Compared to several baselines for panoptic segmentation, our model shows a notable improvement with a PQ boost of +7.8 points and a mIoU increase of +3.2 points. Remarkably, our model is more than 33 times smaller than the highest performing model. Furthermore, we compute panoptic metrics by treating wall, ceiling, and floor as "stuff"

classes to account for their arbitrary boundaries.

In Table 4.3, we present the same metrics evaluated with 6-Fold crossvalidation. This classic setting is typically used to evaluate semantic segmentation; however, we are the first to report panoptic results in this context.

Despite its smaller size, our model achieves high semantic segmentation performance, improving the Area 5 performance of SPT by 1 point and reaching near state-of-the-art performance on the 6-fold evaluation.

ScanNet. As shown in Table 4.4, SuperCluster significantly improves the state-of-the-art of panoptic segmentation by 25.2 PQ points. Our model does None of the reported runtimes include the method's preprocessing times. Thanks to SPT's efficient implementation, our entire preprocessing, including the superpoint partition, is faster or equivalent to all existing 3D segmentation methods. For instance, preprocessing the entirety of S3DIS (271 rooms) takes only 12 minutes with an A40 GPU [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF].

Our model can be trained in an amount of time comparable to its backbone SPT for semantic segmentation [START_REF] Robert | ń Learning multi-view aggregation in the wild for large-scale 3D semantic segmentation ż[END_REF]. One fold of S3DIS takes just under 4 hours, which is substantially quicker than most existing semantic, instance, or panoptic segmentation models. For instance, PointTransformer [START_REF] Zhao | Point Transformer ż[END_REF] trains for 63h and Stratified Transformer [START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF] 216 GPU-h. SuperCluster trains on 6 h on ScanNet, compared to 78 h for Mask3D [START_REF] Schult | ń Mask3D: Mask transformer for 3D semantic instance segmentation ż[END_REF] and 20 h for ISBNet [START_REF] Ngo | ń ISNBET: A 3D point cloud instance segmentation network with instance-aware sampling and box-aware dynamic convolution ż[END_REF].

Ablation Study

We evaluate the impact of our design choice by performing several experiments whose results are given in Table 4.8.

Constant Edge Weights. Replacing all edge weights with a constant value of 1 yields a drop of 4.2 PQ points. This experiment shows the benefit of learning object transitions.

Offset Prediction. Several bottom-up [START_REF] He | Deep learning based 3D segmentation: A survey ż[END_REF] segmentation approaches [START_REF] Jiang | PointGroup: Dual-set point grouping for 3D instance segmentation ż[END_REF][START_REF] Xiang | ń A review of panoptic segmentation for mobile mapping point clouds ż[END_REF][START_REF] Lahoud | ń 3D instance segmentation via multi-task metric learning ż[END_REF][START_REF] Han | ń OccuSeg: Occupancy-aware 3D instance segmentation ż[END_REF] propose clustering points by shifting their positions towards the predicted position of the object centroid. To reproduce this strategy, we adjust the position of x pos s in x along a vector that predicts the center of the majority object. We supervise this prediction with the L1 loss, as it produced the best results among several alternatives that we examined. Despite our efforts, this approach did not improve the results: -1.3 PQ points. We attribute this to the size diversity of objects observed in large-scale scenes (corridors, buildings), resulting in an unstable prediction.

Smaller Superpoints. To demonstrate the benefits of using superpoints, we consider a finer partition with S/P ∼ 15 instead of 30. This requires training with smaller 3 m cylinders instead of 7, decreasing the performance by -1.8

PQ points. This result illustrates that the superpoint paradigm is central to our approach. Clustering Oracle. In a similar vein, we calculate the upper bound of our model by computing the results of the graph clustering with perfect network predictions: x class is set as the one-hot-encoding of the class of the majority object, and the object agreement is set to its true value: a p,q = âp,q . The performance of this oracle (83.6 PQ) shows that our scalable clustering formulation does not significantly compromise the model's precision in its current regime.

Limitations

Our approach, while efficient, is not devoid of constraints. The functional minimized in Equation 4.1 is noncontinuous and nondifferentiable. This hinders the computation of gradients and the possibility of learning the panoptic segmentation end-to-end. Nevertheless, this aspect lends itself to the speed and simplicity of our training process. Although our approach can run on diverse acquisition setups, the superpoint partition is sensitive to low point density and may fail for sparse scans as can be observed on the edge of some KITTI-360 acquisitions.

We use a lightweight SPT network to ensure maximum scalability. This network, while expressive, is not the most powerful existing architecture. There is a potential for improved results using more resource-intensive networks, especially for medium-scale datasets like ScanNet and S3DIS.

Local panoptic supervision does not translate into higher semantic segmentation performance in our experiments.

Conclusion

In this paper, we introduced SuperCluster, a novel approach for 3D panoptic segmentation of large-scale point clouds. We propose a new formulation of this task as a scalable graph clustering problem, bypassing some of the most compute-intensive steps of current panoptic segmentation methods. Our results across multiple benchmarks, including S3DIS, ScanNet, KITTI-360, and DALES, demonstrate that our model achieves state-of-the-art performance while being significantly smaller, scalable, and easier to train.

Despite the considerable industrial applications, large-scale panoptic segmentation has been relatively unexplored by the 3D computer vision community. We hope that our positive results and the state-of-the-art we established on new datasets and settings will encourage the development of future panoptic approaches for large-scale 3D scans. The fast-paced development of dedicated neural architectures for 3D data has led to significant improvements in the automated analysis of large 3D scenes [START_REF] Guo | Deep learning for 3D point clouds: A survey ż[END_REF]. All top-performing methods operate on colorized point clouds, which requires either specialized sensors [START_REF] Woodhouse | A multispectral canopy LiDAR demonstrator project ż[END_REF], or running a colorization step which is often closed-source [306, [START_REF]Faro Freestyle 2 handheld scanner[END_REF]188] and sensor-dependent [START_REF] Julin | Evaluating the quality of TLS point cloud colorization ż[END_REF]. However, while colorized point clouds carry some radiometric information, images combined with dedicated 2D architectures are better suited for learning textural and contextual cues. A promising line of work sets out to leverage the complementarity between 3D point clouds and images by projecting onto 3D points the 2D features learned from real [START_REF] Dai | ń 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation ż[END_REF][START_REF] Hu | Bidirectional projection network for cross dimension scene understanding ż[END_REF][START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF] or virtual images [START_REF] Kundu | ń Virtual multi-view fusion for 3D semantic segmentation ż[END_REF][START_REF] Chiang | A uniőed point-based framework for 3D segmentation ż[END_REF] Combining point clouds and images with arbitrary poses (i.e. in the wild ) as represented in Figure 5.1, involves recovering occlusions and computing a point-pixel mapping, which is typically done using accurate depth maps from specialized sensors [START_REF] Robert | ń Accurate 3D measurement using a structured light system ż[END_REF][START_REF] Cai | RGB-D datasets using Microsoft Kinect or similar sensors: A survey ż[END_REF] or a potentially costly meshing step [START_REF] Boulch | SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks ż[END_REF].

Introduction

Furthermore, when a point is seen in different images simultaneously, the 2D features must be merged in a meaningful way. In the mesh texturation literature, multi-view aggregation is typically addressed by selecting images for each triangle based on their viewing conditions, e.g. distance, viewing angle, or occlusion [START_REF] Allène | ń Seamless image-based texture atlases using multi-band blending ż[END_REF][START_REF] Lempitsky | ń Seamless mosaicing of image-based texture maps ż[END_REF][START_REF] Waechter | ń Let there be color! largescale texturing of 3D reconstructions ż[END_REF]. Hybrid 2D/3D methods for large-scale point cloud analysis usually rely on heuristics to select a fixed number of images per point and pool their features uniformly without considering viewing conditions. Multi-view aggregation has also been extensively studied for shape recognition [START_REF] Feng | GVCNN: Group-view convolutional neural networks for 3D shape recognition ż[END_REF][START_REF] Su | Multi-view convolutional neural networks for 3D shape recognition ż[END_REF][START_REF] Wei | ń View-GCN: View-based graph convolutional network for 3D shape analysis ż[END_REF], albeit in a controlled and synthetic setting not entirely applicable to the analysis of large scenes.

In this work, we propose to learn to merge features from multiple images with a dedicated attention-based scheme. For each 3D point, the information from relevant images is aggregated based on the point's viewing condition.

Thanks to our GPU-based implementation, we can efficiently compute a pointpixel mapping without mesh or true depth maps, and without sacrificing precision. Our model can handle large-scale scenes with an arbitrary number of images per point taken at any position (with camera pose information), which corresponds to a standard industrial operational setting [START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Virtanen | ń Interactive dense point clouds in a game engine ż[END_REF][START_REF] Pepe | ń 3D point cloud model color adjustment by combining terrestrial laser scanner and close range photogrammetry datasets ż[END_REF].

Using only standard 2D and 3D backbone networks, we set a new state-ofthe-art for the S3DIS and KITTI-360 datasets. Our method improves on both standard and hybrid 2D/3D approaches without requiring point cloud colorization, mesh reconstruction, or depth sensors. In this work, we present a novel and modular multi-view aggregation method for semantizing hybrid 2D/3D data based on the viewing conditions of 3D points in images. Our approach combines the following advantages:

-We set a new state-of-the-art for S3DIS 6-fold (74.7 mIoU), and KITTI-360 Test (58.3 mIoU) without using points' colorization.

-Our point-pixel mapping operates directly on 3D point clouds and images without requiring depth maps, meshing, colorization, or virtual view generation.

-Our efficient GPU-based implementation handles arbitrary numbers of 2D views and large 3D point clouds.

Related Work

Point Cloud Colorization. One way of exploiting the complementarity between 3D point clouds and images is to colorize the points. Unlike photogrammetrybased [START_REF] Tomasi | ń Shape and motion from image streams under orthography: A factorization method ż[END_REF] acquisition techniques which naturally produce colorized points, active sensors such as LiDAR [START_REF] Albert V Jelalian | Laser radar systems[END_REF] or time-of-flight cameras [START_REF] Oggier | An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (swissranger) ż[END_REF] do not. In practice, these clouds can be colorized through a nontrivial heuristics-based preprocessing requiring localized RGB images and their camera parameters [START_REF] Julin | Evaluating the quality of TLS point cloud colorization ż[END_REF]. Colorized point cloud datasets [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF][START_REF] Hackel | Semantic3D.Net: A new large-scale point cloud classiőcation benchmark ż[END_REF][START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF][START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF] are frequently used for comparing 3D deep learning methods [START_REF] Charles R Qi | PointNet: Deep learning on point sets for 3D classiőcation and segmentation ż[END_REF][START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF][START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF], which consistently perform better when radiometric information is available [START_REF] Qian | PointNeXt: Revisiting PointNet++ with improved training and scaling strategies ż[END_REF]. In short, point cloud colorization assumes either a specific sensor or heuristics-based preprocessing, and discards dense, contextual, multi-view information carried by images. Hence, 3D analysis methods capable of directly processing raw point clouds and localized images would be less hardware-dependent and more data efficient.

Attention-Based Modality Fusion. Methods using attention mechanisms to learn multi-modal representation have attracted a lot of attention, in particular for combining textual and visual information [START_REF] Caglayan | ń Multimodal attention for neural machine translation ż[END_REF][START_REF] Huang | Attention-based multimodal neural machine translation ż[END_REF][START_REF] Gu | Multimodal affective analysis using hierarchical attention strategy with word-level alignment ż[END_REF] as well as videos [START_REF] Hori | Attention-based multimodal fusion for video description ż[END_REF][START_REF] Long | Multimodal keyless attention fusion for video classiőcation ż[END_REF]. Closer to our setting, Lu et al . [START_REF] Lu | Hierarchical question-image co-attention for visual question answering ż[END_REF] use an attention scheme to select the most relevant parts of an image for visual question answering. Li et al . [START_REF] Li | Attention-based multi-modal fusion network for semantic scene completion ż[END_REF] define a two-branch attention-based modality fusion network merging 2D semantic and 3D occupancy for scene completion.

Such work confirms the relevance of using attention for learning multi-modal representations.

2D/3D Scene Analysis with Deep Learning. Over the last few years, deep networks specifically designed to handle the 3D modality have reached impressive degrees of performance and maturity, see the review of Guo et al . [START_REF] Guo | Deep learning for 3D point clouds: A survey ż[END_REF]. Recent work [START_REF] Dai | ń 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation ż[END_REF][START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF][START_REF] Hu | Bidirectional projection network for cross dimension scene understanding ż[END_REF] propose to use a dedicated 3D network for processing point clouds, while a 2D convolutional network extracts radiometric features which are projected to the point cloud. These methods require the true depth of each pixel to compute the point-pixel mapping, which makes them less applicable in a real-world setting. SnapNet [START_REF] Boulch | SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks ż[END_REF], as well as more recent work [START_REF] Kundu | ń Virtual multi-view fusion for 3D semantic segmentation ż[END_REF][START_REF] Chiang | A uniőed point-based framework for 3D segmentation ż[END_REF] generate virtual views processed by a 2D network and whose predictions are then projected back to the point cloud. These approaches, while performing well, require a costly mesh reconstruction preprocessing to generate meaningful images. Some approaches [START_REF] Hazirbas | FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture ż[END_REF][START_REF] Krispel | FuseSeg: LiDAR point cloud segmentation fusing multi-modal data ż[END_REF] fuse RGB and range images, which requires dedicated sensors and can not handle multiple views with occlusions. Existing hybrid 2D/3D methods rely on a fixed number of images per point chosen with heuristics such as the maximization of unseen points [START_REF] Dai | ń 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation ż[END_REF][START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF][START_REF] Kundu | ń Virtual multi-view fusion for 3D semantic segmentation ż[END_REF]. Then, the different views are merged using pooling operations (max [START_REF] Su | Multi-view convolutional neural networks for 3D shape recognition ż[END_REF][START_REF] Dai | ń 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation ż[END_REF] or sum-pool [START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF]) or based on the 2D features' content [START_REF] Hu | Bidirectional projection network for cross dimension scene understanding ż[END_REF]. To the best of our knowledge, no method has yet been proposed to leverage the viewing conditions for multi-view aggregation for the semantic segmentation of large scenes.

The problem of selecting and merging the best images for a 3D scene has been extensively studied for surface reconstruction and texturing. Images are typically chosen according to the viewing angle with the surface normal [START_REF] Lempitsky | ń Seamless mosaicing of image-based texture maps ż[END_REF][START_REF] Birchőeld | ń Multiway cut for stereo and motion with slanted surfaces ż[END_REF], proximity and resolution [START_REF] Allène | ń Seamless image-based texture atlases using multi-band blending ż[END_REF][START_REF] Buehler | ń Unstructured lumigraph rendering ż[END_REF], geometric and visibility priors [START_REF] Johannes | ń Pixelwise view selection for unstructured multi-view stereo ż[END_REF], as well as crispness [START_REF] Gal | Seamless montage for texturing models ż[END_REF], and consistency with respect to occlusions [START_REF] Waechter | ń Let there be color! largescale texturing of 3D reconstructions ż[END_REF]. While most of these criteria do not directly apply to point clouds, they illustrate the importance of camera pose information for selecting relevant images.

Related to our setting is the Next Best View selection problem [START_REF] William R Scott | ń View planning for automated three-dimensional object reconstruction and inspection ż[END_REF], which consists in planning the camera position giving the most information about an object of interest [START_REF] Connolly | ń The determination of next best views ż[END_REF]. This criterion takes different meanings according to the setting, such as the number of unseen voxels [START_REF] Vasquez-Gomez | ń View/state planning for three-dimensional object reconstruction under uncertainty ż[END_REF], diversity [START_REF] Mokhtarian | ń Automatic selection of optimal views in multi-view object recognition ż[END_REF], information-theoretic measures of uncertainty [START_REF] Isler | An information gain formulation for active volumetric 3D reconstruction ż[END_REF], or can be directly learned end-to-end [START_REF] Mendoza | Supervised learning of the next-best-view for 3D object reconstruction ż[END_REF][START_REF] Wu | ń 3D ShapeNets: A deep representation for volumetric shapes ż[END_REF]. Our setting differs in that the images have already been acquired, and the task is to choose which one contains the most relevant information for each point. We draw inspiration from the end-to-end approaches demonstrating that a neural network can assess the quality of information contained in an image from pose information.

The problem of view selection is also addressed in the literature on shape recognition [START_REF] Su | Multi-view convolutional neural networks for 3D shape recognition ż[END_REF]. Features from different images can be merged based on their similarity [START_REF] Wang | ń Dominant set clustering and pooling for multi-view 3D object recognition ż[END_REF], discriminativity [START_REF] Feng | GVCNN: Group-view convolutional neural networks for 3D shape recognition ż[END_REF], or using patch matching schemes [START_REF] Yang | Learning object bounding boxes for 3D instance segmentation on point clouds ż[END_REF][START_REF] Yu | ń Multi-view harmonized bilinear network for 3D object recognition ż[END_REF] or graph-neural networks [START_REF] Wei | ń View-GCN: View-based graph convolutional network for 3D shape analysis ż[END_REF]. Some methods use 3D features [START_REF] You | PVRNet: Point-view relation neural network for 3D shape recognition ż[END_REF] or camera position [START_REF] Kanezaki | ń RotationNet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints ż[END_REF][START_REF] Hamdi | ń MVTN: Multi-view transformation network for 3D shape recognition ż[END_REF] to select the best views, but no technique yet makes explicit use of the viewing configuration. Furthermore, these methods operate on synthetic views of artificial shapes, which differs from our goal of analyzing large scenes with images in arbitrary poses.

Closer to our problem, Armeni et al . [START_REF] Armeni | ń 3D Scene Graph: A structure for uniőed semantics, 3D space, and camera ż[END_REF] aggregate views using handcrafted heuristics. Bozic et al . [START_REF] Bozic | TransformerFusion: Monocular RGB scene reconstruction using transformers ż[END_REF] use a distance-aware attentive view aggregation for 3D reconstruction, but disregard other viewing conditions.

Method

Let P be a set of 3D points and I a collection of co-registered images, all acquired from the same scene. We characterize points by their position in space, and images by their pixels' RGB values along with intrinsic and extrinsic camera parameters. Our goal is to exploit the correspondence between points and image pixels to perform 3D point cloud semantic segmentation with features learned from both modalities.

Our method starts by computing an occlusion-aware mapping between 3D points and pixels, then uses viewing conditions through an attention scheme to aggregate relevant image features for each 3D point. This approach can be easily integrated into a standard 3D network architecture, allowing us to learn from both point clouds and images simultaneously in an end-to-end fashion.

Point-Image Mapping

We start by efficiently computing a mapping between the images of I and the points of P . We say that a point-image pair (p, i) ∈ P × I is compatible if p is visible in i, i.e. p is in the frustum of i and not occluded. For such a pair, we define the re-projection pix(p, i) as the pixel of i in which p is visible.

Note that as points are zero-dimensional objects (zero-volume), pix(p, i) is a single pixel. We denote by v(p) the views of p, i.e. the set of images in which p is visible.

Point-Pixel Mapping Construction. We operate in a general in the wild multiview setting in which the optical axes of the cameras and the 3D sensor are not necessarily aligned. Consequently, computing the point-image mapping requires a visibility model to detect occlusions. This can be done by computing a full mesh reconstruction from the point clouds or by using a depth map obtained by a camera-aligned depth sensor or other means. In contrast, we propose an efficient implementation of the straightforward Z-buffering method [START_REF] Strasser | ń Fast curve and surface generation for interactive shape design ż[END_REF] to compute the mapping directly from images and point clouds.

For each image i ∈ I, we replace all 3D points in the frustum of i under a pre-determined distance by a square plane section facing towards i and whose size depends on their distance to the sensor and the resolution of the point cloud. We can compute the projection mask-or splat-of each square onto i using the camera parameters of i. We iteratively accumulate all splats in a depth map called Z-buffer by keeping track of the closest point-camera distance for each pixel. Simultaneously, we store corresponding point indices in an index map, along with other relevant point attributes. Once all splats have been accumulated, visible points are the ones whose indices appear in the index map. For each visible point p, we set pix(p, i) as the pixel of i in which p itself is projected. Our GPU-accelerated implementation can process the entire S3DIS dataset [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF] subsampled at 5 cm (12 million points and 1413 high-resolution equirectangular images) within 65 seconds. See Figure 5.2 for an illustration of our mapping construction, and Section 5.4.3 for an analysis of alternate visibility models.

Projection Information. To each compatible point-image pair (p, i), we associate a D-dimensional vector o (p,i) describing the conditions under which the point p is seen in i. In practice, we define this vector as a set of D = 8 handcrafted features qualifying the observation conditions of (p, i).

-Normalized depth (1). An image seeing a point at a distance may contain relevant contextual cues but poor textural information. We -Local geometric descriptors (3). The geometry of a point cloud can impact the quality of its views in images. Indeed, while planar surfaces may be better captured by a camera, a highly irregular surface may present many occlusions or grazing rays. We compute geometric descriptors (linearity, planarity, scattering) based on the eigenvalues of the covariance matrix between a point and its 50 neighbors [START_REF] Demantké | Dimensionality based scale selection in 3D LiDAR point clouds ż[END_REF].

-Viewing angle (1). An image seeing a surface from a right angle may better capture its surroundings than if the view angle is slanted with respect to the surface. We compute the absolute value of the cosine between the viewing angle and the normal estimated from the covariance matrix calculated at the previous step.

-Pixel row (1). To account for potential camera distortion near the top and bottom of the image (e.g. for equirectangular images), we report the row of pixels and divide by the image height (number of rows). Note that we could derive a similar feature for cameras with radial distortion, such as fisheye cameras.

-Local density (1). Density can impact occlusion and be an indicator of the local precision of the 3D sensor. We compute the area of the smallest disk containing the 50th neighbor and normalize it by the square of the voxel grid resolution.

-Occlusion rate (1). Occlusion may significantly impact the quality of the projected image features. We compute the ratio of the 50 nearest neighbors of p also seen in i.

See Section 5.4.3 for an analysis of the impact of these values.

Efficient Implementation. In the Z-buffering step, we only consider points at a maximum distance R = 8 m for indoor scenes and R = 20 m for outdoor settings. We replace the points in image i by cubes oriented towards i and with a size given by the following formula involving dist(p, i) the distance between point p and image i, k = 1 a swell factor ruling how much closer cubes are expanded and c the resolution of the voxel grid, or a typical inter-point distance (2-8cm in our experiments):

size_of_cubes(dist(p, i)) = c(1 + ke -dist(p,i)/R ) . (5.1)
This heuristic increases the size of cubes that are close to the image to ensure that they do hide the cubes behind them. Note that this heuristic operates on the size of the 3D cubes before camera projection and not on their projected pixel masks, which are computed based on camera intrinsic parameters. See Algorithm 2 for the pseudo-code of the mapping computation.

Storing point-pixel mappings for large-scale scenes with many images can be challenging. To minimize the memory impact of such a procedure, we use the Compressed Sparse Row (CSR) format. This allows us to represent the mappings compactly and treat large scenes at once.

Learning Multi-View Aggregation

We denote by {f 2D i } i∈I a set of 2D feature maps of width C associated to the images I, typically obtained with a convolutional neural network (CNN).

Our goal is to transfer these features to the 3D points by exploiting the correspondence between points and images. However, not all viewing images contain equally relevant information for a given 3D point. We propose an attention-based approach to weigh and aggregate features from the viewing images for each point p.

View Features. The mapping pix(p, i) described in Section 5.3.1 allows us to associate image features to each compatible point-image pair (p, i):

f 2D (p,i) = MLP f 2D i [pix(p, i)] , (5.2)
with MLP : R C → R C a Multi-Layer Perceptron (MLP). Learned image features can contain information of different natures: contextual, textural, class-specific, and so on. To reflect this consideration, we split the channels of f 2D (p,i) into K contiguous blocks of ⌊C/K⌋ channels:

f 2D (p,i) = f 2D (p,i),1 , • • • , f 2D (p,i),K . (5.3)
with [ • ] the channel-wise concatenation operator. Each block of channels represents a subset of the image information contained in f 2D .

View Quality. The conditions under which a point is seen in an image can be more or less conducive to certain types of information, see Figure

For example, an image viewing a point from a distance may give important contextual cues, while an image taken close and at a straight angle with respect to the local 3D surface may give detailed textural information. In contrast, an image in which a point's local surface is seen from a slanted angle or under high distortion may not contain relevant information and may need to be discarded. To model these complex dependencies, we propose to predict for each compatible point-image pair (p, i) a set of K quality scores x k (p,i) ∈ R from its viewing conditions o (p,i) defined in Section 5.3.1. The quality x k (p,i) represents the relevance for point p of the information contained in the feature block k of image i .

For each point p, we consider the set v(p) of images in which it is visible.

We propose to learn to predict the view quality x k (p,i) for each feature block k by considering all images i ∈ v(p) simultaneously. Indeed, the relevance of an image can depend on the context of the other views. For example, while a given image may provide less-than-perfect viewing conditions of a given 3D point, it may be the only available image with global information of the point's context. We use a deep set architecture [START_REF] Zaheer | ń Deep sets ż[END_REF] to map the set of viewing conditions {o (p,i) } i∈v(p) to a vector of size K:

z (p,i) = ϕ 1 (o (p,i) )
(5.4)

x (p,i) = ϕ 3 z (p,i) , ϕ 2 max{z (p,i) } i∈v(p) , (5.5)
with ϕ 1 : R D → R M , ϕ 2 : R M → R M , and ϕ 3 : R 2M → R K three MLPs, M the size of the set embedding, and max the channelwise maximum operator for a set of vectors.

View Attention Scores. We can now compute K attention scores a k (p,i) in [0, 1] corresponding to the relative relevance for point p of the kth feature block of image i. The attentions are obtained by applying a softmax function to the quality scores x k (p,i) across the images in v(p). To account for the possibly varying number of views per point, we scale the softmax according to the number of images seeing the point p :

a k (p,i) = softmax 1 |v(p)| x k (p,i) i∈v(p)
.

(

View Gating. A limitation of using a softmax in this context is that the attention scores ãk (p,i) always sum to 1 over v(p) regardless of the overall quality of the image set. Because of occlusion or limited viewpoints, some 3D points may not be seen by any relevant image for a given feature block k (e.g. no close or far images). In this case, it may be beneficial to discard an information block from all images altogether and purely rely on geometry. This allows the 2D network to learn image features without accounting for potentially spreading corrupted information to points with dubious viewing conditions. To this end, we introduce a gating parameter g k p whose role is to block the transfer of the features block k if the overall quality of the image set v(p) is too low:

g k p = ReLU tanh α k max i∈v(p) x k (p,i) + β k , (5.7)
with α, β ∈ R K trainable parameters and ReLU the rectified linear activation [START_REF] Nair | ń Rectiőed linear units improve restricted boltzmann machines ż[END_REF]. If all quality scores x k (p,i) are negative for a given point p and block k, the gating parameter g k p will be exactly zero and block possibly detrimental information due to sub-par viewing conditions.

Attentive Image Feature Pooling. For each point p seen in one or more images, we merge the feature maps f 2D (p,i) from each view (p, i). For each block k, we compute the sum of the view features f 2D (p,i),k weighted by their respective attention scores a k (p,i) and multiplied by the gating parameter g k p . The combined image feature P(f 2D , p) associated to point p is then defined as the channelwise concatenation of the resulting tensors for all blocks:

P(f 2D , p) =   g k p i∈v (p) a k (p,i) f 2D (p,i),k   K k=1 . (5.8)

Bimodal Point-Image Network.

We can use the multi-view feature aggregation method described above to perform semantic segmentation of a point cloud and co-registered images by combining a network operating on 3D point clouds and 2D CNN.

Fusion Strategies. We use a 2D fully convolutional network to compute pixel-wise image feature maps f 2D . We also consider a 3D deep network following the classic U -Net architecture [START_REF] Ronneberger | ń U-Net: Convolutional networks for biomedical image segmentation ż[END_REF] and composed of three parts:

(i) an encoder E 3D mapping the point cloud into a set of 3D feature maps at different resolution (innermost map and skip connections); (ii) a decoder D 3D converting these maps into a 3D feature map at the highest resolution (iii) a classifier C 3D associating to each point a vector for class scores of size N , the number of target classes. As shown in Figure 5.4, we investigate three classic fusion schemes [START_REF] Hazirbas | FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture ż[END_REF][START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF][START_REF] Krispel | FuseSeg: LiDAR point cloud segmentation fusing multi-modal data ż[END_REF], connecting the image features at different points of the 3D network:

(i) directly with the raw 3D features before E 3D (early fusion), (ii) in the skip connections (intermediate fusion) (iii) between the decoder D 3D and the classifier C 3D (late fusion:). See the Section D-2 for the details and equations for these fusion schemes.

Dynamic-Size Image-Batching. The number of images v(p) in which a point p is visible can vary significantly. Furthermore, when dealing with large-scale scenes, only a subset P sample of the 3D scene is typically processed at once (e.g. spherical sampling). For this reason, the part of an image i for which points of P sample are visible can sometimes be only a small fraction of the entire image.

This will typically occur with equirectangular images or when P sample is far away from i. We use the adaptive batching scheme depicted in Figure 5.5 to stabilize memory usage across batches and avoid needless computations on excessively large images. The first step is to crop each image using the smallest window across a fixed set of sizes (e.g. 64 × 64, 128 × 64, etc.) such that the crop contains the bounding box of all seen points of P with a given margin. Observing that the memory consumption of a fully convolutional encoder is linear w.r.t. the number of input pixels, we allocate to each point cloud in the batch a budget of pixels. Images are then chosen randomly by iteratively selecting images with a probability proportional to their number of pixels and to the number of newly seen points in the cloud, until the pixel budget is spent. Finally, the images are organized into different batches according to their sizes, allowing for their simultaneous processing. Note that at inference time, we can take batches as large as the GPU memory allows.

More details on our dynamic-size batching implementation are provided in Section D-3.

Implementation Details

We use sparse encoding for mappings in order to only store compatible point-image pairs. This proves necessary for the large scale, in-the-wild setting with varying number of images seeing each point. The exact network and training configurations are given in Section D-4. Our code is available at https://github.com/drprojects/DeepViewAgg.

Experiments

We propose several experiments on public large-scale semantic segmentation benchmarks to demonstrate the benefits of our deep multi-view aggregation module (DeepViewAgg). Our approach yields significantly better results than our 3D backbone directly operating on colorized point clouds. We set a new state-of-the-art for the highly contested S3DIS benchmark using only standard 2D and 3D architectures combined with our proposed module.

Datasets

S3DIS [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF]. This indoor dataset of office buildings contains over 278 million semantically annotated 3D points across 6 building areas-or folds. A companion dataset can be downloaded at https://github.com/alexsax/2D-3D-Semantics, and contains 1413 equirectangular images. To represent our large-scale, in-the-wild setting, we merge each fold into a large point cloud and discard all room-related information. We apply minor registration adjustments detailed in Section D-5. ScanNet [START_REF] Dai | ń ScanNet: Richly-annotated 3D reconstructions of indoor scenes ż[END_REF]. This indoor dataset contains over 1501 scenes obtained from 2.5 million RGB-D images with pose information. To account for the high redundancy between images, we select one in every 50 image. This dataset deviates slightly from our intended setting as 2D and 3D are derived from the same sensors.

KITTI-360 [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF]. This large outdoor dataset contains over 100k laser scans and 320k images captured with a multi-sensor mobile platform. We use one image every five from the left perspective camera. We report the class-wise performance on the official withheld test set.

General Setting. All datasets provide colorized point clouds obtained with dataset-specific preprocessings. To handle the large size of scans, we define batches using a sampling strategy for S3DIS (2 m-radius spheres) and KITTI-360 (6 m-radius vertical cylinders), while we process ScanNet room-by-room, see Figure 5.6. We down-sample the point clouds for processing (S3DIS: 2cm, ScanNet: 3cm, KITTI-360: 5cm) and interpolate our prediction to full resolution for evaluation. To mitigate the memory impact of the 2D encoder, we also down-sample S3DIS images to 1024 × 512 but keep the full resolution for ScanNet (320 × 240) and KITTI-360 (1408 × 376). In Table 5.1, we compare the performance of our approach and other learning methods on S3DIS, ScanNet Validation, and KITTI-360 Test using the class-wise mean Intersection-over-Union (mIoU) as metric. Our method (DeepViewAgg) uses images in the 2D encoder and raw uncolored point clouds in the 3D encoder. All other approaches, including our backbone (3D Backbone), use the colorized point clouds provided by the datasets.

Quantitative Evaluation

DeepViewAgg sets a new state-of-the-art for S3DIS for all 6 folds and the second-highest performance for the 5th fold. In particular, we outperform the VMVF network [START_REF] Kundu | ń Virtual multi-view fusion for 3D semantic segmentation ż[END_REF], showing that our multi-view aggregation model can overtake methods relying on costly virtual view generation using only available images. Furthermore, VMVF uses true depth maps, colorized point clouds, normals, and room-wise normalized information. In contrast, our method only uses raw XYZ data in the 3D encoder and estimates the mappings. Our approach also overtakes the recent PointTransformer [START_REF] Engel | ń Point transformer ż[END_REF] (PointTrans.) by 1.2 mIoU points, even though this method outperforms our 3D backbone by 4 points on colorized points. Our model also improves the performance of our 3D backbone on the KITTI-360 test set by 4.4 points, illustrating the importance of images for both indoor and outdoor datasets alike.

While giving reasonable results, our method does not perform as well on the validation set of ScanNet comparatively. We outperform the 2D/3D fusion method of BPNet [START_REF] Hu | Bidirectional projection network for cross dimension scene understanding ż[END_REF] when restricted to 3D annotations, illustrating the importance of view selection. We argue that the limited variety in the camera points of view of ScanNet RGB-D scans, as well as their small field-of-view and blurriness reduce the quality of the information provided by images. This is reinforced by the impressive performance of VMVF, which synthesizes its own images with controlled points of view and resolution. See

Analysis

We conduct further analyses on Fold 5 and Fold 2 of S3DIS (subsampled at 5cm for processing) and the validation set of KITTI-360 in Table 5.2. We added Fold 2 along the commonly used Fold 5, as it benefited most from our method, and hence is more conducive to evaluating the impact of our design choices.

Modality Combinations. As observed in Table 5.2, combining a 3D deep network operating on raw 3D features and a 2D network with our method (Best Configuration) improves the performance by over 6 to 15 points compared to the same 3D backbone operating on colorized point clouds alone (XYZRGB).

To illustrate that point colorization is not a trivial task, we train our 3D backbone with point clouds colorized by averaging for each point the color of all pixels in which it is visible (XYZ Average-RGB). Compared to the "official" colored point clouds, we observe a drop of 1 point for Fold 5 and 1.3 point for KITTI-360, but a gain of almost 5 points for Fold 2. This shows how different point cloud colorization schemes can yield vastly different results. Not using any radiometric information and purely relying on 3D points without color (XYZ) decreases the score of XYZRGB by a further 3 to 4 points on S3DIS.

For KITTI-360, XYZ outperforms XYZ Average-RGB, suggesting that poor colorization can even be detrimental.

We also evaluate a scheme in which the 3D network is entirely removed, and 3D points are classified solely based on features coming from a 2D encoderdecoder and our view aggregation module, without any 3D convolution (Pure RGB). This method outperforms even (XYZRGB) for S3DIS, illustrating the relevance of images for point cloud segmentation. On KITTI-360, as many 3D points are not seen by the cameras used, this approach perform worse.

Training our best 2D+3D model with images downsampled by a factor of 2 (Lower Image Resolution) brings a large performance drop. In contrast, using 2 cm of 3D resolution instead of 5 cm (Higher 3D Resolution) has little impact for S3DIS. We conclude that when the images already contain fine-grained information, the impact of the resolution of the 3D voxel grid decreases. Using featurewise max-pooling to merge the views results in a drop of 1 to 1.5 points for S3DIS and 3 points for KITTI-360. This illustrates that as long as we employ proper mapping, batching, and pretraining strategies, even simple pooling operations can perform very well. However, the addition of our model appears necessary to improve the precision even further and reach state-of-the-art results.

Switching our 3D backbone to a lighter version of MinkowskiNet with decreased widths, we observe no significant impact on the prediction quality.

This suggests that we could use our approach successfully with smaller models.

Influence of the Viewing Conditions. We propose to highlight the role viewing conditions descriptor. In Table 5.3, we estimate the usage by our model of each feature as the drop in mIoU on S3DIS Fold 5 & KITTI-360 when they are replaced by their dataset average (e.g. all points appear at the same distance). We also measure the feature sensitivity by averaging the squared partial derivative [95, p. 3.3.1] of the view compatibility score x defined in (4) w.r.t. each view descriptor. We observe that our model makes use of all observation features, and that the compatibility scores are most sensitive to small differences in scattering for S3DIS Fold 5, and depth for KITTI-360

Val.

To visualize the influence of viewing conditions, we represent in Figure 5.8 quality score heatmaps when varying pairs of features for a given view point from S3DIS.

Influence of the Visibility Model. We propose further ablations whose results in Table 5.4. To assess the quality of our visibility model, we propose to compute the point-pixel mappings using the depth maps provided with S3DIS instead of Z-buffering. When running our method with such mappings (Mapping from Depth), we observe lower performances. This can be explained by the fact that depth-based mapping computation is sensitive to minor discrepancies between the depth map and the real point positions. Such a phenomenon can be observed on S3DIS depth images where the surfaces are viewed from a slanted angle, resulting in fewer point-image mappings being recovered. See Figure 5.9 for an illustration of this phenomenon. In conclusion, not only can our approach bypass the need for specialized sensors or costly mesh reconstruction altogether, but our direct point-pixel mapping may yield better results than the provided mappings obtained with more involved methods.

To compare our fusion schemes, we evaluate a model with the intermediate fusion scheme described in D-3 (Intermediate). We observe that, for our module, intermediate fusion does not perform as well as early and late fusion. This could indicate that fusing modalities at their highest respective resolutions yields better results and that matching the encoder levels of 2D and 3D networks may not be straightforward. To ensure that our proposed module captures all radiometric information contained in colorized point clouds, we trained our chosen architecture to run on colorized point clouds and images (XYZRGB + DeepViewAgg). The resulting performance confirms that colorizing 3D points does not bring additional information not already captured by images.

Influence of the Maximum Depth. The maximum point-image depth is chosen as the distance beyond which adjacent 3D points appear in the same image pixel: 8 m for S3DIS sampled at 5 cm with images of width 1024 and 20 m for KITTI-360. As illustrated in Table 5.5, reducing this parameter too much leads to a drop in performance both on S3DIS Fold 5 and KITTI-360, while slight modifications do not significantly affect the results. Since the number of point-image mappings grows quadratically with this parameter, one may consider smaller values to decrease the memory usage or computing time.

Influence of the Number of Images. In contrast to existing methods (e.g.

MVPNet [START_REF] Jaritz | ń Multi-view pointnet for 3D scene understanding ż[END_REF], VMVF [START_REF] Kundu | ń Virtual multi-view fusion for 3D semantic segmentation ż[END_REF], BPNet [START_REF] Hu | Bidirectional projection network for cross dimension scene understanding ż[END_REF]), our set-based, sparse implementa-tion of point-image mappings allows us to have a varying number of views per 3D point. In Table 5.6 we investigate the performance drop w.r.t. our best model when limiting the number of images per point cloud to a fixed number of images and not using dynamic batching. Under these conditions, our performance decreases by 6.8 pts on S3DIS Fold 5 and 3.5 pts on KITTI-360 when using only 3 images, i.e. the configuration of BPNet. For comparison, 3D points of S3DIS are seen in 5.0 images on average (STD 3.3), and 2.5 for KITTI-360 (STD 2.1).

Limitations

While our method does not require sensors with aligned optical axes, true depth maps, or a meshing step, we still need camera poses. In some "in the wild" settings, they may not be available or require a pose estimation and registration step which may be costly and error-prone. Our mapping computation also relies on the assumption that the 2D and 3D modalities are acquired simultaneously.

Our multi-view aggregation method operates purely on viewing conditions and does not take the geometric and radiometric features into account in the computation of attention scores. We implemented a self-attention-based approach using such features, which resulted in a significant increase in memory usage without tangible benefits: the viewing conditions appear to be the most critical factor when selecting and aggregating images features.

Conclusion

We proposed a deep learning-based multi-view aggregation model for the semantic segmentation of large 3D scenes. Our approach uses the viewing condition of 3D points in images to select and merge the most relevant 2D features with 3D information. Combined with standard image and point cloud encoders, our method improves the state-of-the-art for two different datasets. Our full pipeline can run on a point cloud and a set of co-registered images at arbitrary positions without requiring colorization, meshing, or true depth maps. These promising results illustrate the relevancy of using dedicated architectures for extracting information from images even for 3D scene analysis. This concluding chapter recapitulates our contributions in Section 6.1, before sketching out in Section 6.2 future research directions that we identify as promising.

Contributions

This thesis advances the field of 3D computer vision with novel methods for the efficient and scalable analysis of 3D point clouds. Our three contributions can be summarized as follows.

Efficient and Scalable 3D Semantic Segmentation. In Chapter 3, we propose Superpoint Transformer, a novel approach for the efficient semantic segmentation of large 3D point clouds. Our method brings together the best of two worlds: the efficiency of superpoint-based methods and the expressivity of transformer-based methods. We achieve state-of-the-art performance on three 3D semantic segmentation benchmarks while being up to 200× more parameter-efficient and 70× faster to train than competing approaches.

This work illustrates how designing an adequate data structure with strong inductive priors can render multimillion-parameter models futile.

Efficient and Scalable 3D Panoptic Segmentation. In Chapter 4, we formulate 3D panoptic segmentation as a scalable graph partitioning problem, which our small SuperCluster model is trained to address. Contrary to existing 3D instance and panoptic segmentation, our supervision relies on local objectives only, which allows us to circumvent several limitations of competing methods and scale to very large scenes. We reach state-of-the-art performance on four 3D panoptic segmentation benchmarks and show that our method can process scenes of unprecedented size at once on a single consumer-grade GPU.

Learning From Point Clouds and Images in the Wild. In Chapter 5, we introduce DeepViewAgg, an end-to-end multi-view aggregation method for 3D semantic segmentation from images and point clouds. Contrary to existing 2D-3D methods, our approach does not require point cloud colorization, meshing, or depth sensors: only point clouds, images, and their poses. Despite these minimalistic requirements, we achieve state-of-the-art performance on two 3D semantic segmentation benchmarks. In a way, this work can be seen as learning to colorize point clouds with features from arbitrarily posed images.

Perspectives

The works presented in this thesis open the door to several exciting research directions.

Superpoint-Based Learning

We believe that superpoint-based methods introduced in Chapter 3 and Chapter 4 pave the way for promising efficient 3D point cloud processing approaches.

Model Expressivity. Currently, SPT extracts point features using a small PointNet-like model [START_REF] Charles R Qi | PointNet: Deep learning on point sets for 3D classiőcation and segmentation ż[END_REF] for each superpoint taken in isolation. This approach may lack expressivity and ignores the local relationship between points at the border between superpoints. Inspired by other transformer-based architectures for 2D [START_REF] Liu | Swin Transformer: Hierarchical vision transformer using shifted windows ż[END_REF] and 3D [START_REF] Lai | ń Stratiőed transformer for 3D point cloud segmentation ż[END_REF] analysis that rely on convolution-based modules to extract features from the raw input signal, we hypothesize that our model may benefit from a more expressive point encoder inspired by KPConv [START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF] or MinkowskiNet [START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF]. Learnable Partition. Following SGP [START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF], our framework in Chapter 3 relies on the cut-pursuit algorithm [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions on general weighted graphs ż[END_REF][START_REF] Raguet | ń Parallel cut pursuit for minimization of the graph total variation ż[END_REF] for computing a fixed partition, at preprocessing time, and based on handcrafted point features. Obviously enough, learning to partition end-to-end would be an attractive alternative.

However, our cut-pursuit step is currently a non-differentiable operation. To this end, Landrieu et al . [START_REF] Landrieu | ń Point cloud oversegmentation with graphstructured deep metric learning ż[END_REF] propose to pretrain with a dedicated network to learn point features presenting a high contrast at object boundaries, to improve the partition with cut-pursuit. Unfortunately, this approach involves a two-step training and requires instance-level annotations. An end-to-end trainable solution leveraging semantic segmentation labels or no labels at all would be a more convenient solution.

Several directions could be explored. First, the partition could be updated every few training epochs, based on point features learned for the task at hand. Yet, with this approach, the gradient would not be used to adjust the partition itself. An alternative would be to treat the partition step as a blackbox combinatorial optimization step [START_REF] Vlastelica | Differentiation of blackbox combinatorial solvers ż[END_REF]. Finally and more ambitiously, the partition step could be made entirely differentiable. Although this may require substantial theoretical work for cut-pursuit, differentiable alternatives exist for superpixel [START_REF] Jampani | ń Superpixel sampling networks ż[END_REF] or superpoint [START_REF] Hui | Superpoint network for point cloud oversegmentation ż[END_REF] partitions. However, these approaches are essentially variants of k-means that operate on a fixed number of sampled centroids, which limits their ability to scale to point clouds of arbitrary size or to adapt to the geometric complexity.

Super-X Reasoning. While the methods presented in Chapter 3 and Chapter 4 operate on point clouds, adapting our framework to other modalities, such as images, depth maps, or videos would be a natural next step. Essentially, operating on a new modality would be easy and would only require defining an adjacency graph and local features for the partition step, as well as a trainable modality-specific feature extractor, which is likely to be found in the relevant literature.

2D-3D Learning

Our work in Chapter 5 may be further extended for multimodal learning on point clouds and images.

Multi-View Multi-Sensor Aggregation. Some acquisition systems, such as autonomous vehicles, can be equipped with several image sensors with different optical properties. For example, the KITTI-360 [START_REF] Liao | ń KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D ż[END_REF] dataset comprises point clouds and images from perspective and fisheye cameras located at different positions on the vehicle. By extending the observation condition features with information about the sensor of each image, we may be able to train DeepViewAgg to selectively attend to different sensors depending on the viewing conditions. For example, our method may learn that the distortion in fisheye images makes pixel features less reliable as we move away from the center. Meanwhile, the model may also learn that some cameras may be more adequate than others to detect pedestrians, based on their orientation on the acquisition platform.

Multimodal Aggregation. DeepViewAgg proposes to exploit the synergy between point clouds and localized images. However, other modalities may carry complementary information about the 3D scene. Radar sensors, for example, are more robust to visibility and weather conditions than RGB cameras, making them ideal for autonomous driving, as exemplified by the nuScenes [START_REF] Holger | ń nuScenes: A multimodal dataset for autonomous driving ż[END_REF] dataset. Other potentially informative modalities include HD maps, street camera images, aerial images, and satellite images. Provided a feature extractor and a mapping to 3D points for each modality, we believe the attentive, multi-view formulation of our framework could allow for more than two modalities.

Robust 2D-3D Mapping. Similar to other works on 2D-3D learning, the mapping construction in DeepViewAgg makes two important assumptions: static scenes and known camera parameters.

If an object has moved between the acquisition times of the 3D and 2D sensors, our purely geometric mapping construction will be incorrect. Yet, such dynamic scenes are ubiquitous in robotic and autonomous driving scenarios,

where surrounding objects such as vehicles or pedestrians may be dynamic.

Taking temporality into account for multimodal mapping and multi-view aggregation would make for an interesting extension of DeepViewAgg. This problem is related to moving object segmentation [START_REF] Prashant | An end-to-end edge aggregation network for moving object segmentation ż[END_REF][START_REF] Kim | ń Remove, then revert: Static point cloud map construction using multiresolution range images ż[END_REF][START_REF] Chen | Moving object segmentation in 3D LiDAR data: A learningbased approach exploiting sequential data ż[END_REF][START_REF] Kreutz | ń Unsupervised 4D LiDAR moving object segmentation in stationary settings with multivariate occupancy time series ż[END_REF], which is an active research field.

Our mapping computation will also suffer from incorrect camera poses or intrinsics. This typically affects small pixel structures, which correspond to either small or far-away 3D objects. Existing works on 2D-3D registration [START_REF] Li | ń DeepI2P: Image-to-point cloud registration via deep classiőcation ż[END_REF][START_REF] Djahel | ń A 3D segments based algorithm for heterogeneous data registration ż[END_REF] could help alleviate this issue, making our method more robust to camera parameter errors.

Super-X Multimodal Learning. Combining the superpoint and multimodal paradigms developed in this thesis offers multiple possibilities. Using superpoints may increase the robustness of DeepViewAgg. Indeed, aggregating pixel features at the superpoint level rather than for individual points may lead to more robust features. In addition, more robust observation condition features may also be computed at the superpoint level.

Aligning superpixel and superpoint partitions provides a promising framework for large-scale, cross-modal self-supervised learning. For instance, SLidR [START_REF] Sautier | ń Image-to-LiDAR self-supervised distillation for autonomous driving data ż[END_REF] learns to distill image features into a 3D model, by partitioning images with SLIC and projecting the resulting superpixels onto 2.5D LiDAR range images. However, this approach does not handle scenes with multiple arbitrarily-posed 2D views and occlusions. Besides, it relies on images to partition the point cloud, ignoring the available 3D geometric information.

We expect several building blocks introduced in this thesis to be helpful in exploring these directions. We observe that both outdoor datasets strongly benefit from local geometric features, which we hypothesize is due to their lower resolution and noise level.

These results indicate that radiometric features play an important role for all datasets and that geometric features may facilitate learning on noisy or subsampled datasets.

b) Adjacency Features. The analysis of the impact of adjacency features on our model's performance indicates that they play a crucial role in leveraging contextual information from superpoints: removing all adjacency features leads to a significant drop of 3.0 to 6.3 mIoU points on the datasets, as shown in Section 3.3.3. Among the different types of adjacency features, pose features appear particularly useful in characterizing the adjacency relationships between superpoints of S3DIS, while interface features have a smaller impact.

These results suggest that the relative pose of objects in the scene may have more influence on the 3D semantic analysis performed by our model than the precise characterization of their interface. On the other hand, interface and ratio features seem to have more impact on outdoor datasets, while the pose information seems to be less informative in the semantic understanding of the scene.

c) S3DIS Room Partition. The S3DIS dataset is divided into individual rooms aligned along the x and y axes. This setup simplifies the identification of classes such as walls, doors, or windows as they are consistently located at the edge of the room samples. Some methods also add normalized room coordinates to each points. However, we argue that this partition may not generalize well to other environments, such as open offices, industrial facilities, or mobile mapping acquisitions, which cannot naturally be split into rooms.

To address this limitation, we use the absolute room positions to reconstruct the entire floor of each S3DIS area [START_REF] Thomas | KPConv: Flexible and deformable convolution for point clouds ż[END_REF][START_REF] Chaton | Torch-Points3D: A modular multi-task framework for reproducible deep learning on 3D point clouds ż[END_REF]. This enables our model to consider large multi-room samples, resulting in a performance increase of 3.8 points. This highlights the advantage of capturing long-range contextual information. Additionally, we remark that SPT performs better without using room-normalized coordinates, which may lead to overfitting and poor performance on layouts that deviate from the room-based structure of the S3DIS dataset such as large amphitheaters.

B-4 Details on Hierarchical Partitions

We present here a more detailed explanation of the hierarchical partition process. We define for each point c of C a feature f c of dimension D, and G := (C, E, w) is the k-nn adjacency between the points, with w ∈ R E + a nonnegative proximity value. Our goal is to compute a hierarchical multilevel partition of the point cloud into superpoints homogeneous with respect to f at increasing coarseness.

Piecewise Constant Approximation on a Graph. We first explain how to compute a single-level partition of the point cloud. We consider the pointwise features f c as a D-dimensional signal f ∈ R D×|C| defined on the nodes of the weighted graph G := (C, E, w). We first define an energy J (e; f, G, λ) measuring the fidelity between a vertex-valued signal e ∈ R D×|C| and the length of its contours, defined as the weight of the cut between its constant components [START_REF] Landrieu | ń Cut Pursuit: Fast algorithms to learn piecewise constant functions ż[END_REF]: Geometric features ensure that the superpoints are geometrically homogeneous and with simple shapes. We use the normalized dimensionality-based method described in Section 3.2.4. Radiometric features encourage the border of superpoints to follow the color contrast of the scene and are either RGB or intensity values; they must be normalized to fall in the [0,1] range. Lastly, we can add to each point their spatial coordinates with a normalization factor µ in m -1 to limit the size of the superpoints. We recommend setting µ as the inverse of the maximum radius expected for a superpoint: the largest sought object (facade, wall, roof) or an application-dependent constraint.

J (e; f, G, λ) := ∥e -f ∥ 2 + λ (u,v)∈E w u,v [e u ̸ = e v ] , (
The coarseness of the partitions depends on the regularization strength λ as defined in Section 3.2.1. Finer partitions should generally lead to better results but to an increase in training time and memory requirement. We chose a ratio | P 0 | / | P 1 |∼ 30 across all datasets as it proved to be a good compromise between efficiency and precision. Depending on the desired trade-off, different ratios can be chosen by trying other values of λ.

B-6 Implementation Details

We provide the exact parameterization of the SPT architecture used for our experiments. All MLPs in the architecture use LeakyReLU activations and GraphNorm [START_REF] Cai | GraphNorm: A principled approach to accelerating graph neural network training ż[END_REF] normalization. For simplicity, we represent an MLP by the list of its layer widths: [in_channels, hidden_channels, out_channels]. For all datasets, we reduce the output dimension of the point encoder ϕ 0 enc from 128 to 64 [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF]. We find that this does not affect SPT performance while reducing its memory impact.

For ScanNet, we find that using 32 heads instead of 16 and setting D adj = 64 instead of 32 [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF] improves performance.

Object Agreement Head. The object agreement prediction head ϕ object is a normalization-free MLP with LeakyReLU activations and layers [2D, 32, 1],

where D is the output feature dimension of the backbone (i.e. 64 for S3DIS and DALES, and 128 for ScanNet and KITTI-360).

C-4 Detailed Results

We report in We consider P sample a portion of a point cloud to add to the 3D part of a batch. In order to build the image batch we iteratively select images according to the following procedure. We first select the image set I sample seeing at least one point of P sample . For equirectangular images, we rotate the images to place the mappings at the center. We then crop each image i along the Unless specified otherwise, ReLU activation and batch normalization [START_REF] Ioffe | ń Batch normalization: Accelerating deep network training by reducing internal covariate shift ż[END_REF] are used across the architecture.

For our 2D encoders, we use the encoder part of pretrained ResNet18 networks [START_REF] He | Deep residual learning for image recognition ż[END_REF]. For indoor scenes, we use the modified ResNet18 from [START_REF] Csailvision | Semantic-segmentation-pytorch[END_REF] pretrained on ADE20K [START_REF] Zhou | ń Scene parsing through ade20k dataset ż[END_REF], which has 5 layers of output channel sizes [START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Dahl | Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition ż[END_REF][START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Quana | Hierarchical semantic segmentation of urban scene point clouds via group proposal and graph attention network ż[END_REF]512] and resolution scale = [4,4,[START_REF] Alayrac | ń Flamingo: A visual language model for few-shot learning ż[END_REF][START_REF] Alayrac | ń Flamingo: A visual language model for few-shot learning ż[END_REF][START_REF] Alayrac | ń Flamingo: A visual language model for few-shot learning ż[END_REF]. The relatively high resolution of the output feature map allows us to map the 2D features of size C = 512 from the last layer directly to the point cloud without upsampling. For outdoor scenes, we use the ResNet18 from [START_REF]Sfsegnets[END_REF] pretrained on Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding ż[END_REF], which has 5 layers of output channel sizes [START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Dahl | Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition ż[END_REF][START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Quana | Hierarchical semantic segmentation of urban scene point clouds via group proposal and graph attention network ż[END_REF]512] and resolution scale = [4,4,[START_REF] Alayrac | ń Flamingo: A visual language model for few-shot learning ż[END_REF][START_REF] Balestriero | A cookbook of self-supervised learning ż[END_REF][START_REF] Brown | ń Language models are few-shot learners ż[END_REF]. We use pyramid feature pooling [START_REF] Zhao | ń Pyramid scene parsing network ż[END_REF] on layers 1, 2, 3, and 4, which results in a feature vector of size C = 960 passed to the mapped points. For indoor scenes, we use the lowest resolution map of a network [START_REF] Csailvision | Semantic-segmentation-pytorch[END_REF] pretrained on ADE20K [START_REF] Zhou | ń Scene parsing through ade20k dataset ż[END_REF]. For the outdoor scenes, we use pyramid feature pooling [START_REF] Zhao | ń Pyramid scene parsing network ż[END_REF] with a network [START_REF]Sfsegnets[END_REF] pretrained on Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding ż[END_REF]. We use early fusion in all experiments unless specified otherwise.

E-6 Conclusion

Ce chapitre de conclusion récapitule nos contributions, avant d'esquisser les orientations de recherche futures que nous identifions comme prometteuses.

Cette thèse apporte des contributions significatives à la vision par ordinateur 3D, avec trois méthodes novatrices : Ces travaux ouvrent des perspectives pour des recherches futures dans le domaine de la vision 3D.

Apprentissage basé sur les superpoints.

-Expressivité du modèle : Pour améliorer l'expressivité de SPT, l'utilisation d'un encodeur de points plus expressif inspiré de KPConv ou MinkowskiNet est envisagée.

-Tokenisation des nuages de points : La partition des nuages de points en primitives géométriquement homogènes pourrait devenir une étape de prétraitement standard, similaire à la tokenisation pour le traitement du langage naturel.

-Partition apprenable : L'apprentissage de la partition en temps réel est envisagé, bien que la non-différentiabilité de l'étape de partition actuelle soit un défi.

Figure 1 . 1 ś

 11 Figure 1.1 ś Large-Scale 3D Scene Understanding. We develop methods for the efficient semantic analysis of large 3D point clouds. Other methods analyze indoor 3D scenes room by room, ours can consider hundred-room buildings at once, in a few seconds on a single GPU.

Figure 1 . 2 ś

 12 Figure 1.2 ś The Rising Popularity Deep Learning. Historic occurrence of łdeep learningž in academic articles on Google Scholar between 2000 and 2020.

Figure 1 . 3 ś

 13 Figure 1.3 ś Learning Image Rotations. The self-supervised learning task proposed in RotNet[START_REF] Gidaris | ń Unsupervised representation learning by predicting image rotations ż[END_REF] is the following. Given an image rotated by a random multiple of 90°, the model is tasked with learning to recover the rotation angle. The authors show that addressing this task requires learning meaningful semantic representations of the image content: the sky is up, the ground is down, eyes are above the nose which is above the mouth, etc. Source:[START_REF] Gidaris | ń Unsupervised representation learning by predicting image rotations ż[END_REF]

  (a) Structure From Motion[START_REF] Kneip | ń Real-time scalable structure from motion: From fundamental geometric vision to collaborative mapping ż[END_REF] (b) Depth Camera[START_REF] Silberman | ń Indoor segmentation and support inference from RGBD images ż[END_REF] (c) LiDAR[START_REF] Geospatialworld | What is LiDAR technology and how does it work?[END_REF]

Figure 1 . 4 ś

 14 Figure 1.4 ś 3D Point Cloud Acquisition Techniques. Multiple acquisition methods exist to capture 3D scenes. Structure From Motion 1.4a relies on stereo vision to reconstruct a 3D scene from multiple images. Depth cameras 1.4b use structured light patterns to recover depth from a single image. LiDAR 1.4c emits laser pulses and measures the time of ŕight to infer depth.

Figure 1 . 5 ś

 15 Figure 1.5 ś Domains of Application for 3D Point Cloud Processing. Point clouds from diverse acquisition platforms may be used in a variety of applications.

Figure 1 . 6 ś

 16 Figure 1.6 ś Size of Language Models Across Time.The number of parameters of language models has been experiencing exponential growth over the past years. Although scaling model, dataset, and compute has proven to consistently improve performance[START_REF] Kaplan | Scaling laws for neural language models ż[END_REF], these large language models also have negative environmental and social repercussions. Figure taken from a 2020 Microsoft blog post[START_REF] Rosset | Turing-NLG: A 17-billion-parameter language model by microsoft[END_REF]

Figure 1 . 7 ś

 17 Figure 1.7 ś Challenges of 3D Point Cloud Analysis. Some point clouds acquisition characteristics make 3D processing challenging. In this sample from the Semantic3D [111] dataset, we notice the absence of connectivity between points, occlusions, missing data, and radial sampling density.

Figure 1 . 8 ś

 18 Figure 1.8 ś Hierarchical Representation. Our method takes as input a point cloud (a) and computes its hierarchical partition into geometrically homogeneous superpoints at multiple scales: (b) and (c).

Figure 1 . 9 ś

 19 Figure1.9 ś Multimodal Fusion. Point clouds capture the geometry of the scene, while images capture textures and context. The pink halo in the images illustrates which pixels łseež a 3D point from the sample. This mapping is necessary to both point cloud colorization and 2D-3D learning. We seek methods capable of building such mappings and fusing information from both point clouds and arbitrarily-posed images, with minimal sensor and preprocessing requirements.

Figure 2 .

 2 Figure 2.1 ś 3D Deep Learning Methods. A variety of methods have been proposed to extract features from point clouds 2.1a. Some approaches discard information by rendering 2D views of the points 2.1b, or discretizing the cloud in a voxel grid 2.1c. Other methods directly operate on unordered sets of points 2.1d, or generalize discrete 2D convolutions to continuous 3D space 2.1e. Alternatives use graph neural networks 2.1f or transformers 2.1g to reason on a local neighborhood. Closely related to our work, superpoint-based approaches 2.1h reason on a partition of the scene.

Figure 2 . 2 ś

 22 Figure 2.2 ś Pruning.A simpliőed illustration of pruning weights (connections) and neurons (nodes) in a neural network comprising fully connected layers. Source:[START_REF] Menghani | Efficient deep learning: A survey on making deep learning models smaller, faster, and better ż[END_REF]

  saves memory at training time by discarding intermediate activations in the forward pass and recomputing them during backpropagation.

Figure 2 . 3 ś

 23 Figure 2.3 ś Depth-Separable Convolution.By separating the traditional convolution operation into a per-channel convolution followed by a 1x1 channel-mixing operation, depth-separable convolution[START_REF] Chollet | ń Xception: Deep learning with depthwise separable convolutions ż[END_REF] reduce the number of parameters per convolutional layer. Source:[START_REF] Tsang | Review: Xception -with depthwise separable convolution, better than Inception-V3 (image classification)[END_REF]

Figure 2 . 4 ś

 24 Figure 2.4 ś SLIC Superpixel Partition. SLIC[3] is an image partitioning algorithm that segments images into compact, uniform superpixels. Here, two images are partitioned using SLIC with different resolutions. Source:[3]

Figure 2 .

 2 Figure 2.4 for examples of superpixel partitions produced by SLIC. Subsequent works propose different features[START_REF] Li | ń Superpixel segmentation using linear spectral clustering ż[END_REF][START_REF] Liu | Manifold SLIC: A fast method to compute content-sensitive superpixels ż[END_REF], make SLIC faster[2], or

(

  SPT), introduced in Chapter 3, uses a multiscale hierarchical structure that adapts to the local geometry of the data. This hierarchical partition conforms more closely to semantic boundaries than grid-based structures, enabling the network to model the interactions between objects or object parts.

Figure 2 . 5 ś

 25 Figure 2.5 ś Superpoint Graph. Superpoint Graph[START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF] partitions an input point cloud 2.5a into geometrically simple superpoints 2.5b. A superpoint graph 2.5c is then constructed by linking nearby superpoints. A network is trained to transform superpoints into compact embeddings then processed with graph convolutions, and őnally classiőed into semantic labels 2.5d. Source:[START_REF] Landrieu | ń Large-scale point cloud semantic segmentation with superpoint graphs ż[END_REF]

Segmentation 3 Figure 3 . 1 ś

 331 Figure 3.1 ś Model Size vs. Performance. We visualize the performance of different methods on the S3DIS dataset (6-fold validation) in relation to their model size in log-scale. The area of the markers indicates the GPU-time to train on a single fold. Our proposed method Superpoint Transformer (SPT) achieves state-of-the-art with a reduction of up to 200-fold in model size and 70-fold in training time (in GPU-h) compared to recent methods. The even smaller SPT-nano model achieves a fair performance with 26k parameters only.

Figure 3 . 1 .

 31 Our method starts by partitioning a 3D point cloud into a hierarchical superpoint structure that adapts to the local properties of the acquisition at multiple scales simultaneously. To compute this partition efficiently, we propose a new algorithm that is an order of magnitude faster than existing superpoint preprocessing algorithms. Next, we introduce the Superpoint Transformer (SPT) architecture, which uses a sparse self-attention scheme to learn relationships between superpoints at multiple scales. By viewing the semantic segmentation of large point clouds as the classification of a small number of superpoints, our model can accurately classify millions of 3D points simultaneously without relying on sliding windows. SPT achieves near state-of-the-art accuracy on various open benchmarks while being significantly more compact and able to train much quicker than common approaches. The main contributions of this work are as follows:

Figure 3 . 2 .Figure 3 . 2 ś

 3232 Figure 3.2. We first define the concept of hierarchical partitions. Definition 1 Hierarchical Partitions. A partition of a set X is a collection of subsets of X such that each element of X is in one and only one of such

Figure 3 . 3 ś

 33 Figure 3.3 ś Superpoint Transformer. We represent our proposed architecture with two partitions levels P 1 and P 2 . We use a transformer-based module to leverage the context at different scales, leading to large receptive őelds. We only classify the superpoints of the partition P 1 and not individual 3D points, allowing fast training and inference.

Figure 3 . 4 ś

 34 Figure 3.4 ś Point Geometric Features. Given an input cloud (a), the computed PCA-based geometric features (b, c, d, e) and distance to the ground (f) offer a simple characterization of the local geometry around each point.

  Overfitting and Scaling. The superpoint approach drastically simplifies and compresses the training sets: the 274M 3D points of S3DIS are captured by a geometry-driven multilevel graph structure with fewer than 1.25M nodes.While this simplification favors the compactness and speed of the training of the model, this can lead to overfitting when using SPT configurations with more parameters, as shown in Section 3.3.4. Scaling our model to millions of parameters may only yield better results for training sets that are sufficiently large, diverse, and complex.

Figure 3 . 5 śFigure 3 . 6 śFigure 3 . 7 ś

 353637 Figure 3.5 ś Qualitative Results.We represent input samples (with color or intensity) of our approach and its predictions for all three datasets. Additionally, we show the coarsest partition level and demonstrate how superpoints can accurately capture the contours of complex objects and classify them accordingly. Black points are unlabeled in the ground truth. Color legend given in Section A-1.

Figure 4 . 1 ś

 41 Figure 4.1 ś Large-Scale Panoptic Segmentation. We present the results of Super-Cluster for the entire Area 5 of S3DIS [14] (ceiling removed for visualization) with 9.2M points (78M pre-subsampling) and 1863 true łthingsž objects. Our model can process such large scan in one inference on a single V100-32GB GPU in 3.3 seconds and reach a state-of-the-art PQ of 46.3.

Figure 4 . 2 ś

 42 Figure 4.2 ś SuperCluster.We illustrate the sequence of operations of SuperCluster for a simpliőed scene with two objects: a chair and a sofa. (a) showcases the őrst stage of our process, where the point cloud is partitioned into connected superpoints with simple geometric shapes. In (b), we predict a semantic class distribution for each superpoint. In (c), we predict the object agreement for each pair of adjacent superpoints, indicating the likelihood that they belong to the same object. Finally, (d) showcases the output of a graph clustering problem which merges superpoints with compatible class distribution and object agreement while cutting edges at the transition between objects. The resulting superpoint clusters deőne the instances of a panoptic 3D segmentation

2 , ( 4 . 2 ) where y class p is the first C coordinates of y p and y pos p the last 3 ,

 2423 The dissimilarity function d takes into account both the spatial and semantic nature of x: d(x p , y p ) = H(y class p , x class p ) + η∥x pos py pos p ∥ and η ≥ 0 a parameter. The term H(x, y) denotes the cross-entropy between two distributions: H(x, y) = -C c=1 y c log(x c ).

9 Figure 4 . 3 ś

 943 Figure 4.3 ś Superpoint Object Agreement. We compute for each pair of adjacent superpoint (s, t) an object agreement score âs,t . This value is deőned by the average overlap ratio between s and t and their majority-objects obj(t) and obj(s), see Equation4.8.

Figure 4 . 4 ś

 44 Figure 4.4 ś Qualitative Results. We present the panoptic predictions of our model for the four considered datasets. The scenes' size corresponds to a single batch item during training. łStuffž classes are represented with a lower opacity. Color legend given in Section A-1.

Superpoint Oracle.

  Using superpoints greatly improves the efficiency and scalability of SuperCluster. However, since the predictions are made at the superpoint level and never for individual 3D points, the semantic and object purity of the superpoints can restrict the model's performance. To evaluate this impact, we define the superpoint oracle, which assigns each superpoint s the class and index of its majority object obj(s). The resulting performance provides an upper bound of what our model could potentially achieve. The high performance of this oracle (93.4 PQ) indicates that very little precision is lost by working with superpoints.

Figure 4 . 5 ś

 45 Figure 4.5 ś Large-Scale Inference on S3DIS. Largest scan that SuperCluster can segment in one inference on an A40 GPU: 4.6 areas, 21.3m points, 646k superpoints, 5298 target objects, and 4565 predicted objects. Inference takes 7.4 seconds.

Figure 4 . 6 ś

 46 Figure 4.6 ś Large-Scale Inference on ScanNet. Largest number of scans that SuperCluster can segment in one inference on an A40 GPU: 105 scans , 10.9m points, 398k superpoints , 1683 target objects, and 2148 predicted objects. The inference takes 6.8 seconds.

Figure 4 . 7 ś

 47 Figure 4.7 ś Large-Scale Inference on DALES. Largest scan that SuperCluster can segment in one inference on an A40 GPU: 15.3 tiles, 7.8km 2 , 18.0m points, 589k superpoints, 1727 target objects, and 1559 predicted objects. Inference takes 10.1 seconds.

Figure 4 . 8 ś

 48 Figure 4.8 ś Large-Scale Inference on KITTI-360. Largest scan that SuperCluster can segment in one inference on an A40 GPU: 7.5 tiles, 11.0m points, 414k superpoints, 602 target objects, and 1947 predicted objects. Inference takes 6.6 seconds.

Figure 5 . 1 ś

 51 Figure 5.1 ś Combining 2D and 3D Information. We propose to merge the complementary information between point clouds and a set of co-registered images. Using a simple visibility model, we can project 2D features onto the 3D points and use viewing conditions to select features from the most relevant images. We represent images at their position with the symbol and color the 3D points according to the image they are seen in.

Figure 5 . 2 ś

 52 Figure 5.2 ś Mapping Computation. We estimate pixel depth for all (a) images using the co-registered (b) point cloud. We compute (c) Z-buffers with an efficient GPU-accelerated implementation, resulting in depth maps comparable to the (d) true distance given by camera-aligned depth sensors. We use our estimated depth maps to compute point-image mappings. Better seen on a monitor.

Figure 5 . 3 ś

 53 Figure 5.3 ś Multi-View Information. A 3D point is seen in several images with different insights. Here, the green image contains contextual information, while the pink image captures the local texture. The orange image sees the point at a slanted angle and may contain no additional relevant information.

Figure 5 . 4 ś

 54 Figure 5.4 ś Bimodal 2D/3D Architecture. Using our multi-view aggregation module, we combine a 2D convolutional encoder E 2D and a 3D network composed of an encoder E 3D , a decoder D 3D , and a classiőer C 3D . We associate relevant 2D features to each 3D point according to their viewing conditions in each compatible image. We propose three different 2D/3D fusion strategies: early (our choice in the experiments), intermediate, and late fusion.

Figure 5 . 5 ś

 55 Figure 5.5 ś Dynamic Batching. We can improve the quantity of information contained in each training batch by cropping images around the sampled point clouds. We represent a set of 10 images with different crop size őtting in a budget of pixels corresponding to 4 full-size images.

Figure 5 . 6 ś

 56 Figure 5.6 ś Datasets. Illustration of the sampling procedure for all considered datasets with point clouds alongside some of the available images. The 3D components of batches are constituted of spheres for (a) S3DIS, rooms for (b) ScanNet, and cylinders for (c) KITTI-360.

Figure 5

 5

Figure 5 . 7 ś

 57 Figure 5.7 ś Qualitative Illustration. Scenes from our considered datasets (top: S3DIS, middle: ScanNet, bottom: KITTI-360) with (a) colorized point clouds, (b) ground truth point annotations, (c) prediction of the backbone network operating on the colorized point cloud, and (d) our method operating on raw uncolored point clouds and images. Our approach is able to use images to resolve cases in which the geometry is ambiguous or unusual, such as a large amphitheater with tiered rows of seats (top row). Color legend given in Section A-1.

Figure 5 . 8 ś

 58 Figure 5.8 ś Influence of Viewing Descriptors. Given a point-image pair (top left), we compute the quality scores when varying two of its viewing conditions from their initial values •. For simplicity, we omit the inŕuence of other images. We observe feature blocks specializing in retrieving information from views at a given depth range and containing planar objects (bottom left) or blocking straight yet occluded (top right) or sparse and occluded (bottom right) views.

Figure 5 . 9 ś

 59 Figure 5.9 ś Depth-Based Mapping Computation. Based on an input depth map (a), we compute the point-image mappings (b) by searching points within a small margin of the target depth. We note that slight depth discrepancies near slanted surfaces prevents mapping from being recovered. Better seen on a monitor.

Figure A- 1 ś

 1 Figure A-1 ś Colormaps. Throughout all visualizations in this work, we use these colormaps to represent the semantic of each point.

B- 1 )

 1 with λ ∈ R + a regularization strength and [a ̸= b] the function equals to 0 if a = b and 1 otherwise. Minimizers of J are approximations of f that are piecewise constant with respect to a partition with simple contours in G.We can characterize such signal e ∈ R D×|C| by the coarsest partition P e of P and its associated variable f e ∈ R D×|P e | such that e is constant within each segment p of P e with value f e p . The partition P e also induces a graph Ĝe := (P e , E e , w e ) with E e linking the component of P e adjacent in G and w eB-5 Parameterizing the PartitionWe define G as the k = 10-nearest neighbor adjacency graph and set all edge weights w to 1. The point features f p whose piecewise constant approximation yields the partition are of three types: geometric, radiometric, and spatial.

  tightest bounding box containing all seen point of P sample with a minimum margin of m along a fixed set of image size along: crops = {64 × 64, 128 × 64, 128 × 128, 256 × 128, 256 × 256, 512 × 256, 512 × 512, 1024 × 512}. We linear layer followed by a softmax converts the last features into class scores.

For

  our DeepViewAgg module, the extracted image features are converted into view features of size C with the MLP ϕ 0 of size C → C → C. For the computation of quality scores, we use the following MLPs: ϕ 1 : 8 → M → M , ϕ 2 : M → M → M , and ϕ 3 : 2M → M → K with M = 32 and K = 4.

  All models are trained with SGD with an initial learning rate of 0.1 for 200 epochs with decay steps of 0.3 at epoch 80, 120, 160 for indoor datasets, and 20 epochs with decay at epoch 10, 16 for the outdoor dataset. The pretrained 2D networks use a learning rate 100 times smaller than the rest of the model. We use random rotation and jittering for point clouds, random horizontal flip and color jittering for images, and featurewise jittering for descriptors of the

Figure D- 1 ś

 1 Figure D-1 ś Interactive Visualization. We propose interactive visualizations of hybrid 2D/3D data along with predictions of our model. We also provide the code necessary to create more such visualizations.

Chapitre 3 :

 3 Segmentation Sémantique Efficace à Grande Echelle. Présentation de notre architecture de transformer basée sur les superpoints pour effectuer efficacement la segmentation sémantique de scènes 3D à grande échelle. Cette méthode repose sur un algorithme rapide de partitionnement des nuages de points en une structure hiérarchique de superpoints, ainsi que sur un mécanisme d'auto-attention pour apprendre les relations entre les superpoints à plusieurs échelles. Nous démontrons des performances de pointe sur trois référentiels de segmentation sémantique 3D, avec jusqu'à 200× moins de paramètres et jusqu'à 70× d'entraînement plus rapide par rapport aux approches concurrentes.

Chapitre 4 :

 4 Segmentation Panoptique Efficace à Grande Echelle. Formulation de la tâche de segmentation panoptique 3D comme un problème évolutif de partitionnement de graphe, que peut traiter un petit modèle en se basant uniquement sur des objectifs locaux. Notre cadre contourne plusieurs limitations des méthodes concurrentes et peut être étendu naturellement au paradigme des superpoints présenté dans le Chapitre 3, permettant une mise à l'échelle efficace vers des scènes 3D vastes. Nous atteignons des performances de pointe sur quatre référentiels de segmentation panoptique 3D, démontrant l'efficacité

Figure E- 2 śFigure E- 4 ś

 24 Figure E-2 ś Méthodes d'apprentissage profond 3D. Une variété de méthodes ont été proposées pour extraire des descripteurs à partir de nuages de points E-2a. Certaines approches éliminent des informations en rendant des vues 2D des points E-2b, ou en discrétisant le nuage dans une grille de voxels E-2c. D'autres méthodes opèrent directement sur des ensembles non ordonnés de points E-2d, ou généralisent les convolutions discrètes 2D à l'espace continu 3D E-2e. D'autres alternatives utilisent des réseaux neuronaux graphiques E-2f ou des transformers E-2g pour raisonner sur un voisinage local. Étroitement liées à notre travail, les approches basées sur les superpoints E-2h raisonnent sur une partition de la scène.

Figure E- 5 ś

 5 Figure E-5 ś Fusion Multimodale et Multi-vues. Nous proposons de fusionner les informations complémentaires entre les nuages de points et un ensemble d'images localisées. En utilisant un modèle de visibilité simple, nous pouvons projeter les descripteurs 2D sur les points 3D et utiliser les conditions de visualisation pour sélectionner les descripteurs des images les plus pertinentes. Nous représentons les images à leur position avec le symbole et colorons les points 3D en fonction de l'image dans laquelle ils sont vus.

Chapitre 3 :

 3 Segmentation Sémantique Efficace à Grande Echelle. Une approche efficiente pour la segmentation sémantique 3D, combinant superpoints et transformers, surpassant les méthodes concurrentes en performances tout en étant beaucoup plus compacte et rapide à entraîner.

Chapitre 4 :

 4 Segmentation Panoptique Efficace à Grande Echelle. Une formulation novatrice de la segmentation panoptique 3D comme un problème de partitionnement de graphe, avec un modèle léger (SuperCluster) atteignant des performances de pointe et capable de traiter de grandes scènes sur un seul GPU.

Chapitre 5 :

 5 Apprentissage sur Nuages de Points et Images Arbitrairement Localisées. Une méthode d'agrégation multi-vue pour la segmentation sémantique 3D à partir d'images et de nuages de points, dépassant les méthodes existantes sans nécessiter de colorisation ou de maillage des nuages de points.

Other methods directly operate on unordered sets of points 2.1d, or generalize discrete 2D convolutions to continuous 3D space 2.1e. Alternatives use graph neural networks 2.1f or transformers 2.1g to reason on a local neighborhood. Closely related to our work, superpoint-based approaches 2.1h reason on a partition of the scene.

  1 ś 3D Deep Learning Methods. A variety of methods have been proposed to extract features from point clouds 2.1a. Some approaches discard information by rendering 2D views of the points 2.1b, or discretizing the cloud in a voxel grid 2.1c.

Table 3 .

 3 1 ś Partition Configuration. We report the point count of different datasets before and after subsampling, as well as the size of the partitions. small key space of dimension D key = 4, 16 heads, with 3 blocks in the encoder and 1 in the decoder. We set D val = 64 for S3DIS and DALES

	Dataset	Points Subsampled | P 1 | | P 2 |
	S3DIS [13]	273m	32m	979k	292k
	DALES [311, 288] 492m	449m	14.8m 2.56m
	KITTI-360 [197]	919m	432m	16.2m 2.98m

a (210k parameters), and D val = 128 (777k parameters) for KITTI360. See the Section B-6 and our open repository for the detailed configuration of all modules. We also propose SPT-nano, a lightweight version of our model that does not compute point-level features but operates directly on the first partition level P 1 . The value of the maxpool over points in Equation 3.1 for i = 1 is replaced by f 1 , the aggregated handcrafted point features at the level 1 of the partition. This model never considers the full point cloud P 0 but only operates on the partitions. For this model, we set D val = 16 for S3DIS and DALES (26k parameters), and D val = 32 for KITTI360 (70k parameters). Batch Construction. Batches are sampled from large tiles: entire building floors for S3DIS, and full scenes for KITTI-360 or DALES. Each batch is composed of 4 randomly sampled portions of the partition with a radius of tilting, point jitter and handcrafted features dropout. When sampling points within each superpoint, we set n min = 32 and n max = 128.

Table 3 .

 3 2 ś Performance Evaluation. We report the Mean Intersection-over-Union of different methods on three different datasets. SPT performs on par or better than recent methods with signiőcantly fewer parameters. † superpoint-based. ⋆/ * model with 777k/70k parameters.Performance Evaluation. As seen in Table3.2, SPT performs at the state-of-On KITTI-360, SPT outperforms MinkowskiNet despite a size ratio of 49, and surpasses the performance of the even larger multimodal point-image model DeepViewAgg. On DALES, SPT outperforms ConvPoint by more than 12 points with over 21 times fewer parameters. Although SPT is 1.5 points behind KPConv on this dataset, it achieves these results with 67 times fewer parameters. SPT achieves significant performance improvements over all superpoint-based methods on all datasets, ranging from 7 to 14 points. SPT overtakes the SSP and SPNet superpoint methods that learn the partition in a two-stage training setup, leading to preprocessing times of several hours.

	Model	Size ×10 6 6-Fold Area 5 360 val S3DIS KITTI DALES
	PointNet++ [253]	3.0	56.7 -	-	68.3
	† SPG [180]	0.28	62.1 58.0	-	60.6
	ConvPoint [24]	4.7	68.2 -	-	67.4
	† SPG + SSP [177]	0.29	68.4 61.7	-	-
	† SPNet [140]	0.32	68.7 -	-	-
	MinkowskiNet [58, 46] 37.9	69.1 65.4	58.3	-
	RandLANet [133]	1.2	70.0 -	-	-
	KPConv [301]	14.1	70.6 67.1	-	81.1
	Point Trans.[359]	7.8	73.5 70.4	-	-
	RepSurf-U [263]	0.97	74.3 68.9	-	-
	DeepViewAgg [269]	41.2	74.7 67.2	62.1	-
	Strat. Trans. [175, 323] 8.0	74.9 72.0 -	-
	PointNeXt-XL [255]	41.6	74.9 71.1	-	-
	† SPT (ours)	0.21	76.0 68.9	63.5 ⋆ 79.6
	† SPT-nano (ours)	0.026 70.8 64.9	57.2 *	75.2

the-art on two of three datasets despite being a significantly smaller model.

On S3DIS, SPT beats

PointNeXt-XL with 196× fewer parameters. Interestingly, the lightweight SPT-nano model matches KPConv and MinkowskiNet with only 26k parameters. See Figure 3.5 for qualitative illustrations.

Preprocessing Speed. As reported in Table

3

.3, our implementation of the preprocessing is highly efficient. We can compute partitions, superpoint-graphs, and handcrafted features, and perform I/O operations quickly: 12.4 min for S3DIS, 117 for KITTI-360, and 148 for DALES using a server with a 48-core CPU. An 8-core workstation can preprocess S3DIS in 26.6 min. Our preprocessing time is as fast or faster than point-level methods and 7× faster than SuperPoint Graph's, thus alleviating one of the main drawbacks of superpoint-based methods.

Table 3 .

 3 3 ś Efficiency Analysis. We report the preprocessing time for the entire S3DIS dataset and the training and inference time for Area 5. SPT and SPT-nano shows signiőcant speedups in preprocessing, training, and inference times.

		Preprocessing Training Inference
		in min	in GPU-h	in s
	PointNet++ [253]	8.0	6.3	42
	KPConv [301]	23.1	14.1	162
	MinkowskiNet [58]	20.7	28.8	83
	Stratified Trans. [175]	8.0	216.4	30
	Superpoint Graph [180]	89.9	1.3	16
	SPT (ours)	12.4	3.0	2
	SPT-nano (ours)	12.4	1.9	1

in Table

3

.3, which is significantly lower than competing approaches, with a speed-up factor ranging from 8 to 80. All speed measurements were conducted on a single-GPU server (48 cores, 512Go RAM, A40 GPU). Nevertheless, our model can be trained on a standard workstation (8 cores, 64Go, 2080Ti) with smaller batches, taking only 1.5 times longer and with comparable results.

Table 3 .

 3 4 ś Ablation Study. Impact of some of our design choices on the mIoU for all tested datasets.

Table 3 .

 3 5, SPT-128 with D val = 128 (777k param.) performs 1.4 points below D val = 64 on S3DIS.We report a similar behavior for other hyperparameters: in Table3.6, D key = 8 instead of 4 incurs a drop of 1.0, while in Table3.7, N heads = 32 instead of 16 a drop of 0.1 point. For the larger KITTI-360 dataset (13M

nodes), D val = 128 performs 1.9 points above D val = 64, but 5.4 points above D val = 256 (2.7m param.).

Table 3 .

 3 5 ś Impact of Model Scaling. Impact of model size for each dataset. Table 3.6 ś Impact of Query-Key Dimension. Impact of D key on S3DIS 6-Fold.

	Model	Size S3DIS KITTI DALES
	×10 6 6-Fold 360 Val
	SPT-32	0.14	74.5	60.6	78.7
	SPT-64	0.21	76.0	61.6	79.6
	SPT-128 0.77	74.6	63.5	78.8
	SPT-256 1.80	74.0	58.1	77.6
	D key	2	4	8	16
	SPT-64 75.6 76.0 75.0 74.7

Table 3 .

 3 7 ś Impact of Heads Count. Impact of the number of heads N head on the S3DIS 6-Fold performance.

	N head	4	8	16	32
	SPT-64 74.3 75.2 76.0 75.9

Table 3 .

 3 8 ś Ablation on Supervision. Impact of our hierarchical supervision for each dataset.

	Loss	S3DIS KITTI DALES
		6-Fold 360 Val
	freq-P i -P 1 dist-P i -P 2 76.0	63.5	79.6
	freq-P 1	-0.2	-0.8	-0.8
	dist-P i -P 1	-0.8	-1.3	-0.8

  Our model SPT-nano without point encoder relies purely on such features and reaches 70.8 mIoU on S3DIS 6-Fold with only 27k param, illustrating this expressivity.

	Learning Through the Partition. The idea of learning point and adjacency
	features directly end-to-end is a promising research direction to improve our
	model. However, this implies efficiently backpropagating through superpoint
	hard assignments, which remains an open problem. Furthermore, such a
	method would consider individual 3D points during training, which would

necessitate to perform the partitioning step multiple times during training, which may negate the efficiency of our method

Table 4 .

 4 1 ś Graph Clustering Parameters. We provide the graph clustering parameters used for each dataset.

Table 4 .

 4 2 ś S3DIS Area 5. We report the semantic (SS ) and panoptic segmentation results of the top-performing semantic segmentation methods on the őfth area of S3DIS, as well as panoptic segmentation approaches implemented by Xiang et al .[START_REF] Xiang | ń A review of panoptic segmentation for mobile mapping point clouds ż[END_REF]. We provide two panoptic metrics by considering all classes as łthingsž (PS -no "stuff ") and with wall, ceiling and floor as łstuffž (PS ).

	size	SS	PS -no łstuffž		PS
	×10 6 mIoU PQ	RQ	SQ	PQ	RQ	SQ

Table 4 .

 4 3 ś S3DIS 6-Fold. We report the 6-Fold cross-validated semantic and panoptic segmentation results on S3DIS. No panoptic methods were evaluated in this setting to the best of our knowledge.

		size	SS	PS -no łstuffž		PS
		×10 6 mIoU PQ	RQ	SQ	PQ	RQ	SQ
	Semantic segmentation models
	DeepViewAgg [269]	41.2 74.7	-	-	-	-	-	-
	Strat. Trans. [175, 323] 8.0	74.9	-	-	-	-	-	-
	PointNeXt-XL [255]	41.6 74.9	-	-	-	-	-	-
	SPT [267]	0.21 76.0 -	-	-	-	-	-
	Panoptic segmentation models
	SuperCluster (ours) 0.21 75.3	55.9 66.3 83.8 62.7 73.2 84.8

Table 4 .

 4

		size	SS		PS
		×10 6 mIoU PQ	RQ	SQ
	Semantic segmentation models
	KPConv [301]	14.1 69.2	-	-	-
	Point Trans [359]	7.8	70.6	-	-
	Point Trans. v2[334]	11.3 75.4 -	-	-
	OctFormer [322]	44.0 75.7 -	-	-
	Panoptic segmentation models
	SceneGraphFusion [333, 319] 2.9	-	31.5 42.2 72.9
	Panoptic Fusion [231]	†	-	33.5 45.3 73.0
	SuperCluster (ours)	1.0	66.1	58.7 69.1 84.1
	values are not entirely comparable. Still, our model is on par with the fastest
	methods and offers superior scalability.

4 ś ScanNetv2 Val. We report the Semantic Segmentation (SS ) and Panoptic Segmentation (PS ) performance for various methods on the open test set of ScanNetv2. † code and models unavailable.

Table 4 .

 4 5 ś KITTI-360 We report the Semantic Segmentation (SS ) and Panoptic Segmentation (PS ) performance for various methods on the open test set of KITTI-360. No panoptic methods were evaluated on this dataset to the best of our knowledge.

		size	SS		PS
		×10 6 mIoU PQ	RQ	SQ
	Semantic segmentation models
	Minkowski [58]	37.9 58.3	-	-	-
	DeepViewAgg [359]	41.2 62.1	-	-	-
	SPT [267]	0.78 63.5	-	-	-
	Panoptic segmentation models
	SuperCluster (ours) 0.79 62.1	48.3 58.4 75.1

Table 4 .

 4 6 ś DALES We report the Semantic Segmentation (SS ) and Panoptic Segmentation (PS ) performance for various methods on the open test set of DALES. No panoptic methods were evaluated on this dataset to the best of our knowledge.

		size	SS		PS
		×10 6 mIoU PQ	RQ	SQ
	Semantic segmentation models
	ConvPoint [24]	4.7	67.4	-	-	-
	PointNet++ [253]	3.0	68.3	-	-	-
	SPT [267]	0.21 79.6	-	-	-
	KPConv [301]	14.1 81.1 -	-	-
	Panoptic segmentation models
	SuperCluster (ours) 0.21 77.3	61.2 68.6 87.1

Table 4 .

 4 7 ś Runtime. We compare the speed of our model to various instance and panoptic segmentation models. We report the time spent in the backbone network (őrst number) and performing panoptic segmentation (second number) on ScanNet Val. scans.

	⋆ optional CRF post-processing.
		hardware runtime in ms
	Instance segmentation methods	average per scan
	PointGroup [152]	Titan X	452 = 128 + 324
	SoftGroup [317]	Titan X	345 = 152 + 148
	HAIS [49]	Titan X	339 = 154 + 185
	Mask3D [280]	Titan X	339
	ISBNet [234]	Titan X	237 = 152 + 85
	SuperCluster (ours) 1080Ti	238 = 193 + 45
	Panoptic segmentation methods	scan scene0645_01
	PanopticFusion [231]	2×1080Ti 485 = 317 + 168 (+ 4500 ⋆ )
	SuperCluster (ours) 1080Ti	482 = 376 + 106

Table 4 .

 4 8 ś Ablation Study. We report the performance of different experiments on S3DIS Area 5 with wall, ceiling and floor as łstuffž.

	Experiment	PS
		PQ RQ SQ
	Best Model	58.4 68.4 77.8
	Constant Edge Weights 54.2 64.2 76.6
	Offset Prediction	57.1 65.2 77.1
	Smaller Superpoints	56.6 64.6 78.6
	Superpoint Oracle	93.4 99.7 93.7
	Clustering Oracle	83.6 91.7 90.8
	4.4.4 Large-Scale Inference
	Our method can process large 3D point clouds with just one inference. In
	this section, we represent the largest portion of each dataset that SuperCluster

can handle in one inference with an A40 GPU (48G of VRAM). Results for each dataset are presented in Figure 4.5, Figure 4.6, Figure 4.7, and Figure 4.8.

Table 5 .

 5 1 ś Quantitative Evaluation. Mean Intersection-over-Union of different stateof-the-art methods on S3DIS's Fold 5 and 6-fold, ScanNet Val, and KITTI-360 Test. All methods except the last line are trained on colorized point clouds. State-of-the-art, second highest. 1 with 3D supervision only.

	Model	S3DIS Fold 5 6-Fold	ScanNet Val	KITTI 360 Test
	Methods operating on colorized point clouds
	PointNet++ [253]	-	56.7 [46] 67.6 [47]	35.7 [197]
	SPG+SSP [180, 177] 61.7	68.4	-	-
	MinkowskiNet [58]	65.4	65.9[46] 72.4 [233] -
	KPConv [301]	67.1	70.6	69.3 [233] -
	RandLANet [133]	-	70.0	-	-
	PointTrans.[79]	70.4	73.5	-	-
	Our 3D Backbone	64.7	69.5	69.0	53.9
	Methods operating on point clouds and images
	MVPNet [149]	62.4	-	68.3	-
	VMVF [171]	65.4	-	76.4	-
	BPNet [135]	-	-	69.7 1	-
	3D Backbone+ DeepViewAgg (ours)	67.2	74.7	71.0	58.3

Table 5 .

 5 2 ś Ablation Study. Mean IoU comparison of different modalities and design choices on Fold 2 and Fold 5 of S3DIS down-sampled at 5cm for processing and KITTI-360 Val. No Gating, see Equation 5.7) decreases the IoU by 3 points for Fold 2, and 1.1 on KITTI-360. Not using dynamic batches forces us to limit ourselves to 4 full-size images per 3D sphere/cylinder, which results in performance drops of 2 to 7 points. Pretraining the 2D network on related open-access datasets

	Model	S3DIS Fold 2 Fold 5 360 Val KITTI
	Best Configuration	63.2	67.5	57.8
	Modality Combinations
	XYZRGB	-15.9	-6.0	-3.6
	XYZ Average-RGB	-10.8	-7.0	-4.9
	XYZ	-19.5	-9.5	-4.1
	Pure RGB	-5.3	-5.4	-14.5
	Lower Image Resolution	-5.9	-0.8	-0.7
	Higher 3D Resolution	-1.0	-0.3	-
	Design Choices
	Late Fusion	-9.1	-1.0	-1.3
	Only One Group	-4.8	-0.8	-0.4
	No Gating	-3.0	-0.4	-1.1
	No Dynamic Batch	-6.9	-1.9	-4.5
	No Pretraining	-7.2	-6.7	-3.7
	MaxPool	-0.8	-1.5	-2.9
	Smaller 3D backbone	-0.5	-0.7	+0.7
	Design Choices. Using late fusion (Late Fusion) instead of early fusion gives
	comparable results on Fold 5 and KITTI-360, but significantly worse for
	Fold 2 for which the gain of using images is more pronounced. Using only one
	feature group (Only One Group, K = 1 in Equation 5.3) results in a drop of
	4.8 points for Fold 2, highlighting that our method can learn to treat different
	types of radiometric information specifically. Removing the gating mechanism
	(

Table 5 .

 5 3 ś Usage and Sensitivity of Viewing Conditions. Feature usage is reported as a drop in mIoU, and the sensitivity is given as the proportion of squared partial derivative of the compatibility across all features.

	view feature	usage (mIoU drop) S3DIS KITTI-360 S3DIS KITTI-360 sensitivity (in %)
	depth	1.1	1.7	12.6	46.0
	linearity	1.0	0.8	11.9	0.7
	planarity	1.0	1.4	15.8	1.9
	scattering	0.7	1.0	52.7	0.7
	viewing angle	1.3	1.2	2.8	7.4
	pixel row	1.1	0.8	1.6	33.2
	local density	1.2	1.3	0.6	1.8
	occlusion	0.7	0.9	2.0	8.2

Table 5 .

 5 4 ś Supplementary Ablation Study. Mean IoU comparison of different design choices on Fold 2 and Fold 5 of S3DIS down-sampled at 5cm for processing.

	Model	Fold 2 Fold 5
	Best Conőguration	63.1	67.5
	Design Choices
	Mapping from Depth	-5.4	-1.9
	Intermediate	-7.6	-3.2
	XYZRGB + DeepViewAgg -0.5	-0.9

Table 5 . 5 ś

 55 Effect of Maximum Depth. We report the drop in mIoU when removing mappings beyond a threshold distance.

		S3DIS FOLD 5
	Max depth 8	7	6	5	4	3	2
	mIoU drop 0.0 0.0 0.0 0.4 0.6 1.9 8.6
		KITTI-360 Val
	Max depth 20 15 10
	mIoU drop 0.0 0.5 2.6

Table 5 .

 5 6 ś Impact of the Number of Images. We report the drop in mIoU when limiting the number of images per point cloud.

	# images	8	7	6	5	4	3
	S3DIS Fold 5 0.5 1.2 1.7 2.1 3.5 6.8
	KITTI-360	0.5 0.1 0.5 1.3 1.9 3.5

Table B -

 B 1 ś Ablation on Handcrafted Features. Impact of handcrafted features on the mIoU for all tested datasets.

	Experiment	S3DIS KITTI DALES
		6-Fold 360 Val
	Best Model	76.0	63.5	79.6
	a) Point Features
	No radiometric feat.	-2.7	-4.0	-1.2
	No geometric feat.	-0.7	-4.1	-1.4
	b) Adjacency Features
	No interface feat.	-0.2	-0.6	-0.7
	No ratio feat.	-1.1	-2.2	-0.4
	No pose feat.	-5.5	-1.2	-0.8
	c) Room Features
	Room-level samples	-3.8	-	-
	Normalized Room pos.	-0.7	-	-
	B-3 Influence of Handcrafted Features
	In Table B-1, we quantify the impact of the handcrafted features detailed
	in Section 3.2.4 on performance. To this end, we retrain SPT without each
	feature group and evaluate the prediction on S3DIS Area 5.
	a) Point Features. Our experiments show that removing radiometric features
	has a strong impact on performance, with a drop of 2.7 to 4.0 mIoU. In
	contrast, removing geometric features results in a performance drop of 0.7 on
	S3DIS, but 4.1 on KITTI-360.

Table B -

 B 3 ś S3DIS Class-wise Performance. Class-wise mIoU on S3DIS for our Superpoint Transformer.

	SPT-nano	SPT	DeepViewAgg [269]	KPConv [301]	SPG + SSP [177]	MinkowskiNet [58, 269]	ConvPoint [24]	SPG [180]	PointNet [252]		SPT-nano	SPT	Stratified T. [175]	DeepViewAgg [269]	PointTrans.[359]	KPConv [301]	SPG + SSP [177]	MinkowskiNet [58]	SPG [180]	PointNet [252]	Method
	70.8 93.1 96.0 80.9 68.4 54.0 62.2 71.3 76.3 70.8 63.3 74.3 51.9 57.6	76.0 93.9 96.3 84.3 71.4 61.3 70.1 78.2 84.6 74.1 67.8 77.1 63.6 65.0	74.7 90.0 96.1 85.1 66.9 56.3 71.9 78.9 79.7 73.9 69.4 61.1 75.0 65.9	70.6 93.6 92.4 83.1 63.9 54.3 66.1 76.6 57.8 64.0 69.3 74.9 61.3 60.3	68.4 91.7 95.5 80.8 62.2 54.9 58.8 68.4 78.4 69.2 64.3 52.0 54.2 59.2	69.5 91.2 90.6 83.0 59.8 52.3 63.2 75.7 63.2 64.0 69.0 72.1 60.1 59.2	68.2 95.0 97.3 81.7 47.1 34.6 63.2 73.2 75.3 71.8 64.9 59.2 57.6 65.0	62.1 89.9 95.1 76.4 62.8 47.1 55.3 68.4 73.5 69.2 63.2 45.9 8.7 52.9	47.6 88.0 88.7 69.3 42.4 23.1 47.5 51.6 42.0 54.1 38.2 9.6 29.4 35.2	S3DIS 6-FOLD	64.9 92.4 97.1 81.6 0.0 38.2 56.4 58.6 86.3 77.3 69.6 82.5 50.5 53.4	68.9 92.6 97.7 83.5 0.2 42.0 60.6 67.1 88.8 81.0 73.2 86.0 63.1 60.0	72.0 96.2 98.7 85.6 0.0 46.1 60.0 76.8 92.6 84.5 77.8 75.2 78.1 64.0	67.2 87.2 97.3 84.3 0.0 23.4 67.6 72.6 87.8 81.0 76.4 54.9 82.4 58.7	70.4 94.0 98.5 86.3 0.0 38.0 63.4 74.3 89.1 82.4 74.3 80.2 76.0 59.3	67.1 92.8 97.3 82.4 0.0 23.9 58.0 69.0 91.0 81.5 75.3 75.4 66.7 58.9	61.7 91.9 96.7 80.8 0.0 28.8 60.3 57.2 85.5 76.4 70.5 49.1 51.6 53.3	65.4 91.8 98.7 86.2 0.0 34.1 48.9 62.4 81.6 89.8 47.2 74.9 74.4 58.6	58.4 89.4 96.9 78.1 0.0 42.8 48.9 61.6 84.7 75.4 69.8 52.6 2.1 52.2	41.1 88.8 97.3 69.8 0.1 3.9 46.3 10.8 52.6 58.9 40.3 5.9 26.4 33.2	mIoU ceiling floor wall beam column window door chair table bookcase sofa board clutter	S3DIS Area 5

Table B -

 B 4 ś KITTI-360 Class-wise Performance. Class-wise mIoU on KITII-360 for our Superpoint Transformer. Table B-5 ś DALES Class-wise Performance. Class-wise mIoU on DALES for our Superpoint Transformer. use SPT-64 for S3DIS and DALES and SPT-128 for KITTI360 and ScanNet.

	SPT-nano 75.2 96.5 92.6 78.1 35.8 92.1 50.8 59.9 96.0	SPT 79.6 96.7 93.1 86.1 52.4 94.0 52.7 65.3 96.7	KITTI-360 Val KPConv [301] traffic sig. Method mIoU road sidewalk building wall fence pole traffic lig. vegetation terrain person car truck motorcycle bicycle ConvPoint [24] SPG [180] PointCNN [193] 67.4 96.9 91.9 60.6 94.7 87.9 58.4 97.5 91.7 81.1 97.1 94.1 75.5 21.7 86.7 29.6 40.3 62.9 18.7 65.2 33.6 28.5 40.6 40.8 26.7 52.6 57.6 85.3 41.9 95.5 63.5 75.0 96.3 93.4 95.7 96.6	MinkowskiNet [58, 269] 54.2 90.6 74.4 84.5 45.3 42.9 52.7 0.5 38.6 87.6 70.3 26.9 87.3 66.0 28.2 17.2 PointNet++ [253] 68.3 94.1 91.2 75.4 30.3 79.9 46.2 40.0 89.1	DeepViewAgg [269] 57.8 93.5 77.5 89.3 53.5 47.1 55.6 18.0 44.5 91.8 71.8 40.2 87.8 30.8 39.6 26.1 Method mIoU ground vegetation car truck power line fence pole building	SPT 63.5 93.3 79.3 90.8 56.2 45.7 52.8 20.4 51.4 89.8 73.6 61.6 95.1 79.0 53.1 10.9 DALES	SPT-nano 57.2 91.7 74.7 87.8 49.3 38.8 49.0 12.2 39.2 88.0 69.5 39.9 94.2 80.1 33.7 10.4

  Table C-1, Table C-2, Table C-3, Table C-4, and Table C-5 the average and per-class performances of SuperCluster on each dataset.

Table C -

 C 1 ś S3DIS Class-wise Performance. We report the average and per-class panoptic quality (PQ), recognition quality (RQ), segmentation quality (SQ), precision (Prec), and recall (Rec) performance of SuperCluster on S3DIS. We indicate łstuffž classes with †.

	PQ 58.4 93.8 96.2 84.0 0.0 48.5 64.7 45.3 64.3 40.1 47.7 62.9	Metric Avg. ceiling † floor † wall † beam column window door chair table bookcase sofa board clutter	S3DIS Area 5

Table C -

 C 2 ś S3DIS Class-wise Performance Without "Stuff ". We report the average and per-class panoptic quality (PQ), recognition quality (RQ), segmentation quality (SQ), precision (Prec), and recall (Rec) performance of SuperCluster on S3DIS without łstuffž classes. Table C-3 ś ScanNetv2 Val. Class-wise Performance. We report the average and per-class panoptic quality (PQ), recognition quality (RQ), segmentation quality (SQ), precision (Prec), and recall (Rec) performance of SuperCluster on ScanNet. We indicate łstuffž classes with †. Avg. wall † ŕoor † cabinet bed chair sofa table door window bookshelf picture counter desk curtain refrigerator shower toilet sink bathtub otherfurniture PQ

	S3DIS Area 5 -no "stuff"	Metric Avg. ceiling floor wall beam column window door chair table bookcase sofa board clutter	50.1 46.9 69.5 39.0 0.0 45.7 68.1 47.9 64.2 41.1 48.6 66.2 74.8 39.1 Prec. Rec. 76.7 93.1 100.0 69.7 81.1 79.0 80.0 68.0 65.9 63.4 75.7 64.9 68.4 60.1 58.0 94.3 82.8 98.3 86.0 64.3 84.6 98.7 55.7 75.0 63.9 78.4 69.9 44.2 48.0 68.8 23.7 51.0 65.4 43.3 57.9 85.7 100.0 51.6 PQ 76.7 74.2	60.1 52.0 78.4 49.3 0.0 58.6 80.8 60.2 75.9 53.5 57.6 81.8 85.7 47.8 SQ 84.1 82.6 92.4 81.6 90.2 86.8 87.2 85.5 80.3 81.5 91.4 79.7 73.5 79.1 82.6 89.4 85.5 89.3 79.1 RQ 81.8	76.6 90.3 88.6 79.1 0.0 78.0 84.2 79.6 84.5 76.8 84.3 80.9 87.3 81.8 RQ 69.1 88.7 99.3 61.9 77.9 70.7 79.2 68.9 52.9 54.6 72.1 34.7 58.4 62.6 49.6 71.7 84.2 99.2 64.5 SQ 75.4	63.6 45.5 70.6 43.7 0.0 62.1 90.5 68.7 76.7 58.5 74.4 81.8 94.3 59.9 Metric 58.7 73.3 58.4 60.5 88.2 56.6 0.0 55.4 73.1 53.5 75.2 49.3 47.0 81.8 78.6 39.8 91.8 50.5 70.3 61.3 69.0 58.9 42.5 44.5 65.9 27.7 42.9 49.6 40.9 64.1 72.0 88.6 51.0 Prec. Rec. 61.7	S3DIS 6-FOLD -no "stuff"	55.9 68.6 64.1 40.0 65.6 64.0 70.1 42.7 48.0 48.3 43.7 55.4 69.4 46.7 PQ	66.3 74.8 72.6 50.8 74.3 76.7 81.8 57.1 57.3 62.8 52.5 64.6 80.5 56.0 RQ	83.8 91.8 88.2 78.6 88.3 83.4 85.8 74.7 83.7 76.9 83.2 85.7 86.2 83.4 SQ	72.8 76.4 69.8 50.2 77.9 78.3 86.7 68.8 70.7 63.9 67.0 72.7 90.8 73.1 Prec.	61.7 73.3 75.7 51.4 71.1 75.2 77.4 48.8 48.2 61.8 43.1 58.2 72.3 45.4 Rec.
			46.8	69.5	83.4	56.0	46.7

Table C -

 C 4 ś KITTI-360 Val. Class-wise Performance. We report the average and per-class panoptic quality (PQ), recognition quality (RQ), segmentation quality (SQ), precision (Prec), and recall (Rec) performance of SuperCluster on the KITTI-360 Validation set. We indicate łstuffž classes with †.

	5.4	7.1	75.5	10.0	5.6
	55.5	66.7	83.3	73.3	61.1
	81.5	83.3	97.8	90.9	76.9
	84.2	89.1	94.4	92.4	86.1
	19.4	20.7	93.6	23.1	18.8
	49.5	68.5	72.3	76.0	62.3
	89.5	100.0	89.5	100.0	100.0
	30.5	40.8	74.6	43.5	38.5
	0.0	0.0	0.0	0.0	0.0
	43.4	65.6	66.2	65.6	65.6
	20.0	33.3	60.0	33.3	33.3
	31.5	51.4	61.2	51.9	50.9
	44.8	53.4	83.9	48.4	59.5
	75.6	95.3	79.3	96.2	94.4
	48.3 94.3	58.4 100.0	75.1 94.3	60.3 100.0	56.9 100.0
	PQ	RQ	SQ	Prec.	Rec.

Metric Avg. road † sidewalk † building wall † fence † pole † traffic lig. † traffic sig. † vegetation † terrain † person † car truck † motorcycle † bicycle †

Table C -

 C 5 ś DALES Class-wise Performance. We report the average and per-class panoptic quality (PQ), recognition quality (RQ), segmentation quality (SQ), precision (Prec), and recall (Rec) performance of SuperCluster on DALES. We indicate łstuffž classes with †. Dynamic-Size Image-Batching Input: P sample point cloud, I image set Parameters: B budget of pixels, m border margin I

	with f 3D 0 the raw 3D features. The decoder and classifiers follow the same
	organization than the 3D backbone.
	D-3 Dynamic-Size Image-Batching
	Algorithm D-3
	Metric Avg. ground † vegetation † car truck power line fence pole building
	PQ	61.2	95.6	90.3	70.9 45.0	18.8	23.5 64.3	81.5
	RQ	68.6	100.0	99.0	78.4 51.1	23.1	31.3 79.6	86.6
	SQ	87.1	95.6	91.2	90.4 88.2	81.3	75.0 80.8	94.1
	Prec.	68.5	100.0	99.0	87.3 55.1	16.3	24.2 81.5	84.7
	Rec.	71.0	100.0	99.0	71.1 47.5	39.7	44.3 77.8	88.5

sample ← {i | i ∈ I and ∃p ∈ P sample s.t i ∈ v(p)} scores ← array of 0 of size I sample for i ∈ I sample do i ← tightest crop(i) to contain P sample with margin m scores[i] ← score(i, P sample , I sample , ∅) end for batch ← [] while B > 0 and length I sample > 0 do pick i ∈ I sample randomly w.r.t. scores batch ← [batch, i] I sample ← I sample \ {i} B ← Barea(i) for i ∈ I sample do scores[i] ← score(i, P sample , I sample , batch) end for end while

Table D -

 D 2 ś ScanNet Val. Class-wise Performance. Class-wise mIoU on ScanNet Val. for our 3D backbone network with and without learned multi-view aggregation. ScanNet Method Avg wall ŕoor cabinet bed chair sofa table door window bookshelf picture counter desk curtain refrigerator shower cur. toilet sink bathtub other à entraîner des modèles 3D massifs avec d'énormes jeux de données, nous préconisons des approches sobres et efficientes, adaptées aux chercheurs et praticiens disposant de ressources limitées. Les méthodes proposées sont conçues pour traiter des scènes 3D étendues sans compromettre les performances, avec une évaluation sur des jeux de données publics et un code open source pour la reproductibilité. En résumé, cette thèse vise des solutions accessibles, efficaces et respectueuses de l'environnement pour l'analyse de nuages de points 3D. Dans ce chapitre introductif, nous situons d'abord cette thèse dans le contexte de l'histoire récente de l'apprentissage profond en 3D. Ensuite, nous exposons nos motivations pour travailler sur l'analyse efficace de grands nuages de points 3D, ainsi que les défis correspondants à nos objectifs. Nous présentons ensuite les principales contributions de cette thèse. Enfin, nous résumons la structure du présent document. Au cours de la dernière décennie, la recherche en apprentissage profond a connu des avancées notables, passant par les réseaux neuronaux convolutifs à l'adoption généralisée des architectures de type transformer et à la révolution de l'apprentissage auto-supervisé. Le domaine a été marqué par la montée en puissance de modèles toujours plus grands et par l'essor de l'apprentissage autosupervisé, mais ces avancées ont été principalement axées sur le traitement du texte et de l'image. En parallèle, la vision par ordinateur 3D a évolué avec l'explosion des données 3D et l'émergence d'architectures adaptées, bien que les jeux de données annotés restent limités. Cette thèse se positionne dans ce contexte, cherchant à développer des méthodes d'apprentissage profond 3D efficaces malgré des données moins abondantes et des ressources plus modestes. Cette thèse se concentre sur des méthodes efficaces pour analyser de grands nuages de points 3D, notamment dans des domaines tels que la conduite autonome et la gestion urbaine. Nous justifions cette approche en considération des tendances actuelles de l'apprentissage profond 3D, soulignant la nécessité de l'efficacité face à la course à la taille des modèles. Pour guider notre travail, nous proposons une typologie de cinq dimensions d'efficacité : mémoire, calcul, matériel, données et travail humain. En résumé, notre objectif est de développer des techniques d'apprentissage profond 3D efficientes pour traiter efficacement les vastes scènes 3D du monde réel. Les nuages de points offrent une représentation imparfaite des scènes 3D en raison de l'absence de connectivité entre les points, entraînant des défis tels que la perte de détails géométriques fins et des caractéristiques distinctes selon les techniques d'acquisition. Pour surmonter ces défis, cette thèse se concentre sur trois aspects clés : le raisonnement multiniveau efficace, la fusion multimodale efficace et une mise en oeuvre efficiente. Le raisonnement multiniveau efficace repose sur une représentation de données hiérarchique adaptative à la complexité géométrique de la scène. La fusion multimodale efficace propose une méthode pour extraire et fusionner efficacement des descripteurs à partir de nuages de points et d'images, sans dépendre d'une acquisition spécifique ou de prétraitements coûteux. Enfin, une mise en oeuvre efficace est soulignée comme un aspect essentiel, nécessitant des efforts d'ingénierie algorithmique pour surmonter les défis liés à la nature non structurée et variante en taille des nuages de points 3D. Ces contributions visent à relever les défis liés au traitement efficace des vastes scènes 3D du monde réel. Cette thèse propose deux contributions majeures pour l'analyse efficace de scènes 3D à grande échelle. La première consiste en un modèle léger pour l'analyse sémantique des nuages de points, surpassant les méthodes existantes avec moins de paramètres et des temps plus rapides. La seconde est une architecture capable d'extraire des informations à partir de nuages de points 3D et d'images 2D de manière conjointe, avec une implémentation parcimonieuse et parallélisée. En résumé, ces contributions exploitent les structures des nuages de points 3D et des données multimodales pour créer des architectures d'apprentissage profond performantes, efficientes et évolutives.Chapitre 2 : État de l'Art. Introduction aux principales familles de modèles d'apprentissage profond 3D. Revue des stratégies existantes pour l'apprentissage profond efficace sur images et nuages de points. Présentation des méthodes basées sur les superpoints pour le traitement 3D efficace et aperçu des approches d'apprentissage multimodal avec les nuages de points.

	51.0	56.3
	85.3	82.8
	62.6	66.2
	89.9	89.5
	67.4	71.4
	46.2	57.4
	70.4	71.9
	60.8	55.6
	60.3	62.2
	28.4	39.9
	80.0	77.8
	60.2	66.9
	61.4	63.4
	70.3	71.9
	80.7	78.0
	88.9	90.1
	77.9	78.9
	62.2	57.8
	94.5	94.4
	3D Backbone 69.0 82.7	+ DeepViewAgg 71.0 84.3

(a) True depth (b) Depth-based visibility

HD, les images de rue et les images satellite.-Mapping robuste 2D-3D : L'intégration de la temporalité pour prendre en compte les mouvements d'objets et la robustesse aux erreurs de paramètres de la caméra est recommandée.-Apprentissage multimodal Super-X : La combinaison des paradigmes Super-X et multimodal offre des possibilités intéressantes pour l'apprentissage auto-supervisé à grande échelle et l'adaptation naturelle à d'autres modalités.En résumé, ces perspectives suggèrent des directions prometteuses pour de futurs développements en vision par ordinateur 3D et en apprentissage multimodal.

Acknowledgements

As this PhD nears its end, I have a lot of gratitude to express. Acknowledgements. This work was funded by ENGIE Lab CRIGEN. This work was supported by ANR project READY3D ANR-19-CE23-0007, and was granted access to the HPC resources of IDRIS under the allocation AD011013388R1 made by GENCI. We thank Bruno Vallet, Romain Loiseau and Ewelina Rupnik for inspiring discussions and valuable feedback.

Acknowledgements. This work was funded by ENGIE Lab CRIGEN. This work was supported by ANR project READY3D ANR-19-CE23-0007, and was granted access to the HPC resources of IDRIS under the allocation AD011013388R1 made by GENCI. We thank Mathieu Brédif and Bruno Vallet for inspiring discussions and valuable feedback. Acknowledgements This work was funded by ENGIE Lab CRIGEN and carried on in the LASTIG research unit of Université Paris-Est. The authors wish to thank AI4GEO for sharing their computing resources. AI4GEO is a project funded by the French future investment program led by the Secretariat General for Investment and operated by public investment bank Bpifrance.

We thank Philippe Calvez, Dmitriy Slutskiy, Marcos Gomes-Borges, Gisela Lechuga, Romain Loiseau, Vivien Sainte Fare Garnot and Ewelina Rupnik for inspiring discussions and valuable feedback.

Abstract

We introduce a highly efficient method for panoptic segmentation of large 3D point clouds by redefining this task as a scalable graph clustering problem. This approach can be trained using only local auxiliary tasks, thereby eliminating the resource-intensive instance-matching step during training. Moreover, our formulation can easily be adapted to the superpoint paradigm, further increasing its efficiency. This allows our model to process scenes with millions of points and thousands of objects in a single inference.

Our method, called SuperCluster, achieves a new state-of-the-art panoptic segmentation performance for two indoor scanning datasets: 50.1 PQ (+7.8) for S3DIS Area 5, and 58.7 PQ (+25.2) for ScanNetV2. We also set the first state-of-the-art for two large-scale mobile mapping benchmarks: KITTI-360 and DALES. With only 209k parameters, our model is over 30 times smaller than the best-competing method and trains up to 15 times faster. Our code and pretrained models are available at github.com/drprojects/super_cluster. This chapter's work was initially presented in: Damien Robert, Hugo Raguet, Loic Landrieu, "Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering", 3DV, 2024.

Semantic segmentation models SPT [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF] 0.21 not perform as well as large networks designed for semantic segmentation but provides decent results with a small backbone of just 1M parameters.

DALES and KITTI-360. To the best of our knowledge, SuperCluster is the first method capable of processing the large tiles of the DALES and KITTI-360 data sets at once, thus establishing the first panoptic state-of-the-art for these datasets given in Table 4.5 and Table 4.6.

Inference and Training Speed. In Table 4.7, we compare the inference speed of our approach with state-of-the-art instance and panoptic segmentation algorithms. Although we used our smallest GPU (a 1080Ti) to replicate the setting used to measure most of the approaches' speed (a Titan-X), the

Appendices

Appendix A

Additional Notes

In this appendix, we detail the colormaps usedfor each dataset in Section A-1.

A-1 Dataset Colormaps

We use the same colormaps for each dataset across all chapters. These colormaps are detailed in Table A-1.

Appendix B Additional Results on

Efficient and Scalable 3D Semantic In this document, we introduce our interactive visualization tool (Section B-1), share our source code (Section B-2), provide an analysis (Section B-3) of all handcrafted features used by our method, detail the partition process (Section B-4), and provide guidelines on how to choose the partition's hyperparameters (Section B-5). Finally, we clarify our architecture parameters (Section B-6) and detail the class-wise performance of our approach on each dataset (Section B-7).

B-1 Interactive Visualization

We release for this project an interactive plotly visualization tool that produces HTML files compatible with any browser. As shown in Figure B-1, we can visualize samples from S3DIS, KITTI-360, and DALES with different point attributes and from any angle. These visualizations were instrumental in designing and validating our model, and we hope that they will facilitate the reader's understanding as well.

B-2 Source Code

We make our source code publicly available at https://github.com/drprojects/superpoint_transformer.

The code provides all necessary instructions for installing and navigating the project, simple commands to reproduce our main results on all datasets, ready-to-use pretrained models, and ready-to-use notebooks.

Our method is developed in PyTorch and relies on PyTorch Geometric, PyTorch Lightning, and Hydra.

the weight of the cut between adjacent elements of P e :

We denote by partition (e) the function mapping e to these uniquely defined variables: We first define Ĝ0 as the point-level adjacency graph Ĝ and f 0 as f . We can now define the levels of a hierarchical partition P i for i ∈ [1, I]:

Given that the optimization problems defined in Equation B-5 for i > 1 operate on the component graphs Ĝi , which are smaller than Ĝ0 , the first partition is the most demanding in terms of computation.

Note that we used the hat notation Ĝi , because these graphs are only used for computing the hierarchical partitions P i , and should be distinguished from the superpoint graphs G i on which is based our self-attention mechanism, constructed from P i as explained in Section 3.2.5.

B-7 Detailed Results

We report in

C-2 Source Code

We make our source code available to the reviewers at github.com/drprojects/super_cluster.

The code provides all necessary instructions for installing and navigating the project, simple commands to reproduce our main results on all datasets, and ready-to-use notebooks.

Our method is developed in PyTorch and relies on PyTorch Geometric, PyTorch Lightning, and Hydra.

C-3 Implementation Details

We provide the exact parameterization of the SuperCluster architecture used for our experiments. For simplicity, we represent an MLP by the list of its layer widths: [in_channels, hidden_channels, out_channels].

Backbone. Our backbone model follows the same model configurations as the Superpoint Transformer [START_REF] Robert | ń Efficient 3D semantic segmentation with superpoint transformer ż[END_REF] with minor modifications, described below. We Finally, we provide detailed class-wise results for each dataset in Section D-6.

D-1 Interactive Visualization and Code

We release our code at https://github.com/drprojects/DeepViewAgg. The provided code allows for reproduction of our experiments and inference using pretrained models.

Our repository also contains interactive visualizations as HTML files,

showing different images and model predictions for spheres sampled in S3DIS, as shown in Figure D-1. This tool makes it easier to see the additional insights brought by images than the visuals included in the chapter.

D-2 Fusion schemes

We denote by {f 2D i } i∈I a set of 2D feature maps of width C associated with the images I, typically obtained with a convolutional neural network.

f 3D designates the raw feature of the point cloud given by the sensor: position, but also intensity/reflectance if available (not used in this work). We denote by P(f 2D , P ) the projection of the learned image features f 2D onto the point cloud P by our multi-view aggregation technique. The early and late fusion schemes can be written as follows:

For the intermediate fusion scheme, the 2D and 3D features are merged directly in the 3D encoder. Our 3D backbone follows a classic U-Net architecture, and its encoder is organized in L levels {E 3D l } L l=1 processing maps of increasingly coarse resolution. Each level l > 1 is composed of of a downsampling module down l , typically strided convolutions (down 1 = Id), and a convolutional module conv l , typically a sequence of ResNet blocks. The 2D encoder is also composed of L levels {E 2D l } L l=1 corresponding to the different image resolutions. We propose to match the 2D and 3D levels at full resolution (1024 × 512 and 2 cm for S3DIS), and all subsequent levels after the same number of 2D/3D downsampling. At each level l = 1 . . . L, we concatenate the downsampled higher resolution 3D map f 3D l-1 with the map f 2D l obtained from the images at the matched resolution:

then associate with each cropped image a score defined as follows:

with area(i) the area of the cropped image i in pixels, batch the current image batch, unseen(i, P sample , batch) the number of points of P sample seen in image i but not in any image of the current batch, and λ = 2 a parameter controlling the trade-off between maximum area and maximum coverage. The current image batch is initialized as an empty set, but the scores must be updated as it is filled. The images are chosen randomly with a probability proportional to their score. We chose in all experiments a margin m = 8 pixels and a budget corresponding to 4 full resolution mapping.

D-4 Implementation Details

Our method is developed in Pytorch and is implemented within the opensource framework Torch-Points3d [START_REF] Chaton | Torch-Points3D: A modular multi-task framework for reproducible deep learning on 3D point clouds ż[END_REF].

For our backbone 3D network, we use TorchPoint3D's [START_REF] Chaton | Torch-Points3D: A modular multi-task framework for reproducible deep learning on 3D point clouds ż[END_REF] Res16UNet34 implementation of MinkowskiNet [START_REF] Choy | ń 4D spatio-temporal convnets: Minkowski convolutional neural networks ż[END_REF]. This UNet-like architecture comprises 5 encoding layers and 5 decoding layers. Encoding layers are composed of a strided convolution of kernel_size = [3, 2, 2, 2, 2] and stride = [1, 2, 2, 2, 2] followed by N = [0, 2, 3, 4, 6] ResNet blocks [START_REF] He | Identity mappings in deep residual networks ż[END_REF] of channel size [START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Brown | ń Language models are few-shot learners ż[END_REF][START_REF] Dahl | Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition ż[END_REF][START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Quana | Hierarchical semantic segmentation of urban scene point clouds via group proposal and graph attention network ż[END_REF], respectively. The decoding layers are built in the same manner, with a strided transposed convolution of kernel_size = [2, 2, 2,

and stride = [2, 2, 2, 2, 1] as their first operation, followed by a concatenation with the corresponding skipped-features from the encoder and N = 1 ResNet block of channel size [START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Hodgetts | ń Laser scanning and digital outcrop geology in the petroleum industry: A review ż[END_REF][START_REF] Giattino | ń Artiőcial intelligence ż[END_REF][START_REF] Giattino | ń Artiőcial intelligence ż[END_REF][START_REF] Giattino | ń Artiőcial intelligence ż[END_REF], respectively. A fully connected viewing conditions.

For more details on the implementation, we refer the reader to our provided code.

D-5 S3DIS Adjustment

We adjusted some room and image positions in S3DIS [START_REF] Armeni | ń 3D semantic parsing of large-scale indoor spaces ż[END_REF] and https: //github.com/alexsax/2D-3D-Semantics to recover mappings between points and equirectangular images. More specifically, we rotate hallway_11 from Area_2 and hallway_6 from Area_5 by 180°around the Z-axis, and we shift and rotate all images in Area_5b by the same manually-found corrective offset and angle. These fixes are all available in our repository.

D-6 Detailed Results

We report in Table D-1, Table D-2, and Table D-3 the class-wise performance across all datasets. We see a clear improvement for indoor datasets for classes such as windows, boards, and pictures. These are expected results because these classes are hard to parse in 3D but easily identified in 2D.

Besides, we can see that S3DIS's classes such as beams, columns, chairs, and tables also benefit from the contextual information provided by images.

For the KITTI-360 dataset, the multimodal model outperforms the 3D-only baseline for all classes. We can see the benefit of image features on small objects or underrepresented classes in 3D, such as traffic signs, persons, trucks, motorcycles, and bicycles. Nous développons des méthodes pour l'analyse sémantique efficace de grands nuages de points 3D. Ici, nous présentons les résultats de notre travail sur la segmentation panoptique, sur S3DIS [START_REF] Armeni | ń 3D Scene Graph: A structure for uniőed semantics, 3D space, and camera ż[END_REF]. Alors que la plupart des méthodes opèrent sur une pièce à la fois, la nôtre peut en considérer des centaines. En particulier, nous atteignons l'état de l'art tout en traitant des bâtiments de dizaines de millions de points en une seule inférence, sur un seul GPU, en seulement 7.4 secondes.

Cette thèse aborde l'analyse automatisée de scènes 3D, pouvant trouver des applications dans des domaines tels que la surveillance de l'environnement et la planification urbaine. Contrairement à la tendance actuelle consistant de notre méthode à l'entraînement et à l'inférence. -Raisonnement Super-X : L'extension du cadre à d'autres modalités, comme les images, est suggérée, ouvrant la voie à des adaptations pour des modalités multiples.

Apprentissage Multimodal 2D-3D.

-Aggrégation multi-vue multi-capteurs : L'adaptation de DeepViewAgg pour tenir compte de plusieurs capteurs d'image avec des propriétés optiques différentes est suggérée.

-Ajout de modalités : L'extension de DeepViewAgg à d'autres modalités complémentaires est proposée, telles que les capteurs radar, les cartes