Ever-increasing demands for solutions that describe our environment and the resources it contains, require technologies that support efficient and comprehensive description, leading to a better content-understanding. Optical technologies, the combination of these technologies and effective processing are crucial in this context. The focus of this thesis lies on 3D scanning and hyperspectral technologies. Rapid developments in hyperspectral imaging are opening up new possibilities for better understanding the physical aspects of materials and scenes in a wide range of applications due to their high spatial and spectral resolutions, while 3D technologies help to understand scenes in a more detailed way by using geometrical, topological and depth information. The investigations of this thesis aim at the combined use of 3D and hyperspectral data and demonstrates the potential and added value of a combined approach by means of different applications. Special focus is given to the identification and extraction of features in both domains and the use of these features to detect objects of interest.

More specifically, we propose different approaches to combine 3D and hyperspectral data depending on the HSI/3D technologies used and show how each sensor could compensate the weaknesses of the other. Furthermore, a new shape and rule-based method for the analysis of spectral signatures was developed and presented. The strengths and weaknesses compared to existing approaches are discussed and the outperformance compared to SVM methods are demonstrated on the basis of practical findings from the field of cultural heritage and waste management.

Additionally, a newly developed analytical method based on 3D and hyperspectral characteristics is presented. The evaluation of this methodology is based on a practical example from the field of WEEE and focuses on the separation of materials like plastics, PCBs and electronic components on PCBs. The results obtained confirms that an improvement of classification results could be achieved compared to previously proposed methods.

The claim of the individual methods and processes developed in this thesis is general validity and simple transferability to any field of application.

v

Résumé

La demande sans cesse croissante de solutions permettant de décrire notre environnement et les ressources qu'il contient nécessite des technologies qui permettent une description efficace et complète, conduisant à une meilleure compréhension du contenu. Les technologies optiques, la combinaison de ces technologies et un traitement efficace sont cruciaux dans ce contexte. Cette thèse se concentre sur les technologies 3D et les technologies hyper-spectrales (HSI). Tandis que les technologies 3D aident à comprendre les scènes de manière plus détaillée en utilisant des informations géométriques, topologiques et de profondeur, les développements rapides de l'imagerie hyper-spectrale ouvrent de nouvelles possibilités pour mieux comprendre les aspects physiques des matériaux et des scènes dans un large éventail d'applications grâce à leurs hautes résolutions spatiales et spectrales. Les travaux de recherches de cette thèse visent . Representation of an HSI and the comparison with a RGB image. Each wavelength λ is represented as an image with x and y dimensions. The spectral component is presented as a vector for each pixel [START_REF] Lu | Medical hyperspectral imaging: A review[END_REF] Figure 3.2. The effect of using 1 st and 2 nd order derivatives to eliminate additive (green) and additive plus multiplicative (red) spectral reflectance shifts. The blue graph is the spectra without any offsets, the black dotted line is the zero line [START_REF] Rinnan | Review of the most common preprocessing techniques for near-infrared spectra[END_REF]. Comparison of different smoothing grades using Savitzky-Golay filter on continuum removed spectra. An increase in the smoothing grade increases the noise reduction, but also the possibility that the signal will be distorted by the smoothing. ... Selection of significant bands for shape description using extreme values of 2 nd derivative. Shape describing features are the bands location (min/max), the curvature values at these locations (red lines on the top of green areas) and the direction of curvature values (up for convex and down for concave behaviour). Adapted from [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF] Each plastic category has a specific spectral response. This specific shape is also reflected in the shapes of the continuum-removed reflectance spectra. This means that the rule (based on a combination of conditions) defined for a category is specific to that category and there is no risk of confusion because the shapes are distinct enough. Considering all materials which need to be classified in the formation of rules has the advantage that rules can be set up more efficiently and conditions based on significant bands can be kept low. [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF] 1), inula conyza (2), campanula rotundifolia (3), moss (4), geranium robertianum (5), asplenium trichomanes (6), green sedum (7), moss & red sedum (8), moss (9), moss & lichen (10) [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]. 57

List of Figures

Figure 4.19. Spectra of measured plants. Two different cameras were used to acquire the spectra (Specim FX10 and Specim FX17). The discontinuities in the range of 900 nm and the greater noise in the VIS range can be explained by the differences in spectral sensitivity of the cameras. In particular, the range below 500 nm exhibits strong noise effects and was not considered in the rule formation. Taraxacum (1), inula conyza (2), campanula rotundifolia (3), moss (4), geranium robertianum (5), asplenium trichomanes (6), green sedum (7), red sedum (8), moss (9), lichen (10) [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]. .......... 58 As a result, the rule defined for a category is specific to that category and there is no possibility of confusion because the shapes are distinct enough. Such rules can therefore only be used if the shapes of the spectral curves are all sufficiently different (Polat et al., 2021a) Steps of the rule-based classification process using spectral and spatial features for the rule formation. The geometrical information is not only used for structuring but also support the classification of components that cannot be classified from their spectral signature only (Polat et al., 2021a) ecob 7). Red arrows point to false classified components. Adapted from (Polat et al., 2021a). .......... Figure 5.12. Misclassification due to geometric similarities in the point cloud (Polat et al., 2021a) In this context I would also like to thank Sven Kaulfersch and Matthias Brantzen. Both of them, in particular Sven Kaulfersch, were a very big help to me in setting up the hyperspectral lab at i3mainz. I would also like to thank my colleagues Dr. Jean-Jacques Ponciano and Dr.

Claire Prudhomme for their support and valuable input throughout the entire process.

Finally, a very heartfelt thank you to my friends and family. Thank you for your patience and understanding. And thank you for always motivating me.

INTRODUCTION (EN FRANÇAIS)

xxi

CONTEXTE ET MOTIVATION

CHAPTER 1: INTRODUCTION

CONTEXT AND MOTIVATION

Nature is a source of inspiration and has been used as a basis for many technological innovations. Be it the bat that serves as inspiration for parking sensors based on sonar principles or bird bones that inspire aircraft construction, nature serves as a model. Inspiration for this work can also be derived from nature, specifically from humans. Humans use a combination of different senses to perceive their environment and to make decisions based on their knowledge [START_REF] Elmenreich | An Introduction to Sensor Fusion[END_REF]. For instance, assessing the quality of food is based on a combination of senses: sight, smell and taste. A purely visual assessment of the external form can lead to a wrong judgement and only in combination with smell or taste is it possible to really determine whether food is still edible or not. The combination of different senses therefore enables a more reliable statement. This circumstance can also be transferred to sensors. Increasing quality requirements in various areas of industry demand technologies that meet these high-quality standards. Sensor combinations have the potential to increase the performance of systems. The basic idea behind this is the mutual support and the capability of one source to compensate the error of another. The combination of heterogeneous information sources and the resulting diversity of information can help to better understand the underlying data and to optimize processes like feature extraction, classification and segmentation [START_REF] Snidaro | Data Fusion in Modern Surveillance[END_REF].

In this thesis we adopt the idea of sensor fusion and propose an optical sensor concept based on a combination of 3D sensor technology and hyperspectral sensor technology to provide a better understanding of the phenomenon under consideration. Here, one of the objectives was to develop a system that can be used for various study cases. The 3D sensor technology provides topological, depth and spatial information while hyperspectral imaging (HSI) provides physical information. Together, these sources provide a detailed dataset that is rich in information and of great value for a wide range of issues from different fields of applications. Notable among others are the studies of [START_REF] Roscher | Detection of disease symptoms on hyperspectral 3D plant models[END_REF] and [START_REF] Morsy | Multispectral LiDAR Data for Land Cover Classification of Urban Areas[END_REF].

Both works show the benefits of combining geometry and spectral data. To give an example, the work of [START_REF] Roscher | Detection of disease symptoms on hyperspectral 3D plant models[END_REF] used geometry in terms of inclination angles in combination with spectral data for the early detection of disease symptoms on plants, while in [START_REF] Morsy | Multispectral LiDAR Data for Land Cover Classification of Urban Areas[END_REF] a combination of multispectral LiDAR data was used to classify the land cover of an urban area.

General concept of the proposed method is the fusion of datasets and the subsequent extraction of spectral and spatial characteristics. In this context, the extraction of spectral characteristics is carried out using a newly developed approach using shape-based properties of spectral curves such as curvature points, curvature values, curvature direction and spectral values. The advantage of taking shapes into account, particularly when combined with highresolution spectral data, is obvious: spectra are an expression for the mixed reflectivity of a given object's constituent elements (e.g. molecules, pigments, cell structure, water content), which means that each individual element has only a proportional influence on a spectrum.

Modifications in the composition of the elements subsequently change mainly the mixture of all spectral contributions, resulting in local or regional variations and changing the shape of a spectrum. and the flexible extensibility of the rule sets [START_REF] Tan | Introduction to data mining[END_REF].

Local changes in

Special attention is and has to be paid to the generalisability of the classification task and the possibility to cover different types of application. Object characteristics are useful to divide datasets into groups with the same behaviour and to classify them accordingly based on predefined group properties. To ensure generalisation of this process, the features relevant for describing group properties must be carefully selected and go through an iterative and flexible process until the final classification. That is, it is first necessary to consider what kind of objects to recognise and what they have in common. This would lead to a first generic set of features that is applicable to different use cases in common. This process of general feature identification continues until it reaches a point where application-specific elements need to be used. Thus, ensuring a general classification process consists of maximising the number of general valuable features and identifying a maximum of common aspects between all objects of interest.

A classification process helps to structure objects with certain characteristic features and to group them together. A structured data basis has the advantage of obtaining a better understanding of the data basis and thus to simplify the process of problem-solving for differ-ent fields of application. Typically, all objects have a specific shape and a material composition. This ensures a generalisation, as 3D and HSI are in principle sensitive to characteristics in these areas. The fact that the added value of 3D information varies depending on the application must also be considered and is demonstrated by various use cases.

THESIS OUTLINE

This thesis investigates the combined use of 3D and hyperspectral data and demonstrates the potential and added value of a combined approach through various applications.

Particular focus is placed on the identification and extraction of characteristics in both domains and the use of these characteristics for the detection of objects of interest.

Chapter 2 gives an overview of existing 3D and HSI technologies, the characteristic properties of these technologies and the methods required to combine the datasets. The sensor systems used and potential combinations depending on the chosen use cases are shown and illustrated in terms of approach.

Chapter 3 contains a literature review of methods for pre-processing, feature extraction and selection of significant bands in HSI. Furthermore, an overview of existing approaches for the classification of multidimensional data is provided.

Chapter 4 consists of the first main contribution of this thesis. The proposed shape-based approach for analysing spectral signatures in combination with a rule-based classification approach is described and demonstrated based on practical examples.

The combined use of 3D and hyperspectral data is part of chapter 5. This chapter corresponds to the second main contribution of this thesis. The advantages of a combined approach are demonstrated in this section using a practical example from the field of waste sorting. With a selection of different applications, it is shown that the added value of 3D data mainly depends on the selected use cases.

Chapter 6 concludes the findings with regard to the potential of the combined use of 3D

and HSI and provides an outlook on future directions of this research.

CHAPTER 2: TECHNICAL BACKGROUND

This chapter outlines the technological principles that are involved when combining sensor datasets from different sources. Section 2.1 gives an overview of the relevant 3D and HSI technologies used in this work. It should be mentioned at this point that various sensor combinations have been considered in this thesis and depending on the use cases, an adapted sensor combination was chosen. The principles for the registration of datasets, based on the chosen sensor combination, are described in section 2.2 and covers a total of 3 use cases from the fields of waste sorting, cultural heritage and forensics.

SENSORS

Spectral Information

Spectral imaging is a combination of spectroscopy and digital imaging. While a spectrometer only provides a single spectrum over a defined range of the electromagnetic spectrum, the combination of a dispersive element (i.e. prism or a diffraction grating) with digital imaging provides a spectrum for each pixel of the image [START_REF] Garini | Spectral imaging: Principles and Applications[END_REF]. In particular, the developments of the last decades have helped to improve the understanding of object characteristics and scenes. Spectral resolution is a key issue here. While multispectral systems only cover a dozen spectral bands, hyperspectral sensor systems allow a much higher spectral resolution. With several hundred narrow channels, HSI opens up completely new possibilities for analysis in a wide variety of application fields [START_REF] Khan | Modern Trends in Hyperspectral Image Analysis: A Review[END_REF] including the evaluation of food quality and safety [START_REF] Al-Sarayreh | Detection of Red-Meat Adulteration by Deep Spectral-Spatial Features in Hyperspectral Images[END_REF][START_REF] Pineda | Calyx and Stem Discrimination for Apple Quality Control Using Hyperspectral Imaging[END_REF]Q. Wang & Yu, 2019), medicine [START_REF] Ding | Hyperspectral Imaging With Burn Contour Extraction for Burn Wound Depth Assessment[END_REF][START_REF] Halicek | In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer[END_REF][START_REF] Wahabzada | Monitoring wound healing in a 3D wound model by hyperspectral imaging and efficient clustering[END_REF], forensic [START_REF] Cadd | Age Determination of Blood-Stained Fingerprints Using Visible Wavelength Reflectance Hyperspectral Imaging[END_REF][START_REF] D ; Edelman | Hyperspectral Imaging for Non-Contact Analysis of Forensic Traces[END_REF][START_REF] Kuula | Using VIS/NIR and IR spectral cameras for detecting and separating crime scene details[END_REF][START_REF] Majda | Hyperspectral imaging and multivariate analysis in the dried blood spots investigations[END_REF][START_REF] Yang | Crime Scene Blood Evidence Detection Using Spectral Imaging[END_REF], waste sorting (Bonifazi & Serranti, 2014;[START_REF] Moroni | Pet and PVC separation with hyperspectral imagery[END_REF][START_REF] Wu | Auto-sorting commonly recovered plastics from waste household appliances and electronics using near-infrared spectroscopy[END_REF][START_REF] Zhu | Characterization of microplastics on filter substrates based on hyperspectral imaging: Laboratory assessments[END_REF], precision agriculture [START_REF] Honkavaara | Remote Sensing of 3-D Geometry and Surface Moisture of a Peat Production Area Using Hyperspectral Frame Cameras in Visible to Short-Wave Infrared Spectral Ranges Onboard a Small Unmanned Airborne Vehicle (UAV)[END_REF][START_REF] Morsy | Multispectral LiDAR Data for Land Cover Classification of Urban Areas[END_REF][START_REF] Roscher | Detection of disease symptoms on hyperspectral 3D plant models[END_REF][START_REF] Vermeulen | Assessment of pesticide coating on cereal seeds by near infrared hyperspectral imaging[END_REF] and the art market [START_REF] Cucci | Reflectance Hyperspectral Imaging for Investigation of Works of Art: Old Master Paintings and Illuminated Manuscripts[END_REF][START_REF] Daniel | Hyperspectral imaging applied to the analysis of Goya paintings in the Museum of Zaragoza (Spain)[END_REF][START_REF] Pillay | Hyperspectral imaging of art: Acquisition and calibration workflows[END_REF][START_REF] Polak | Hyperspectral imaging combined with data classification techniques as an aid for artwork authentication[END_REF].

The structure of a HSI is similar to a RGB image and is represented as a data cube (also called hypercube), which is composed of vectors containing spectral information ( ) as well as two-dimensional spatial information ( and ) (Di [START_REF] Wu | Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment: A Review -Part I: Fundamentals[END_REF]. As presented in Figure 2.1, while an RGB image consists of 3 bands, an HSI comprises several hundred bands due to the higher spectral resolution. In both cases, the images consist of a series of sub-images, which are arranged as a sequence and correspond to a specific wavelength. In the case of RGB images, these are the wideband wavelengths red, green and blue. Each sub-image represents the spatial distribution of the spectral intensity related to the corresponding wavelength. Consequently, each HSI can be considered either as a spatial image of a specific wavelength or as a spectrum for each individual pixel of the image (Di [START_REF] Wu | Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment: A Review -Part I: Fundamentals[END_REF].

Figure 2.1. Representation of an HSI and the comparison with a RGB image. Each wavelength is represented as an image with x and y dimensions. The spectral component is presented as a vector for each pixel [START_REF] Lu | Medical hyperspectral imaging: A review[END_REF].

A HSI always have the same data cube structure. However, the construction of such a data cube differs in the acquisition strategy. A categorisation into 4 main acquisition strategies is made by [START_REF] Wu | Advanced applications of hyperspectral imaging technology for food quality and safety analysis and assessment: A Review -Part I: Fundamentals[END_REF] and is presented in Figure 2.2. A distinction is made between systems that scan sequentially and those that simultaneously record spatial and spectral information (Snapshot). Sequentially scanning systems can again be broken down into subcategories. Systems that scan point-based (Whiskbroom), line-based (Pushbroom) and frame-based (Spectral Scanning). The advantages of these systems are primarily their high spatial and spectral resolution. However, a high spatial and spectral resolution also means an increased amount of data to be processed. Large amounts of data are always a problem in terms of real-time capability. Consequently, these systems have only limited suitability for real-time applications. Snapshot systems, on the other hand, are very well suited for real-time applications due to their simultaneous acquisition capability. The disadvantage of these systems, however, is their low spectral and spatial resolution. Basically, it can be stated here that the choice of hyperspectral cameras should be made carefully, depending on the intended application.

The camera systems used in this work are two pushbroom cameras from Specim Ltd. (Oulu, Finland) and a spectral scanning system from Senop Oy (Basware, Finland). The sensors mentioned here were the best solution from the actual state of the art and were selected in the course of extensive research and analysis with regard to their specifications and the needs of the use cases exploited in this thesis. A detailed technical specification is given in for outdoor use to a limited extent or can only be used outdoors with significant effort, spectral scanning systems are specially designed for such applications. The choice of camera systems with regard to the requirements not only facilitates the measurement process itself, but is also advantageous with regard to the technical aspects described in chapter 2.2 when combining 3D and HSI data. The Specim FX10 captures the spectral signature from 400 nm to 1000 nm (224 bands) with a high spatial resolution of 1024 pixels per line, while the Specim FX17 captures the spectral signature from 900 nm to 1700 nm (224 bands) with a spatial resolution of 640 pixels per line. The combination of both systems allows the coverage of a wide electromagnetic spectrum. While the Specim FX10 mainly covers the VIS range and gives basic information about the colour of objects, the extension into the IR using the Specim FX17 allows much more potential in terms of capturing physical information. By extending the electromagnetic spectrum to NIR range materials and material compositions can be identified by a clear fingerprint.

Using these fingerprints can lead to advantages in a wide variety of applications.

Especially with regard to pushbroom cameras, many factors have to be considered for a clean data acquisition. Furthermore, there must be a linear movement between the object to be measured and the camera. For this purpose, the cameras were integrated into a fixed measurement setup with a linear traversing unit. Each camera looks vertically downwards at the carrier of the traversing unit. On the other hand, a lighting unit that corresponds to the working range of the camera is necessary. A total of 6 halogen spotlights (DECOSTAR 51 ALU 35W 12V 36deg GU5.3 lamps from Osram GmbH, Germany) were used for this purpose, especially for the Specim FX17, as it operates in the infrared range and halogen light sources cover the range deep into the infrared. A comparison of different light sources was carried out in the study by [START_REF] Mühle | Workflow and hardware for intraoperative hyperspectral data acquisition in neurosurgery[END_REF] and showed that the used halogen spotlights show a relatively constant increase in irradiance between 500 and 1000 nm. should be noted that if using a combination of the datasets, the different spatial resolutions can lead to a loss or change of spectral and spatial information. The effect of varying pixel sizes, however, depends on the heterogeneity or homogeneity of the objects to be captured. If a surface is homogeneous, e.g. a plastic part, the influence of an enlarged pixel is negligible because the homogeneity does not change the spectral content. For heterogeneous surfaces, on the other hand, the change in the size of a pixel can lead to changes in the spectral content, which can consequently also influence the quality of further analysis processes. The Specim FX10 (1024 px/line) has an object size of 0.22 mm per pixel, while the Specim FX17 (640 px/line) has 0.36 mm. To achieve homogeneous exposure of the objects, 3 halogen spots are mounted on both sides and illuminate the field of view of the cameras at an angle of about 45°.

Since the observation of the entire spectrum is of interest for many applications and is covered by two cameras in this setup, the combination of the two HSI data cubes is done by circular targets. These targets are visible in both HSIs and are used to estimate the transformation parameters. The result is a combined HSI with spectral signatures for each pixel covering the spectral region from 400 to 1700 nm. An example for combined spectral signatures is shown in Figure 2.4 and presents the spectral signature of two pixels for dandelion. At this point, it should be mentioned that the wavelengths below 500 nm were not included in the further analysis due to the stronger noise. For a clean fusion of the spectra, it is important that both cameras have been radiometrically calibrated under identical light conditions and by using the same calibration tile. Otherwise, an offset of the spectra in the overlapping range from 900 -1000 nm may occur. The radiometric calibration is a necessary step to convert unitless intensity values into normalized reflectance values to prevent spectral non-uniformity of the illumination device and to eliminate the influence of the dark current. For this purpose, calibration tiles with a matte, Lambertian reflecting surface are typically used, so that the reflected light has an almost equal intensity in all directions. The calibration process itself requires the acquisition of a reference image and a dark current image at a uniform exposure time [START_REF] Pillay | Hyperspectral imaging of art: Acquisition and calibration workflows[END_REF][START_REF] Shaikh | Calibration of a Hyper-Spectral Imaging System Using a Low-Cost Reference[END_REF]. In contrast to pushbroom sensors, the Senop HSC-2 camera shown in Figure 2.5 is designed to capture spectral information also in outdoor environments and based on a tuneable Fabry-Pérot interferometer (FPI) developed by the VTT Technical Research Centre of Finland (VTT) [START_REF] Mäkeläinen | 2D Hyperspectral Frame Imager Camera Data in Photogrammetric Mosaicking[END_REF][START_REF] Rissanen | VTT's Fabry-Perot interferometer technologies for hyperspectral imaging and mobile sensing applications[END_REF]. The spectral operating range of this camera is between 500 nm and 900 nm. Main advantage of the HSC-2 is the simultaneous acquisition of the spatial component (1-megapixel resolution) and the flexibility to freely select the needed spectral bands. The camera allows a very narrow band acquisition (up to 1000 freely selectable spectral bands), so that a very high spectral resolution can be achieved. With the selection of a smaller number of bands, however, it is also possible to realise real-time acquisitions (149 frames/s) and, in particular, drone applications.

Spatial Information

For the acquisition of 3D geometry, numerous 3D scanning systems based on different technologies are available. Each of these systems has its strengths and weaknesses in terms of resolution, working distance, accuracy, noise performance and limitations related to the nature of objects (e.g. shiny or transparent objects) [START_REF] Georgopoulos | Assessing the performance of a structured light scanner[END_REF]. The choice of 3D

technology should therefore also depend on the intended application

The use cases addressed in this thesis have different requirements for the measurement processes and the quality of the data basis. Let's take the application case from the field of waste sorting. The quality of the 3D data is an important factor influencing the success of sorting small electronic objects from the e-waste recycling sector. Small components such as capacitors or resistors can only be accurately described on the basis of their geometric properties if they can also be acquired with high spatial resolution and the required depth accuracy.

In this respect, a device should be selected that meets these requirements and is capable of being integrated into the measurement process in the context of a laboratory application.

Thus, in order to fulfill the requirements of the waste sorting application, a structured light scanner from GOM GmbH (Braunschweig, Germany) is used to capture the geometry and is part of the measurement system illustrated in Figure 2.3. The Atos Core 500 is a highresolution optical system with a maximum resolution of 0.195 mm and a depth accuracy of 0.05 mm. This 3D scanner consists of two stereoscopic cameras with 5-megapixel resolution and a blue LED light projector which projects structured light onto the object. The projected light, which essentially encodes the surface of the object, is captured by the cameras. Due to the structured light patterns on the object surface and the known physical relationship of the cameras to each other, the required 3D information can be determined using triangulation. The result of the scanning with the structured light scanner is a 3D point cloud.

In the used setup, the 3D sensor covers a measuring area of 450 x 335 mm and achieves a measuring point distance of 0.2 mm. The final 3D point cloud consists of several individual scans and was combined by using circular target points. The acquisition, registration and processing of the scans is done by using the GOM ATOS Professional 2019 software. The required number of scans varies depending on the object. While flat objects can be captured well with a small number of scans, objects with complex geometry require more acquisitions from different views. For the subsequent registration process between HSI and 3D, the objects of interest have not to be moved. For a complete registration of the geometry, either the angle of the sensor has to be adjusted or the objects have to be moved by the carrier unit. These limitations are among the reasons why structured light sensors should be used indoor and in static situations. In outdoor use, for example, problems such as sunlight or changing illumination conditions depending on clouds are to be dealt with.

This means that for outdoor use, further requirements have to be considered. As an example, take the application in the field of cultural heritage. The idea is to use a 3D technology that guarantees outdoor recognition and is insensitive to sunlight or changing light conditions.

Furthermore, the application intends to use HSI and 3D in the context of a drone application.

However, the carrying capacity of drones is limited, which means that the choice of sensors must be carefully considered. Furthermore, real-time acquisition must be guaranteed in the context of a drone application. In addition, as this thesis mainly focuses on the use of combined 3D and HSI datasets, it is important to choose 3D technologies that allow easy and straightforward registration with the hyperspectral sensor technology used. Therefore, in the context of the frame-based HSI sensor (Senop HSC-2), a technology called Structure from Motion (SfM) is used to generate 3D point clouds. SfM is an image-based method and has the advantage that the Senop HSC-2 sensor is perfectly adequate for generating spectral and spatial 3D data. This would cover the problems with regard to carrying capacity and also real time acquisition when using a drone. The detailed procedure of combining 3D and HSI is described in the following sections. It is also possible to combine the Senop HSC-2 with the structured light scanner or other 3D scanning technologies (e.g. lidar, time-of-flight), but as already mentioned, the choice of sensor technology must be suitable for the specific use case. For the use case investigated in this thesis, the combination of the Senop HSC-2 and SfM method is the most flexible and suitable.

SfM is an alternative methodology for obtaining 3D information by considering multiple images of an object or scene and is based on two key-fields: photogrammetry and computer vision [START_REF] Eltner | Structure from motion photogrammetric technique[END_REF][START_REF] Luhmann | Nahbereichsphotogrammetrie: Grundlagen -Methoden -Beispiele (4[END_REF]. A prerequisite for 3D reconstruction is that the object to be reconstructed is captured from multiple locations to ensure that the entire object information is available multiple times in different overlapping images. The evaluation of the images is done in a first step by detecting and extracting features in each individual image.

After finding corresponding features distributed over all images, these homologous image points are used to reconstruct a photogrammetric model. The bundle adjustment that takes place in the next step of the evaluation primarily has the aim of minimising errors of observed image coordinates and determining 3D object coordinates. The result of the bundle adjustment includes not only the adjusted object coordinates, but also the internal camera parameters (focal length, principal point and distortion parameters) and exterior orientations describing the position (three shifts) and orientation (three rotations) of all images, so that the geometric state of the camera at the time of acquisition can also be determined. The further processing of the obtained 3D object points can ultimately be carried out in any way. Generally, SfM is successful if a sufficient surface texture is present, as this method relies mainly on the presence of image features, if a dense image overlap is achieved and if the camera used is either calibrated in advance or at least remains mechanically stable during image acquisition. Due to the type of feature extraction and the measure used to calculate the similarity between the images, the achievable accuracy is general around 0.5 -2 pixels [START_REF] Luhmann | Nahbereichsphotogrammetrie: Grundlagen -Methoden -Beispiele (4[END_REF].

COMBINING SPATIAL AND SPECTRAL DATASETS

In this thesis, three different use cases from completely different fields of application, namely forensics, waste sorting and cultural heritage, are used to illustrate the advantages of a combined 3D and HSI approach. An overview of the use cases, the used sensor technologies and the investigations carried out is given in Table 2.2. Depending on the application, the sensor technology used and also the process of dataset fusion differs.

Investigations based on used cases

Waste Sorting

Used sensors:

Structured Light Scanner & Pushbroom Camera (900 -1700 nm)

 Investigations include spectral and spatial domains. The combination of 3D and HSI leads to advantages in the classification of materials and components.

Cultural Heritage

Used sensors and methods:

Outdoor: SfM & Spectral Scanning Camera (500 -900 nm)

Laboratory: Pushbroom Cameras (400 -1700 nm)

 Investigations are based on spectral domain. The 3D component is mainly used for localization.

Forensic

Used sensors:

Pushbroom Cameras (400 -1700 nm)

 Investigations are based on spectral domain. The 3D component would be for localization.

For example, use cases in the field of waste sorting require a fixed structure with conveyor belt, controlled lighting conditions and corresponding sensors. Pushbroom sensors are ideal in this context, as the movement between object and camera is given by the conveyor belt. The use of pushbroom sensors in outdoor environments on the other hand, would complicate the acquisition process, as the necessary prerequisites are not given. We can consider drone applications. Pushbroom sensor technology requires linear, uniform movement for an accurate acquisition. This is difficult to achieve with a drone. In this context, the use of snapshot systems is more appropriate. In the following, some general concepts for sensor fusion will be presented and for each of the three use cases, a sensor combination and the corresponding methodology for the fusion of the datasets are proposed.

Registration

The fusion or registration of different datasets is part of many studies and is the process of determining the geometric relationship between two or more sensors. This relationship describes precisely the rigid transformation between two sensors (e.g. 2D-to-2D, 2D-to-3D, 3Dto-3D) and includes 3 translations and 3 rotations (i.e. extrinsic parameters). For the determination of the extrinsic parameters, different approaches exist. However, the basic prerequisite for all approaches is the presence of identical features in both datasets. Features are used to geometrically link datasets and do not have to be specific 2D/3D control points. Rather, they are selected in such a way that a sufficient object area is covered and a rigid connection of both datasets to each other is ensured. The distribution of the features should be chosen in such a way that a stable geometric coupling of the datasets can be guaranteed [START_REF] Luhmann | Nahbereichsphotogrammetrie: Grundlagen -Methoden -Beispiele (4[END_REF].

A robust and proven approach is to equip the scene to be captured with markers/targets or to use reference objects like checkerboards. [START_REF] Geiger | Automatic camera and range sensor calibration using a single shot[END_REF] for example, uses several checkerboards distributed in space to determine the exterior orientation between a camera and a range sensor. In the study of [START_REF] Liao | Extrinsic Calibration of Lidar and Camera with Polygon[END_REF] a polygon board is used to determine the exterior orientation between a camera and a 3D Lidar by using lines as features. There are many other works that solve the correspondence between the sensors in a similar way (e.g. [START_REF] Fuersattel | Accurate laser scanner to camera calibration with application to range sensor evaluation[END_REF][START_REF] Kummerle | Automatic Calibration of Multiple Cameras and Depth Sensors with a Spherical Target[END_REF][START_REF] Peikui | Improving High-Throughput Phenotyping Using Fusion of Close-Range Hyperspectral Camera and Low-Cost Depth Sensor[END_REF][START_REF] Schilling | Entwicklung von Verfahren zur rechnergestützten Verknüpfung von Bild-und Laserscandaten auf Basis von Passobjekten[END_REF]), but such an approach is associated with effort, either because the scene to be captured must be provided with markers or because special reference objects are necessary. An alternative to markers or reference objects are the automatic detection of natural points by using features like SURF [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] or SIFT [START_REF] Lowe | Object recognition from local scale-invariant features[END_REF]. Methods using SURF and SIFT features for the fusion of HSI and Laser Scans are part of the studies of [START_REF] Nieto | 3D Geological modelling using laser and hyperspectral data[END_REF][START_REF] Zhang | Automatic Fusion of Hyperspectral Images and Laser Scans Using Feature Points[END_REF]. Such an approach, however, depends on lighting conditions and requires the presence of sufficient texture, as robust feature extraction and a stable correspondence is not achievable if the texture of the scene or object is weak. A well-established approach in the medical field for the registration of CT and MRI images is Mutual Information (MI) [START_REF] Kosiński | Robust Image Registration Based on Mutual Information Measure[END_REF][START_REF] Studholme | An overlap invariant entropy measure of 3D medical image alignment[END_REF]. In [START_REF] Pandey | Automatic Extrinsic Calibration of Vision and Lidar by Maximizing Mutual Information[END_REF], MI is used to establish the geometric relationship between an omnidirectional camera and a 3D lidar sensor mounted on the roof of a vehicle. Another MI based method for the determination of the geometric relationship between a camera and a Lidar sensor in a natural environment is given in (Taylor & Nieto). In [START_REF] Igelbrink | Markerless Ad-Hoc Calibration of a Hyperspectral Camera and a 3D Laser Scanner[END_REF], MI is used for the registration between a pushbroom hyperspectral camera and a 3D laser scanner.

Both hyperspectral camera and laser scanner are in a fixed relation mounted on a mobile robot and used to create panoramic images of the scene and then to register them with each other using MI. The fixed sensor arrangement is crucial to the success of the method and allows the derivation of assumptions about the model parameters, which are necessary for a global registration using the MI-based method.

The techniques for registration used in this work have been developed depending on the use cases in focus and represent the best solution in relation to the individual situations. In the following sections, two approaches for the registration of HSI and 3D data are presented. The first solution is optimised for the waste sorting use case and covers a laboratory solution. By using a fixed measurement setup (sensors fixed in a frame with perpendicular acquisition direction), the registration of the datasets is done by tie points distributed in the measurement field. The second solution addresses the cultural heritage use case and the potential use of HSI and 3D in the context of a drone application. The solution is therefore based purely on HSI sensor technology. The 3D data is generated by using natural features (e.g. SIFT/SURF) in the HSI dataset and the use of these in the context of photogrammetric approaches. The registration problem is therefore solved directly by this process and offers enormous advantages in terms of an outdoor application, as no manually applied tie points are necessary and no further 3D sensor technology needs to be installed as part of a drone system.

The detailed description of the flexible and generally valid methods used in this work for combining 3D and HSI is given in the following sections.

Waste Sorting Use Case

The first use case is in the field of waste sorting and essentially deals with the sorting of electronic wastes. Technological innovations, short production cycles and, above all, ever shorter product life cycles are reasons for an increased amount of electronic wastes. Worldwide, 53.6 million tonnes of electronic waste were produced alone in 2019, but only 17.4% of it was recycled [START_REF] Forti | The Global E-waste Monitor 2020: Quantities, flows, and the circular economy potential[END_REF]. Electronic wastes that are not recycled ends up in incineration plants and thus not only harm the environment but also humans, as toxic substances are released during the incineration process. This makes recycling processes that help to achieve sustainability more important. The main electronic waste components to be separated can be divided into cables, plastics and printed circuit boards (PCB) as shown in Figure 2.6. While the cables can be "easily" separated from the remaining parts due to their geometry, the separation of plastics and PCBs, which represent a highly valuable resource consisting of polymers and precious metals [START_REF] Calvini | Growing applications of hyperspectral and multispectral imaging[END_REF], in particular is challenging. The focus of this use case is therefore on the separation of plastics and PCBs, whereby the identification of individual components on PCBs such as capacitors, integrated circuits or inductors presents a further particular challenge. To achieve this, the measuring system shown in Figure 2.3 has been used for the acquisition of real electronic waste samples consisting of plastics and PCBs.

The wavelength required is limited to the NIR range, as all components should be separated according to their physical information, i.e. according to their material composition. Therefore, in this use case, only the Specim FX17, which operates in the range from 900 -1700 nm, is used for the spectral measurements.

The acquired datasets were radiometrically normalised by using a dark reference image for the dark current (closed shutter) and a white reference image to reduce the influence of intensity variability. A 99% Spectralon TM reflectance tile (Specim), was used for the white calibration. The dimensions of the calibration tile are 300 x 25 x 10 mm (length x width x height) and cover the entire visual field of both cameras. For the acquisition of the geometry, the integrated structured light scanner Atos Core 500 was used.

Like described in section 2.2.1. the datasets obtained from both sensors have been combined using tie points (i.e. 2D/3D features) selected in 3D and HSI. Due to the specific technical design of the measurement system, the objects are acquired both hyperspectral and threedimensional in one sequential measurement process. That means, the objects on the object carrier are not allowed to move during the measurement process. Only the object carrier is moved. The spectral data is acquired one after the other. The object carrier is finally positioned under the structured light scanner so that the objects are also captured geometrically. Figure 2.7 shows such a tie point with a diameter in object space of 7 mm. The determination of the tie points is done automatically in 3D with sub-pixel accuracy using the GOM ATOS Professional 2019 software. In the HSI, on the other hand, the centres of the tie points are based on a manual tie point picking. The Root Mean Square Error (RMSE) for the tie points is generally between 0.5 and 1 pixel, depending on the accuracy of the manually picked points.

The quality of the registration is significantly influenced by the care with which the points are selected. The registration accuracy is of crucial importance and directly influences all further

processes. An optimisation of the target detection through automated circle detection would be desirable in order to increase the accuracy of the fusion process. Especially at the transitions between different elements the influence of a low registration accuracy would be noticeable.

As an example, in Figure 2.8 (a) we have a circuit board with tracks. We can see that the transitions from the tracks to the board are not neatly assigned. However, we also see that the actually straight traces in the HSI are displayed with a small shift in the combined dataset.

In addition to the influence of the registration accuracy, this is also due to the different spatial resolutions of the underlying datasets. While the HSI with a resolution of 413 x 514 pixels displays fine tracks in a high resolution, the transformation of the HSI data to the 3D dataset (x-y-plane of 450 x 250 pixels) results in a loss of spatial resolution. This leads to the HSI shown in Figure 2.8 (c) after the transformation. This transformed dataset forms the basis for the assignment between 3D point and spectral signature. In this particular case, the influence of the registration accuracy plays a minor role and the differences in spatial resolution between the datasets have a much greater influence on the quality of the merged data.

After successful fusion of the datasets in x-y plane, the result is a 3D point cloud with an underlying hyperspectral image. Due to the different resolutions of the datasets, the automated assignment between 3D point and spectral signature is done pixel by pixel by checking the neighbourhood. Concretely, this means that for each x-y-coordinate of each individual point of the point cloud, the nearest neighbouring pixel of the HSI is assigned. By following this procedure, a spectral signature can be assigned to each individual 3D point. An example of a combined dataset can be found in Figure 2.9. The 8 parameters of a projective geometric transformation and the accuracy of the transformation parameters for these objects are presented in Table 2.3. The configuration of the measurement system in combination with the registration methodology enables a simple fusion of 3D and HSI data. The underlying sensor technology in particular offers great potential for a wide variety of applications, due to the high spectral

Cultural Heritage Use Case

The second use case is in the field of cultural heritage and is based on a research activity done by the Institute for Stone Conservation2 (Mainz, Germany). Weather conditions such as heat, cold and rain often lead to pronounced damage on a large number of castles and ruins.

The tops of the walls are particularly affected, as this is where the greatest damage occurs due to climatic effects. But it is not only climatic conditions that cause damage. Certain types of vegetation (e.g. roots of trees) can also be damaging. As an example, Figure 2.10 shows a wall section of the Balduinseck castle ruin (Buch, Germany) with flat-growing vegetation (e.g. dandelions) and vegetation in form of shrubs and trees.

The preservation of castles and ruins and the repair of damage requires a careful approach. Before any restoration work can begin, the affected areas must be cleared of their natural vegetation in a first step. After the restoration work has been completed, however, the treated areas are re-vegetated, because naturally vegetation provides effective protection against temperature changes and moisture in the masonry. However, it should also be noted here that a distinction must be made between protective and harmful vegetation species. While mosses and lichens tend to be protective species, trees and shrubs are species that cause damage through their roots. The investigations in this use case thus aim to examine how well spectral and geometrical information could be used to achieve a separation between different plant species. In order to generate a combined dataset consisting of hyperspectral and geometric information, the Senop HSC-2 camera has been used in combination with the SfM method. The generation of dense 3D point clouds is based on the processing of the hyperspectral images with Agisoft Metashape version 1.7.2 (Agisoft LLC., St. Petersburg, Russia). The automatic processing includes image alignment by calculating the camera position (three shifts and three rotations) using feature points and feature matching between all images, and the generation of a dense point cloud using depth maps calculated from stereo matching.

An example point cloud with texture calculated from 8 overlapping hyperspectral images acquired with the Senop HSC-2 are shown in Figure 2.11. For the assessment of the global accuracy of the SfM solution, the total reprojection error serves as an indicator [START_REF] Altuntas | Georeferencing and reprojection error investigation on image based 3D digitization and mapping of historical buildings[END_REF]. This error represents the Euclidean distance between a measured image point and the reprojection of the corresponding estimated 3D point. In the example shown in Figure 2.11, the total reprojection error is less than 0.5 pixel. The accuracy of SfM is usually given as 0.5 to 2 pixels for commercially available digital cameras. This depends on the camera, the total number of images, the geometry of ray intersection and the quality of the interior and exterior orientations. Considering the fact that wind-induced movement of plants also plays a role in outdoor measurements, the achieved error of 0.5 pixel can be considered quite reasonable. With consideration of the optical properties of the Senop HSC-2 and at an approximate working distance of about 2 m, the size of a pixel in the object is 1.29 mm. This is quite acceptable for the detection of mossy surfaces or harmful larger woods. It should also be noted that the error achievable here only affects the quality of the 3D geometry. Through the process of fusing 3D and HSI which will be described in the following, this error does not affect the assignment between 3D point and spectral signature. An essential factor during the measurement process is the acquisition of the objects or the scenery under stable lighting conditions. While in laboratory applications the lighting conditions can be controlled reliably, this is difficult when measuring outdoors under natural lighting conditions. Changing light conditions due to the rotation of the earth and consequently changes in the position of the sun and climatic changes lead to variations in the light conditions [START_REF] Eckhard | Outdoor scene reflectance measurements using a Bragg-grating-based hyperspectral imager[END_REF]. In order to enable comparable spectral measurements, the normalisation of the HSI is essential.

To reduce the influence of intensity variability, all acquired HSI were radiometrically normalised by using a white reference reflectance target with 75% reflectivity. The used target (500 x 500 mm) is an EnsoMOSAIC3 Agri reflectance target and is made of material with nearly Lambertian reflection properties.

Let's summarise the data base so far. We have individual radiometrically normalised HSI of the scene and we have generated a 3D point cloud of the captured scene through the HSI images. Different ways to solve the registration problem between HSI and 3D point cloud have already been discussed in section 2.2.1. Most of these possibilities involve additional effort and require, for example, the equipping of the scene to be captured with targets and a prior calibration of the sensors used. With the approach chosen in this thesis, we reduce the effort as the 3D point cloud is generated directly from the HSI and thus only one sensor serves as a basis. This has the advantage that only a flexible and simple procedure has to be found to establish the correspondence between a 3D point and the spectral signature. In this context, in particular the depth maps are of enormous importance. A depth map is a 2D image where the intensity values at each index of the image reflect the depth of the object from the camera.

Agisoft Metashape permits the export of depth images for each used image in the original resolution. Using the intrinsic camera parameters (focal length and principal point in x, y) estimated simultaneously during the SfM 3D reconstruction process, the depth information can be converted into 3D data using Equations 2.1 [START_REF] João | An open-source tool for distributed viewing of kinect data on the web[END_REF]).

= - *

(2.1)

= - *

The intrinsic camera parameters used are the principal point ( , ) and the focal length ( ). and correspond to the position on the image (1024 x 1024 pixels). The parameter correspond to the depth of each pixel at position and . The desired 3D information for each individual pixel of the image is given by , and .

Through this approach we have a direct correspondence between the underlying images and the 3D information. Since the calculations are based on the hyperspectral images, we also have a direct correspondence between the 3D information and the spectral information. An example of this correspondence is shown in Figure 2.12 in the form of a 3D point cloud with average reflectance value over the entire spectrum (500 -900 nm) shown per each 3D point.

The approach proposed here for combining 3D and HSI is based on a flexible and simple method. The prerequisite, however, is that the objects or the scene to be captured have enough texture for feature detection. This is a general problem with image-based approaches. Poor texture or highly homogeneous areas make it difficult to find characteristics in the images, which in turn leads to incorrect calculations and consequently leads to failure of the approach.

Figure 2.12. 3D point cloud obtained from depth image and HSI visualised with average reflectance value over the entire spectrum (500 -900 nm) shown for each 3D point.

Forensic Use Case

The third use case is from the field of forensics and was realised in cooperation with the Hessian State Criminal Police Office (LKA Hessen, Germany) and the Federal Criminal

Police Office (BKA Wiesbaden, Germany). Finding traces such as blood, fingerprints or gunshot residue makes it possible to reconstruct the crime course and provides important insights that can contribute to solving the crime.

The investigations in this use case are mainly aimed at finding traces of blood. Chemical substances such as luminol, which emits a bright blue light when in contact with blood, are usually used to search for traces of blood [START_REF] Manoharan | The Chemiluminescent imaging of blood on washed cotton and polyester fabrics[END_REF]. Depending on the area under investigation, this can be very time-consuming and costly. An alternative non-invasive technique for trace detection is HSI. A number of studies have investigated the potential of HSI in forensic evidence and have shown that HSI is not only suitable for finding blood traces, but can also be used to determine the age of blood traces [START_REF] Cadd | Age Determination of Blood-Stained Fingerprints Using Visible Wavelength Reflectance Hyperspectral Imaging[END_REF][START_REF] D ; Edelman | Hyperspectral Imaging for Non-Contact Analysis of Forensic Traces[END_REF][START_REF] Edelman | Identification and age estimation of blood stains on colored backgrounds by near infrared spectroscopy[END_REF][START_REF] Majda | Hyperspectral imaging and multivariate analysis in the dried blood spots investigations[END_REF][START_REF] Yang | Crime Scene Blood Evidence Detection Using Spectral Imaging[END_REF]. The detection of hidden blood traces, on the other hand, has not yet been investigated. A removal of traces by the culprit to cover up the crime can make the forensic examination even more difficult. The removal of traces for example can be the painting over of surfaces or the setting of areas and objects on fire. Therefore, the focus of our investigations is on the detection of blood traces hidden by paint and soot and mainly based on HSI acquisitions in the laboratory using the measurement system shown in Figure 2.3.

The practical procedure for crime scene surveying usually involves the use of technology to capture the scenery in 3D. These can be 3D laser scanners or image-based methods like SfM. For the combined use of 3D and HSI (whereby 3D is only used for localisation), various sensor combinations are possible. For trace detection in the visible range of the electromagnetic spectrum, a procedure as described in section 2.2.3 based on the Senop HSC-2 camera and SfM method is conceivable. For the capture of a broader electromagnetic spectrum (e.g.

UV, NIR), pushbroom sensor technology must be used. In this case, a combination with 3D laser scanners would be conceivable. Methodologies for combining the aforementioned sensor systems can be found, for example, in [START_REF] Igelbrink | Markerless Ad-Hoc Calibration of a Hyperspectral Camera and a 3D Laser Scanner[END_REF][START_REF] Nieto | 3D Geological modelling using laser and hyperspectral data[END_REF][START_REF] Wendel | Extrinsic Parameter Calibration for Line Scanning Cameras on Ground Vehicles[END_REF] and are not part of the investigations in this use case.

CHAPTER 3: STATE OF THE ART -SPECTRAL ANALYSIS AND CLASSIFICATION

HSI is rich in spectral and 2D spatial information and thus a valuable data basis for a large number of applications, such as disease diagnostics (Z. Liu et al., 2012;[START_REF] Ernesto | Multispectral Endoscopy to Identify Precancerous Lesions in Gastric Mucosa[END_REF][START_REF] Wahabzada | Monitoring wound healing in a 3D wound model by hyperspectral imaging and efficient clustering[END_REF] or food quality control [START_REF] Fan | Prediction of soluble solids content of apple using the combination of spectra and textural features of hyperspectral reflectance imaging data[END_REF][START_REF] Huang | Rapid and non-invasive quantification of intramuscular fat content of intact pork cuts[END_REF].

However, only HSI database is not enough to achieve the specific objectives of these applications. Only in combination with the appropriate processing methods is it possible to exploit the full potential of a HSI dataset.

Therefore, this chapter outlines a literature review of different methods to analyse spectral signatures. The pre-processing of HSI, spectral features extraction and selection of significant bands in HSI are described in section 3.1. Furthermore, an overview of existing approaches for the classification of multidimensional data is provided in section 3.2.

ANALYSIS OF SPECTRAL SIGNATURES

As with all measurement processes, hyperspectral acquisition processes and the resulting HSI are also affected by undesired noise. The source of the presence of noise can vary and can be divided into random noise, caused by changes in the observed properties of the sample and environmental factors such as temperature variations, and systematic noise, resulting from changes in instrumentation due to ageing detectors, inherent instabilities or transformation between different HSI systems [START_REF] Dai | Recent Advances in De-Noising Methods and Their Applications in Hyperspectral Image Processing for the Food Industry[END_REF]. Other influences that affect the raw spectra are, for example, background interferences or overlapping absorption bands of other components.

Consequently, it is common practice to perform correction methods to reduce, eliminate or standardize these influences and to obtain valuable information about the properties of the observed sample.

Well-known methods for pre-processing are e.g. scatter correction methods like Standard Normal Variate (SNV) [START_REF] Barnes | Standard Normal Variate Transformation and De-Trending of Near-Infrared Diffuse Reflectance Spectra[END_REF], baseline correction methods like spectral derivative transformation, methods for random noise reduction like Savitzky-Golay smoothing [START_REF] Savitzky | Smoothing and Differentiation of Data by Simplified Least Squares Procedures[END_REF]) and many others [START_REF] Rinnan | Review of the most common preprocessing techniques for near-infrared spectra[END_REF]. An example for a pre-processed spectral signature based on Polypropylene (PP) is given in In particular, derivatives have been used for decades in the context of pre-processing due to their robustness to spectral shifts. An example of the ability of derivatives to eliminate spectral shifts (additive and multiplicative effects) is shown in Figure 3.2.

The amount of methods available often makes it difficult to choose the optimal technique, especially as there is no optimal universal approach. The type and extent of pre-processing methods are data specific and therefore require a careful selection strategy i.e. the choice of processing steps must be made depending on what needs to be achieved (e.g. elimination of disturbing effects, highlighting of areas). To give an example, [START_REF] Heil | An Evaluation of Different NIR-Spectral Pre-Treatments to Derive the Soil Parameters C and N of a Humus-Clay-Rich Soil[END_REF]) conduct an evaluation of different pre-processing methods in their study in order to determine their suitability for deriving parameters of soil properties using NIR spectra and to determine which wavelengths are most important for predicting the parameters. The results of this study showed that the varying pre-processing methods have a significant impact on the performance of the selected parameters. The choice of pre-processing steps should therefore always depend on the defined objective and should aim to improve the results in order to achieve this objective.

Figure 3.2. The effect of using 1 st and 2 nd order derivatives to eliminate additive (green) and additive plus multiplicative (red) spectral reflectance shifts. The blue graph is the spectra without any offsets, the black dotted line is the zero line [START_REF] Rinnan | Review of the most common preprocessing techniques for near-infrared spectra[END_REF].

A further method for the pre-processing of spectra is a normalization procedure called Continuum Removal (CR). The advantage of this method is that it highlights the absorption peaks and allows a better quantification after removing the overall concave shape of spectral curves [START_REF] Clark | Imaging spectroscopy: Earth and planetary remote sensing with the USGS Tetracorder and expert systems[END_REF][START_REF] Clark | Reflectance spectroscopy: Quantitative analysis techniques for remote sensing applications[END_REF]). An illustration for the spectral signature of PP is given in Figure 3.3. In the research of [START_REF] Kruse | Spectral-feature-based analysis of reflectance and emission spectral libraries and imaging spectrometer data[END_REF], CR is used to highlight the absorption peaks and to detect features based on them. The features such as band position, band depth, band width and band asymmetry are used to characterise the absorption peaks.

As already mentioned, hyperspectral images are data cubes with a high information content. Let's take an example of a data cube taken with the Senop HSC-2 camera. The camera provides a spatial resolution of 1024 x 1024 pixels and covers the spectral range of 500 -900 nm. Assuming a spectral resolution of 1 nm, that would mean 400 bands for each spectral signature. This means that we would have a total of 1.048.576 pixels, with each individual pixel comprising a spectral vector of 400 bands. Processing such a large amount of information is not only a problem in terms of performance, but also leads to the well-known problem known as Curse of Dimensionality (also called Hughes Phenomenon) (Richard [START_REF] Bellman | Dynamic programming: Series: Princeton Landmarks in Mathematics and Physics[END_REF]. The high-dimensional space leads to problems in data processing. Taking classification as an example, the use of HSI in its raw form leads to negative effects such as a decrease in classification accuracy, an increase in the search space and an increase in the complexity of the model, with the result that the computational effort of the model tends to increase exponentially [START_REF] Ren | Partitioned Relief-F Method for Dimensionality Reduction of Hyperspectral Images[END_REF]. To prevent these effects, it is therefore essential to perform a dimensional reduction.

The reduction of the dimension can be achieved through two different methods, feature extraction and feature selection. Feature extraction improves relevant bands through arithmetic operations or projects the data onto a new feature space while retaining the discriminatory information (B. [START_REF] Kumar | Feature extraction for hyperspectral image classification: a review[END_REF]. The most commonly known feature extraction methods include Principal Component Analysis (PCA) [START_REF] Jiang | SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery[END_REF], Minimum Noise Fraction (MNF) (G. [START_REF] Luo | Minimum Noise Fraction versus Principal Component Analysis as a Preprocessing Step for Hyperspectral Imagery Denoising[END_REF], Independent Component Analysis (ICA) (J. [START_REF] Wang | Independent component analysis-based dimensionality reduction with applications in hyperspectral image analysis[END_REF]) and many other algorithms [START_REF] Ren | Partitioned Relief-F Method for Dimensionality Reduction of Hyperspectral Images[END_REF]). An overview of features extraction techniques, a detailed description and a list of strengths and weaknesses can be found in the study by (B. [START_REF] Kumar | Feature extraction for hyperspectral image classification: a review[END_REF].

A major disadvantage of feature extraction methods, however, is the fact that these techniques usually change the physical meaning of the original spectra, since the spectral bands in low-dimensional space do not correspond to individual original bands, but to their linear combinations [START_REF] Du | Similarity-Based Unsupervised Band Selection for Hyperspectral Image Analysis[END_REF][START_REF] Habermann | Supervised band selection in hyperspectral images using single-layer neural networks[END_REF].

The second category of dimension reduction methods, named features selection (also called bands selection), uses the selection of a representative subset from the original set of bands by preserving the physical characteristics of the original spectral bands. In the review article by [START_REF] Sawant | A survey of band selection techniques for hyperspectral image classification[END_REF], different features selection methods are listed and considered in terms of their challenges. The authors summarise that in most feature selection methods, bands are selected individually without considering the relationships between them and thus, the selected bands do not represent the characteristics of the original data. Furthermore, water absorption or noisy bands must be removed manually, as they have a low discriminatory power. However, manual removal of these low discriminatory bands is a very expensive, timeconsuming and requires expert knowledge. The authors also mention other aspects, such as the fact that the determination of the most appropriate number of bands is still a challenging issue [START_REF] Torres | Spatial Spectral Band Selection for Enhanced Hyperspectral Remote Sensing Classification Applications[END_REF].

Basically, it can be concluded that the number of existent studies (e.g. [START_REF] Benabadji | Unsupervised hyperspectral band selection by combination of unmixing and sequential clustering techniques[END_REF][START_REF] Li | Unsupervised Band Selection of Hyperspectral Images via Multi-Dictionary Sparse Representation[END_REF][START_REF] Luo | Unsupervised Band Selection Method Based on Importance-Assisted Column Subset Selection[END_REF][START_REF] Su | A Saliency-Based Band Selection Approach for Hyperspectral Imagery Inspired by Scale Selection[END_REF][START_REF] Su | Band Ranking via Extended Coefficient of Variation for Hyperspectral Band Selection[END_REF][START_REF] Sun | Adaptive Distance-Based Band Hierarchy (ADBH) for Effective Hyperspectral Band Selection[END_REF] Q. [START_REF] Ren | Partitioned Relief-F Method for Dimensionality Reduction of Hyperspectral Images[END_REF]) demonstrates the lack of a satisfactory, robust and reliable methodology and that HSI bands selection is one of the most challenging open issues in spectral imaging.

The methods described so far are purely statistical procedures for determining characteristics of spectral signatures and consider only a small set of characteristic values. An alternative to statistical methods is the consideration of physical effects coming from the object and its reflective behaviour and their representation in the shape of spectral signatures.

Spectral signatures are an expression of the mixed reflectance of the elements of which an object is composed (e.g. molecules, pigments, cell structure, water content), with the result that each individual component has only a proportional influence on a spectrum. Modifications in the composition of the elements then mainly change the mixture of all spectral contributions, leading to local or regional variations as well as changes in the shape of a spectrum [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]. The consideration of shape describing features can therefore be a beneficial approach and was investigated in [START_REF] Boochs | Shape of the red edge as vitality indicator for plants[END_REF][START_REF] Chen | Remote sensing for vegetation monitoring in carbon capture storage regions: A review[END_REF][START_REF] Kruse | Spectral-feature-based analysis of reflectance and emission spectral libraries and imaging spectrometer data[END_REF].

For example, the work of [START_REF] Chen | Remote sensing for vegetation monitoring in carbon capture storage regions: A review[END_REF]) consists of an approach that makes full use of the spectral signature shape by using a code to parameterise the spectral shape. The use of spectral shape features (e.g. band position, band depth, band width and band asymmetry) for the description of absorption peaks of spectral signatures was investigated in [START_REF] Kruse | Spectral-feature-based analysis of reflectance and emission spectral libraries and imaging spectrometer data[END_REF]. The disadvantage of both approaches is the effort required to determine the shape describing features of the spectral signatures. In both approaches, tables are used to store the features followed by a comparison between the stored features of the reference spectra and the spectra of interest, which also makes the classification process a time-consuming task.

CLASSIFICATION OF SPECTRAL SIGNATURES

The term classification describes the assignment of each pixel (consisting of a spectrum) of a HSI to a predefined class label. The result of such a classification procedure is a classification map with the dimensions of the underlying HSI and the aim of finding and grouping objects based on similarities, identifying materials or objects detection processes.

There are a considerable number of studies on the classification of HSI. Examples include the classification of HSI using indices [START_REF] Mahlein | Development of spectral indices for detecting and identifying plant diseases[END_REF][START_REF] Shirzadifar | Development of spectral indices for identifying glyphosate-resistant weeds[END_REF], on the basis of expert knowledge [START_REF] Andrés | Ontology-based classification of remote sensing images using spectral rules[END_REF][START_REF] Ghazaryan | A rule-based approach for crop identification using multi-temporal and multi-sensor phenological metrics[END_REF] or on the basis of machine learning techniques. In particular, machine learning based techniques are among the most widely used approaches and include methods such as k-nearest neighbours (KNN) or support vector machines (SVM), up to deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN). Each has its strengths and weaknesses and has been subject of numerous studies and comparisons in articles such as [START_REF] Chutia | Hyperspectral Remote Sensing Classifications: A Perspective Survey[END_REF]Ghamisi, Plaza, et al., 2017;[START_REF] Ghamisi | Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art[END_REF][START_REF] Gogineni | Hyperspectral Image Classification[END_REF][START_REF] Jia | A survey: Deep learning for hyperspectral image classification with few labeled samples[END_REF][START_REF] Ms | Classification Analysis on Hyperspectral imagery data for accuracy improvement[END_REF][START_REF] Paoletti | Deep learning classifiers for hyperspectral imaging: A review[END_REF].

Fundamentally, dealing with HSI classification is a challenging task due to the high dimensionality, limited or unbalanced training datasets, spectral variability and mixed pixels [START_REF] Gogineni | Hyperspectral Image Classification[END_REF]. The most common problem is the Curse of Dimensionality described in section 3.1.

Especially in the context of machine learning based approaches, the influence of training datasets is a problem. Machine learning approaches require a large number of training datasets to train models well and to tune the numerous parameters. However, the process of providing training datasets requires a lot of effort in annotation, which in practice is often very timeconsuming and also requires expert knowledge. Furthermore, it is important to ensure that training datasets are truly representative and captures all influences. If this is not the case, it will affect the quality when applied to data with slightly varying content. These restrictions often lead to an insufficient base of training datasets [START_REF] Jia | A survey: Deep learning for hyperspectral image classification with few labeled samples[END_REF]. Approaches to address this problem in regard to deep learning classification approaches are part of the study in [START_REF] Jia | A survey: Deep learning for hyperspectral image classification with few labeled samples[END_REF]. The authors examined different approaches based on few label classifications and concluded that while the use of few annotated data saves a lot of time and effort, it also quickly introduces new challenges such as over-fitting [START_REF] Molinier | Avoiding Overfitting When Applying Spectral-Spatial Deep Learning Methods on Hyperspectral Images with Limited Labels[END_REF] and weak generalisation.

An alternative to machine learning based classifiers are indices [START_REF] Huete | Development of vegetation and soil indices for MODIS-EOS[END_REF][START_REF] Mahlein | Development of spectral indices for detecting and identifying plant diseases[END_REF][START_REF] Meng | Development of Spectral Disease Indices for Southern Corn Rust Detection and Severity Classification[END_REF][START_REF] Varade | Development of Spectral Indexes in Hyperspectral Imagery for Land Cover Assessment[END_REF]. Indices combine a few wavelengths of the captured electromagnetic spectrum and have been successfully used in many remote sensing applications for distinguishing land cover types. One of the best known from the field of remote sensing is the Normalized Vegetation Index (NDVI) [START_REF] Rouse | Monitoring the Vernal Advancement and Retrogradation (Green Wave Effect) of Natural Vegetation [Great Plains Corridor[END_REF], which combines only a few broad bands of the NIR and RED spectrum and effectively classifies green areas by combining them. Other examples for indices are the Normalized Differential Water Index (NDWI), which is sensitive to the change in the water content of leaves [START_REF] Gao | NDWI-A normalized difference water index for remote sensing of vegetation liquid water from space[END_REF] and the Normalized Difference Built-up Index (NDBI) [START_REF] Zha | Use of normalized difference built-up index in automatically mapping urban areas from TM imagery[END_REF] for detecting urban areas [START_REF] Qian | A new index to differentiate tree and grass based on high resolution image and object-based methods[END_REF].

Indices originally were based on multispectral imaging and use only a limited number of broad spectral bands to obtain separations for example between areas with vegetation or built-up areas. A more detailed separation (e.g. the separation of different vegetation types) is not possible due to the reduced number of spectral broadband wavelengths. HSI, on the other hand, offers the enormous advantage of a high-resolution spectral component and the potential to use it for a more detailed classification of objects. This makes it possible to visualise typical small spectral effects of an object, whereas this is not possible with low spectral resolution.

The use of broad bandwidths as is the case with the use of indices therefore contradicts the original use of HSI [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

One more classification type is based on the establishment of rules. The assignment of individual pixels to a class is done using various "IF-THEN" expressions. One of the advantages of this approach is the simple interpretability of the established rules. The performance, the ability to deal with redundant and irrelevant attributes and the flexible extensibility of rule sets are further advantages [START_REF] Tan | Introduction to data mining[END_REF].

The typical expression of an IF-THEN rule is of the form left-hand-side (LHS)  righthand-side (RHS), where LHS (also called rule antecedent or pre-condition) is a set of conditions that must be satisfied to derive a conclusion represented by RHS (also called rule consequent). The LHS usually describes a combination of several individual conditions that combine relations by logical operators (e.g. AND, OR), while the LHS stands for a class value [START_REF] Al-Masri | Survey of Rule-Based Systems[END_REF][START_REF] Tan | Introduction to data mining[END_REF][START_REF] Tung | Rule-based Classification[END_REF]. An example of a general IF-THEN rule consisting of 2 conditions is given in Equation 3.1.

( ) ( ) → ( ) (3.1)
The existence of expert knowledge is necessary for the establishment of rules. By analysing the data basis in detail and acquiring knowledge, the understanding for the data will be improved. This is not only beneficial for problem solving, but also allows to structure and simplify existing problems [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]. Studies from the field of remote sensing that take advantage of knowledge can be found in [START_REF] Andrés | Ontology-based classification of remote sensing images using spectral rules[END_REF][START_REF] Berhane | Decision-Tree, Rule-Based, and Random Forest Classification of High-Resolution Multispectral Imagery for Wetland Mapping and Inventory[END_REF][START_REF] Cui | Knowledge and Geo-Object Based Graph Convolutional Network for Remote Sensing Semantic Segmentation[END_REF][START_REF] Ghazaryan | A rule-based approach for crop identification using multi-temporal and multi-sensor phenological metrics[END_REF][START_REF] Houhoulis | Detecting wetland change: A rule-based approach using NWI and SPOT-XS data[END_REF][START_REF] Liu | Multitask Deep Learning With Spectral Knowledge for Hyperspectral Image Classification[END_REF].

In comparison to machine learning methods, rule formation does not require an extensive annotated data basis in order to train the model. As mentioned before, rule-based approaches rely much more on the acquired knowledge of experts. This requires a fundamental understanding of the problem and the underlying data. Such a process of knowledge acquisition can seem time consuming and effortful, but it is a clear advantage when considering the benefits of using expert knowledge to structure and simplify problems. This can be confirmed, among other things, by a study of [START_REF] Ponciano | Object Semantic Segmentation in Point Clouds-Comparison of a Deep Learning and a Knowledge-Based Method[END_REF]. One of the aims of the study was a comparison of deep learning and a knowledge-based method. It demonstrates that a rule-based method has the potential to even be better than machine learning-based methods. The same conclusion was reached in the study by [START_REF] Qian | A new index to differentiate tree and grass based on high resolution image and object-based methods[END_REF]. A tree-grass differentiation index (TGDI) was developed and the performance was evaluated using a rule-based approach and a machine learning-based (SVM) approach. The results obtained show that the rule-based approach leads to more satisfactory results.

An additional enormous advantage is the fact that conventional machine learning based methods only consider the spectral intensity or reflectance values of HSI datasets for the model building. In a rule-based approach, there are no restrictions on the use of characteristics and relations. If we take a spectral signature, in addition to the intensity or reflectance values, relationships between individual bands or shape-describing features can also be included in rule formation and this ultimately leads to a more accurate description of the spectra.

CHAPTER 4: SPECTRAL SIGNATURE ANALYSIS

Spectral signatures are a representation of reflectance values in a certain range of the electromagnetic spectrum. Each material has its own unique spectral signature that can be used to uniquely classify materials or objects. The higher the underlying spectral resolution, the more classification information can be derived from the spectral signatures [START_REF] Khan | Modern Trends in Hyperspectral Image Analysis: A Review[END_REF].

HSI provides base rich in information through the high-resolution spectral component.

The processing of this valuable data base is not always straightforward due to the high dimensionality and requires that alternative approaches should be considered in addition to the methods mentioned in chapter 3.

An alternative approach is to use spectral characteristics that allow a description of the geometric shape of spectral signatures. This chapter therefore deals with the proposed shapebased approach for analysing spectral signatures in combination with a rule-based classification approach. The performance and flexibility are demonstrated using practical examples from two use cases mentioned earlier. In the first use case from the field of waste sorting, the approach is used to separate different types of plastics, while in the second use case from the field of cultural heritage, the focus is on plants and the separation of different species.

SHAPE-BASED ANALYSIS

There are many studies that use a classical approach consisting of dimension reduction (e.g. features extraction and features selection methods) followed by machine learning based classification. Studies that use the advantages of the geometry of spectral curves, on the other hand, are not very common when dealing with hyperspectral data. The advantages of a geometry or shape-based approach are obvious. Depending on the object or material, spectral signatures have a specific geometric curve and express the spectral reflectivity of this object/material through this curve.

Reasons for different spectral signatures are a combination of molecules in materials and the morphological structure of them. Differences in molecules lead to differences in spectral signatures. In turn, the morphological structure is important due to the resulting light path created by reflection, absorption, transmission and deflection by the various elements of an object. Particularly with regard to high-resolution spectral signatures, local or regional changes can be detected due to the high information content and used for the identification of materials through geometric characteristics.

In this thesis, the focus is on characteristics such as curvature points, curvature values, curvature behaviour and spectral reflectance values, that clearly describe the shape of the spectral signature. Spectral signatures do not only differ with regard to different materials or objects, but can also vary for one and the same object or material. The reason for this, is the fact that spectra of one pixel are usually a mixture of a collection of constituent spectra, also called endmembers. Such mixtures occur for a variety of reasons, including the limited spatial resolution of the sensor, the intrinsic variability of the materials and atmospheric scattering effects [START_REF] Scheunders | Contributions of Machine Learning to Remote Sensing Data Analysis[END_REF]. The technique for extracting endmembers and estimating their abundance for each pixel is called spectral unmixing and is for example discussed in [START_REF] Borsoi | Spectral Variability in Hyperspectral Data Unmixing: A Comprehensive Review[END_REF][START_REF] Drumetz | Variability of the endmembers in spectral unmixing[END_REF][START_REF] Keshava | Spectral unmixing[END_REF].

As an example, from the cultural heritage use case, a plant consists of different elements (e.g. molecules, pigments, cell structure, water content) and the influence of each of these elements has a proportional effect on the reflectivity. Changes in the individual elements, such as the cell structure due to ageing of the plant or changes in the pigmentation due to stronger solar radiation, also lead to changes in the geometric course of the spectral signature of the plant. Consequently, different spectral signatures are obtained for one and the same plant. Another example of mixed spectra are shadow areas. The shadow can also be considered as an endmember whose influence must be considered when processing the underlying data. Ideally, it should be eliminated in advance. Even though the hyperspectral sensors used have high spatial resolutions, mixed spectra can still be occurred at the edge between two materials. In the waste sorting use case for example, there may be a mixture of two or more materials at the pixel level, which complicates the classification task.

Even though spectral unmixing is not part of this thesis, depending on the use cases discussed, it becomes obvious that we have to deal with different types of spectrally mixed pixels and that we have to take them into account when analysing the spectral signatures.

Pre-Processing

The selection of the pre-processing steps should be chosen depending on what is to be achieved as a result. In our case, we want to use curvature elements to describe the shape of spectral signatures. Pre-processing should therefore aim to emphasise the shape of the spectral signature and to be robust against spectral shifts. Before doing so, it is necessary to eliminate disturbing effects. This means that the pre-processing steps includes smoothing to eliminate existing random noise in the data by using the Savitzky-Golay method [START_REF] Savitzky | Smoothing and Differentiation of Data by Simplified Least Squares Procedures[END_REF] followed by the Continuum Removal [START_REF] Clark | Imaging spectroscopy: Earth and planetary remote sensing with the USGS Tetracorder and expert systems[END_REF][START_REF] Clark | Reflectance spectroscopy: Quantitative analysis techniques for remote sensing applications[END_REF]. Compared to other methods of smoothing (e.g. Moving Average linear filter [START_REF] Smith | The scientist and engineer's guide to digital signal processing[END_REF] or

Gaussian kernel filter (Wenqian [START_REF] Dong | Hyperspectral pansharpening based on guided filter and Gaussian filter[END_REF]), Savitzky-Golay smoothing is not so effective at reducing noise, but is much more effective at keeping the shape of the original signal (Y. [START_REF] Liu | Applications of Savitzky-Golay Filter for Seismic Random Noise Reduction[END_REF], which is preferable in regard to the shape-based approach proposed in this thesis.

Figure 4.1. Comparison of different smoothing grades using Savitzky-Golay filter on continuum removed spectra. An increase in the smoothing grade increases the noise reduction, but also the possibility that the signal will be distorted by the smoothing.

The Continuum Removal process is used to highlight the geometric shape of the spectral signature, because after removing the concave overall shape of the spectral signatures, a better quantification of the highlighted absorption peaks is achieved. With respect to the proposed shape-based approach, this is also particularly advantageous as it makes the geometric shape more distinct and rules based on curvature features can be developed much more efficiently.

An example of continuum removed spectra and the influence of the strength of the smoothing is shown in Figure 4.1. Depicted are continuum removed spectra for the polymer PP and 3 different grades of smoothing. The choice of the degree of smoothing plays an essential part in this process, since the larger the window length, the greater the noise reduction, but the higher is the chance that the signal will become distorted by the smoothing. This can cause a loss of fine changes in shape that may be helpful in separating objects. For spectra that vary greatly, subtle changes do not need to be captured, but for spectra that are very similar, these subtle differences can be the key to success.

It should be noted at this point that there is no optimum value for the degree of filtration.

The strength of the filtering only affects the level of detail of the spectral signature. It is only important that there is enough detail to be able to apply rules for separation between different spectra. It is also important that the spectra used for classification have the same degree of preprocessing as the spectra used for rule formation. As an example, consider the spectra shown in Figure 4.1. If the slightly smoothed spectral signature shown in green (almost same as original blue spectra) was used for rule formation and applied to spectra that were filtered more strongly (e.g. red and magenta spectra), the rule would not be effective because a change in shape occurs in the range at 1200 nm. So as long as the pre-processing of the spectra for rule formation is identical to the pre-processing of the spectra to be classified, the approach proposed here is robust with respect to the choice of parameters regarding the pre-processing. Dealing with spectral shifts is also an important issue that can be a problem, especially for objects with different surface conditions (e.g. dirty areas or areas with shadow). This effect is illustrated in As can be seen, the strong spectral variations of the sample with the inhomogeneous surface can also be eliminated by the 2 nd derivative. Dealing with these variations is important because the proposed methodology uses the 2 nd derivative to determine the significant bands.

Curvature Features

The approach proposed in this thesis is to use shape characteristics to describe the unambiguously course of spectral signatures. As mentioned above, the course of spectral signatures varies due to changes in the material composition of objects. These changes, local or regional, inevitably lead to changes in curvature behaviour, with the result that the curvature κ becomes an important parameter for the description of the shape and the modelling of spectral signature changes.

Seen mathematically, the curvature κ is the change in a curve that occurs when the curve is traversed and can be expressed in parametric form for each point ( ), ( ) using Equation 4.1 [START_REF] Simmons | Calculus with analytic geometry[END_REF]. The points on the variables and stand for the 1 st and 2 nd order derivatives.

= | ̇ ̈-̇ | ( ̇ + ̇ ) (4.1)
To better illustrate the curvature, Figure 4.3 shows a part of the spectral signature of polystyrene with curves and straight lines. Since there is no curvature in a straight line, the curvature at these points is zero. In all other areas, the curve changes from point to point and indicates how strongly the curve at a point deviates from a straight line, which can be visualised by circles. The size of the circle is computed in regards to its fitting to the curve at a point (i.e. to maximise the size of the share border). A small circle has a greater curvature than a large circle, because a small circle deviates stronger from a straight line than a large circle. The curvature values for each point of the spectral signature determined using Equation 4.1 and represented by green lines, where the length of the lines is the measure of the strength of the curvature. And in addition, the curve behaviour, which can be derived from the direction of the curvature values and is an important source of information for describing the shape because it helps to distinguish between concave and convex curve behaviour. The red lines represent the significant band positions with a curvature value that is above a threshold value defined in advance and should be set in dependence of the underlying spectra. These red lines are the basis for the following rule formation process.

RULE-BASED CLASSIFICATION

The advantages of rule-based approaches were discussed in chapter 3.2. The geometric shape of spectral signatures only changes when the internal composition of the object or material changes. In this respect, the characteristics like the strength, the direction and the band position are stable elements that allow objects/materials to be classified even under different conditions. The robustness of the method proposed will be demonstrated in the experiments presented in section 4.3. Based on these characteristics rules for classification are established.

For this, only the significant bands (red lines) are included in terms of their characteristics, since these reflect both the concave and convex behaviour and a significant change in curvature. To minimise the complexity of rule formation and keep it as simple as possible, all other bands (green lines) are not included in the rule formation.

As already mentioned, the selection of the significant bands is made by setting a threshold, which mainly depends on the curve shape. This threshold is not a sensitive value that would influence the success of the methodology. Rather, it serves to identify significant bands for rule formation in a simple way. Even if the threshold is set too low and the number of significant bands is increased as a result, this does not pose a problem due to the prior analysis of the data by an expert before rule formation.

In the example shown in Figure 4.4 the used threshold is 0.1. A smaller threshold value would increase the number of significant bands and thus allow a more detailed description of the curve. However, this is not always absolutely necessary. For strongly varying spectra, it is therefore absolutely sufficient to keep the number of significant bands low. For a better understanding, Figure 4.5 shows the spectral signatures of two different types of plastics, namely polystyrene (PS) and polyethylene (PE). It can be seen that several significant bands were identified based on the local extrema of the 2 nd derivative (red and blue dots), but only the bands with a high curvature value were selected, as these are sufficient to describe the geometric shape of both spectra in order to make a separation between the different plastic types.

→

Based on these significant bands and their characteristics, rules consisting of logically related conditions can now be formulated. Referring again to the spectra shown in Figure 4.5, the rules listed in Table 4.1 can be established for PE and PS. It should be noted that even if a large number of significant bands are available to describe spectra, it is not necessary to include all significant bands (red lines) in the form of conditions in the rule formation. It is more important to analyse the database in advance and to use the knowledge gained to establish rules that contribute to a clean classification even with a small number of conditions.

Each of the rules includes several conditions that are linked by a logical AND operator.

Each individual condition represents one of the significant bands and each of these conditions must be fulfilled for an assignment to the corresponding class. Of course, it is possible that for some materials few of these conditions are satisfied by different materials, but never all of them. It is therefore important to define conditions for significant bands covering the entire spectral signature. The robustness and global applicability of the rules should always have first priority. For example, if the rule formation also considers particularly small curvatures, it could be that this small curvature is not detected in a measurement under different conditions due to weak light conditions. As a consequence, the rule would not apply. It is therefore all the more important to ensure that the rules are unambiguous and insensitive to external influences.

In order to achieve this, characteristics must be selected for the rule formation that are stable and does not change in the variation of the data. The proposed processing and a subsequent analysis of the spectra allow the identification of robust, i.e. unique and stable, features that can be used for rule formulation. An overview of the continuum removed spectra for all 9 objects is shown in Figure 4.7. The strength of the Curvature Value (CV) is expressed by the parameter CV at the location of the corresponding significant band. The sign in front of the threshold value indicates the direction of the curvature value and is used to distinguish between convex and concave curve behaviour. In the case of a negative threshold value, the downward red lines (concave behaviour) are detected, while all conditions with a positive threshold value detect the upward red lines (convex behaviour). Depending on the curvature behaviour, this must be greater or less than the specified threshold value of 0.1. The number of conditions depends essentially on the data basis. Strategically, it should be clear which materials are involved in the classification process. Prior knowledge of the target structures is crucial for the success of the classification. The determination of the spectral characteristics of the involved materials/objects is the next step, followed by the actual analysis of these characteristics. It is important to recognise common characteristics and to take them into account in an efficient way in the rule formation. If the characteristics of the spectral signatures are unique and different, rules can be established based on an arbitrary number of characteristics. If the characteristics of the spectral signatures are identical, hierarchical processing steps can be incorporated. Take, for example, plants whose water content is identical to a certain extent, but the pigments or cell structure are different. In such a case, a first division of the plants on the basis of their water content can be made by means of a hierarchical approach and then further differentiated within the groups.

This procedure from coarse to fine increases the performance and can lead to increased generalisability.

In addition to the form-describing characteristics used here, other conditions can also be integrated in the rule formation. These can be e. One of the essential conditions that should be considered when creating rules is complexity. The number of logically linked conditions of a rule should be kept within limits in order to reduce complexity and to formulate rules in a comprehensible way. This can be done in two ways. A low threshold value for the selection of significant bands (red lines) can be set.

This causes a high number of significant bands to appear, because even not to distinct curvatures will be considered. The choice of bands for rule formation must be made through a careful analysis of the spectral signatures of all materials present and on significant differences between the individual spectral signatures. This has the advantage of making the rules more robust and avoiding unnecessarily complex rules. The other way could be of interest in the context of automatic rule formation and would be based on a high threshold. A high threshold would mean that only the most pronounced curvatures of the signatures would be selected and thus a small number of the significant bands would already be available in advance. These significant bands could be used as the basis for an automated approach. However, this approach only makes sense if the spectral signatures of the materials differ strongly in their shape, so that only the significant curvatures are actually sufficient for separation.

EXPERIMENTAL RESULTS

This subsection covers all practical investigations regarding the use cases from the fields of waste sorting and cultural heritage. The evaluation of the shape-based characteristics in combination with rule-based classification is carried out on datasets from the fields of waste sorting and cultural heritage. Additional practical investigations for a use case in the forensic field consist of investigations regarding the detectability of hidden blood traces. The findings of these investigations are documented in 0, as they are based only on visual examinations and the methodologies proposed in this thesis were not applied.

Waste Sorting Use Case

Data Set

Based on the proposed approach, shape-based rules for the separation of different plas- The first dataset shows an image of seven classes of different plastic specimens and has a resolution of 661 x 500 pixels and 229 bands with a spectral range from 900 nm to 1700 nm.

The second dataset consists of plastic parts and circuit boards from shredded electronic waste and has a spatial resolution of 1253 x 578 pixels and 229 bands. Due to the setup of the used Specim FX17, the spatial resolution of one pixel is 0.36 mm. Naturally, the number of pixels per object varies depending on the object size. To give an example, the plastic samples in the first dataset consist on average of about 4800 pixels. The rule formation for the different plastic types are based on the radiometrically normalised reflectance spectra shown in Figure 4.9. The continuum removed spectra for each plastic type and the rules derived from them are given in Appendix A.

Figure 4.9. Mean reflection spectra of parts of the available plastics used for rule formation. Each plastic category has a specific spectral response. This specific shape is also reflected in the shapes of the continuum-removed reflectance spectra. This means that the rule (based on a combination of conditions) defined for a category is specific to that category and there is no risk of confusion because the shapes are distinct enough. Considering all materials which need to be classified in the formation of rules has the advantage that rules can be set up more efficiently and conditions based on significant bands can be kept low. [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

Results and Discussion

Both HSI datasets were processed pixel by pixel using the rules listed in Appendix A.

The two datasets differ fundamentally in the surface properties of the samples. While the objects shown in Figure 4.8 (a) are homogeneous and clean, the real waste samples shown in the spectral signature and to high variability within identical material compositions, as besides the material there are additional spectral endmembers such as shadows or dirt. The influence of this effect and the use of the 2 nd derivative to overcome this issue is discussed in section 4.1.1. The results described in the following confirm that the procedure we have chosen is robust to the spectral shifts mentioned. First, the classification map for dataset of plastic spec-imen is shown in Figure 4.10. Basically, it can be seen that all samples were classified correctly. However, a closer look especially at the borders reveals a few problem areas. These problem areas mainly include the occurrence of shadow effects. Some of the samples (e.g. PS)

do not lie planar on the underlying ground, instead they have a light bend, which in turn leads to the aforementioned cast shadows. The presence of shadows is generally a problem in visionbased applications and leads to changes in the radiometric information in the affected areas [START_REF] Mostafa | A Review on Various Shadow Detection and Compensation Techniques in Remote Sensing Images[END_REF][START_REF] Shahtahmassebi | Review of shadow detection and de-shadowing methods in remote sensing[END_REF]. This effect influences processes such as classifications and leads to the misclassified pixels at the borders of the samples (shadow area in background assigned as sample material) that can be seen in the classification map in Figure 4.10.

Figure 4.10. Classification result using shape-based rules [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

Another point concerns the PF samples present in this dataset. Looking at the spectrum for PF shown in Appendix A, Figure A3, it is apparent that the sensor did not provide useful spectral information for this sample due to a lack of reflectivity. This is caused by the fact that the PF samples used here are composed of black colourants and due to the colouring, there is no reflection on the surface of the samples. This is a well-known problem when dealing with spectral measurement systems. Black pigmented samples absorb the light shining on the surface from the UV to the NIR. No reflection means there is no light that can be detected by the sensor, and thus no spectral information that can be used for classification [START_REF] Dvorak | Development of NIR Detectable Black Plastic Packaging[END_REF][START_REF] Rozenstein | Development of a new approach based on midwave infrared spectroscopy for post-consumer black plastic waste sorting in the recycling industry[END_REF]. Nevertheless, it is evident from the classification map that the PF samples were classified correctly. This is due to the flexibility of rule-based approaches. Even if there is only limited reflection, a classification can still be made possible by setting up appropriate rules. With the PF-Black rule set up here, however, not only black PF samples would be detected, but also all black plastics within a dataset. That is why this rule has to be seen as exclusively true for this example and cannot be generalised without further investigations.

Figure 4.11. Classification result for real waste samples consisting of plastics and printed circuit boards using shape-based rules [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]).

If we now look at the classification map of the dataset consisting of real waste given in In order to be able to make some statistical statements about the classification accuracy in addition to the visual assessment and to compare the shape-based approach presented in this thesis with standard classification methods in terms of performance, a benchmark with stateof-the-art strategies is needed. The reference data required for this was generated through a manual labelling of both datasets using the MATLAB Image Labeler application.

Among the most popular and common classifiers, a machine learning based supervised method called SVM [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF][START_REF] El Bouchefry | Learning in Big Data: Introduction to Machine Learning[END_REF] was used for the comparison. The software fluxTrainer by LuxFlux GmbH 7 supports state-of-the-art solutions for the processing of hyperspectral datasets and provides also an SVM implementation equivalent to other state-of-the-art implementations. The quality of the results of a SVM classifier depends mainly on the parameterisation and the underlying training data (Ghamisi, Plaza, et al., 2017). Since the software used does not support functions for determining optimal parameter sets, various parameterisations using trial-and-error tuning were used as a basis for the calculations and optimised accordingly on the basis of visual assessment of the resulting classification maps. Basically, the manual search for optimal hyperparameters is part of the state-ofthe-art and can be associated with an increased effort of trial-and-error. However, this procedure can also be advantageous, as the ability of the expert to understand the mathematical process of the procedure and the influence of the parameters on the methodology can significantly improve the search. The training datasets were created by manual selection of corresponding areas of the samples for each class. In the case of the plastic specimen dataset, one material sample of each material class was manually labelled and used as a training set. In the case of the real waste data base, a different strategy has to be followed due to the lack of homogeneity. For each class, small subsets of each sample were manually selected, with attention paid to particularly 7 www.luxflux.de critical areas such as depth-related shadow areas, dirty areas and damaged surfaces. An overview of the classes and the amount of underlying training data is listed in Table 4.2.

The assessment of the classification is carried out visually on the basis of the classification maps and on the basis of the derived metrics using the confusion matrix estimated from the ground truth classification map and the predicted classification map [START_REF] Al-Jabery | Data analysis and machine learning tools in MATLAB and Python[END_REF][START_REF] Grandini | Metrics for Multi-Class Classification: an Overview[END_REF][START_REF] Ting | Confusion Matrix[END_REF]. The predicted classification maps for the plastic specimen dataset using different kernels and classification thresholds are shown in Figure 4.15 and for the real waste dataset in Figure 4.16. The corresponding statistical values can be found in Table 4.3 (plastic specimen samples) and Table 4.4 (real waste samples). (e) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: RBF; C-Value: 0.2; Threshold: 0.3. (f) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: RBF; C-Value: 0.2; Threshold: 0.4. (g) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: RBF; C-Value: 0.2; Threshold: 0.5. (h) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: RBF; C-Value: 0.2; Threshold: 0.6.

Considering the statistics for both datasets, it can be stated that in principle a high degree of Overall Accuracy (OA) can be achieved for both methods and that the results are comparable. However, a closer look at the other metrics like F1-Score or Kappa shows that the rulebased approach performs slightly better in comparison to the SVM classifier. This can also be seen in the classification maps. In both datasets, the shape-based results appear more homogeneous and the handling of pixels at the object borders appears more robust. Especially in the dataset with the plastic specimen samples and the apparent shadow regions at the borders, misclassifications of the border pixels occur with the SVM method, while the rule-based approach also classifies these edge pixels and some shadow pixels as corresponding material. A more detailed view of these problem areas can be seen in Figure 4.13. Despite the fact that the real waste dataset is much more challenging due to the factors already mentioned, such as dirt, depth changes and damaged surfaces, the statistics are even better. Especially when comparing the classification maps, the better performance of the rulebased approach can be seen with regard to the high variability of the underlying spectral signatures. The different sets of parameters of the SVM classifier show that a compromise has to be found between misclassified pixels at the edges of the objects and unclassified pixels on the objects. As an example, in Figure 4.14 (b), object pixels are relatively well classified.

However, misclassifications at the edges are also present. But trying to eliminate these misclassifications at the edges also leads to the loss of object pixels like seen in Figure 4.14 (f).

This effect is mainly due to the problem of the high variability of the spectra. The shape-based approach is better able to handle these variations, as the shape of the spectra is always the same despite the variation in reflectivity. Only the degree of reflectivity changes, not the shape. This means that theoretically only one spectrum per material is necessary for rule formation. With the SVM method, on the other hand, it would have to be ensured that the dataset with which the model is trained covers all spectral variations. Fulfilling this condition, could be a very challenging and time-consuming task. We can take the real waste dataset. Due to the already mentioned challenges regarding dirt, shadows etc. we have a very inhomogeneous dataset. In order to estimate a reliable model, the training dataset must therefore take all these inhomogeneities into account. For this purpose, as already mentioned, different areas of each object were integrated into the training process. Additional objects, however, mean further variations and need further training, especially when it concerns the processing of waste.

Cultural Heritage Use Case

Data Set

The approach proposed in this thesis, wants to be of general value and applicability and has already shown convincing results for the waste sorting use case. As complete different and challenging objects it will now be applied to plants. For this purpose, various plants growing on a section of the wall of the Balduinseck castle ruins and shown in Figure 4.17 were examined. In order to be able to use a broad electromagnetic spectrum for the classification, the plants were acquired under laboratory conditions using the measurement system shown in Fig-

ure 2.3. Figure 4.18 shows the plants found on the wall section and thus also the data basis.

Both hyperspectral camera systems, the Specim FX10 and the Specim FX17 were used for the acquisition. The data basis for the proposed approach therefore consists of a combined HSI with a resolution of 1220 x 640 pixels and a total of 462 bands covering the spectral range from 400 nm to 1700 nm. ). The discontinuities in the range of 900 nm and the greater noise in the VIS range can be explained by the differences in spectral sensitivity of the cameras. In particular, the range below 500 nm exhibits strong noise effects and was not considered in the rule formation. Taraxacum (1), inula conyza (2), campanula rotundifolia (3), moss (4), geranium robertianum ( 5), asplenium trichomanes ( 6), green sedum ( 7), red sedum (8), moss (9), lichen (10) [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

Results and Discussion

Classifying different plant species is very challenging due to the very similar shape of spectral signatures of plants as illustrated in These spectral characteristics were used to establish rules for separating the plant species. A documentation of the rules and their order of application (Figure B11), as well as the Continuum Removed Reflectance Spectra of the involved plants can be found in Appendix B.

Due to the very similar geometric course of the spectral signatures, a simplification of the dataset was carried out in a first step by using shape-describing characteristics. This was done considering the shape differences above 950 nm to divide the plant species into two groups.

The strategy followed here is of hierarchical nature and is of interest in the context of the general applicability of the methodology, as the use of established rules achieves a general separation of the plants into two categories. The 950 nm was specifically chosen because studies have shown the existence of 4 water absorption bands. These are at 970 nm, 1200 nm, 1400nm, and 1900 nm [START_REF] Pu | Hyperspectral remote sensing: Fundamentals and practices[END_REF][START_REF] Ustin | How the Optical Properties of Leaves Modify the Absorption and Scattering of Energy and Enhance Leaf Functionality[END_REF][START_REF] Zhao | Application of Hyperspectral Imaging for Assessment of Tomato Leaf Water Status in Plant Factories[END_REF]. By using the spectral signature above 950 nm, the separation into two groups based on the cell structure and water content of the plants can be performed. The result of this simplifying step is presented in Figure 4.20 and shows the distinguished plants based on the similarities in cell structure.

For the further separation into subcategories, Continuum Removed Reflectance Values (CRRV) were used in addition to shape-describing characteristics like the curvature. The rules established allow to subdivide the two groups into further subcategories, namely moss, lichen, red sedum, green sedum and two further groups, namely green leaves and a group consisting of geranium robertianum and campanula rotundifolia. Green sedum, moss and red sedum fall into the group with higher water content (stronger absorption in the bands at 970 nm and 1200 nm). The other plants, i.e. geranium robertianum, lichen, taraxacum, inula conyza, campanula rotundifolia and asplenium trichomanes assigned to the second group with lower absorption at the mentioned water absorption bands. A more detailed categorisation of green-leaf plants, such as the distinction between taraxacum and inula conyza, has its limits due to the similar spectral characteristics. Nevertheless, the classification map in Figure 4.21

shows that a prior analysis of the spectra makes it possible to divide them into subcategories.

In contrast to the waste sorting use case where only the NIR spectra were considered, in this particular case the VIS range provides additional useful information that can be used for the classification. We see, for example, that moss can be clearly distinguished from green leaves due to the cell structure and the lower reflectivity around 550 nm. Red sedum can also be clearly distinguished from other plants due to its red colouring. Lichens have a similar cell structure to green-leaf plants, but differ in pigmentation. These differences were used in the rule formation process to identify the lichens in object number 10. To assess the classification accuracy, also this dataset was annotated using the MATLAB Image Labeler application and metrics were obtained using the confusion matrix. If we look at the statistics in Table 4.5, we see that the statistical (OA: 92.74 %) results

for this use case are not as good as the results from the waste sorting use case and that in contrast to our shape-based method better results are achieved with the SVM classifier. This is due to the very similar geometric spectral conditions and the fact that the use of shapedescribing elements for rule formation is limited. Nevertheless, the potential of the rule-based approach to provide useful results is still apparent and depends mainly on the previous analytical efforts. The rules established in this context are simple and based on obvious characteristics. Better results could be achieved if more complex rules were established.

Furthermore, it must also be taken into account that the spectral properties of plants strongly vary, also within one plant species [START_REF] Klančnik | Leaf spectral signatures differ in plant species colonizing habitats along a hydrological gradient[END_REF]. An example for such a variation is given in Figure 4.23 for moss.

Figure 4.23. Spectral variations in VIS for one type of moss regarding all pixels in the red box. Main reasons for this strong variations are for example optical properties of leaves and the direction of the leaf to the light beam [START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

In contrast to the waste sorting use case and the variations depending on shadow and dirt, the spectral variations here are not only due to a spectral shift, but to a mixing of spectral signatures due to the intrinsic variability of the plant. The diversity of plant characteristics is an expression of the different adaptation strategies of the individual plant species [START_REF] Serbin | Remotely estimating photosynthetic capacity, and its response to temperature, in vegetation canopies using imaging spectroscopy[END_REF]. As already mentioned, the spectral signature of a plant is mainly influenced by their pigments, their internal cell structure and nutrient content [START_REF] Carvalho Lopes | Classification and Authentication of Plants by Chemometric Analysis of Spectral Data[END_REF]. Environmental influences, stress or even the ageing of the plants lead to changes in these factors and inevitably also to changes in the spectral signature. Healthy plants, for example, have a high chlorophyll concentration. A stress-related undersupply (lack of chlorophyll) becomes visible in the form of a yellowing of the leaf. This inevitably leads to a change of the spectral signature in the VIS range (Yun Chen et al., 2019). Also related to colouring are changes in the seasons. While leaves are green in summer, they change colour as they move into autumn. The cell structure of a plant changes, for example, with the phenological ageing process. The optical properties of leaves, the direction of the leaf to the light beam, all these are also factors that influence the spectral signature [START_REF] Ustin | How the Optical Properties of Leaves Modify the Absorption and Scattering of Energy and Enhance Leaf Functionality[END_REF]. Consequently, all these factors complicate the classification process and do not really allow a generalisation of rules only based on spectral signatures. However, the application of rules in the context of a hierarchical strategy has the potential to overcome these challenges. By analysing common elements in different plant spectra and applying appropriate rules, a rough separation into groups can be made, as shown in our example. These groups serve as a basis for further refinement and help to simplify and manage complex structures. A further possibility to deal with such heterogeneous spectra, could be the addition of rules based on texture features or other local image features [START_REF] Beghin | Shape and Texture Based Plant Leaf Classification. Advanced Concepts for Intelligent Vision Systems[END_REF][START_REF] Salve | Multimodal plant recognition through hybrid feature fusion technique using imaging and non-imaging hyper-spectral data[END_REF][START_REF] Siravenha | Plant Classification from Leaf Textures[END_REF]/2016).

DISCUSSION

The first main contribution of this thesis was addressed within this chapter. The concept, potential and effectiveness of a new shape-based approach for the extraction of spectral characteristics contained in HSI data in combination with a rule-based classification was demonstrated by experimental results using completely different datasets from the fields of cultural heritage and waste sorting. Based on previous analysis of the spectral signatures of materials/objects of interest, form-descriptive characteristics such as curvature points, curvature values and curvature behaviour were selected and used to develop rules in a supervised way on the basis of available knowledge. The major advantage of this method is the ability to automatically derive these mentioned characteristics in a simple way, as they are basic mathematical processes relying on the 1 st and 2 nd derivative of spectral signatures. The results from the two use cases, especially in the field of waste sorting, are significant and confirm the effectiveness of this method. Even though in this thesis the rules were created manually on the basis of automatically extracted characteristics and expert knowledge, a completely automated rule formation process is also possible. The selection of the shape characteristics mainly depends on the strength of the curvatures of the spectra and how high or low the threshold value is set to obtain the significant bands. Based on this, significant shape-describing bands are determined and used for the rule formation. The advantage of a manually operated procedure based on knowledge is that not all of the bands that have been identified as significant have to be considered in the rule formation. Only as many bands as necessary to separate it from other materials/objects have to be used. The aim is thus that the uniqueness of spectra for individual objects are captured by a minimum number of form descriptive feature elements to express correctly the shape of the spectra, but always under the claim of general applicability also to other cases. Meeting this demand through an automated approach has its challenges.

A possible automatic approach would be to express all bands considered significant, based on their position and behaviour (positive or negative sign of the curvature value), in a rule using logical AND operator. In such a process, the choice of parameters is crucial as they influence the number of significant bands. The incorrect choice of values for parameters can lead to a high number of significant bands and consequently to unnecessarily complex rules.

Let us assume that the smoothing parameter is set too low. This leads to many small local curvatures being considered and, depending on the threshold for the curvature value, to a high number of bands being considered significant. In the case of strongly varying spectra, however, it is not necessary to capture every smallest local change in shape. The success of an automated process therefore essentially depends on the parameters that are defined in advance.

A further disadvantage that can arise with the automatic formulation of rules is that the number of rules increases with the number of categories. This not only leads to complex rules, but can also lead to rules being in conflict with each other. Modelling the rules based on expert knowledge and the use of hierarchical strategies can avoid this, as the interaction of the different factors is well understood, such that the simplest concept that gives the best results can be formulated and complexity can be reduced. Nevertheless, the possibility of automation is not excluded, as it eliminates the previous step of data analysis. However, this is only recommended if the number of categories is manageable.

Classical approaches based on machine learning also do not require a prior analysis of the data basis, but are based on a large number of training datasets that are used to estimate the model parameters. However, the collection of training datasets which contain all relevant influences and are representative for the complete dataset is not always simple and can be time-consuming (e.g. for annotation). Furthermore, due to the high dimensions of a hyperspectral dataset, a dimension reduction is mandatory. This is usually based on purely statistical procedures and aims to reduce the dataset to a number of significant bands. The disadvantage of these dimension reduction is often the loss of information related to the true physical form of the spectra. One more point is that these approaches focus purely on the intensity values, while global or even local neighbourhood relationships are completely or partially disregarded.

With the proposed approach presented here, the physical shape of spectra can be represented on the basis of a few true significant bands. The smallest changes in the curve can be identified and modelled. Due to the flexible approach of rule formation, further characteristics can be added to the shape-describing characteristics mentioned so far, such as relations between significant bands or band ranges (e.g. as with indices), intensities or local or global relationships.

However, this requires a prior analysis of the spectral signatures and the establishment of rules based on expert knowledge. This process can be seen as time-consuming, but is preferable in view of the advantages.

Analysing the spectral signatures in advance has the further advantage that the full spectrum does not have to be used for classification. In case significant shape differences are found within a limited spectral range, a consideration of only this range is sufficient to establish the rules based on it. This can lead to an increase in performance, especially with regard to applications in industrial practice where fast processing is required. But even without the reduction of the spectral range, the performance of the approach presented here is satisfactory. The processing time for the cultural heritage dataset (1220 x 640 pixels and 462 bands) shown in Figure 4.18 on a machine with an Intel® Core TM i7-10700 CPU @ 2.90-GHz and 16 GB RAM using the SVM classifier takes 2856.15 seconds. In contrast, the pixel-by-pixel processing and checking for conditions on the same machine using our proposed method takes 116.30 seconds. Compared to this, the shape-based classification in (Yuanyuan [START_REF] Chen | Remote sensing for vegetation monitoring in carbon capture storage regions: A review[END_REF] takes about 9 minutes for a smaller dataset of 512 x 512 pixels and 6 bands (computer configuration:

CPU 2.93 GHz and installed memory 4.00 GB). A further comparison with an investigation using a classical classification method such as SVM also shows better performance. The processing time for a similar dataset consisting of 1168 x 696 pixels and 520 bands is 2980.21 seconds using 30 bands and 9381.37 seconds using all bands of the dataset (computer configuration: Intel® Core TM i7-6800k 3.40-GHz CPU and 64 GB RAM) (Q. [START_REF] Ren | Partitioned Relief-F Method for Dimensionality Reduction of Hyperspectral Images[END_REF].

The reason why this shape-based approach delivers promising results is obvious. While spectral values can vary strongly depending on different factors (e.g. shadows, dirt, light conditions, etc.) and result in spectral shifts, the characteristic shape elements of spectra only change when there is a composition change in the materials or objects. The typical shape of a spectral signature is always the same for one and the same composition. With the basic idea to use the shape of spectral signatures, we therefore have an analytical method that is robust to spectral variations and allows an unambiguous description of the spectra on the basis of geometric characteristics. Although there is a plethora of studies dealing with the analysis and classification of spectral signatures, to the best of our knowledge there is no comparable analytical method to the one presented here.

In the following, this will be demonstrated by a specific example. To illustrate the added value of a combined 3D and HSI approach, we therefore take the use case from the field of waste sorting, as in this context both spectral and spatial information are used for the classification of objects, which allows us to better illustrate the benefit of such a combined dataset.

The general concept of the proposed method presented in Figure 5.1 is to fuse the datasets, to simplify and structure them and to extract features in the 3D and hyperspectral domains for the subsequent classification process. Even though the focus here is on waste sorting, In the following, we would also like to go into more detail about the advantages and the general role of 3D data. When looking at a group of objects distributed in space, several useful characteristics can be found in 3D. A spatial position, a size, a shape, distances between objects and useful topological information. All these characteristics can be used to simplify the aimed goal of classification by using for processes like background differentiation or obtaining shape parameters. These geometric characteristics are derived by considering the local neighbourhood of each 3D point. Invariant moments representing the geometric characteristics of the local 3D structures can be calculated for each point based on this neighbourhood (M. [START_REF] Weinmann | Feature relevance assessment for the semantic interpretation of 3D point cloud data[END_REF]. The spatial characteristics considered in this thesis are mainly derived based on normal vectors and eigenvalue-based 3D features, which are nowadays commonly used in lidar data processing. For instance, the change in curvature , verticality , linearity , planarity , omnivariance , anisotropy and scattering . The 3D feature set mentioned here consists of geometric features based on normalized eigenvalues ( , , ) of the local structure tensor and the corresponding eigenvectors ( , , ) of the covariance tensor for a given 3D point and its closest neighbours. The mathematical expression for each feature is given in Equations 5.1 to 5.8 and is derived from [START_REF] Hackel | Fast Semantic Segmentation of 3D Point Clouds with Strongly Varying Density[END_REF][START_REF] Weinmann | Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers[END_REF]. A visualisation of the features omnivariance, verticality and normal change rate can be seen in Figure 5. However, due to the flexibility of rule-based approaches, there are no limits and in addition to these local geometric features, other characteristics, like mentioned in (Weihua [START_REF] Dong | Selection of LiDAR geometric features with adaptive neighborhood size for urban land cover classification[END_REF], can also be used in order to consider more properties of the 3D point cloud.

In the study of (M. [START_REF] Weinmann | Geometric Features and their relevance for 3D Point Cloud Classification[END_REF], angular features [START_REF] Munoz | Contextual classification with functional Max-Margin Markov Networks[END_REF], height and plane features [START_REF] Niemeyer | Conditional Random Fields for the Classification of Lidar Point Clouds[END_REF], moments and height features [START_REF] Hackel | Fast Semantic Segmentation of 3D Point Clouds with Strongly Varying Density[END_REF] and specific descriptors involving surface features, slope, height features, vertical profiles and 2D

projections [START_REF] Guo | Classification of airborne laser scanning data using JointBoost[END_REF] are mentioned as possible characteristics for 3D point classification. The choice of 3D characteristics should be made depending on the data basis and adjusted accordingly to what exactly is to be described.

A rule-based approach is again used for the classification process. The basis for the conditions of the rules are the features from the spectral domain (i.e. curvature values, curvature behaviour, etc.), as well as normal vectors and eigenvalue-based features extracted from the 3D dataset. The rules developed in this thesis are based on knowledge established for electronic waste sorting. A use of the procedure for other applications requires an adaptation of the rules to the respective application based on knowledge from the relevant field.

WASTE SORTING

Problem of identifying and separating PCBs

In the area of waste management, in addition to the separation of plastics, the recognition of PCBs is also necessary for efficient recycling in Waste Electrical and Electronic Equipment (WEEE). PCBs are composed of various polymers and precious metals, which makes the PCB itself a very valuable resource [START_REF] Calvini | Growing applications of hyperspectral and multispectral imaging[END_REF]. However, PCBs also pose an enormous challenge due to the various components found in (e.g. copper traces) and on the PCBs (e.g. resistors, capacitors, etc.). Various studies following different strategies have been taken to address this issue and either focus on a specific type of component (e.g. Surface-Mounted Devices (SMD) or Through Hole Components (THC)) or are intended for inspection tasks. For instance, the work of (W. [START_REF] Li | SMD segmentation for automated PCB recycling[END_REF] proposes an approach for the segmentation of SMDs on PCBs. A segmentation based on the assembly paint and colour distribution is used to detect two types of SMDs, such as small devices like resistors and integrated circuits. Also focused on the detection of SMDs is the study by [START_REF] Jessurun | SHADE: Automated Refinement of PCB Component Estimates Using Detected Shadows[END_REF] using the shadow cast of SMDs, which is used to separate SMDs from components that are not of interest. The study of [START_REF] Herchenbach | Segmentation and classification of THCs on PCBAs[END_REF] uses a combination of RGB images and depth images to detect THCs. Also based on depth images of the Microsoft Kinect TM sensor and pixel classification is the study of (D. [START_REF] Li | Semantic Segmentation of a Printed Circuit Board for Component Recognition Based on Depth Images[END_REF] which deals with the detectability of different electronic components with regard to inspection tasks. Inspection tasks are based on data from intact and clean PCBs. A major challenge when it comes to waste management are the deformed, demolished and soiled objects as a result of processing at the landfill. Due to the non-uniform shape of the individual components, an approach based purely on 3D geometry or even 2D shape is not very effective.

Moreover, the studies mentioned so far do not allow the separation of the components in regard to their material composition, as they are based purely on depth information or images. Capacitors mounted on circuit boards, for example, differ not only in their shape but also in their material composition. To be able to separate successfully in such cases, the use of physical information provided by HSI is more goal-oriented.

HSI technologies enable the acquisition of physical information and are also the subject of various studies. Due to technological advances in instrumentation in recent years, they can also be easily integrated into sorting processes [START_REF] Ebayyeh | A Review and Analysis of Automatic Optical Inspection and Quality Monitoring Methods in Electronics Industry[END_REF][START_REF] Khan | Modern Trends in Hyperspectral Image Analysis: A Review[END_REF].

For example, the studies of [START_REF] Ibrahim | Spectral imaging method for material classification and inspection of printed circuit boards[END_REF][START_REF] Palmieri | Recycling-oriented characterization of plastic frames and printed circuit boards from mobile phones by electronic and chemical imaging[END_REF] use spectral information to classify metals on PCBs. Combining HSI with RGB images, can be found in [START_REF] Sudharshan | Object Detection Routine for Material Streams Combining RGB and Hyperspectral Reflectance Data Based on Guided Object Localization[END_REF], it proposes an object detection approach to detect recyclable components on PCBs.

The authors used the spectral features of the components (integrated circuits, gold connectors and electrolytic capacitors) to locate them, while the spatial features given by the RGB image are used to classify the located components. A machine learning based approach is used for the classification. Results showed, however, that not all components on the PCBs are recog-

nised. An explanation given by the authors is that this is due to the very small training dataset available and the parameters chosen for the classification process. This explanation is reasonable and reflects the general problems of machine learning based methods, which have already been discussed in section 3.2. The dataset used in this mentioned study is based on intact PCBs.

With regard to objects from landfill, a machine learning based approach is even more problematic due to the strong variations within the components caused by contamination, demolition and deformation.

Fundamentally, the classification of PCBs and components on them is a difficult task due to the complexity of PCBs both in terms of their structure and the large variety of electronic components with small size and broad chemical composition [START_REF] Palmieri | Recycling-oriented characterization of plastic frames and printed circuit boards from mobile phones by electronic and chemical imaging[END_REF]. For

PCBs from the waste management sector, this task is even more challenging.

Combined Data Basis

Data Set Acquisition and Fusion

The experimental investigations are based on the randomly selected 19 objects shown in Figure 5.3. To be seen are 5 plastic parts and 14 circuit boards. The circuit boards are equipped with various electronic components such as resistors, capacitors, transistors and others. Since the items were shredded when they were processed at the landfill, they are partially damaged and dirty. The components on the circuit boards also do not have a clean geometry and are randomly oriented. Since in this case the relevant information is primarily related to the physical properties of the materials, the objects were spectrally acquired with the Specim FX17 using the measurement system shown in Figure 2.3. The obtained HSI has a spatial resolution of 636 x 1118 pixels and covers 224 bands per pixel in a wavelength range from 900 to 1700 nm. The radiometric normalisation was done using a 99% Spectralon TM white calibration tile.

For the acquisition of the 3D geometry, the structured light scanner Atos Core 500 integrated in the measuring system is used. The resulting unordered point cloud consists of

Mutual reinforcing use of 3D and spectral data

Improvement of data base by 3D information

To simplify the further processing of the data basis, it is advisable to remove irrelevant areas of the dataset. The 3D component is very helpful in this respect and allows the reduction of the dataset in a very simple way. This is not only useful in terms of performance, but also allows focusing on the objects that are of real interest. By using well known 3D processing algorithms such as 3D plane estimation [START_REF] Torr | MLESAC: A New Robust Estimator with Application to Estimating Image Geometry[END_REF], the background, which

does not provide any useful information for the classification process, can be easily eliminated by fitting a plane to the background describing points and eliminating all points within a specific threshold belonging to the plane. In principle, a simple threshold value for the depth information can also be used to remove the background. However, a perpendicular orientation of the sensor is necessary for this procedure. The use of an estimated plane and the definition of a threshold value related to the plane eliminates this constraint and is a more robust approach that is independent of the orientation between the sensor and the object plane. The removal of the background not only reduces the dataset but also improves the data basis for the further procedure. With the background, disturbing shadow areas that occur in the hyperspectral domain are also removed. In general, areas of shadow are a major problem in dataset processing, as they reduce the reliability and success rate of algorithms for object detection, classification and other types of processing. Consequently, removing shadow areas is an important step that improves image quality [START_REF] Mostafa | A Review on Various Shadow Detection and Compensation Techniques in Remote Sensing Images[END_REF][START_REF] Shahtahmassebi | Review of shadow detection and de-shadowing methods in remote sensing[END_REF] as well as the basis for further analysis steps. We have seen the influence of disturbing shadow effects in the experimental part of the analysis of spectral data in section 4. With the elimination of the background, it is possible to use further processing methods like 3D clustering algorithms to partition the point cloud [START_REF] Lari | An Adaptive Approach for Segmentation of 3D Laser Point Cloud[END_REF][START_REF] Nguyen | 3D point cloud segmentation: A survey[END_REF][START_REF] Rodriguez | Clustering algorithms: A comparative approach[END_REF][START_REF] Sanchez | A Linear Programming Approach for 3D Point Cloud Simplification[END_REF]. The primary task of cluster algorithms is to find structures with different properties within datasets and to discover relationships in order to be able to group objects. Cluster strategies differ in their approaches and strategies, but always with the goal of grouping objects that are nearly identical. Due to the homogeneity of the data in our dataset, a clustering approach based on Euclidean distance was chosen to group the individual objects. The prerequisite for the application of the mentioned cluster methodology is the step of background removal described beforehand. This causes 3D points describing an object to be in close spatial neighbourhood to each other, while 3D points of different ob- The clustering is done after removing the board points using a plane estimation method. Adapted from (Polat et al., 2021a).

It shows various components on a PCB and the result of the clustering process after removing the board. Due to the different depth levels of the components, a separation between the objects captured in 3D is possible despite partially overlapping components. This procedure reaches its limits with components that are in direct neighbourhood to each other and at the same depth. This is apparent for example from the result in Figure 5.6 (d). The two capacitors are recognised as one component because of their identical depths. In such cases, classification using only 3D features would most likely lead to incorrect results. Nevertheless, due to the hyperspectral component, these capacitors can be clearly classified as capacitors.

Rule-based classification

After the data has been combined, processed and simplified, the next step is the classification process. Similarly, this is based on a rule-based approach as described in chapter 4. Conditions are formulated and related to each other in order to establish rules for classifying materials. The difference is that the information available for rule formation is now much greater. The combined dataset, let's call it a hyperspectral point cloud, provides a source of valuable spectral as well as geometric information. This allows rules based on conditions that take into account the topology, depth and spatiality of the geometry like described in the beginning of chapter 5, as well as the physical characteristics of materials. The rule formation concerning the spectral dimension is performed in the same way as described in chapter 4.

Spectral signatures are examined and based on the shape of the spectral curves, curvaturebased features (i.e. curvature values, curvature behaviour, etc.) are used. These are extended or supplemented by rules based on spatial information and described by geometric characteristics considering the 3D point cloud.

With the knowledge we have acquired about the underlying objects and their spectral and spatial characteristics, we have a good basis for rule formation. We are aware of the objects and know which properties can be described most ideally by which domain to optimise the identification of objects or components of interest. The advantage of using features from both domains lies in the mutual support of the data. Let's take the colour black as an example.

Materials that are pigmented black (e.g. microchips on PCBs) do not provide useful spectral information that can be used in the classification because of the strong light absorption. Consequently, this would lead to unclassified or misclassified points. The formulation of rules that apply in this case and are based on geometric features would lead to an improvement of the classification.

In concrete terms, we follow again a hierarchical process und apply first spectral rules.

As long as objects cannot be classified by their spectral signature, the next step is to check rules for validity based on geometric characteristics. For example, let's take microchips. Microchips are generally black, meaning spectral signatures will not be helpful. However, they also have a unique geometry, which consists of a rectangular shape with a planar surface. The rule for such a microchip could be formulated as shown in Equation 5.9.

IF (omnivariance > threshold A)

(change in curvature < threshold B) THEN class →

(5.9)

Following the principle that rules should always reflect a logical approach and be based on the typical characteristics of the objects to be classified, the features 'omnivariance' and 'change in curvature' are used for the rule formation of microchips. The feature 'omnivariance' allows to distinguish between a linear, an aerial or a volumetric distribution of points. Low values correspond to flat regions and linear structures, whereas higher values are expected for areas with a volumetric point distribution (H. [START_REF] Gross | Segmentation of tree regions using data of a full-waveform laser[END_REF][START_REF] Niemeyer | Conditional Random Fields for the Classification of Lidar Point Clouds[END_REF]. In the case of the microchip, the threshold A for this parameter was experimentally set to 10. The 'change in curvature' feature is used to describe the planar surface of the microchip. This means, the threshold value for this feature must be set low (threshold B was set experimentally to 0.045), because a planar surface has no curvature. The threshold values for both parameters were determined by analyses based on the available dataset Different data bases resulting from different sensor technologies would only require the threshold values to be adjusted, as the features used reflect the properties of the objects.

Figure 5.7. Spectral signatures used for the classification. As a result, the rule defined for a category is specific to that category and there is no possibility of confusion because the shapes are distinct enough. Such rules can therefore only be used if the shapes of the spectral curves are all sufficiently different (Polat et al., 2021a).

For the underlying dataset used in this thesis, 8 rules were established based on spectral characteristics. The spectral signatures of the materials (e.g. capacitors, plastic parts, board parts) are shown in Figure 5.7. Furthermore, 5 rules based on geometric features were set up to identify objects such as microchips, black capacitors or also black connectors. A summary of the rules and the sequence of application is documented in Appendix C.

Results and Discussion

The results of the classification are presented step by step in this section. In this process, the established rules are passed through using a hierarchical strategy and the classification result is subsequently refined comprehensively utilizing the advantages of spatial and spectral information. Hierarchical processing has the benefit to simplify the general value, because following a strategy going from course and more important to fine and more specific content, which makes such processes more robust and increase the value of generalisability as it becomes easier to extend to other cases. A further benefit is the reduction of calculation effort and the simultaneously increase of the classification performance. Spatial data, for example, provide an excellent basis for reducing the database and external error factors so that the focus can be directed to the essential objects. For the classification of objects, on the other hand, the spectral dimension is a valuable option. By processing the underlying information sources in a logical and meaningful way and by exploiting the advantages of each of these sources, the process of finding a solution can generally be optimised, irrespective of the intended application.

The scheme for the individual stages is illustrated in Figure 5.8 and can be understood as a hierarchical process which iterative refine starting with simple and dominant conditions and continues towards more complex and compound conditions.

Figure 5.8. Steps of the rule-based classification process using spectral and spatial features for the rule formation. The geometrical information is not only used for structuring but also support the classification of components that cannot be classified from their spectral signature only (Polat et al., 2021a).

Step 1: Separation between plastic and PCB

The distinction between plastics and PCBs is the first step in the classification process due to the significant spectral differences between these objects. Understanding the spectral behaviour of plastics allows classifying them accurately. Advantageous in this context is the pre-processing of the hyperspectral point cloud into individual clusters. The principle of clustering chosen in our case is possible due to the given conditions. The objects lie on a plane surface, which allows us to remove the background and then group them based on simple distance considerations between neighbouring points. Typically, objects lie on planar surfaces, so for the majority of use cases (e.g. objects on a conveyor belt in industrial context) such an approach is applicable.

Figure 5.9. Example of result of classification (after step 1) of plastic objects by checking average spectra for each cluster (Polat et al., 2021a).

In cases where these assumptions do not apply, alternative methods must be used. For example, a possibility to remove the background could be to consider the colour and/or material composition of the background. In this case, the hyperspectral camera could be used to remove the background, so that cluster methods can again be used to group the objects. Creating groups makes it possible to consider all 3D points of an object and consequently the entire spectral information of this object. In the case of plastics, it is thus possible to obtain a clean classification for the entire object with a single query regarding the averaged spectral values for all 3D points describing the plastic sample. This works because plastics usually consist of only one component, whereas PCBs in general consist of different material compositions and objects (board and different components). The result of this first rule query is shown in Figure 5.9 and includes an accurate classification of all 5 plastic samples (PS, PP, PA). This not only has advantages in terms of classification, but also offers a significant advantage in terms of performance, since it is not necessary to check rules for each individual 3D point.

Step 2: Board and component classification using HSI

In the next step of classification, the focus is on the remaining objects. By assuming that all unclassified objects must be samples made of compound materials, the further processing of each cluster (object) is carried out. We know that PCBs usually consist of a board and various electronic components on the board. This means that each of the clusters that still need to be classified must be processed in a further step to distinguish between board and components.

Figure 5.10. Result of classification for PCBs using spectral features (after step 2). Classification of 3 board types and 4 different types of components on board (PS, ecob 1, ecob 2 and ecob 3) (Polat et al., 2021a). This is done using the same approach as for background removal. By fitting a plane within each cluster, a separation is made between the actual board of a PCB and the individual electronic components on board (ecob). By reusing cluster analysis techniques, the components of each PCB become separated again like shown in Figure 5.6 (c). The result is a separation of the components on the board, containing geometrical and physical information for each of them inside a cluster. The next step in the rule-based classification is to check the average spectral signature for each board and component. All boards and components that provide sufficient spectral information are included in this classification step. The result can be seen in Figure 5.10 and includes the classification of 3 different board types, 3 different capacitor types and one connector made of PS. All other components that were not assigned to a class cannot be processed due to missing spectral information and must therefore be considered in the next step by means of geometric features.

Figure 5.11. Result of classification for PCBs using 3D features (after step 3). Classification of 4 different types of components on board (ecob 4, ecob 5, ecob 6 and ecob 7). Red arrows point to false classified components. Adapted from (Polat et al., 2021a).

Step 3: Component classification using 3D information

In the last classification step, a further refinement is made on the basis of existing knowledge. Through the cluster analysis in advance, we know that the remaining objects must be parts of a PCB. Rules based on geometric features and topological relationships allow fur-ther classification of these remaining objects. By applying a set of 4 additional rules, the remaining objects can be separated into groups of round capacitors, rectangular and planar objects like microchips, resistors and other components like potentiometers and connectors.

A look at the resulting classification map in Figure 5.11 shows that some misclassifications (red arrows) have occurred in this step of the classification. This is due to similarities in geometry that occur. The connector shown in Figure 5.12 and the horizontal radial capacitor have a very similar 3D geometry. This circumstance leads to the aforementioned misclassification, in which the connector was assigned to the class of radial capacitors (ecob 5).

Furthermore, this step also shows that the classification of PCB parts in WEEE is very challenging due to contamination and damage. Even if not all objects, especially the small ones, on the PCB have been classified, the overall classification result with an Overall Accuracy (OA) of 98.24% is satisfactory.

Figure 5.12. Misclassification due to geometric similarities in the point cloud (Polat et al., 2021a).

One of the most important aspects concerning 3D data is the fact that the chosen sensor technology is a crucial factor regarding the success rate. A 3D capture technology with a low resolution and depth accuracy (e.g. Microsoft KinectTM) will not provide a data basis that allows to capture small objects sufficiently. The precise acquisition of the individual objects is essential for the description of the geometry. With the structured light scanner integrated in our system, the limits become apparent with the very small elements like resistors (green box), capacitors (yellow box) and metals (red and blue box) as presented in Figure 5.13. The smallest object that could still be classified is the group of black resistors (ecob 7). Looking at the statistics in Table 5.1, we see metrics for each individual class. While the larger objects have high overall statistics, the OA of the resistors is 65.95%. Therefore, for satisfactory results, the choice of sensor should be well-considered and made depending on the intended application.

But even if a distinction between these elements does not seem possible in detail, these elements can be identified as components of PCBs by the given topological relationships, which could be advantageous for further analyses (e.g. determination of different metals).

Figure 5.13. Small components like metals (red and blue box), resistors (green box) and capacitors (yellow box) are difficult to distinguish because of limits in resolution (Polat et al., 2021a).

To assess the classification accuracy, the dataset was annotated and metrics were derived using the confusion matrix. The annotated dataset was also used for comparison on a SVM-based classification approach (C-SVC, RBF-Kernel, C: 2048, gamma: 2.8284). In this comparison, we mainly aim to show that the combined approach (HSI and 3D) provides advantages in terms of classification results compared to a purely HSI-based classification. The comparison was done using the LIBSVM library [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF] for MATLAB. Tenfold cross-validation was used to determine the optimized hyperparameters. Randomly 10% of the samples of each class were selected for the training dataset. The results of this comparison can be found in Table 5.2 and confirm the assumption that a combined approach (OA: 98.24%) and mutual support to compensate for weaknesses is advantageous compared to a purely HSI-based processing (OA: 88.65%).

DISCUSSION

In this chapter, the second main contribution of this PhD work was addressed and it was

shown by means of practical experiments that a combination of 3D and HSI and the analytical method we chose for processing the combined dataset, yields convincing results for the classification of samples in the WEEE sector. Although the flexibility and performance of our methodology is illustrated by an example from the waste sorting use case, this approach is easily transferable to other applications, because one of the key requirements of the methodology is generalisability. This includes two essential aspects, the generation of the combined 3D and HSI datasets and the processing of the combined dataset. Mainly the base is the unique spatial and physical (reflective) characteristics of objects. To this the acquisition and processing are adapted and provide these good results which will also be found in other applications where objects can be discriminated through spatial and physical differences. The gener-ation of a combined dataset is independent of the selected sensor technology. The only important point is that an HSI and a 3D point cloud are provided by the used sensors. However, the choice of sensor technology also defines the limits of the analytical part. Therefore, the choice of sensor technology should always depend on the intended application and consider the demands on the resolution and quality of the data. Let's take the small components like resistors or tiny capacitors from our waste sorting use case as an example. With a 3D sensor based on the Time of Flight (ToF) principle, the detection of these very small components would not be of sufficient quality to allow classification using rules based on 3D features.

Even the structured light scanner we use reaches its limits with these objects. Thus, if the focus should really be on the detection of such small objects, sensor technologies should be used that allow depth detection in the micrometre range (e.g. line laser scanners based on triangulation). If the combined dataset is available with the necessary accuracy, the processing can be carried out according to the same principle as proposed.

To get the most out of a combined 3D and HSI dataset, it is important for the strategic approach to analyse the underlying data sufficiently. Expert knowledge regarding the properties of objects helps to improve the results by developing rules adjusted on these properties.

Adding 3D information in particular is helpful in structuring and simplifying the datasets like shown in the experimental part by removing the background, areas of shadows and by separating objects trough cluster methods. In turn, hyperspectral data helps to determine the type of material and thus refines the classification. And more importantly, each of the very distinct datasets supports the other and helps to overcome the weaknesses of the other. Again, an example from the use case of waste sorting: A large number of the components that are mounted on PCBs have a black material colour. Due to the high light absorption, the use of purely spectral information is not advisable with these components. The SVM comparison presented in Table 5.2 confirms this, in which only HSI data was used for classification. Classifying based purely on 3D data can be successful in separating objects (e.g. board and components on the board), but would fail, for example, in distinguishing between different board materials or compositions.

For the use case that we selected, a total of 13 rules were developed and thus a separation of the existing objects (i.e. plastics, PCBs and electronic components) into 13 classes was carried out like presented in the classification map shown in Figure 5.11. This is not only due to the carefully defined rules but also to the iterative strategy of passing hierarchically from more dominant to fewer dominant rules.

In principle, care should be taken to avoid over-fitting when creating rules, as overly sensitive threshold values can lead to incorrect classifications when applied to new datasets and should therefore be selected carefully. It therefore makes more sense to proceed hierarchically and to structure and simplify the data in advance. This allows iterative refinement from the general to the specific and counteracts precisely problems like over-fitting and a high complexity of rules.

A further advantage of the proposed method is the flexibility, which permits expandability to include other materials and components. It should be noted, however, that the rules used for this dataset cannot be applied directly to a dataset based on a different 3D sensor technology. The threshold values set are the result of an extensive analysis of the database and may have to be adapted in the case of other sensor technologies. The necessity and degree of adaptation depends mainly on differences in resolution, accuracy and noise behaviour of the sensor technology used. The same applies to the transfer to other applications. Depending on the intended application, it is necessary to pay attention to correct modelling. It is important, therefore, to study the spatial and spectral properties of the objects in advance and to investigate to what extent the acquired data can make these properties visible. In summary, the limitations of our proposed method depend mainly on three factors: The first factor is the capacity of the sensor technologies used, the second the amount of work done in advance to analyse the data and model the rules and processes and finally the acquisition scenario, which in our case is optimised for the use in laboratory. This is not straightforwardly transferable to applications for example in an outdoor scenario and therefore requires an adaptation of the methodology to the circumstances (e.g. light conditions, sensor orientation, natural uneven structures, etc.).

And even the transferability to an industrial application is not necessarily given straight away and may require adaptations to the specific conditions of the industrial environment. In addition, we would like to mention that to our best knowledge, no comparable analytical strategy for the analysis and processing of combined spectral and 3D information is available or has been used.

At this point, we would like to explicitly address the waste sorting use case and the practical applicability of the proposed technique in industry. In order to be able to ensure an industrial use, several factors have to be considered. These include not only the speed (several metres per second) at which sorting processes are carried out, but also aspects relating to sensor integration and also the handling of dust and dirt (especially with regard to optical systems) in the industrial environment. As mentioned earlier, our proposed approach offers the flexibility to choose any sensor technology. The structured light scanner used in our experimental part is unsuitable for fast processing, as the measurement process does not occur in real time. There are, however, alternatives that can be used in industrial production lines, such as investigated in the studies of [START_REF] Bian | On-line detection device for high temperature forgings based on laser triangulation[END_REF][START_REF] Sioma | Automated Control of Surface Defects on Ceramic Tiles Using 3D Image Analysis[END_REF]. An investigation of different 3D sensor technologies and their applicability in the context of industrial production lines is the content of [START_REF] Sioma | 3D imaging methods in quality inspection systems[END_REF]. Four technologies for the generation of 3D data, based on laser triangulation, Time-of-Flight (ToF), Shape-from-Focus (SFF) and stereovision, are identified. Especially laser scanners based on triangulation are very common in production lines, as such systems are able to achieve depth resolutions up to a few micrometres and are a robust and standardised technology. Developments in the field of hyperspectral sensor technology are also increasingly making it possible to integrate them into industrial production lines. In the food industry in particular, hyperspectral cameras are used for quality control or inspection tasks [START_REF] Al-Sarayreh | Detection of Red-Meat Adulteration by Deep Spectral-Spatial Features in Hyperspectral Images[END_REF][START_REF] Boldrini | Hyperspectral Imaging: A Review of Best Practice, Performance and Pitfalls for in-line and on-line Applications[END_REF][START_REF] Ma | Advanced Techniques for Hyperspectral Imaging in the Food Industry: Principles and Recent Applications[END_REF]. The Specim FX17 used in our measuring system is with 670 FPS at full resolution certainly suitable for industrial applications. A further improvement in FPS is possible by reducing the spatial and/or spectral resolution. Reducing the spectral component in particular is quite conceivable with the proposed shape-based/rulebased method described in chapter 4, since a prior analysis of the data allows the restriction to a few bands or band regions.

Another advantage of our method is the handling of soiling on the objects and also on the conveyor belt. We discussed the influence of dirt on the behaviour of spectra in chapter 4

and showed that the shape-based approach is robust to such effects. The reason for this is that dirt only affects the reflectance values and causes a spectral shift, but the shape remains the same. This means that even if the signal is weak, as long as the shape is maintained, the rules apply. Furthermore, the influence of dirt on the conveyor belt as well as on objects can be minimised or eliminated through the 3D information. The 3D information allows us to group components into clusters and to use the average spectral signature of this cluster for the rule check. So, the influence of soiled regions only becomes significant if the object is entirely soiled. The situation is similar for dirt on the conveyor belt. Due to the 3D depth information, the background, i.e. the conveyor belt, can be removed by a simple threshold value related to the depth. This means that the problem of dirt on the conveyor belt is insignificant.

And finally, the processing time must also be mentioned. Performing 3D processing and point-by-point rule checking with MATLAB on a machine with an Intel® Core™ i7-10700 CPU @ 2.90 GHz and 16 GB RAM required 110.29 seconds for the dataset shown in Figure 5.4 without background (752,253 3D points and 224 bands). Compared to this, processing the same dataset (HSI only) on the same machine with SVM (LIBSVM for MATLAB) takes 2025.07 seconds.

Even if 110.29 seconds can be considered a high value, especially for a practical industrial application, it must also be considered that a full-spectral and high-resolution 3D point cloud serves as the data basis. Improvements in processing time are conceivable and could be achieved by reducing for example the dataset (e.g. reducing the spectral resolution), by using high-performance processing hardware and through more efficient programming (e.g. parallel computing).

CHAPTER 6: CONCLUSION

In this thesis, we investigated the potential of combining 3D and HSI sensors in the context of various application fields. Several points, namely the data acquisition, the data fusion, the analysis of spectral signatures and the processing of the combined hyperspectral point cloud were addressed for this purpose. First, various general concepts for data acquisition and data fusion were presented depending on the intended application. In the context of laboratory applications, hyperspectral pushbroom sensors were used in combination with a structured light scanner to produce hyperspectral point clouds. In contrast, for outdoor measurements, the concept of fusion based on a hyperspectral scanning system in combination with the SFM method was presented. Both concepts for data fusion are general and follow a simple approach that can be easily transferred to practical applications. Second, new methods for analysing hyperspectral signatures and processing hyperspectral point clouds were presented. These new developed methods are the main contributions of this thesis. Established methods for the analysis of spectral signatures are not always effective for the processing of hyperspectral data due to the factors discussed in the previous sections (e.g. curse of dimensionality, loss of information, amount of needed trainings datasets, etc.). The developed method, which is based on a concept of using shape-describing features of the spectral signatures and rule-based classification, is an alternative to established methods and has delivered convincing results, especially for the waste sorting use case. The strengths and weaknesses of this method can be summarised as follows:

Strengths



In contrast to established machine learning methods, not only intensity or reflectance values are considered, but also the entire geometric course of the curve. This allows a more robust handling of spectral information.



Dirt and/or shadows can lead to shifts in the spectral signatures, while the geometric course of the spectra remains the same despite these influences. By considering only the spectral shape, our approach is robust to spectral shifts.



The formation of rules does not require a large number of reference datasets or trainings datasets. Basically, a single spectrum is sufficient to express conditions linked to the shape of the spectrum as a rule. However, using a larger database allows a deeper understanding of the underlying objects and the derivation of stable rules.



The rule formation is based on expert knowledge. Preliminary analyses allow efficient processing of the data and help to simplify and structure it. This not only has a positive effect on the result, but can also lead to a reduction of the spectral data to significant bands or regions and thus to an increase in performance.



Hyperspectral data has the significant advantage of high spectral resolution. In contrast to established methods such as indices, shape-based analysis can consider the finest differences in spectral signatures, while indices only consider spectral broadband regions.

Weaknesses



For a robust classification, distinct spectral signatures of different materials are advantageous. Materials that differ only minimally in their fingerprint can also be separated, but this can increase the complexity of the rule.



The number of materials or objects to be classified should be limited. A large number of classes can also lead to complex rules or lead to conflicts between them. An increased number of classes also increases the analysis effort in advance. the reduction of data to increase performance, to simplify the dataset or to structure it, could be improved. The use of the analytical method in experimental results from the waste sorting use case confirms the applicability and effectiveness of the method. Nevertheless, it must also be mentioned that the experimental investigations so far and the measurement system used are solutions to demonstrate theoretical feasibility in industrial sorting or inspection applications.

A step that needs to be considered in the future is therefore the optimisation of our approach for industrial needs. This includes, on the one hand, the adaptation of sensor technology, but also the optimisation of processes and methods. In order to meet the demands on quality and also with regard to the required speed, further investigations and measurements should be carried out with regard to the sensor technology used. The structured light scanner used in this work should be replaced by a 3D sensor usable for fast sorting processes. Conceivable in this context would be the use of laser scanners based on triangulation. Furthermore, it is necessary to consider the integration of sensors into existing sorting systems. The optimal integration of the 3D and HSI sensors as well as the illumination unit are crucial with regard to the quality of the acquired data. Based on the acquired data, processes and parameters of the methodology presented in this thesis needs to be adapted to the new data basis and optimized if necessary.

These include both the processes for simplifying and structuring the data basis as well as the rules used.

The use in practice would also have to be addressed for the application in the field of cultural heritage. The intention in practice is to use UAS-based data acquisition. Due to the incompatibility of the Senop HSC-2 hyperspectral camera with our drone, the investigations are limited to the laboratory. The practical inspection of wall tops using UAS and the application of our shape-based/rule-based method for the detection of harmful plants is therefore also an open issue that still needs to be addressed. As can be seen from the images, the bloodstains were applied in the form of thicker drops and smears. The thicker bloodstains could not be completely covered despite several layers of paint. The smears, on the other hand, are no longer visible after the paint has been applied.

At the time of these measurements, the Specim FX10 and FX17 sensors shown in Figure 2.3 was not yet available, so the sensor system shown in Figure D2 was used for the acquisition process. Shown is the mobile Specim IQ camera, which works in the wavelength range from 400 to 1000 nm. The resulting datasets are spatially resolved to 512 x 512 pixels and cover a total number of 204 spectral bands. The radiometric normalisation of the HSI was carried out using a 99% Spectralon TM white reference (Specim) square panel (100 mm x 100 mm), which is acquired simultaneously with the captured scene. The acquisition of the dark reference image is part of the data recording workflow inside the camera and is automatically recorded (front lens cap closes). More specifications of the camera are listed in Table D1. A comprehensive evaluation of the Specim IQ can be found in [START_REF] Behmann | Specim IQ: Evaluation of a New, Miniaturized Handheld Hyperspectral Camera and Its Application for Plant Phenotyping and Disease Detection[END_REF]. After getting the systems shown in Figure 2.3, the samples were additionally measured with the Specim FX17 in order to also investigate the range between 900 and 1700 nm. The Specim FX17 HSI cubes have a spatial resolution of 640 x 587 pixels and covers 224 bands.

In addition to the colour-covered samples, blood traces covered with soot also were prepared. As can be seen in Figure D3, glass slides were covered with traces of blood for this purpose and subsequently sooted by using a candle. The sooting process was carried out by the German Federal Criminal Police Office (BKA). The spectral investigations of the sooted specimens were done by the Specim FX10 and the Specim FX17 and cover the electromagnetic spectrum from 400 to 1700 nm.

Table D1. Specifications Specim IQ.

Results and Discussion

In order to be able to make a statement concerning the possibility of detecting blood traces covered with paint and soot using HSI, the hyperspectral cubes were decomposed into sub-images representing the spatial distribution of the spectral intensity related to the corresponding wavelength from 400 to 1700 nm. To improve the quality of the sub-images, contrast enhancement was also applied. When looking at the sub-images shown in Figure D4, it can be seen that some of the blood traces are actually visible. This seems to be the case in the shortwave range at around 415 nm and also at a wavelength of around 1586 nm. A noticeable aspect, however, is that the effect of visibility is dependent on certain factors like the thickness of the bloodstains applied, the colouring of paint and the chemical composition of the paint. The thicker drops of blood that were painted over with white paint, for example, can also be seen clearly in the RGB, as it was not possible to cover them completely with paint despite several layers. In this respect, the appearance in the HSI is not really surprising. All other traces that were covered with white paint are not really visible in the HSI. The high reflectivity of the white paint could be a reason for this. However, if we look at the bloodstains that have been covered with purple paint, we can see that in some cases even the very well covered, slightly the different paintings has an influence on the recognisability can also be seen in the board covered with green and blue paint. While the traces under the blue colour are clearly recognisable at a wavelength of 415 nm, with the green colour only the thick drops of blood can be seen. At a wavelength of 1586 nm, on the other hand, not even the thick drops of blood are visible. However, colouring should not play a significant role in this wavelength, as spectral investigations in the NIR are primarily aimed at physical aspects. The composition of the colours must therefore also have an influence. Table D2 shows the composition of the 4 colours used.

Table D2. Chemical composition of the colours used. it is obvious that this circumstance is due to the composition of the material.

Paint

In addition, the number of layers of paint applied is also a factor. Complete coverage of bloodstains requires at least several layers of paint even for bloodstains that are not very pronounced. In Figure D5 we see an example of bloodstains that were painted over with the green paint. A clean coverage by the green paint, such that no blood traces could be seen under the paint with the naked eye, could only be achieved with 3 layers of paint. Also shown are the averaged reflectance values of the HSI after each individual layer. Obviously, as the thickness of the layer increases, the visibility in the HSI decreases. A statement about the usefulness of HSI for the detection of bloodstains that have been covered with paint is rather difficult at this point. Even if we can see from the experiments that there is a certain potential, further intensive investigations are still essential. Especially with regard to the variety of different colours and manufacturers and the manufacturer-specific composition of chemical substances.

The situation is similar with the soot-covered samples. The spectral examinations of the samples in the wavelength range from 400 to 1700 nm did not allow any conclusions to be drawn about traces underneath the soot, neither in the case of the thick traces nor in the case of the thin ones. The reason for this, however, lies in the fact that soot absorbs strongly in the UV, VIS and NIR regions like carbon black. The problem of the strong absorption of black coloured materials is well known in the field of waste management [START_REF] Dvorak | Development of NIR Detectable Black Plastic Packaging[END_REF][START_REF] Rozenstein | Development of a new approach based on midwave infrared spectroscopy for post-consumer black plastic waste sorting in the recycling industry[END_REF]. It is similar with soot, as its deep blackness absorbs all light [START_REF] Chudnovsky | Reflectance spectroscopy of indoor settled dust in Tel Aviv, Israel: comparison between the spring and the summer seasons[END_REF] like shown in Figure D6. Studies in the field of waste management have shown that the Midwave Infrared Spectrum (MWIR) is very suitable for using spectral signatures in the processing of black materials [START_REF] Becker | Detection of Black Plastics in the Middle Infrared Spectrum (MIR) Using Photon Up-Conversion Technique for Polymer Recycling Purposes[END_REF][START_REF] Signoret | Mir spectral characterization of plastic to enable discrimination in an industrial recycling context: Ii. Specific case of polyolefins[END_REF]. This could also be useful in the case of soot-covered bloodstains and should be considered in further investigations.

Conclusion

Our research shows some potential for the use of HSI in terms of hidden blood traces.

However, much more intensive investigations are still needed. Especially with regard to bloodstains that are covered with soot. We know that light from the UV to SWIR range is completely or very strongly absorbed, but we also know that MWIR is suitable for separating black objects. Even though it was impossible for us to carry out the investigations due to the lack of a suitable sensor in the MWIR, further investigations are being planned by our project partner.

In the case that HSI covering the MWIR proves to be feasible, the practical applicability at a crime scene would still have to be investigated. This includes, in particular, issues relating to the illumination at a crime scene and the consideration of the heat generated by lighting necessary for the MWIR.

Figure 2 .

 2 Figure 2.1. Representation of an HSI and the comparison with a RGB image. Each wavelength λ is represented as an image with x and y dimensions. The spectral component is presented as a vector for each pixel (Lu & Fei, 2014). ..............................

  Figure 2.1. Representation of an HSI and the comparison with a RGB image. Each wavelength λ is represented as an image with x and y dimensions. The spectral component is presented as a vector for each pixel (Lu & Fei, 2014). ..............................

  Figure 2.1. Representation of an HSI and the comparison with a RGB image. Each wavelength λ is represented as an image with x and y dimensions. The spectral component is presented as a vector for each pixel (Lu & Fei, 2014). ..............................

Figure 2 . 2 .

 22 Figure 2.2. Hyperspectral camera types and their respective acquisition and data storage methods. (a) Whiskbroom camera; (b) Pushbroom camera; (c) Hyperspectral camera based on spectral scanning; (d) Snapshot camera (Halicek et al., 2019). ........................

Figure 2 . 3 .

 23 Figure 2.3. Measuring system consisting of an illumination unit, linear traversing unit and two pushbroom cameras for the acquisition of spectral information and a structured light scanner for the acquisition of 3D information. ........................................................

Figure 2 . 4 .

 24 Figure 2.4. Combined spectral signature of dandelion. ............................................................

Figure 2

 2 Figure 2.5. Frame-based hyperspectral camera Senop HSC-2................................................

Figure 2 . 6 .

 26 Figure 2.6. Components of electronic wastes. Cables (left), plastics (middle) and printed circuit boards (right). ......................................................................................................

Figure 2

 2 Figure 2.7. Representation of a tie point used for the fusion process in HSI captured by the NIR camera. ...................................................................................................................

Figure 2

 2 Figure 2.8. 3D point cloud of a PCB with registered HSI (a), single HSI visualised by the average reflectance value over the entire spectrum (900 -1700 nm) (b) and HSI after the transformation to the 3D point cloud (c). .................................................................

Figure 2 . 9 .

 29 Figure 2.9. Waste samples represented as RGB image (at left) and combined 3D point cloud and HSI (at right) with average reflectance value over the entire spectrum (900 -1700 nm) shown per pixel for (a) PCB; (b) plastic. ................................................................

Figure 2 .

 2 Figure 2.10. Vegetation on parts of the walls of the Balduinseck castle ruins in Buch (Germany). .....................................................................................................................

Figure 2 .

 2 Figure 2.11. Calculated point cloud using the 8 hyperspectral images presented right. The blue rectangles represent the camera position and orientation for each image. The white square shows the reflectance target. ...............................................................................

Figure 2 .

 2 Figure2.12. 3D point cloud obtained from depth image and HSI visualised with average reflectance value over the entire spectrum (500 -900 nm) shown for each 3D point. ..

Figure 3

 3 Figure 3.1. Reflectance Spectra of PP presented in (a) and processed with SNV (b), 1st order derivative (c) and 2 nd order derivative (d). .....................................................................

Figure 3 . 3 .

 33 Figure 3.3. The use of Continuum Removal for quantification of absorption peaks by connecting the maxima of spectra with straight lines (left), setting the convex hull to 100% and subtracting the original spectra (right) (Polat et al., 2021b). .........................

Figure

  Figure 4.1. Comparison of different smoothing grades using Savitzky-Golay filter oncontinuum removed spectra. An increase in the smoothing grade increases the noise reduction, but also the possibility that the signal will be distorted by the smoothing. ...

Figure 4 . 2 .

 42 Figure 4.2. Representation of the spectral variability for 2 different samples with (a) homogeneous and clean surface; (b) inhomogeneous surface due to dirt, height differences and shadows (spectra of all pixels inside the red box of both samples); (c) and (d) illustrate the corresponding 2 nd derivatives. .......................................................

Figure 4

 4 Figure 4.3. Meaning of curvature illustrated by the spectral signature of polystyrene. ..........

Figure 4

 4 Figure 4.5. Continuum Removed Reflectance spectra (black line), calculated curvature values (green lines), local extrema with concave behaviour (red dots) and convex behaviour (blue dots) for (a) Polystyrene (PS); (b) Polyethylene (PE) (Polat et al., 2021b). ..........

Figure 4 . 6 .

 46 Figure 4.6. Variation of spectra for polypropylene depending on different colouring, dirt, shadows and damaged surfaces. .....................................................................................

Figure 4 . 7 .

 47 Figure 4.7. Continuum removed spectra for different objects consisting of polypropylene and affected by dirt, shadow areas, different colorization and damages surfaces. The analysis of such an overview is helpful to identify robust spectral features for the rule formulation. ....................................................................................................................

Figure 4

 4 Figure 4.8. RGB images of captured specimens. (a) Seven classes of different plastics consisting of phenolic resin (PF), polyamide (PA), polyvinylchloride (PVC), polyethylene (PE), polymethylmethacrylate (PMMA), polyester resins (UP) and polystyrene (PS); (b) Randomly selected real waste plastic and circuit boards. ............

Figure 4

 4 Figure 4.9. Mean reflection spectra of parts of the available plastics used for rule formation.Each plastic category has a specific spectral response. This specific shape is also reflected in the shapes of the continuum-removed reflectance spectra. This means that the rule (based on a combination of conditions) defined for a category is specific to that category and there is no risk of confusion because the shapes are distinct enough. Considering all materials which need to be classified in the formation of rules has the advantage that rules can be set up more efficiently and conditions based on significant bands can be kept low.[START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]. ..................................................................

  Figure 4.9. Mean reflection spectra of parts of the available plastics used for rule formation.Each plastic category has a specific spectral response. This specific shape is also reflected in the shapes of the continuum-removed reflectance spectra. This means that the rule (based on a combination of conditions) defined for a category is specific to that category and there is no risk of confusion because the shapes are distinct enough. Considering all materials which need to be classified in the formation of rules has the advantage that rules can be set up more efficiently and conditions based on significant bands can be kept low.[START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF]. ..................................................................

Figure 4 .

 4 Figure 4.10. Classification result using shape-based rules (Polat et al., 2021b). ....................

Figure 4 .

 4 Figure 4.11. Classification result for real waste samples consisting of plastics and printed circuit boards using shape-based rules (Polat et al., 2021b). ..........................................

Figure 4 .

 4 Figure 4.12. Visualisation of Reflectance Spectra (a); Continuum Removed Reflectance (b) and 2 nd derivatives of PCBs (c) for objects 20 -27. Bands for identification of PCBs are at 1401 nm and 1429 nm. ...............................................................................................

Figure 4 .

 4 Figure 4.13. Detailed view of the classification results for proposed rule-based method; SVM (a): RBF kernel, C-Value of 1 and classification threshold of 0.5; SVM (b): linear kernel, C-Value of 1 and classification threshold of 0.3; SVM (c) polynomial kernel, C-Value of 1 and classification threshold of 0.3. ...............................................................

Figure 4 .

 4 Figure 4.14. Detailed view of classification results using: (a) proposed shape-based method; (b) SVM with RBF kernel, C-Value of 0.2 and classification threshold of 0.2; (c) RBF kernel, C-Value of 0.2 and classification threshold of 0.3; (d) RBF kernel, C-Value of 0.2 and classification threshold of 0.4; (e) RBF kernel, C-Value of 0.2 and classification threshold of 0.5; (f) RBF kernel, C-Value of 0.2 and classification threshold of 0.6 .....

Figure 4 .

 4 Figure 4.15. Classification results using: (a) proposed shape-based method; (b) SVM with RBF kernel, C-Value of 1 and classification threshold of 0.5; (c) linear kernel, C-Value of 1 and classification threshold of 0.3; (d) polynomial kernel, C-Value of 1 and classification threshold of 0.3. (Polat et al., 2021b) .......................................................

Figure 4 .

 4 Figure 4.16. Classification results using: (a) proposed shape-based method; (b) SVM with RBF kernel, C-Value of 0.2 and classification threshold of 0.2; (c) RBF kernel, C-Value of 0.2 and classification threshold of 0.3; (d) RBF kernel, C-Value of 0.2 and classification threshold of 0.4; (e) RBF kernel, C-Value of 0.2 and classification

Figure 4 .

 4 Figure 4.20. Using shape-based characteristics to divide the plants in two groups. ............... 59 Figure 4.21. Rule-based classification result using shape-based properties for plants in VIS and NIR (500 -1700 nm). .............................................................................................. 60 Figure 4.22. SVM classification result for plants using VIS and NIR (500 -1700 nm). ....... 61 Figure 4.23. Spectral variations in VIS for one type of moss regarding all pixels in the red box. Main reasons for this strong variations are for example optical properties of leaves and the direction of the leaf to the light beam (Polat et al., 2021b). .............................. 62 Figure 5.1. Workflow consisting of data acquisition, data fusion, the use of 3D data to simplify and structure the data basis and the final classification based on obtained HSI and 3D features (Polat et al., 2021a). ............................................................................. 67 Figure 5.2. Visualization of 3D features. Omnivariance (left), Verticality (middle) and Normal Change Rate (right) (Polat et al., 2021a). .......................................................... 68 Figure 5.3. Acquired datasets of electronic waste consisting of waste plastics and PCBs: (a) RGB; (b) 3D point cloud with a resolution of 0.2 mm and a depth accuracy of 0.05 mm; (c) HSI with a resolution of 636 x 1118 pixels and 224 bands in the wavelength range from 900 -1700 nm (Polat et al., 2021a). ...................................................................... 71 Figure 5.4. Combined 3D point cloud and HSI. Each 3D point is assigned by a spectrum. In this figure, the average value over the entire spectrum is shown per 3D point (Polat et al., 2021a). ...................................................................................................................... 72 Figure 5.5. Influence of shadow on the classification process. Here shadow occurs only on the neighbouring area surrounding the object. (a) Classification of pixels values belonging to the class PS using spectral data. (b) Classification of pixels values belonging to the class PS after removal of the background (Polat et al., 2021a). .......... 73 Figure 5.6. Examples of results of 3D point cloud clustering: (a) Background removed dataset; (b) Clustered objects; (c) PCB with electronic components and clustering result; (d) PCB with electronic components and failed separation of capacitors. The clustering is done after removing the board points using a plane estimation method. Adapted from (Polat et al., 2021a). ....................................................................................................... 74 Figure 5.7. Spectral signatures used for the classification. As a result, the rule defined for a category is specific to that category and there is no possibility of confusion because the shapes are distinct enough. Such rules can therefore only be used if the shapes of the spectral curves are all sufficiently different (Polat et al., 2021a). .................................. 76

  Figure 5.7. Spectral signatures used for the classification. As a result, the rule defined for a category is specific to that category and there is no possibility of confusion because the shapes are distinct enough. Such rules can therefore only be used if the shapes of the spectral curves are all sufficiently different (Polat et al., 2021a). .................................. 76 Figure 5.8. Steps of the rule-based classification process using spectral and spatial features for the rule formation. The geometrical information is not only used for structuring but also support the classification of components that cannot be classified from their spectral signature only (Polat et al., 2021a). .................................................................. 77

  the neighbourhood are also the key element in terms of spatial characteristic extraction in the context of 3D point clouds. Point clouds are a representation of 3D information and allow the extraction of geometric features that can be used to describe the properties of objects and scenes. These properties are derived through local geometric relationships between individual 3D points. This means that objects/scenes can be described unambiguously on the basis of their properties, such as their linearity, flatness or change in curvature. All extracted spectral and spatial characteristics are integrated into a rule-based approach to classify objects of interest. Main advantage of rule-based classifiers is that classification rules can be established very easily. Based on knowledge acquired through prior analysis of the data base, conditions can be established and correlated for rule formulation. Other advantages are the performance, the ability to deal with redundant and irrelevant attributes,

Figure 2

 2 Figure 2.2. Hyperspectral camera types and their respective acquisition and data storage methods. (a) Whiskbroom camera; (b) Pushbroom camera; (c) Hyperspectral camera based on spectral scanning; (d) Snapshot camera (Halicek et al., 2019).

Figure 2

 2 Figure 2.3. Measuring system consisting of an illumination unit, linear traversing unit and two pushbroom cameras for the acquisition of spectral information and a structured light scanner for the acquisition of 3D information.

Figure 2

 2 Figure 2.4. Combined spectral signature of dandelion.

Figure 2

 2 Figure 2.5. Frame-based hyperspectral camera Senop HSC-2 1 .

Figure 2

 2 Figure 2.6. Components of electronic wastes. Cables (left), plastics (middle) and printed circuit boards (right).

Figure 2 . 7 .

 27 Figure 2.7. Representation of a tie point used for the fusion process in HSI captured by the NIR camera.

Figure 2

 2 Figure 2.8. 3D point cloud of a PCB with registered HSI (a), single HSI visualised by the average reflectance value over the entire spectrum (900 -1700 nm) (b) and HSI after the transformation to the 3D point cloud (c).

Figure 2 .

 2 Figure 2.10. Vegetation on parts of the walls of the Balduinseck castle ruins in Buch (Germany).

Figure 2 .

 2 Figure 2.11. Calculated point cloud using the 8 hyperspectral images presented right. The blue rectangles represent the camera position and orientation for each image. The white square shows the reflectance target.

  Figure 3.1. Reflectance Spectra of PP presented in (a) and processed with SNV (b), 1st order derivative (c) and 2 nd order derivative (d).

Figure 3

 3 Figure 3.3. The use of Continuum Removal for quantification of absorption peaks by connecting the maxima of spectra with straight lines (left), setting the convex hull to 100% and subtracting the original spectra (right) (Polat et al., 2021b).

  Figure 4.2. Representation of the spectral variability for 2 different samples with (a) homogeneous and clean surface; (b) inhomogeneous surface due to dirt, height differences and shadows (spectra of all pixels inside the red box of both samples); (c) and (d) illustrate the corresponding 2 nd derivatives.

  Figure 4.2 (a) and (b), which shows examples of areas of a sample with a homogeneous and clean surface and a sample with an inhomogeneous surface. All spectral signatures within the red box are shown in the graph. While all spectral signatures of the sample with the homogeneous surface are very similar, for the real waste sample a high spectral shift can be seen due to the differences in depth, surface location, shadow effects and soiling.

Figure 4

 4 Figure 4.2 (c) and (d) illustrate the 2 nd derivatives of the spectra. As can be seen, the strong

Figure 4

 4 Figure 4.3. Meaning of curvature illustrated by the spectral signature of polystyrene.

Figure 4 . 4 .

 44 Figure 4.4. Selection of significant bands for shape description using extreme values of 2 nd derivative.Shape describing features are the bands location (min/max), the curvature values at these locations (red lines on the top of green areas) and the direction of curvature values (up for convex and down for concave behaviour). Adapted from[START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

  Figure 4.5. Continuum Removed Reflectance spectra (black line), calculated curvature values (green lines), local extrema with concave behaviour (red dots) and convex behaviour (blue dots) for (a) Polystyrene (PS); (b) Polyethylene (PE) (Polat et al., 2021b).

Figure 4 . 6 .

 46 Figure 4.6. Variation of spectra for polypropylene depending on different colouring, dirt, shadows and damaged surfaces.

Figure 4

 4 Figure 4.7. Continuum removed spectra for different objects consisting of polypropylene and affected by dirt, shadow areas, different colorization and damages surfaces. The analysis of such an overview is helpful to identify robust spectral features for the rule formulation.

  g. conditions based on Continuum Removed Reflectance Values (CRRV), radiometrically normalized reflectance values or also relationships between individual bands or averaged reflectivity in different regions as indices like NDVI do. Due to the flexibility of rule-based approaches, there are no limitations in this respect.

  Figure 4.8. RGB images of captured specimens. (a) Seven classes of different plastics 4 consisting of phenolic resin (PF), polyamide (PA), polyvinylchloride (PVC), polyethylene (PE), polymethylmethacrylate (PMMA), polyester resins (UP) and polystyrene (PS); (b) Randomly selected real waste 5 plastic and circuit boards.

Figure 4

 4 Figure 4.8 (b) are characterised through dirty surfaces, damaged surface parts, depth differences and shadow effects due to the object geometry. All these factors can lead to changes in

Figure 4 .Figure 4

 44 Figure4.11, we see convincing results despite the existing challenges (e.g. shadow, dirt, depth). Since the objects in this dataset are randomly selected and consist on plastic parts (objects 1 -19) and circuit boards (objects 20 -27), not all types of plastic from the plastic specimen dataset are represented. Instead, three additional types of plastic are present. While two of the types could be identified as PP and acrylonitrile butadiene styrene (ABS) on the basis of literature[START_REF] Wu | Auto-sorting commonly recovered plastics from waste household appliances and electronics using near-infrared spectroscopy[END_REF][START_REF] Zhu | Plastic solid waste identification system based on near infrared spectroscopy in combination with support vector machine[END_REF], object number 5 was left out of the classification due to a lack of reference. Also excluded from the classification are the black coloured plastic parts (objects 1 and 17). Since the background used in this dataset also consists of black plastic, a separation in this special case proves to be difficult. In addition to the rules for the detection of plastics, a rule for the detection of circuit boards has also been established. The main problem in handling PCBs is the great variance in composition depending on materials and components from different manufacturers, different appliances and different age of PCBs

  (a) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: RBF; C-Value: 1; Threshold: 0.5. (b) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: linear; C-Value: 1; Threshold: 0.3. (c) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: polynomial; C-Value: 1; Threshold: 0.3.

Figure 4 .

 4 Figure 4.13. Detailed view of the classification results for proposed rule-based method; SVM (a): RBF kernel, C-Value of 1 and classification threshold of 0.5; SVM (b): linear kernel, C-Value of 1 and classification threshold of 0.3; SVM (c) polynomial kernel, C-Value of 1 and classification threshold of 0.3.

  Figure 4.14. Detailed view of classification results using: (a) proposed shape-based method; (b) SVM with RBF kernel, C-Value of 0.2 and classification threshold of 0.2; (c) RBF kernel, C-Value of 0.2 and classification threshold of 0.3; (d) RBF kernel, C-Value of 0.2 and classification threshold of 0.4; (e) RBF kernel, C-Value of 0.2 and classification threshold of 0.5; (f) RBF kernel, C-Value of 0.2 and classification threshold of 0.6

Figure 4 .

 4 Figure 4.17. A section of the wall of the Balduinseck castle ruins covered with vegetation.

Figure 4 .

 4 Figure 4.18. Different plant species consisting of taraxacum (1), inula conyza (2), campanula rotundifolia (3), moss (4), geranium robertianum (5), asplenium trichomanes (6), green sedum (7), moss & red sedum (8), moss (9), moss & lichen (10)[START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

Figure 4 .

 4 Figure 4.19. Spectra of measured plants.Two different cameras were used to acquire the spectra (Specim FX10 and Specim FX17). The discontinuities in the range of 900 nm and the greater noise in the VIS range can be explained by the differences in spectral sensitivity of the cameras. In particular, the range below 500 nm exhibits strong noise effects and was not considered in the rule formation. Taraxacum (1), inula conyza (2), campanula rotundifolia (3), moss (4), geranium robertianum (5), asplenium trichomanes (6), green sedum (7), red sedum (8), moss (9), lichen (10)[START_REF] Polat | Rule-Based Classification of Hyperspectral Imaging Data[END_REF].

  Figure 4.19. It shows the spectral signature for 1 pixel of each plant shown in Figure 4.18. Different regions of the electromagnetic spectrum cover different properties of the plants. While in the VIS range the pigments of the plants influences the spectral characteristics, in the NIR it is the cell structure and the water content (Yun Chen et al., 2019).

Figure 4 .

 4 Figure 4.20. Using shape-based characteristics to divide the plants in two groups.

Figure 4 .

 4 Figure 4.21. Rule-based classification result using shape-based properties for plants in VIS and NIR (500 -1700 nm).

  To have a comparison to a state-of-the-art classifier, the annotated dataset was also used on a SVM-based classification approach (C-SVC, RBF-Kernel, C: 2048, gamma: 0.0883) using the LIBSVM library[START_REF] Chang | LIBSVM: A library for support vector machines[END_REF] for MATLAB. Ten-fold cross-validation was used to determine the optimized hyperparameters. Randomly 10% of the samples of each class were selected for the training dataset. The statistical values of this comparison can be found in Table4.5. The classification map of the SVM classification is shown in Figure4.23.

Figure 4 .

 4 Figure 4.22. SVM classification result for plants using VIS and NIR (500 -1700 nm).

  care must be taken to ensure that the individual steps can be generalised. This should ensure the transferability or expandability of the concept to other areas of application. Typically, all objects have a shape and a material composition. This guarantees generalisation, as 3D and HSI are in principle sensitive to features in these domains. The extraction of features from the HSI domain and their use in the context of classification was discussed in detail in chapter 4.

Figure 5 . 2 .

 52 Figure 5.2. Visualization of 3D features. Omnivariance (left), Verticality (middle) and Normal Change Rate (right) (Polat et al., 2021a).

  Figure 5.3. Acquired datasets of electronic waste consisting of waste plastics and PCBs: (a) RGB; (b) 3D point cloud with a resolution of 0.2 mm and a depth accuracy of 0.05 mm; (c) HSI with a resolution of 636 x 1118 pixels and 224 bands in the wavelength range from 900 -1700 nm (Polat et al., 2021a).

  1.746.741 points with a resolution of 0.2 mm and a maximum point-to-point distance of 0.5 mm. The specifications of the sensors and the description of the methodology used for the registration of the hyperspectral data and the 3D point cloud are described in detail in chapter 2. The combined dataset with a spectral signature for each 3D point of the point cloud is presented in Figure5.4. The visualized colour for each 3D point is the average reflectance value over the entire NIR spectrum (900 -1700 nm). High reflectivity areas are shown in red, while the transition to dark blue represents decreasing reflectivity. The dark blue areas are mainly present on the circuit boards and represent black electronic components with a very low spectral reflectance.

Figure 5

 5 Figure 5.4. Combined 3D point cloud and HSI. Each 3D point is assigned by a spectrum. In this figure, the average value over the entire spectrum is shown per 3D point (Polat et al., 2021a).

  Figure 5.5. Influence of shadow on the classification process. Here shadow occurs only on the neighbouring area surrounding the object. (a) Classification of pixels values belonging to the class PS using spectral data. (b) Classification of pixels values belonging to the class PS after removal of the background (Polat et al., 2021a).

FigureFigure 5

 5 Figure 5.6 (c).

  The shape-based analysis of spectra is one of the essential elements of the second main contribution of this PhD work, a generalized analytical method based on the combined hyperspectral point cloud. Rules based on 3D features and shape-based hyperspectral characteristics were established and the potential of this methodology was demonstrated and discussed in the context of the waste sorting use case. It has been shown that by exploiting the key properties of both sensor technologies, a maximum of efficiency and flexibility can be reached in the processing of the hyperspectral point cloud. Processes such as the removal of irrelevant areas,

FaiblessesFigure A1 .Figure A2 .Figure A3 .Figure A4 .FigureFigure A6 .Figure A7 .Figure

 A1A2A3A4A6A7 Figure A1. Continuum removed reflectance spectra and shape-based rule for PA.

Figure B3 .

 B3 Figure B3. Continuum removed reflectance spectra of campanula rotundifolia in the spectral range of 450 -1700 nm.

Figure B4 .

 B4 Figure B4. Continuum removed reflectance spectra of moss (object number 4) in the spectral range of 450 -1700 nm.

Figure B5 .

 B5 Figure B5. Continuum removed reflectance spectra of geranium robertianum in the spectral range of 450 -1700 nm.

Figure B6 .

 B6 Figure B6. Continuum removed reflectance spectra of asplenium trichomanes in the spectral range of 450 -1700 nm.

Figure B7 .

 B7 Figure B7. Continuum removed reflectance spectra of green sedum in the spectral range of 450 -1700 nm.

Figure B8 .

 B8 Figure B8. Continuum removed reflectance spectra of red sedum in the spectral range of 450 -1700 nm.

Figure B9 .

 B9 Figure B9. Continuum removed reflectance spectra of moss (object number 9) in the spectral range of 450 -1700 nm.

Figure B10 .Figure B11 .Figure C1 .Figure C3 .Figure C4 .FigureFigure C6 .Figure C7 .Figure

 B10B11C1C3C4C6C7 Figure B10. Continuum removed reflectance spectra of lichen in the spectral range of 450 -1700 nm.

Figure D2 .

 D2 Figure D2. Mobile pushbroom sensor Specim IQ.

  are also visible. Especially at a wavelength range of 1586 nm. The fact that

  -2-methyl-2H-isothiazol-3-one; 2-methyl-2H-isothiazol-3-one (3:1) 1,2-benzisothiazol-3(2H)-one; 2-methyl-2H-isothiazol-3-one (3:1) white / green 5-Chloro-2-methyl-4-isothiazolin-3-one; 2-Methyl-1,2-thiazol-3(2H)-one (3:1) 1,2-benzisothiazol-3(2H)-one; 2-methyl-2H-isothiazol-3-one (3:1)Blue and purple have a composition that is identical, as do white and green. Since the colours green and white are the examples where no blood traces can be seen and blue and purple are the colours where traces are recognizable in the corresponding wavelength ranges,

  Figure D4. RGB image of bloodstains (a); Spatial distribution of spectral intensity at 415 nm for white color (left), purple color (middle) and green / blue color (right) (b); Spatial distribution of spectral intensity at 1586 nm for white color (left), purple color (middle) and green / blue color (right) (c).

Figure D5 .

 D5 Figure D5. Representation of layers of colour. Layers 1 to 3 are the spatial representations of the averaged reflectance values over the entire wavelength range from 900 to 1700 nm for each pixel. The green colour is the representation after 3 layers of colour.

Figure D6 .

 D6 Figure D6. Reflectance spectra of soot (all pixels inside the red box).

  . ..................................................................................................................

  Figure5.9. Example of result of classification (after step 1) of plastic objects by checking average spectra for each cluster(Polat et al., 2021a). ..................................................... Figure 5.10. Result of classification for PCBs using spectral features (after step 2). Classification of 3 board types and 4 different types of components on board (PS, ecob 1, ecob 2 and ecob 3) (Polat et al., 2021a). .....................................................................

	Figure 5.11. Result of classification for PCBs using 3D features (after step 3). Classification
	of 4 different types of components on board (ecob 4, ecob 5, ecob 6 and

. .................................................................. 77

  . ............................................................................................................................Figure 5.13. Small components like metals (red and blue box), resistors (green box) and capacitors (yellow box) are difficult to distinguish because of limits in resolution (Polat et al., 2021a). ..................................................................................................................

xiii

List of Tables

Table 2 .

 2 1. Specifications of the HSI sensors used. .................................................................Table 2.2. Overview about the use cases, the used sensor technologies and the investigations regarding the use cases. .................................................................................................. Table 2.3. Results of 2D-Projective Transformation .............................................................. Table 4.1. Rules based on spectral shape characteristics for PS and PE. ................................ Table 4.2. Number of samples and training samples for each class. ....................................... Table 4.3. Classification accuracies for plastic dataset. Better results are highlighted in bold. ........................................................................................................................................ Table 4.4. Classification results for real waste dataset. Better results are highlighted in bold. ........................................................................................................................................ to take this opportunity to thank all the institutions, companies and individuals without whose help, support and guidance the preparation of this thesis would not have been possible. First and foremost, I would like to thank the state of Rhineland-Palatinate and the European Union (ERDF) for providing the financial resources within the framework of the "InnoProm -Innovation and Promotion" funding programme. I would like to thank V&R Vision & Robotics GmbH for the cooperation within the InnoProm project and would also like to thank them for their advisory and supportive technical input. My deepest thanks and gratitude go to my supervisor Prof. Alain Trémeau and co-supervisor Prof. Frank Boochs. Their support and guidance over the last 3 years have been essential to the success of my entire PhD process. Thank you for your time, patience and effort.

	Table 4.5. Classification accuracies for plant dataset using (a) Rule-based approach; (b) SVM

classification (C-SVC, RBF-Kernel, C: 2048, gamma: 0.0883). Better results are highlighted in bold. ........................................................................................................

Table 5.1. Classification accuracies for each class based on 3D and HSI data (Polat et al., 2021a). ............................................................................................................................ Table 5.2. Classification accuracies for real electronic wastes using (a) Rule-based approach with combined 3D and HSI dataset; (b) SVM classification (C-SVC, RBF-Kernel, C: 2048, gamma: 2.8284) with HSI dataset. Better results are highlighted in bold (Polat et al., 2021a). ...................................................................................................................... In addition, I would like to thank all the people who were involved in finding application scenarios and who gave me the opportunity to gain insights into practice and gave me access to samples and measurement systems. This includes employees of the Institute for Stone Conservation in Mainz (IFS), the German Federal Criminal Police Office (BKA), the Hessian State Criminal Police Office (LKA), Pellenc ST and TOMRA Systems GmbH. This thesis was mainly carried out at the Institute for Spatial Information and Surveying Technology (i3mainz) of the University of Applied Sciences Mainz, with an interim stay at the Hubert Curien Laboratory of the University Jean Monnet in St. Etienne. I would like to thank all those who have supported me during this PhD process and provided me with access to laboratories and equipment. My special thanks go to Prof. Philippe Colantoni, who gave me access to hyperspectral sensor technology at the Hubert Curien Laboratory, and Prof. Kai-Christian Bruhn, whose support made it possible to set up a hyperspectral lab at the i3mainz.

  La nature est une source d'inspiration qui a servi de base à de nombreuses innovations technologiques. Que ce soit la chauve-souris, source d'inspiration pour les capteurs de stationnement basés sur les principes du sonar ou la morphologie des ailes d'oiseaux ayant inspiré la construction d'avions, la nature sert de modèle. L'inspiration pour ce travail peut également provenir de la nature, plus précisément de l'homme. Les humains utilisent une combinaison de différents sens pour percevoir leur environnement et prendre des décisions basées sur leurs connaissances[START_REF] Elmenreich | An Introduction to Sensor Fusion[END_REF]. Par exemple, l'évaluation de la qualité d'un aliment repose sur une combinaison de sens : la vue, l'odorat et le goût. Une évaluation purement visuelle de l'aspect extérieure peut conduire à un jugement erroné (ex des illusions d'optiques, trompes de la méthode proposée est la fusion d'ensembles de données et l'extraction ultérieure de caractéristiques spectrales et spatiales. Dans ce contexte, l'extraction des caractéristiques spectrales est effectuée à l'aide d'une approche nouvelle utilisant les propriétés locales des courbes spectrales basées sur la forme, telles que les points de courbure, les valeurs de courbure, la direction de la courbure et les valeurs spectrales. L'avantage de la prise en compte de la forme locale, en particulier lorsqu'elle est associée à des données spectrales à haute résolution, est évidente : les spectres sont l'expression de la réflectivité mixte des éléments constitutifs d'un objet donné (par exemple, les molécules, les pigments, la structure cellulaire, la teneur en eau), ce qui signifie que le spectre global peut être vue comme une somme pondérée des spectres de ces constituants et que chaque élément n'a qu'une influence proportionnelle à son poids dans la combinaison. Les modifications de la composition spectral des éléments individuels modifient ensuite principalement le mélange de toutes les contributions spectrales, ce qui entraîne des variations locales ou régionales et change la forme d'un spectre. de caractéristiques générales se poursuit jusqu'à ce qu'il atteigne un point où des éléments spécifiques à l'application doivent être utilisés. Ainsi, assurer un processus de classification général consiste à maximiser le nombre de caractéristiques générales valables et à identifier un maximum d'aspects communs entre tous les objets d'intérêt.Un processus de classification permet de structurer des objets présentant certains traits caractéristiques et de les regrouper. Une base de données structurée présente l'avantage d'obtenir une meilleure compréhension de la base de données et donc de simplifier le processus de résolution des problèmes dans différents domaines d'application. En général, tous les objets ont une forme spécifique et une composition matérielle. Cela garantit une généralisation, car la 3D et l'HSI sont en principe sensibles aux caractéristiques de ces domaines. Le fait que la valeur ajoutée des informations 3D varie en fonction de l'application doit également être pris en compte et est démontré par divers cas d'utilisation.

l'oeil) et ce n'est qu'en combinant avec l'odorat ou le goût qu'il est possible de déterminer réellement si un aliment est encore comestible ou non. La combinaison de différents sens permet donc une décision plus fiable. Cet aspect peut également être transposé aux capteurs. Les exigences de qualité croissante dans divers domaines de l'industrie exigent des technologies qui répondent à ces normes de haute qualité. La combinaison de capteurs offre un potentiel d'augmenter les performances des systèmes. L'idée de base est le soutien mutuel et la capacité d'une source à compenser les faiblesses d'une autre. La combinaison de sources d'information hétérogènes et la diversité des informations qui en résulte peuvent aider à mieux comprendre les données sous-jacentes et à optimiser des processus tels que l'extraction de caractéristiques, la classification et la segmentation

[START_REF] Snidaro | Data Fusion in Modern Surveillance[END_REF]

.

Dans cette thèse, nous adoptons l'idée de la fusion de capteurs et proposons un concept de capteur optique basé sur une combinaison des technologies de capteurs 3D et hyperspectraux afin de fournir une meilleure compréhension du phénomène étudié. Ici, l'un des objectifs est de développer un système qui puisse être généralisé à différents cas d'étude. La technologie des capteurs 3D fournit des informations topologiques, de profondeur et spatiales tandis que l'imagerie hyper-spectrale (HSI) fournit des informations physiques. Ensemble, ces sources fournissent un ensemble de données détaillées, riches en information et d'une grande valeur pour un large éventail de questions relevant de différents domaines d'application. On peut citer notamment les études de

[START_REF] Roscher | Detection of disease symptoms on hyperspectral 3D plant models[END_REF]

et

[START_REF] Morsy | Multispectral LiDAR Data for Land Cover Classification of Urban Areas[END_REF]

. Ces deux travaux montrent les avantages de la combinaison des données géométriques et spectrales. Pour donner un exemple, le travail de

[START_REF] Roscher | Detection of disease symptoms on hyperspectral 3D plant models[END_REF]

a utilisé la géométrie en termes d'angles d'inclinaison en combinaison avec des données spectrales pour la détection précoce des symptômes de maladie sur les plantes, tandis que dans

[START_REF] Morsy | Multispectral LiDAR Data for Land Cover Classification of Urban Areas[END_REF]

une combinaison de données LiDAR et multi-spectrales (MSI) a été utilisée pour classifier la couverture terrestre d'une zone urbaine. xxii Le concept général Les changements locaux dans le voisinage spatial d'un objet sont également un élément clé en termes d'extraction de caractéristiques spatiales dans le contexte des nuages de points 3D. Les nuages de points sont une représentation d'informations 3D. Ils permettent l'extraction de caractéristiques géométriques et topologiques qui peuvent être utilisées pour décrire certaines propriétés des objets et des scènes. Ces propriétés sont dérivées des relations géométriques locales entre les différents points 3D. Cela signifie que les objets/scènes peuvent être décrits sans ambiguïté sur la base de leurs propriétés, telles que leur linéarité, leur planéité ou leur changement de courbure. Toutes les caractéristiques spectrales et spatiales extraites sont intégrées dans une approche basée sur des règles de décision pour classer les objets d'intérêt. Le principal avantage des classificateurs à base de règles de décision est que les règles de classification peuvent être établies très facilement. Des conditions issues des connaissances acquises par une analyse préalable de la base de données peuvent être établies et corrélées pour la formulation de règles. Les autres avantages sont la performance, la capacité à traiter les attributs redondants et non pertinents, et l'extension flexible des ensembles de règles (Tan et al., 2019). Une attention particulière est et doit être accordée à la généralisation de la tâche de classification et à la possibilité de couvrir différents types d'applications. Les caractéristiques des objets sont utiles pour diviser un ensemble de données en groupes ayant des analogies et pour les classer en conséquence sur la base de propriétés de groupe prédéfinies. Pour assurer la généralisation de ce processus, les caractéristiques pertinentes pour décrire les propriétés des groupes doivent être soigneusement sélectionnées et passer par un processus itératif et flexible jusqu'à la classification finale. En d'autres termes, il est d'abord nécessaire de réfléchir au type d'objets à reconnaître et à ce qu'ils ont en commun afin d'obtenir un premier ensemble générique de caractéristiques applicables à différents cas d'utilisation en commun. Ce processus d'identification PLAN DE LA THÈSE Cette thèse étudie l'utilisation combinée de données 3D et hyper-spectrales. Elle démontre également le potentiel et la valeur ajoutée d'une approche combinée à travers diverses applications. Un accent particulier est mis sur l'identification et l'extraction de caractéristiques dans ces deux domaines et l'utilisation de ces caractéristiques pour la détection d'objets d'intérêt. Le chapitre 2 donne un aperçu des technologies 3D et d'HSI existantes, des propriétés caractéristiques de ces technologies, et des méthodes requises pour combiner ces deux ensembles de données. Les systèmes de détection utilisés et les combinaisons possibles en fonction des cas d'utilisation choisis sont présentés et illustrés en termes d'approche. Le chapitre 3 contient une revue de la littérature sur les méthodes de pré-traitement, d'extraction de caractéristiques et de sélection de bandes significatives en HSI. En outre, un aperçu des approches existantes pour la classification des données multidimensionnelles est fourni. Le chapitre 4 constitue la première contribution principale de cette thèse. L'approche, basée sur la forme locale, proposée pour analyser les signatures spectrales en combinaison avec une approche de classification, basée sur des règles de décision, est décrite et démontrée sur la base d'exemples pratiques. L'utilisation combinée de données 3D et hyper-spectrales fait partie du chapitre 5. Ce chapitre correspond à la deuxième contribution principale de cette thèse. Les avantages d'une approche combinée sont démontrés dans ce chapitre à l'aide d'un exemple pratique issu du domaine du tri de déchets électroniques et plastiques. Avec une sélection de différentes applications, il est démontré que la valeur ajoutée des données 3D dépend principalement des cas d'utilisation sélectionnés. xxiv Le chapitre 6 conclut sur les résultats concernant le potentiel de l'utilisation combinée de la 3D et de l'HSI. Il fournit également une perspective sur les orientations futures de cette recherche.

Table 2

 2

	.1. In particular, they were selected under consideration of performance features like
	the full spectral range, spatial and spectral resolution, field of view (FOV), system complexity,
	signal o noise ratio (SNR), wavelength selectability and others. The use cases addressed in this

Table 2 .1. Specifications of the HSI sensors used. Specim FX10 Specim FX17 Senop HSC-2

 2

	Sensor Type	Pushbroom Scanning	Pushbroom Scanning	Spectral Scanning
		line-based	line-based	frame-based
	Spectral range	400 -1000 nm	900 -1700 nm	500 -900 nm
	Spectral resolution	5.5 nm	8 nm	5 -8 nm
	(FWHM)
	Spectral bands	224	224	Up to 1000 bands
	Spatial resolution	1024 pixels per line	640 pixels per line	1024 x 1024 pixels
	Image speed	330 FPS (full range)	670 FPS (full range)	149 FPS
	Field of View (FOV)	38°	38°	36.8°

Table 2 .

 2 2. Overview about the use cases, the used sensor technologies and the investigations regarding the use cases.

Table 2 .

 2 3. Results of 2D-Projective Transformation

	Parameter	a	b	c	d	e	F	g	h
	Estimated	0.372	0.004	-2.1e-06	0.000	-0.374	-1.7e-05	-29.365	141.750
	Accuracy	0.002	0.001	8.9e-06	0.001	0.001	8.2e-06	0.282	0.241

Table 4

 4

	: -.	< -. AND	: -.	< -. AND
	: -.	< -. AND	: -.	< -. AND
	: -.	< -. AND	: -.	< -. AND
	: + .	> + . AND	: + .	> + . AND
	: + .	> + . AND	: + .	> + .
	: + .	> + .
	→

.1. Rules based on spectral shape characteristics for PS and PE.

Table 4 .

 4 2. Number of samples and training samples for each class.

		Class	Number of Samples	Number of Training Sam-ples
		PA	16276	3783
		PE	16212	3659
	Homogenous plastic dataset	PF-Black PMMA PS	11157 15779 16877	4930 4071 4123
		PVC	16645	4869
		UP	16098	4613
		PA	13343	2114
	Real waste	PS	81238	10444
	dataset	PP	116620	11936
		ABS	11703	1638

Table 4 .

 4 3. Classification accuracies for plastic dataset. Better results are highlighted in bold.

	Metrics	Rule-based Method	SVM (a)	SVM (b)	SVM (c)
	Overall Accuracy	0.9694	0.9647	0.9641	0.9641
	Precision	0.9604	0.9731	0.9403	0.9403
	Sensitivity	0.9551	0.9272	0.9623	0.9623
	False Positive Rate	0.0077	0.0114	0.0076	0.0076
	F1-Score	0.9564	0.9487	0.9503	0.9503
	Kappa	0.8603	0.8384	0.8358	0.8359

Table 4 .

 4 4. Classification results for real waste dataset. Better results are highlighted in bold.

	Metrics	Rule-based Method	SVM (a)	SVM (b)	SVM (c)	SVM (d)	SVM (e)
	Overall Accuracy	0.9842	0.9770	0.9826	0.9805	0.9760	0.9586
	Precision	0.9842	0.9398	0.9698	0.9807	0.9856	0.9854
	Sensitivity	0.9498	0.9477	0.9434	0.9353	0.9208	0.8831
	False Positive Rate	0.0075	0.0093	0.0098	0.0119	0.0152	0.0267
	F1-Score	0.9661	0.9396	0.9540	0.9556	0.9504	0.9290
	Kappa	0.9506	0.9283	0.9456	0.9392	0.9249	0.8707

(d) Solver: C-SVC; Classification Type: One-Versus-All; Kernel: RBF; C-Value: 0.2; Threshold: 0.2.

Table 4 .

 4

5. Classification accuracies for plant dataset using (a) Rule-based approach; (b) SVM classification (C-SVC, RBF-Kernel, C: 2048, gamma: 0.0883). Better results are highlighted in bold.

Metrics (

a) Rule-based (b) SVM

	Overall Accuracy	0.9274	0.9800
	Precision	0.6384	0.9312
	Sensitivity	0.7055	0.9024
	False Positive Rate	0.0207	0.0071
	F1-Score	0.6518	0.9164
	Kappa	0.7037	0.9184

Table 5 .

 5 1. Classification accuracies for each class based on 3D and HSI data(Polat et al., 2021a).

	Class	OA of Single	Error of Single	F1-Score	Kappa
	PA	0.9969	0.0030	0.9975	0.9584
	PP	0.9961	0.0380	0.9966	0.8781
	PS	0.9861	0.0138	0.9794	0.9723
	Board type 1	0.9075	0.0924	0.9327	0.7950
	Board type 2	0.9560	0.0439	0.9755	0.9628
	Board type 3	0.9689	0.0311	0.9789	0.9325
	ecob 1	0.8936	0.1063	0.9240	0.9947
	ecob 2	0.7206	0.2793	0.7594	0.9947
	ecob 3	0.7764	0.2235	0.7948	0.9970
	ecob 4	0.9849	0.0150	0.9094	0.9745
	ecob 5	0.8100	0.1899	0.7931	0.9848
	ecob 6	0.7114	0.2885	0.7127	0.9890
	ecob 7	0.6595	0.3404	0.5831	0.9981

Table 5 .

 5 2. Classification accuracies for real electronic wastes using (a) Rule-based approach with combined 3D and HSI dataset; (b) SVM classification (C-SVC, RBF-Kernel, C: 2048, gamma: 2.8284) with HSI dataset. Better results are highlighted in bold(Polat et al., 2021a).

	Metrics	(a) Rule-based (3D + HSI)	(b) SVM (HSI)
	Overall Accuracy	0.9824	0.8865
	Precision	0.8812	0.7812
	Sensitivity	0.8834	0.6576
	False Positive Rate	0.0023	0.0124
	F1-Score	0.8810	0.6956
	Kappa	0.8672	0.2011

  Dans cette thèse, nous avons étudié le potentiel de la combinaison de capteurs 3D et d'HSI dans le cadre de différents domaines d'application. Plusieurs points, à savoir l'acquisition des données, la fusion des données, l'analyse des signatures spectrales et le traitement de nuages de points hyperspectraux combiné ont été abordés dans ce but. Tout d'abord, différents concepts généraux pour l'acquisition et la fusion des données ont été présentés en fonction de l'application visée. Dans le cadre d'applications en laboratoire, des capteurs hyperspectraux à balayage ont été utilisés en combinaison avec un scanner à lumière structurée pour produire des nuages de points hyperspectraux. En revanche, pour les mesures en extérieur, le concept de fusion basé sur un système de balayage hyperspectral en combinaison avec la méthode SFM a été présenté. Les deux concepts de fusion de données sont généraux et suivent une approche simple qui peut être facilement transférée à d'autres applications pratiques. Deuxièmement, de nouvelles méthodes d'analyse des signatures hyper-spectrales et de traitement des nuages de points hyperspectraux ont été présentées. Ces nouvelles méthodes développées sont les principales contributions de cette thèse. Les méthodes établies pour l'analyse des signatures spec-

à l'utilisation combinée des données 3D et hyper-spectrales. Ils visent également à démontrer le potentiel et la valeur ajoutée d'une approche combinée dans le contexte de différentes applications. Une attention particulière est accordée à l'identification et à l'extraction de caractéristiques dans les deux domaines et à l'utilisation de ces caractéristiques pour détecter des objets d'intérêt. Plus spécifiquement, nous proposons différentes approches pour combiner les données 3D et hyper-spectrales en fonction des technologies 3D et d'imagerie hyper-spectrale (HSI) utilisées et montrons comment chaque capteur peut compenser les faiblesses de l'autre. De plus, une nouvelle méthode basée sur des critères de forme dédiés à la classification de signatures spectrales et des règles de décision liés à l'analyse des signatures spectrales a été développée et présentée. Les forces et les faiblesses de cette méthode par rapport aux approches existantes sont discutées. Les expérimentations réalisées, dans le domaine du patrimoine culturel et du tri de déchets plastiques et électroniques, démontrent que la performance et l'efficacité de la méthode proposée sont supérieures à celles des méthodes de machines à vecteurs de support (SVM). En outre, une nouvelle méthode d'analyse basée sur les caractéristiques 3D et hyperspectrales est présentée. L'évaluation de cette méthode est basée sur un exemple pratique du domaine des déchet d'équipements électriques et électroniques (WEEE) et se concentre sur la séparation de matériaux comme les plastiques, les carte à circuit imprimé (PCB) et les composants électroniques sur PCB. Les résultats obtenus confirment qu'une amélioration des résultats de classification a pu être obtenue par rapport aux méthodes proposées précédemment. L'avantage des méthodes et processus individuels développés dans cette thèse est qu'ils peuvent être transposé directement à tout autre domaine d'application que ceux investigué, et généralisé à d'autres cas d'étude sans adaptation préalable.

www.senop.fi

www.ifs-mainz.de

www.mosaicmill.com

Plastic samples were kindly provided by PlasticsEurope Deutschland e.V. (www.plasticseurope.org).

Real waste samples were kindly provided by Pellenc ST (www.pellencst.com).

www.mathworks.com

Acknowledgements

CHAPTER 5: COMBINED USE OF SPATIAL AND SPECTRAL INFORMATION

The potential of HSI analysis and processing is central subject of chapter 4 and has been discussed on the basis of various use cases. However, HSI also has its limitations in certain situations. Here, for example, the strong absorption of light from the UV to the SWIR of dark or black materials [START_REF] Dvorak | Development of NIR Detectable Black Plastic Packaging[END_REF][START_REF] Rozenstein | Development of a new approach based on midwave infrared spectroscopy for post-consumer black plastic waste sorting in the recycling industry[END_REF] and the influence of shadows on the quality of the hyperspectral data processing should be mentioned. A possibility to deal with these weaknesses will be proposed in this chapter and based on a combination of 3D geometry and HSI. 3D sensors allow to derive topological, depth and spatial information from their data. Combining this valuable information with physical information provided by HSI can lead to higher analysis potential and improve processes such as classification, segmentation or even scene understanding. In the same way as HSI sensors, 3D sensors also have their limitations. For example, a structured light scanner has difficulty detecting transparent objects, whereas HSI sensors are not affected by this. The idea of such a combination is therefore the mutual support among the used sensor technologies to compensate for existing weaknesses and thereby be able to generate more effective solutions to problems.

Figure 5.1. Workflow consisting of data acquisition, data fusion, the use of 3D data to simplify and structure the data basis and the final classification based on obtained HSI and 3D features (Polat et al., 2021a).

CONCLUSION (EN FRANÇAIS)