Specular reflection detection and removal is a fundamental yet non-trivial problem in the image processing domain, including applications for segmentation, object detection and decision-making systems. Most systems overlook the particular scenario and ignore input images with specular highlights instead of mitigating them in the pre-processing stage. This work presents techniques developed for accurately segmenting specular regions in real-world images and generating specularity-free images from a single image input without any additional guidance or parameters.

For reliable specularity detection we developed an efficient Specularity Segmentation (SpecSeg) deep neural network based on the U-net architecture. SpecSeg has a fast inference time of 3.1ms and can be trained in only 40 minutes. We also develop a fast colour Weighted Median Inpainting (WMI) method to quickly inpaint large regions of affected specular regions with approximated colour. For specular mitigation, we developed a multi-domain Specular Highlight Mitigation Generative Adversarial Network (SHMGAN) trained using multiple polarimetric images, for synthesizing specularity-free images from a single image input. We take advantage of the inherently polarized nature of specular highlights and varying illumination information captured using polarizer filters. No external label or additional input is required for the removal of specularity as the SHMGAN network uses a dynamically generated self-attention mask for detecting specular regions. Both networks are trained and tested on self-acquired and publicly available datasets of real-world images. The images generated after specular mitigation are realistic and have minimal noise, distortions and aberrations compared to the existing state-of-the-art methods.

First and foremost, I would like to thank Almighty Allah for giving me the strength and resolve to accomplish this work with perseverance, due diligence and determination. I would like to express my deepest gratitude to Prof. Samia Ainouz for her invaluable feedback, guidance and patience throughout my PhD. I would like to thank her for having faith in me and my work this entire time and for helping me adapt and adjust to the new experiences in France. She also generously provided knowledge and expertise on the technical and non-technical sides of the research and helped me overcome the difficulties faced. This endeavour would not have been possible without Prof. Fabrice Mériaudeau, whose guidance, encouragement and continuous support throughout these years have been essential to the existence of this thesis. I have had the luck and honour to be under his guidance since my Master's at UTP. I will forever remain indebted to his invaluable suggestions, wisdom and enlightening discussions leading to this thesis and hopefully in the future as well.

I would like to express my utmost appreciation and gratitude to my supervisors, Assoc. Prof. Naufal Bin. M. Saad and Assoc. Prof. Syed Saad Azhar Ali. Their guidance, never-ending dedication, motivation and cooperation throughout this period have been essential to this thesis. I would like to extend my sincere thanks to Assoc.

Prof. Nidal Kamel guided me at the start of my PhD and introduced me to in-depth mathematical concepts that paved the way to achieve the current results.

Being part of a cotutelle between UTP Malaysia and INSA France allowed me the opportunity to have splendid experiences halfway around the world, familiarizing iii myself with different cultures, learning foreign languages, exploring cuisines and most importantly, meeting new friends and colleagues. This experience has had an incredibly positive effect and widened perspectives on a personal level. To this end, I would like to extend my sincere thanks to the administrative staff for their help, cooperation and support. In particular, Brigette and Sandra (INSA) and Mr Fahmi (UTP) deserve special mention for facilitating and expediting all administrative processes and queries.

I am also thankful to Vitor Ramos for his insights, thoughtful discussions and sharing of ideas related to the research topic at the very early stages of my PhD, as they gave me confidence and guidance to orient the research correctly during the early stages. I would like to extend my sincere thanks to Amjad Khan for his thoughtful discussions, late-night feedback sessions, and moral support. I consider myself lucky to have had his acquaintance and friendship! I also had the pleasure of working with Cyprian Ruffino, who helped me immensely with his expertise in solving technical errors during the implementation stage.

My heartfelt thanks go to all my fellow lab mates and friends, the faculty members and colleagues at both LITIS(INSA) and CISIR(UTP) for their help, support and the great times we had together. The trips to the waterfalls in Perak and the laser tag and bowling sessions at River Droit hold a special place in my heart and memory. Special mention and thanks, in no particular order, are due to all CISIR colleagues and friends, including Khaleel, Alam, Danish, and Dr Khurram Altaf for their moral and technical support and for making Malaysia a home-away-from-home, especially during the lockdown. At INSA, I would like to thank Maël, Matthias, Sandratra, Henrique, Matthieu, Imane, Robin, Tsiry, Amit, Dhruv, Mukesh and Usman for the thoughtful technical discussions and inspiring contemporary pop-culture insights, bearing with my poor french and guidance to adapt to the new environment throughout this period.

And last but not least, I would like to thank and humbly dedicate this thesis to my mother (late) and father. No words describe how grateful I am for their selfless love, prayers, efforts, wishes, wisdom, sacrifices and support. I would also like to thank my brother and wife for their patience, never-ending moral support and encouragement during my studies. And especially to my daughter for the continuously revitalizing and driving motivation to complete this PhD.

Resume

La détection et la suppression des reflets spéculaires est un problème fondamental mais non trivial dans le domaine du traitement d'images, y compris dans les applications pour la segmentation d'images, la détection d'objets et les systèmes de décision basés sur l'image. La plupart des systèmes négligent ce scénario particulier et ignorent les images d'entrée présentant des reflets spéculaires au lieu de les atténuer au stade du prétraitement. Ce travail présente des techniques développées pour segmenter avec précision les régions spéculaires dans les images du monde réel et générer des images sans spécularité à partir d'une seule image d'entrée sans aucune indication ou paramètre supplémentaire. Pour une détection fiable de la spécularité, nous avons développé un réseau neuronal profond efficace de segmentation de la spécularité (SpecSeg) basé sur l'architecture U-net. SpecSeg a un temps d'inférence rapide de 3.1ms et peut être entraîné en seulement 40 minutes. Nous développons également une méthode rapide de peinture médiane pondérée (WMI) en couleur pour peindre rapidement de grandes régions spéculaires affectées avec une couleur approximative. Pour l'atténuation des effets spéculaires, nous avons mis au point un réseau adversarial génératif multi-domaines (SHMGAN) entraîné à l'aide de plusieurs images polarimétriques, afin de synthétiser des images sans spécularité à partir d'une seule image. Nous tirons parti de la nature intrinsèquement polarisée des reflets spéculaires et des informations d'illumination variables capturées à l'aide de filtres polarisants. Aucune étiquette externe ou entrée supplémentaire n'est requise pour la suppression de la spécularité car SHMGAN utilise un masque d'auto-attention généré dynamiquement pour détecter les régions spéculaires. Les deux réseaux sont entraînés et testés sur des ensembles de données d'images du monde réel acquises par les utilisateurs eux-mêmes et accessibles au public. Les images générées après l'atténuation de la spécularité sont réalistes et présentent un bruit, des distorsions et des aberrations minimes par rapport aux méthodes de pointe existantes.

xi

List of Tables

4.11

Visual comparison of testing on the PSD dataset [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. The methods compared include both traditional image processing techniques [START_REF] Shen | Chromaticity-Based Separation of Reflection Components in a Single Image[END_REF][START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF][START_REF] Yang | Real-Time Specular Highlight Removal Using Bilateral Filtering[END_REF] and modern GAN based methods [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF][START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] [START_REF] Shen | Chromaticity-Based Separation of Reflection Components in a Single Image[END_REF][START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF][START_REF] Yang | Real-Time Specular Highlight Removal Using Bilateral Filtering[END_REF] and modern GAN based methods [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF][START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. . . . . . . . . . . .

Digital imaging and specular reflections

"Light is the colour of the translucent", a rather ambiguous definition of light as a natural phenomenon, given by Aristotle in his book On Sense and the Sensible in 350 BC. He believed that light was one of his four elements that composed matter and was an essential property of various substances when subject to any reaction.

Aristotle based his theories on his greek predecessors, Empedocles, who had proposed that light streaming out of our eyes and touching objects caused human vision. While flawed, this theory of treating light as rays became the fundamental hypothesis on which later philosophers and mathematicians would construct some of the most important discoveries in the fields of light, vision, and optics. In the 11th century, Arab scientist Ibn-al-Haytham (also known as Alhazen) pioneered the camera obscura or pinhole camera by generating a flipped image based on the prevalent theory that light travels in a straight line. This became one of the most significant concepts in all optics and imaging domains, inspiring research by many notable inventors and visionaries, including Leonardo Da-Vinci in the 15th century, leading up to the modern era of digital imaging. During this time, an alternate approach was also put forward with treatment of light as a wave as opposed to a beam of particles travelling in a straight line. This alternate treatment allowed the explanation of some basic phenomena of light that were not explainable, considering it as particulate in nature. Properties such as double refraction, diffraction and bringing behaviours, among others, were only possible if light was treated as a wave. The wave theory was initially established by Augustin Fresnel in the early 18th century, leading to the proposal of the electromagnetic theory by James Clerk Maxwell in 1876. Although wave theory is generally correct when light propagation is described (and of other electromagnetic waves), it fails when other light properties are to be explained, especially the interaction of light with matter. In modern physics, this duality of the nature of light is now widely accepted and used as deemed feasible.

Explaining how light travels through a medium and interacts with materials upon striking a surface has been a prime area of understanding of classical physics and a precursor for all photography and imaging research. Because of the physical nature of light, there are two fundamental properties when interacting with any object, refraction and reflection. refraction of light is the change in the direction of light occurring at the boundary of the medium it strikes as it passes through it, whereas However, the particle theory additionally suggests that if the surface is very rough, the light particles reflect back at varying angles scattering the light. This fits very closely to experimental observation and makes the case for a particulate nature of light far more substantial with regards to the reflection phenomenon than it is for refraction.

As will be discussed in the subsequent chapters, both these concepts play a pivotal role in shaping the different models for reflection of light. However, to generalise the concept, the reflection of light can be roughly categorised into two types of reflections, specular reflection and diffuse reflection, which are both explored in depth in the subsequent section.

Specular and diffuse reflection components

Reflection is the inherent property of all waves, whether electromagnetic (such as light) or particulate (such as sound) in nature. In the case of visible light, which exhibits both electromagnetic wave and particulate nature, when it strikes a surface, three distinct possibilities can occur simultaneously but in varying proportions, depending on several factors. One possibility is that all or part of the incident light is transmitted through the material. Secondly, it can be absorbed by the material or lastly, all or part of the light can be reflected back to the incident medium from the surface of the surface. Several factors decide the ratio of each of these possibilities that applies to the incident light. The major factors include the material of the surface, the material of the body and the angle of incident light about the surface normal.

Specular highlights 1 are the shiny gloss and reflections in an image caused due to light bouncing directly off the body surface without interacting with the material of the surface or the material underneath the surface. Generally, the term 'specular reflection' defines the mirror-like reflection of waves from any surface about the surface normal, as shown by the Figure 1.2. Due to this, the colour of specular reflection generally represents the colour of the illuminant light source as it is reflected to the imaging device without undergoing any interaction. In contrast, diffuse reflections are the rays of light that undergo two types of interactions that work according to the law of reflection. Firstly, depending on the material of the surface, some of the light rays penetrate below the surface and interact with the material of the object before being reflected back to the viewer. Due to the interaction of light with the material of the object, the diffuse light represents the colour of the object. Secondly, if the surface is rough, some light rays are dispersed during the reflection phase due to the surface roughness represented by the micro-facets. This causes the light to reach the viewer or sensor in a weaker diffused state, i.e. the rays scatter.

Polarization and Specular Reflection

Specular reflection components of light are usually strong in intensity and significantly polarised compared to the diffuse component of light [START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF]. This makes Circular Polarizer (CPL) filters an effective way to remove specular highlights in traditional photography manually. A CPL is basically a linear polarizer that can be rotated manually to remove reflections or haze prior to taking the picture. By using a rotating linear polariser, we can actively cancel out the polarised specular highlights for a particular polarizer angle ϕ pol , partially solving the problem of specular highlight mitigation.

Normal light from most sources is unpolarised and has equal irradiance in all directions. Unpolarized light specularly reflected from a reflective surface becomes partially polarized [START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF]. The angle of polarisation is variable as it depends on the surface orientation as well as the orientation of the illuminating source. Therefore, the specularly reflected light wave is a combination of polarised and unpolarised com-

I s = I sc + I sv (cos 2 ϕ pol -α ) (1.1)
The intensity I of each pixel p is linked to the AoP (α), DoP (ρ) and polarizer angle (ϕ pol ) by eqn. 1.2.

I p (ϕ pol ) = I 2 I t ot al 1 + ρ cos 2ϕ pol -2α (1.2)
For an image observed through a polarizer filter, the light intensity fluctuates sinusoidally as a function of the polarizer angle (ϕ pol ), where the peak of the sinusoid is the maximum intensity of light I max , and the sum of the diffuse reflection, polarised (I sv ) and unpolarised (I sc ) specular highlight [START_REF] Wolff | Polarization-Based Material Classification from Specular Reflection[END_REF] as shown in fig. 2.6. It is important to note that the unpolarised specular reflection components (I sc ) pass through unhindered through a polariser filter, as is also visible in the Figure 1.3. This is why a single polariser filter cannot completely mitigate specular reflections, and additional methods are still required for mitigation.

Research Motivation

Specular highlights have especially gained importance for image processing since the advent of digital image acquisition sensors. They are a highly informative feature as they convey photometric properties and are used in determining shape of objects [START_REF] Klinker | A Physical Approach to Color Image Understanding[END_REF], surface orientations [4] and estimation of illumination chromaticity [START_REF] Tan | Separation of Highlight Reflections on Textured Surfaces[END_REF] etc. Camera technology has come a long way since these times, but the fundamental physics behind cameras has not changed all that much. Besides the addition of lenses to focus light and to replace the wall of the camera obscura with light-sensitive materials to capture the photograph, the concept that light travels through a transmission medium in a straight line still applies. Modern cameras replace the imaging medium and photography films with imaging sensors that are light-sensitive counterparts of traditional photography films. These sensor pixels are prone to saturation when exposed to very strong lighting such as the ones constituted by specular reflections, resulting in sensor clipping. When the sensor pixels saturate, they non only lose the colour and textural information below the specular pixels but also cause discontinuities in the image leading to extremely bright patches in the image where the specular reflection occurs. These discontinuous regions pose a significant challenge for computer vision algorithms which is why most algorithms have to ignore all cases where there is a specular reflection in the image. However, ignoring specular highlights is not a feasible solution with the utility of modern computer vision algorithms in a wide variety of real-world applications. These include applications where human life is directly impacted, such as autonomous vehicles and life-saving medical imaging applications. Mitigation of specular highlights is inherently an undetermined system and thus has a non-trivial solution, requiring research to develop rigorous and robust solutions.

Problem formulation

The objectives of a specular highlight mitigation method is ideally two-fold. Firstly, the aim is to accurately and correctly identify the specular pixels in an image; without being affected by the shape, texture or colour of the underlying objects. Secondly, and most importantly, the 'Mitigation' part is to recover the underlying object's diffuse colour and texture as close to the actual object as possible. As will be explored in detail in sections 2.

Problem statements

1) Existing specular highlight detection methods are unable to detect and segregate the specular reflections accurately from images taken in uncontrolled environments.

2) In real-world images, the results of the recovered diffuse image are often not representative of the original diffuse colour of the body, and there is an evident loss of colour information, especially in areas of strong specular reflections.

Research Questions

There are three main questions tackled by this research work that are enumerated below:

1) How can we accurately separate specular pixels in any real-world image? Segmenting specular pixels is a non-trivial problem due to their similar nature to lighter coloured regions and other brightly-lit areas and the developed techniques would be required to precisely give repeatable results in a wide variety of scenarios.

2) How can modern polarimetric cameras with on-sensor polarizer filters be utilized to find a robust and efficient specular mitigation method?

3) What are the most effective methods that can be explored and utilized for specular highlight mitigation? Can traditional filtering or inpainting methods sufficiently recover the affected colour information or can we utilize state-ofthe-art generative adversarial networks to provide an effective and robust solution?

Research Objectives

This work is aimed to contribute to this need for specular highlight segmentation and mitigation by accurately segmenting and mitigating the specular highlights from real-world images. As will be explored in the in-depth literature review in Chapter 2, utilisation of conventional image processing methods is not favourable for a robust solution that applies to a wide assortment of real-world images. This leads to exploring and developing deep-learning-based solutions for more widespread applicability. Thus the main objectives of this thesis can be defined as follows:

• Objective 1: To develop a deep learning-based segmentation network for highly accurate detection of specular highlights in real-world images at near real-time performance.

• Objective 2: To utilize polarimetric imaging and leverage specular highlight polarization properties to learn accurate diffuse colour recovery.

• Objective 3: To develop a deep learning-based image translation network for mitigating the detected Specular highlights and generating specularity-free images from a single input image.

Hypothesis

As specular highlights are polarized in nature, utilizing polarimetry to learn the true diffuse colour is the primary way that specular reflections can be mitigated naturally. This makes polarimetric solutions as a very viable go-to method that is accurate and physically plausible. The research gap in mitigating specular highlights can be addressed using classical methods as well as modern deep learning approaches.

There are benefits and trade-offs of selecting one method over the other, such as computational power required, training data requirements and most importantly, the robustness of the methods to meet the end objectives. While deep learning based methods provide several benefits over the classical computer vision methods, it is still worth it to explore the problem from a classical standpoint as some solutions can be reasonable for meeting the relaxed requirements of some problems. To summarize, the three main hypothesis of the thesis can be enumerated as follows:

H1: We can develop a lean deep-learning network for specular pixel detection and segmentation that is fast to train and is able to accurately detect specular pixels in any real-world image.

H2: An initial mitigation solution can be based on a fast and simple inpainting method on the affected area by utilizing the colour information of the surrounding border pixels of the detected specular highlight.

H3:

Utilizing the state-of-the-art generative adversarial networks, we can develop a multi-domain image-to-image translation network, capable of generating specular-free images from a single input RGB image by learning the illumination variation in polarimetric images.

Research Contributions

Thesis organisation

A visual representation of the flow of the thesis can be given by the flow chart in Fig.

Chapter 2:

The various physical models of light reflection such as the Torrence-Sparrow and Dichromatic Reflection Model (DRM) are explained in detail along with other relevant core concepts. In particular, the phenomenon of polarisation is discussed in depth as it is of special importance due to the highly polarised nature of specular highlights. The disadvantages and issues caused by specular highlights are discussed forming the core justification for the developed work All prior literature focused on detecting specular highlights is reviewed in depth, including both classical image processing methods as well as deep learning-based solutions available.

Mitigation of specular reflections is the core focus of this thesis and the prior literature over the years that is relevant to specular highlight removal is explored in depth, including both classical and deep learning-based state-of-the-art methods.

Lastly, multi-domain and multi-modal generative adversarial networks are briefly

Chapter Abstract

In this chapter, we set the basis of diffuse colour and specular reflection from a physical and mathematical standpoint.

Physical Model of Light Reflection

To understand and model the reflections mathematically, several different models have been proposed over the years, with varying complexity and application areas in mind. Some earlier works were proposed with more ideal assumptions, whereas several physics-based models were later developed that, under nominal assumptions, are able to describe the interaction of light following the laws of physics.

Models such as Lambertian, Phong, Cook-Torrance, Blinn, and lastly, the Dichromatic Reflection Model are generally the ones utilised in various applications. These models are used in a wide variety of applications, including but not limited to digital imaging, 3D rendering, medical imaging, augmented and virtual reality etc.

However, some models have been traditionally more preferred than others by researchers and the industry as they can represent and reproduce the interactions of light accurately without requiring significant computational power. Some of the core physics-based models that are most relevant to the problem of understanding and mitigating specular reflections in images are detailed in the subsequent subsections.

Torrance-Sparrow microfacet model

The Torrance-Sparrow model [START_REF] Kenneth E Torrance | Polarization, Directional Distribution, and Off-Specular Peak Phenomena in Light Reflected from Roughened Surfaces[END_REF] is one of the most popular models for represent-

ρ λ θ i , θ r , φ = F β, n λ , k λ D(α)G θ i , θ r 4 cos θ i cos θ r (2.1)
Where:

• D is the distribution function of the micro-facets. The constant α indicates the roughness of the material, with 0 indicating ideal smooth surfaces and 1 indicating maximum roughness. • G is the geometric attenuation term, which deals with how the individual micro-facets shadow and mask each other depending on the incident (θ i ) and reflected (θ r ) angles.

• F is the Fresnel function that depends on the incident angle β and the complex index of refraction (n λ , k λ ) at wavelength λ.

The microfacet model plays an essential role in understanding how reflections are created, as reflection is an important visual cue for perception and depth for human vision [START_REF] Hu | Visual Cues for Perceiving Distances from Objects to Surfaces[END_REF]. These cues are also required to be reproduced realistically to enhance the visual quality and realism of artificially generated scenes and images. The BRDF function is the most common way to generate rendering equations to model the behaviour of light upon interaction with material in 3D space and differentiate between diffuse reflection and specular reflection in the scene. This reinforces the reason BRDF's have been one of the central models in the development of computer graphics and rendering domain, where the objective is to model and reproduce lighting effects in a scene. This includes both realistic diffuse and specular reflections on all the objects in the scene as they heavily contribute to the realism of modern 3D graphics and renderings.

Dichromatic Reflection Model (DRM)

The decomposition of an image into specular and diffuse images was first proposed by Shafer et al. [START_REF] Steven | Using Color to Separate Reflection Components[END_REF] as the Dichromatic Reflection Model (DRM). The DRM is a linear additive model according to which there are two luminance components. The body reflectance comprises the part of wavelengths of the visible spectrum that is reflected after interacting with the particles of the body below the surface and therefore represents the colour of the target body. The interface reflectance is the part of the wavelength that is reflected directly from the surface and represents the illuminant's colour as shown in the Figure 2.2. The DRM is inherently an underdetermined system with a non-trivial solution for separating the two components of an image. The model can be defined by the following equation:

L(λ, i , e, g ) = m d (i , e, g )c d (λ) + m s (i , e, g )c s (λ) (2.2)
Where d, s stand for the diffuse (body) and specular (interface) components, c d , c s are the diffuse and specular spectral power distributions and m d , m s are the geometric scale factors respectively. The light wavelength is denoted by λ and i , e, g are angles of the incident light, emitted light and phase angle (with respect to the surface normal), respectively. A matte surface is comprised mostly of the body reflectance, whereas specular surfaces contain a combination of both spectral and diffuse components. Furthermore, the probabilistic independence of specular and diffuse highlight is not constant as it depends on whether the surface is textured or smooth, as shown in Figure 2.3. The DRM model as proposed is valid for optically inhomogeneous materials only [START_REF] Steven | Using Color to Separate Reflection Components[END_REF]. These materials are a composites of two or more materials with different dielectric response functions and their appearance varies from point to point [START_REF] David | Optical Effects in Solids[END_REF]. Examples of inhomogeneous materials include most daily life materials including most paints, varnishes, paper, ceramics, plastics etc.

The model is based on three core assumptions. Firstly, the reflection from the surface is invariant with respect to rotation around the surface normal, and there are no inter-reflections among surfaces. Secondly, the body reflection is Lambertian 1 , which means that the brightness is independent of the viewing direction. Moreover, the specular reflection has the same colour as the illumination and tends to be polarised [START_REF] Steven | Using Color to Separate Reflection Components[END_REF]. While most of the assumptions seem to limit the model's applicability to real-world problems, these assumptions allow the generalisation of the 1 Lambertian models describe a perfectly diffuse surface that scatters incident illumination isotropically (equally in all directions) independent of the viewer's position. Although this reflection model is not physically plausible, it is a reasonable approximation to many real-world surfaces such as matte paint.

model and increase its applicability to a wide assortment of problems. Thus even after several ideal assumptions, the DRM model has broad applicability to understanding and mitigating specular highlights. A quick comparison of both DRM and Torrance-Sparrow microfacet models for defining specular highlights is given in table 2.1.

Intrinsic image decomposition and specular reflections

Another way to describe illuminations and reflections in a scene is by separating them into intrinsic images, a term initially coined by Barrow and Tenenbaum [START_REF] Barrow | Recovering Intrinsic Scene Characteristics from Images[END_REF].

Intrinsic images define scene characteristics into a model comprised of multiple independent images, where all images can be interpreted from the base illumination image. With intrinsic characteristics, they essentially referred to a set of features such as surface reflectance, distance or surface orientation, and incident illumination in a scene. They noted that humans are exceptional at judging an object's reflectance despite significant changes in illumination of the scene, a skill known as "lightness constancy" [START_REF] Edward H Adelson | The Perception of Shading and Reflectance[END_REF]. We also have the ability to estimate intrinsic characteristics from an image and do not seem to require familiarity with the scene or with objects contained therein. Another important observation was that while luminance can be directly observed, reflectance and illumination can only be derived by perceptual processes as it provides visual cues required for scene understanding. These observations led them to believe that separating the scene into separate characteristics from image intensities would provide substantial advantages for an effective visual system. A visual system must begin with the observed luminance image, I(x, y), and infer the underlying shading and reflectance images, s(x, y) and r (x, y). In a scene consisting of Lambertian surfaces illuminated by a single distant light source, the observed luminance image I(x, y) is a set of intensity value pixels that encode all the intrinsic attributes of the corresponding scene point. The image can be defined as the product of the reflectance (albedo) image, r (x, y), the shading image s(x, y) and the addition of a specular image c(x, y) [START_REF] Son | Toward Specular Removal from Natural Images Based on Statistical Reflection Models[END_REF][START_REF] Grosse | Ground Truth Dataset and Baseline Evaluations for Intrinsic Image Algorithms[END_REF], as given in The reflectance image comprises all material-dependent properties and remains constant under different illumination, whereas Shading image comprises all light dependent properties. Furthermore, the shading image itself is the product of the luminous flux φ v , and the cosine of the angle of incidence (i.e. the dot product of the surface normal n(x,y) and the illumination direction L.) as given in equation 2.5. Note that both the surface normal n(x,y) and the illumination direction L are 3D vectors [START_REF] Edward H Adelson | The Perception of Shading and Reflectance[END_REF].

s(x, y) = φ v n(x,y) • L (2.5)
Selected intrinsic images from the MIT Intrinsic Image dataset [START_REF] Grosse | Ground Truth Dataset and Baseline Evaluations for Intrinsic Image Algorithms[END_REF]

Specular highlight detection and segmentation

Regions with specular reflections in an image are generally unwanted yet mostly unavoidable feature. This is why the problem of Specular highlight detection is challenging and has been an area of progressive research for both traditional photography and digital imaging since their inception. Specular reflections are extremely hard to avoid in real-world conditions since they depend on several factors, including variables related to the illuminating source as well as the target object in the scene. These factors include the azimuthal and zenith orientations of the illuminating source and the object as the primary factor in the presence of specular highlight, alongside factors such as the material of the surfaces interacting with the light. In most natural world conditions, one or more of these factors are uncontrollable, which makes the presence of specular reflections impossible to avoid.

Specular highlights are a highly informative feature, and they have an important role in the fields of image processing and computer graphics. Specular reflections are essential for human vision as they provide powerful visual cues about the shape of the objects, the material of the object, and the location of the illuminating light source. However, apart from specific applications, specular reflection is generally considered an undesirable feature in the image processing domain, causing loss of chromatic and textural information that is often vital to applications [START_REF] Klinker | A Physical Approach to Color Image Understanding[END_REF].

In digital imaging, the requirement of detecting the occurrence of specular reflections and identifying all the corresponding affected pixels accurately becomes an essential and formidable task. The affected specular pixels have to be segmented before any processing algorithm can be used to remove the specularity and mitigate the undesirable effects of the reflection. This chapter addresses this task by proposing a state-of-the-art solution that can precisely identify and segment all pixels in an image affected by specular reflections.

Accurately segmenting and detecting specular pixels in an image is challenging for several reasons. sections.

Classical specular detection and segmentation methods

Specular highlight segmentation has proven to be an extremely challenging problem over the years since it is an ill-posed problem. While specular reflections are easily distinguishable by human vision, it is a tough ask for digital image processing systems. Traditional techniques have always been based on simplifying the problem in some manner, including assumptions regarding the colour of light, the transmission medium and its refractive index, the object's material, etc. While most assumptions are valid for solving a problem, they are mostly unrealistic and do not represent an accurate real-world scenario. The DRM model has proven to be a reasonably accurate model to explain the causes of specular reflections and thus forms the basis of a large selection of detection and mitigation techniques. The subsequent sections will review the most used methods and techniques proposed by research works over the years.

Segmenting specular highlights using chromaticity

As explored in depth in section 2.1.3, Shafer et al. [START_REF] Steven | Using Color to Separate Reflection Components[END_REF] were the first ones to propose the DRM, which became the fundamental model for understanding and explaining nearly all reflection models. Their breakthrough paper used the spectral distribution of light and its colour coordinates to identify and separate the colour pixels into diffuse and specular components. Unlike previous models like the Phong model [START_REF] Bui Tuong | Illumination for Computer Generated Pictures[END_REF] which uses specific reflectance functions to predict the reflection amount, DRM is based on the physical model of reflection, making it more intuitive and realistic.

Klinker et al. [START_REF] Klinker | A Physical Approach to Color Image Understanding[END_REF] based their work on DRM and showed that the colour histogram of an image forms a T-shaped distribution with uniform diffuse regions. This also results in the formation of linear clusters with diffuse and specular pixels. Using geometric heuristics instead of colour information, they estimate a single global diffuse colour, which can be extended to several segmented regions of homogeneous diffuse colour and estimate the body and reflection components. Klinker and Shafer et al. [START_REF] Klinker | The Measurement of Highlights in Color Images[END_REF] also proposed modelling of highlights as a linear combination of both surface and body reflections and modelled camera properties to account for camera limitations and showed that generating the intrinsic images from a single image was possible. Schluns et al. [START_REF] Schlüns | Analysis of 2D Color Spaces for Highlight Elimination in 3D Shape Reconstruction[END_REF] proposed segmentation of specular regions by transforming the 3-dimensional colour-space to consecutive two-dimensional descriptors and thresholding specular pixels based on the projection distances. Bajscy et al. [START_REF] Bajcsy | Detection of Diffuse and Specular Interface Reflections and Inter-Reflections by Color Image Segmentation[END_REF] defined a Spectral Scene Radiance Model called S-space, which is a direct transformation from the RGB space using three orthogonal basis functions. Assuming a white illuminating source and analyzing a colour image in the S-space, the specular reflection pixel clusters in the S space align with the brightness axis. They can be used to segment out the specular pixels. However, the assumption of pure white global illumination and uniformly single-coloured, non-textured objects limit the application. Yang et al. [START_REF] Yang | A New Projection Space for Separation of Specular-Diffuse Reflection Components in Color Images[END_REF] proposed a new colour space called Chromaticity Coefficient of Variation (Ch-Cv) for specular reflection removal.Using their developed colour space, they propose a slope-based region growing method to separate each pixel's specular and diffuse components.

Tan and Ikeuchi et al. [START_REF] Robby | Separating Reflection Components of Textured Surfaces Using a Single Image[END_REF] proposed a method based on the difference in logarithmic differentiation of the normalized input and specular-free images. Yoon et al. [START_REF] Kuk-Jin Yoon | Fast Separation of Reflection Components Using a Specularity-Invariant Image Representation[END_REF] were the first to introduce the two-band Pseudo Specular Free (PSF) image obtained by subtracting the minimum of the three RGB channel values from each pixel. These values are then compared to neighbour intensity ratios to their corresponding ratios in the PSF representation for separating highlight pixels. The Pseudo Specular Free (PSF) image is thus defined as the component of the RGB image without the specular component. The PSF image can be calculated by taking the minimum of each pixel in all three channels. Cai [START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF] approached the removal problem by first segmenting into mixed specular and diffuse and purely diffuse; however they corrected the values of specular regions by solving for a constant gain with regards to the modified specular-free image. Yang et al. [START_REF] Yang | Efficient and Robust Specular Highlight Removal[END_REF] achieved highlight removal by applying a joint bilateral filter to smooth out the maximum chromaticity regions of the observed image, using a PSF image to guide the filter. Suo et al. [START_REF] An | Fast and High Quality Highlight Removal from A Single Image[END_REF] extended the DRM in terms of L2 normalization by formulating the problem such that the illuminant is orthogonal to the chromaticity.

    I r (p) I g (p) I b (p)     = m d (p)     Λ r (p) Λ g (p) Λ b (p)     + m s (p)     Γ r (p) Γ g (p) Γ b (p)     (2.
Their approach also requires clustering for the estimation of region-specific purely diffuse colours. Kim et al. [START_REF] Kim | Specular Reflection Separation Using Dark Channel Prior[END_REF] introduced the concept of utilising the Dark Channel Prior (DCP) concept, originally proposed for haze removal [START_REF] He | Single Image Haze Removal Using Dark Channel Prior[END_REF]. Kim et al. observed that the dark channel provided an approximate specular-free image for most natural images. They used this idea and approached the problem from an optimization standpoint by formulating it in terms of a TV-L1 and TV-L2 optimization problem in a Maximum Aposteriori (MAP) framework, which yields pleasing results.

Specular segmentation using polarization

The concept of polarisation is directly related to the problem of specular highlight segmentation due to the highly polarised nature of specular reflections [START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF]. Due to this, significant research has been done for segmenting and removing specular highlights in images using polarisation, which requires special consideration. It is noteworthy that the significance of DRM is further increased since it can be used in conjunction with the polarised nature of specular reflection to explain the occurrence and mitigation of specular reflections. Wolf et al. [START_REF] Wolff | Polarization-Based Material Classification from Specular Reflection[END_REF] were one of the earliest to use polariser images for the classification of materials in images by using the Fresnel reflection model. They monitored the variation of light by capturing multiple images while rotating the polariser filter in front of a camera and noted that the brightness of diffuse materials varied as the polariser was rotated. They also noted that the variation between the minimum and maximum intensity captured fluctuates in a sinusoidal pattern as a function of the polariser angle. Nayar et al. [START_REF] Shree K Nayar | Separation of Reflection Components Using Color and Polarization[END_REF] were

Low-rank approximations and other approaches

Over the years, one of the popular methods of solving the specular reflection problem has been to treat it as noise in an image and utilize techniques that can mitigate the effect of noise in images. By assuming specular reflections as noise, methods such as noise filtering, low-rank approximations and other minimization techniques can be used to approximate the image data, freeing it from the effects of noise. Zhang et al. [START_REF] Zhang | Reflection Component Separation Using Statistical Analysis and Polarisation[END_REF] Using the K-nearest-neighbour algorithm (KNN), they replaced highlight pixels with body-reflection pixels removing specular highlights in RGB images. Guo et al. [START_REF] Guo | Single Image Highlight Removal with a Sparse and Low-Rank Reflection Model[END_REF] introduced a sparse and low-rank formulation and incorporated advances in nonnegativity and matrix factorization. They also introduce two auxiliary variables to incorporate these formulations and iteratively solve the optimization problem. [START_REF] Fu | Specular Highlight Removal for Real-world Images[END_REF] proposed an energy minimization framework for simultaneously estimating the diffuse and specular highlight images from a single image and then recovering the diffuse colour.

Multiple-images based methods

As shown by the DRM model, specular highlights are dependent on the incidence and reflection angles between the light source and the observing camera or sensor. This implies that any change in this angle can lead to diminishing or removing specular highlight regions. Following this concept, an alternate approach to treat- 2.2.

Deep learning based methods

As we can see in the preceding section, a significant amount of work has been done on detecting and segmenting specular highlights using classical image processing techniques. While there have been many studies of specular highlight detection over the years, most classical methods conduct a visual evaluation on a few selected images, mostly without annotated ground truth or highlight masks. This has led to a very unrealistic quantitative evaluation of highlight detection algorithms on realworld images where the lighting can vary significantly from ideal conditions. Specular reflections caused by inter-reflections between objects or due to light reflecting off other surfaces in the scene cause multiple issues, which are often not addressed by classical methods. During the last decade, the benefits of machine learning have become quite evident with a substantial impact, especially in the fields of image processing. Furthermore, deep learning has seen a significant amount of growth and development not only in the core techniques but also in frameworks for implementing efficient and robust deep-learning implementations.

Several solutions have been proposed in recent years to accurately identify the specular pixels in medical images by leveraging machine learning algorithms. Sanchez et al. [START_REF] Sánchez | Bright Spot Regions Segmentation and Classification for Specular Highlights Detection in Colonoscopy Videos[END_REF] used a two-stage segmentation and classification approach to identify specular regions in colonoscopic images and then filtered through a linear SVM classifier. Akbari et al. [START_REF] Akbari | Adaptive Specular Reflection XXIII BIBLIOGRAPHY Detection and Inpainting in Colonoscopy Video Frames[END_REF] utilized an adaptation between RGB and HSV colour spaces using a non-linear SVM classifier and then inpainted the detected regions. One of the earliest methods toward a more generalized and innovative specular highlight detection method was proposed by Lee et al. [START_REF] Lee | Removal of Specular Reflections in Tooth Color Image by Perceptron Neural Nets[END_REF] which implemented detection of specular reflections by a single layer perceptron. Moving forward toward the stateof-the-art deep learning methods, Funke et al. [START_REF] Funke | Generative Adversarial Networks for Specular Highlight Removal in Endoscopic Images[END_REF] were the first ones to utilize a Cycle-Consistent Generative Adversarial Network (CycleGAN) to localize specular regions for endoscopic images. Their method used data with weak labels indicating the presence or absence of specular highlight in a training image only.

It is worth mentioning that typically most of the state-of-the-art deep-learningbased methods are geared towards training the network to mitigate the specular highlights using supervised or unsupervised training methods. This means that very few works exclusively focused on deep learning methods to detect specular pixels, which is the focus of this work. One of the recent papers that focused on detecting specular highlights in real-world images was proposed by Fu et al. [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF].

The proposed a Specular Highlight Detection Network (SHDNet), that used multiscale contrast features to detect specular pixels that are scale agnostic. SHDNET uses a convenient and embeds a multi-scale context contrasted feature network for successfully detecting specular highlights in real-world images. The authors also present a large-scale dataset of roughly real-world images, which include manually annotated highlight regions. In addition to the primary dataset, they also prepared a testing dataset of 500 images in the wild called the WHU-TRIW dataset. Fu et al. [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] proposed another large-scale dataset comprising 16k real-world images alongside a multi-task network for Joint Specular Highlight Detection and Removal (JSHDR). Bajcsy et al. [START_REF] Bajcsy | Detection of Diffuse and Specular Interface Reflections and Inter-Reflections by Color Image Segmentation[END_REF] 1996 Segmentation Segmentation by Hue, Saturation

S-space

Park et a. [START_REF] Park | A Truncated Least Squares Approach to the Detection of Specular Highlights in Color Images[END_REF] 2003 Segmentation Least Squares, PCA RGB Umeyama et al. [START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF] 2004 Separation Polarization, ICA Greyscale Tan et al. [START_REF] Robby | Separating Reflection Components of Textured Surfaces Using a Single Image[END_REF] 2005 Separation Chromaticity, Colour Spaces RGB Tan et al. [START_REF] Tan | Separation of Highlight Reflections on Textured Surfaces[END_REF] 2006 Separation Spatial Colour Distributions RGB Zhouyu et al. [START_REF] Fu | Specular Free Spectral Imaging Using Orthogonal Subspace Projection[END_REF] 2006 Segmentation Subspace projection RGB Shen et al. [START_REF] Shen | Chromaticity-Based Separation of Reflection Components in a Single Image[END_REF] 2008 Separation Chromaticity based RGB Maxwell et al. [START_REF] Maxwell | A Bi-Illuminant Dichromatic Reflection Model for Understanding Images[END_REF] 2008 Segmentation Bi-illuminant DRM RGB Shen et al. [START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF] 2009 Separation Pixel clustering RGBD Mesloushi et al. [START_REF] Meslouhi | Automatic Detection and Inpainting of Specular Reflections for Colposcopic Images[END_REF] 2011 Segmentation Chromaticity CIE XYZ Yang et al. [START_REF] Yang | Separating Specular and Diffuse Reflection Components in the HSI Color Space[END_REF] 2013 Separation Region growing algorithm HSI Kim et al. [START_REF] Kim | Specular Reflection Separation Using Dark Channel Prior[END_REF] 2013 Segmentation Dark Channel Prior RGB Zou et al. [START_REF] Zou | Specularity Removal Using Dark Channel Prior[END_REF] 2013 Segmentation Dark Channel Prior RGB Akashi et al. [START_REF] Akashi | Separation of Reflection Components by Sparse Non-Negative Matrix Factorization[END_REF] 2016 Segmentation NMF RGB Shah et al. [START_REF] Syed | Removal of Specular Reflections from Image Sequences Using Feature Correspondences[END_REF] 2017 Segmentation SIFT in sequential images RGB Yamamoto et al. [START_REF] Yamamoto | Efficient Improvement Method for Separation of Reflection Components Based on an Energy Function[END_REF] 2017 Separation SVD, Energy minimization RGB Alsaleh et al. [START_REF] Samar | Dim the Lights! -Low-Rank Prior Temporal Data for Specular-Free Video Recovery[END_REF] 2019 Separation Low-Rank Temporal Data RGB Fu et al. [START_REF] Fu | Specular Highlight Removal for Real-world Images[END_REF] 2019 Separation Optimization RGB Li et al. [START_REF] Li | Specular Reflections Removal for Endoscopic Image Sequences With Adaptive-RPCA Decomposition[END_REF] 2020 Separation RPCA RGB Son et al. [START_REF] Son | Toward Specular Removal from Natural Images Based on Statistical Reflection Models[END_REF] 2020 Separation convex optimization RGB Ramos et al. [START_REF] Vítor | Real-Time Highlight Removal From a Single Image[END_REF] 2021 Separation histogram matching YCbCr Haefner et al. [START_REF] Haefner | Recovering Real-World Reflectance Properties and Shading From HDR Imagery[END_REF] 2021 Separation HDR Imaging for separation RGB Bonekamp et al. [START_REF] Bonekamp | Multi-Image Optimization Based Specular Reflection Removal from Non-Dielectric Surfaces[END_REF] 2021 Separation Multi-Image Optimization RGB Kim et al. [START_REF] Kim | Specular Detection on Glossy Surface Using Geometric Characteristics of Specularity in Top-View Images[END_REF] 2021 Segmentation Geometric estimation RGB Ramos et al. [START_REF] Vítor | Real-Time Highlight Removal From a Single Image[END_REF] 2021 Separation histogram matching YCbCr Tominaga et al. [START_REF] Tominaga | Spectral-Reflectance Estimation under Multiple Light Sources[END_REF] 2021 Segmentation Iterative estimation process RGB Wen et al. [START_REF] Wen | Polarization Guided Specular Reflection Separation[END_REF] 2021 Separation Polarization RGB Li Furukawa [START_REF] Li | DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces[END_REF] 2022 Separation RPCA, Photometric Stereo RGB CHAPTER 2. LITERATURE REVIEW

Limitations of the current state of the art

The accurate detection of specular highlights is significant in many applications.

Classical methods for accurately detecting specular highlights have difficulty detecting pixels accurately in a wide variety of scenes containing lighter coloured objects, bright backgrounds, or complex-shaped objects with irregular specular reflections. One of the significant issues faced by the classical techniques is the robustness and generalization of techniques. While the methodologies are based on firm mathematical foundations and optimization techniques, they are mostly based on assumptions that significantly limit their applications to general real-world images that are not part of the work's dataset. Thus while the results are significantly better on the selected set of images, they do not apply to any general image taken from a generic camera under uncontrolled settings. Several research works based on treating specular reflections using colour space transformations attempted to understand and tackle the problem purely from an objective often tested on a minimal set of images and failed to work beyond their preferred set. Methods based on polarization classically use a manual polarizer filter that is rotated to acquire images ad different polarimetric angles. This means that the images are temporally incoherent, and unless taken of a static object under a static and controlled environment, the images face alignment issues where pixels do not share the same spatial instance between the polar images. This also limits the number of images that can be acquired as a significant amount of effort is required to take a broad and generalized dataset.

Several assumptions are also made for classical methods to work, which are sometimes not reflective of real-world conditions. For, e.g., a single Illumination is mostly assumed with a non-existent or minimum amount of inter-reflections from surrounding surfaces. The illuminants selected are assumed to be of pure white colour with known spectral power distribution (SPD) to simplify all chromaticity-based methods. It is further assumed that each segmented cluster has uniform diffuse chromaticity. While being very helpful for modelling the problem of specular highlight, these and other assumptions are not reflective of the randomness of realworld images and limit the generalization and applicability of methods. Since most limitations are not considered for deep-learning-based methods, it is pretty clear that modern state-of-the-art methods are significantly more robust and can cater to a much more comprehensive range of images. These limitations are only enhanced in the presence of outdoor images, which have both intense illumination and interreflections in an uncontrolled and often stochastic environment. Outdoor environments have illumination from the sun as an omnidirectional light source, causing light to bounce off in often undesirable directions and strength. Strong light sources also result in larger specular regions in images, which makes the regions easily visible but also easily confused with the objects in the scene, as well as causing a significant loss of information in the area, which hinders the recovery of colour and other information in the affected region. Modern deep learning methods are a natural progression to robust intelligent solutions that can distinguish specularity from the background. However, deep learning methods are also challenged with some limitations.

Specular highlight mitigation

The core contribution of this thesis is on the mitigation of specular reflections, as will be explored in the following sections. As we have seen in the previous chapter, there has been a significant amount of research in detecting specular regions; however, once the affected regions are detected, there are two options available for algorithms to explore. One option is to use that information to ignore the affected areas and pixels and process the image with the remaining information. The second and the most favourable option is that we are able to remove the specularity from the image and recover the information that is in place of the affected region. The subsequent sections will, first of all, explore the prior literature over the years that is relevant to specular highlight removal in depth, including both classical and deep learning-based state-of-the-art methods. Based on the literature review, we explore a fast diffuse colour inpainting method that utilizes the detected regions from our proposed SpecSeg network and inpaints the affected regions with an estimated diffuse colour inferred from the boundary regions. The advantages and limitations of this method are also discussed and are followed by details of our developed Specular Highlight Mitigation Generative Adversarial Network (SHMGAN). By leveraging the advantages of deep convolutional networks, we are able to mitigate the affected region and recover the lost information successfully.

Classical methods of specular highlight mitigation Using colour space transformation

There have been several studies to mitigate specular highlights over the years. With the advent of digital imaging, colour spaces were a prime area of research to interpret the pixel information captured from sensors. This also led to increased interest in using various transformations and interpretations to convert the captured information from the RGB colour space to various other spaces and separate the specu- [21] introduced a Chromaticity Coefficient of Variation (Ch-Cv) colour space where the surface points with the same diffuse chromaticity have the same slope. Using a slope-based region growing method in the specular regions, they were able to separate the reflection components for each segmented region. They also explored the separation of specular highlights in HSI colour space [START_REF] Yang | Separating Specular and Diffuse Reflection Components in the HSI Color Space[END_REF], where they use a regiongrowing algorithm to locate adjacent pixels with similar diffuse chromaticity. Yu et al. [START_REF] Yu | Efficient Highlight Removal of Metal Surfaces[END_REF] also utilized HSV space to remove specular reflection from metallic surfaces.

Akbar et al. [START_REF] Akbar | Sparse Coded Decomposition for Single Image-Based Specular Removal[END_REF] introduced a new XYZ colour space transformation from RGB and showed that using sparse coded decomposition, a specular-free image can be generated for each RGB channel.More recently, [START_REF] Huang | An Effective Algorithm for Specular Reflection Image Enhancement[END_REF] and [START_REF] Vítor | Real-Time Highlight Removal From a Single Image[END_REF] utilized the YCbCr colour space for generating specularity free images. YCbCr colour space has the property of being specular-free in the Y (Luma) channel [START_REF] Vítor | Single Image Highlight Removal for Real-Time Image Processing Pipelines[END_REF]. For a pixel p, the DRM model defined by equation (2.2) in YCbCr colour space can be written as equation 2.9:

    I y (p) I cb (p) I cr (p)     = A     I r (p) I g (p) I b (p)     = A     I r d (p) + I s (p) I g d (p) + I s (p) I bd (p) + I s (p)     + offset (2.
I cb (p) I cr (p)     = A     I r d (p) I g d (p) I bd (p)     +     I s (p) 0 0     + offset (2.11)
This makes the usage of the Y channel quite enticing for any algorithms for specular reflection removal, as will be seen in the subsequent subsections.

Noise filtering and inpainting

Several authors proposed treating specularity as noise in an image and have shown the benefits of using noise filters such as low-pass filters to remove specular pixels to an extent. Filtering techniques such as joint bilateral filtering [START_REF] Jung | Specular Highlight Removal Using Reflection Component Separation and Joint Bilateral Filtering[END_REF][START_REF] Yang | Efficient and Robust Specular Highlight Removal[END_REF][START_REF] Yang | Real-Time Specular Highlight Removal Using Bilateral Filtering[END_REF] have been shown to perform in near real-time on videos owing to the high performance implementations of traditional filtering. Inpainting methods have also been a very popular research area as they are very close to the idea of taking the surrounding spatial and/or temporal information and filling in the affected areas. Field imaging is a modern branch of computational photography where a microlens array is used to capture the direction of light in addition to its intensity in a spatially-coherent set of images. This allows post-capture applications, such as refocusing and altering viewpoints. Wang et al. [START_REF] Wang | Light Field Imaging Based Accurate Image Specular Highlight Removal[END_REF] applied light field imaging for specularity removal in HSI colour space. Other works [START_REF] Wang | Light Field Imaging Based Accurate Image Specular Highlight Removal[END_REF][START_REF] Alperovich | Shadow and Specularity Priors for Intrinsic Light Field Decomposition[END_REF][START_REF] Yu | Light-Field Features for Robotic Vision in the Presence of Refractive Objects[END_REF][START_REF] Feng | Specular Highlight Removal of Light Field Based on Dichromatic Reflection and Total Variation Optimizations[END_REF]] also show the benefits of multiple views from a single camera to restore highlight information.

However, since these cameras are specialized equipment and emerging technology, the utilization of light field cameras is not as widespread. Several classical methods such as pixel clustering [START_REF] An | Fast and High Quality Highlight Removal from A Single Image[END_REF][START_REF] Antonio | Real-Time High-Quality Specular Highlight Removal Using Efficient Pixel Clustering[END_REF], histogram matching [START_REF] Vítor | Single Image Highlight Removal for Real-Time Image Processing Pipelines[END_REF], intensity ratios [START_REF] Shen | Real-Time Highlight Removal Using Intensity Ratio[END_REF] and low-rank optimization methods [START_REF] Shirai | Local Covariance Filtering for Color Images[END_REF][START_REF] Shakeri | Highlight Specular Reflection Separation Based on Tensor Low-Rank and Sparse Decomposition Using Polarimetric Cues[END_REF][START_REF] Xia | A Global Optimization Method for Specular Highlight Removal From a Single Image[END_REF] have been shown to remove specular highlights with varying effectiveness on a limited set of images. Ramos et al. [START_REF] Ramos | SIHR: A MATLAB/GNU Octave Toolbox for Single Image Highlight Removal[END_REF] provided a publicly available repository of several DRM-based specular highlight mitigation methods for performance comparison.

Polarization and specular highlights

In 1852, George Gabriel Stokes initially established a mathematical description of this incoherent or partially polarised nature of light. Later in 1890, Henri Poincaré proposed the existence of the state of polarization and later proposed a spherical representation to describe these states of polarisation known as the Poincare Sphere. Polarisation is a natural property of transverse waves that specifies the geometrical orientation of the oscillations. As light is fundamentally also a transverse electromagnetic wave with oscillations that are orthogonal to the direction of mo-tion, it possesses a state of polarisation characterising the vibrational orientations of the electric field component [START_REF] Wolff | Polarization-Based Material Classification from Specular Reflection[END_REF]. An electromagnetic wave such as light consists of a coupled oscillating electric field E and magnetic vector field B which are always orthogonal to each other and are defined by Maxwell's equations. By convention, however, the polarisation of electromagnetic waves refers to the direction of the electric field E and is described by projections of the electric field vectors in an

xy plane that is orthogonal to the direction of propagation of the wave. For the electromagnetic field E, let z be the direction of propagation of the wave and ω be the frequency of the wave along the time axis t . Then the electric field vector can be written as two components, E x and E y such that

E(z, t ) = (E x (z, t ), E y (z, t )) (2.12)
Where E x and E y represent the scalar components along the axes x and y respectively. The components can also be written in a column vector notation as:

E(z, t ) =     E x (z, t ) E y (z, t ) E z (t )     =     E x cos (ωt -kz) E y cos ωt -kz + ϕ 0     (2.13)
Where E x and E y represent the maximum amplitudes of each component, ϕ = ϕ y -ϕ x represent the phase shift between the two components, ω is the angular frequency, k is the wave number (spatial frequency) that is directly related to the wavelength λ by k = 2π λ n, where n is the refractive index of the propagation medium.

The polarization state of an electromagnetic wave given by equations 2.12 and 2.13 is defined by the curve of the tip of the resultant electric field vector E as a function of time t , projected into the plane orthogonal to the direction of propagation z [START_REF] Giménez | Caractérisation et étalonnage des systèmes d'imagerie de Stokes à matrice de filtres polariseurs[END_REF].

Considering a uni-directional wave and projecting the waves on an x-y plane, we get scalar projections defined by the following equations.

E x (t ) = E x cos(ωt ) (2.14) E y (t ) = E y cos(ωt + φ) (2.15)
If φ varies with time such that -1 ≤ sin(φ) ≤ 1, then the wave is said to be unpolarized as tip of the resultant vector obeys a random trajectory over time. If the angle is sin(φ) = ±1, the wave is said to be circularly polarized, with +1 being right circular and -1 being left circular. By convention, the direction of rotation is determined to be clockwise and counterclockwise when looking in the direction of propagation. If, however, the angle sin(φ) is zero, the electric field oscillates along a straight line and is said to be linearly polarised. For a linearly polarized light, the plotted amplitude and direction does not change over time. Waves that have orthogonal components that are out of phase but have the same amplitude will result in a superposition of a circularly polarised wave. For image processing, however, assuming the rays are linearly polarised only and ignoring circular polarisation is sufficient for describing most cases and is, therefore, the most widely accepted usage for processing polarised images. Since an electromagnetic wave exhibits polarised nature, its orthogonal components described above can be attenuated such that all components of the wave are cancelled out except one as shown in Figure 2 with an angle of polarisation ψ, passing through a perfect polariser with optical axis θ, produces an outgoing polarisation state whose intensity I out as described by the equation:

I out = I i n cos 2 (ψ -θ) (2.16)
Thus the resultant of components E x and E y at polariser angle θ of the incoming wave, coinciding with the angle of polarisation ψ of the filter, will pass through the polariser and the remaining components will be attenuated by the polariser.

Stokes parameters

An alternative model for describing the states of polarisation of electromagnetic waves was introduced by George Gabriel Stokes in 1852 and called Stokes parameters. To remove the dependence on instantaneous time functions E x and E y , Stokes proposed a matrix based on four elements arranged in a column matrix called polarisation space that can be used to construct four real parameters called the Stokes parameters for describing polarisation. A stokes vector is estimated by combining the measurement of the polarisation of light from the projection of different polarisation filters. The resulting intensity measurement I taken by the polariser filter relative to polarisation states S can be defined by equations 2.17 and 2.18. To calculate Stokes vectors, a series of measurements must be taken with a set of Q polariser filters. It is assumed that during the whole measurement process, the incident Stokes vector is the same for all the filters in use.

I = AS (2.17)     I 1 . . . I Q     =     a 0,0 a 0,1 a 0,2 a 0,3 . . . . . . . . . . . . a Q-1,0 a Q-1,1 a Q-1,2 a Q-1,3            S 0 S 1 S 2 S 3        (2.

18)

The analyser matrix A is characterised by the number of polariser filters Q oriented at different angles θ i . The matrix becomes nonsingular for Q ≥ 4. Note that if Q > 4, the analyser matrix A cannot be inverted normally, however it is possible to use the Moore-Penrose pseudo-inverse (A + ) to calculate the stokes vector as given by the equations 2.19.

S = A + I

where

A + = (A T A) -1 .A T (2.19)
Thus a stokes vectors comprising of four stokes components S = [S 0 , S 1 , S 2 , S 3 ] T can be formed from 4 polarimetric angle measurements. Each Stokes parameter has a physical interpretation related and can be expressed as sums and differences of

Circular S0 1 1 1 1 1 1 1 S1 0 1 -1 0 0 0 0 S2 0 0 0 1 -1 0 0 S3 0 0 0 0 0 -1 1
electric field intensities as given in equation 2. [START_REF] Bajcsy | Detection of Diffuse and Specular Interface Reflections and Inter-Reflections by Color Image Segmentation[END_REF].

S =        S 0 S 1 S 2 S 3        =        E 2 x + E 2 y E 2 x -E 2 y 2E x E y cos(φ) 2E x E y sin(φ)        (2.20)
• S 0 represents the total intensity of light, irrespective of the state of polarization. This value is always 0 < S 0 ≤ 1.

• S 1 represents the intensity of linear horizontal or vertical polarization

• S 2 represents the intensity of linearly polarised light at ±45 •

• S 3 gives the difference of the right minus the left circularly polarised light.

Furthermore, in the case of a fully polarised light, the Stokes parameters obey the following physical admissibility constraints given by equation 2.21 [START_REF] Ruffino | Physically-Admissible Polarimetric Data Augmentation for Road-Scene Analysis[END_REF].

S 0 > 0 S 2 0 ≥ S 2 1 + S 2 2 + S 2 3 (2.21)
Substituting for Q = 4 stokes parameters in equation 2.17 the image intensity vector for 4 polarimetric angles 0 • , 45

I =        I 0 I 45 I 90 I 135        = A i d eal        S 0 S 1 S 2 S 3        (2.

22)

Where A i d eal is the matrix defining an ideal linear polarizer and defined as equation 2.23. A summary of the representation of the various polarisation states related to the Stokes parameters as calculated from different values of the A i d eal matrix is given by the table 2.4.

A ideal = 1 2        1 1 0 0 1 0 1 0 1 -1 0 0 1 0 -1 0        (2.23)
The equation 2.20 can also be rewritten as sums and differences of the pixel intensities of the four polarimetric images as shown in equation 2. [START_REF] Shen | Chromaticity-Based Separation of Reflection Components in a Single Image[END_REF]. angles. This is applicable for the acquisition of both greyscale and RGB images from polariser filters on standard imaging sensors. Additionally, the images' maximum and minimum intensity pixel values can be calculated from the Stokes parameters using equation 2.25.

S =        S 0 S 1 S 2 S 3        =        I 0 + I 90 I 0 -I 90 I 45 -I 135 I l -I r        (2.
I max = 1 2 S 0 + S 2 1 + S 2 2 I min = 1 2 S 0 -S 2 1 + S 2 2 (2.25)
In general usage, the normalised Stokes vector is used. Normalisation is done with respect to S 0 and represents the state of polarisation-independent from the inten-sity of light, as well as bounds the vector components to the [-1, 1] interval.

S =        1 S 1 /S 0 S 2 /S 0 S 3 /S 0        (2.26)
For the fully polarized light the Stokes vector S has components 1, 0, 0, 1 T , whereas the Stokes vector for unpolarized light is always of the form 1, 0, 0, 0 T .

Degree and angle of polarization

The degree of polarisation (DoP) ρ is a measure of the extent to which the light is polarised and is defined as the ratio of the polarised component state to the total intensity. The DoP varies between 0 for unpolarised light and 1 for fully polarised light. Similarly, we can also define the DoLP by taking the circular polarisation component S 3 = 0. The DoP in pixel intensity is given by equation 2.

ϕ pol = 1 2 arctan S 2 S 1 , 0 ≤ ϕ pol < π (2.29)

Polarimetric cameras and imaging sensors

Polarisers have been widely used in imaging systems, both traditional and digital.

Traditional polarisers are freely rotating filters mounted on standard camera lenses and can be manually rotated to any desired angle. While this allows capturing an image at any polariser angle, it limits capturing only one polarimetric angle at one instance. Furthermore, since the rotation of the polarimetric filter is manual, there is a chance of slightly altering the camera's field of view during the rotation. Therefore capturing four spatially and temporally coherent polarimetric images is impossible with externally mounted polariser filters.

With the development of advanced digital imaging sensors, polarisers that are fitted directly to the sensor (i.e., on-sensor filters) have also been developed. These polarizers consist of four different filters on one pixel and are pre-aligned at 0°, 45°, 90°, and 135°. Such filters are mounted on the sensor permanently and integrated into the vision system. The lifespan of the filters, which are protected from the external environment, is longer than that of external polarizers. This setup captures four spatially and temporally coherent images at four different polarimetric angles in a single shot. However, the image resolution is a quarter of the image sensor, due to the separation of the images by demosaicing. A generic on-sensor polarimetric filter configuration is shown in Figure 2.8.

Mitigation of specular highlights using Polarization

As has already been established that light from most sources is unpolarized and has equal irradiance in all directions. However, unpolarized light specularly reflected from a reflective surface becomes partially polarized [START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF]. For an image observed through a polarizer filter, the intensity of reflected light cancelled out by the polarizer fluctuates sinusoidally as a function of the polarization angle as shown in Fig.

2.6. Nayer et al. [START_REF] Shree K Nayar | Separation of Reflection Components Using Color and Polarization[END_REF] were one of the first to use colour and polarization information simultaneously to obtain constraints on the reflection components. Umeyama et al. [START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF] showed that diffuse and specular components could be separated by applying ICA to the images observed through a polarizer. Kim et al. [START_REF] Dae | Variational Specular Separation Using Color and Polarization[END_REF] applied PDEs, whereas Wang et al. [START_REF] Wang | Polarization-Based Specularity Removal Method with Global Energy Minimization[END_REF][START_REF] Wang | Specularity Removal: A Global Energy Minimization Approach Based on Polarization Imaging[END_REF] applied global energy minimization to polarizer images for inpainting the specular regions with diffuse colour.

Deep learning based methods

One of the earliest methods toward a more generalized and smart specular highlight detection method was proposed by Lee et al. [START_REF] Lee | Removal of Specular Reflections in Tooth Color Image by Perceptron Neural Nets[END_REF] which implemented the detec- Jimenez-Martin et al. [START_REF] Jimenez-Martin | Specular Reflections Removal in Colposcopic Images Based on Neural Networks: Supervised Training with No Ground Truth Previous Knowledge[END_REF] showed that it was possible for specular reflections re-

Multi-domain Generative adversarial networks

Before we delve into the details of image segmentation and mitigation methods using deep learning methods, it will be appropriate to explain some basic nomen- Recently, an in-depth analysis of the mathematics of GANs was done by Wang [START_REF] Wang | A Mathematical Introduction to Generative Adversarial Nets (GAN)[END_REF] giving deep insight into the mathematics and learning of weights by backpropaga-

CycleGAN and StarGAN

Zhu et al [START_REF] Zhu | Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[END_REF] proposed an important variant of the vanilla GAN, known as Cycle-GAN. It is an extension of Pix2Pix architecture [START_REF] Isola | Imageto-Image Translation with Conditional Adversarial Networks[END_REF] which involves simultaneous training of two generator models and two discriminator models. CycleGAN is a type of generative adversarial network for unpaired image-to-image translation. Cycle-GAN learns a mapping for two domains G : X → Y and F : Y → X and then uses the intuition that these mappings should be reversible and equally applicable to both domains, as shown in the Figure2.11. This is achieved through a cycle consistency loss as defined by equation 2.31. StarGAN is a robust and scaleable approach able to perform image-to-image translation among multiple domains using a single model and can generate higher visual 2.12.

For a multi-domain translation problem, if there are domains that are missing or imbalanced, it often introduces large amounts of bias in the trained network. To impute the missing data, Lee et al. proposed Collaborative Generative Adversarial Network (CollaGAN) [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF]. CollaGAN, as shown in Figure 2.13, treats the image imputation problem as a multi-domain images-to-image translation task so that a single generator and discriminator network can successfully estimate the missing data using the remaining clean data set. Since the missing data domain is not difficult to estimate a priori, the imputation algorithm can estimate the missing data in any domain by exploiting the data in the remaining domains. CollaGAN retains the one-generator architecture similar to StarGAN, which is more memory-efficient than CycleGAN but requires all the domains as input to generate the missing domain. A brief comparison of StarGAN and CollaGAN is given in table 2.7

Self-attention mechanisms in deep-learning networks

The human visual attention system allows us to focus on regions with high importance simultaneously while perceiving the surrounding image and the background of lower importance. Similarly, we can infer the relationship between words in one sentence or close context. Based on this premise, attention in deep learning can be broadly interpreted as a vector of importance weights in order to predict or infer one element, such as a pixel in an image or a word in a sentence. Attention helps estimate how strongly the vector is correlated with other elements and take the sum of their weighted values as the approximation of the target.

The concept of Attention was introduced initially in the Natural language Processing (NLP) domain in the paper by Sutskever et al [START_REF] Sutskever | Sequence to Sequence Learning with Neural Networks[END_REF]. Their work was motivated by how humans correlate words in one sentence or pay visual attention to different regions of an image, and aimed to transform arbitrary length input sequences to output sequences of words to form a sentence. Looking towards the breakthrough results in automated machine translations [START_REF] Vaswani | Attention Is All You Need[END_REF], attention mechanism made its way into computer vision by Xu et al. [START_REF] Xu | Show, Attend and Tell: Neural Image Caption Generation with Visual Attention[END_REF], followed by several other works implementing some form of attention mechanism into their networks. Attention can be divided into several categories such as Self-Attention, Global or Soft attention or local or hard attention, each with different concepts as described in the table 2.8. Out of these, Self-attention for generative adversarial networks was first intorduced by Zhang et al. [START_REF] Zhang | Self-Attention Generative Adversarial Networks[END_REF] in their SAGAN network, and added self-attention layers in both their generator and discriminator to better model relationships between spatial regions. Later Radford et al. [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF] in their network DCGAN added a soft self-attention mechanism to learn the positional relationship between pixels. This allowed them to handle the details of the generated images in a better way. The SAGAN adopts the non-local neural network to apply the attention computation. The convolutional image feature maps is branched out into three copies, corresponding to the concepts of key, value, and query in the transformer: Mathematically, it can be defined by equation 2.34 Then we apply the dot-product attention to output the selfattention feature maps:

o j = W N i =1 α i , j h (x i ) (2.34)
where α i , j is the weighted attention of the i-th value when calculating the j-th location, multiplied by W, a 1 × 1 convolutional filter.

Popular datasets for specular highlight research

For recent deep-learning-based methods, one of the main requirements of the development, training and testing of networks is the availability of large datasets that accurately encompass various specular reflections and their respective ground truths. While acquiring real-world images with specular reflections is relatively easy, acquiring the pure diffuse images of the same scene is a complicated task. This is because the only way to achieve images without specular reflections in natural, uncontrolled environments is to acquire images using a polarizer. However, even with a polarizer, completely removing all specular reflections in a scene is not easily achieved due to the presence of multiple light sources and random orientations and materials of the objects in the scene. This makes the acquisition of comprehensive datasets challenging and amplifies the importance and significance of datasets painstakingly acquired by researchers.

One of the reasons for the lack of research using polarimetric information for specular highlight mitigation has been due to expensive polarimetric cameras and the effort required to capture multiple images by manually rotating the polarizing filter. This leads to the lack of availability of large annotated datasets with specular highlights that can be used for research and explicitly training deep learning networks. Several solutions are now available that are cost-effective and have sensors with multiple polarizer angles embedded into the image sensor in a super-pixel configuration. Therefore, a single image can capture four polarimetric angles in one instance, and the resulting raw image can be demosaiced to get four polarimetric images, as shown in Fig. 2.8. The cameras were used to capture the four polarimetric angles simultaneously in a single acquisition generating a super-pixel image that can be demosaiced according to the Bayer pattern, as shown in Fig. 2.8. Simultaneous acquisition of all polarimetric angles provides spatial and temporal coherency among the polarimetric images and removes the need to register or align the images Figure 2.14: Set of classical images used for specular highlight mitigation in literature manually, as well as reducing any blur due to accidental unwanted camera movements etc., such as the ones faced by [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. This ensures that the only variation in the image is luminance, including specular reflection variations between the polarimetric images.

Additionally, most publicly available datasets lack pure diffuse images (groundtruth) by using cross-polarization (images captured with a polarized light source) and thus are insufficient for training machine learning algorithms for specularity removal. Some works have recently made available reliable datasets, thereby further enabling research. A comprehensive table of relevant datasets for specular highlight mitigation is given in table 2.9 summarising the essential characteristics of the datasets. The most extensive dataset of real-world images with a matching specular mask was recently made available by Fu et al. titled Whu-Specular dataset [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF].

They curated a large dataset of approximately 500 real-world images in the wild with specularity and have provided manually annotated specular pixel masks for each image. The authors also proposed a deep learning-based SHDNET that uses multiscale context contrasted features to detect specular highlights. Similarly, [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] also generated a large-scale real-world highlight dataset of around 4500 images containing a wide variety of material categories, with diverse highlight shapes and appearances; with each image provided with its annotated ground truth. Wu et al. [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] provide a large Paired Specular-Diffuse (PSD) dataset consisting of roughly 12000 images acquired over 12 polarimetric angles and 1600 paired specular-diffuse pairs of images. The pure diffuse images are acquired using cross-polarization, which ensures the removal of specular highlights in the diffuse images. A smaller dataset of 40 scenes captured with polarimetric cameras is also provided by Qui et al. [START_REF] Qiu | Polarization Demosaicking for Monochrome and Color Polarization Focal Plane Arrays[END_REF] containing several objects of varying materials with specularity. A quick summary of the popular datasets and sets of images used in literature can be seen in table 2.9. Some of the earliest literature on specular highlight mitigation only acquired a few images due to the lack of polarizers and digital imaging equipment. However, repeated usage of the early images made them popular for usage by literature and are still used to this date. A few of the example images are attached as Fig. 2.14.

Datasets for specular highlight mitigation

One of the reasons for the lack of research using polarimetric information for specular highlight mitigation has been due to expensive polarimetric cameras and the effort required to capture multiple images by manually rotating the polarizing filter. This leads to the lack of availability of large annotated datasets with specular highlights that can be used for research and explicitly training deep learning networks. Several solutions are now available that are cost-effective and have sensors with multiple polarizer angles embedded into the image sensor in a super-pixel configuration. Therefore, a single image can capture four polarimetric angles in one instance, and the resulting raw image can be demosaiced to get four polarimetric images, as shown in Fig. 2.8. The cameras were used to capture the four polarimetric angles simultaneously in a single acquisition generating a super-pixel image that can be demosaiced according to the Bayer pattern, as shown in Fig. 2.8. Simultaneous acquisition of all polarimetric angles provides spatial and temporal coherency among the polarimetric images and removes the need to register or align the images manually, as well as reducing any blur due to accidental unwanted camera movements etc., such as the ones faced by [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. This ensures that the only variation in the image is luminance, including specular reflection variations between the polarimetric images.

Additionally, most publicly available datasets lack pure diffuse images (groundtruth) by using cross-polarization (images captured with a polarized light source) and thus are insufficient for training machine learning algorithms for specularity removal. Some works have recently made available reliable datasets, thereby further enabling research. Fu et al. [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] have curated a large dataset of approximately 500 real-world images in the wild with specularity and have provided manually annotated specular pixel masks for each image. Similarly, [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] also generated a largescale real-world highlight dataset of around 4500 images containing a wide variety of material categories, with diverse highlight shapes and appearances; with each [START_REF] Grosse | Ground Truth Dataset and Baseline Evaluations for Intrinsic Image Algorithms[END_REF] 2009 RW 20

MB

IIW [START_REF] Bell | Intrinsic Images in the Wild[END_REF] 2014 RW 5000

1.5 GB CVC-ClinicDB [START_REF] Bernal | WM-DOVA Maps for Accurate Polyp Highlighting in Colonoscopy: Validation vs. Saliency Maps from XXXII BIBLIOGRAPHY Physicians[END_REF] 2015 MI 612 263 MB CVC-ClinicSpec [START_REF] Sánchez | Bright Spot Regions Segmentation and Classification for Specular Highlights Detection in Colonoscopy Videos[END_REF] 2017 MI 59 6 MB LIME [START_REF] Meka | LIME: Live Intrinsic Material Estimation[END_REF] 2018 Syn, RW 10k, 45

GB Polarization

Image dataset [START_REF] Qiu | Polarization Demosaicking for Monochrome and Color Polarization Focal Plane Arrays[END_REF] 2019 RW 40 935 MB

Whu Specular [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] 2020 RW 4310 2 GB PolaBot [START_REF] Blanchon | Polarimetric Image Augmentation[END_REF] 2020 RW 177 584 MB SHIQ [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] 2021 RW 10825 10.8 GB Whu TRIW [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] 2021 RW 500 835 MB PSD-dataset [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] 2021 RW 13,380

7.4 GB 2022 SIHR [START_REF] Wang | SIHRNet: A Fully Convolutional Network for Single Image Highlight Removal with a Real-World Dataset[END_REF] 2022 RW 200

MB SHMGAN (Ours)

2022 RW 330 2.3 GB 5 RW: Real-world 6 MI: Medical Imaging 7 Syn: Synthetic Images image provided with its annotated ground truth. Wu et al. [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] provide a large PSD dataset consisting of roughly 12000 images acquired over 12 polarimetric angles and 1600 paired specular-diffuse pairs of images. The pure diffuse images are acquired using cross-polarization, which ensures the removal of specular highlights in the diffuse images. A smaller dataset of 40 scenes captured with polarimetric cameras is also provided by Qui et al. [START_REF] Qiu | Polarization Demosaicking for Monochrome and Color Polarization Focal Plane Arrays[END_REF] containing several objects of varying materials with specularity.

Criticism on state-of-the-art

The accurate detection of specular highlights is significant in many applications and therefore has been an area of research for several decades. The current state-ofthe-art for specular detection and mitigation faces several challenges and issues as highlighted below.

Issues with current specular detection methods

Generalized and robust solutions

Classical methods for accurately detecting specular highlights have difficulty detecting pixels accurately in a wide variety of scenes containing lighter coloured objects, bright backgrounds, or complex-shaped objects with irregular specular reflections. One of the significant issues faced by the classical techniques is the robustness and generalisation of the techniques. While the methodologies are based on firm mathematical foundations and optimisation techniques, they are primarily based on assumptions that significantly limit their applications to general realworld images that are not part of their dataset. Thus, while the results are significantly better on the selected set of images, they do not apply to any general image taken from a generic camera under uncontrolled settings. Multiple research works on treating specular reflections using colour space transformations attempted to understand and tackle the problem purely from an objective often tested on a minimal set of images which fails to work beyond their preferred set. Methods based on polarisation classically use a manual polariser filter that is rotated to acquire images at different polarimetric angles. This means that the images are temporally incoherent, and unless taken of a static object under a static and controlled environment, the images face alignment issues where pixels do not share the same spatial instance between the polar images. This also limits the number of images that can be acquired as a significant amount of effort is required to take a broad and generalised dataset.

Illumination colour and SPD assumptions

Several assumptions are also made for classical methods to work, which are sometimes not reflective of real-world conditions, e.g., a single illumination is mostly assumed with a non-existent or minimal amount of inter-reflections from surrounding surfaces. The illuminants selected are assumed to be of pure white colour with known spectral power distribution (SPD) to simplify all chromaticity-based methods. It is further assumed that each segmented cluster has uniform diffuse chromaticity. While being very helpful for modelling the problem of specular highlight, these and other assumptions do not reflect real-world images' randomness and limit the generalisation and applicability of methods. Since most limitations are not considered for deep learning-based methods, it is quite clear that modern state-of-the-art methods are significantly more robust and can cater to a much more comprehensive range of images. However, limitations are enhanced in the presence of outdoor images, which have both strong illumination and inter-reflections in an uncontrolled and often stochastic environment. Simplifying the proposed networks to reduce the number of existing parameters and upgrade the presumption rapidity for its usages on the mobile computing programs [START_REF] Dong | Location-Aware Single Image Reflection Removal[END_REF][START_REF] Zheng | Single Image Reflection Removal with Absorption Effect[END_REF].

Outdoor environments

Outdoor environments have illumination from the sun as an omnidirectional light source, causing light to bounce off in often undesirable directions and strength. The synthesized and real-world images in some datasets of the proposed algorithms are all indoor scenes that may not be suitable for outdoor scenes. These proposed networks were not very effective and almost failed in outdoor scenes [START_REF] Zhang | Fast User-Guided Single Image Reflection Removal via Edge-Aware Cascaded Networks[END_REF]. Strong light sources also result in more significant specular regions in images, which makes the regions easily visible but also easily confused with the objects in the scene, as well as causing a significant loss of information in the area, which hinders the recovery of colour and other information in the affected region.

Limitations in mitigation of specular reflections

Imaging and scene conditions

Most proposed methods for specular highlight nitigation are not only unable to properly mitigate specular highlights but often have adverse effects on the image, such as altering contrast and distorting the colour of the objects in the scene [START_REF] Tan | Separation of Highlight Reflections on Textured Surfaces[END_REF].

Several proposed solutions are highly dependent on illumination conditions, reflectance, material properties and colour of the source lighting. Furthermore, existing traditional methods cannot often distinguish coloured specular reflections in images. Similar to the issues for specular highlight detection, image regions affected by extreme specular highlights and saturation are especially poorly estimated by most algorithms that entirely or partially fail to estimate and restore the underlying colour of the affected objects. Generally, accurate and reliable methods trade-off highlight-mitigation accuracy with speed and thus are not suited for real-time applications. These problems are unsolved and indicate that specular highlight mitigation is an area with a notable gap in the availability of a fast and accurate detection and mitigation method.

Limited datasets, handcrafted features and deep neural networks

The state of the art in specular highlight mitigation are deep neural networks which have shown to provide reasonable amount of mitigation to most non-ideal images with strong reflections [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF][START_REF] Hou | Text-Aware Single Image Specular Highlight Removal[END_REF][START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] and provide confidence for future development in the field. In some other networks, the diversity of scenarios and capturing settings for the images which are included in the synthetic dataset needs to be improved. These problems in data generation may restrict the generalization ability of the dataset [START_REF] Kim | Single Image Reflection Removal With Physically-Based Training Images[END_REF]. When the whole images are dominated by reflection or ghosting reflection, which makes it so hazy and blurry, or the reflection layers and background are overlapped, the effectiveness of the proposed networks may drop, and these networks may not be able to completely remove the reflections, and the evaluated background still remains visible residual edges. Also, the proposed technique may have some problems with gradient disappearing when the deep learning technique is trained directly on the images [START_REF] Zhang | Fast User-Guided Single Image Reflection Removal via Edge-Aware Cascaded Networks[END_REF][START_REF] Chang | Joint Reflection Removal and Depth Estimation from a Single Image[END_REF]. According to the fact that some of the presented networks are operating based on the extracted edges, these algorithms may not work properly whenever there is a loss of edge information, or the edge information is low-confident [START_REF] Zhang | Fast User-Guided Single Image Reflection Removal via Edge-Aware Cascaded Networks[END_REF]. Some of the proposed networks rely on handcrafted features. Proposing and designing a more hand-free and automated reflection removal algorithm than the proposed ones, which can free users from guidance and suppress reflection with high quality, can be mentioned as a future direction, and it is expected to successfully deal with the limitations in challenging reflection removal tasks [START_REF] Zhang | Fast User-Guided Single Image Reflection Removal via Edge-Aware Cascaded Networks[END_REF].

Summary

In this chapter, we firstly explored in depth the various physical models of reflection of light that have been developed to explain the phenomenon in a generalized manner. Specifically, the DRM model by Shafer et al. has proven to be simple yet diverse enough for the development of accurate detection and mitigation methods. The importance of specular highlight detection and mitigation, the relevant literature and its real-world applications are explored in depth. Detection and segmentation of specular reflections was explored in detail. Both classical and deep-learning solutions were reviewed, giving a broad overview as well as deep insight on the benefits and pitfalls of classical computer vision based methods. The polarisation character-istics that are inherent to reflection were specially studied in depth as they provide a promising and a plausible way for removal of specular reflections based on physical properties of light. The limitations of the current state-of-the-art methods on detection and segmentation were also explored based on the extensive literature and research works available. Having developed a strong foundation of the current methods in vogue and their benefits, in the next chapter, we will addressthe detection and mitigation of the detected specular highlights by developing separate methods that are specialized for each task, and extend them to a wide range of images taken under various conditions.

Chapter 3

Methodology "There's a benefit to losing; You get to learn from your mistakes!"

Megamind

Chapter abstract

In this chapter, we tackle the problem of detecting, segmenting and mitigating specular highlight pixels in real-world images. With extensive literature review done in chapter 2, traditional image processing methods are shown to be inadequate for a generic solution that works under a wide assortment of real-world images. We take inspiration from the current state-of-the-art and apply the philosophy of Occam's razor to simply the network and the resources required to get at-par or better results than competing methods. To this end, we develop Specular Segmentation (SpecSeg), a fast-

to-train yet highly effective deep learning network that is able to accurately detect and segment out the specular pixels and regions with precision. We also show a fast diffuse colour inpainting method that utilises the detected regions from our developed SpecSeg network and inpaints the affected regions with an estimated diffuse colour inferred from the boundary regions. The advantages and limitations of this classical computer-vision based method are also discussed establishing the need for deep neural networks for a robust solution. We leverage deep neural networks and take advantage of the varying illumination information in polarimetric images for synthesizing specular free images. For this, we develop a multi-domain attention-based

Specular Highlight Mitigation Generative Adversarial Network (SHMGAN) trained using multiple polarimetric images simultaneously. The developed network uses a dynamically generated attention mask and requires no manual input for segmenting specular pixels. The network is able to learn the illumination variation between the four polarimetric images and a pseudo-diffuse image without requiring extensive training data or time. Once trained, SHMGAN is able to generate specular-free images from a single RGB image as input; without requiring any additional external or pixel labels.

Overview

Exploring the state-of-the-art methods used to detect and mitigate specular highlights, it is evident that classical computer vision methods have been developed with an in-depth understanding and mathematical modelling of light reflection, polarization and other physical properties. However, the robustness of such methods is still found lacking as they are mostly applicable to limited images, taken in mostly controlled environments. Over the recent years, deep-learning-based solutions have not only shown spectacular results but also have shown to work on a wide variety of images that were not possible by classical methods. taking inspiration from the state-of-the-art, we explore and develop three methodologies in this chapter. Deep-learning based specular highlight detection and segmentation network, a fast inpainting method using calculated weighted median, and lastly a multi-domain generative adversarial network that uses polarimetric information for learning to mitigate specular highlights. An overview of the methodology chapter and the three methods developed is given in Figure 3.1. The developed methods are explored in depth, addressing the details and reasons for the selection of various parameters and concepts that have been learned from the literature as well as from experimentation.

Specular Segmentation (SpecSeg) network

In this section, we explore the motivation and details of the different building blocks of the developed SpecSeg Network in detail and the unique characteristics of Spec-Seg Network itself. We conclude this section with a detailed comparison between the developed SpecSeg Network and state-of-the-art networks for specular highlight segmentation.

Motivation

As already detailed in the preceding section, accurate detection and segmentation of specular pixels from real-world images have significant implications in various fields. This work intends to fill the research gap and add to the current state of the art in specular highlight segmentation. To achieve this, we propose a specular highlight segmentation network that is simple to model, fast to train and works on images used in literature as well as a wide variety of general real-world images. Most state-of-the-art deep learning models are complex structurally, with a complex organization of deep hidden layers and innovative, unique features such as attention and other methods. Secondly, due to their complex design, they require a significant time to train and fine-tune due to a large number of hyperparameters in the model and deep neural network layer structure. This, in turn, causes significant hindrances in research and development due to unoptimized training times required while expected nominal results are not achieved. Furthermore, complex and deep networks also mandate the utilization of expensive and powerful hardware, consuming a lot of power while training and re-training.

We avoid both these pitfalls by our developed Specular Highlight Segmentation Network (SpecSeg Network for short), based on the proven U-Net model, which is a highly reliable yet straightforward model that was initially proposed for medical segmentation [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF]. Our experiments show that this decision makes the specular highlight detection network simple to build and requires significantly less time and fewer resources to train. This enables increased experimentation and re-training opportunities without trading accuracy or precision from the existing state-of-theart methods. Furthermore, we also show that using SpecSeg Network; it is possible to detect specular highlights after a speedy training process on a relatively small dataset and generate accurate detection results on real-world images. The affected pixels are accurately marked in a wide assortment of images taken in random uncontrolled settings and improve upon the existing state-of-the-art in specular highlight detection.

U-Net and image segmentation

With a brief introduction of the basics of deep learning covered in the previous section, we now move on to our application-specific convnets, namely the networks used for image segmentation. There are three essential computer vision tasks: image classification, image segmentation, and object detection. In image classification, the goal is to assign one or more labels to each image pixel, depending on whether the problem is a single-label classification or multi-label classification problem. In an image segmentation task, the goal is to "segment" or "partition" an image into different areas and provide an outline of the region within the image, such as background and foreground, specular and non-specular etc. Each segment can then be used for algorithms for analysis or other tasks, such as mitigation in the case of specular highlights. An image segmentation task generates a segmentation mask, which is the same size as the input image and can be a binary or multichannel image based on a single or multi-class segmentation problem. For the case of specular highlight segmentation, the required mask has to be a binary image, segmenting each pixel in either specular or non-specular categories. To differentiate between image classification and segmentation, the former process groups all the relevant regions into a groups or categories whereas segmentation only separates out all regions of interest in an image. Both image classification and object detection are a precursor of image segmentation as both techniques must occur before segmentation can begin. One of the first CNN architectures to allow automatic end-to-end semantic segmentation is the FCN [START_REF] Long | Fully Convolutional Networks for Semantic Segmentation[END_REF]. FCNs are derived from deep classification models such as VGG16 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF], AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] or GoogLeNet [START_REF] Szegedy | Going Deeper with Convolutions[END_REF], by removing the corresponding classification layers, i.e., replacing their fully connected layers with convolutional ones, and plugging in an upsampling path that is dedicated to transforming coarse outputs into dense predictions. . With its ability to extract multi-scale features, fully connected networks set a milestone in segmentation approaches and paved the way for encoder-decoder segmentation networks. To increase depth and precision within the learnt contextual features, many works within the field advocate going deeper with FCN layers ( [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF]; [START_REF] Szegedy | Going Deeper with Convolutions[END_REF]). Improving a model's prediction ability by adding deeper hidden layers to a fully connected layer is a task of increasing difficulty. One side effect of adding the said deeper layers is the loss of global and spatial information leading the network [START_REF] Jurdi | BB-UNet: U-Net With Bounding Box Prior[END_REF] and prone to produce fuzzy or blurred predictions and segmentations. Moreover, deepening the convolutional network will often increase the model's complexity, thus subjecting the training to additional challenges such as vanishing gradients. As a result, deep FCNs may suffer from performance saturation or degradation while training. To address these issues, many FCN improved variants have emerged, among which is the very well-known U-Net introduced by Ronneberger et al. [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] Since its inception, U-Net has proven to be a breakthrough for segmentation tasks and has been instrumental in paving the way for developing a more advanced encoder-decoder style of networks. The network is named after the U-shape of the hidden layers, combining an encoder-decoder arrangement with convolution, activation and pooling operations between its successive hidden layers. With its peculiar arrangement, this specific architecture allows the network to propagate context information to higher resolution layers by introducing skip-connections between the encoder and decoder parts. The encoder-decoder generator architecture takes an image as input and downsampling it over a few layers until it becomes a bottleneck layer. The representation is then upsampled over a few layers before outputting the final image with the desired size. The U-Net model architecture is very similar in that it involves downsampling to a bottleneck and upsampling again to an output image. However, links or skip connections are made between layers of the same size in the encoder and the decoder. It learns to segment images in an end-to-end setting, i.e. the network input is a raw image (which can be in a single or multi-channel colour space), and the output image is in the form of a segmentation map. The traditional U-Net is able to segment multiple objects in an image even if their boundaries are colliding. Skip connections allow high-level features from the encoder to be passed on to the decoder's generated outputs and significantly affect the quality and accuracy of the U-Net output [START_REF] He | Identity Mappings in Deep Residual Networks[END_REF]. By parsing the input image through down convolutions and pooling in an encoder, the network learns to identify the target regions in a scale-agnostic manner. The U-Net network has been shown to work with high accuracy and detect objects with substantial shape variations, weak borders and inset or overlapping objects. Due to these properties, the U-Net forms the primary building block of our developed SpecSeg Network for de-tecting specular highlights in real-world images.

SpecSeg network model and implementation

Since its inception, U-Net has proven to be a breakthrough for segmentation tasks and has been instrumental in paving the way for the development of a more advanced encoder-decoder style of networks. The network is named after the U-shape of the hidden layers, combining an encoder-decoder arrangement for downsampling the input to a bottleneck and upsampling again to an output image, with convolution, activation and pooling operations between its successive hidden layers. Skip connections allow the network to propagate context information from higher resolution layers to the decoder's generated outputs and significantly affect the quality and accuracy of the U-Net output [START_REF] He | Identity Mappings in Deep Residual Networks[END_REF]. By parsing the input image through down convolutions and pooling in an encoder, the network learns to identify the target regions in a scale-agnostic manner. The network thus learns to segment images in an end-to-end setting, i.e. the network input is a raw image (which can be in a single or multi-channel colour space), and the output image is in the form of a segmentation map. The U-Net network has been shown to work with high accuracy and detect objects with substantial shape variations, weak borders and inset or overlapping objects. It has also been shown in literature that a simple but properly trained U-Net architecture can match and even surpass the state-of-theart approaches for image segmentation [START_REF] Galdran | State-of-the-Art Retinal Vessel Segmentation with Minimalistic Models[END_REF]. Due to these properties, the U-Net forms the primary building block of our developed SpecSeg Network network for detecting specular highlights in real-world images. The developed deep convolutional network layout is shown in 3.2, and the following sections discuss the design and reasons for selecting the hyper-parameters.

Encoder and decoder blocks

SpecSeg comprises of 5 encoder blocks and 4 decoder blocks based on the classical U-Net pattern, and each path from the encoder is passed to the decoder via a skip connection. Each encoder block consists of two 2D convolutional layers with filters (k) = 3 and stride (s) = 3 with same padding and uses ReLU activation in the output of each convolutional layer. The (3 × 3) filter has been inspired by the original proposed U-Net configuration. However, a stride of (3 × 3) is added to avoid overlap when convolving the filter, as it was experimentally determined to give the most favourable results during testing and evaluation. While in the original paper, Ronneberger et al. [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] propose unpadded convolutions in the encoder section, it has been shown [START_REF] Alsallakh | Mind the Pad -CNNs Can Develop Blind Spots[END_REF] that the choice of padding has a direct effect on the performance of a model. Without padding, the input layer volume size reduces too quickly as a deeper network is designed. Stacking multiple unpadded layers also ignores the image's border pixels, resulting in a loss of learnable information around the borders.

Since specular highlights can also extend to the border of the input images, adding padding around the border increases the chances of detecting specular pixels near the border of the input image.

An incremental dropout of 10%, 20% and 30% respectively is also introduced between the two convolutional layers of the first, third and fifth encoder block to improve the robustness of the learned features. By incrementally increasing the dropout, the network is able to learn sparser representations of the high-level features and in return, improves the accuracy of the detection of specular pixels. The training was done on a batch size of 16, and a BN layer was introduced in the encoder sections before the pooling layer. BN has proven to be a reliable normalization method for segmentation networks [START_REF] Chollet | Deep Learning with Python[END_REF], and the same was confirmed by our experimentation as well, making it a sound choice. Lastly, to reduce the variance and computational complexity as we go deeper in the U-Net, we need to reduce the size of the feature map. This is achieved with a MaxPooling layer which selects the maximum value out of a 2×2 block, reducing the size of the feature set. Maxpooling ensures that the most critical features (denoted by the maximum valued pixels) are taken from each block only.

The decoder block mostly mirrors the encoder block setup defined above with a few notable changes. Firstly the decoder performs an upscaling operation. This is done using 2D transpose convolutional layers with filters k = 2 and stride s = 2. A similar incremental dropout between two consecutive convolutional layers is also used. However, the final convolutional layer uses filter and strides of k = 1, s = 1 respectively and sigmoid activation to generate a 256 × 256 × 16 mask images of the entire batch similar in size the input images.

Thus the overall U-Net structure takes batches of 16 images of resolution 256 × 256 as input and generates mask images as output for all of the 16 images while learning the weights during the downscaling-upscaling operations in the encoder-decoder

Loss functions

As the training progresses and the network learns the weights related to the feature maps, the error for the model's current state must be estimated repeatedly. This is part of the optimization algorithm being employed. The set of functions used to estimate this error is called a loss function. Loss functions depend profoundly on the problem being solved and are often tailored to the task at hand. As deep learning has progressed, researchers have developed and proposed several known loss functions over the years that have shown to be very reliable for particular problems. Furthermore, total losses formed by weighted additions of different losses have proven very useful. For specular highlight segmentation, we selected a linear combination of Dice similarity coefficient (DSC) [START_REF] Milletari | V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation[END_REF], and Focal loss [START_REF] Lin | Focal Loss for Dense Object Detection[END_REF] as experiments proved that the combination of these losses showed the best segmentation results.

Dice similarity coefficient: DSC is a spatial overlap index developed to measure the pixel-level similarity between two images, where one is generally the binary mask image. DSC loss function has values ranging between 0-1. Lower values indicate minimum spatial overlap between two sets of binary segmentation results, whereas larger values nearing 1 indicate increasing overlap, where 1 represents 100% complete overlap. The Dice similarity coefficient has been adopted widely in biomedical segmentation problems where manually annotated lesions or cancerous cell datasets are available to train segmentation algorithms. Mathematically, the dice similarity loss (or dice loss for short) is defined as 3.1.

L Di ce (p, p) = 1 - 2 p h,w ph,w p h,w + ph,w (3.1)
The loss is calculated in terms of the per-pixel classification of TP, TN, FP and FN.

Where p is the ground truth, p is the predicted probability and p h,w ∈ {0, 1} and 0 ≤ ph,w ≤ 1

Focal loss: [START_REF] Lin | Focal Loss for Dense Object Detection[END_REF] addresses class imbalance during training by applying a modulating term to the cross entropy loss to focus learning on hard misclassified samples.

Alternatively, it can also be visualized as a dynamically-scaled cross-entropy loss, where the scaling factor decays to zero as confidence in the correct class increases.

Intuitively, this scaling factor automatically down-weights the contribution of easier training samples and rapidly converges the model to focus on harder examples.

Mathematically focal loss can be defined as:

L Foc al (p, p) = -α(1 -p) γ p log( p) -α(1 -p) pγ log(1 -p) (3.2)
Total loss: By adding the losses mentioned above, we can create a total loss that calculates the true positive segmented pixels and enables the network to focus on the misclassified samples of the training dataset. The dice loss maximizes the overlap between predicted and actual labels, whereas the focal loss addresses class imbalance by reducing the effect of biased or skewed classification on the predicted results. The total loss function is defined as a linear combination of both the Dice loss and Focal loss and is used for backpropagating over all learnable parameters.

L Tot al = κ d L Di ce + κ f L Foc al (3.3)
This weighted combination of both dice and focal losses provides the most accurate segmentation results, as will be shown in the results section. After several experiments carried out to find the most suitable weights, κ d = κ f = 1 was found to be the most suitable set of weights with repeatable results.

Results of specular highlight segmentation

As already highlighted in the proposed flowchart at section 1.8, specular highlight segmentation is a precursor to specular highlight mitigation techniques developed in Section 3. The results and discussions on the image segmentation are discussed at length in section 4.2.

Weighted-median inpainting for specular highlight removal

One of the most feasible methods to mitigate specular reflection is inpainting, where the target region pixels are replaced by colour information that is inferred by the method in use. This replacement can be iteratively or in one go depending on the method in use. However, iterative inpainting methods have shown to be significantly slower in estimating the diffuse colour. So one of the target objectives of any inpainting method is to improve the estimation time with a little trade-off to the estimated colour accuracy. Taking inspiration from the state-of-the-art inpainting methods, we propose a fast inpainting method that infers the colour information of the affected region from the neighbouring pixels of the affected region and is explained in depth in the subsequent sections.

YCbCr colour space for illumination separation

While RGB colour space is the defacto standard for developing image processing algorithms, there are several disadvantages related to the colour space. One of the primary issues is that the three-channel colour space combines the illumination information and the colour information of the objects. While this is not a concern for most applications in the computer vision domain, it is not beneficial for our particular problem, which requires colour information to separate out the specular effects of the illuminant. An alternative to the RGB colourspace is the YCbCr space. It is defined by a coordinate transformation of the associated RGB colour space and has a similar three-channel configuration, and is visualized in the Figure 3. In order to separate out the illumination information from the colour, we proposed using the YCbCr colour space for specular highlight segmentation. This decision was made because of the fact that it has been shown mathematically that the CbCr channels in the YCbCr colour space are free of the specular highlights [START_REF] Vítor | Single Image Highlight Removal for Real-Time Image Processing Pipelines[END_REF]. Another pertinent observation regarding specular highlights in the YCbCr domain is that due to the way RGB colour space is oriented in YCbCr space, the black-white (0 to 255) RGB colour axis is arranged along the Z-axis (Y-Channel), whereas the Cb and Cr axes values are at 0.5. Due to this arrangement, whenever a region is affected by a white specular highlight, the pixel colour of that region in YCbCr space always shifts towards this central axis, and the intensity of the specular highlight is along the Y-Channel. This colour shift information can be useful since the colour of the specular region can be inferred from the pixel values of the surrounding region where the shift approximately starts. The observation holds in cases where the specular highlight saturates the imaging sensor, and there is a strong peak in the Y-channel along with a clear shift of the surrounding region in Cb and Cr channels to 0.5. This concept lies at the core of our developed inpainting method, where the specular region is first segmented out in the image, and then the colour of the surrounding region (from where the colour shifts) is estimated.

Segmenting specular highlights using Y-Channel

The developed method for specular highlight inpainting is composed of two major stages, namely, specular segmentation and weighted median inpainting. A complete flowchart of the entire process is shown in Fig. 3.4 and the pseudocode of the developed method is also given in Algorithm 1. The first stage of the proposed method is separating diffuse and specular images using the Modified Specular-Free (MSF) image technique by Shen et al. [START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF]. However, in contrast to using the three channels of the RGB colour space, the proposed method utilizes only the Y-Channel from the YCbCr colour space. This allows the processing of the luma channel of the images without affecting the colour information by utilizing only the Y-channel.

Ideally, the output of the MSF using an RGB image as input is a diffuse image. However, since only the luma (Y) channel is used for generating the specular free image, YCbCr colour space can be written as equation 2.9. Also part of the first stage is to generate a specular mask of the affected pixels. Several classical methods from the ones listed in Table 2.2 were tested to get an accurate masks. However most of the Figure 3.5: Mesh displaying the distance transform methods failed to be robust enough for a generalized and reliable solution. Therefore, we used the output of the developed SpecSeg network as it proved to be robust and accurate as will be shown in the quantitative and qualitative results in section 4.2. The correct segmentation of the specular image is critical and has a significant impact on the final diffuse image as better segmentation results in reliable thresholding and estimation of the colour from the surrounding region.

the
Since real-world images can contain multiple objects or regions in a scene that are affected by specular reflection, each area will have its own base diffuse colour, which is not necessarily unique. To cater for this issue, the mask is then used to generate three different sub-images, namely region segments, region boundaries and distance transform images. Firstly, the specular image is segmented into separate regions using the mask based on contiguous connectivity, and each region can be treated as a separate entity for performing all subsequent calculations. Secondly, the boundary pixels of the segmented specular regions are identified. Lastly, a normalized weight matrix is generated using the Euclidean distance transform of each region, where the weight increases from zero at the boundary pixels to the centre of the region. These three sub-images are used together for generating the inpainting matrix in the next stage.

In diffuse images generated in the first stage, there is a clear loss of colour infor-mation in places where the pixels were affected by specular highlights. This is because specular reflection distorts the colour information of the body, and once the specular highlights are removed, most methods do not attempt to explicitly improve upon the colour information. For our case, since only the Y-Channel is utilized for generating the specular image leaving the colour channels Cb and Cr untouched, the situation is the same and needs to be followed up with a method to improve the colour information recovery in the diffuse image. In the second stage, once different regions in an image are segmented using the mask, each region can be processed independently of the other. The localized colour information of each segmented region can be theoretically approximated by the median colour of that segment.

However, this colour information is lost due to the superposition of the illuminant colour. The segmented diffuse image (obtained after removing specular highlights) also contains colour information of the region surrounding the specular highlight.

The median of this surrounding region can, therefore, be considered as representing the diffuse colour of the region. A median is preferred over a mean to avoid any skewing of colour data in the presence of dark or extremely bright pixels in the surrounding region of the specular highlight. However, filling the entire region with a constant colour value results in a discontinuous colour patch. Therefore the inpainted value needs to be adjusted so that it gradually increases from the boundary towards the calculated median value.

Utilizing this concept, the three sub-images mentioned above are used to inpaint the specular regions with a Weighted-Median (WM) of the information in the three channels (not to be confused with the statistical quantity weighted median). The weighted median is calculated using the euclidean distance transform, as proposed by Maurer et al. [START_REF] Calvin R Maurer | A Linear Time Algorithm for Computing Exact Euclidean Distance Transforms of Binary Images in Arbitrary Dimensions[END_REF], which is a fast sequential algorithm to calculate the exact euclidean distance transform of pixels in a binary image defined by Equation 3.4, where w n is the weight of the i , j coordinates of the pixel of a k dimensional binary image. The p value of 1, 2 and ∞ are known as the Manhattan, Euclidean, and chessboard distances, respectively.

∆ i , j = k n=1 |w n i n -j n | p 1/p (3.4)
The distance transform results in floating-point distances between the current pixel and the nearest non-zero pixel of the binary image and can be treated as a weight Ω(i ) ← r eg i ons(I specul ar (i )) 9:

∂Ω(i ) ← bound ar y(I specul ar (i )) 10:

∆(i ) ← nor mal i ze(∆ x i , y i × I specul ar (i )) 11: μr (i ) = med i an(I(x, y))∀(x, y) ∈ Ω(i ) 12: μb (i ) = med i an(I(x, y))∀(x, y) ∈ ∂Ω(i ) 13: for c = 1...3 do 14: I c (x, y) = (∆(x, y). * μr ) + ((1 -∆(x, y)). * μb ) 15:
end for 16: end for 17: I d (YCbCr ) ← conc at (I c=1..3 ) 18: I RGB ← conver t (I d (YCbCr ) ) matrix for multiplication with the calculated region median. The benefit of using a distance transform over a centroid-based approach is that weights conform to the shape of the region instead of a fixed centre of mass. Only utilizing this weighted matrix for inpainting still results in an unnatural transition of colour and causes a colour discontinuity in the image. To circumvent this, a normalized weighted matrix is utilized such that the values along the boundaries transition gradually from the edge towards the centre of the region and ensure that the pixels values at the edge of the region do not go below the minimum colour of the boundary. This can be visualized using the gradient colour image in Fig. 3.5, where the normalized weights increase along the edge of the region towards the median value in the centre. Consider an image I(m,n) such that I(i , j ) is the pixel location inside Ω (the area to be inpainted) and ∂ Ω is the said regions' boundary. Then the median of the region μr and boundary μb pixels can be calculated using Eqn's 3.5 and 3.6 respectively :

μr = median(I(i , j )) for ∀(i , j ) ∈ Ω (3.5) μb = median(I(i , j )) for ∀(i , j ) ∈ ∂Ω (3.6)
Let ∆ be the distance transform matrix for the region. Multiplying ∆ with the median value of the region and boundary values, as shown in Eqn. 3.7, it can be ensured that the inpainted colour gradually scales up from the boundary colour to the centre of the region (median value) for each pixel in the region.

W m,n (i , j ) = (∆(i , j ). * μr ) + ((1 -∆(i , j )). * μb ) (3.7)
This weighted median matrix is applied to all three channels, which can then be combined to form an updated diffuse image, inpainted with the estimated colour of the body.

Summary -WM inpainting for specular mitigation

The developed method developed in the previous section is a relatively fast inpainting method that utilizes the accurate segmentation information from SpecSeg network and inpaints using the surrounding colour pixel information to fill in the affected regions. As will be shown in the results section 4, inpainting is a fast and moderately accurate method to mitigate the specular regions and estimate the diffuse colour that is used to replace the specular reflection region. The estimated colour information is a good approximation of the diffuse colour of the body as it depends on the surrounding pixel colours and attempts to smoothly reproduce the colour from the boundaries to the centre of the affected region. While the colour recovered is a good approximation, as discussed in section 4.3, there are several limitations and drawbacks, similar to those associated with classical computer vision algorithms for specular highlight removal. Firstly, the mitigation is not able to recover the textural information of the surface as it does not cater for the surrounding texture of the region. Only plain coloured surfaces are restored using inpainting. Additionally, the recovery is dependent on the pixel-accurate segmentation of the specular region so that the correct boundaries are identified for colour recovery.

Any erroneous pixel misclassified as non-specular affects the final recovery as the colour information propagates while inpainting the region.

These limitations in the simplistic weighted median inpainting method lead us to further explore methods that are agnostic to prior segmentation of specular regions as well as being able to recover textural information intelligently. The modern state-of-the-art methods enable intelligent learning networks that can be trained to work with very little prior information required for restoring damaged images.

These methods fall under the broad category of image-to-image translation networks, which learn the information from the images in the training dataset to translate the learned information to other images that are given as inputs. An in-depth explanation of the developed network, including development techniques and network structure, is given in section 3.4.

Specular Highlight Mitigation GAN (SHMGAN)

Over the years, a wide variety of classical methods have been developed for mitigation of specular highlights in images as surveyed in section 2.3.1. Some of the proposed works also showed high performance in the images that were selected for evaluating the performance. However, almost all classical methods have shown to be significantly less robust to images taken in an uncontrolled environment, which have several issues varying from large saturated specular regions to multiple colour specular reflections from non-dielectric objects such as metals. The DRM model was developed for di-electric materials since the reflectivity of metals causes significant challenges due to inter-reflections from the surroundings. While DRM is extremely successful in modelling the problem of specular reflections, there is a need for developing material and model-agnostic methods that are robust to the number and colour of lights in the scene as well as the types of materials of the target object. Keeping this in view, we propose the utilization of deep learning based models for mitigation. There are several state-of-the-art networks that can be used to treat similar problems, as highlighted in depth in section 2.3.4. However, by leveraging the strengths of generative models such as GANs, it might be possible to train networks that learn the underlying diffuse colour of objects without delving into the segmentation of intrinsic sub-components.

In the following sections, the developed generative adversarial network for the task of mitigating specular highlights in real-world images is described in depth. The main objective of developing the network is that there are the network is able to mitigate specular highlights in natural real-world images. Additionally, there are no requirements for prior segmentation of the affected region or any limitations to the amount or type of illuminants used for the acquisition of the image.

The developed network is a multi-domain image-to-image translation generative adversarial network using a Single Generator-Discriminator pair inspired by Star-GAN by Chi et al. [START_REF] Choi | StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation[END_REF]. Similar to StarGAN which is aimed at image to image translation between multiple domains, SHMGAN is aimed towards image synthesis after learning the variation between multiple domains (in this case polarimetric images from different angles) and generate specular-free images from a single RGB image input. SHMGAN utilizes polarimetric images in YCbCr colour space and learns the variation in illumination in the Y channel of the images. This allows the separation of the colour components from the illumination such that any alteration to the luma does not distort the hue of the pixels. The CbCr channels also have the property of being specularity free as shown by Ramos et al. [START_REF] Vítor | Single Image Highlight Removal for Real-Time Image Processing Pipelines[END_REF]. The developed network generates cyclic Y-channel images and uses a combination of self-attention and multiple losses to remove the specularity from the input RGB images. To give a complete overview of the developed network, we first establish the polarimetric images used as input, followed by the details of the network architecture of the deep neural network as shown in Fig. 3.6 and 3.7.

Polarimetric images

The SHMGAN generator-discriminator pair is modelled to learn the illumination variation between 5 input images. We use four orthogonal pairs of polarimetric images I 0,45,90,135 to capture the maximum variation in specular highlights. The cameras used for acquiring our data were polarimetric colour and monochrome Polar-Cam cameras from 4DTechnology and demosaiced to get the four polarimetric images as shown in Fig. 2.8. This facilitated in capturing of spatial and temporally coherent polarimetric images in various settings. To ensure replication of real-world settings, a significant part of the data collected was using unpolarized lights to reproduce the images acquired under natural conditions. A smaller set of images was also captured using cross-polarization by using a single polarized light source to illuminate the objects, which maximizes the chance of getting a pure diffuse image in at least one of the four polarimetric channels. Objects included in the data acquisition consisted of different materials, including plastic, metals, glass and other transparent objects. Images were also acquired outdoors under sunny conditions to capture severe specular highlights of cars and signboards. Some of the various images captured can be seen in Fig. 2.7.

Pseudo-diffuse image

In addition to the four polarimetric images, a pseudo-diffuse image I ED is estimated and passed on to the network as the fifth input domain. This is based on the intuition explained by Eqn. 1.1 that the illumination is a linear sum of the diffuse image, polarized specular highlight and unpolarized specular highlight. The pseudodiffuse image (ED) is calculated by taking the element-wise minimum of the four polar images I ϕ pol shown in Eqn. 3.8; resulting in the removal of maximum polarized specular highlights. It is considered the target image for the network to generate as specified by the target on-hot encoded labels.

I ED (x,y) = mi n(I ϕ pol ( x,y) ) ∀ ϕ pol (3.8)
This estimated diffuse image can be considered an initial solution to the specularfree image as it contains the least specular reflection component and is required to aid the convergence of SHMGAN towards the desired specular-free image. Providing this estimated diffuse image also circumvents the requirement of a fully-diffuse image as the target domain, which can only be acquired by image acquisition under strict cross-polarization conditions and a single illuminant. As a result, realworld images taken under various conditions, including multiple unpolarized light sources and brightly illuminated outdoor environments, can be used, and a target domain can be provided to the developed SHMGAN. Furthermore, since the network is designed to learn over a large number of polarimetric images, inaccuracies (such as small-scale specularities) in the pseudo-diffuse image are easily compensated while learning.

SHMGAN network structure

Generator

The developed SHMGAN network consists of a single Generator (G) and a Discriminator (D). The generator is based on the U-net structure and consists of 5 encoders, four decoders and one residual block, with each path from the encoder passed to the decoder via a modified skip connection. All blocks consist of 2D convolutions layers with LeakyReLU activation followed by an Instance Normalization (IN) [START_REF] Ulyanov | Instance Normalization: The Missing Ingredient for Fast Stylization[END_REF] layer. The encoder uses average pooling to downsize the layers, whereas the de-

Self-attention mechanism

For SHMGAN, we use the output of the pre-trained SpecSeg network developed in section 3.2. The SpecSeg network is trained on the Why-Specular dataset as described in chapter 2 and outputs a binary specularity mask. The mask is Maxpooled and convolved with two 2-D convolutional layers with k=3, s=3 and 'same' padding, followed by a LeakyReLU acivation. The attention masks calculated at the encoder side are then added to the corresponding decoder side.

Discriminator

The discriminator is comprised of 5 blocks of 2D convolutional layers with kernel sizes k = 4 and s = 2, followed by an instance normalization and LeakyReLU activation and the same padding. The self-attention mechanism is also added to the discriminator before the last convolutional layer block. The discriminator has two outputs; one is the classification of the real or fake (D cl s f ) while the other is the target label of the generated images (D ad v ). The real/fake classification is a 2D convolutional layer with Leaky ReLU activation, whereas the target label classification is done using a dense layer with softmax activation.

One-hot encoding

All input images are converted to the YCbCr colour space and normalized to [0, 1] before being input to the generator but converted back to RGB before being fed to The one-hot encoded channels are used to designate the target images for the generator as well as the target label for the discriminator. We use a mini-batch size of 1,

where each batch is considered as a set of the five spatially and temporally coherent input images I 0,45,90,135,ED .

Network losses

In order to train the single generator-discriminator pair, we introduce several losses and minimization constraints. An ablation study is explored in this paper in section 4.4.4 to explore the effect of the losses L further.

Multiple cyclic consistency loss

The classic cyclic consistency loss proposed by CycleGAN has to be modified to cater for multiple inputs. For a multi-input system, assuming one of the output from the generator is xa then the combinations of cyclic reconstructed images from the generator can be represented by xb|a , xc|a , xd|a and xe|a where each of the domains ae can be represented as the set of Eqn. 3.9 [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF].

xb|a = G ({ xa , x c , x d } ; b) xc|a = G ({ xa , x b , x d } ; c) xd|a = G ({ xa , x b , x d } ; d ) xe|a = G ({ xa , x b , x c } ; e) (3.9)
The multiple cyclic consistency loss L c yc,a for a multi-input system can then be defined as:

L c yc,a = x b -xb|a 1 + x c -xc|a 1 + x d -xd|a 1 + x e -xe|a 1 (3.10)
where 1 is the L 1 -norm.

L 1 loss

To ensure a sound quality generation of images similar to the input image, the L 1 loss is used to guide the generator. It is defined by Eqn. 3.11, which is the absolute difference between the generated cyclic images and the original polarimetric images, and the error is back-propagated, allowing the generator to synthesize realistic images. L 1 loss has shown to perform significantly better than L 2 loss and encourages lesser blurring in the output images [START_REF] Isola | Imageto-Image Translation with Conditional Adversarial Networks[END_REF] as was also confirmed during our experimentation.

L L 1 = n i =1 y i -x i 1 (3.11)

Structural similarity (SSIM) loss

Structural Similarity Index (SSIM) is one of the most robust and state-of-the-art metrics to measure image quality. The commonly used L2 loss, widely used for image restoration tasks, has been reported to cause blurriness and artefacts in the results [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF]. SSIM is one of the core perceptual metrics for realistic image generation, and it is also differentiable so that it can be back-propagated. The SSIM for pixel p is defined between 0 and 1, and the loss function for SSIM can be calculated as Eqn.

3.13.

L SSIM = SSIM(X, Y) (3.12) = -log 1 2|P| p∈P(x,y) (1 + SSIM(p)) (3.13)
where P(x, y)) denotes the pixel location set and |P| its cardinality.

Specular loss

The intuition behind this loss is to force the generator and discriminator to specifically compare the regions with specular highlights so that they are generated and Algorithm 2 SHMGAN algorithm overview. All experiments use m, n = 128, batch size of 1, ADAM optimiser with β 1 = 0.5, β 2 = 0.99, l r g en = 2e -6 , l r d i sc = 1e -6 , decaying every 10k steps with a base of 0.95.

Input: Four Polarimetric and one pseudo-diffuse RGB image Output: Specular-free RGB image 1: for k = 1 . . . epochs do 2:

x i ← Sample mini-batch i = 0, 45, 90, 135, ED 3:

χ r / f ,χ cl s f ← D(x i ) 4:
x i ← YCbCr colorspace, normalize to [0

CbCr av g = ( 5 i =1 x i CbCr )/5

6:

OH l abel = (0, 0, 0, 0, 1) :=1 m×n if l abel ∈ Target domain, 0 m×n if l abel ∉ Target domain.

7:

x i ← concatenate (x i , OH l abel ) 8:

I g en ← G(x i ) 9:
I g en ← concatenate (x i , CbCr av g ) 10:

χ r / f , χ cl s f ← D(I g en )

I c yc ← Randomly replace I g en in x i 12:

for q = 1 . . . 5 do

I c ycl i cRGB q ← concatenate (I c yc q , CbCr av g )

I c ycl i cRGB q ← G(I c ycl i cRGB q )

15:

χ r / f , χ cl s f ← D(I c ycl i cRGB q ) 16:
χ r / f , χ cl s f ← D(I or i g i nal q ) 17:

end for 18:

Calculate L t ot al = L g en + L d i sc 19: ω G , ω D ← Adam(L t ot al ) 20: ω G , ω D ← clip (ω G , ω D , -1, 1)
21: end for evaluated correctly, respectively. To promote this, we introduce a weighted L 2 norm of the masked regions of both the original and the generated images. Let M be the binary specular candidate (from the self-attention mechanism (section 2.4.1)), I c yc y be the cyclic Y-channel and I n be the original polarimetric images (where n = 0, 45, 90, 135, ED). Then the specular loss can be calculated by the L 2 norm of element-wise multiplication of the specular candidate and the input images, as given by eqn. 3.14.

L SpecLoss = λ 1 (M I c yc y ) -λ 2 (M I n ) 2 (3.14)

Style transfer loss

Style Transfer is a technique popularised by the advancement in GANs for stylized generation of images with the same "content" as a base image but the "style" of another image. Traditionally style transfer is implemented by taking the weights from multiple frozen layers of pre-trained VGG networks to learn and superimpose the style onto the target image.

Using the same analogy, we treat the Estimated Diffuse (ED) image as the specularfree 'style' that we want to superimpose on the input image 'content'. L st yl e = 1 4n 2 m 2 G st yl e -G cont ent 2 (3.17) where n, m are the image dimensions, ω 1 and ω 2 are the weights with values 1 and 100, respectively. The weights were selected after experimentation based on the proposed ratio ω 2 /ω 1 of 2 to 3 orders of magnitudes between them [START_REF] Gatys | A Neural Algorithm of Artistic Style[END_REF].

Discriminator loss

The discriminator loss L d i sc is defined as the sum of the individual losses of the two discriminator outputs D ad v and D cl s f . The D ad v loss is the adversarial loss of the discriminator, classifying the generated images as real or fake. It is calculated using the Least Square GAN (LSGAN) loss [START_REF] Mao | Least Squares Generative Adversarial Networks[END_REF] which has proven to cater to the vanishing gradient problem in adversarial loss and is beneficial for GAN convergence. Intuitively, LSGAN wants the target discriminator label for real images to be 1 and generated images to be 0. Furthermore, for the generator, it wants the target label for generated images to be 1 which can be calculated as follows:

min D L LSGAN (D) = 1 2 E x∼p data (x) (D(x) -1) 2 + 1 2 E z∼p z (z) (D(G(z))) 2 min G L LSGAN (G) = 1 2 E z∼p z (z) (D(G(z)) -1) 2 .
(3.18)

The classification loss D cl s f calculates the probability of each generated image belonging to a particular class by calculating the cross entropy between the generated image and the target labels and passes it through a softmax function as defined by eqn. 3.19, where χ x j represents the one-hot target labels.

D cl s f = 1 5 5 i =1 exp D (x i ) T χ (x i ) 5 i =1 exp f (D i ) T χ x j (3.19)
We also add the specular loss and style transfer loss from the generator for improving the generator performance.

Total SHMGAN loss

The total loss can be defined as the sum of the total generator and discriminator loss. These losses are calculated as follows:

L t ot al = L g en + L d i sc (3.20)

L d i sc = Σ(γ 1 L cl ass , γ 2 L SpecLoss , γ 3 L St yl eTx ) (3.21) L g en = Σ(λ 1 L c ycl i c , λ 2 L L 1 , λ 3 L SSIM , λ 4 L SpecLoss , λ 5 L St yl eTx ) (3.22)
Where γ 1 = 1, γ 2 = 10, γ 3 = 5, λ 1 = 1, λ 2 = 10, λ 3 = 5, λ 4 = λ 5 = 10 from experimentation.

A detailed flowchart of the developed network is shown in Fig. 3.8, and the proposed training method is detailed in Algorithm 2.

SHMGAN hyper-parameter selection and implementation

The network was implemented in Tensorflow 2 and trained on a single Nvidia RTX3070 GPU with 8GB memory for 140 epochs. The model was optimized using ADAM optimizer with β 1 = 0.5 and β 2 = 0.99. Individual learning rates for training the generator and discriminator were used as suggested by the Two Time-scale In-house dataset 330

Datasets used for evaluation

To qualitatively and quantitatively compare the generated images, datasets comprising real-world images, both with and without ground truth, were used. A detailed table of the dataset and the exact number of images used for training and testing are given in Table 3.1. The most extensive dataset available with pure diffuse ground truth images was recently made available by Wu et al. [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. The authors provide 12 polarimetric images per scene with 30 deg increments; however, we only selected four orthogonal angles for training, namely I 0,60,90,150 and the pseudodiffuse I ED image was calculated using these images. These polarimetric angles were selected to capture maximum specularity variation in the orthogonal images.

Data was also acquired in-house using monochrome and colour polarimetric cameras in various settings and lighting conditions, as described in section 3.4.1. The data acquired consists of 330 images captured with multiple light sources and were used for testing purposes.

The training was thus done on a total of 1295 images, including the datasets mentioned in Table 3.1. To qualitatively and quantitatively compare the generated images, datasets comprising real-world images, both with and without ground truth, were used. The WHU-Specular dataset [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] provides 4310 real-world image pairs containing specular reflections of various intensities along with manually labelled ground truth masks for each image. The training set consisted of 3017 images, out of which 10% randomly selected images were used for validation. The remaining 1293 images were used for testing the network. For testing, the SpecularityNet was also trained separately on the dataset provided by the authors and quantitative analysis was done by training the developed network on the same dataset. Other large datasets comprising real-world images with manually labelled specular pixels are Whu-Specular, and TRIIW datasets [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF][START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF]. No ground-truth diffuse images are provided in these datasets; therefore, they can only be used to qualitatively test the results of specular highlight mitigation on real-world images taken under random conditions. The results on all these datasets are compared qualitatively and quantitatively and presented in the following sections. We compare our developed network's results with classical and state-of-the-art data-driven specular reflection mitigation methods. Classical methods based on chromaticity [START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF], bilateral filtering [START_REF] Yang | A New Projection Space for Separation of Specular-Diffuse Reflection Components in Color Images[END_REF] were used. For deep learning-based comparison, SpecularityNet [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] was the most relevant to the developed method and target application. All networks were trained and tested on the same dataset and resolution for a fair comparison. All metrics were calculated in MATLAB 2021a. Note that While the developed network also takes inspiration from CollaGAN [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF], it cannot be used for direct comparison to SHMGAN results since CollaGAN is targeted at image imputation and requires multiple image inputs (all the domains) for generating the missing domain as opposed to the single-input single-output concept of our developed network.

Metrics used for evaluation

In order to evaluate the performance of any segmentation algorithm, several metrics have been developed over the years. While a qualitative review of Image segmentation gives a broad overview of the success or failure of a method, it is biased toward human perception and the ability to see fine details. Quantitative segmentation requires a ground truth label or mask image that exactly marks the pixels as falling into a category. Semantic segmentation requires multiple masks for a multiclass classification problem for specular segmentation, but for our particular case, a binary mask is enough to classify a pixel as specular or non-specular. With the ground truth available, the accuracy of the segmented images can be evaluated using several metrics. The most popular metrics used are explored below in brief so they can allow us to understand the results in a better way.

While several qualitative measures have been introduced over the years to measure

Summary

Specular highlight mitigation is a challenging problem with non-trivial solutions and affects real-world images and modern vision-based applications. Detecting and mitigating specular highlights using state-of-the-art deep learning networks have been quite promising and have shown significant improvement over the classical methods. In this chapter, three distinct methodologies were developed to achieve the goals of specular highlight detection and mitigation. Firstly, to detect specular pixels in a wide variety of real-world images independent of the number, colour, or type of illuminating source, we propose an efficient Specular Segmentation (SpecSeg) network based on the U-net architecture that is expeditious to train on nominal-sized datasets. The proposed network can detect pixels strongly affected by specular highlights with a high degree of precision, as shown by comparison with the state-of-the-art methods. We also proposed a fast Weighted Median

Inpainting method for replacing the affected pixels with the colour of the region that is approximated from the boundary pixels. The method is fast and quite effective in inpainting small regions that are comprised of a single colour. Lastly, for a more

Results and Discussions

"The important thing about a problem is not its solution but the strength we gain in finding the solution."

Seneca

Chapter abstract

This chapter culminates the results of the three methods proposed in this thesis. The chapter starts with accurate segmentation of specular highlights using the developed SpecSeg network (section 3.2) in real-world indoor and outdoor images, which are analyzed qualitatively and quantitatively by comparing to the modern state-of-theart competing methods. The segmented results are used for mitigating specular reflections in images using the proposed weighted-median inpainting method. We also show a fast diffuse colour inpainting method that utilises the detected regions from our developed SpecSeg network and inpaints the affected regions with an estimated diffuse colour inferred from the boundary regions, followed by the results of our de- The tota loss is a sum of Dice similarity coefficient (DSC) [START_REF] Milletari | V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation[END_REF] and Focal loss [START_REF] Lin | Focal Loss for Dense Object Detection[END_REF] as described in section 3.2 as the combination of both dice and focal losses with α = 0.25 and γ = 2.0 proved to be efficient and highly effective for binary segmentation problem of segmenting specular pixels. The dice loss maximizes the overlap be- Several datasets with specular masks are available publicly for testing, as detailed in the table 3.1, but for testing and comparison, two of the most recent datasets were used; namely Whu-Specular dataset [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF], and SHIQ dataset [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF]. The datasets were split into train, validation sets in 80%, 10% ratio respectively, whereas the initially provided Test sets with each dataset were used for testing. The qualitative and quantitative results are discussed in the following subsections in detail.

Qualitative results

The results of segmenting specular highlights using SpecSeg network on the Whu-Specular dataset [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] are shown in Figure 4.2 and on the SIHQ dataset [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] in Figure 4.3. The input image in the top row, followed by the manually generated mask of the specular highlights in the second row as given in the Whu-Specular dataset by Fu et al. [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF]. The last row is the predicted specular pixels from our SpecSeg network. Visually comparing with the manually annotated masks, we can see that the network can detect all specular regions and generate masks closely resembling the groundtruth images. The detection of specular regions is valid for various materials in the images, including plastic, wood, metallic and ceramic objects of irregular shape.

Even small specular regions in the images are detected quite accurately. Furthermore, the images are taken under natural lighting conditions and have an unknown number and orientation of light sources. This results in specular pixels of various intensities and colours depending on the illuminating source colour. Additionally, specular highlights on light-coloured surfaces are also detected accurately, which is often hard for most conventional algorithms. Note that the manually annotated masks result from human visual interpretation of specular pixels in an image and are therefore susceptible to misrepresentation, especially around the region borders. While the highly saturated pixels are easy to identify and mark, the distinction becomes significantly challenging and blurry around the edges of the specular region, where the falloff to diffuse colour can be soft enough such that some pixels may be wrongly marked as specular and vice versa. This is challenging in real-world images because there are multiple light sources in various orientations and of different strengths. As opposed to medical image masks, where there is a single illumination positioned nearly concentric with the camera for acquiring endoscopic and colonoscopic images. This results in very sharp specular boundaries that medical experts can mark, resulting in the masks being highly accurate, making the qualitative analysis easier and quantitative analysis more meaningful. Despite these shortcomings, the manually annotated masks provided are an excellent baseline for evaluating all qualitative and quantitative segmentation methods. Looking at a few

Quantitative results

The quantitative results of the testing done on the datasets are presented in Table 4.1. A statistical summary of the results achieved is also presented in Fig. 4.6. The quantitative comprising was done using three metrics: S-measure, mean F-measure (meanF), and MAE. Several segmentation methods were evaluated by Fu et al. [START_REF] Fu | Specular Highlight Removal for Real-world Images[END_REF] and have been directly included here from their works for a broader comparison. In However, to our knowledge, this work is the first to present an accurate specular highlight detection network that works on indoor as well as outdoor images with reasonably accurate results on the latter, despite no availability of any large outdoor specular dataset available to train the network.

Performance comparison

One of the most significant caveats of deep learning is the significantly staggeringly

Ablation studies

In

Mitigation of specular highlights using weightedmedian inpainting

Testing the weighted median inpainting method was also done on several datasets, as mentioned in table 3.1. Testing on a large set of real-world images helped prove the effectiveness of the simple method while also reinforcing the reasons for the inability of classical methods to deal with real-world images at large. The results and the limitations of the classical inpainting method are discussed in the following subsections.

Qualitative results

The developed method's qualitative comparison of specular and diffuse images is shown in Fig. 4.7. The top row is the input RGB image, whereas the bottom row is with the specular highlights inpainted using the colour information using the re-

Generating specular-free images using SHMGAN

The SHMGAN network was implemented using Python and Tensorflow 2.8, similar to SpecSeg network. However, Tensorflow's Functional API was used to develop the multi-input GAN network as it provides versatility and customization to develop any level of deep and complex networks.

The SHM network model has been discussed in depth in section 3. The model was optimized using ADAM optimizer with β 1 = 0.5 and β 2 = 0.99. Individual learning rates for training the generator and discriminator were used as suggested by the Two Time-scale Update Rule (TTUR) rule [START_REF] Heusel | GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium[END_REF] as it helps in convergence. The starting learning rates for generator and discriminator were 2e -6 and 1e -6 respectively, decaying every 10, 000 steps with a base of 0.95. Both discriminator and generator were trained simultaneously with a batch size of 1. The network kernels were initialized with mean µ = 0 and standard deviation σ = 0.02 as proposed by DCGAN [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF]. The images were resized to a resolution of 128×128, and all processing, such as resizing, conversion to YCbCr etc., were done at runtime. To improve training, the images were augmented by random flipping. A dropout of 20% was used before the discriminator's dense layers along with L2 regularization. Input randomization is implemented by substituting the labels of the cyclic images with zeros. Other techniques such as label smoothing, gradient clipping, adding noise to the inputs etc., were also utilized to improve the training results. The total loss can be defined as the sum of the total generator and discriminator loss and is defined by We compare our developed network's results with classical and state-of-the-art data-driven specular reflection mitigation methods. Classical methods based on chromaticity [START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF], bilateral filtering [START_REF] Yang | A New Projection Space for Separation of Specular-Diffuse Reflection Components in Color Images[END_REF] were used. For deep learning-based comparison, SpecularityNet [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] was used as being the most relevant to the developed method as well as the target application. All networks were trained and tested on the same dataset and resolution for a fair comparison, and all metrics were calculated in MATLAB 2021a. Note that While the developed network also takes inspiration from CollaGAN [START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF], it cannot be used for direct comparison to SHMGAN results since CollaGAN is targeted at image imputation and requires multiple image inputs (all the domains) for generating the missing domain as opposed to the single-input single-output concept of our developed network.

Qualitative results

As the developed network is developed as a multi-input CycleGAN, SHMGAN generates images across all input images in a cyclic fashion. While we are primarily interested in generating the specular-free images from the network, other polarimetric images are also generated, and the results can be seen in Fig. 4.10. The realistic images show variation in the illumination in all polarimetric domains, verifying that the network learns the variation similar to polarimetric images. The network attempts to recreate the sinusoidal variation in illumination across all domains, with the strength varying from image to image.

It is pertinent to mention that since there are no polarimetric constraints such as angle of polarization, degree of polarization, stokes parameters etc., enforced on image generation, the resulting images cannot be evaluated as true representatives of the polarimetric images. During the development of the network, experiments were conducted to quantify and relate the artificially synthesized images as possibly converging towards realistic polarimetric images, including experimenting with various physical constraints. However, generating polarimetric images is an ill-posed problem from a physics perspective. Thus, while the GAN produces visually convincing

Ablation studies

Several ablation studies were conducted to verify the developed SHMGAN network.

The network architecture and losses were selectively removed and replaced to verify the effect and significance of each sub-part of the developed network that achieves the best results.

• To validate the usage of multiple polarimetric images, a network was trained with two images only, the diffuse image and a single RGB image. As expected, the network was unable to mitigate specular reflections since it was unable to learn enough variation in illumination between images as shown in the Figure 4.17

Network losses

To verify the benefit of the individual loss function, an ablation study was done by varying the losses such as the SSIM loss, specular loss etc. as shown in the Figure 4.19

• The SSIM loss enables the network to generate more realistic results, closer to the diffuse ground truth.

• The specular loss enables the network to focus on the specular highlight regions and learn the variation of illumination between the polarimetric images and the estimated diffuse image.

• Overall, the proposed total loss results in visually appealing images after the After only 10 epochs, we can see that not only the images start to look washed out with reduced colour generation, the network also starts to generate dark blobs and regions. In the developed SHMGAN, the images are scaled to [0, 1] range and then standardized by using equation 4.1 in all the presented results.

Summary

In this chapter, we qualitatively and quantitatively analyzed the different methodologies developed in Chapter 3. We show that the proposed SpecSeg network is significantly fast to train with limited images and accurately detects specular reflections in real-world images with no restriction on illumination conditions for image acquisition. We achieve very low scores that are comparable to the state-ofthe-art methods. Furthermore, the proposed network's training time requirement and inference performance are significantly better than other competing networks trained and tested on comparable hardware and it is able to train in just 40 minutes.

Qualitatively, the segmented specular highlights are comparable with state-of-theart specular detection methods provided in the literature. We also show that the proposed network can detect specular highlights in outdoor images taken under extremely bright conditions, with good results. To our knowledge, no other prior work has presented a specular highlight detection network that works on indoor and outdoor images with reasonably accurate results on both conditions. Once the specular pixels are detected, we show that our proposed Weighted Median Inpainting method can fill in the estimate diffuse colour with reasonable accuracy in regions that are texture-less. But for a more robust solution, inpainting does not work as dynamically for complex images. To improve the robustness, we show that the developed SHMGAN network is able to remove specular highlights in images without any additional input. The network is trained to take advantage of the varying illumination information in polarimetric images and synthesises a specular free images.

No manual segmentation or marking is required for the specular pixels in the scene, and the network learns by a self-attention mechanism by utilizing the developed SpecSeg network, as described in section 3.2. SHMGAN outperforms state of the art approaches and is able to mitigate specular reflections on objects, independent of the material of the object or colour of the illuminating light sources. Extensive qualitative and quantitative testing is done on real-world images from in-house collected dataset as well as publicly available datasets to verify the results. The resulting images were realistic visually, with noticeably 9.33% higher signal to noise ratio, low artefacts and chromatic aberrations as compared to other state of the art methods.

Testing was also done on outdoor images captured under bright sunny conditions, something that has not been reported by any other work to our knowledge. SHMGAN generates a specular-free image from a single input RGB image with a mean improvement of 9.33% mean SSIM score over the stateof-the-art methods.

Both qualitative and quantitative comparisons and ablation studies of both Spec-Seg and SHMGAN networks on publicly available datasets were performed. We show that our methods perform well on real-world images and are able to generate images that mitigate the specular reflections in the affected region. In summary, we demonstrate in this thesis that polarimetric imaging can be highly beneficial to learning specular variation, and the illumination variation can be learnt by generative adversarial networks and regenerated without any additional input.

Application pipeline of SpecSeg and SHMGAN

Specular highlight mitigation is a challenging problem with non-trivial solutions and affects real-world images and modern vision-based applications. Developing successful mitigation techniques means that they must be usable in any image processing pipeline without causing any significant disruption or major rework. Keeping this in mind, the developed networks in this thesis can be integrated easily into any standard image or video processing method, as shown in the Figure 5.1. An image can be parsed through SpecSeg network, which would be able to detect the presence of specularity in an image due to the significantly fast inference time, as already proven in table 4.2. The image or video frame can be ignored if there is no specularity. Alternatively, if the image contains specularity, it can be passed on to SHMGAN, which uses the specular generated by the specular mask to generate the same image without specularity. This generated image can then be fed back to the regular pipeline for further processing as the target application requires. As the inference time for detection and generation is relatively less time-consuming, there should be a minimal impact on the processing performance of the pipeline while improving the results due to the removal of specular reflection from the images.

Limitations and Future Work

During this thesis, several limitations were identified that directly affected the research for specular highlight mitigation.

Firstly, the proposed SpecSeg network achieves reliable and accurate with an order of magnitude improvement in training time with the state-of-the-art methods, in the the quantitative results it was unable to outperform the method proposed by Fu et al. [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] in terms of detection accuracy and mean error scores. While the resulting metrics are very close, there is still room for fine-tuning and improvement and we are quite confident that SpecSeg can be improved to outperform all competing networks with relative ease. For generation of polarimetric images, the developed SHMGAN generates all polarimetric angles for the input images; however, no physical polarimetric constraints are provided to ensure the physical plausibility of the generated images to be real polarimetric images. Previous Efforts have been made to use polarimetric admissibility conditions to try to generate polarimetric images from a single input image [START_REF] Blin | A New Multimodal RGB and Polarimetric Image Dataset for Road Scenes Analysis[END_REF][START_REF] Blin | How Polarimetry May Contribute to Deep Road Scene Analysis in Adverse Weather Conditions[END_REF] especially for adverse weather conditions, however there is currently no feasible way to verify the polarimetric properties of the generated images with real-world polarimetric images. This was also corroborated during our research.

However, this is an exciting and open problem and can significantly impact future understanding of polarimetric image generation and validation methods. The proposed pipeline for implementation of polarimetric images is feasible to implement; however, the implementation was out of the scope of this research thesis as the focus of this research thesis was only on the accurate detection and removal of specular reflections in images. Real-time implementation requires significant optimization and is dependent on several hardware and software constraints and parameters and is very likely to be explored in the future. Especially the implementation to outdoor images is an area that requires further exploration as outdoor images taken on a bright sunny day produce significantly larger specular regions due to strong direct and inter-reflections between objects. This can prove to be a vital area of research for robotics as well as ADAS systems. In their original paper [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF], the authors claim that BN reduces the internal covariate shift of the network and has a regularisation effect because it is computed over mini-batches and not the entire data set.

Part IV Appendix

Data augmentation

For training a deep learning network, it is always beneficial to have large amounts of data with enough variation so that it can be generalised to the problem. This enables the neural network to learn the important features and characteristics of the data without over or under fitting. However, acquiring more data is not always possible or feasible due to many reasons. An alternative to acquiring more data is making the existing dataset larger with simple transformations to the existing data.

If, for example, an entire image is shifted left by a few pixels, the change is imperceptible to a network that only sees intensity values and numbers. This shift can be fairly significant as the classification or label of the image does not change while VIII

2. 1 105 4. 1 116 4. 2 118 4. 3 131 xviiiof Figures 1 . 1 51 2. 8 77 3. 2 83 3. 3 85 3. 4 87 3. 5 95 3. 7 96 3. 8 4 . 9

 11051116211831311151877283385487595796849 figuration to capture polarimetric images in each colour. . . . . . . . . 52 2.9 A general end-to-end flow for developing a deep-learning-based solution from dataset to the required output. . . . . . . . . . . . . . . . . . . 59 2.10 A generic generative adversarial network (GAN) with a single generator-discriminator pair. . . . . . . . . . . . . . . . . . . . . . . . . . 62 2.11 The CycleGAN architecture as proposed by Zhu et al [104] uses a two generators with feedback from two separate discriminators to train them in a cyclic fashion. Each additional domain requires a separate generator-discriminator pair. . . . . . . . . . . . . . . . . . . . . . . . . . 63 2.12 StarGAN architecture as proposed by Choi et al. [127] trains a single generator-discriminator pair, replacing the requirement of a separate pair per domain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 2.13 CollaGAN by Lee et al. [117] improves StarGAN for image imputation of a missing domain among multiple inputs. . . . . . . . . . . . . . . . . 65 2.14 Set of classical images used for specular highlight mitigation in literature 68

4. 14 139 A. 1 1 1 . 2 1 . 3 8

 141391112138 Results of detected specularity by self-attention mechanism and diffuse images by SHMGAN, on the TRIW dataset. . . . . . . . . . . . . . . 4.15 Results of detected specularity by self-attention mechanism and diffuse images by SHMGAN, on the SHIQ dataset. . . . . . . . . . . . . . . 4.16 Summary of quantitative results comparing the spread of results of the developed network with SpecularityNet. in the (a) PSD dataset and (b) in-house dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.17 Results of ablation study after 70 epochs of training with only two input images (RGB and I ED ) instead of the developed five images show that even after extended training, the network generates images with artefacts and is unable to remove specular reflections effectively. . . . . 4.18 Results of ablation study after 70 epochs of training without selfattention show that the specular reflections are not fully mitigated, and the images generated have distorted colours. . . . . . . . . . . . . . xxii LIST OF FIGURES 4.19 Results of ablation study after 70 epochs of training with various loss combinations. (a) For ablation results without specular loss (b) ablation results without SSIM loss. . . . . . . . . . . . . . . . . . . . . . . . . 133 4.20 Ablation study (after 50 epochs) of clipping gradients before backpropagating weights. Clipping the gradients to [0, 1] resulting in exponentially increase in the generator loss and produced poor resulting images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4.21 Ablation study of benefits of image standardization on GAN generation. Non standardizing images results in loss of colour generation and large blobs only after ten epochs. . . . . . . . . . . . . . . . . . . . . 135 5.1 Pipeline implementing specular highlight detection and mitigation networks. Due to their fast inference times, developed networks Spec-Seg and SHMGAN can easily be integrated into existing pipelines for specular highlight removal. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure depicting a generic deep convolutional network configuration as pieces of LEGO ® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . II A.2 Figure depicting a generic kernel filter in a convolutional neural network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III A.3 Figure depicting upscaling or encoding using a generic transposed convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . III A.4 A generic residual connection layout where the input features are added to the output layer before passing to the succeeding layers. . . . IV A.5 Visualising dropout between CNN layers. Red nodes represent dropped nodes that are randomly selected at every pass during the training period, and all connections are severed for that training pass. . VI A.6 Popular activation layers used in CNNs. . . . . . . . . . . . . . . . . . . . VI A.7 Example of various non-destructive and destructive transformations for data augmentation that can be used for increasing the dataset size without causing the network to overfitting trained weights. . . . . . . . VIII Digital imaging and specular reflections . . . . . . . . . . . . . . . 1.1.1 Specular and diffuse reflection components . . . . . . . . . . 1.1.2 Polarization and Specular Reflection . . . . . . . . . . . . . . Research Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Problem statements . . . . . . . . . . . . . . . . . . . . . . . . Thesis organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 1 . 1 :

 11 Figure 1.1: According to the law of reflection, the angle of incidence θ i and angle of reflection θ r are symmetric about the surface normal n.

Figure 1 . 2 :

 12 Figure 1.2: Figure depicting diffuse and specular reflection components from incident light sources.

  ponents. According to Fresnel's theory, this specularly reflected light wave can be written as a combination of constant diffuse component I d and a varying specular component I s , which is a sum of a constant component I sc and a cosine term I sv that varies with the difference of polarizer orientation ϕ pol and the Angle of Polarization1 It should be noted that in literature, the terms 'specular reflection' and 'specular highlight' have been used interchangeably by several authors. Following the same convention, throughout this text, both these terms are also used interchangeably, and unless otherwise specified, it means the strong reflections in any scene, represented by near-saturation pixels in the image.

Figure 1 . 3 :

 13 Figure 1.3: Variation in specularity with the variation of polarisation angle (orange areas) in uncontrolled environments. Note that unpolarised light causes specular reflection regardless of polarisation filter angle (blue areas)

1. 4 1 :

 41 and the organisation and contents of each chapter are detailed below: Chapter Chapter 1 setup the introduction to the problem being catered in the thesis, i.e specular highlights. The basic physical model for defining highlights,are explained, establishing the causes of the highlight formation in imaging. The research motivation and research questions to be answered are laid down along with the research objectives and the proposed hypothesis.

Figure 1 . 4 : 3 : 4 : 5 :"

 14345 Figure 1.4: A flowchart of the thesis organization.

  photography, graphics and cutting-edge applications such as Virtual Reality (VR) and gaming engines. The model assumes that a surface is composed of a distribution of randomly oriented, mirror-like micro-facets, as shown in Figure2.1. The perceived specular reflection from any surface is the resultant sum of reflection of the incident light rays from these mirror-like surfaces. Cook et al.[START_REF] Cook | A Reflectance Model for Computer Graphics[END_REF] proposed a distribution function of the reflected light alongside changes in the chromaticity as the incident angle of light changes. The distribution function is called the Bi-directional Reflectance Distribution Function (BRDF) given by equation 2.1.

Figure 2 . 1 :

 21 Figure 2.1: A visual representation of mirofacets as proposed by the Torrence-Sparrow model. The microfacets are probabilistic in nature and cause light rays to reflect at random directions.

Figure 2 . 2 :

 22 Figure 2.2: The Dichromatic Reflection Model (DRM) represents specularly reflected light components about the surface normal n at the same angle as the incident light rays.

Figure 2 . 3 :

 23 Figure 2.3: Variation of specular and diffuse reflections as surface roughness varies from perfect specular to perfect Lambertian.

Figure 2 . 4 :

 24 Figure 2.4: Shading and reflectance intrinsic image samples from the MIT Intrinsic image dataset [14].

Figure 2 . 5 :

 25 Figure 2.5: Real-world examples of specular reflection in indoor and outdoor images, caused by ambient and multiple point light sources.

6 )I

 6 min (p) = min c∈{r,g ,b} I c (p) (2.7) the PSF image I is simply the original image I minus the original image entry-wise, i.e. Ĩ(p) = I(p) -I min (p) (2.8) Shen et al. [24] modified the PSF image by Yoon et al. to make its chromaticity robust to noise by adding an offset factor and solving the DRM equation as a leastsquare problem for mixed specular-diffuse regions. The pure diffuse regions have the least distance in chromaticity coordinates from their solution. Later, Shen and

  one of the first to use polarization and colour information simultaneously to separate the diffuse and specular reflection components by capturing at least six images captured at different polarizer angles. They use polarization to acquire independent local estimates of the colour of the specular component, forcing each image pixel to lie in a linear colour subspace and then thresholding it to achieve the desired separation. Kim et al.[START_REF] Dae | Variational Specular Separation Using Color and Polarization[END_REF] extended Nayar et al.'s work by dividing the colour space into a specular line space and a diffuse plane space. The diffuse pixels are selected by thresholding the intensity variation while rotating the polariser. The spatial variation in the specular components is then smoothed out using an energy function. Umeyama et al.[START_REF] Umeyama | Separation of Diffuse and Specular Components of Surface Reflection by Use of Polarization and Statistical Analysis of Images[END_REF] applied Independent Component Analysis (ICA) to images captured through a rotating polarizer to separate the diffuse and specular components. More recently, Wen et al.[START_REF] Wen | Polarization Guided Specular Reflection Separation[END_REF] proposed a polarisation-guided model that can be used to cluster pixels with similar diffuse colours. They formulated the problem in an optimized global energy minimization function, resulting in specular reflection separation in images. As seen in the above literature review, the development of classical methods that use polarization for specular reflection segmentation has reduced significantly, favouring the state-of-the-art deep learning processes in vogue. Deep learning methods have been proven to generate significantly improved results using the additional information provided by polarisation imaging. A more in-depth overview of such methods is provided in section 2.3.4.

  treated separation of specular reflection as a blind source separation problem from polarization images. Using Singular Value Decomposition (SVD), they are able to separate the two components using three images captured at different polarizer angles. Akashi and Okatani[START_REF] Akashi | Separation of Reflection Components by Sparse Non-Negative Matrix Factorization[END_REF] introduced a framework that incorporated Non-Negative Matrix Factorization (NNMF) with a sparsity constraint that limited the number of colours used to compose the image, taking advantage of the fact that natural images have a limited number and composition of colours. One of the bases of the factorization was the illuminant itself, and a cost function was formulated to penalize the use of illuminant colour. Bochko et al.[START_REF] Bochko | Highlight Analysis Using a Mixture Model of Probabilistic PCA[END_REF] used Probabilistic Principal Component Analysis (PCA) to cluster the data into the highlight and body-reflection computing the covariance matrices eigenvectors of the clusters.

  ing specular reflections is to capture multiple images from different angles either by taking multiple images or using light field cameras specializing in taking multifocal but spatially coherent images. Lee and Bajcsy et al.[START_REF] Wook | Detection of Specularity Using Color and Multiple Views[END_REF] .captured spectral scene radiance from different views and proposed a spectral differencing algorithm to compute a minimum spectral distance that represents specular reflections above a threshold. Wang et al.[START_REF] Wang | Light Field Imaging Based Accurate Image Specular Highlight Removal[END_REF] used the state of the art light field cameras to capture multi-focal light field images and apply depth estimation to cluster the specular pixels into saturated and unsaturated and do a colour variance analysis to recover diffuse colour information. Islam et al.[START_REF] Islam | Specular Reflection Detection and Inpainting in Transparent Object through MSPLFI[END_REF] also used a Multi-spectral Polarimetric Light Field Imagery (MSPLFI) setup to segment out specular components of a transparent object. They showed that polarimetry combined with the multispectral aspect added to light field cameras effectively separates the specular reflection part quite reliably. However, the requirement of multi-spectral expensive light field cameras is a limitation for general-purpose imaging. Zhouyu et al.[START_REF] Fu | Specular Free Spectral Imaging Using Orthogonal Subspace Projection[END_REF] used orthogonal subspace projection representation for removing specular reelections in hyperspectral images, based on the DRM that was valid for single and multi-colour illuminants.With time, there has been an immense interest in developing deep learning-based techniques for segmenting specular highlights in various applications. Two areas stand out in particular for this task, medical imaging and real-world imaging. Specular highlight segmentation in medical imaging is especially critical, as all invasive and non-invasive medical imaging procedures are generally done with a single camera and a concentric light source that is attached to the camera. Procedures such as endoscopy and colonoscopy are examples of such procedures. Detection of accurate specular highlights is especially critical in medical imaging, where such procedures are affected by extreme specular reflections from the singular light source with the camera and can result in incorrect identification of regions of interest. Arnold et al.[START_REF] Arnold | Automatic Segmentation and Inpainting of Specular Highlights for Endoscopic Imaging[END_REF] proposed a segmentation method based on non-linear filtering and colour image thresholding of endoscopic images and then inpainting to fill in the damaged areas. Alsaleg et al.[START_REF] Samar | Adaptive Segmentation and Mask-Specific Sobolev Inpainting of Specular Highlights for Endoscopic Images[END_REF] used a colour-adaptive threshold and a gradient-based edge detector to detect the specular regions in endoscopic images and then inpainting them. As can be seen, there has been a significant amount of research over the years, and multiple ways and techniques have been attempted to segment out the damaged pixels in an image. A summary of the classical methods for specular highlight segmentation is given in Table

  lar and diffuse components of the DRM model. Bajscy et al. [20] presented a linear basis model based on DRM with an orthogonal HSL colour space, where weighting factors for surface reflectance lead to the identification of specular and diffuse components based on the difference of the saturation values. Mallick et al. proposed using transformations in the SUV colour space [66, 67] and using Spatio-temporal information, proposed a partial differential equation that iteratively erodes the specular component at each pixel to find the maximum diffuse chromaticity. Yang et al.

Figure 2 . 6 :

 26 Figure 2.6: Polarised nature of specular reflection after passing through a polarizer filter causes it to oscillate in a sinusoidal pattern as a function of the polarizer angle ϕ pol .

  24) Where I 0 and I 90 represent the light intensities polarised along the horizontal and vertical axis x and y respectively, I 45 and I 135 represent the light intensities polarised at ±45 • , and I l and I r represent the light intensities polarised in a left circular and right circular state. The equation 2.24 allows estimation of stokes parameters from intensity images taken from 4 linear polariser filters, oriented at a difference of 45 •

  27 and in terms of stokes parameters by equation 2.28, where 0 ≤ ρ ≤ 1. ρ = I max -I min I max + I min (2.27) ρ l i near = S 2 1 + S 2 2 /S 0 (2.28) When the polarisation planes of the lens and the light source are in parallel with each other, this combination results in the brightest possible image with the elimination of the non-polarised ambient light. The DoP can be calculated for each image pixel, thus forming an image called the DoP image. The Angle of Polarisation (AoP) ϕ pol is the direction of the reflected light's polarisation. The AoP is determined by the angle of an unpolarised incident light, the surface orientation of the object and the material of the object. This causes the AoP to vary randomly from pixel to pixel for rough surfaces due to the existence of microfacets. For a smooth surface, however, the AoP changes smoothly and continuously [91]. The AoP can be extracted from each image pixel, thus forming an image called the AoP image. AoP can be calculated as equation 2.29 and a comparison of AoP, DoP and the stokes parameters is given in Figure 2.7.

Figure 2 . 7 :

 27 Figure 2.7: Comparison of all polarimetric angles I 0 , I 45 , I 90 , I 135 and calculated parameters such as AoP, DoP and linear stokes parameters S0, S1 and S2. Notice that the DoP can be interpreted easily indicating the highly polarized areas as the brightest in the image however AoP is more difficult to physically interpret without any reference object with known AoP.

Figure 2 . 8 :

 28 Figure 2.8: Modern on-sensor polarimetric filters are able to capture 4 polarimetric images that are spatially and temporally coherent in both greyscale and RGB colourspace (depending on the sensor configuration). The raw images can then be demosaiced to recover four separate polarimetric angle images. The colour Bayer sensors use a super-pixel configuration to capture polarimetric images in each colour.

  tion of specular reflections by a single layer. Almost a decade later, a lot of attention towards application of deep learning towards specular highlight segmentation as already discussed in depth in sections 2.2.2 and table 2.3. Similarly, the benefits of deep learning in image generation and image-to-image translation were being explored with promising results in applications such as image inpainting[START_REF] Yeh | Semantic Image Inpainting with Deep Generative Models[END_REF].Georgoulis et al.[4] introduced a deep convolutional neural network that directly predicts a reflectance map from the input image itself using supervised learning.The usage and power of deep CNN networks have really exploded since the proposal of GAN by Goodfellow et al.[START_REF] Goodfellow | Generative Adversarial Networks[END_REF], and CycleGAN[START_REF] Zhu | Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[END_REF], image synthesis and image to image translation have gained massive popularity among researchers for various applications. One of the initial usages of GANs for specular highlight mitigation was by Funke et al.[START_REF] Funke | Generative Adversarial Networks for Specular Highlight Removal in Endoscopic Images[END_REF] using a CycleGAN for generating specular free endoscopic images using manually labelled data for training the network. Lin et al.[105] developed a GAN network and trained it on a synthetic dataset of 3D models to estimate diffuse components on real and synthetic test images. Their discriminator was trained as a multi-class discriminator instead of a binary one. The authors proposed this to help the discriminator pinpoint the desired manifold with a multiclass classification layer. Xu et al.[START_REF] Shijian | CDFF-Net: Cumulative Dense Feature Fusion for Single Image Specular Highlight Removal[END_REF] proposed their CDFF-Net and also trained it on a synthetic dataset for specular highlight removal. They proposed a cumulative dense feature connection between each downsampling layer of a pre-trained VGG-al.[START_REF] Hirpa Madessa | A Deep Learning Approach for Specular Highlight Removal from Transmissive Materials[END_REF] proposed a deep learning-based inpainting method to inpaint an automatically generated semantic mask on the specular pixels with a partial weighted convolution operation. They generated their mask using the classical Otsu's binarization method with a global image thresholding technique. They also explained that during their experiments, a regional convolutional Mask R-CNN network by He et al.[START_REF] He | Mask R-CNN[END_REF] was unable to generate accurate specular masks and failed to detect small specular regions. Recently, Fu et al.[START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] released their specular highlight detection network (SHDNet) comprised of multi-scale context contrasted features to detect specular highlights from real-world images. They also released two large datasets that have aided research for specular highlight detection greatly. The first dataset, titled the "WHU-Specular dataset", consists of over 4000 images with manually annotated specular masks for each image. They also released a second smaller dataset called the WHU-TRIIW dataset consisting of 500 real-world images containing specular regions of varying size and strengths. The dataset, however, does not contain any specular masks for the specular regions in the images. The Whu-specular dataset especially has been key in training and testing our developed SpecSeg network as detailed in sections 3.2 and 4.2. Muhammad et al.[START_REF] Muhammad | Spec-Net and Spec-CGAN: Deep XXIX BIBLIOGRAPHY Learning Models for Specularity Removal from Faces[END_REF] developed two networks, Spec-Net and Spec-CGAN, for specularity removal from monochrome and RGB face images, respectively. They also introduced the Spec-Face dataset containing 2805 real-world face images from 187 3D models. Wu et al.[START_REF] Wu | Deep Specular Highlight Removal for Single Real-World Image[END_REF] trained their specular highlight removal GAN network on 600 captured images with encouraging results. Fu et al.[START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] later proposed a multi-task network for JSHDR alongside a dataset named SIHQ. The SHIQ dataset consists of 10,825 images cropped from MIT's Multi-Illumination Images in the Wild (MIW) dataset[START_REF] Murmann | A Dataset of Multi-Illumination Images in the Wild[END_REF]. Each image is then processed to generate a highlight-free image, a specular illumination image and a highlight binary mask. Hou et al.[START_REF] Hou | Text-Aware Single Image Specular Highlight Removal[END_REF] introduced an application-oriented network to improve the accuracy of text detection by removing highlights from images with text. They proposed a two-stage framework with highlight detection and removal implemented as separate sub-networks called Net D and Net R respectively.

  moval in colonoscopic images by training their GAN network with specular masks generated from thresholding the maximum intensity in the images. Another medical imagining-oriented solution was proposed by Monkam et al.[START_REF] Monkam | EasySpec: Automatic Specular Reflection Detection and Suppression from Endoscopic Images[END_REF] whose twostage network called EasySpec inpaints the specular regions in endoscopic images. Specular reflection datasets have been very sparse in terms of the number of realworld images until recently. Another large dataset for rectifying this situation was proposed by Wu et al.[START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] with their Paired Specular-Diffuse (PSD) dataset. The PSD dataset is the largest polarization image dataset to date, consisting of 13,380 images. The PSD dataset consists of manually acquired a set of 12 polarimetric images per scene of real-world objects taken at a polarizer angle of ϕ pol = 30

Figure 2 . 9 :

 29 Figure 2.9: A general end-to-end flow for developing a deep-learning-based solution from dataset to the required output.

2 . 4 . 1 E

 241 clature and concepts related to deep learning basics and the methods developed in particular for image segmentation. Appendix A covers some core deep learning concepts relevant to the developed method. While deep learning has grasped the attention of academia, the public and the industry, it is not the first successful form of machine learning. However, it is safe to say that many of the machine learning algorithms used in the industry today are adapting to deep neural network-based algorithms. For a long time, the missing piece for this adaptation was an efficient way to train large neural networks. This changed in the mid-1980s, when multiple people independently rediscovered the Backpropagation algorithm using gradientdescent optimisation and started applying it to neural networks. The first successful practical application of neural nets came in 1989 from Bell Labs when Yann Le-Cun et al. introduced LeNet[START_REF] Lecun | Backpropagation Applied to Handwritten Zip Code Recognition[END_REF] and combined convolutional neural networks and backpropagation and applied them to classify handwritten digits. In 2011, Dan Ciresan from Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA)[START_REF] Cireşan | Multi-Column Deep Neural Networks for Image Classification[END_REF] proposed DanNet, the first pure deep convolutional neural network (CNN) to win a computer vision contest and became the first practical precursor of modern deep learning. The "deep" in "deep learning" is not a reference to any kind of deeper un-derstanding achieved by the approach. Instead, it stands for this idea of successive layers of representations that contribute to a model of the data. The first practical success of modern deep learning came in 2012 with the entry of Geoffrey Hinton's group[START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] in the yearly large-scale image-classification challenge ILSVRC (Ima-geNet Large Scale Visual Recognition Challenge), who achieved a top-five accuracy of 83.6% which was a significant breakthrough at that time. Since 2012, deep CNN have become the go-to algorithm that is used almost universally for all computer vision tasks, natural language processing and other applications. The fundamental difference between a densely connected layer and a convolution layer is that dense layers learn global patterns in their input feature space, whereas convolution layers learn local patterns in the feature space[START_REF] Chollet | Deep Learning with Python[END_REF]. These learned patterns are translation invariant and are spatially hierarchical. Arguably, the two main drivers for this have been the large-scale availability of processing hardware (i.e. GPUs) and digital data, which has led to significant investments in further development. A general process of a deep learning process is visualised in the flowchart 2.9. We first establish the basics of CNN and then present the intuition behind them to solve more complex problems. Relevant concepts and explanations of parameters for CNNs such as filter size, stride, batch size, padding and poling are presented first, followed by details on the types of activation layers used. These basic notions are essential to understanding the functioning of image segmentation algorithms. The theory behind all these concepts supports the different experiments carried out for specular highlight segmentation (Chapter 2) and mitigation (Chapter 3). Generative Adversarial Networks (GANs) Deep learning-based image processing really came under the spotlight after the development of methods that fall under the image-to-image translation category. In image-to-image translation, networks are fed input images and, based on the learned weights, transform the image after the application of filters. While image processing filters have been around for decades, deep-learning-based image-toimage translation filters' robustness and quality were unmatched by any prior solution and ushered in a new era in image transformation techniques. One of the methods that led to massive popularity among researchers for various applications was the proposal of GAN by Goodfellow et al. [103]. GANs are algorithmic architectures that use two neural networks, pitting one against the other (thus the "adversar-ial") in order to generate new, synthetic instances of data that can pass for real data. They are used widely in image generation, video generation and voice generation. Both generator and discriminator work in tandem and learn to mimic the distribution of the training dataset. GANs are part of image synthesis or image-to-image translation networks. GANs are unsupervised learning algorithms that use a supervised loss as part of the training. The latter appears to be where you are getting hung up. When we talk about supervised learning, we are usually talking about learning to predict a label associated with the data. The goal is for the model to generalise to new data. In the GAN case, we do not have either of these components. The data comes in with no labels, and we are not trying to generalize any kind of prediction to new data. The goal is for the GAN to model what the data looks like (i.e., density estimation), and be able to generate new examples of what it has learned. The GAN sets up a supervised learning problem to do unsupervised learning, generates fake/randomlooking data, and determines if a sample is generated fake or real data. This is a supervised component, yes. However, it is not the goal of the GAN, and the labels are trivial. The idea of using a supervised component for an unsupervised task is not particularly new. Random Forests have done this for a long time for outlier detection (also trained on random data vs real data), and the One-Class SVM for outlier detection is technically trained in a supervised fashion with the original data being the real class and a single point at the origin of the space (i.e., the zero vector) treated as the outlier class. The significant difference of GANs with traditional deep learning networks lies in the requirement of a cost function. While traditional cost functions need to be carefully designed for best results, GANs learn the latent distribution of the training dataset on their own, based on the min-max game of the generator and discriminator guided by an objective function(s). Mathematically, both generator and discriminator are engaged in a min-max game over an objective function their losses, respectively, in order to achieve their target results as described by equation 2.30, where D(x) is the discriminator, G(x) is the discriminator, and E x denotes the expectation with respect to the distribution x. x∼µ [log D(x)] + E z∼γ [log(1 -D(G(z)))](2.30)

Figure 2 .

 2 Figure 2.10: A generic generative adversarial network (GAN) with a single generatordiscriminator pair.

Figure 2 . 11 :

 211 Figure 2.11: The CycleGAN architecture as proposed by Zhu et al [104] uses a two generators with feedback from two separate discriminators to train them in a cyclic fashion. Each additional domain requires a separate generator-discriminator pair.

Figure 2 . 12 :

 212 Figure 2.12: StarGAN architecture as proposed by Choi et al. [127] trains a single generatordiscriminator pair, replacing the requirement of a separate pair per domain.

Figure 2 . 13 :

 213 Figure 2.13: CollaGAN by Lee et al. [117] improves StarGAN for image imputation of a missing domain among multiple inputs.

Contents 3 . 1 81 3. 3 85 3. 3 . 1 104 3. 5 105 Figure 3 . 1 :

 318138531104510531 Figure 3.1: A flowchart of the methodology showing the three developed namely, WMI inpainting method, SpecSeg and SHMGAN networks.

Figure 3 . 2 :

 32 Figure 3.2: SpecSeg configuration based on the U-Net architechture

Figure 3 . 3 :

 33 Figure 3.3: YCbCr colour space transformation from RGB colour space.

  3. YCbCr wasdeveloped as a practical approximation to colour processing and perceptual uniformity. YCbCr is used to separate out a luma signal (Y) and two chroma components (CB and CR). Luma is the weighted sum of RGB components of a colour image after gamma correction. If the weighted sum is only of the non-gamma-corrected RGB values, then it is called 'relative luminance'.

Figure 3 . 4 :

 34 Figure 3.4: Flowchart explaining the weighted median inpainting method.

  algorithm output is a single channel luma image with receded intensity values in specular regions of the image. This reduction automatically reduces the specular pixel intensities in the image. After processing the MSF using the Y channel, the chroma channels Cb and Cr from the original image are combined with the resulting MSF image to get the final diffuse image. This combined image is the diffuse component without specular highlights, and the specular image can then be separated by taking the difference of this concatenated diffuse image from the original image in YCbCr space. For a pixel p, the DRM model defined by equation 2.2 in

Algorithm 1

 1 Weighted median inpainting psuedo-code. Input: An RGB image I of size m, n and pixel p Output: Specular and Diffuse images in RGB colour space 1: read I ← image 2: η = 0.5 3: I YCbCr ← Y, Cb, Cr 4: MSF ← I Y pmi n(I p ) * β s where β s = (I Ymi n -η). * (I Ymi n > η) 5: I d (YCbCr ) ← conc at (MSF, Cb, Cr ) 6: I specul ar ← SpecSeg (r g b2g r e y(I)) 7: for num r eg i ons do 8:

Figure 3 . 6 :

 36 Figure 3.6: The developed SHMGAN generator network consists of 4 decoder-encoder blocks with skip connections, and outputs a 128×128×1 greyscale image.

Figure 3 . 7 :

 37 Figure 3.7: The developed SHMGAN discriminator network consists of four blocks with selfattention layer between third and fourth blocks. Outputs of the discriminator are real/fake probability and predicted class of the image.

  the discriminator. The five Y-channels of the input images I 0,45,90,135,ED are concatenated along the channel dimension to form a 5D tensor. The images are then one-hot encoded along the channel dimension as binary matrices with the exact dimensions as the input image for a tensor of dimensions (b, n, m, 10) where b is the mini-batch size, n, m are the image dimensions and the channel dimension is 10.

  The content loss is the intermediate and high-level feature representation of the input image calculated as the L2 loss of the input image I i nput to the generated cyclic diffuse image I c ycED . The style loss for SHMGAN is calculated by a weighted L2 loss of the Gram matrices of the style G st yl e and content G cont ent images. The Gram matrices provide the cross-correlation between vectorized style and content images. The total style loss L St yl eTx is calculated as a sum of the individual style. L st yl e and content L cont ent losses. L St yl eTx = ω 1 L st yl e + ω 2 L cont ent (3.15) L cont ent = I i nput -I c ycED 2 (3.16)

Figure 3 . 8 :

 38 Figure 3.8: Flowchart explaining the working of SHMGAN. All original and generated polarimtric images are passed through the discriminator in the forward and cyclic pass, but the discriminator weights are only updated using real images.

  robust specular highlight mitigation, we developed a deep generative adversarial network called SHMGAN with a dynamically generated self-attention mechanism to remove specular highlights in images. The network is trained to take advantage of the varying illumination information in polarimetric images and synthesises a specular free image from a single image input. No manual segmentation or marking is required for the specular pixels in the scene. The network is composed of a single generator-discriminator pair, eliminating the need for a separate network pair per polarimetric angle. As we will show in the following chapter, both Spec-Seg and SHMGAN networks outperform state-of-the-art approaches and are able to detect and mitigate specular reflections in scenes, independent of the material of the object or the colour of the illuminating light sources. Extensive qualitative and quantitative testing done on real-world images from inhouse collected dataset as well as publicly available datasets verify the results.

1

 1 veloped multi-domain SHMGAN adversarial network (section 3.4). The qualitative and quantitative results are complimented with an extended ablation study and discussed in depth. The results show that SHMGAN can successfully learn the variation of illumination from polarimetric images and apply the learned weights to mitigate Network implementation and training The network was implemented using Python language and Tensorflow 2.8, a popular, free, open-source software library for machine learning and deep learning library developed by Google. Tensorflow's Sequential API was used to develop the U-net based architecture of SpecSeg network as it provided easy and highperformance execution of the relatively more straightforward network. The SpecSeg network model has been discussed in depth in section 3.2. A brief summary of the network is presented here for context, along with reasons for selecting the specific hyperparameters for training. Definitions and benefits of the parameters used are discussed in depth in section A.1.3. SpecSeg uses the U-net architecture to allow the propagation of context information to higher resolution layers. It comprises of 5 encoders and 4 decoder blocks, with each path from the encoder passed to the decoder via a skip connection. Each encoder block consists of two 2D convolutional layers with filters (k) = 3 and stride (s) = 3 with ReLU activation and same padding. To improve the robustness of the learned layers, an incremental dropout of 10, 20 and 30% respectively is also introduced between the two convolutional layers of the first, third and fifth encoder block for regularization. This is followed by a batch normalization layer and a MaxPooling layer to downscale the layer for the subsequent layers. The decoder block upscales the layers using 2D transpose convolutional layers (k = 2, s = 2) with a similar incremental dropout between two consecutive convolutional layers. The final convolutional layer uses k = 1, s = 1 to generate a 256 × 256 image after sigmoid activation. The model was optimized using Adaptive moment estimation (ADAM) optimizer with β 1 = 0.9 and β 2 = 0.999. A batch size of 16 was used for training the network. All network kernels were initialized with a normal distribution. The initialization of kernels with a normalized range of known standard deviation is an essential factor in achieving the proper training in deep learning. It is one of the many granular implementation techniques that have been acquired after many observations and experiments.

  tween predicted and actual labels, whereas the focal loss addresses class imbalance by reducing the effect of biased or skewed classification on the predicted results.As deep learning training and evaluation depends significantly on the specialized hardware used, the choice of hardware is an explicit consideration in measuring the network's performance and comparing it with competing methods. All training and testing for SpecSeg network was done on the Nvidia P100 card, released in April 2016 and based on Nvidia's proprietary Pascal Architecture.

Figure 4 . 1 :

 41 Figure 4.1: Segmentation results of SpecSeg network as compared to manually labelled ground truths in the Whu-Specular dataset[START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]

Figure 4 . 2 :

 42 Figure 4.2: Segmentation results of SpecSeg network as compared to manually labelled ground truths in the Whu-Specular dataset[START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]

Figure 4 . 3 :

 43 Figure 4.3: Segmentation results of SpecSeg network as compared to manually labelled Ground Truths (GT) in the SIHQ dataset[START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF]

Figure 4 . 4 :

 44 Figure 4.4: Zoomed-in ground truth (GT) and prediction (Pred) views of the marked sections in RGB images. SpecSeg network is successfully able to detect regions that are (a) on light-coloured objects, (b) small in size, (c) in multiple blocks with cavities inside specular regions, (d) clipped around the edges of the image, (e) detect specularity correctly from images on a white background.

  their paper, all learning-based methods were retrained on the same dataset (WHU-Specular dataset), and the authors fine-tuned the hyperparameters to give the best possible results. SpecSeg was also trained on the same training dataset, and the same validation and test sets were used to generate a fair comparison. The results of the segmentation masks generated by SpecSeg are significantly better than the classical methods. The results are also comparable to other SOTA deep learning-based

Figure 4 . 5 :

 45 Figure 4.5: Specular segmentation results on outdoor images acquired on a sunny day and under clear sky conditions. Specular reflections detected under extreme conditions are plausible and significantly better than any other state-of-the-art technique. Note that brightly lit regions such as the sky or water puddles are not detected as specular regions.

Figure 4 . 6 :

 46 Figure 4.6: (a) A summary of the metrics over the entire dataset. (b) Training and validation losses after 200 epochs. The training was stopped after 200 epochs to avoid overfitting by the network.

  large times required for training the networks. To compare training time with the other methods, our developed network was trained on the Whu-specular training dataset for 200 epochs for a mere 40 minutes on a P100 (Pascal architecture). In comparison, the SHDNet achieved its results after training for 100 epochs in 80 hours on a GTX-1080Ti (also Pascal architecture). This significantly reduces training time without the need for additional computational power to achieve comparable segmentation results. For training and inference comparison, Fu et al. [63] trained and tested their network on the NVIDIA GeForce GTX 1080Ti, which was released in March 2017 and is based on the Pascal Architecture by Nvidia. in comparison, our training and testing were done on the NVIDIA P100, released in April 2016 and also based on Pascal Architecture. Having the same architecture helps to maintain similarity in the performance, allowing to compute performance metrics to be as close as possible. Note that the authors of SHDNet have not provided their PyTorch or

  gional pixels. The developed weighted median inpainting method shows that the specular regions are replaced with a reasonably good estimate of the homogenous diffuse colour of the underlying object. The proposed weighted median inpainting equation can be interpreted as lowering the intensity peak in the Y channel while increasing or decreasing the colour values in the Cb and Cr channels to gravitate towards the colour values of the surrounding region. This enables significantly improved lost colour information restoration in specular regions without generating sharp discontinuities in the resulting diffuse images.

Figure 4 . 7 :

 47 Figure 4.7: Specular mitigation results by using weighted median inpainting method

Figure 4 . 8 :

 48 Figure 4.8: Limitations of weighted median inpainting method

  4. A brief summary of the network is presented here for context, along with reasons for selecting the specific hyperparameters for training. Definitions and benefits of the most essential hyperparameters used are discussed in depth in section A.1.3. SHMGAN utilizes polarimetric images in YCbCr colour space and learns the variation in illumination in the Y channel of the images. It comprises the Y (luma), Cb (bluedifference) and Cr (red-difference) chroma components, where the luma encodes the light intensity in the scene in the form of a grey channel. The developed SHM-GAN network consists of a single Generator (G) and Discriminator (D), modelled to learn the illumination variation between 5 input images. We use four orthogonal pairs of polarimetric images I 0,45,90,135 . In addition to the four polarimetric images, a pseudo-diffuse image I ED is estimated and passed on to the network as the fifth input domain. This estimated diffuse image can be considered an initial solution to the specular-free image as it contains the least specular reflection component and is required to aid the convergence of SHMGAN towards the desired specular-free image. The generator is based on the U-net structure and consists of 5 encoders, 4 decoders and 1 residual block, with each path from the encoder passed to the decoder via a modified skip connection. All blocks consist of 2D convolutions layers with LeakyReLU activation followed by an Instance Normalization (IN) [154] layer. The encoder uses average pooling to downsize the layers, whereas the decoder uses a stride (s) of 2 to upscale the inputs. The output layer of the generator is a dense layer with filter (k) with k = 1, s = 1. The decoder consists of a series of 5 convolution layers with s = 2, and the 2D convolutions blocks use Leaky ReLU activation functions followed by Instance Normalization. The U-net skip connections are modified by the element-wise addition of a dynamically generated self-attention mask that enhances the attention of the generator network on specular highlights as described in section 3.4. The discriminator is comprised of 5 blocks of 2D convolutional layers with kernel sizes k = 4 and s = 2, followed by an instance normalization and LeakyReLU activation and the same padding. The self-attention mechanism is also added to the discriminator before the last convolutional layer block. The discriminator has two outputs; one is the classification of the real or fake (D cl s f ) while the other is the target label of the generated images (D ad v ). The real/fake classification is a 2D convolutional layer with Leaky ReLU activation, whereas the target label classification is done using a dense layer with softmax activation. All input images are converted to the YCbCr colour space and normalized to [0, 1] before being input to the generator but converted back to RGB before being fed to the discriminator. The five Y-channels of the input images I 0,45,90,135,ED are concatenated along the channel dimension to form a 5D tensor. The images are then one-hot encoded along the channel dimension as binary matrices with the exact dimensions as the input image for a tensor of dimensions (b, n, m, 10) where b is the mini-batch size, n, m are the image dimensions and the channel dimension is 10. The one-hot encoded channels are used to designate the generator's target images and the discriminator's target label. We use a mini-batch size of 1, where each batch is considered as a set of the five spatially and temporally coherent input images I 0,45,90,135,ED .

equation 3 .

 3 22 in section 3.4.4. The network was trained on a single Nvidia RTX3070 GPU with 8GB memory for 140 epochs, as well as the Nvidia P100 with 16GB RAM for comparison. The total training time for 140 epochs was around 24 hours. The total memory consumed was around 5GB for an image resolution of 128 × 128. The psuedocode for SHMGAN is given in Algorithm 2. Input randomization is implemented by substituting the labels of the cyclic images with zeros. Other techniques such as label smoothing, gradient clipping, adding noise to the inputs etc., were also utilized to improve the training results. A detailed flow chart of the implementation is shown in Fig. 3.4 and the related pseudocode is given as Algorithm 2. As shown by the flowchart, the SHMGAN generator is called twice to generate images from augmented inputs ( Algorithm 2 line numbers 8, 14 for the input image and cyclic image generation respectively), whereas the discriminator is called four times (Algorithm 2 line numbers 3, 10, 15, 16 for learning to discriminate original images, generated image, cyclic image and target images re-

Figure 4 . 9 :

 49 Figure 4.9: Confusion matrix of the training SHMGAN on PSD dataset.

Figure 4 . 10 :

 410 Figure 4.10: All polarimetric angles generated by SHMGAN network. The polarimetric images generated are realistic and have a variation of specular illumination in all polar angles.However, this variation cannot be considered as physically accurate as the target image was only the diffuse image, and no polarimetric constraints are provided to ensure physically accurate generation.

Figure 4 . 11 :

 411 Figure 4.11: Visual comparison of testing on the PSD dataset[START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. The methods compared include both traditional image processing techniques[START_REF] Shen | Chromaticity-Based Separation of Reflection Components in a Single Image[END_REF][START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF][START_REF] Yang | Real-Time Specular Highlight Removal Using Bilateral Filtering[END_REF] and modern GAN based methods[START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF][START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF].

Figure 4 . 12 :

 412 Figure 4.12: Visual comparison of testing on data collected outdoors. The classical image processing methods are unable to perform due to the presence of large regions of brightly lit areas in the scene. SpecularityNET has some visible distortions in the images, whereas the developed network is able to generate images with slightly reduced reflections but without noticeable distortions.

Figure 4 . 13 :

 413 Figure 4.13: Visual comparison of testing on the in-house dataset. The methods compared include both traditional image processing techniques[START_REF] Shen | Chromaticity-Based Separation of Reflection Components in a Single Image[END_REF][START_REF] Shen | Simple and Efficient Method for Specularity Removal in an Image[END_REF][START_REF] Yang | Real-Time Specular Highlight Removal Using Bilateral Filtering[END_REF] and modern GAN based methods[START_REF] Lee | Col-laGAN: Collaborative GAN for Missing Image Data Imputation[END_REF][START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF].

Figure 4 . 14 :

 414 Figure 4.14: Results of detected specularity by self-attention mechanism and diffuse images by SHMGAN, on the TRIW dataset.

Figure 4 . 15 :

 415 Figure 4.15: Results of detected specularity by self-attention mechanism and diffuse images by SHMGAN, on the SHIQ dataset.

4. 4 . 3

 43 Quantitative resultsFor qualitative comparison, the metrics used for comparison are Mean Square Error (MSE), Structural Similarity Index (SSIM), Peak Signal to noise ratio (PSNR) and ∆E. ∆E is the measure of change in visual perception of two given colours calculated in the Lab colour space. On a typical scale, the ∆E value will range from 0 to 100, with lower values representing a lower colour difference. Two different CIE standards were used to compare the ∆E results since both are widely used; however, the values are calculated slightly differently in each standard.As can be seen, by the results in tables 4.5, for the PSD dataset, SHMGAN can outperform the classical and state-of-the-art techniques. While the SSIM and PSNR are very close to[START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF], the images produced by SHMGAN have lesser colour distortion and a higher PSNR. The mean ∆E of the image generated by SHMGAN are lower than the competing methods signifying lesser colour aberrations and distortions in both datasets. As expected, the classical methods of specular highlight mitigation methods cannot perform on most real-world images, as represented by their low overall scores. For the in-house dataset, overall, SHMGAN performs better or at par with all competing methods by generating images with low noise and MSE. The ∆E is also at par with the competing methods. The complete picture of testing GAN net-

  values. Therefore we also present a summary of the trend over the entire test set in Figure4.16. The generated specular-free images by our developed network has a tighter spread of PSNR, MSE and SSIM values with fewer outliers, strengthening the confidence in the quality of the resulting images. The network is also able to produce realistic colours, but the overall spread of the resulting images shows some room for improvement in future works.

  . Polarimetric images allow the network to learn the subtle variations in illumination in the different angles, each cancelling out a portion of specular reflection in that angle.

Figure 4 . 16 :

 416 Figure 4.16: Summary of quantitative results comparing the spread of results of the developed network with SpecularityNet. in the (a) PSD dataset and (b) in-house dataset.

Figure 4 . 17 :

 417 Figure 4.17: Results of ablation study after 70 epochs of training with only two input images (RGB and I ED ) instead of the developed five images show that even after extended training, the network generates images with artefacts and is unable to remove specular reflections effectively.

Figure 4 . 18 :

 418 Figure 4.18: Results of ablation study after 70 epochs of training without self-attention show that the specular reflections are not fully mitigated, and the images generated have distorted colours.

Figure 4 . 19 :

 419 Figure 4.19: Results of ablation study after 70 epochs of training with various loss combinations. (a) For ablation results without specular loss (b) ablation results without SSIM loss.

Figure 4 . 20 :

 420 Figure 4.20: Ablation study (after 50 epochs) of clipping gradients before backpropagating weights. Clipping the gradients to [0, 1] resulting in exponentially increase in the generator loss and produced poor resulting images.

Figure 4 .

 4 Figure 4.20 shows the effect of gradient clipping on the network. Before the learned weights are backpropogated and the weights are updated, clipping the gradient to certain bounds is recommended to avoid the vanishing gradient problem. The Figure shows the comparison of using the [0, 1] bounds versus [1, -1]. As the results clearly show that clipping the gradients to [0, 1] causes the generator training loss to grow uncontrollably out of bounds, resulting in the generator to output poor images. The Figure shows the results after training 50 epochs only, as training any

Figure 4 . 21 :

 421 Figure 4.21: Ablation study of benefits of image standardization on GAN generation. Non standardizing images results in loss of colour generation and large blobs only after ten epochs.

• Objective 1 :-• 2 :• Objective 3 :

 123 ing research work. Both the objectives and the resulting work of this thesis can be summarized as given below: To develop a deep learning-based segmentation network for highly accurate detection of specular highlights in real-world images at near real-time performance. -Contribution: A fast and accurate Specular Highlight Segmentation Network (SpecSeg) was developed that is able to accurately detect and segment out the specular pixels with a very low MAE score of 8 × 10 -3 and mean inference time of 3.1ms. Objective To utilize polarimetric imaging and leverage specular highlight polarization properties to learn accurate diffuse colour recovery. To develop a deep learning-based image translation network for mitigating the detected Specular highlights and generating specularity-free images from a single input image.-Contribution: Two methods for Specular highlight mitigation were developed. A fast diffuse colour inpainting method that utilizes the detected regions from our developed SpecSeg network and inpaints the affected regions with an estimated diffuse colour inferred from the boundary regions. And a Multi-domain Specular Highlight Mitigation Generative Adversarial Network (SHMGAN). SHMGAN is trained with datasets of real-world polarimetric images and is able to learn the variation of polarized specular reflections between the different polarimetric images.

Figure 5 . 1 :

 51 Figure 5.1: Pipeline implementing specular highlight detection and mitigation networks. Due to their fast inference times, developed networks SpecSeg and SHMGAN can easily be integrated into existing pipelines for specular highlight removal.

Furthermore, generating higher

  resolution images right now requires significantly large amounts of memory as well as computing power. This is due to the requirement of deep CNN layers. While it has been shown by state-of-the-art generative models such as DALL•E [167] and DALL•E 2 [168], generating high-resolution models is quite possible with the availability of large data centres and TPU clusters for training and inference. The current CNN-based generative adversarial networks are extremely hard to train and require a significant amount of fine-tuning to get realistic and plausible results. Several up-and-coming alternative models to CNN networks are being developed or adapted from other applications. Transformers are networks originally intended for NLP tasks that have shown promising results for high-resolution image generation. Recently, Diffusion Models have emerged as a powerful class of generative learning methods. These models, also known as denoising diffusion models or score-based generative models, demonstrate surprisingly high sample quality, often outperforming traditional generative adversarial networks. Both these models can be explored for specular highlight mitigation models to extend current work performance. Additionally, as the Tensorflow framework is in a constant state of active development, some of the performance-related issues are being catered to in forthcoming updates. However, some cross-compatibility issues of CUDA, NumPy and other packages require the constraining of the version used for developing and training the model. It is hoped that with Tensorflow 3.0 release in the near future, several performance issues should be tackled alongside compatibility with newer packages allowing for reduced training times.

Figure A. 2 :

 2 Figure A.2: Figure depicting a generic kernel filter in a convolutional neural network.

Figure A. 3 :Figure A. 5 :

 35 Figure A.3: Figure depicting upscaling or encoding using a generic transposed convolution

Figure A. 6 :

 6 Figure A.6: Popular activation layers used in CNNs.

Figure A. 7 :

 7 Figure A.7: Example of various non-destructive and destructive transformations for data augmentation that can be used for increasing the dataset size without causing the network to overfitting trained weights.

  . . . . . . . . . . . .

4.12 Visual comparison of testing on data collected outdoors. The classical image processing methods are unable to perform due to the presence of large regions of brightly lit areas in the scene. SpecularityNET has some visible distortions in the images, whereas the developed network is able to generate images with slightly reduced reflections but without noticeable distortions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 Visual comparison of testing on the in-house dataset. The methods compared include both traditional image processing techniques

  2, segmentation of specular highlights has seen significant research; however, a significantly large amount of techniques work only on ideal images taken under controlled environments as opposed to real-world images captured under uncontrolled situations. Image regions affected by very strong specular highlights are poorly estimated by most algorithms which completely or partially fail to estimate and restore the underlying colour of the affected objects.Most of the accurate and reliable methods trade-off highlight mitigation accuracy with speed and thus are not suited for real-time applications. The recovered diffuse images after mitigation are often not representative of the original diffuse colour of the body, and there is an apparent loss of colour information, especially in areas of strong specular reflections. Furthermore, enforcing specular highlight mit-

igation to conform to unrealistic constraints such as single scene illuminant and polarized light source does not represent real-world conditions for the application of the state-of-the-art algorithms. Most methods often have adverse effects on the recovered image, such as altering contrast and distorting the colour of the objects in the scene. These problems are yet unsolved and indicate that specular highlight mitigation is an area with a notable gap in the availability of a fast and accurate segmentation and mitigation method.

  images and uses the specularity mask generated by SpecSeg network as a self-guided attention layer to learn to mitigate the specular reflections. The scope of the thesis has been limited to using only real-world images and avoiding synthetic training images. Additionally, medical image segmentation and generation were considered out of scope of the current problem domain, even though the methods developed herein can also be trained and tested for medical imaging applications. The research resulted in several peer-reviewed publications that are listed in the list of publications.

Two CNN-based networks deep learning are developed in this thesis to attain these objectives and alleviate the effects of specular reflections. The first is a SpecSeg Network that is able to accurately detect specular reflections in input real-world images without any additional guidance or labelling. A second deep generative adversarial network called SHMGAN is also developed that is able to generate a specularity-free image from a single input image. The SHMGAN is trained to learn the illumination variation among polarimetric

39 2.3 Specular highlight mitigation . . . . . . . . . . . . . . . . . . . . . . 40

  Torrance-Sparrow microfacet model . . . . . . . . . . . . . . . 21 2.1.2 Dichromatic Reflection Model (DRM) . . . . . . . . . . . . . . 23 2.1.3 Intrinsic image decomposition and specular reflections . . . 25 Limitations of the current state of the art . . . . . . . . . . . .

	2.2.1 Classical specular detection and segmentation methods . . . 29
	2.2.2 Deep learning based methods . . . . . . . . . . . . . . . . . . 35
	2.2.3 2.3.1 Classical methods of specular highlight mitigation . . . . . . 41
	2.3.2 Polarization and specular highlights . . . . . . . . . . . . . . . 43
	2.3.3 Mitigation of specular highlights using Polarization . . . . . 50
	2.3.4 Deep learning based methods . . . . . . . . . . . . . . . . . . 55
	2

The concepts explored herein build the framework of the problem addressed in this thesis as well as establish the reasons and background for selecting the segmentation and mitigation solutions proposed in the thesis. Empirical and physical concepts such as intrinsic images, micro-facets and chromaticity-based models are conceptualised in-depth, setting up the causes and impact of diffuse colour and specular reflections in images. Properties of strong specular highlights that help understanding the issues caused by specular reflections are elucidated in detail. The phenomenon of light polarisation is explored in depth, and its relation to the highly polarised reflection components in images is also demonstrated. Prior literature focused on detecting specular highlights is reviewed in depth, including both classical image processing methods as well as deep learning-based solutions available. Mitigation of specular reflections is the core focus of this thesis and the prior literature over the years that is relevant to specular highlight removal is explored in depth, including both classical and deep learning-based state-of-the-art methods. Multi-domain and multi-modal generative adversarial networks are explained briefly along with an details of the popular datasets used in literature for specular highlight detection and mitigation. Lastly, an in-depth criticism and limitations of the state-of-the-art is discussed.

Contents 2.1 Physical Model of Light Reflection . . . . . . . . . . . . . . . . . . . 20 2.1.1 2.2 Specular highlight detection and segmentation . . . . . . . . . . . 27

.4 Multi-domain Generative adversarial networks . . . . . . . . . . . 59

	2.4.1 Generative Adversarial Networks (GANs) . . . . . . . . . . . . 60
	2.4.2 Popular datasets for specular highlight research . . . . . . . . 67

2.5 Datasets for specular highlight mitigation . . . . . . . . . . . . . . 69 2.6 Criticism on state-of-the-art . . . . . . . . . . . . . . . . . . . . . . 70

	2.6.1 Issues with current specular detection methods . . . . . . . . 71
	2.6.2 Limitations in mitigation of specular reflections . . . . . . . . 72
	2

.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

Table 2

 2

	.1: Summary of DRM vs microfacet model for defining specular highlights in images
		Dichromatic Reflection Model (DRM) Cook-Torrance Model
	Model	Physics based simpler model for sur-	More realistic physical based model
		face reflections	for surface reflections
	Material	Does not depend on material-related	Includes effect of material properties
	Interaction	properties	such as refraction, surface roughness,
			Fresnel conductance
	Surface In-	Assumes body is Lambertian, scatters	Body is composed of micro-facets
	teraction	diffuse illumination isotropically	with interreflection, shadow and
			masking effects
	Usage	Useful for simplifying light models	Generally used for accurate rendering
		and studying specular reflections on	of specularity from non-Lambertian
		surfaces	surfaces with high performance.

Table 2 .

 2 2: Summary of prominent non-deep learning based methods for specular highlight segmentation

	Name	Year	Category 1,2	Technique	Color
					space

They propose a Dilated Spatial Contextual Feature Aggregation (DSCFA) to detect and accurately remove highlights of varying sizes. A comprehensive list of the relevant deep learning-based methods on real-world images (excluding medical imaging systems) for detecting specular highlights in images is presented in table 2.3.

Table 2 .

 2 3: Summary of influential deep learning based methods for specular highlight segmentation

	Author	Year	Category 3,4	Type 5	Architecture	Losses		Eval Metrics
	Lee et al. [61]	2010	Segmentation, Mitigation RW	Single layer perceptron	-		-
	Sanchez et al. [59]	2017	Segmentation	MIS	SVM	-		DICE
	Akbari et al. [60]	2018	Segmentation	MIS	SVM	-		DICE, Specificity, Precision
	Funke et al. [62]	2018	Segmentation, Mitigation MIS	SpecGAN	Cyclic loss	MSE PSNR, SSIM
	Fu et al. [63]	2020	Segmentation	RW	SHDNet	BCE, IOUE	F-measure, MAE, S-measure
	Fu et al. [64]	2021	Segmentation, Mitigation RW	JSHDR	BCE, L2		Accuracy, BER
	Monkam et al. [65] 2021	Segmentation, Mitigation MIS	Scaled-UNet, GatedResUNet Mask,	Valid,	SNR, DICE, SSIM, IoU
						Perceptual,
						Style,	Total
						variation

Mitigation: Methods that generate diffuse images.

Segmentation: Methods that segment out specular pixels from the original image Type: Real-world (RW) images or Medical Imaging Systems (MIS)

  If the illuminant I s is normalized for an RGB image, the chroma channels (Cb and Cr) are invariant to specularity as RGB values in Cb and Cr sum to zero using the transformation matrix. Because of the way the transformation matrix has been developed for the YCbCr colour space, after multiplication, the resulting equation only contains the specular component in the Y channel, whereas the specular component in the Cb and Cr channels are removed as shown in equation 2.11.

				0.299	0.587	0.114	
		A =	 	-0.173 -0.339	0.511	 	(2.10)
							
				0.511 -0.428 -0.083
		I y (p)
	  

9)

Where d, s are the diffuse and specular components respectively, normalized offset is defined by [0, 128/255, 128/255] T and A is the direct transform matrix to convert from RGB to YCbCr given by equation 2.10.

Table 2 .

 2 4: Different polarisation states as represented by elements of stokes vector

	Stokes	Unpolarized	Linear	Linear	Linear	Linear	Left	Right
		light	Hori-	Vertical	+45	-45	Circular
			zontal

Table 2 .

 2 5: Summary of various classical computer vision techniques for specular highlight mitigation in literature.

	Author	Year Technique	Colorspace Evaluation Metrics
	Kim et al. [31]	2002 Energy minimization and polarization	RGB	-
	Umeyama et al. [1] 2004 Energy minimization and polarization	Greyscale	-
	Mallick et al.[66]	2005 Eroding S channel	RGB, SUV	-
	Mallick et al.[67]	2006 Inpainting using PDE	RGB, SUV	-
	Ortiz et al. [94]	2006 Intensity-Saturation diagram	Greyscale	-
	Yoon et al. [23]	2006 Specularity-Invariant Value and Ratio	RGB	-
	Shen et al. [25]	2009 Modified specular-free (MSF) image	RGB	-
	Yang et al. [73]	2010 Bilateral filtering	RGB	PSNR
	Jung et al [72]	2012 Joint Bilateral Filtering	RGB	-
	Zhang et al. [33]	2011 Statistical analysis, polarization	RGB	-
	Shen et al/ [84]	2013 Intensity ratio	RGB	PSNR
	Kim et al. [28]	2013 Dark channel prior	RGB	-
	Yang et al. [21]	2013 Lease square chromaticity error	Ch-CV	-
	Yang et a;l. [42]	2013 Region-Growing Algorithm	HSI	PSNR
	Zou et al [43]	2013 Dark Channel Prior	RGB	-
	Nguyen et al. [95]	2014 Tensor voting	RGB	MSE
	Yu et al. [68]	2014 Inpainting	HSV	MOS
	An et al. [27]	2015 Clustering using K-Means	RGB	PSNR
	Fang et al. [78]	2015 Dark channel prior, polarization	RGB	-
	Yang et al. [26]	2015 Bilateral Filter	RGB	PSNR
	Zhao et al. [96]	2015 Local Structural Similarity	RGB	CurveletQA
	Akbar et al. [69]	2016 Sparse Coded Decomposition	ZYZ	MAE
	53			continued . . .

Table 2 .

 2 • . Being the biggest publicly available polarimetric dataset to date, the PSD dataset adds a much-needed boost to research on polarimetric images, especially for data-centric methods such as deep CNN-based networks. Wu et al. also proposed a GAN network for specularity removal. Xu et al. [115] also proposed a GAN network; however, it is trained to work on greyscale images only. Of the most recent research works, Wang et al. [116] released SIHRNet, which utilizes extracted layers from a pretrained VGG-19 network trained on ImageNet dataset and adds nine convolutional blocks to generate specular free images from an input image. 6: Table of prominent research works on specular highlight mitigation by deep learning

	Name	Year	Network	Loss Functions	Eval Metrics
	Funke et al. [62]	2018 SpecGAN	Cyclic loss	MSE, PSNR, SSIM
	Lee et al. [117]	2019 CollaGAN	Cycli loss SSIM
	Lin et al. [105]	2019 SRN	Content loss, adversarial loss	L2, DSSIM
	Xu et al. [106]	2019 CDFF-Net	L1 loss, perceptual loss	PSNR, SSIM, RMSE
	Muhammad et al. [109]	2020 SpecCGAN		FID, LMSE, PSNR, SSIM
	Wu et al. [110]	2020 GAN	Cyclic, RaSGAN, Identity Loss	DSSIM, MSE, PSNR
	Fu et al. [64]	2022 JSHDR	BCE IOUE	F-measure, MAE, S-measure
	Hou et al. [112]	2021 Net D , Net R	Highlight Detection, Reconstruction, Text-Related F-measure, PSNR, Precision, Recall,
					SSIM
	Jimenez-Martin et al. [113]	2021 GAN	MSE (L2 norm)	MSE
	Monkam et al. [65]	2021 Unet	Mask, Valid, Perceptual, Style, Total Variation	Dice Coefficient, IOU, SNR, SSIM
	Wu et al. [114]	2021 GAN	Adversarial, Feature, Focal, Pixel	MSE, PSNR, SSIM
	Yoo et al. [118]	2021 EfficientNet, Unet	Color constancy, coefficient, reconstruction, tem-	Mean, Median, Tri-mean
				poral regularization
	Daher et al. [119]	2021 Temporal GAN	Temporal GAN Inpainting	RMSE
	Xu et al. [115]	2022 CycleGAN	MSE, SSIM, Attention, adversarial	RMSE
	Wang et al. [116]

Table 2 .

 2 7: Brief comparison of StarGAN and CollaGAN networks.

	Property	StarGAN		CollaGAN
	Category	Image to Image translation	Image Imputation
	Inputs Required	Single RGB image		Multiple RGB images (all domains)
	Training Inputs	Multi-domain RGB images	Multi-domain RGB images
	Working Color space RGB		RGB/YCbCr
	Output Type	RGB		RGB / Grayscale
	Key Feature	Cross-domain,	image-to-image	Missing	domain	imputation
		translation		among multiple domains
	quality images compared to a vanilla CycleGAN. An example of the StarGAN model
	is given in Figure

Table 2 .

 2 8: Different types of attention in deep neural networks

	Atention type	Description
	Self-Attention	Relating different positions of the same input sequence. In theory, self-
		attention can adopt loss function by replacing the target sequence with
		the same input sequence.
	Global/Soft Attention Attending to the entire input state space.
	Local/Hard Attention Attending to the part of input state space i.e. a patch of the input image

Table 2 .

 2 9: Table listing notable datasets with publicly available specular highlight imaging datasets especially used for machine learning algorithms

	Dataset Name	Year Cat. 5,6,7	Total	Specular	Diffuse	Test-	Size
			images	Mask	Image	train
						split
	Spec-DB [38]	2003 RW	300				10 MB
	MIT Intrinsic Images
	dataset

Table 3 .

 3 1: Summary of the datasets used for training and testing

		Mode	Dataset	Images Mask
	SpecSeg	Training	Whu-Specular Dataset	4310
			Whu-Specular [63]	1293
		Testing	PSD	38
			TRIIW [64]	500
	WMI 1		Classical images	50	Partial
		Testing	Whu-Specular dataset	1293
			In-house dataset	330
	SHMGAN	Training	PSD dataset train set	3072
			Whu-Specular Dataset	1293
		Testing	TRIW PSD dataset test set	500 54

Table 3 .

 3 2: Table of different evaluation metrics used in literature. indicates higher value is better (generally scaled to 1), whereas means a lower value is better (generally scaled to 0). generative models like GANs, there is no consensus as to which measure best captures the strengths and limitations of generative models. As in other areas of computer vision and machine learning, it is critical to settle on one or few suitable measures to steer the progress in this field. Qualitative metrics are very subjective and often have a human bias associated with them. Quantitative measures are less subjective and do not directly correspond to how humans perceive and judge images. However, several perceptually meaningful image similarity measures make the results more intuitive to how we perceive an image. Borji et al.

	Error	Abbreviation Better if Compare with
	Jaccard index / IOU	IOU	Mask
	Dice Coefficient / F1 Score	F1 Score	Mask
	Precision, Recall	PR	Mask
	F-measure	F	Mask
	Mean Absolute Error	MAE	Mask
	Root Mean Squared Error	RMSE	Mask
	Peak Signal to Noise Ratio	PSNR	Image
	Delta E	DE	Image
	Structural Similarity	SSIM	Image
	the performance of [158] extensively reviewed the pros and cons of GAN evaluation metrics, including
	the traditional log-likelihood, image quality metrics such as Peak Signal to Noise
	Ratio (PSNR), Structural Similarity Index Measure(SSIM) and Precision, Recall and
	F1 scores etc. A detailed explanation of each of the metrics used for analysis is also
	presented in Appendix B.

Table 4 .

 4 1: Qualitative comparison of SpecSeg network to classical and deep learning stateof-the-art methods

	Metrics	Year	Type	S-m 1	meanF 1 MAE 2

Table 4 .

 4 2: Training time comparison of different segmentation networks It is clear from the results in table 4.2 that the training time required by SpecSeg is an Order of magnitudes better than all other competing networks. Furthermore, the inference time is also faster than the competing

	Author	Network	GPU	Epochs	Training	Inference
					Time	Time
	Monkam et al. [65]	ScaledUNetGTX 2080Ti	50	-	3.43 ms
	Ronneberger et al.	UI-Net	NVidia Titan	-	10 h	14.13
	[143]
	Fu et al. [63]	SHDNet	GTX1080Ti	100	80 h	-
	Fu et al. [64]	JSHDR	GTX 2080Ti	100	3 days	-
	Proposed	SpecSeg	Nvidia P100	140	40 mins	3.1ms
	Tensorflow implementation code for public access, so retraining their network on
	any dataset was impossible.

networks. As noted above, since the code or training weights of SHDNet or JSHDR have not been provided publicly, it was impossible to retrain and test on the same hardware for a 100% fair comparison. However, the hardware used is comparable and can be treated as similar for all intents and purposes for deep neural network training.

  order to test the developed network, an in-depth ablation study was carried out by varying different aspects of the network. As shown by the performance comparison in table 4.2, the training time for the network is very low, which significantly helps in testing different configurations and hyper-parameter tuning of the network. Several variations were constructed by editing the activation functions of the SpecSeg network. A separate training session also noted the benefit of using batch normalization. Additionally, varying the loss functions with alternate losses versus the proposed joint loss L Tot al was also studied. A comparison of different metrics calculated from the resulting ablation studies is shown in the table 4.3. The proposed losses combined with batch normalization and LRelu activation give the best results for PSNR, MSE, Dice and S-measure scores, whereas the SSIM score is lower only by a negligible amount. Using Leaky ReLU activation gives the overall best scores as it avoids the vanishing gradient problem. The combination of dice and focal losses appear to converge successfully towards the best results on the testTable 4.3: Ablation study results of different variations of the SpecSeg network

		PSNR ↑	SSIM ↑	MSE ↓	F_m ↑	S_m ↑
	No BN	23.7213	0.9539	0.0092	0.4662	0.6643
	BN+SparseCE loss	25.2064	0.9628	0.0076	0.5072	0.6598
	Elu Activation	21.9828	0.9494	0.0122	0.4308	0.6138
	Linear Activation	24.0415	0.9609	0.0092	0.0067	0.5214
	Baseline(BN+LReLU+L Tot al ) 25.2211	0.9625	0.0073	0.5278	0.6761
	dataset. All ablation tests were carried out on the same hardware and did not see
	any change in training time.

Table 4 .

 4 5: Mean qualitative comparison of the generated test images from PSD test dataset and selected appropriate methods with the best results in bold text.

	Method			PSD Test set
		PSNR	SSIM	MSE	∆E CIE76	∆E CIE94
	Shen et al. [25]	18.5649 0.8478 0.02227 72.0296	46.2681
	Yang et al. [73]	14.9773 0.8065 0.03742 86.9731	43.2086
	SpecularityNET [114] 17.8149 0.8305 0.02727 73.3086	48.2662
	SHMGAN	19.5700 0.8625 0.0153 68.2535 42.9445
			Inhouse dataset
	Shen et al. [25]	13.215	0.5737	0.0688	68.1804 39.0154
	Yang et al. [73]	11.585	0.5007	0.0975	69.7143	37.8468
	SpecularityNET [114] 13.7769	0.6415 0.0609	74.0787	40.7022
	SHMGAN	15.0282 0.6322	0.0449 72.6276	37.9153

works over a large number of images is often not fully represented by single mean

Separation: Methods that separate distinct specular and diffuse images that are additive.

Segmentation: Methods that segment out specular pixels from the original image, but do not generate diffuse image.

SIHRNet Structure loss, Feature loss, Region loss SSIM, PSNR Xu et al. [115] 2022 Attentive GAN MSE Loss, SSIM Loss SSIM, PSNR, MSE

OH: One Hot

2-D matrix labels 2 r/f : Real/Fake classification result of the discriminator

Weighted median Inpainting

Acknowledgements

List of Algorithms

List of publications

During this Ph.D, the following research papers have been accepted and submitted for publication:

[1] A. Anwer, S. Ainouz, M. N. M. Saad, S. S. A. Ali, and F. Meriaudeau, "SpecSeg network for specular highlight detection and segmentation in real-world images," Sensors, vol. 22, no. 17, 2022, doi: 10.3390/s22176552.

[2] A. Anwer, S. Ainouz, N. M. Saad, S. S. A, Ali, and F. Meriaudeau, "SHMGAN: Joint Network for Specular highlight detection and mitigation in real-world images,"

Neurocomputing, Auguest. 2022 (Submitted). [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF] 54 Yes No Whu-Specular [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF] 1293 No Yes TRIW [START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF] 500 No No spectively), during a single training step. The combined loss function is then calculated as described in section 3.4.4 and back-propagated.

Datasets and methods for training and testing

To qualitatively and quantitatively compare the generated images, datasets comprising real-world images, both with and without ground truth, were used. A detailed table of the dataset and the exact number of images used for training and testing are given in table 3.1 and a confusion matrix of the training result is displayed in Figure 4.9. The most extensive dataset available with pure diffuse ground truth images was recently made available by Wu et al. [START_REF] Wu | Single-Image Specular Highlight Removal via Real-World Dataset Construction[END_REF]. The authors provide 12 polarimetric images per scene with 30 deg increments; however, we only selected four orthogonal angles for training, namely I 0,60,90,150 and the pseudo-diffuse I ED image was calculated using these images. These polarimetric angles were selected to capture maximum specularity variation in the orthogonal images. Data was also acquired in-house using monochrome and colour polarimetric cameras in various settings and lighting conditions, as described in section 3. For testing, the SpecularityNet was also trained separately on the Whu-Specular dataset and quantitative analysis was done by training the developed network on the same dataset. Other large datasets comprising real-world images with manually labelled specular pixels are Whu-Specular, and TRIIW datasets [START_REF] Fu | Learning to Detect Specular Highlights from Real-World Images[END_REF][START_REF] Fu | A Multi-Task Network for Joint Specular Highlight Detection and Removal[END_REF]. No groundtruth diffuse images are provided in these datasets; therefore, they can only be used

Conclusions

The main objective tackled by this thesis was the challenging problem of specular highlight detection and mitigation using state-of-the-art deep-learning-based solutions. There were three main research questions selected as the focal point of this research work, as described in section 1.4. Namely, how can we accurately separate specular pixels in any real-world image? And how can polarimetric cameras with on-sensor polarizer filters be utilized to find a robust and efficient specular mitigation method? And lastly, what are the most effective methods that can be explored and utilized for specular highlight mitigation? The answer these questions, two broad objectives were selected as the main focus of this thesis and the ensu-

The input tensor can be a representation of anything such as text or image however, we will primarily be considering image as input tensors. A key property of CNN's is that the whole network is fully differentiable. This allows for backpropogation, a widely used algorithm to train neural networks. Backpropagation computes the gradient of the loss function f with respect to the weights of the network w i in an efficient manner than direct computation of the gradients with respect to each weight individually. The gradients are calculated for each layer and iterated back- to recognise a pattern in any location in an image. Spatial hierarchy allows the network to learn increasingly complex and abstract concepts. Intuitively, the network will learn filters that activate when they see some type of visual features, such as an edge of some orientation or a blotch of some colour on the first layer, and eventually, the entire patterns on higher layers of the network. A convolutional layer has an entire set of filters, and each of them produces a separate 2D activation map that is stacked along the depth dimension to produce the activation maps as an output tensor.

Each layer of a CNN can also be thought of as individual pieces of LEGO ® with two spatial axes (height and width) and a depth or channel axis, that are stacked one on

A.1.2 Residual networks or skip connections

Neurons in a fully connected (or dense) layer have full connections to all activations in the previous layer, as seen in regular Neural Networks. Their activations can hence be computed with a matrix multiplication followed by a bias offset. In CNN, however, the convolution operation is the sum of element-wise multiplication between the sliding window and the filter and can be defined as A.2. Residual or skip connections provide another path for data to reach the latter parts of the neural network by skipping some layers.

Where

A.1.3 CNN hyperparameters

Some additional and critical parameters for CNN are padding, stride, batch size and pooling, which can be explained as follows.

Padding

Padding consists of adding an appropriate number of rows and columns on each side of the input feature map so as to make it possible to fit centre convolution windows around every input tile. A padding of 'same' means to add padding so that the output has the exact dimensions as the input. A 'valid' padding means no padding is used (only valid window locations are used). The output size after the padding operation can be calculated by A.3.

Where O is the output size, W is the input dimension, k is the filter size, p is padding, and s is the stride.

Stride

In simplest terms, a stride is the jump distance between the current and following pixel positions on which the convolution is applied. It has the additional effect of downscaling the shape of the output feature map by the stride factor. Strided convolution is generally not used in classification problems but significantly impacts the segmentation and generation of images. The output of strided feature maps are explained by A.4 where s is the convolution stride.

Pooling

Pooling is typically used to reduce the dimensions of feature maps as the network depth increases. This reduction in spatial dimension is generally done after the nonlinear convolution and also aids the reduction of computational resources. Pooling is done by dividing the image into ω p × ω q windows and applying either the maxi-

V

Activation functions

An activation function is an operation applied on a neural network node and is defined as the weighted sum of the input layers into an output layer or node. A network may have three types of layers: *input layers* that take raw input from the domain, *hidden layers* that take input from another layer and pass output to another layer, and *output layers* that make a prediction. All hidden layers typically use the same activation function. The output layer will typically use a different activation function from the hidden layers and is dependent upon the type of prediction required by the model. Furthermore, activation functions are differentiable, so they can be used for back-propagation to update the layers' weights. Activation functions are a critical part of the design of a neural network. And the choice of activation function impacts the capability and performance of the neural network, and different activation functions may be used in different parts of the model to achieve various goals.

An activation function f is mathematically defined as equation A.6 [START_REF] Dumoulin | A Guide to Convolution Arithmetic for Deep Learning[END_REF].

where x = [x 1 , x -2, . . . x m ] T is the input neuron, y is the neuron output, w = [w 1 , w -2, . . . w n ] T are the weights, w 0 is the bias. Several activation functions have been proposed over the years, each having its benefits and usage in various scenarios.

Some popular activation functions and their profiles are illustrated in A.6.

Batch and Batch Normalization (BN)

Before the batch images are passed on to the next block of the network, they have to be normalised as a pre-processing step to standardise the weights. This enables faster training speeds and allows for using higher learning rates. The mainstream normalisation technique widely adopted for convnets is BN Proposed by Google in 2015 [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF], BN normalises all images across the batch and spatial locations and can accelerate a model's converging speed while alleviating issues such as Gradient Dispersion, making it easier to train models. Mathematically, batch normalisation is defined as A.7:

B.1.3 Precision and recall

Precision is how precise the model is, as given by the number of positive prediction values. It is a measure of the quality of the predictions made by the algorithm. i.e.

out of all the positive predictions, how many are True Positives predictions. The higher the precision score, the better. On the other hand, Recall is the True Positive Rate, i.e. out of all the actual positives, how many are True Positives predictions.

Both can be calculated using equations B. The SSIM index ranges between 0 and 1, with the value of 1 representing two identical sets of images and therefore indicating perfect structural similarity.