Directrice De Thèse

Prof Trocan Maria

Devant

M Dariusz

Krol Rapporteur

M Nistor

Grozavu Rapporteur

M Dan

Istrate Examinateur

M Christophe

Marsala Examinateur

M Thomas

Frossard Directeur

Lab Innovation

Tinubu M Martin

Sheppard Directeur

Risk Tinubu

Mme Natalia Díaz-Rodríguez

Co-Encadrante Mme

Directrice

An EXplainable Artificial Intelligence Credit Rating System

Over the past few years, the trade finance gap has surged to an alarming $1.5 trillion, underscoring a growing crisis in global commerce. This gap is particularly detrimental to small and medium-sized enterprises (SMEs), which often find it difficult to access trade finance. Traditional credit scoring systems, which are the backbone of trade finance, are not always tailored to assess the creditworthiness of SMEs adequately.

The term credit scoring stands for the methods and techniques used to evaluate the creditworthiness of individuals or business. The score generated is then used by financial institutions to make decisions on loan approvals, interest rates, and credit limits. Credit scoring present several characteristics that makes it a challenging task. First, the lack of explainability in complex machine learning models often results in less acceptance of credit assessments, particulary among stakeholders who require transparent decision-making process. This opacity can be an obstacle in the widespread adoption of advanced scoring techniques. Another significant challenge is the variability in data availability across countries and the often incomplete financial records of SME's which makes it difficult to develop universally applicable models.

In this thesis, we initially tackled the issue of explainability by employing state-ofthe-art techniques in Explainable Artificial Intelligence (XAI). We introduced a novel strategy that involved comparing the explanations generated by machine learning models with the criteria used by credit experts. This comparative analysis revealed a divergence between the model's reasoning and the expert's judgment, underscoring the necessity of incorporating expert criteria into the training phase of the model. The findings suggest that aligning machine-generated explanations with human expertise could be a pivotal step in enhancing the model's acceptance and trustworthiness.

Subsequently, we shifted our focus to address the challenge of sparse or incomplete financial data. We incorporated textual credit assessments into the credit scoring model using cutting-edge Natural Language Processing (NLP) techniques. Our results demonstrated that models trained with both financial data and textual credit assessments outperformed those relying solely on financial data. Moreover, we showed that our approach could effectively generate credit scores using only textual risk assessments, thereby offering a viable solution for scenarios where traditional financial metrics are unavailable or insufficient.

Tout d'abord, je souhaite exprimer ma profonde gratitude envers ma directrice de thèse, Prof. Maria Trocan, ma superviseuse, Dr. Natalia Diaz-Rodriguez. Leur soutien constant, leur encouragement et leur vaste expertise ont été des éléments clés tout au long de mon aventure doctorale. Les conseils judicieux de Prof. Trocan et l'orientation de Dr. Diaz-Rodriguez m'ont été d'un grand secours durant les étapes de recherche et de composition de ma thèse. Travailler sous la direction de deux experts a été une expérience exigeante : il m'a fallu naviguer et évaluer une variété de propositions et d'approches. Néanmoins, cette collaboration m'a été extrêmement bénéfique : elle m'a offert l'opportunité d'apprendre de différentes personnalités et de me familiariser avec diverses méthodes de travail et de recherche. Je tiens également à remercier Patricia Conde-Cespedes pour son expertise dans le domaine, qui a enrichi significativement mon travail de recherche.

Je tiens à exprimer ma sincère gratitude à Tinubu pour m'avoir offert la chance de me consacrer à un sujet qui non seulement a un impact socioéconomique significatif, mais qui constitue également un challenge stimulant en matière de recherche de solutions. Plus spécifiquement, mes remerciements vont à Thomas Frossard, le directeur du laboratoire où j'ai réalisé mon doctorat. Sa constante motivation, son aptitude à suggérer des idées novatrices et son soutien inébranlable ont été des piliers essentiels de mon parcours académique. Paper à Paper.

Enfin, et c'est le plus important, je souhaite adresser une mention spéciale à ceux qui constituent le socle de ma vie et la source d'énergie qui m'a permis de continuer à croire en moi-même dans les moments les plus difficiles : ma famille. À mon frère Ayman, qui a toujours été là, dans les bons comme dans les mauvais moments. À ma soeur Miriam, pour son soutien sans faille. Je suis fier de vous avoir comme frère et soeur. Pour mon père, une éternité ne suffirait pas pour exprimer toute ma gratitude pour ce qu'il a fait pour moi et ce qu'il m'a appris. Si je devais choisir une chose à retenir, ce serait les valeurs d'effort et de travail qu'il m'a inculquées. Et enfin, le plus difficile, à ma mère, qui restera toujours dans mon coeur. Son soutien inconditionnel et sa foi inébranlable en moi sont des forces sans lesquelles je ne serais pas la personne que je suis aujourd'hui. Je t'aime.

iii

Résumé

Au cours des dernières années, le déficit de financement du commerce a atteint le chiffre alarmant de 1 500 milliards de dollars, soulignant une crise croissante dans le commerce mondial. Ce déficit est particulièrement préjudiciable aux petites et moyennes entreprises (PME), qui éprouvent souvent des difficultés à accéder au financement du commerce. Les systèmes traditionnels d'évaluation du crédit, qui constituent l'épine dorsale du financement du commerce, ne sont pas toujours adaptés pour évaluer correctement la solvabilité des PME.

Le terme "credit scoring" désigne les méthodes et techniques utilisées pour évaluer la solvabilité des individus ou des entreprises. Le score généré est ensuite utilisé par les institutions financières pour prendre des décisions sur l'approbation des prêts, les taux d'intérêt et les limites de crédit. L'évaluation du crédit présente plusieurs caractéristiques qui en font une tâche difficile. Tout d'abord, le manque d'explicabilité des modèles complexes d'apprentissage automatique entraîne souvent une moindre acceptation des évaluations de crédit, en particulier parmi les parties prenantes qui exigent un processus décisionnel transparent. Cette opacité peut constituer un obstacle à l'adoption généralisée de techniques d'évaluation avancées. Un autre défi important est la variabilité de la disponibilité des données entre les pays et les dossiers financiers souvent incomplets des PME, ce qui rend difficile le développement de modèles universellement applicables.

Dans cette thèse, nous avons d'abord abordé la question de l'explicabilité en utilisant des techniques de pointe dans le domaine de l'intelligence artificielle explicable (XAI). Nous avons introduit une nouvelle stratégie consistant à comparer les explications générées par les modèles d'apprentissage automatique avec les critères utilisés par les experts en crédit. Cette analyse comparative a révélé une divergence entre le raisonnement du modèle et le jugement de l'expert, soulignant la nécessité d'incorporer les critères de l'expert dans la phase de formation du modèle. Les résultats suggèrent que l'alignement des explications générées par la machine sur l'expertise humaine pourrait être une étape cruciale dans l'amélioration de l'acceptation et de la fiabilité du modèle.

Par la suite, nous nous sommes concentrés sur le défi que représentent les données financières éparses ou incomplètes. Nous avons incorporé des évaluations de crédit textuelles dans le modèle d'évaluation du crédit en utilisant des techniques de pointe de traitement du langage naturel (NLP). Nos résultats ont démontré que les modèles formés à la fois avec des données financières et des évaluations de crédit textuelles étaient plus performants que ceux qui s'appuyaient uniquement sur des données financières. En outre, nous avons montré que notre approche pouvait effectivement générer des scores de crédit en utilisant uniquement des évaluations de risque textuelles, offrant ainsi une solution viable pour les scénarios dans lesquels les mesures financières traditionnelles ne sont pas disponibles ou insuffisantes.

Chapter 1 Introduction

The process of granting credit to lenders is undeniably one of the most essential business activities in the financial sector. This practice not only leads to the generation of significant profits for a multitude of entities, including banks, other financial institutions, and shareholders, but it also plays a huge role in contributing positively to the community.

While the advantages are numerous, this activity is also accompanied by substantial risks. One needs only to reflect on the recent financial crises that shook the global market to understand the magnitude of these risks. These crises led to devastating losses on a global scale, pushing banks and financial institutions into a state of heightened alertness. For example, the principal factor that led to the financial crisis of 2008 was the underestimation of mortgage risk of default [START_REF] Albanesi | Credit Growth and the Financial Crisis: A New Narrative[END_REF]. This increased attention became particularly focused on the credit risk models that these institutions employed. It has become abundantly clear, especially in the wake of these financial downturns, that banks have to be more discerning than ever. They recognize the imperative need to incorporate rigorous credit evaluation models into their systems. Such prudence is essential whether they're contemplating granting a loan to a single individual or to a large corporate entity.

Credit Scoring

Credit scoring serves as a crucial tool in the assessment of the risk associated with lending money. It involves the use of statistical models that incorporate various factors such as the borrower's credit history, current financial standing, and other relevant socio-economic indicators. The result is a numerical score, often referred to as a credit score, which provides a quantified assessment of the likelihood of the borrower defaulting on a loan.

Chapter 1. Introduction

In an increasingly interconnected and globalized financial environment, the necessity to assess the creditworthiness of individuals and corporations has never been more paramount. Digitalization has been a driving force behind the evolution and sophistication of credit scoring, fundamentally transforming the way credit risk is assessed and managed.

Leveraging the power of big data involves utilizing cost-effective and cutting-edge processing techniques. These methods enable users to gain deeper insights, which in turn bolsters decision-making and drives process automation. With the rise of digital technologies, vast amounts of data have become accessible [START_REF] Hasan | Current landscape and influence of big data on finance[END_REF]. This spans not only traditional credit information but also extends to unconventional digital footprints like online shopping behavior and social media activities.

Structured data is akin to an organized library of information, where each piece of data adheres to a specific, predefined format. To draw a clearer picture, imagine a meticulously managed database where each entry has distinct fields like company's name, address, and net worth. This arrangement makes structured data straightforward to search, access, and manage. Conversely, unstructured data is more akin to a vast ocean, teeming with a myriad of varied entities. The term "unstructured" implies that this kind of data doesn't conform to a standard format or structure. It encompasses a vast array of formats ranging from emails and images to audio recordings, videos, and even readings from various sensors. This diverse nature of unstructured data introduces layers of complexity. For example, the unstructured data utilized for credit scoring can encompass company news, consumer feedback on products, or even risk assessment documents crafted by credit risk specialists ( [START_REF] Jain | Leveraging unstructured data to improve customer engagement and revenue in financial institutions: A deep reinforcement learning approach to personalized transaction recommendations[END_REF], [START_REF] Muthukumar | A stochastic time series model for predicting financial trends using nlp[END_REF]).

Credit Scoring

The landscape of credit scoring has undergone a radical transformation in recent years, primarily driven by the dual forces of the widespread availability of data and significant leaps in machine learning techniques. Traditionally, credit scoring relied on a relatively limited set of parameters, most of which were directly linked to an individual's or entity's financial history and behavior [START_REF] Altman | FINANCIAL RATIOS, DISCRIMINANT ANALYSIS AND THE PREDICTION OF CORPORATE BANKRUPTCY[END_REF].

Classical models might have primarily focused on past loan payment records, current debts, income levels, and perhaps a few other factors. However, the digital revolution has brought about an era where vast quantities of data-ranging (e.g., social media activities) are easily accessible and can be harvested for insights. This data explosion, when viewed from the lens of credit scoring, offers a gold mine of nuanced information that could potentially reveal a lot more about an individual's credibility and their likelihood to repay a debt. However, having access to a large volume of data is only one part of the equation. To transform this data into actionable insights, there's a need for sophisticated analytical tools-and this is where the advancements in machine learning come into play a huge role. Machine learning, with its ability to discern patterns from seemingly unrelated data points, can take these vast datasets and process them to extract meaningful patterns conversely to traditional credit scoring models that are able to capture linear relantionships [START_REF] Beaver | Financial ratios as predictors of failure[END_REF].

More than just processing, machine learning models can learn and adapt over time, becoming increasingly accurate as they are fed more data. This dynamic nature of machine learning means that credit scoring models can now evolve in real-time, adapting to new financial behaviors and trends as they emerge. A credit scoring system that undergoes regular retraining can adjust to evolving economic scenarios, thereby mitigating significant losses.

In the complex domain of financial lending, the methodology deployed for credit scoring is significantly influenced by the nature of the counterpart, i.e., the specific entity seeking credit. This differentiation in counterparts mandates a diverse set of considerations, evaluation criteria, and bespoke models to accurately gauge creditworthiness. Given that each counterpart presents its own set of financial behaviors, variables, risk profiles, and borrowing histories, a comprehensive and tailored approach becomes essential for credit risk assessment. This section will systematically unpack the various credit scoring methodologies, each uniquely adapted based on the distinct characteristics of different counterparts, to elucidate the underlying complexities and specificities involved.

Individual Consumer Credit Scoring

Individual Consumer Credit Scoring is a system used to assess the creditworthiness of individual consumers, typically for personal financial products such as credit cards, mortgages, and loans. The important parts of individual consumer credit scoring include several factors relating to a person's management of money. The credit history is vital, as it records past payments and any failures to pay, giving an idea of past reliability. Next, the credit utilization ratio looks at existing debts compared to credit limits, showing how a person manages current debt. The time someone has had credit accounts, the various kinds of credit they have (like credit cards or loans), and any recent requests for new credit are also considered.

Company Credit Scoring

In the realm of corporate finance and investment, company scoring stands as a pivotal mechanism for assessing the financial health and creditworthiness of businesses. This evaluation serves as the bedrock upon which critical decisions such as loan approvals, investment allocations, and risk assessments are made ( [START_REF] Khemir | The influence of ESG information on investment allocation decisions[END_REF], [START_REF] Patel | Caste-based discrimination, microfinance credit scores, and microfinance loan approvals among females in india[END_REF]). However, it's imperative to recognize that not all companies are created equal, especially when it comes to their status as either listed or non-listed entities. This distinction has far-reaching implications for how these companies are scored, the data available for analysis, and the subsequent interpretations and applications of these scores.

Listed Companies

Listed companies, commonly known as publicly traded companies, are corporations that have successfully completed an Initial Public Offering (IPO) and have their shares available for trading on stock exchanges. These companies operate under a stringent set of regulatory requirements and are obligated to disclose a wide range of financial and operational information to both the public and regulatory authorities.

One of the defining characteristics of listed companies is their high level of transparency. They are required to publish financial reports on a quarterly and annual basis, which include comprehensive financial statements such as income statements, balance sheets, and cash flow statements. This transparency is further reinforced by the regulatory oversight they are subject to, often by bodies like the Securities and Exchange Chapter 1. Introduction Commission (SEC) in the United States or the Autorité des marchés financiers (AMF) in France.

Another key feature is their market capitalization, which provides a readily available measure of the company's value and is calculated by multiplying the stock price by the total number of outstanding shares. The ownership of these companies is usually distributed among a diverse set of shareholders, including institutional investors, retail investors, and company insiders. Additionally, the shares of listed companies are easily bought or sold on stock exchanges, providing a high level of liquidity. Being in the public eye also means that these companies are subject to market sentiment, news coverage, and the opinions of financial analysts.

When it comes to scoring listed companies, several unique implications arise. The abundant availability of financial data allows for a more nuanced and accurate scoring process. Financial metrics such as Price-to-Earnings (P/E) ratios and Earnings Before Interest and Taxes (EBIT) are readily accessible for analysis. Stock performance and market capitalization can also serve as supplementary indicators of a company's financial health and are often incorporated into scoring models. The stringent regulatory environment and mandatory financial audits add an extra layer of credibility to the financial statements, which is usually reflected positively in the credit score.

Investor sentiment, as gauged through stock price and trading volume, can also play a significant role [START_REF] Zhou | Measuring investor sentiment[END_REF]. While positive sentiment can enhance a company's perceived creditworthiness, negative sentiment can have the opposite effect [START_REF] Serafeim | Public sentiment and the price of corporate sustainability[END_REF].

The real-time nature of stock markets means that credit scores for listed companies are dynamic and require frequent updates to reflect current market conditions. The wealth of data often allows for the use of standardized scoring models like the Altman Z-score [START_REF] Altman | FINANCIAL RATIOS, DISCRIMINANT ANALYSIS AND THE PREDICTION OF CORPORATE BANKRUPTCY[END_REF] or Merton's model [START_REF] Merton | On the pricing of corporate debt: the risk structure of interest rates[END_REF], which are widely recognized and accepted in the financial industry.

The characteristics of listed companies, such as transparency, regulatory oversight, and market-driven data, contribute to a more robust and dynamic credit scoring process. However, this also introduces additional variables like market sentiment and stock performance, requiring a more complex and nuanced approach to credit scoring.

Non-listed companies

Non-listed companies-commonly referred to as privately-held or unlisted entities constitute a distinct category of business organizations that are not traded on public stock exchanges. This absence of public trading engenders a markedly different operational landscape compared to publicly listed corporations, which are subject to rigorous regulatory oversight and disclosure mandates. The limited amount of regulatory restrictions gives non-listed companies greater freedom in their operations, allowing them to be more flexible in their strategic planning.

Principal Company Credit Scoring Agencies

Ownership structures in non-listed companies are frequently characterized by a high degree of concentration, often vested in a limited cohort of stakeholders such as founding members, familial networks, or private equity consortia. This concentrated ownership paradigm expedites decision-making processes but concurrently engenders a liquidity constraint, as the shares are not readily tradable on public markets.

Unlike their publicly-listed counterparts, which can readily augment capital through the issuance of additional shares, non-listed entities predominantly rely on alternative financial instruments such as debt, venture capital, or private equity for capital infusion.

The absence of a publicly accessible trading platform further complicates the task of ascertaining an accurate market valuation, although it liberates these entities from the exigencies of quarterly financial reporting, thereby enabling a more long-term strategic focus.

The evaluation or 'scoring' of non-listed companies poses a unique set of challenges. The scarcity of publicly available financial information forces a dependence on self-reported data, which might not be as reliable or standardized as the financial statements that publicly-listed companies disclose. This lack of transparency in financial matters, combined with minimal regulatory oversight, complicates the process of assessing risk, often requiring an in-depth due diligence approach. The shares' lack of liquidity and the generally higher perceived risk often result in an increased cost of capital, which could negatively impact the company's competitive standing and growth prospects. On the other hand, the lesser regulatory constraints provide these companies with a level of operational flexibility that can be a strategic asset, especially in fast-changing industries.

Traditional credit scoring models might not be entirely suitable for evaluating these companies, so using alternative methods like machine learning algorithms based on unconventional data sources could yield more precise and nuanced evaluations.

Principal Company Credit Scoring Agencies

The task of appraising a corporation's creditworthiness is an intricate and specialized endeavor, predominantly executed by agencies that are expressly devoted to the formulation of corporate credit ratings. These entities utilize a set of methodologies that are markedly divergent from those applied in the assessment of individual creditworthiness. Unlike the latter, the methodologies for corporate credit evaluation are sophisticated. These specialized agencies scrutinize an array of variables, encompassing a firm's fiscal robustness, operational efficacy, and market standing, to formulate a credit rating that cogently articulates the corporation's aptitude for fulfilling its monetary commitments.

The evaluative frameworks employed by these agencies amalgamate an extensive spec-Chapter 1. Introduction trum of considerations, ranging from quantitative financial metrics and business risk profiles to qualitative assessments of managerial competence and sector-specific vicissitudes. In doing so, they endeavor to furnish investors, financial institutions, and other pertinent stakeholders with a comprehensive and nuanced understanding of the corporation's credit risk. Consequently, the role of these principal agencies in the realm of corporate credit scoring is not merely instrumental but also highly specialized, necessitating a distinct arsenal of analytical methodologies and evaluative criteria vis-à-vis those employed for individual credit assessments.

Standard & Poor's (S&P), one of the preeminent agencies in the domain of corporate credit ratings, employs a multifaceted and rigorous methodology to assess a corporation's creditworthiness. At the core of this evaluative framework are four principal dimensions.

First, the Business Risk Profile is scrutinized to gauge both the inherent risks associated with the industry in which the corporation operates and its competitive positioning within that sector. This involves a nuanced analysis of market dynamics, consumer demand, and competitive advantages or disadvantages.

Second, the Financial Risk Profile is meticulously assessed, focusing on key financial ratios such as debt-to-equity and liquidity ratios, as well as the corporation's cash flow patterns and overall financial stability. This provides a quantitative foundation for the credit rating, offering insights into the firm's fiscal health and its ability to meet short-term and long-term obligations.

Third, the Country Risk dimension takes into account the macroeconomic and geopolitical landscape of the nation in which the corporation conducts its business. Factors such as economic growth rates, political stability, and regulatory frameworks are considered to understand how they might impact the company's financial standing.

Lastly, the Management and Governance aspect is evaluated to ascertain the efficacy of the company's organizational structures, leadership quality, and governance protocols. This qualitative assessment aims to shed light on the strategic direction of the company and the competence of its management team in steering the organization towards financial stability and growth.

In addition to the methodologies employed by Standard & Poor's, it's worth noting that other credit rating agencies (i.e., Moody's and Fitch) also incorporate additional factors into their evaluative frameworks. Specifically, many agencies give weight to External Influences, which encompass a broad range of macroeconomic variables, industry-specific trends, and geopolitical risks. This aspect of the methodology aims to capture the broader contextual factors that could affect a corporation's financial stability. For instance, fluctuations in commodity prices, shifts in consumer preferences, or geopolitical tensions can all have a significant impact on a company's creditworthiness.

Another critical factor considered is Legal and Regulatory Risks. This involves a thorough examination of the potential impact of changes in laws and regulations on a com-1.5. Trade finance and the importance of credit scoring pany's operations and, consequently, its ability to meet financial obligations. Regulatory shifts, such as changes in environmental standards or trade tariffs, can have immediate and long-term effects on a company's profitability and cash flow, thereby affecting its credit rating.

One of the contemporary challenges confronting credit rating agencies is the integration of Environmental, Social, and Governance (ESG) factors into their traditional methodologies for assessing credit risk. As societal awareness and regulatory focus on sustainability and ethical governance intensify, the imperative to incorporate ESG criteria has become increasingly salient. These factors range from a company's environmental impact and carbon footprint to its labor practices, ethical conduct, and governance structures.

The challenge lies in quantifying these often qualitative factors and seamlessly incorporating them into existing evaluative frameworks that have historically prioritized financial metrics. The objective is to offer a more holistic view of a company's risk profile, one that accounts not only for financial stability but also for long-term sustainability and ethical considerations.

This evolution in methodology is not merely a response to changing investor preferences but is also a recognition of the long-term financial risks associated with poor ESG performance. Therefore, the integration of ESG ratings is becoming a critical dimension in the comprehensive assessment of a corporation's creditworthiness, marking a significant paradigm shift in the field of corporate credit risk assessment.

Trade finance and the importance of credit scoring

Trade finance frequently confronts a pronounced disparity between the demand for and supply of credit. This gap is especially acute in emerging markets and among small and medium-sized enterprises (SMEs), where access to trade finance is often constrained by various factors such as regulatory hurdles, lack of financial infrastructure, and perceived high risks from lenders perspectives [START_REF] Wto | Trade finance and SMEs Bridging the gaps in provision[END_REF].

Financial institutions may be reticent to extend credit due to a lack of reliable information on the creditworthiness of trading entities, particularly SMEs, which may not have an extensive credit history or may operate in jurisdictions that are perceived to be riskier.

Additionally, the complexities of cross-border transactions, including currency exchange risks and differing legal frameworks, further exacerbate the challenge of meeting the burgeoning demand for trade finance. This situation creates a self-perpetuating cycle where the absence of readily available trade finance restricts the ability of companies, especially SMEs, to engage in international trade, thereby limiting their growth prospects Figure 1.4: Simplified structure of a credit in trade finance. Source: WTO [START_REF] Wto | Trade finance and SMEs Bridging the gaps in provision[END_REF] and, in turn, making them less attractive candidates for trade finance.

As a concrete illustration of the challenges previously discussed, consider the plight of small and medium-sized enterprises (SMEs) in developing nations, where the obstacles to obtaining trade finance are particularly pronounced. In Africa, the estimated shortfall in trade finance is a staggering US$ 120 billion, which accounts for approximately onethird of the entire trade finance market across the continent. The situation is even more acute in developing regions of Asia, where the unmet demand for trade finance soars to an astronomical US$ 700 billion. These figures not only underscore the magnitude of the trade finance gap but also highlight the acute challenges faced by SMEs in these regions in accessing the financial resources needed to engage in international trade.

On the other hand, In certain large developed countries, as many as one-third of small and medium-sized enterprises (SMEs) grapple with difficulties in securing trade finance. To put this into perspective, SMEs are responsible for a significant portion of exports from these developed economies; they account for 20% of all U.S. exports and an even higher 40% of exports from the European Union. These statistics emphasize that the issue of limited access to trade finance is not confined to developing regions but is also a pertinent concern in developed economies, affecting a substantial number of SMEs that are key contributors to international trade. 1.6. Machine Learning and Deep Learning in Credit Scoring: Challenges and Impediments to Implementation

Machine Learning and Deep Learning in Credit Scoring: Challenges and Impediments to Implementation

Certainly, the advent of machine learning and deep learning technologies has undeniably revolutionized the field of credit scoring. However, the industry is still grappling with several challenges that require attention for further progress.

Limited access to financial data is a significant hurdle. Data privacy concerns have made many individuals reluctant to share their personal financial information, thereby limiting the amount of data available for building robust credit scoring models. Additionally, especially for new borrowers or those without a traditional credit history, the available financial data is often sparse, making it difficult to assess creditworthiness accurately.

Financial data is also often stored in isolated databases across different institutions, complicating the task of obtaining a comprehensive view of an individual's financial health. For international borrowers, the absence of a standardized global system for financial data can further complicate credit scoring.

Another challenge lies in the inability of classical models to capture complex patterns in financial data. Traditional models like logistic regression or linear models are often too simplistic to capture the intricate, non-linear relationships that exist.

While machine learning models are capable of capturing these complex patterns, they are also prone to overfitting, especially when the available data is limited or imbalanced. Classical models also often rely on a limited set of features, ignoring potentially important variables that could improve the model's predictive power.

Strong regulation and the need for model transparency also pose challenges. Financial institutions are subject to stringent regulations, such as the General Data Protection Regulation (GDPR) in Europe. GDPR imposes strict regulations on the collection, storage, and processing of personal data, including financial information used in credit scoring. This has led to heightened requirements for transparency and explainability in credit scoring algorithms. Financial institutions are now obligated to provide clear justifications for their credit decisions, which poses a challenge for complex machine learning and deep learning models [START_REF] Ali | Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence[END_REF].

Advanced machine learning and deep learning models are often criticized for being "black boxes," making it difficult to interpret their decisions and thereby posing a challenge in meeting regulatory requirements for transparency. There is also an increasing focus on ethical AI, demanding that credit scoring models be free from biases related to gender, race, or socio-economic status. Complying with these ethical standards while maintaining high accuracy is a significant challenge. Moreover, different countries have different regulations concerning credit scoring, making it challenging for financial institu-Figure 1.5: Illustration of the balance between accuracy and interpretability. Source: [START_REF] Ali | Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence[END_REF] tions that operate internationally to maintain a consistent and compliant approach.

Problem Statement

In previous sections, we have delved into the principal real-world stakes associated with credit scoring, highlighting its pivotal role in various economic and financial contexts. We mentioned the biggest challenges of implementing machine-learning based credit scoring models.

This thesis aims to tackle several specific challenges that were encountered by Tinubu2 when developing a credit scoring system. These challenges serve as real-world examples that highlight the complexities involved in implementing machine learning models for credit risk assessment.

One of the most volatile factors affecting credit scoring is the pace of economic changes. Economic conditions can shift rapidly, making previously reliable models obsolete or less accurate. This necessitates frequent updates and recalibrations to the credit scoring models to keep them relevant and reliable.

Another challenge is the lack of comprehensive financial features. In many cases, essential financial data may be missing or incomplete, making it difficult to assess creditworthiness accurately. This issue of missing values can severely impact the model's performance and may require sophisticated techniques for imputation or feature engineering.

In the field of credit scoring, it's common to encounter imbalanced datasets where the ratio of defaulted to non-defaulted cases is highly skewed. In Tinubu's experience, the default to non-default ratio was around approximately 1%, making it a significant challenge to train a model that can effectively identify the minority class of defaulted cases without being overwhelmed by the majority class leading the model to overfit.

Problem Statement

Lastly, one of the critical issues is that the credit scoring models often yield credit ratings without providing any explanations or justifications. This lack of interpretability can be a significant drawback, especially when financial institutions or borrowers seek to understand the factors contributing to a particular credit score.

Contribution and Structure of the Manuscript

In this thesis, we present two main contributions to the field of credit scoring, particularly focusing on the challenges and opportunities arising from the application of machine learning techniques. These contributions are structured across six different chapters, each designed to delve into specific aspects of the problem, methodologies, and solutions. The chapters are organized in a way that provides a comprehensive understanding of the current landscape, the challenges witnessed by industry experts (i.e., Tinubu), and the approaches we propose to address these issues.

In the first chapter of this thesis, we lay the groundwork by introducing the field of machine learning, which serves as the cornerstone for the entire manuscript. We delve into the essential concepts, terminologies, and methodologies that are fundamental to understanding the subsequent chapters.

In the second chapter of this thesis, we exmine into the impact of the COVID-19 crisis on the default rates of companies, with a specific focus on those based in France and operating in various economic sectors. Through a comprehensive analysis, we examine how the pandemic has influenced the financial stability of businesses, leading to changes in their default percentages. This investigation not only provides a nuanced understanding of the crisis's economic repercussions but also serves as a foundational study for developing more resilient and adaptive credit scoring models that can better handle such unprecedented events.

Development of an explainable credit scoring model

In the third chapter of this thesis, we focus on a comparative analysis between Tinubu's existing automatic credit rating system and a machine learning-based model that we develop. Both systems aim to assess the risk that a borrower will default, but they do so in different ways. Tinubu's current system provides a qualitative output, offering a risk assessment in a more descriptive manner. In contrast, our machine learning-based model uses the same input variables but aims to provide both a quantitative risk assessment and explanations for its predictions. One of the key aspects of this chapter is the comparison of explanations generated by the machine learning model with the criteria used by risk analysts. By doing so, we aim to identify any divergences between the machine learning model's rationale and the human experts' judgments. This comparative approach serves to validate the effectiveness and interpretability of the machine learning model while also highlighting areas for potential improvement or alignment with human expertise.

Chapter 1. Introduction

In the fourth chapter, we take a different approach compared to the previous chapters by expanding our focus to multiple industries, each with its own unique set of challenges and characteristics. Unlike the previous chapter, which concentrated on a limited set of financial features, this chapter explores a broader feature space that encompasses variables relevant to various industries. One of the key findings in this chapter is the divergence in terms of explanations across different industries. We analyze how the machine learning model's rationale for credit risk assessment varies depending on the industry in question, providing valuable insights into the adaptability and limitations of the model in diverse settings. Additionally, we demonstrate the positive impact of this broader approach on model performance. By incorporating a more extensive set of features and considering industry-specific nuances, we show that the machine learning model's predictive accuracy improves, thereby confirming the utility of a multi-industry approach in credit scoring.

In the fifth chapter, we shift our focus from predicting the likelihood of a company going into default to predicting the future credit rating of the company. This represents a significant departure from the objectives of the previous chapters. Here, we treat each company as a time series, utilizing their historical ratings as input features to forecast future ratings. This time-series approach is particularly effective for companies that have a long track record of credit ratings, as it allows the model to capture temporal patterns and trends that may not be evident in a cross-sectional analysis. By focusing on predicting future ratings rather than default likelihood, we aim to provide a more nuanced understanding of a company's financial health over time. The results indicate that this time-series approach is highly effective for a subset of companies with an extensive history of ratings, thereby offering a new dimension to credit risk assessment that complements the more traditional methods explored in earlier chapters.

Multimodal credit risk scoring

In the sixth and final chapter, we venture into the realm of incorporating textual data into the development of credit scoring models, with the ambitious goal of potentially substituting traditional financial features with textual information. Initially, we demonstrate that by adding human-generated credit reports to the existing set of financial features, the model's capability to accurately identify companies that will go into default is significantly enhanced. This suggests that textual data, often rich in qualitative insights, adds a layer of depth to the model that numerical financial features alone may not capture. Furthermore, we make a groundbreaking revelation: it is possible to identify highly risky companies, those with poor ratings according to Tinubu's expertise, without relying on financial features at all. By solely utilizing textual data, our model is still able to flag companies that are at high risk of default or financial instability. This finding has profound implications for the field of credit scoring, especially in scenarios where financial data may be incomplete, unreliable, or unavailable. It opens up new avenues for research and application, underscoring the untapped potential of textual information in enhancing the robustness and versatility of credit scoring models.

Chapter 2

Related Work

In this particular chapter, we take a comprehensive approach to formally define the problem of credit scoring through mathematical formulations. Our aim is to provide a rigorous foundation that sets the stage for the empirical work that follows. Alongside this, we also delve into an extensive review of the existing literature and the current state of the art in several interconnected disciplines. These include not only credit scoring but also machine learning, explainable artificial intelligence, and natural language processing. This multi-disciplinary review serves to contextualize our work within the larger academic and industry landscapes.

Machine Learning Terminology and Specifics of Credit Scoring

When defining a credit scoring model, there are two distinct approaches based on different underlying philosophies: Point-in-Time (PIT) and Through-the-Cycle (TTC) [START_REF] Rikkers | The influence of rating philosophy on regulatory capital and procyclicality[END_REF]. The PIT approach focuses on assessing the credit risk of a borrower at a specific moment in time, taking into account both cyclical and structural factors. This means that the PIT model is sensitive to current economic conditions and can fluctuate based on short-term changes in the borrower's financial situation or the broader economy. On the other hand, the TTC approach aims to evaluate the credit risk of a borrower over an entire economic cycle. It smooths out short-term fluctuations and focuses on the borrower's long-term ability to meet financial obligations, irrespective of current economic conditions. While PIT models are generally more accurate for immediate risk assessment and decisionmaking, TTC models are often used for strategic planning and capital allocation, as they provide a more stable and long-term view of credit risk

In light of these considerations, our work will primarily focus on Point-in-Time (PIT) approaches to credit scoring. This decision is motivated by the fact that companies Chapter 2. Related Work typically seek short-term credits, making the immediate and current assessment of credit risk particularly relevant. The PIT model's sensitivity to current economic conditions and short-term changes in a company's financial status makes it a more suitable framework for evaluating the types of credit commonly demanded by businesses In this section, the focus shifts to a mathematical formalization of machine learning and deep learning tasks as they pertain to credit scoring. The aim of this section is to give a thorough but easy-to-understand foundation on how to define the credit scoring problem and to translate the business problem into a machine-learning task.

The issue of credit scoring can be rigorously defined in mathematical terms, typically falling under the category of supervised learning. In the realm of machine learning, supervised learning tasks are those where the model is trained on a dataset that includes both the input variables and the corresponding output labels. More specifically, credit scoring is most commonly approached as a classification problem, although there are instances where it can also be treated as a regression problem.

In a classification framework, the aim is to categorize borrowers into distinct classes, such as 'low risk' or 'high risk,' based on their financial attributes. On the other hand, in a regression framework, the objective might be to predict a continuous outcome, such as the probability of default. To encapsulate these concepts and provide a structured approach to the problem, a simplified mathematical model is often employed.

Credit Scoring formalization

In the context of this thesis, it is imperative to establish a clear and consistent set of variable definitions that will be employed throughout the entire work. These variables serve as the foundational elements for constructing and evaluating machine learning models aimed at predicting credit risk.

• X : This is known as the feature space. It represents all the possible financial characteristics or attributes that company might have. Examples of these features include net worth, country in which the company operates, and sales. Each company will have a unique combination of these features, which will be used to assess their creditworthiness.

• Y: This is the label space, which is usually a binary set {0, 1}. Since the objective is to predict whether a company will go into a default the year following the financial statemtents publication, the label 1 is used to indicate that a company goes into default the year frollowing the assessment, while the number 0 signifies that the company does not go into default. These labels serve as the outcomes that the machine learning model aims to predict based on the features.

• (x i , y i ): This represents a single labeled example in the dataset. Here, x i is a feature vector that belongs to the feature space X and contains the financial attributes for 2.1. Machine Learning Terminology and Specifics of Credit Scoring the i-th borrower. y i is the corresponding label for that borrower and belongs to the label space Y. The feature vector and label together provide a complete snapshot of the i-th borrower's financial situation and credit risk level.

• D = {(x 1 , y 1 ), (x 2 , y 2 ), . . . , (x n , y n )}: This is the dataset, which is a collection of n labeled examples. Each labeled example consists of a feature vector and a corresponding label, as described above. The machine learning model uses this dataset to learn the relationships between the features and labels, with the aim of making accurate credit risk predictions for new, unseen borrowers.

The formal objective of a machine learning-based credit scoring model is to construct a predictive function f : X → Y.

The function f aims to map each borrower's feature vector x ∈ X to a corresponding label y ∈ Y, such that the prediction f (x) approximates the true label y as closely as possible. In the context of credit scoring, the problem can be simplified to a classification task where the objective is to estimate the conditional probability p(y | x). However, this estimation can also be interpreted as a regression problem. Specifically, the goal is to find a function f (x) that serves as the best approximation for F (x), which is the true but unknown function governing the relationship between the features x and the labels y.

Bias-Variance Trade-off

However, real-world scenarios introduce a considerable amount of noise, which can be attributed to factors such as missing financial features, incorrect labeling, and so on. We define as the irreducible error that we cannot recover from, such that the relationship between the features x and the labels y can be expressed as y = F (x) + .

One fundamental concept in machine learning and statistics is the bias-variance tradeoff. It describes two sources of errors that affect the performance of predictive models. The bias represents the error to overly simplistic assumptions in the learning algorithm. For instance, high bias can cause the model to miss relevant relations between features and target outputs leading to what is called underfitting. On the othre hand, there is the error caused by a learning algorithm with a significant level of complexity. In this case, models with high variance suffer from overfitting (i.e., the model learns the random noise from the training data wich leads to poor performances in unseen data).

The expected test error E[Error] can be decomposed as:

E[Error] = Bias 2 + Variance + Where: Chapter 2. Related Work Bias 2 ( f ) = E[ f (x)] -f (x) 2 Variance( f ) = E[ f (x) -E[ f (x)]] 2
The trade-off involves minimizing E[Error] by balancing Bias and Variance (see Fig. This loss function quantifies the discrepancy between the predicted labels and the true labels. The ultimate goal is to minimize L while ensuring that the model generalizes well to new, unseen data. 20

Standard models and its application in credit scoring

Loss Functions

We discuss various loss functions commonly used in machine learning:

• Logistic Loss: log(f (x), y) = log(1 + e -yf (x) ), often used in neural networks and logistic regression [START_REF] Bahnsen | Example-dependent cost-sensitive logistic regression for credit scoring[END_REF], measures the difference between the predicted probabilities and the actual labels. It is both differentiable and convex, making it well-suited for optimization algorithms.

• Squared Error: se(f (x), y) = (y -f (x)) 2 , the standard loss for regression tasks.

In this manuscript, we use it in Gradient Boosting [START_REF] Chen | XGBoost: A scalable tree boosting system[END_REF] for classification tasks.

• Hinge Loss: hinge(f (x), y) = max(0, 1 -yf (x)), primarly used in Support Vector Machines (SVM) [START_REF] Lee | A study on l2-loss (squared hinge-loss) multiclass svm[END_REF], [START_REF] Xu | Robust support vector machines based on the rescaled hinge loss function[END_REF]. This loss is zero when y and f (x) agree and linearly increases when they disagree.

• Exponential Loss: exp(f (x), y) = e -yf (x) , used in Adaboost [START_REF] Hastie | Multi-class adaboost[END_REF], [START_REF] Chengsheng | Adaboost typical algorithm and its application research[END_REF] . This loss is challenging to optimize due to its exponential nature. The choice of a loss function plays a critical role in determining the accuracy of a predictive model. Different loss functions have varying sensitivities to outliers and model complexity, which can significantly impact the model's performance [START_REF] Muthukumar | Classification vs regression in overparameterized regimes: Does the loss function matter?[END_REF].

Standard models and its application in credit scoring

Chapter 2. Related Work

The Altman Z-Score model holds the distinction of being one of the first credit scoring models to gain widespread acceptance and usage. Developed by Edward I. Altman in [START_REF] Altman | FINANCIAL RATIOS, DISCRIMINANT ANALYSIS AND THE PREDICTION OF CORPORATE BANKRUPTCY[END_REF], this model was initially designed to predict the likelihood of a publicly traded manufacturing company going bankrupt within a two-year period. The model employs multiple financial ratios that are derived from a company's balance sheet and income statement, such as liquidity, profitability, and leverage ratios.

Linear Models

During the 1980s and 1990s, statistical methods such as Logistic Regression( [START_REF] Bolton | Logistic regression and its application in credit scoring[END_REF], [START_REF] Steenackers | A credit scoring model for personal loans[END_REF], [START_REF] Bensic | Modelling small-business credit scoring by using logistic regression, neural networks and decision trees[END_REF]) and Linear Discriminant Analysis (LDA) ( [START_REF] Eisenbeis | Problems in applying discriminant analysis in credit scoring models[END_REF], [START_REF] Mylonakis | Evaluating the likelihood of using linear discriminant analysis as a commercial bank card owners credit scoring model[END_REF]) gained prominence in the credit scoring field. These models brought greater flexibility in accommodating various types of data and quickly became popular choices for consumer credit scoring. Logistic Regression, in particular, emerged as the industry standard for an extended period, largely due to its capability to offer probability estimates of default or non-default.

Mathematically, Logistic Regression models the log-odds of the probability P (y = 1|x) as a linear combination of the features x:

log P (y = 1|x) 1 -P (y = 1|x) = β 0 + β 1 x 1 + β 2 x 2 + . . . + β n x n
The probability P (y = 1|x) can then be obtained by transforming the log-odds back:

P (y = 1|x) = 1 1 + e -(β 0 +β 1 x 1 +β 2 x 2 +...+βnxn)
On the other hand, Linear Discriminant Analysis (LDA) aims to find a linear combination of features that best separates two or more classes. The discriminant function for LDA is given by:

D(x) = w T x + w 0
Here, w is the weight vector, and w 0 is the bias term. The class label y is then determined based on the sign of D(x).

Regularized Linear Models such as Ridge [START_REF] Hoerl | Ridge regression: Biased estimation for nonorthogonal problems[END_REF] and Lasso [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] Regression have emerged as powerful alternatives to traditional methods like Logistic Regression and Linear Discriminant Analysis. Specifically, these regularized models excel in managing multicollinearity among features, a challenge that can compromise the performance of Logistic Regression and LDA [START_REF] Salehi | The impact of regularization on highdimensional logistic regression[END_REF]. By introducing regularization terms, Ridge and Lasso effectively shrink the coefficients of less important features, thereby preventing overfitting.

Decision Trees

Decision trees are a type of supervised learning algorithm predominantly used for classification problems (e.g, [START_REF] Charbuty | Classification based on decision tree algorithm for machine learning[END_REF]), but can also be employed for regression tasks (e.g., [START_REF] Czajkowski | The role of decision tree representation in regression problems-an evolutionary perspective[END_REF]).

Standard models and its application in credit scoring

Mathematically, a decision tree partitions the feature space into disjoint regions (see Fig. 2.3). For classification tasks, each leaf node in the tree represents a class label, while for regression tasks, it represents a real value. The feature space is divided based on certain criteria that aim to maximize the information gain or minimize impurity. Two of the most commonly used criteria are Gini impurity and entropy.

The mathematical formulation for Gini impurity for a node t is given by:

Gini(t) = 1 - c i=1 p 2 i
Where p i is the proportion of samples that belong to class c for the node t.

Entropy for a node t is calculated as:

Entropy(t) = - c i=1 p i log 2 (p i )
Information Gain, which is the reduction in entropy or Gini impurity, is then used to decide which feature to split on at each step. The feature with the highest information gain is chosen for the split.

Information Gain = Entropy(t) - child nodes |D v | |D| × Entropy(v)
Where D is the dataset, D v is the subset of D at child node v. In the context of credit scoring, decision trees offer several advantages. They are interpretable, which aligns well with the regulatory requirements in financial sectors. They can handle both categorical and numerical features, making them versatile for different types of financial data [START_REF] Kingsford | What are decision trees?[END_REF]. Moreover, decision trees can capture non-linear relationships in the data, which is often the case in credit risk assessment.

Chapter 2. Related Work However, a single decision tree is often prone to overfitting [START_REF] Kingsford | What are decision trees?[END_REF], especially when the tree is deep. This is why ensemble methods like Random Forests [START_REF] Breiman | Random forests[END_REF], which aggregate the predictions of multiple decision trees, are often used in credit scoring to improve the model's generalization ability. In [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF], it was demonstrated that ensemble models exhibit greater predictive power compared to standalone decision trees when applied to credit scoring tasks.

Regulatory constraints (e.g., [START_REF] Laurent Dupont | Gouvernance des algorithmes d'intelligence artificielle dans le secteur financier[END_REF]) often limit the use of advanced scoring methods for traditional credit products. Despite their higher predictive accuracy, complex models face adoption challenges. Simpler, more interpretable models like logistic regression or decision trees are thus more commonly used in the industry due to their regulatory compliance.

Support Vector Machines

Support Vector Machines [START_REF] Baesens | Benchmarking state-of-the-art classification algorithms for credit scoring[END_REF], commonly known as SVMs, serve as a category of supervised machine learning models applicable to both classification and regression problems. From a mathematical standpoint, the core aim of an SVM is to identify the optimal hyperplane that most effectively segregates data into distinct classes. While in a twodimensional setting this separating hyperplane manifests as a simple line, in spaces with higher dimensions it takes the form of a more complex plane or even a multi-dimensional hyperplane.

The equation of the hyperplane is given by:

w • x -b = 0
Where w is the weight vector, x is the input vector, and b is the bias term.

The objective is to maximize the margin, which is the distance between the closest points (support vectors) of the different classes to the hyperplane. Mathematically, the margin M is calculated as:

M = 2 w
The optimization problem can be formulated as:

Minimize c 1 2 w 2 Subject to y i ( w • x i -b) ≥ 1, i = 1, . . . , n
Here, y i are the labels, and x i are the data points.

SVMs come with multiple benefits. They perform well in situations where the data has many variables, a common scenario in financial data that includes diverse factors like income levels, credit history, and job status. Additionally, SVMs can handle complex, non-linear relationships between these variables by employing specialized mathematical functions known as kernels, such as the Radial Basis Function (RBF), polynomial, or 2.3. Ensemble models in credit scoring sigmoid kernels. The authors in [START_REF] Al-Mejibli | Performance evaluation of kernels in support vector machine[END_REF] emphasize the significant impact that choosing the right kernel has on the overall performance of the model. SVM combined with feature selection techniques have shown promising results in credit scoring tasks. This approach enhances model performance by focusing on relevant financial variables. The outcome is a more accurate and efficient credit risk assessment [START_REF] Chen | Combination of feature selection approaches with SVM in credit scoring[END_REF]. Furthermore in [START_REF] Han | Orthogonal support vector machine for credit scoring[END_REF], the authors concluded that using logistic regression to filter dummy variables and orthogonal feature extraction improved the performance of Support Vector Machines. This approach not only simplified the model and sped up convergence but also yielded superior result However, one of the challenges with SVMs in credit scoring is interpretability. Unlike decision trees or logistic regression models, SVMs are often considered as "black-box" models, making them less transparent. This could be a concern in financial sectors where interpretability is crucial for regulatory compliance. In recent works, the authors have addressed this concrete problem by analyzing the weights of a svm model [START_REF] Shirataki | A study on interpretability of decision of machine learning[END_REF].

Ensemble models in credit scoring

In the previous section, we highlighted that decision trees often struggle with managing the bias-variance trade-off effectively. This is a crucial aspect, especially in the context of credit scoring, where both underfitting and overfitting can lead to significant financial implications. Interestingly, in [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF] the authors conducted a comparative study found that ensemble models are far more adept at handling the bias-variance trade-off in credit scoring scenarios. Given this backdrop, this section is dedicated to introducing ensemble learning models and elucidating their growing importance in modern machine learningbased credit scoring systems.

The core principle is simple yet effective: by combining the predictions of multiple models, we can often achieve greater accuracy than relying on a single model. We will specifically focus on two widely-used ensemble learning techniques: bagging and boosting.

Bagging

In ensemble methods like bagging [START_REF] Breiman | Bagging predictors[END_REF], the core idea is to amalgamate the predictions from multiple classifiers to arrive at a more accurate final prediction. What sets bagging apart is its unique approach to training each classifier on a distinct subset of the original data set, denoted as X , which is a proper subset of X . This is achieved through a technique known as 'sampling with replacement,' where data points are randomly selected from SS and then returned, allowing for the possibility of multiple selections of the same data point.

Chapter 2. Related Work

One of the advantages of decision trees is the ease of training but it comes with an important inconvenient: overfitting (see section 2.2).Bagging addresses this issue by introducing an element of randomness through its sampling technique. This randomness foment diversity among the individual trees, making the ensemble model more resilient to overfitting, even if each constituent tree is highly variable. The final bagged model is thus more robust and less prone to overfitting, offering a balanced approach to tree-based classification (e.g., [START_REF] Zhang | Vertical bagging decision trees model for credit scoring[END_REF]) or regression tasks (e.g., [START_REF] Branco | Rebagg: Resampled bagging for imbalanced regression[END_REF]).

Introduced in [START_REF] Breiman | Random forests[END_REF], Random forest is the most widely adopted bagging method. Employing the same variable definiton presented in section 2.1 and defining variables inherent to Random Forest, the algorithm can be formalized as follows:

• T : Number of trees in the forest.

• m: Number of features selected at each split.

1. Bootstrap Sampling: For each tree t, draw a bootstrap sample D t of size n from D with replacement.

Tree Building:

For each bootstrap sample D t , grow a decision tree f t (x) as follows:

• At each node, randomly select m features without replacement.

• Split the node using the feature that provides the best split according to some criterion (see section 2.3).

• Recur for the child nodes until a stopping criterion is met (e.g., maximum depth, minimum samples at leaf).

Ensemble Prediction:

The Random Forest F (x) makes a prediction by averaging the predictions of all individual trees for regression or by majority voting for classification:

F (x) = 1 T T t=1 f t (x) (Regression) F (x) = mode{f 1 (x), f 2 (x), . . . , f T (x)} (Classification)
Various studies have leveraged the advantages offered by the random forest algorithm. In [START_REF] Zhang | A novel credit scoring model based on optimized random forest[END_REF], the authors have introduced a novel credit scoring model based in random forest and feature selection. This random forest based model improves the accuracy compared to different state-of-the-art machine learning models. In a comparative study of a large list of machine learning models, the authors [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF] found that random forest is one of the most performant models.

Boosting

The idea of boosting was first articulated by [START_REF] Freund | Experiments with a new boosting algorithm[END_REF], showing that one could theoretically improve the efficacy of any learning model by amalgamating an array of weak classifiers. This theoretical underpinning has subsequently catalyzed the creation of a range of boosting algorithms.

Boosting is a sophisticated family of ensemble learning algorithms designed to enhance the predictive capabilities of weak or base learners. The fundamental principle that underpins boosting is the iterative training of a series of weak learners, where each learner in the sequence is specifically trained to correct the errors or misclassifications made by its predecessor. This iterative correction process allows the algorithm to focus on the more challenging or nuanced aspects of the data, thereby improving the model's overall performance. Once this iterative training is complete, the algorithm combines the outputs of all the weak learners to construct a single, more robust and accurate, strong learner [START_REF] Freund | Experiments with a new boosting algorithm[END_REF].

In mathematical terms, the boosting algorithm can be formalized as follows. Let F (x) represent the final strong learner we aim to construct. This strong learner is essentially a weighted sum of T weak learners f t (x), each associated with a weight α t . Mathematically, this can be expressed as:

F (x) = T t=1 α t f t (x)
Here, T is the total number of iterations or weak learners in the ensemble, f t (x) represents the t th weak learner, and α t is the weight assigned to this weak learner. These weights α t are computed based on the performance of each weak learner, effectively serving as a measure of its contribution to the final strong learner. The better a weak learner is at making accurate predictions, the higher its corresponding weight α t will be.

Adaptive Boosting (AdaBoost) is one of the pioneering algorithms in the field of boosting and has gained widespread popularity since its introduction in [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF]. As one of the first algorithms to successfully demonstrate the power of boosting techniques, it has been extensively used in various applications, including credit scoring [START_REF] Wang | A comparative assessment of ensemble learning for credit scoring[END_REF], healthcare [START_REF] Hatwell | Ada-whips: explaining adaboost classification with applications in the health sciences[END_REF], and sentiment analysis [START_REF] Xiao | Multi-view adaboost for multilingual subjectivity analysis[END_REF].

The AdaBoost algorithm starts by assigning equal weights to all the training samples. These weights are then adjusted at each iteration to give more importance to the samples that were misclassified by the previous weak learner.

Mathematically, the Adaboost algorithm is constructed as follows:

1. Initialize Weights: Initially, each sample i is given an equal weight w i = 1 n .

Chapter 2. Related Work

Iterative Training:

For t = 1, 2, . . . , T (where T is the total number of iterations or weak learners):

• Train a weak learner f t (x) using the weighted samples.

• Calculate the weighted error t of f t (x) as the sum of the weights of the misclassified samples.

• Update the weight α t for the weak learner f t (x) as

α t = 1 2 ln 1-t t .
• Update the sample weights w i to give more importance to the misclassified samples.

Final Model:

The final strong learner F (x) is a weighted sum of the weak learners:

F (x) = T t=1 α t f t (x)
The sign of F (x) will give the final classification.

Another family of boosting methods is Gradient Boosting [START_REF] Friedman | Greedy function approximation: a gradient boosting machine[END_REF], which aims to construct a robust predictive model by combining multiple weak or base learners. Unlike AdaBoost, which adjusts the weights of individual samples to focus on difficult-to-classify instances, Gradient Boosting concentrates on the errors or residuals made by the preceding models in the ensemble. The algorithm iteratively fits new models to these residuals, thereby correcting the mistakes of the existing ensemble of models.

In Gradient Boosting, several key characteristics distinguish it from other ensemble methods like AdaBoost, and these can be mathematically formalized.

First, the concept of "Residual Fitting" is central to the algorithm. In this approach, each new weak learner is trained to fit the residuals r i = y i -F m-1 (x i ) of the previous ensemble model F m-1 (x) at each iteration m. This allows the model to focus on the errors made by the preceding learners.

Second, Gradient Boosting optimizes a differentiable loss function L(y, F (x)), offering more flexibility in model optimization compared to AdaBoost's sample re-weighting technique. The loss function is minimized through gradient descent, and the weak learners are fit to the negative gradients.

Third, the algorithm incorporates a "Learning Rate" η to control the contributions of each weak learner, thereby adding a regularization component to the model. Mathematically, the ensemble model at each iteration m is updated as

F m (x) = F m-1 (x) + η • f m (x),
where f m (x) is the weak learner at iteration m.

Lastly, Gradient Boosting is highly versatile; it can be applied to both classification and regression tasks and can accommodate a wide range of loss functions. Gradient Boosting has been widely applied across various domains, demonstrating its versatility and robustness. In the field of healthcare, [START_REF] Zhang | Predictive analytics with gradient boosting in clinical medicine[END_REF] gradient boosting has been employed in 2.4. eXplainable Artifical Intelligence clinical. In finance, [START_REF] Bussmann | Explainable machine learning in credit risk management[END_REF] utilized the algorithm for credit risk assessment, highlighting its superiority over traditional methods. In natural language processing, the technique has been used for detecting fake news [START_REF] Reddy | A novel approach to detect fake news using extreme gradient boosting[END_REF].

eXplainable Artifical Intelligence

Explainable Artificial Intelligence (XAI) has emerged as a critical subfield of machine learning, particularly in contexts where understanding the decision-making process of algorithms is not just beneficial but essential. The objective is to bridge the gap between the performance of a model and the human understanding of how that model arrives at a particular decision.

The emergence of the field of Explainable Artificial Intelligence (XAI) can be traced back to the quest for increasingly accurate machine learning models. As models become more sophisticated to capture complex patterns and nuances in data, they inherently become less understandable to humans [START_REF] Dosilovic | Explainable artificial intelligence: A survey[END_REF]. Recent research endeavors have focused on establishing a comprehensive taxonomy for explainable models within the realm of artificial intelligence [START_REF] Arrieta | Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai[END_REF]. First concept that needs to be clear when dealing with XAI is the definition of explainability and interpretability. Explainability refers to the extent to which the internal workings of a machine learning or artificial intelligence system can be revealed or understood. An explainable model provides insights into its decision-making process, often Chapter 2. Related Work through visual or textual explanations, so that a human can understand why a particular decision or prediction has been made [START_REF] Gunning | Xai-explainable artificial intelligence[END_REF].

On the othre hand, interpretability is the degree to which a human can understand the cause-and-effect relationship between variables in a machine learning model. An interpretable model is one where the internal mechanics are transparent enough that they can be easily scrutinized to understand how input variables are transformed into an output decision. Interpretability is often a desired feature in models where safety or critical decision-making is involved, as it allows for easier debugging and trust-building (e.g., medicine applications [START_REF] Shaban-Nejad | Explainability and interpretability: keys to deep medicine[END_REF]).

Models such as decision trees, logistic regression, and linear discriminant analysis are inherently interpretable. This means that their internal workings are transparent, allowing for a straightforward understanding of how input features relate to output predictions (see Fig. 2.4).

Due to their increasing complexity, ensemble models often require post-hoc explanations to make their predictions understandable and actionable. As explained in previous sections, ensemble methods combine multiple base learners, making the decision-making process intricate and less transparent. Post-hoc models come into play here as separate explanatory models that are applied after the original model has made its predictions. These post-hoc models aim to approximate the complex model's behavior in a more interpretable form, shedding light on the important features and decision paths. For example in [START_REF] Demajo | Explainable ai for interpretable credit scoring[END_REF], the authors provides a series of different type of explanations of a gradient boosting based model.

On of the most widely used post-hoc models for explaining the output of any machine learning model is SHapley Additive exPlanations (SHAP) [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF]. It employs game theory to fairly allocate contributions of each feature for a particular prediction. Several. Despite its widespread use and versatility, SHAP is not without its drawbacks. One notable limitation is the interpretation of the SHAP scores themselves. While these scores offer a ranking of feature importance, the actual numerical values can be misleading, requiring the end user to focus more on the order of the features rather than the scores. Another issue is the assumption of feature independence, which may not hold true in many real-world scenarios, potentially leading to inaccurate or misleading explanations [START_REF] Salih | Commentary on explainable artificial intelligence methods: Shap and lime[END_REF]. Additionally, SHAP can be computationally expensive, especially for complex models and large datasets, making it less feasible for real-time or resource-constrained applications. Recent works have focused in accelerating shap value computation for tree-based models [START_REF] Yang | Fast treeshap: Accelerating shap value computation for trees[END_REF].

In different research papers, the SHAP has been employed for understanding credit score. In [START_REF] Bussmann | Explainable AI in Credit Risk Management[END_REF], they used SHAP to find out which factors, like income or debt, are most important for a company's credit score. In [START_REF] Hjelkrem | Explaining deep learning models for credit scoring with shap: A case study using open banking data[END_REF], the authors focused on customer credit scoring The study went beyond merely identifying important features; it also generated local explanations for individual loan decisions.

Natural Language Processing

Natural Language Processing

The rapid advancements in the field of Natural Language Processing (NLP) have not only revolutionized various sectors but have also caught the attention of the finance industry. In this section, we will specifically focus on how these NLP methodologies have been increasingly integrated into the finance sector, including their role in credit scoring, risk assessment, and financial analytics.

One of the first NLP models to be adopted is the Bag-of-Words (BoW) [START_REF] Zhang | Understanding bag-of-words model: a statistical framework[END_REF]. The model represents text data as a 'bag' or collection of individual words, disregarding grammar and word order. It converts text into numerical vectors by counting the frequency of each word. It has been used for different application like, text classification [START_REF] Hacohen-Kerner | The influence of preprocessing on text classification using a bag-of-words representation[END_REF], and spam filtering [START_REF] Cormack | Spam filtering for short messages[END_REF].

Term Frequency-Inverse Document Frequency (TF-IDF) wich is an extension of BoW it not only counts the frequency of each word (like BoW) but also weights it based on its importance in the document relative to a collection of documents (corpus). It has been widely used for information retrieval [START_REF] Thakkar | Predicting stock trend using an integrated term frequency-inverse document frequency-based feature weight matrix with neural networks[END_REF]. The latter work applied directly the information retrieved from the document to perform stock trend. Before the advent of Transformer models, the most widely used family of models for tasks such as machine translation were Word Embeddings. These models, including popular variants like Word2Vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF] and GloVe [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF], map words into a continuous vector space. They capture semantic meanings and relationships between words based on their co-occurrence in text. An intersting application of Word2Vec has been developed in [START_REF] Xue | Stock market reactions to the covid-19 pandemic: The moderating role of corporate big data strategies based on word2vec[END_REF]. The aurhors used this algorithm to identify stock market reactions to the COVID-19 pandemic.

In recent years, Transformer models [START_REF] Vaswani | Attention is all you need[END_REF] have completely revolutionized the domain of Natural Language Processing (NLP). These groundbreaking architectures (see Fig. 2.5) have set new benchmarks and reshaped the landscape of what's possible in various NLP tasks. Bidirectional Encoder Representations from Transformers (BERT) [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF] a pretrained transformer has been widely used in finance. In [START_REF] Chen | Stock movement prediction with financial news using contextualized embedding from bert[END_REF], the authors employed BERT to predict stock movement. They showed that BERT is able to achieve and even outperforms state-of-the-art models.

Chapter 3

Analyzing the Impact of the COVID-19 Outbreak on Companies Default Rates

Introduction

In previous sections, we discussed how the advent of a crisis (i.e., 2008 financial crisis), has the potential to dramatically alter the landscape of the business network. In this chapter, we analyze the impact on default behavior before and after the Covid-19 outbreak from the point of Credit Insurers (CI) and Export Credit Agencies (ECA)

The business network of a country is mainly composed of Small and Medium-sized Enterprises (SMEs). They are considered the driving force of the country. In large and established economies, like France, the percentage of SMEs represents 99.8 % of the total registered enterprises and The SMEs generate 48.5 % of the total employment [START_REF]Les PME et TPE en France : une situation financière améliorée et un accès au crédit plus facile[END_REF].

The definition of SMEs is not universal. However, the most common criteria used to distinguish them is the number of total employees [START_REF] Hatten | Coursemate with online interactive business plan and liveplan, 1 term (6 months) printed access card for hatten's small business management: Entrepreneurship and beyond[END_REF] generally between 10 and 250. The simplicity of its business structure allows the company to be very flexible. Commonly, SMEs rely on the personal assets of owners to finance the company. Their lack of external financing may pose an extra risk, especially when the local economy is not stable.

On March 11, the Director-General of the World Health Organization (WHO) declared Covid-19 a pandemic, causing a severe shutdown of the global economy in an effort to contain the virus. From an economical point of view, and due to their size, SMEs were strongly affected by the temporary shutdown. Given the importance of SMEs in the global economy, it is necessary to quantify the impact of Covid-19 on the SMEs' network to understand and estimate the repercussion of the pandemic.

It is important to remark that the impact of Covid-19 on company's default depends Chapter 3. Analyzing the Impact of the COVID-19 Outbreak on Companies Default Rates on the activity sectors. SMEs default occurs when the company is not able to refund the full amount of the loan. The Covid-19 impact on the SMEs' financial soundness is not homogeneous across sectors. As far as we know, no previous research has investigated the current impact of the Covid-19 outbreak on the activity sector for the French economy.

We analyze the four most represented economic sectors: Wholesale & retail trade, repair of motor vehicles; Construction; Manufacturing; Accommodation and food service activities. These sectors have been chosen regarding the number of companies that are operating in that sector. A large number of researches have been focused on the impact of Covid-19 in the economy. In this section, we present literature on the economic impact of Covid-19 with 3.3. Methodology a special focus on its impact on SME's. One of the first studies on the impact of Covid-19 on SMEs [START_REF] Lu | The perceived impact of the Covid-19 epidemic: evidence from a sample of 4807 SMEs in Sichuan Province, China[END_REF], the authors established, through the analysis of the answers of 4807 Chinese companies to a questionnaire, that SMEs were in a near-bankruptcy situation, mainly due to the inability to recover economic activity as well as the fact that they had to continue paying fixed costs with minimal or even no income. A similar analysis conducted by [START_REF] Grondys | Risk Assessment of the SME Sector Operations during the COVID-19 Pandemic[END_REF] shows that the main risk faced by polish SMEs during Covid-19 is a strong competition as well as the increases in energy prices and the low-profit margin. There are studies such as [START_REF] Juergensen | European SMEs amidst the COVID-19 crisis: assessing impact and policy responses[END_REF], in which the researchers focus on the impact on the Manufacturing sector. On one hand, the authors state that in the short term the most important problem faced by SMEs is logistics management, although the impact differs between different sectors. On the other hand, they establish that in the long term the problem will depend more on the type of SME, concluding that a policy is needed that goes beyond the survival of European SMEs and that focuses on promoting the growth of companies through innovation and internationalization. More extended analysis was conducted in [START_REF] Kalemli-Ozcan | COVID-19 and SME Failures[END_REF]. In this study, the authors use cost minimization and measure each firm's liquidity shortfall during and after COVID-19. They estimate a large increase in the failure rate of SMEs absent of government support. It is found that the most affected sectors are Accommodation & Food Services, Arts, Entertainment & Recreation, Education, and Other Services.

Literature Review

Recent studies ( [START_REF] Qadi | Feature contribution alignment with expert knowledge for artificial intelligence credit scoring[END_REF], [START_REF] Bussmann | Explainable AI in Fintech Risk Management[END_REF]) have been focused on the application of Artificial Intelligence methods for predicting the default of European SMEs. In [START_REF] El-Qadi | Sectorial analysis impact on the development of credit scoring machine learning models[END_REF], the authors concluded that the estimation of the probability of default can be improved by creating more granular models (i.e., a model by sector). Nevertheless, the analysis conducted to not focus on the impact of economically challenging periods (i.e., Covid-19).

Methodology

The proposed experimental framework in this chapter can be divided into three main parts: first we collect and clean the data provided by Tinubu. Then we analyze two types of companies: a sector-centered analysis in which we analyze the default for the chosen sectors. To observe the impact of the Covid-19 outbreak we consider the time frame 2018-2022. The second analysis is more company-centered. We consider just the companies with the sector and default information for the complete period of 2008-2022 which will allow us to observe the complete default trend and the impact of Covid-19.

Data Processing

The first step is to clean the data. We remove the data points (i.e., company information) that are duplicated. We will focus on the default evolution year by year, so each company is represented by one unique row. In the case in which the company's status changed from solvent to default in a concrete year, the company is considered in payment default.

The next step is to concentrate on French companies. The original dataset contains several European countries. However, most of the companies are based in France, so for this analysis, we will keep the companies that operate in France.

The sector activity information is encoded into two features in the original dataset: the activity codification and the activity code. For french companies, the activity codification used is the NAF2 1 . The sectors are defined by the NAF2 which is the French nomenclature of business activities adopted since 2008. This activity codification contains 4 different levels of precision. In this chapter, we will analyze the global sector classification. We 3.4. Results map the sector activity code into the real name of the sectors.

Sector Default Evolution

The first default analysis will consider companies that belong to the same sector for a given year. In this case, we analyze the default by year as well as the volume of companies by sector (i.e., the number of companies assessed by Tinubu's Credit Risk System and Risk Analysts).

Analyzing the Default of Companies by Sector for the Period 2008-2022

The next analysis proposed in this work consists of keeping companies with the sector and default information for the period 2008-2022. This more detailed analysis will help us obtain global conclusions about the sector's behavior.

Results

In this section we will present the results yielded by the proposed framework.

Sector Default Evolution

In Fig. 3.1 we present the default by sector and the number of companies analyzed for each sector before and after Covid-19. It is important to remark that if a certain company defaulted during the year 2020, the default information will be published the next year.

In other words, the year 2021 contains what happened during the year 2020.

We focus our analysis on 4 different sectors: Wholesale & retail trade, repair of motor vehicles; Construction; Manufacturing; Accommodation and food service activities. These sectors have been chosen regarding the number of companies that are operating in that sector. They represent 77.3 % of the raw dataset.

What is interesting about the figures in Fig. 3.1 is that there is a significant decrease in terms of both default rate and size of assessed companies by Tinubu's that affects all the different sectors in a similar way. In the graph, we can see how the decrease is accentuated from 2020 onwards which can be explained by the first effects on the SME's French network by Covid-19. What should be expected, since Covid-19 impacted negatively the french economy, is an increase in the default rate which is the opposite Chapter 3. Analyzing the Impact of the COVID-19 Outbreak on Companies Default Rates of what we obtained from analyzing the dataset. However, the result can be rationally explained by the policies imposed by the french government during the Covid period to avoid the default of a large number of companies.

Evolution of Companies by Sector for the Period 2008-2022

For each sector, we analyze the same companies for the period 2008-2022. We group the companies according to the sector in which they operate. In Fig. 3.2 we provide the results obtained after analyzing the companies with available data for each sector. As can be seen from Fig . 3.2, for all sectors analyzed except for the accommodation and food service activities, the default rate suffers a slight increase that starts in 2020. For the accommodation and food service activities (see Fig 3.2(d)) the default rate varies strongly due to the limited amount of data available.

Discussion

The analysis of the data provided by Tinubu shows the economic impact of Covid-19 on french companies from the point of Credit Insurers and Export Credit Agencies. From the first analysis, we found that there has been a decrease in the activity related to the sectors (i.e., Wholesale & retail trade, repair of motor vehicles; Construction; Manufacturing; Accommodation & food service activities) that are emphasized in the period 2020-2021. This behavior can be explained by the fact that the CI and the ECA risk aversion increased due to the uncertainties associated with the pandemic.

Moreover, the default rate found on the same dataset (see Fig. 3.1) is the opposite of what was expected. However, there is an explanation for this behavior and it is associated with the policies imposed by the french government. During the pandemic, the french government established a solidarity fund for SMEs particularly affected by the economy.

In the second analysis, we focused on the company's evolution in the period from 2008 to 2022. Regarding Fig. 3.1 and Fig. 3.2, the most surprising difference is the opposite behavior in terms of default when analyzing the companies using the different proposed frameworks. The French companies' solidarity fund varies depending on the size of the companies. Small companies are more supported financially than medium companies. The relation between the size of the company and its age explains the fact that the companies analyzed using a historic of 14 years reduced the size of the dataset biasing the dataset towards more established companies and thus less supported by the solidarity fund implemented by the french government.

3.6. Conclusions

Conclusions

This chapter has proposed a novel approach to analyze the economic impact of the Covid-19 Pandemic. The conducted analysis focus on the impact of the default of the SME from the point of view of Credit Insurers and Export Credit Agencies. The analyzed sectors in this chapter were: Wholesale & retail trade, repair of motor vehicles; Construction; Manufacturing; Accommodation & food service activities It is particularly interesting the divergence between the two different experiments. It was found that the trends in the default rate before and after 2020 are completely different. This is due to the bias introduced to the analysis when focusing on companies with at least 14 years of available data. Small companies are, to a large extent, removed for the second analysis (see Fig. 3.2). Another important result to highlight is the decrease in the default rate during the pandemic period and how the policies implemented by the french government has helped small-sized companies has helped companies to avoid default. This chapter represents our effort to explore empirically the effects on the SME network caused by the Covid-19 outbreak. We consider it necessary to continue to analyze the default by sector previous to and after Covid-19 from the point of view of other relevant financial institutions. This will help us understand the real impact of Covid-19 on the french economy.

Chapter 4

Aligning Feature Contributions with Expert Knowledge in Artificial Intelligence-Based Credit Scoring

Introduction

As previously introduced, credit risk assessment is a cornerstone activity for banks, financial institutions, and insurance companies. The methodology for evaluating credit risk varies based on the type of counterpart involved, which can be broadly categorized into three groups: publicly traded corporations, small and medium-sized enterprises (SMEs), and individual consumers. The risk assessment for each of these counterparts can be conducted either through the expertise of a credit risk analyst or via a mathematical approach.

The objective of credit scoring is to assess the probability that a borrower will show some undesirable behavior in the future [START_REF] Hand | Statistical classification methods in consumer credit scoring: a review[END_REF]. The nature of available data to estimate the probability of default rely on the counterpart.

For publicly traded companies the literature is focused on two different approaches: the first one starts with the Z-Score [START_REF] Altman | Financial ratios, discriminant analysis and the prediction of corporate bankruptcy[END_REF], a model that predicts insol vency using historical accounting data. The second approach relies on securities market information [START_REF] Merton | On the pricing of corporate debt: the risk structure of interest rates[END_REF]. In order to assess the credit worthiness of a company, financial institutions use financial indicators (i.e., financial ratios computed using financial statements) for business loans, and both personal and financial information for consumer lending.

To highlight the relevance of developing a credit score model, [START_REF] Albanesi | Credit growth and the financial crisis: A new narrative[END_REF] shows that during the 2007-2009 housing crisis there was a marked rise in mortgage delinquencies and foreclosures among high credit score borrowers, suggesting that credit scoring models at Intelligence-Based Credit Scoring the time did not accurately reflect the probability of default for these borrowers. After the 2008 crisis, the financial institutions became more risk-averse, which provoked a substantial increase on the barriers in the process of acquiring credit [START_REF] Cowling | Small business financing in the uk before and during the current financial crisis[END_REF].

As shown in [START_REF] Rikkers | The influence of rating philosophy on regulatory capital and procyclicality[END_REF] it is important, when developing a model that estimates the probability of default (PD) or an internal rating, to decide whether to grade borrowers using their current situation or (point-in-time, PIT) or their expected condition over a cycle (through-the-cycle, TTC). Classical Credit Rating Companies use Credit Scorecards to evaluate the risk of a counterpart. This algorithm takes as input financial information and provides a qualitative estimation of the probability of default for a company.

In the last decades, a growing number of approaches has been developed to model the credit quality of a company by exploring statistical techniques. There are three main generations of statistical techniques [START_REF] Kim | Corporate default predictions using machine learning: Literature review[END_REF]: Discriminant Analysis [START_REF] Altman | Financial ratios, discriminant analysis and the prediction of corporate bankruptcy[END_REF], Binary Response Models [START_REF] Ohlson | Financial ratios and the probabilistic prediction of bankruptcy[END_REF], and Hazard Models [START_REF] Chava | Bankruptcy Prediction with Industry Effects*[END_REF].

Machine learning algorithms have shown an increase in the prediction power for Credit Risk Modeling [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF].Although they improve the existing credit scoring models, the AIpowered systems are regarded with suspicion because they do not provide reliable explanations for the score they provide. In this context, eXplainable Artificial Intelligence (XAI) has rapidly gained interest in the financial field.

In this chapter, we use historical financial data to predict the default of a company in one-year horizon. We focus on companies based in Europe (mainly France). The information used for modeling the default is shown in Table 4.9. As we can see in ??, The dataset provided is highly imbalanced. We apply several machine learning (ML) techniques as well as resampling techniques to address this problem. Finally, we combine the best model with the SHAP technique [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF], a XAI method widely used for model interpretation based on feature attribution.

To summarize, in this chapter:

• We analyze a large dataset (around 100.000 companies) based on different European countries. Most of the literature on credit scoring algorithms predicts consumers' default, while in this work we focus on a large variety of companies (from small companies to big corporations).

• We map our machine learning model probabilities to risk score labels to compare it with an established companies credit risk scoring system.

• We perform an interpretability study, using the well-known Shapley value analysis (SHAP) , in order to understand why the algorithm made a certain decision and compare the most important features of the developed model with the expertise of several risk analysts.

Related Work: EXplainable AI for Credit Scoring

In the last decade the intersection between machine learning and the credit risk community has improved the performance of the credit risk models. In this section we present the main works on credit scoring focusing on machine learning models and non traded companies (i.e., companies that have their shares listed on any stock exchange). Also, we show previous studies on eXplainable Artificial Intelligence in finance.

Machine Learning for Credit Risk Scoring

During the last decade the relevance of Machine Learning (ML) has grown exponentially across all industries. The first intersection between finance, in particular credit scoring, and ML industries was in the 80s. Some ML algorithms used in credit scoring are decision trees [START_REF] Makowski | Credit scoring branches out[END_REF], kernel-based algorithms such as Support Vector Machine (SVM) [START_REF] Baesens | Benchmarking state-of-the-art classification algorithms for credit scoring[END_REF]. Recently, more sophisticated ML-based models have been applied to credit scoring. In [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF] they compare a list of 41 different ML models for consumer credit scoring. The results show that the Random Forest Algorithm, a random version of bagged decision trees [START_REF] Breiman | Random forests[END_REF] outperforms the classical and widely used Logistic Regression (LR).

The scarcity of data for assessing the credit risk of non publicly traded companies has provoked research to be more focused on consumer lending. Nonetheless, some works focus on this subject. In [START_REF] Bussmann | Explainable machine learning in credit risk management[END_REF], they analyze a dataset of companies based in Southern Europe for the year 2015. They use Extreme Gradient Boosting [START_REF] Chen | Xgboost[END_REF] for predicting whether a company will default the next year. [START_REF] Provenzano | Machine learning approach for credit scoring[END_REF] build a model to predict the default of a company in a one year horizon using a dataset composed of Italian companies over the period 2011-2017. For PD modeling they use a boosting method called LightGBM [START_REF] Ke | Lightgbm: A highly efficient gradient boosting decision tree[END_REF]. [START_REF] Moscatelli | Corporate default forecasting with machine learning[END_REF] they consider that a company is defaulted for the given year if the ratio of non-performing credits to total credit drawn is greater than 5%. Their best results has been obtained using Random Forest.

Companies' default, as well as consumers default, are rare events, and thus, when treating with these datasets is important to address potential data collection and reporting bias. In Table 4.1 we summarize the different datasets used in the literature. Several techniques are applied to tackle the imbalance problem . One of the common solutions is generating synthetic data of the minority class (SMOTE) [START_REF] Chawla | Smote: Synthetic minority over-sampling technique[END_REF]. In [START_REF] Islam | Investigating bankruptcy prediction models in the presence of extreme class imbalance and multiple stages of economy[END_REF] they show that applying SMOTE [START_REF] Chawla | Smote: Synthetic minority over-sampling technique[END_REF] in training stage improves the performance of a large list of ML models for bankruptcy modeling.

EXplainable Artificial Intelligence (XAI) in Finance

The pursuit of highly performant machine learning algorithms has derived in complex systems that are harder to interpret and therefore to trust [START_REF] Dosilovic | Explainable artificial intelligence: A survey[END_REF]. The challenge for today's ML-based credit scoring models and more generally the implementation of AI-powered systems in the financial industry is to meet strong regulations(e.g., General Data Protection Regulations (GDPR), Basel III, Solvency II). To ensure the correct, ethical and responsible development of AI in finance the implemented systems need to be explainable and interpretable. Recent works [START_REF] Kuiper | Exploring explainable ai in the financial sector: Perspectives of banks and supervisory authorities[END_REF] discuss the requirements an AI-based system needs to meet to guarantee the fair functioning of the system. Several techniques have been developed in order to clarify opaque models interpretability problem [START_REF] Guidotti | A survey of methods for explaining black box models[END_REF]. One of these techniques is SHAP (SHapley Additive exPlanation). SHAP [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF] is a framework used for interpreting predictions based on game theory. It falls into the Post-hoc explainability methods taxonomy of XAI. These family of methods target to explain the output of models that are not readily interpretable by design [START_REF] Arrieta | Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai[END_REF]. Recent works [START_REF] Demajo | Explainable ai for interpretable credit scoring[END_REF] emphasize the importance of understanding the decision-making process for ML based credit risk models. and how addressing this problem could benefit the implementations of more machine learning models in the credit risk industry.

Methodology: Black Box Models and XAI

In this section, we present the methods we use in our study. First, we start by presenting the State-of-the-Art ML models, and then we describe how we prepared our data for the modeling step.For the the data preprocessing we used Python frameworks Pandas 1 and Numpy 2 . We show a procedure to perform data augmentation and generate synthetic data for the minority class, based on oversampling, in order to improve the model robustness and performance metrics. Imblearn 3 is the framework used for data augmentation that contains the SMOTE method. We first present the different evaluation metrics used to compare the different ML models. We then briefly describe the explainability framework used in this work, SHAP values. In the last stage we detail how we conduct a survey among several credit risk analyst experts. The survey will be used to compare the results of the explainability framework with the human expert explanations.

Data Preprocessing Pipeline for Company Credit Risk Scoring

Data Cleaning: The data used in this study is provided by Tinubu Square4 , a company that provides companies credit risk opinions as a service. Tinubu's database is composed of financial and non-financial information about a large set of companies.The initial dataset is composed of 6,051,844 data points and 17 variables. To develop the PD model, we use the financial variables in Table 4.9. , therefore, from the original dataset we will keep those evaluations that were made using financial variables. We are interested in those companies for which we have all financial information available. At this stage, we have a total of 1,415,610 assessments (i.e., data points); the number of unique companies assessed is 418,516.

Data Labeling: The next step is to create the target variable of our problem. We are considering the problem of modeling the default of a company knowing its previous financial statements. Financial statements of a company non publicly traded are published yearly. Since in this study we are interested in short term PD modeling, we fix the time horizon to one year (e.g., given the financial statements of a company closed closing financial statements of a company in the year 2012, we want to estimate the probability of default for the year 2013). We select companies with financial data available for two consecutive years, knowing that the first year the company has to be a non defaulted company. This means that we check if the Out of business variable in to No), and then, if in the second year the company's Out of business = Yes, then we set the target variable to 1; otherwise, we set the target variable to 0.

In Fig. 4.1 we observe that the years with around 100K companies assessed, the default rate is in the range [1.5%-2%]. During the period between 2015-2018, the amount of assessed companies is significantly lower than the other years, and consequently the % of defaulted companies varies heavily.

All original financial variables (see Table 4.7) are important for predicting whether a company will incur into a default. Since the number of missing values is significant (see Table 4.8), We keep those companies we have all financial variables needed to compute the ratios in Table 4.9.

For analyzing the results of the interpolation, we remove companies with missing information. We select the feature i that will be interpolated. We split the data into two datasets: a first set (95% of the data) with non-missing values and a second dataset whose feature i is full of missing values. We interpolate feature i using kNN [START_REF] Troyanskaya | Missing value estimation methods for dna microarrays[END_REF]. Finally, Table 4.2: Financial ratios used in the financial industry. This ratios are the inputs of Tinubu's Scorecard Rating Algorithm and the ML models used to predict the default of a given company. we score the company with Tinubu's algorithm and compare the rating obtained for the interpolated company and the original rating of the considered company.

Features Definition

Data Transformation: First, we start by encoding the categorical features. The only categorical variable present in Table 4.7 is the country code. We use the one-hot encoding technique to create a new column for each country. This variable will take the value 1 if the company is located in the considered country (when a model is built per country), and 0 otherwise. We split the dataset into two main sets: the first set contains data between 2004-2012. We use this data to train and test (70% for training and 30% for testing). The data between 2013-2018 will be used to validate our model.

Data Normalization:

The transformed data in Table 4.9 contains noise and has a different scale. We scale the data using the Standard Scaler [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. This helps to reduce the noise by transforming the data distribution into a new one with a mean 0 and standard deviation of 1.

Data Oversampling: The SMOTE oversampling technique [102] consists of oversampling the minority class by taking each minority class sample and introducing synthetic examples along the line segments joining any/all of the k minority class nearest neighbors.

Depending on the amount of over-sampling required, neighbors from the k nearest neighbors are randomly chosen.We resample the training set using SMOTE with the parameter k=10 and the ratio between the minority class and majority class in the resampled set to 0.5 5 . Intelligence-Based Credit Scoring

PD Modeling using Machine Learning Models

In this part, we present the classification models used to predict the probability of default. The output of all models is a probability of default. We reduce our problem to a binary classification model, we establish that if this probability is greater or equal to 0.5, then the model is predicting that the company will be in default the next year.

Logistic Regression (LR): a linear model that makes a prediction by computing a weighted sum of the input features. The output of this model is a probability for binary classification (whether the company will fall into default or not). This probability can be mapped by setting a threshold. The threshold used for predicting whether a company will be in default is 0.5. If the probability is greater than 0.5, then the model predicts the company as defaulted, i.e., bankrupt. Logistic regression has been widely used in the credit risk prediction domain due to its simplicity and interpretability [START_REF] Moscatelli | Corporate default forecasting with machine learning[END_REF].

AdaBoost (AB):

A tree-based ensemble method whose algorithm trains a decision tree and then tries to fix the errors by training sequentially decision trees over the predecessor tree errors. For the hyperparameter optimization step, we use Grid Search 5 fold crossvalidation. We obtain the best results with a Learning rate=0.8 and n estimators=100.

Random Forest (RF): this algorithm generates trees from a random sample of the training data. For each generated tree, the algorithm randomly selects an attribute for splitting the tree. This randomness reduces the overfitting of the model by decreasing the correlation between trees. The output of a Random Forest is the average of all trees predictions. Between all the parameters tested using Grid Search 5 fold cross-validation for the training set, best results (5.1) were obtained with n estimators=1500.

Gradient Boosting (XGBoost): XGBoost consists of a sequential combination of weak learners in order to create a robust model. The difference between AdaBoost and Gradient Boost is that the drawback of having weak learners is detected using gradient descent. A variant of the Gradient Boosting, the extreme Gradient boosting has been used in this work. This variant uses a more regularized model formalization to control overfitting. In Table 5.1 results associated to the XGB model were obtained using the hyperparameters: learning rate=0.1, n estimators=100, max depth=10, subsample=1, colsample bytree=1 and gamma=0.7.

All methods employed used the implementations provided by Scikit Learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF]. XG-Boost uses the one concretely from [START_REF] Chen | Xgboost[END_REF]. We compare the performance of the different machine learning models using the following classification metrics: precision, recall and the Area Under the Receiver Operating Characteristics (AUC).

Data Oversampling using SMOTE

The SMOTE oversampling technique [START_REF] Chawla | Smote: Synthetic minority over-sampling technique[END_REF] consists on oversampling the minority class by taking each minority class sample and introducing synthetic examples along the line segments joining any/all of the k minority class nearest neighbors. Depending on the amount of over-sampling required, neighbors from the k nearest neighbors are randomly chosen. We resample the training set using SMOTE with the parameter k=10 and the ratio between the minority class and majority class in the resampled set to 0.56 .

• Accuracy: defined as the ratio of correct predictions • Precision: the proportion of correctly predicted classes over the total of data points.

• Recall: the proportion of defaulted companies that are correctly predicted by the model.

• F1-Score: the weighted average of the precision and recall.

• AUC : measures model's ability to discriminate between cases. It is the area under the Receiver Operating Characteristic.

SHAP for Model Explanations

To understand the outputs of the ML model we employ SHAP [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF], a framework for interpreting model predictions. SHAP uses a game-theoretic approach that explains the contribution of each feature to the final output of a given model. This method will ascertain which financial features are the most relevant to predict the default of a company.

Human Expertise Alignment: Introducing Credit Risk Analysts Expertise

We asked several Tinubu's risk analysts (to be more precise 4 different risk analysts) to weigh what variables are the most important to rate a company. We asked them to distribute 100 points between all the variables that have been used for training our ML model. Then we compute the sum of all weights given by the different risk analysts. This sum represents the level of importance for each feature. This value will be used to rank the features by importance degree and it allows us to compare the human expert ranking with the feature importance for our model given by the SHAP value The main point of this article is to compare the explanations given by the ML model (i.e., in this case the XGBoost) after applying the SHAP [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF] framework with the credit risk analyst opinion. The way this comparison has been conducted is as follows: we asked several Tinubu's risk analysts (to be more precise 4 different risk analysts) to weight what variables are the most important in order to rate a company. We asked them to distribute 100 points between all the variables that has been used for training our ML model. Then we compute the sum of all weights given by the different risk analysts. This sum represents the level of importance for each feature. This value will be used to rank the features by importance degree and it allow us to compare the human expert ranking with the feature importance for our model given by the SHAP value.

Results

In this section, we present the results of the models we described in the previous section for different settings. Then we compare the results of the best model with Tinubu's Rating System. Finally, we discuss the differences between the decision-making process of our ML model, Tinubu's mathematical Credit Rating System, and the credit risk experts.

Analyzing the Impact of Interpolation in the Credit Score

We analyze the impact of using an interpolation method based on kNN algorithm [START_REF] Troyanskaya | Missing value estimation methods for dna microarrays[END_REF].

In this case, we consider the companies with all features available. We remove around 5% The columns of the confusion matrix represent the rating given by the Tinubu's system after the feature interpolation. Rows correspond to the rating given by Tinubu's algorithm with the original feature (the number of missing values of the considered features, see Fig. 4.2) of a given variable. Then we interpolate using kNN with k = 2. The number of neighbors chosen is the one that minimizes the number of companies that after the interpolation has changed their ratings using Tinubu's credit scoring algorithm. At this point, we run Tinubu's algorithm and analyze the change of ratings for the companies that have been interpolated. Results in (Fig. 4.4a, Fig. 4.4b and Fig. 4.4c) mainly show that the risk score of interpolated companies tends to be slightly worse. However, it is interesting to keep in mind that we prefer a more conservative score.

Performance of ML Models for PD Modeling

In Table 5.1, we compare several ML-models using different data strategies: without resampling (WRS), resampling the training data with SMOTE (RS) and resampling the training data and measuring the model performance over the validation dataset (RS+VS). Analyzing Table 5.1, we observe that the ML models do not recognize the companies that will default the next year if we train them with the original data. However, there is an improvement for models trained with generated synthetic data of the minority class (i.e., the. defaulted companies). What we notice is that for the validation set, which is composed of companies between 2013-2018, the XGBoost performs better than all other tested models. The results in

Mapping XGBoost Probabilities to Tinubu's Grades

Tinubu's Scorecard Algorithm is the internal proprietary rating system used at Tinubu Square to evaluate the credit risk of a debtor (a company). This rating is a descriptive way to present the probability of default of the assessed entity. Tinubu's rating scale uses letters to establish the level of credit-worthiness of a company. On one hand, A is given for companies that the model estimates the probability of default is close to zero. On the other hand, companies considered very likely to incur into default are rated with letter F. The X score is given to companies whose data needed to compute the PD is not available. Our goal is to compare the two models: Tinubu's Scorecard Algorithm and the best machine learning model (i.e., the XGBoost model). Nevertheless, the outputs of both models (Tinubu Scorecard Algorithm and ML model) are naturally different. The Tinubu Scorecard Algorithm model outputs a category, while the ML model outputs a continuous variable (probability of default). To compare both models we need to create a mapping to assign to each score letter a probability of default. As we consider the gold standard of the financial experts model of Tinubu our ground truth, this PD will need to be matched by the ML model output. Given a Tinubu's rating class we compute the average of the probabilities of the companies being in default next year (i.e., Y t = 1).

Results

µ(R)

= 1 N n-1 i=0 P i (Y t+1 = 1 | Rate = R) (4.1)
where R is the rate yielded by Tinubu's Scorecard Algorithm and rating R ∈ {A, .., F }. µ(R) represents the mean of the probabilities of default given by the XGBoost model when the yielded rate by the Tinubu's Scorecard is R.

R = argmin R P(Y t+1 | X t ) -µ(R) (4.2)
Then for each company, we estimate the rating by searching for the rate that minimizes the difference between the PD of the company given by the ML model and the average probability estimated by the ML model of all different Tinubu's Rating classes (see equation 5.2). The estimated rating R represents the rating yielded by the XGBoost after the mapping process. In Figure 4.5 we observe that the ML model is able to assign correctly low-risk labels to companies (A and B labels). However, the ML model overestimates the risk of a significant number of companies. This is shown in Table 4.5 in column F. In the context of Credit Scoring is important to remark the impact of underestimating the risk. The critical case occurs when the model estimates that a company has a low or near zero risk of being in default the year after the assessment, and the reality is that such company is much riskier than predicted by the model. This is why we consider that the XGBoost model works relatively well, in comparison with a well established Rating Algorithm (Tinubu's Scorecard Algorithm) that has been rating companies since early 2000s and is still being used nowadays.

Explaining our PD model: SHAP Value Analysis

We analyze how the ML model has arrived at the results shown in Tables 5.1 and 4.5. Features in Fig. 5.4 are sorted according to their relevance (i.e., SHAP mean absolute Intelligence-Based Credit Scoring value). The most relevant (R3 profitability ratio) to the less important ratio (country code J). This analysis highlights the fact that, for the ML model (i.e., the XGBoost model), companies based in country K (i.e., the Legal Entity (LGE) COUNTRY CODE K binary variable is equal to 1) have a lower probability of default than companies based in other analyzed countries. This fact matches with the way Tinubu's algorithm works. For companies with financial ratios relatively similar, the Tinubu's Scorecard Algorithm will give a better rating to those companies based in country K. The contribution of a given feature whose data points lie next to other data points of opposite color in the same axis show that both high and low values of that feature have influenced similarly the model outcome, and thus their contribution may be contextually better determined by considering generic values of other out-weighting features that have a stronger contribution for that data point.

Feature Contribution Analysis: Assessing Explanations from risk analyst experts vs ML models

To assess the explainability of our ML model with the respective mathematical model of Tinubu Square, we asked 4 different analysts from Tinubu Square to give a weight for all features in Table 4.9. Each analyst has 100 points to distribute between the features. All participants of this survey filled the Table independently. As all analysts work at Tinubu Square, we expect relatively uniform criteria. Comparing the weights given by the analysts in Table 4.6, and the importance of each feature for the ML model (see SHAP analysis in Fig. 5.4), we conclude that the decision-making process diverges among machine and human models. Generally, the assessment of the credit-worthiness of a given company is made by a human expert if the amount of credit demanded by the assessed company is relevant. Therefore, the dataset of companies treated by the risk analyst may be biased towards companies that can afford larger interest rates or higher insurance premiums. For all analysts the R 3 Profitability feature is irrelevant, while for our ML model is the most important one. Analyzing the most important features for both the humanexpert and the ML model we discover that the only feature considered relevant when assessing a company credit worthiness for both human experts and the ML model is the Chapter 4. Aligning Feature Contributions with Expert Knowledge in Artificial Intelligence-Based Credit Scoring R 1 Solvency, which measures the ability of a company to meet short-term debts. From a credit risk analyst stand point, the most important features are those related to shortterm activities (i.e., R 1 Liquidity, R 2 Liquidity and R 2 Solvency). However for our ML model, the features that contribute the most to the model output are the features that relate to activities extended in time, i.e., long term capabilities of the firm, for instance, R 3 Profitability, Time in Business and location. Table 4.6: Weight of each feature given by 4 different risk analyst expert (i.e, R.E) at Tinubu Square. The risk analyst expert distribute 100 points between all the features. Features are ordered by decreasing importance for the risk analysts (i.e., total points the feature has received from all experts)The sum of weights given by the analysts is the same for R 2 Solvency and R 2 Profitability. We ranked R 2 Solvency above R 2 Profitability because the weights are similar. In Table 4.6 we rank the relevance of the features depending on the risk analysts. The latter act as a proxy of the gold standard credit risk opinions. Their expertise is captured by the Tinubu internal Scorecard Algorithm.

Features

Discussion

The first issue concerns the usage of a highly imbalanced dataset. It is worth noting that the task of predicting the probability of default entails having to deal with the bias inherent to the nature of the data since the percentage of default companies is very low concerning successful companies Therefore the ML model may have problems differentiating between both defaulted and non-defaulted companies. Since defaults are very rare events, resampling data by generating new synthetic data of the minority class (SMOTE) may not improve models' performance because each event of default has its characteristics in a particular context, and the concept of default may differ from one country to another. It is worth highlighting the comparison between the Tinubu's Scorecard algorithm and the ML model (i.e., the XGBoost model).

The comparison between Tinubu's Scorecard algorithm and the ML model shows that 4.6. Conclusion and Future work the ML model does underestimate the number of highly ranked firms (i.e., ranked with top score A, B and C by Tinubu's Scorecard Algorithm). On the other hand, we observe that the ML (XGBoost) model rates a considerable number of companies with the F score, while the Tinubu Scorecard Algorithm maps the same companies mainly to scores in the range [B-D]. This behavior can be interpreted as the ML model is significantly more conservative than Tinubu's Scorecard Algorithm. The property of being risk-averse is therefore desirable in our context, and thus, beneficial due to safety reasons.

The comparison showed that the criteria between human experts and the ML model are quite different. In particular: a. The ML model excels at being able to capture the longer term abilities of a company, for the features considered by the Tinubu risk analyst experts. The latter focuses more on shorter-term variables such as R 1 Liquidity, or R 2 Liquidity, while the ML model attributes higher relevance to life-long attributes of the firm. b. We found interesting the fact that, without explicitly implementing any constraint, XGBoost arrives at the conclusion that companies that exercise their activities in Country K are more likely to avoid financial problems, while this behavior is explicitly encoded and accounted for by Tinubu's Scorecard Algorithm. c. Since credit risk scoring algorithms, especially those focused on companies rather than individuals, are in the very early stages, we acknowledge the abilities of our ML model to remain conservative when estimating risk. This is a desirable property of such complex models, since it is preferable to avoid critically large economical losses

Conclusion and Future work

Several state-of-the-art machine learning models have been proposed to model the probability of default of a company the year after it has been assessed. For the Tinubu dataset used, we remark the importance of dealing with a high imbalanced dataset.

In order to evaluate our ML model's PD, casted as a regression problem, with risk analyst experts categorical scores predicting a PD in form of a score, we mapped the probabilities given by the best ML model (i.e., XGBoost, as it showed best results) to compare our ML model with to Tinubu's Scorecard model score labels. This mapping shows that the ML model is able to tell apart companies with low risk of default (i.e., companies rated with an A and B with the Tinubu's Scorecard Algorithm) with respect to from the rest of companies.

One of the biggest challenges for the introduction of machine learning based models in the credit scoring field, in particular for companies credit scoring, is the lack of credibility, trust, and explainability. We addressed this problem by using the explainable framework of SHAP analysis. [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF]. Assessing the results of the SHAP analysis, we conclude that the difference between companies based on the country K and the rest of the companies may require a deeper analysis, since the hypothesis of different country regulations could affect the companies default analysis. The analysis of explanations given by a SHAP analysis of feature contributions concerning explanations backed up by the expertise of a pool of credit analysts showed a certain divergence between what variables should be considered more relevant when assessing a company.

Future works should focus on studying how inductive biases can infuse expert knowledge into the ML model, for instance, by introducing credit expert opinions and preferences (at the data annotation and model design stages) to improve the model performance. Other potential avenue of research is designing a sound basis for causal explainability that can be verified and certified by human experts. Moreover, it may be interesting to further analyze the geographical locality context, i.e., contextually and historically, by assessing companies both by risk analysts and the ML model. Finally, we hope future work designs explainability metrics to programmatically assess the quality of an explanation given by a black-box model that learns, evolves and degrades over time, to continuously assess its fidelity and alignment with human expert opinions. This way we will be able to deploy AI systems that both humans and experts can mutually improve, support, and trust. Represents the percentage of the gross income that goes to debt payments Liquidity R 1 :

Total Current Assets Total Current Liabilities

The current ratio measures the ability to pay short term obligations (within one year) Liquidity R 2 :

Cash Liquid Assets Sales

Liquidity indicator that represents the percentage of liquid assets over the revenues of the company Profitability R 1 :

Working Capital Sales

Shows the relationship between the funds used to finance company's activities and the revenues a company generates as a result Profitability R 2 :

Net Income Is an indicator of company's profitability Profitability R 3 :

Gross Income Total Assets

Measures how effectively a company is using its assets to generate earning Time in business Assessment year -incorporation year Years in business Sales evolution Current sales -previous year sales Measures the sales evolution Country code Country codification Country abbreviation in which the company is located

Chapter 5

Sectorial Analysis Impact on the Development of Credit Scoring Machine Learning Models

Introduction

As mentioned previously, in trade finance, the ability to accurately assess the creditworthiness of companies in various sectors becomes increasingly complex during economic downturns (e.g., financial crisis 2008, COVID-19 outbreak).

In [START_REF] Albanesi | Credit Growth and the Financial Crisis: A New Narrative[END_REF], the authors find that the main issue that led to the financial crisis of 2008 was the underestimation of mortgage risk of default during the credit growth experienced between 2001-2007. Consequently, financial institutions became more risk-averse, which provoked a substantial increase in the barriers in the process of acquiring credits [START_REF] Cowling | Small business financing in the UK before and during the current financial crisis[END_REF].

The 2008 global crisis not only affected the financial system but also international trade. Concretely, in [START_REF] Shelburne | The Global Financial Crisis and Its Impact on Trade: The World and the European Emerging Economies[END_REF], the authors found that in Europe the emerging economies were severely affected by the financial crisis. In [113], the authors conclude that financial crises also affect negatively in terms of trading costs for those countries with stable and secure trading relations. On the other hand, international trade has been associated with positive impacts on growth for Small, Medium Enterprises (SMEs). As stated in [113], there are two major challenges in international trade: the counterparty risks are relatively high for exporters and importers with limited capacities and resources. SMEs are, in a large set of cases, constrained in terms of working capital. The authors consider that digitalization can increase SMEs' access to trade finance and thus accelerate the growth of the economy.

Financial and insurance institutions can help SMEs to access trade finance by financing Learning Models their activities. It is in this context that the Credit Scoring systems play an important role. They are the systems that serve as a backbone for decision-making.

Historically, credit scoring systems have been based on statistical techniques [START_REF] Altman | FINANCIAL RATIOS, DISCRIMINANT ANALYSIS AND THE PREDICTION OF CORPORATE BANKRUPTCY[END_REF][START_REF] Ohlson | Financial Ratios and the Probabilistic Prediction of Bankruptcy[END_REF][START_REF] Kim | Corporate Default Predictions Using Machine Learning: Literature Review[END_REF]. Recently, machine learning (ML) algorithms have shown an improved capacity for estimating the PD [START_REF] Provenzano | Machine Learning approach for Credit Scoring[END_REF] in different contexts. One of the main problems of MLbased approaches is the trade-off between the performance of ML-based models and the explainability given by those models [START_REF] Provenzano | Machine Learning approach for Credit Scoring[END_REF][START_REF] Babenko | Classical Machine Learning Methods in Economics Research: Macro and Micro Level Examples[END_REF].

Previous work on credit scoring has mainly focused on consumer credit scoring. Few researchers have addressed the problem of estimating the creditworthiness of SMEs ( [START_REF] Bussmann | Explainable AI in Credit Risk Management[END_REF][START_REF] Provenzano | Machine Learning approach for Credit Scoring[END_REF]). As mentioned previously, this is an important constraint for SMEs to get access to trade finance. This chapter examines the creditworthiness of a relatively large set, compared to previous literature, of companies based in France. Furthermore, the study conducted in this chapter addresses two main questions:

1. RQ1. Are sectorial models more appropriate than global models for predicting the company's default? : We focus our analysis on the economic sectors and the impact on the model appropriateness. The chapter is structured as follows: first, we present the previous research in the field of credit scoring based on ML models. Then, we explain the methodology we used to solve the problem. In the next section, we show the results obtained using the chosen approach. Finally, we present the conclusion and potential hints for future work.

Related Work

In this section, we present the previous work that has been done in the intersection of ML, credit scoring, and eXplainable Artificial Intelligence (XAI).

In [START_REF] Nehrebecka | PREDICTING THE DEFAULT RISK OF COMPANIES[END_REF], the authors analyze and compare the performance of a LOGIT model and a Support Vector Machine (SVM) model over a dataset composed of Polish companies. Both models tend to have higher accuracy in the training set. This is mainly due to the presence of a significantly low percentage of defaulted companies (i.e., 3%). Similar 5.3. Methodology work has been done in [START_REF] Svabova | Business Failure Prediction for Slovak Small and Medium-Sized Companies[END_REF] for Slovakian companies. In this chapter, the authors add the featured sector to their analysis and also analyze the default over two different time frames: one year and two years. They combined two linear models. The results showed that in terms of explanatory variables both models are similar.

Besides these approaches, there has been some interesting research that focuses on the dependence between sectors and countries.Namely the work in [START_REF] Coquidé | Interdependence of Sectors of Economic Activities for World Countries from the Reduced Google Matrix Analysis of WTO Data[END_REF]. In this chapter, the authors study the interdependence and interactions of economic sectors of several countries (e.g., USA, Russia, and China). By applying the Google Matrix Algorithm, they found that globally speaking there is a strong sensitivity of the chemicals, metals, energies, and food sectors to the price increase of the petroleum sectors. In contrast, in [START_REF] Chen | Measuring fairness in credit scoring[END_REF] The authors found that the economic sector features (i.e., country and sector features) are not relevant in terms of predicting the default. In [START_REF] Provenzano | Machine Learning approach for Credit Scoring[END_REF], the analysis of the sector is introduced to the ML models using Sentence Embeddings and Autoencoders.

Furthermore in recent works [START_REF] Bussmann | Explainable AI in Credit Risk Management[END_REF][START_REF] Provenzano | Machine Learning approach for Credit Scoring[END_REF][START_REF] Chen | Measuring fairness in credit scoring[END_REF], they apply a widely used post-hoc XAI method: SHapley Additive exPlanations (SHAP) Values [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF]. This method is used to understand the global behavior of the model.

Methodology

We propose an experimental framework that can be split into three different main parts: in the first part we proceed to analyze the raw data provided by Tinubu Square, a company that specializes in providing credit risk opinions to its clients. 1 . This study consists of three main stages. The first stage consists of obtaining, from the raw data, the data in the format necessary to create the ML-based credit scoring models.

In the second stage, we compare the performance of six different state-of-the-art MLbased models for predicting a binary class (i.e., whether a company will be in default the year after the release of its balance sheet): Logistic Regression, K-Nearest Neighbors, Support Vector Machine, Random Forest, LightGBM, and XGBoost. The models will be compared using the following metrics: F1-Score, Accuracy and Area Under the Receiver Operating Characteristic (AUROC). To improve the appropriateness of the models, we create a model by economic sector (i.e., mainly the four most represented economic sectors in Tinubu's dataset). Then, we compare the performance year by year. Furthermore, we compare the results of the best model with the credit rating given by human credit risk analysts by mapping the probabilities of default given by the model with the credit rating accorded by the risk analyst.

In the last stage, we compute the SHAP values for each economic sector model. We compare the model's global explanations to find common patterns in models' decisionmaking.

Data Description

In this part, we describe, step-by-step, how to obtain the data that will be used to compute the PD. Raw Data is composed of 9 951 981 financial and non-financial information of companies based in Europe, mainly in France. The raw dataset is composed of 81 features.

Data Cleaning: In this step, we remove the duplicated data points and remove the features that will not be used during the modeling stage. Since the features related to the sector are encoded using the NAF2 activity codification, we map the code to the name of the sector. We focus the analysis on the companies based in France.

Data Labeling: The raw data contains the financial and non-financial information about the companies for the period 2008-2020. We will analyze just those companies for which we have two consecutive financial statements. We will add to the credit record the binary variable N extyearStatus. This feature will take the value 1 if the company goes into default the next second year and 0 else. After this stage, the remaining dataset is composed of 171 109 credit records and 32 different financial features.

Sector Analysis: First, we analyze the distribution of economic sectors. We analyze graphically the default evolution over the period 2008-2020 for the sectors with more than 10 000 credit records.

Encoding Categorical Features: As we mentioned before, we start by creating a model over the whole dataset. For this model, we will need to encode the categorical features. In this dataset, there are two categorical features: the activity code feature and the credit rating given by Tinubu. We encode the activity feature using the One-Hot encoding technique (i.e., for the model that englobes all economic sectors) which creates a new column for each of the categories of the features. The column takes the value 1 if the company operates in the considered sector and 0 else. The rating given by Tinubu is a feature that takes 6 different values: from A (i.e., well-established company) to F (i.e., companies with a high PD). We will use Ordinal Encoding to encode these features. Splitting the Data: We will use the 70% for training the models and 30% for testing the models

Machine Learning Algorithms for Credit Scoring

In this section, we present briefly the models we employed in this chapter. We also define the metrics we will use to compare the results. The outcome of the models is a probability of default. Since our target variable is binary, we need to map the output of the model to binary distribution. For instance, all models will consider that a company will go into default if the PD is equal to or greater than 0.5.

Logistic Regression (LR):

LR is a linear model whose output is a weighted sum of independent variables. The output of the model is mapped to a probability using the sigmoid function. Random Forest (RF): Random Forest combines many simple decision trees, using the Bagging procedure, to increase their prediction power. In addition, the trees are generated more randomly, choosing arbitrarily the split variable at each node.

Methodology

Light Gradient Boosting (LightGBM): LightGBM is a gradient boosting framework based on decision trees to increase the efficiency of the model and reduce memory usage. It uses two different techniques: Gradient-based One Side Sampling and Exclusive Feature Bundling (EFB) eXtreme Gradient Boosting (XGBoost): The model utilizes the boosting procedure as an aggregation technique: small trees are sequentially added to the model to reduce the loss while keeping the previous trees fixed. Each tree focuses more on the individuals which have been badly predicted from the previous trees.

For all models, we performed hyperparameter tuning using the package hyperopt [START_REF] Bergstra | Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures[END_REF]. For the implementation of kNN, LR, SVM and RF we used the scikit-learn package [START_REF] Buitinck | API design for machine learning software: experiences from the scikit-learn project[END_REF]. XGBoost and LightGBM have been implemented using [START_REF] Chen | XGBoost: A scalable tree boosting system[END_REF] and [START_REF] Ke | Lightgbm: A highly efficient gradient boosting decision tree[END_REF]. The metrics used for evaluating the performance of the different ML models are: the Area Under the Receiver Operating Characteristic (AUC ROC), F1-Score 2 and the Accuracy.

Explaining Model Behavior using SHAP Values

SHAP is a widely used post-hoc explainability framework used for understanding the inner functioning of any model. Developed by Lundberg et al. [START_REF] Lundberg | A unified approach to interpreting model predictions[END_REF], it is a method that uses a game-theory approach to assign to each feature an importance value for a particular prediction. We will use this framework to compare the explanations given by all the different developed models.

Generating Ratings from Models Outputs

As we mentioned before, Tinubu provides credit risk ratings using labels that range from [A-F]. However, the output of the models is a PD. To compare both we need to map the PD to Tinubu's risk class. In this work, we propose the next mapping: Given a Tinubu's risk class, we generate a PD for all companies that have been rated with this label. Then, we compute the mean of the PD associated with this rating. where R is the rating yielded by Tinubu's Scorecard Algorithm and rating R ∈ {A, .., F }. µ(R) represents the mean of the probabilities of default given by the XGBoost model when the yielded rate by the Tinubu's Scorecard is R.

µ(R) = 1 N n-1 i=0 P i (Y t+1 = 1 | Rate = R) (5.1) 2 F 1 -Score =
R = argmin R P(Y t+1 | X t ) -µ(R) (5.2)
Once calculated the mean for each risk class, we estimate the rating for every company by searching which rating class minimizes the distance between the PD for the company and the PD mean for that risk class.

Results

In this section, we analyze the impact of the sector on the percentage of defaulted companies to establish whether the sector variable is important when predicting the default Learning Models of a company. We present the results obtained over the raw dataset after applying the data preprocessing pipeline shown in the previous section. The output of the most performant model (i.e., the probability of default) will be compared with the rating given by several risk analysts. Since the nature of both risk analyses is different, we proceed to map the probability of default into a credit risk rating. We simulate a scenario in which the most important features for the ML-based model (i.e., the most important features using the SHAP analysis) are missing. Finally, we compare the rating generated by both models, with the original data and the interpolated data, with the rating given by the risk analysis.

Default Analysis by Sector

.2 shows in fact that the percentage of defaulted companies for the analyzed period (e.g., 2008-2020) differs considerably among sectors. In figure 5.1, we can observe the list of sectors in the dataset as well as the distribution. As we can observe, the percentage of default varies considerably between the analyzed sectors in figure 5.2. In this work, we will focus on the 4 principal sectors: Manufacturing, Construction, Accommodation and Food Services, and the Wholesale Retail Trade and Repair of Motor Vehicles.

Model Performance

In table 5.1, we compare the results obtained for the six state-of-the-art machine learning models over the test set. It is important to remark that all these models have been trained using the whole dataset (i.e., all sectors included in the training set). The results The ensemble methods outperform the more classical ML models (i.e., SVM, LR, and KNN). For the economic sector analysis, we will develop the ensemble methods we employed previously. In Fig. 5.3 we compare the performance of the models for the manufacturing, wholesale, construction, accommodation, and food sectors over the period 2008-2020. We can conclude that the three models used have similar F1-Score being the XGBoost-based model, the one that generally presents the best results for all the different analyzed sectors.

Results

Shap Analysis

In this part, we explore and understand the model's behavior by applying the shap framework. As we can observe, the most relevant features for the model (i.e., XGBoost) for all sectors (see Fig. 5.5a are the Tinubu's Rating, Retained Earnings and the EBE3 Cash and Liquid Assets and the Current Ratio. The analysis also highlights how the magnitude of different features changes the output of the model. For example, the higher the risk according to Tinubu's rating (i.e., higher values after encoding of the Rating feature) the greater the likelihood of going into default the next year. It is important to remark that, from a risk point of view, it makes sense since the most relevant financial features for the model are the ones mentioned before. These features are related to the company's capabilities to pay short term debts, and hence, paying its debts.

On the other hand, if we diagnose the explanations given by SHAP values for the 4 different sectors in Fig. 5.5, we can see that the main difference, in terms of the most relevant features, is in the wholesale and retail trade sectors (see Fig. 5.5b). This sector is the only who does not consider Tinubu's Rating feature as the most important variable 5.5. Discussion for predicting the default. For all other sectors, the model's explanations, in terms of feature importance, vary slightly in comparison to the model that englobes all sectors.

Comparing the Risk Analysts risk scoring with the MLbased Credit Scoring System

As mentioned in previous sections, Tinubu measures the risk using a descriptive way in which if it considers that a company will likely incur a default in the short term, it will rate the company with the letter F. On the other hand, if it considers that the likelihood of the company not repaying its debts in the short term is close to zero, they rate the company with the letter A.

The nature of the ML-based model is quantitative, in other words, it outputs a probability of default. Since both methods are different by nature, we will use the mapping described in the section methodology to compare both the ML model and the rating generated by Tinubu.

Comparing Fig. 5.6a, Fig. 5.6b and Fig. 5.6c, the results show that the mapping is significantly better for the models that have been developed for each sector than the model that englobes all the sectors. Tinubu's extreme classes (i.e., A and F) are better mapped using the sectorial approach proposed in this work.

Discussion

The analysis of the data provided by Tinubu shows that the percentage of companies going into default varies significantly among the different economic sectors. This discriminant factor may play an important role in the early detection of possible company failures at a one-year horizon. The comparison of the results between a global model and a more granular model by economic sector shows that there is a slight improvement in the ability of the models to discriminate between healthy firms and firms with a high probability of default (see Fig. 5.3 vs Table 5.1)). It is important to remark when comparing the ability of the model to reproduce the rating given by Tinubu, that there is an important improvement when developing a sectorial model (see Fig. 5.5)) (RQ1). On the other hand, and thanks to the SHAP explanatory framework, it is possible to observe the divergence that exists between the sectorial models in terms of the most important variables. Mainly what we show in this study is that, except for the wholesale sector, all the models consider the Tinubu rating as the most important factor when predicting companies' default. However, for the wholesale and retail trade sector, this variable is the fifth most important variable, which leads us to think that perhaps for that sector the Tinubu rating is not sufficiently discriminating. It is important to note that, as a general rule, the models use more short-term financial variables (i.e., EBE, Retained Earnings, Current Ratio) which can be considered correct analysis from a financial point of view since the default is predicted for 12 months. (RQ2) Last but not least, after applying the mapping proposed in this chapter, the comparison of the sectorial models concerning the global model shows a better alignment with Tinubu's credit risk model.

Conclusion

In this chapter, we have proposed a different approach for the creation of ML-based bankruptcy models. We have focused on the study of the sectorial behavior of firms to determine the probability of bankruptcy in one year.

Chapter 6

Credit Risk Scoring Forecasting using a Time Series Approach

Introduction

In this chapter, we diverge significantly from the approach taken in previous chapters by concentrating specifically on companies with a long history of credit ratings. The primary objective here is to predict the future credit rating of these companies.

When it comes to companies, the set of financial data provided by companies differs from country to country [START_REF] Stability | Review of country-by-country reporting requirements for extractive and logging industries[END_REF]. It depends on the financial regulation adopted by the country in which the company is based. An important feature when analyzing companies is whether they are publicly traded or not. While public companies must publish audited financial statements (in most countries according to the International Financial Reporting Standards (IFRS)), the data available and analysis of their creditworthiness are completely different with respect to Small and Medium Enterprises (SME). In some countries like the United States, SMEs have no requirements [START_REF]SME accounting requirements: basing policy on evidence[END_REF] in terms of publishing their financial data. In other cases where the data is partially available (i.e., some financial features are missing). In [START_REF] Florez-Lopez | Effects of missing data in credit risk scoring. A comparative analysis of methods to achieve robustness in the absence of sufficient data[END_REF] different data imputation techniques have been used in order to address this problem. This work has focused its efforts on a consumer's credit scoring. The latter has been widely treated in the literature ( [START_REF] Giudici | Network Based Scoring Models to Improve Credit Risk Management in Peer to Peer Lending Platforms[END_REF], [START_REF] Wang | Improving Investment Suggestions for Peer-to-Peer Lending via Integrating Credit Scoring into Profit Scoring[END_REF]). Some efforts have been made in order to create a credit scoring system for companies [START_REF] Provenzano | An Artificial Intelligence approach to Shadow Rating[END_REF]. Nonetheless the focus on previous studies has been to build highly accurate Machine-Learning (ML) based models using a large list of financial features [START_REF] Qadi | Feature contribution alignment with expert knowledge for artificial intelligence credit scoring[END_REF]. The problem with this approach is that they consider companies as data points and do not take into account potential trends when assessing the PD of a company.

Artificial intelligence (AI) has been widely used in the financial industry for financial In this chapter we present several contributions that are listed as follows:

• We analyze a large list of SMEs based mainly in Southern Europe.

• Our proposed framework analyzes the past behavior of companies and does not consider companies as an isolated data point.

• We forecast the rating of several companies using both a statistical traditional model (i.e., ARMA) and a ML-base model (i.e., Gradient Boosting).

• The proposed framework does not depend on the financial data, it depends on the historical behavior of companies.

• We analyze the results of the model using an out-of-time sample and compare them with the rating given by a company specialized on credit scoring.

Related Work

In this section we present the main works that address machine learning for credit scoring as well as the application of AI-based models for financial forecasting.

Credit Scoring

As defined in the previous section, credit scoring aims at measuring the risk for a bank or, more generally, a credit institution to grant a loan to an applicant. The most widely used algorithms for assessing the PD are logistic regression and linear discriminant ( [START_REF] Altman | FINANCIAL RATIOS, DISCRIMINANT ANALYSIS AND THE PREDICTION OF CORPORATE BANKRUPTCY[END_REF], [START_REF] Memic | Assessing Credit Default using Logistic Regression and Multiple Discriminant Analysis:Empirical Evidence from Bosnia and Herzegovina[END_REF]).

The main reason why these machine learning-based models have been widely adopted in the financial industry is their simplicity, their ease of use. The latter models are very limited since they are not able to capture nonlinear relationships between features. This limitation has been addressed in the literature by applying more sophisticated machine learning-based models: Random Forests [START_REF] Breiman | Random forests[END_REF], Gradient Boosting ( [START_REF] Ke | Light-GBM: A Highly Efficient Gradient Boosting Decision Tree[END_REF], [START_REF] Chen | XGBoost: A Scalable Tree Boosting System[END_REF]) and kernel-based algorithms such as Support Vector Machine (SVM) [START_REF] Baesens | Benchmarking state-of-the-art classification algorithms for credit scoring[END_REF]. For credit scoring applications, ensemble ML-based models have shown an impressive increase in terms of accuracy when predicting whether a customer will repay the loan or not [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF].

Forecasting in the Financial Industry

Financial time series forecasting has been a hot topic during the last decade. With the rise of machine learning and deep learning models, researchers have been focused on applying these models to predict the evolution of different stock markets( [START_REF] Niaki | Forecasting S&P 500 index using artificial neural networks and design of experiments[END_REF], [START_REF] Menon | NSE Stock Market Prediction Using Deep-Learning Models[END_REF]). In some sense, the stock price evolution of a company can be interpreted as what the market thinks of the activities developed by the company, and thus its credit worthiness. Indeed, when there is an important event that may negatively affect the activities of a certain company, the stock price is trending downwards.

Methodology

In this section, we present the method used in this study. We start by presenting the raw data. Then we show the procedure employed to transform the data in order to feed the models. Finally, we introduce the models used for this work and the metrics employed to evaluate the models.

Data

The data used for this work has been provided by Tinubu Square, a company which provides credit risk assessments of potential trade partners to its customers. Tinubu Square has an internal credit risk model that has been used for 20 years to assess the creditworthiness of a company. This internal model uses a large list of financial variables to compute a score. Then, the score (i.e., EVL_LEVEL variable) is mapped to a letter which is the final result of Tinubu's internal model. This rating represents the probability of default in a descriptive way. Highly risked companies have lower scores and are represented with the rating letter F. On the other hand, companies whose PD is close to zero are represented with higher scores and with the rating letter A.

The purpose of this work is to create a forecasting model to predict the rating evolution of companies in Tinubu's portfolio. In order to predict the future rating of a company, we need to convert the original dataset into a time series dataset. Each row of this new dataset represents a company and the columns represent the year the company has been rated.

Forecasting Time Series

In this part of the study we present the models proposed to address the problem of forecasting the creditworthiness of a company.

AutoRegressive Moving Average

AutoRegressive Moving Average (ARMA) is a combination of two different models: an autoregressive model (AR) and a moving average model (MA). Mathematically, ARMA processes result from the sum of both processes an AR of order p and MA of order q. An ARMA(p,q) model combines both the AR(p) and MA(q) models as follows:

The AR model assumes that we can model a time series x t using the last p observations of the given times series plus an additional term, the white noise error t (see eq.6.1).

AR(p)

: x t = c + p i=1 φ i x t-i + t (6.1)
On the other hand, the MA model considers that the current value of the time series x t is affected with the previous q white noise errors (see eq.6.2).

M A(q) : x t = µ q i=1 θ i t-i + t (6.2) 78 6.3. Methodology

x t = c + p i=1 φ i x t-i + t + q i=1 θ i t-i (6.
3)

The parameters p and q can be determined using different methods. By observing the graph of the Autocorrelation Function (ACF) and the Partial Autocorrelation Function (PACF) both parameters can be estimated. In this work we will estimate the parameters using a more analytical approach by computing the Akaike Information Criterion (AIC) [START_REF] Akaike | A new look at the statistical model identification[END_REF]. AIC is a statistical measure that allows the comparison between statistical models to determine which model best fits the data series. AIC considers both model goodness of fit and model complexity.

AIC = 2k -2 log(L) (6.4)
where k is the number of the parameters of the statistical model and L is the maximum value of the likelihood function for the model. The firs term represents the complexity of the model while the second term in Eq (6.4) represents how well the model fits the data.

Gradient Boosting

The eXtreme Gradient Boosting (XGBoost) is a machine-learning based algorithm that consists of a sequential combination of weak learners that corrects the errors of the previous weak learner.

XGBoost is an open-source framework proposed by [START_REF] Chen | XGBoost: A Scalable Tree Boosting System[END_REF] that has been widely used in machine learning competitions.

The outcome of a XGBoost composed of K weak learners for a data set with n instances and m features D = (x i , y i )(|D| = n, x i ∈ R m , y i ∈ R) is represented mathematically as follows:

ŷi = Ω(X i ) = K k=1 f k (X i ), f k ∈ F (6.5)
where F = {f (x) = w q(x) }(q : R m → T, w ∈ R T ) is the space regression of trees. T is the number of leaves in the tree and q represents the structure of each tree that maps an example to the corresponding leaf index [START_REF] Chen | XGBoost: A Scalable Tree Boosting System[END_REF]. For each independent tree f k there is an independent q and leaf weights w q .

L t = i (y i , ŷi ) + k Ω(f k ) (6.6)
where l is the loss function, y i is the target value and ŷi is the prediction.Ω is a term that penalizes the complexity of the function. This term is introduced to avoid overfitting.

For the hyperparameter estimation, we used scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] to perform a Grid Search, which consists of creating all possible hyperparameter combinations of a list of values predetermined by the user.

Comparing the Forecasted Values

To assess the performance of both proposed models, we will split the dataset into two different parts: the train set will contain the score, and thus the rating, of companies during the period 2008-2017. The rest of the dataset, the test set, will be composed by the rating of the same companies scored by Tinubu during the period 2018-2019 (see Fig. 6.2).

We will compare graphically the results of both models. For the ARMA model, we will keep one ARMA by company with the hyperparameters p and q that minimize the AIC for the time series between the period 2008-2017. For each model we generate two predictions that correspond to the years 2018 and 2019. Then we compare the forecast of both models graphically.

The two models proposed in this chapter for forecasting the risk class of companies are different by nature. For the ARMA model we need to create several models with hyperparameters p and q and choose the one whose AIC value is the lowest. Then we will present the difference between models prediction and the Tinubu's Score, the target variable.

The prediction of the model with respect to the actual value of the test set using gr. This is a regression metric that estimates the distance between the true value and the predicted value. This metric is not used for regression problems where the target variable could take values close to zero. Since the target variable (i.e., Tinubu's Score) is in the range the MAPE is an interesting measure to use. MAPE is defined as follows:

Results

In this section we present the results of the models that have been described in the previous section. First, we present the results of each model individually and then we compare the forecasted ratings for both models using several companies.

Assessing the Performance of ARMA

As we mentioned previously, we will create several ARMA models with different hyperparameters (i.e., p and q). For instance, we created 9 different ARMA models and considered the best model the one with the lowest AIC. In table 6.2, we show the results obtained for the company with the id 462792. For this particular company the hyperparameters with the lowest AIC are p = 2 and q = 3. When we forecast Tinubu's Score for the years 2018 and 2019 we observe that the model underestimates the risk Fig. 6.3. The ARMA captures the smooth deterioration of the company for the year 2018. However, the difference between the model in 2019 is significant.

Forecasting Tinubu's Score Using XGBoost

In this part of the study, we present the results obtained with XGBoost. In Table 6.3 we show the optimal hyperparameters found using the Grid Search optimization.

As we can see in Fig. 6.4, the XGBoost considers that the company with the ID 462792 will slightly improve during the next two years. The rating given by Tinubu and the rating generated by our XGBoost model diverges for the year 2019. This difference Table 6.2: For each company in the dataset, we create 9 models. The best model is the one with the lowest AIC value. The AIC has been computed over the train set which consists of companies rated every year by Tinubu Square during the period 2008-2017. We show the difference between the predicted rating and the real rating. p and q are the hyperparameters of the ARMA model.

LGE_ID can be explained by the fact there has been an event that has adversely affected the company's activity.

Analyzing the Models Ratings

As we mentioned in previous sections, Tinubu's rating system reflects the PD in a descriptive way. In this section we will present several examples of comparison between the behavior of both models for different companies with the rating given by Tinubu for the test period (see Fig. 6.2).

In Fig. 6.5(a) we compare the xgboost with an ARMA with p, q = 2. The results show that, in this particular case, both models slightly underestimate the risk. On the other side, we observe that both models overestimate the risk.

For the company 684690 (see Fig. 6.5(b)), the results show a divergence between the ARMA model with p, q = 1 and the XGBoost model. The latter can capture the evolution of the rating during the years 2018 and 2019 with a narrow difference between the predicted score and the Score given by Tinubu. On the other side, the ARMA model, even if it captures the trend, clearly overestimates the company's creditworthiness.

Nonetheless there are cases where both models underestimate the risk. This is the case in Fig. 6.6(a). Both models consider the creditworthiness of the company for the next two years to have a downward trend.

(a) Predicted rating evolution for company 55190 (b) Predicted rating evolution for company 684690 Figure 6.5: Comparison of the forecasted Tinubu's rating for the period 2018-2019 for the companies with the ID 55190 and 684690 respectively using the proposed models in this work: ARMA and XGBoost Another relevant case is presented in Fig. 6.7(b). In this one, the ARMA with p = 1, q = 3 highly underestimates the risk by giving an A rate to a company for which Tinubu's rating for the year 2019 is E.

Conclusions

In this work we proposed a framework for the credit scoring that differs from previous work. The approach considered consists of forecasting the credit rating of a large number of companies using an historic index given by Tinubu Square for the period 2008-2019. We used the period 2018-2019 to compare both forecasting models with the target value which is Tinubu's rating. We observed that for the set of companies analyzed, both models tend to slightly underestimate the risk. For companies specialized in credit risk, models that overestimate the risk are preferable to models that underestimate the risk.

Conclusions

This is mainly because underestimating the risk leads to potential economic losses. On the other hand, models that are strongly conservative (i.e., models that overestimate the risk) are not desirable for credit scoring since they impact the economic activity by reducing the volume of credits approved and thus decreasing the business between SMEs. Considering this criteria and observing the results of our work we can conclude that the machine learning-based models are closer to Tinubu standards in terms of credit scoring. It is important to remark on the fact that wealthy companies (i.e., companies with the rating classes A-B) are more stable over the period analyzed than companies with a lower rating. Future work should focus on the adequate evaluation of the performance of the models proposed in this work. It will be interesting to introduce deep learning models and compare them with results yielded by the models proposed.

Chapter 7

Multimodal Credit Risk Scoring

Introduction

As highlighted in the introduction, one of the significant challenges in credit scoring is the lack of comprehensive financial data. This scarcity often hampers the effectiveness of traditional credit scoring models, which rely heavily on financial metrics. To address this issue, researchers in the field are increasingly exploring alternative types of data that can supplement or even replace traditional financial indicators.

Traditional credit scoring models primarily rely on tabular financial data, such as credit history, income, and debt-to-income ratio [START_REF] Addo | Credit Risk Analysis Using Machine and Deep Learning Models[END_REF]. However, these models may not capture the full picture of a borrower's creditworthiness, especially when dealing with complex and dynamic markets.

Textual data, such as news articles, press releases, and social media posts, can provide valuable insights into potential risks that may impact a borrower's ability to repay a loan [START_REF] Niu | Credit Scoring Using Machine Learning by Combing Social Network Information: Evidence from Peer-to-Peer Lending[END_REF]. By combining these textual risk assessments with traditional financial data, financial institutions can improve the accuracy and reliability of credit risk scoring algorithms, leading to better lending decisions and reduced credit losses.

In this context, we propose a new framework that intends to leverage the non-structured information available in the comments made by risk analysts by combining it with tabular financial data.

The chapter is structured as follows: first, we present the previous research in the field of credit scoring and in Natural Language Processing (NLP). Then, we present our framework and the results yielded. Finally, we present the conclusion and potential hints for future work.

We compare the impact of adding the text feature using different Natural Language Processing techniques on the performance of several state-of-the-art models.

Related Work

Credit Scoring

Credit scoring is an important tool for financial institutions and is used to assess the creditworthiness of potential borrowers. Classical credit scoring models are based on statistical techniques such as Linear Discriminant Analysis, Logistic Regression ( [START_REF] Altman | FINANCIAL RATIOS, DISCRIMINANT ANALYSIS AND THE PREDICTION OF CORPORATE BANKRUPTCY[END_REF], [START_REF] Sohn | Technology credit scoring model with fuzzy logistic regression[END_REF]).

Recently, there is a new wave of Machine Learning based algorithms that are used to estimate the probability of default. In [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF], the authors have shown that high-level sub-symbolic algorithms outperform statistical-based models.

Recent works focus on the use of financial tabular data for predicting companies' future status (i.e., default/non-default). The authors in [START_REF] Addo | Credit Risk Analysis Using Machine and Deep Learning Models[END_REF], center their efforts on predicting the default using a set of 81 financial features. They conclude that among all tested models, tree-based models are the best in terms of performance. In [START_REF] Bussmann | Explainable AI in Fintech Risk Management[END_REF], the authors employed a similar approach for predicting the default of companies one year later after they published their financial sheets. An interesting different approach has been employed by the authors in [START_REF] Qadi | Feature contribution alignment with expert knowledge for artificial intelligence credit scoring[END_REF]. They compare the output of a machine learningbased credit scoring model with the rating given by a credit risk company. The results show that the model distinguishes extremely rated companies.

A large number of works have shown that XGBoost [START_REF] Chen | XGBoost: A Scalable Tree Boosting System[END_REF], a boosting method, outperforms classical machine-learning-based methods (i.e., logistic regression, decision trees neural networks) for the task of predicting the default.

Natural Language Processing

There have been many significant advances in the field of natural language processing (NLP) in recent years. One of the most important developments has been the advancement of deep learning (DL) techniques [START_REF] Alzubaidi | Review of deep learning: concepts, CNN architectures, challenges, applications, future directions[END_REF], which have led to significant improvements in the accuracy and effectiveness of NLP models [START_REF] Otter | A Survey of the Usages of Deep Learning in Natural Language Processing[END_REF].

During the 1980s, the field of NLP experienced the first revolution with the introduction of statistical models such as Hidden Markov Models (HMM) [START_REF] Rabiner | A tutorial on hidden Markov models and selected applications in speech recognition[END_REF] for speech recognition and N-grams models [START_REF] Brown | A statistical approach to machine translation[END_REF] for machine translation. Afterward, and as a result of the exponential growth of computational power, the deep learning models have rapidly

Data Overview

The dataset used in this work is provided by Tinubu, a company specializing in credit risk assessments. The original dataset contains 4951 credit assessments. For each assessment, we have 34 standard features that represent the companies information (e.g., creation year, the country in which it operates), its financial statements (e.g., net worth, current assets), and a text feature (e.g., risk analyst comments), that typically contains a description of the company, the business context, and its activities. Each data point corresponds to the year in which the company has been assessed by credit risk analysts. Word2vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF], [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF]. The target variable of our models is whether the company is in financial embarrassment1 or being out of business. We create the default variable by setting it to 1 if the considered company is in financial embarrassment or out of business the year after the assessment and 0 otherwise.

Data Preprocessing

The preprocessing step can be split into two main parts depending on the applied text feature model. For GLoVe and Word2vec, we reduce the dimensional space of the text embedding using Principal Component Analysis ( [START_REF] Jolliffe | Principal component analysis: a review and recent developments[END_REF], [START_REF] Raunak | Effective dimensionality reduction for word embeddings[END_REF]). For non-tree-based ML models, we scale the data using Standard Scaler [START_REF] Buitinck | API design for machine learning software: experiences from the scikit-learn project[END_REF].

The categorical features (i.e., the country in which the company is based) are encoded using the one-hot encoding technique. This method creates a new feature in the dataset for each feature label. It takes the value one if the company is based on the considered country. It is important to remark that the categorical feature derived from applying the Sentiment Analysis model to the text feature is also encoded using the same technique.

Finally, once we preprocessed our data, we split the dataset into three parts: an outof-time dataset which includes the data for the period 2017-2019 that will be employed to valid model performance; and a train/test dataset (i.e., 70% for training) that covers the period (2010-2016).

Models

The following part presents the state-of-the-art machine learning-based models trained to predict the default the year after the assessment using the data processing pipeline explained in the previous section. The outcome of the models is the probability of being in out of business or financial embarrassments the year after the risk assessment. This probability is mapped to a binary value, 1 (i.e., default) if the probability is greater or equal to 0.5.

Logistic Regression: we implement the logistic regression using the Ridge regularisation that helps the model prevent multicollinearity. We also set the class weight parameter to balanced to adjust weights inversely proportional to class frequencies.

Random Forest:

We set the number of trees to 100 and the class weight to balanced to deal with the imbalance in the target variable. s represents the embedding dimension (i.e., s=200 for GLoVe and s=300 for Word2Vec). The variable r represents the word embedding space after the dimension reduction (i.e., r=5).

Support Vector Machine (SVM):

.

Neural Network:

We trained a neural network using stochastic gradient descent. The neural network is composed by 4 hidden layers with 128, 256, 512, and 128 neurons. The activation function is the ReLU function.

Model Performance

Two different strategies have been used to evaluate the performance of the developed models. The first model evaluation comes from the 5 cross validation that has been employed using the train/test dataset. We also compare the performance of the models using the out-of-time sample.

For the train/test set, we employed three different classification metrics: precision, recall, and F1-Score. Precision measures the proportion of true positives among all predicted positives. Recall measures the proportion of true positives that were correctly identified. F1-Score is calculated as the harmonic mean of precision and recall.

Results

Exploratory Data Analysis

Our approach starts by analyzing the target variable and the text feature which is the risk analyst comments (e.g., see Fig. 7.1). As we can see in Fig. 7.2, the default rate experience a downward trend from 2010 to 2018. However, during 2014 and 2017 the default rate did not fit the trend.

In Fig. 7.3, we show the length distribution of the analyzed risk analyst comments. The median of this distribution is 693 characters. Just 10% of the risk analysis has a length bigger than 5 pages (i.e., 2600 characters).

Model Performance

For straightforward comparison, we employ the most commonly used classification evaluation metrics that lie in [0,1]. Higher values indicate superior performance. Tab. 7.2 display the accuracy of the different models with the different text embedding techniques. The overall performance of the models improves when we raw embed the text using Word2vec and GLoVe, especially for the logistic regression and the SVM.

The imbalanced characteristics of the dataset may bias the models to predict the most represented class (i.e., the non-defaulted class). In Tab. 7.3, we compare the precision of

Conclusions

In conclusion, this chapter proposes a new approach to credit scoring that combines financial tabular data with credit risk textual assessment using word embedding techniques and sentiment analysis. Moreover, we addressed the problem of representing the text in a high dimensional space by using dimension reduction techniques (i.e., Principal Component Analysis).

Experimental results demonstrate that the addition of the textual feature slightly improves model performance. This suggests that incorporating credit risk textual assessment can provide additional information to financial institutions for more informed credit decisions. Furthermore, we experimentally demonstrate that the sentiment analysis approach tends to yield better results in comparison to the word embedding techniques. This is due to the fact that word embeddings, in essence, map words into a high-dimensional space where semantically similar words are placed closely together. However, these techniques focus primarily on individual words or at most, phrases. They do not inherently account for the larger syntactic or semantic context that extends beyond individual words and thus they might not adequately encapsulate the overall sentiment of a text, leading to a loss of crucial information.

In addition to improving model performance, a performant credit scoring system that combines financial tabular data with credit risk textual assessment can help credit risk analysts to focus on more critical cases, leaving the smaller cases to the system. The framework proposed in this chapter offers a promising approach for credit scoring and can be applied to other text classification tasks in finance and beyond. Future research should explore the use of more advanced techniques for sentiment analysis and word embedding (i.e. LSTM [START_REF] Gers | Learning to Forget: Continual Prediction with LSTM[END_REF], GRU [START_REF] Chung | Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling[END_REF]), as well as the incorporation of other qualitative factors such as news articles and social media data to further improve model performance.

Chapter 8. Predicting Corporate Solvency using Sentiment Analysis of Risk Analyst Textual Assessments

In this paper, we introduce a novel framework that incorporates credit risk text assessments made by credit analysts to fine-tune a pre-trained financial sentiment analysis model. Finally, we compare the sentiment analysis of the text with the real rating given by the risk analysts. This chapter is presented as follows: first, we present recent progress in the field of credit scoring, and more specifically, the intersection between this field and the Natural Language Processing field. Then, we explain step-by-step the processes used to prepare the data and to train the model. The next stage of this paper consists of showing and analyzing the results yielded by the proposed framework. Finally, we discuss the results and potential improvements.

Related Work

Artificial Intelligence in Credit Scoring

Artificial intelligence (AI) has emerged as a promising tool for credit scoring due to its ability to handle large amounts of data, learn from patterns, and make accurate predictions. Several studies have explored the use of AI models in credit scoring and their potential benefits.

In [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF], the authors have shown that the random forest model [START_REF] Breiman | Random forests[END_REF] outperforms classical machine-learning algorithms (ML). Although the authors assert that gradient boosting methods usually produce superior results, they suggest that the use of deep learning should be taken into consideration due to its greater dependence on the specific financial application [START_REF] Schmitt | Deep learning vs. gradient boosting: Benchmarking state-of-the-art machine learning algorithms for credit scoring[END_REF].

In another study conducted in [START_REF] Addi | An ontology-based model for credit scoring knowledge in microfinance: Towards a better decision making[END_REF], the authors review multiple models to identify elements that influence credit scores in a microfinance setting, with the aim of constructing an ontological model to assist institutions in decision-making progress.

In the realm of credit scoring research, prticularly focusing on company credit scoring, several recent studies need to be mentioned. In [START_REF] Bussmann | Explainable AI in Fintech Risk Management[END_REF], the authors investigate using a subset of data to predict the likelihood of default within one year. This study utilizes an XGBoost model, achieving promising results in predicting default probabilities.

Another interesting approach proposed in [START_REF] Provenzano | Machine learning approach for credit scoring[END_REF] aims to reproduce rating agency ratings by employing a comprehensive list of financial variables. This approach demonstrates the potential for replicating the complex decision-making processes of rating agencies with machine learning models.

Natural Language Processing

NLP is becoming increasingly important for the finance industry due to the vast amounts of unstructured data that are generated in this field. Financial institutions generate huge volumes of data in the form of news articles, social media posts, financial statements, and other documents, making it difficult for humans to analyze and extract insights effectively [START_REF] Goldstein | Big data in finance[END_REF].

Different applications of NLP techniques have been studied. In [START_REF] Muthukumar | A stochastic time series model for predicting financial trends using nlp[END_REF], the authors combine sentiment analysis and financial numeric data to predict short and long-term aerospace stock trends. Another study has shown the effectiveness of incorporating sentiment features in forecasting models [START_REF] Gupta | Sentiment analysis for stock price prediction[END_REF]. An interesting approach has been introduced by [START_REF] Zheng | Doc2edag: An end-to-end document-level framework for chinese financial event extraction[END_REF]. In this work, the authors have developed an end-to-end model that uses formal documents (e.g., news articles) to extract and update information about customers. The model is composed of a transformer encoder [START_REF] Vaswani | Attention is all you need[END_REF] and the BiLSTM-CRF event recognizer [START_REF] Huang | Bidirectional lstm-crf models for sequence tagging[END_REF].

One of the most widely used models in NLP is Bidirectional Encoder Representations from Transformer (BERT). It is a pre-trained model developed by Google and based on the transformer architecture [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]. In [START_REF] Hazourli | FinancialBERT -A Pretrained Language Model for Financial Text Mining[END_REF], the authors present FinancialBERT, a domain-specific language representation model pre-trained on a large financial corpus.

Methodology

In this section, we present the proposed framework. During the first stage, we prepare the text that will be used to train the model. Then, we split the data into two datasets: the train set (i.e., 70% of the data) and the test set (i.e., 30%). The next step consists on fine-tune a pre-trained sentiment analysis model. Finally, we evaluate the performance of the model using the test set. Chapter 8. Predicting Corporate Solvency using Sentiment Analysis of Risk Analyst Textual Assessments

Data

As mentioned in in the introduction and in previous chapters the data used for this work has been provided by Tinubu1 . One of the main activities at Tinubu is to quantitatively give credit risk recommendations. The risk analysts analyze the company using different information (i.e., financial statements, social media, news) and then they write a report in which they synthesize the company and give a recommendation in the form of a rating that range from A (i.e., very low default risk) to F (i.e., extremely high default risk).

Model Architecture

The architecture of the model used is inspired by the FinancialBERT model [START_REF] Hazourli | FinancialBERT -A Pretrained Language Model for Financial Text Mining[END_REF].

The model is fine-tuned on a sentiment analysis task using a supervised learning approach. The fine-tuning architecture used is the same as that used in BERT, with a dense layer added after the last hidden state of the [CLS] token. The model is trained on the labeled credit risk text assessments. As mentioned previously, each text is labeled depending on the final rating given by the expert. The loss function used is cross-entropy. In our particular case, high risked companies should be labeled by the model as negative, neutral for companies with moderate risk and low-risk companies with a positive label.

The training is performed with a batch size of 8, a maximum sequence length of is equal to the longest text assessment, and a learning rate of 2e-5 for 5 epochs.

Experimental Context

We define three different experiments:

Fine-tuned FinancialBERT

In this experiment, we use the fine-tuned FinancialBERT model to classify financial text into three different classes: neutral, positive, and negative. We map each class into a risk

Sentiment Analysis using Phrase-Level Processing

We propose in this work a second approach in which we compute the sentiment analysis of the fine-tuned model over each phrase of the text. We define three different approaches Let's define the target feature as follows:

y i =        1 Positive 0 Neutral -1 Negative (8.1)
where y i is the label for each phrase. Then we also define Y as the label of the complete text. The output of the model is a probability of belonging to each class P[y i ] = P i 1. Mean Aggreagation: Y = N i=0 y i P i × where N is the number of phrases present in the text.

2. Linear Growing Importance: Y = N i=0 w i P i y i . w i is the importance of each phrase and is calculated

w i = i N (N -1) + 1 2N
3. Exponential Growing Importance: Y = N i=0 w i P i y i . w i is the importance of each phrase and is calculated w i = r i (1-r) 1-r N

Sentiment Analysis using End-of-Text Content

This approach is based on the idea that the final phrases of a text can provide valuable information about the overall sentiment expressed in the text. To test this hypothesis, we compute the sentiment analysis of the last paragraph on credit risk assessments.

Results

This section is structured as follows: first, we compare the labels generated by the finetuned FinancialBERT with the labels given associated with each risk assessment. Then, we compare the results of Fine-Tuned model using three different Phrase-Level strategies proposed in this study.

Fine-Tuned FinancialBERT

In Fig. 8.3, The results of the confusion matrix indicate that the model is overestimating the class "neutral". This means that the model is frequently predicting that the sentiment of the text is neutral, even when it is actually positive or negative. This overestimation of Textual Assessments the neutral class in the confusion matrix could be explained by the neutral tone often used by risk analysts in their texts. Risk text analysis tend to avoid expressions of emotions or bias, thus it is complicated for the model to find the correlation between the risk classes and the sentiment analysis of the text. It is worth noting that the extreme errors in the confusion matrix are not very high. This means that the model is not making a large number of extreme errors, such as confusing positive sentiment with high-risk companies.

Pharse-Level Sentiment Analysis

In this section, we compare the three different methods to compute the sentiment analysis using a phrase-level approach. It is worth mentioning that the model used is the fine-tuned FinancialBERT.

Comparing Fig. 8.3 and Fig. 8.4, we observe that the phrase-level sentiment analysis improves the ability to predict the extreme positive class. This indicates that the model is better able to identify positive sentiment in the text data. However, the model overestimates the risk of bankruptcy for high-risk companies and categorizes them as neutral.

Among the three phrase-level strategies used, it is worth highlighting that the strategy that produces the best results is one where the weighting of each phrase grows exponentially (see Fig. 8.4c) This can be explained by observing the typical structure of the analyses performed by experts (see Fig. 8.1). The first part of the text usually summarizes the entity. In the second paragraph, the figures are presented. Finally, the expert closes the text with his or her opinion on the solvency of the company and whether it is correct to grant it credit. This structure of the text highlights the importance of assigning greater weight to the last phrases, as they often contain the expert's opinion and assessment of the company's solvency. By giving greater weight to these phrases, the model can better capture the sentiment expressed in the text, leading to improved performance.

End-of-Text Sentiment Analysis

As observed in the previous results (see Fig. 8.4), the best results are yielded by the exponential growing importance approach. This means that the last phrases of the text tend to better discriminate the risk associated with the text. The improvement in the results in Fig. 8.5 compared to Fig. 8.3 can be explained by the fact the end-of-text strategy provides a focused and relevant representation of the sentiment expressed in the text. By using the last paragraph, the model is able to effectively determine the sentiment of the text.

Conclusion and Perspectives

In the introduction of this thesis, we emphasized the pivotal role that credit scoring systems play in fostering economic growth. Moreover, we showed that, from governmental (i.e., Banque de France) to intergovernmental organizations (i.e., World Trade Organization) highlight the importance of robust and ethical credit scoring.

In this thesis, we initiated our investigation by scrutinizing the impact of the COVID-19 outbreak on our dataset. The primary objective was to assess whether the pandemic had a substantial influence on the default rates of companies, which could potentially compromise the generalizability of our predictive models. We revealed that to achieve robust and generalizable default predictions, it is imperative to develop granular models. Specifically, we found that creating models tailored to individual trade sectors significantly enhances predictive accuracy. This granularity allows the models to capture the unique risk factors and economic dynamics that are specific to each sector, thereby making them more resilient to broad economic shocks, such as the COVID-19 pandemic.

In addition to the need for sector-specific granularity, one of the main contributions of this thesis is the importance of aligning the machine learning models with human risk criteria. This alignment is crucial for capturing the domain expertise that risk analysts bring to the table. This compatibility between machine-generated insights and human expertise serves to enhance the model's credibility, making it more acceptable to stakeholders and more effective in real-world applications.

In real-world scenarios, the variability in regulations concerning access to public data for small and medium-sized enterprises (SMEs) poses a significant challenge, especially at the country level. This variability often results in a lack of financial data, making traditional credit scoring methods less applicable. This leads us to the second major contribution of our framework: the development of credit scoring models that do not rely solely on financial data.

Our contribution is fundamentally rooted in the application of Natural Language Processing (NLP) to textual data generated by risk analysts. Recognizing the value of this expert-generated content, we have proposed various methods to systematically analyze Textual Assessments and interpret the text. These methods serve to extract meaningful insights and features that can be incorporated into our credit scoring models. Ultimately, we have developed a comprehensive framework that leverages these NLP-derived features to score companies that have not publicly disclosed their financial data. This framework not only enhances the robustness of our credit scoring system but also significantly broadens its applicability, especially for companies operating in regulatory environments where financial data disclosure is not mandatory.

Perspectives and Future Work

Future work should concentrate on several key areas to further refine and improve the framework. One primary focus should be the incorporation of credit risk expertise directly into the training loop of the machine learning models. This would serve to align the models more closely with the domain-specific knowledge and criteria used by risk analysts.

Incorporating the unique expertise of risk analysts into machine learning models is particularly important because each company has its own distinct approach to training these professionals. This specialized training is often a reflection of the company's specific organizational philosophies, risk tolerances, and even regulatory requirements. By embedding this domain-specific knowledge into the machine learning model, one essentially captures the nuances and subtleties that come with years of specialized training and experience.

Future efforts should also prioritize the creation of dynamic strategies for monitoring and updating machine learning-based credit scoring models. The aim is to ensure that these models can adapt swiftly to shifts in economic paradigms. Economic landscapes are constantly evolving due to various factors such as technological advancements, regulatory changes, and global events like pandemics or financial crises. Traditional models may become obsolete or less accurate when faced with such rapid changes. Therefore, it's crucial to develop adaptive algorithms that can quickly incorporate new data and insights, allowing for real-time adjustments to the model's predictive capabilities.

Another avenue for future investigation that remains unexplored following the work conducted in this thesis is online data, such as social media text data and news, for credit scoring. Utilizing these additional data sources could be particularly beneficial in scenarios where financial data is sparse or unavailable. By incorporating textual data from diverse and publicly accessible platforms, we can potentially create more robust and comprehensive credit scoring models. This approach could offer a significant improvement in the accuracy and reliability of credit assessments, especially in contexts where traditional financial metrics are lacking or incomplete

Figure 1 . 1 :

 11 Figure 1.1: Volume of structured data compared to unstructured dataduring the last decade( [3])

Figure 1 . 2 :

 12 Figure 1.2: FICO Score description ( [8])

  2.1).

Figure 2 . 1 :

 21 Figure 2.1: Illustration of the common intuition for the bias-variance tradeoff. Source:[START_REF] Fortmann-Roe | Understan-ding the Bias-Variance Tradeoff[END_REF]

Figure 2 . 2 :

 22 Figure 2.2: Loss functions described in this manuscript.

Figure 2 . 3 :

 23 Figure 2.3: Example of a decision tree in customer credit scoring[START_REF] Bengio | Decision trees do not generalize to new variations[END_REF]

Figure 2 . 4 :

 24 Figure 2.4: Visual Representation of the Transparency Levels Across Various Machine Learning Models: (a) Linear regression; (b) Decision trees; (c) K-Nearest Neighbors; (d) Rule-based Learners; (e) Generalized Additive Models; (f) Bayesian Models.Source:[START_REF] Arrieta | Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai[END_REF]

Figure 2 . 5 :

 25 Figure 2.5: Transformers architecture.Source:[START_REF] Vaswani | Attention is all you need[END_REF]

  (a) Wholesale and retail trade, repair of motor vehicles (b) Construction industry (c) Manufacturing sector (d) Accommodation and food service activities

Figure 3 . 1 :

 31 Figure 3.1: The right y-axis corresponds to the percentage of default in the sector (red line) and the left y-axis the total of companies belonging to the sector that have been assessed (green line). This analysis shows how the default by sector varies between 2018 and 2021.

Chapter 3 .

 3 Analyzing the Impact of the COVID-19 Outbreak on Companies Default Rates (a) Wholesale and retail trade, repair of motor vehicles (b) Construction industry (c) Manufacturing sector (d) Accommodation and food service activities

Figure 3 . 2 :

 32 Figure 3.2: The right y-axis corresponds to the percentage of default in the sector (red line) and the left y-axis the total of companies belonging to the sector that have been assessed (green line). This analysis shows how the default by sector varies over time (2008 till 2021).

Chapter 4 .

 4 Aligning Feature Contributions with Expert Knowledge in Artificial Intelligence-Based Credit Scoring

Figure 4 . 3 :

 43 Figure 4.3: Data preparation flowchart including the years over which train and test data are split.

Figure 4 . 4 :

 44 Figure 4.4:The columns of the confusion matrix represent the rating given by the Tinubu's system after the feature interpolation. Rows correspond to the rating given by Tinubu's algorithm with the original feature

Figure 4 . 5 :

 45 Figure 4.5: Confusion matrix for mapping Tinubu's Scorecard algorithm risk labels to the ML model probabilities.

Figure 4 . 6 :

 46 Figure 4.6: Contribution of each explanatory feature to the final prediction based on Shapley analysis of contribution decomposition for the default prediction. Features are sorted according to the their relevance (i.e., SHAP average absolute value)

Chapter 4 .

 4 Aligning Feature Contributions with Expert Knowledge in Artificial Intelligence-Based Credit Scoring

2 .

 2 RQ2. What are the features that drive the model to consider a company highly risked for the different sectors? Are those features the same for each sector? Is there a difference between sector models and the global economic model in terms of most relevant features?: We compare the main factors that lead the model to rate a company by sector and analyze the divergence between economic sectors.

Figure 5 . 1 :

 51 Figure 5.1: Companies distribution by sector.

  (a) Wholesale and retail trade, repair of motor vehicles (b) Construction industry (c) Manufacturing sector (d) Accommodation and food service activities

Figure 5 . 2 :

 52 Figure 5.2:The right y-axis corresponds to the percentage of default in the sector (red line) and the left y-axis the total of companies belonging to the sector that have been assessed (green line). This analysis shows how the default by sector varies over time.

  True Positives True Positives+0.5 * (False Positives+False Negatives) 5.4. Results (a) Wholesale and retail trade, repair of motor vehicles (b) Construction industry (c) Manufacturing sector (d) Accommodation and food sevice activities

Figure 5 . 3 :

 53 Figure 5.3: Comparison of the AUC ROC over the test set for XGBoost, LightGBM and Random Forest over the period 2008-2020

Figure 5 . 4 :

 54 Figure 5.4: Contribution of each explanatory feature to the final prediction (i.e., the XGBoost model based on Shapley. analysis of contribution decomposition for the default prediction.)

  (a) All sectors (b) Wholesale and retail trade, repair of motor vehicles (c) Construction industry (d) Manufacturing sector (e) Accommodation and food service activities

Figure 5 . 5 :

 55 Figure 5.5: Contribution of each explanatory feature to the final prediction (i.e., the XGBoost model) based on Shapley analysis of contribution decomposition for the default prediction for every analyzed economical sector.

  (a) All Economic Sectors (b) Wholesale and retail trade, repair of motor vehicles (c) Construction industry (d) Manufacturing sector (e) Accommodation and food service activities.

Figure 5 . 6 :

 56 Figure 5.6: Comparison of the Rating given by Tinubu's Risk Analysts for 4 different economic sectors with the mapped PD of the XGboost model trained using companies of the given sector. The values shown are normalized by rows, and represent the fraction of ratings given by the human risk analyst (i.e., the rows) that matches the rating given by the XGBoost model (i.e., the columns).

Figure 5 . 7 :

 57 Figure 5.7: Comparison of the Rating given by Tinubu's Risk Analysts with the mapped PD of the XGboost model. The values shown are normalized by rows. This represent the fraction of ratings given by the human risk analyst that matches the rating given by the XGBoost model.

Chapter 6 .Figure 6 . 1 :

 661 Figure 6.1: Example of two companies rated by Tinubu Square every year during the period 2008-2019. The EVL_LEVEL represents the score given by Tinubu before being mapped to a rating class. The FIN_STATEMENT_DATE is the date the company has published its financial statements.

Figure 6 . 2 :

 62 Figure 6.2: Proposed splitting strategy

Figure 6 . 3 :

 63 Figure 6.3: Results of the prediction of the rating of the two next years for company 462792 using an ARMA model with hyperparameters p=2, q=3 which are the hyperparameters with the lowest AIC for models trained with the data between (2008-2017).

Figure 6 . 4 :

 64 Figure 6.4: Rating prediction for the company 462792 for the years 2018 and 2019 using XGBoost with the hyperparameters presented in Table 6.3.

Figure 6 . 6 :Figure 6 . 7 :

 6667 Figure 6.6: Comparison of the forecasted Tinubu's rating for the period 2018-2019 for the companies with the ID 162009 and 165482 respectively using the proposed models in this work: ARMA and XGBoost

Figure 7 . 1 :

 71 Figure 7.1: Text assessment example generated by a risk analyst.

Figure 7 . 2 :

 72 Figure 7.2: Number of companies assessed each year. The default rate represents the percentage of companies that will experience financial difficulties the year after the assessment.

  In this case, we have set the hyperparameter class_weight = 'balanced'. The other hyperparameters are the default ones. XGBoost: This algorithm employs a sequential combination of 1000 weak learners. The hyperparameters of the xgboost are the follow: n_estimators = 1000, learning_rate = 0.1 and scale_pos_weight = 12. The latter help the model with the imbalance dataset nad is usually set to #P ositive Instances #N egative Instances . LightGBM: We set three different hyperparameters for this gradient boosting-based model: n_estimators = 100, learning_rate = 0.1 and class_weight = 'balanced'.

  (a) Text processing using word embedding (b) Text processing using word embedding and dimension reduction (c) Text processing using sentiment analysis

Figure 7 . 4 :

 74 Figure 7.4: Conceptual overview of the proposed framework. m, n represent respectively the number of assessments and the number of financial features available in the dataset.s represents the embedding dimension (i.e., s=200 for GLoVe and s=300 for Word2Vec). The variable r represents the word embedding space after the dimension reduction (i.e., r=5)..

Figure 8 . 1 :

 81 Figure 8.1: Text assessment example generated by a risk analyst

Figure 8 . 2 :

 82 Figure 8.2: Logarithmic Distribution of Text Size (in Words)

Figure 8 . 3 :

 83 Figure 8.3: The columns represent the sentiment analysis of the Fine-Tuned Financial-BERT. The rows represent the risks associated with each assessed company

  (a) Mean phrases strategy (b) Linear Growing Importance (c) Exponential Growing Importance

Figure 8 . 4 :

 84 Figure 8.4: The columns represent the sentiment analysis of the model. The rows represent the risks associated with each assessed company

8. 4 .

 4 Resultsto label the text the split.

Figure 8 . 5 :

 85 Figure 8.5: The columns represent the sentiment analysis of the End-of-Text Fine-Tuned FinancialBERT. The rows represent the risks associated with each assessed company

Table 4 .

 4 1: Datasets used for companies credit scoring modeling using machine learningbased models. Imbalance Ratio is the ratio between the instances of the majority class and the instances of the minority class.

	Dataset Reference	Dataset Size	Features	Imbalance Ra-
				tio
	Bussmann et al. 2020 [57]	15,045	Not specified	8.17
	Addo, Guegan, and Hassani 2018 [104] 117,019	181	65.67
	Provenzano et al. 2020 [99]	919,636	179	65
	Moscatelli et al. 2020 [101]	∼ 250,000	26	65
	Dataset used in this chapter	138,419	15 (Table 4.7)	114.75

  Solvency R 1 :

		Net Worth
		Total Assets
	Solvency R 2 :	Financial Debt Gross Income
	Liquidity R 1 :	Total Current Assets Total Current Liabilities
	Liquidity R 2 :	Cash Liquid Assets Sales
	Profitability R 1 :	Working Capital Sales
	Profitability R 2 :	Net Income
	Profitability R 3 :	Gross Income Total Assets
	Time in business	Assessment year -incorporation year
	Sales evolution	Current sales -previous year sales
	Country code	Country codification

Table 5

 5

	Chapter 4. Aligning Feature Contributions with Expert Knowledge in Artificial
	Intelligence-Based Credit Scoring

.

[START_REF] Albanesi | Credit Growth and the Financial Crisis: A New Narrative[END_REF]

show that the XGBoost model can to detect defaulted companies with higher precision (i.e., recall) than the rest of ML models.

Table 4 .

 4 3: Models performance using the features in Table4.9.

	Model

Table 4 .

 4

	Rating Scale	Probability of default level
	A	Little/no default risk
	B	Low default risk
	C	Average default risk
	D	Above average default risk
	E	Increasing and high default risk
	F	Extremely high default risk or in default
	X	Excluded companies from the analysis
		due to lack of data

4: Rating scale for tinubu's algorithm.

Table 4 .

 4

	Tinubu	P(Y t+1 = 1)
	Score	Interval lower bound Interval Upper Bound
	A	0	0.0828
	B	0.0828	0.1411
	C	0.1411	0.2029
	D	0.2029	0.2486
	E	0.2486	0.285
	F	0.285	1

5: Scorecard mapping: we assign a Tinubu-defined credit risk score to each probability yield by XGBoost.

Table 4 .

 4 7: Definition of original financial variables. This variables are used to compute the financial ratios in table4.9.

	Feature name	Feature Description
	LGE ID	Identification number. This value is unique for each company
	Statement date	Corresponds to the date in which the financial data was published
	Out of business indicator	Binary variable: Yes if the company is currently defaulted
	Country code	Abbreviation of the country in which the company is based
	Total Employees	Number of employees
	Net worth	Total amount of Equity
	Total Assets	Refers to the total amount of assets owed by the entity
	Gross Income	Amount of money earned before taxes
	Total Liabilities	Combined debts a company owes
	Current Ratio	A liquid ratio that measures the ability of a company to pay short-term
	Cash and Liquid Assets	Refers to assets that can be readily convert to cash
	Sales	Net sales for the period after returns, allowances, and discounts are de-
		ducted
	Working Capital	Capital of the financial activity period
	Net Income	Amount left over after all expenses and taxes are deducted
	Incorporation Year	The year the business incorporated
	Previous Sales	Financial statement date
	58

Table 4 .

 4 8: Percentage of missing values for each financial variable by year.

	Financial Variable	2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
	Total Employees	6.23	6.1	10.5	21.22 25.51 27.94 32.05 38.37 37.76 35.6
	Net worth	0.77	6.94	7.94	5.57	0.97	0.54	0.71	0.23	0.1	0.07
	Total Assets	0.84	6.93	7.61	5.58	1.03	0.66	1.13	0.85	0.79	0.78
	Gross Income	14.74 14.35 18.84 13.31 8.99	12.4	13.13 12.03 13.78 13.75
	Total Liabilities	0.87	6.97	7.49	5.27	1.18	0.86	1.37	1.06	1.04	1.01
	L1 Ratio	77.59 82.23 77.82 80.47 40.12 11.43 6.47	4.43	4.61	4.04
	Cash and Liquid Assets	4.44	10.74 11.12 8.95	4.48	4.11	4.53	4.47	4.28	4.03
	Sales	12.5	8.7	11.79 10.02 6.72	8.82	10.92 9.1	10.12 10.47
	Working Capital	7.79	13.27 30.49 52.29 58.9	60.1	63.7	69.33 70.87 71.69
	Net Income	6.37	10.16 13.76 10.53 6.07	8.95	9.15	7.71	8.5	8.76
	Previous Sales	18.53 14.99 18.49 17.36 12.3	15.2	14.4	11.74 12.52 12.41

Table 4 .

 4 9: Financial ratios used in the financial industry. This ratios are the inputs of Tinubu's Scorecard Rating Algorithm and the ML models used to predict the default of a given company.

	Features	Definition	Description
	Solvency R 1 :	Net Worth Total Assets	Measures enterprise ability to
			meet current debt obligations.
			High S 1 values is indicative of
			greater solvency
	Solvency R 2 :	Financial Debt Gross Income

Table 5 .

 5 1: Performance of the Machine Learning Models for All Economical Sectors

	Model		Performance Metrics
		Accuracy	AUC ROC	F1-Score
	LR	0.695	0.722	0.14
	SVM	0.653	0.698	0.07218
	KNN	0.954	0.579	0.05703
	Random Forest 0.792	0.755	0.18434
	LightGBM	0.836	0.764	0.19804
	XGBoost	0.913	0.777	0.22732

Table 6 .

 6 1: Transformed data is the result of keeping companies rated every year during the period 2008-2019. Each row of the dataset represents an assessment of the company. Time-Series dataset is a dataset in which each row represents a company and the columns the year the company has been rated

	Dataset	Size
	Original Dataset	1399179
	Transformed Dataset 40772
	Time-Series Dataset	3395

Table 6 .

 6 3: Optimal XGBoost hyperparameters found using Grid Search.

		2018	2019	Prediction 2018 Prediction 2019 p q	AIC
	462792	233 (B) 213 (C)	234 (B)	245 (B)	1 1	65.2517
	462792	233 (B) 213 (C)	234 (B)	245 (B)	2 1	67.2505
	462792	233 (B) 213 (C)	233 (B)	244 (B)	3 1	69.0822
	462792	233 (B) 213 (C)	233 (B)	249 (A)	1 2	59.9773
	462792	233 (B) 213 (C)	235 (B)	259 (A)	1 3	57.8751
	462792	233 (B) 213 (C)	233 (B)	246 (B)	2 2	61.4822
	462792	233 (B) 213 (C)	234 (B)	253 (A)	3 3	60.3457
	462792	233 (B) 213 (C)	233 (B)	246 (B)	3 2	63.482
	462792	233 (B) 213 (C)	236 (B)	253 (A)	2 3 58.4614
				Hyperparameter Value
				n estimators	150
				learning rate	0.2
				max depth	8

A FICO Score is a three-digit number based on the information in your credit reports. It helps lenders determine how likely you are to repay a loan.

Tinubu is a company specialzed in credit scoring assessments

1.7. Problem StatementWe conclude the manuscript with a general summary that encapsulates the key findings, contributions, and implications of the research conducted. In addition to summarizing the work, we also present open questions and future perspectives that arise from our research.

Linear models have long served as a foundational element in the domain of credit scoring, dating back to several decades.

https://www.insee.fr/fr/information/2120875

https://pandas.pydata.org/

https://numpy.org/

https://imbalanced-learn.org/stable/

https://www.tinubu.com/

After trying different combinations for both hyperparameters, these values obtain the best results for the tested ML models.

After trying different combinations for both hyperparameters, these values obtain the best results for the tested ML models.

https://www.tinubu.com/

EBE: Excedent Brut d'exploitation. It is the equivalent of the EBITDA (Earnings Before Interests Taxes, Depreciation, and Amortization)

Financial embarrassment refers to a state of financial difficulty. Companies in financial embarrassment may have problems refunding their loans

The data is not publicly available

Remerciements

Chapter 4. Aligning Feature Contributions with Expert Knowledge in Artificial Intelligence-Based Credit Scoring Figure 4.1: For a given year t, the default rate represents the percentage of defaulted companies in year t + 1 over all companies rated the year t and t + 1. Chapter 5. Sectorial Analysis Impact on the Development of Credit Scoring Machine Learning Models Furthermore, we have shown that the models show certain divergences in terms of the variables considered most important for decision-making. In credit scoring, companies are evaluated using three different analyses: analysis of the company's accounts, analysis of the socio-economic situation of the country to which it belongs, and finally, the analysis of the sector in which it operates. In this work, we have analyzed the company's accounts as well as the sector, focusing on the creation of a model by sector. Future work should expand the framework to include the socio-economic analysis of the country to develop a model that applies to multiple countries. The analysis in this chapter has focused on the specific analysis of the impact on the economic sector. However, it is important to note that the sectors present a certain dependence among them. Future work should also address the issue of interdependence between sectors. gained interest for solving NLP tasks like Information Retrieval (IR) [START_REF] Shen | Learning semantic representations using convolutional neural networks for web search[END_REF], Named Entity Recognition (NER) [START_REF] Santos | Boosting Named Entity Recognition with Neural Character Embeddings[END_REF] or Text Classification [START_REF] Conde-Cespedes | Detection of Suspicious Accounts on Twitter Using Word2Vec and Sentiment Analysis[END_REF].

Both Machine-Learning (ML) and DL models take as input numerical features. Two popular approaches are used in NLP for text representation and feature extraction. The methods employed for representing text and words in a numerical format (i.e., highdimensional vectors) that aim to capture the meaning and context of the words are called Word Embeddings (WE). NLP researchers have adopted several methods for creating word embeddings (i.e., Word2vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF], GLoVe [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF], BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF]). More classical approaches like TF-IDF [START_REF] Berger | Information retrieval as statistical translation[END_REF] represent the text as a sparse vector of word frequencies.

For the TF-IDF technique [START_REF] Berger | Information retrieval as statistical translation[END_REF] the measurement of the importance of a word in a document is made by combining the term frequency (TF) and the inverse document frequency (IDF), which refers to how common or rare a word is across the entire document. Word2vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF] is a WE-method that represents words in a continuous vector space in which words that have similar meanings are closer in the vector space. GloVe [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF] is based on the idea of factorizing a large matrix of word-word co-occurrence counts, where the matrix is constructed from a large corpus of text. The co-occurrence information between words indicates how often they appear together in the corpus. BERT (Bidirectional Encoder Representations from Transformers) is a state-of-the-art natural language processing model developed by Google in 2018. It is a pre-trained language model that uses a deep neural network architecture called the transformer, which allows it to capture the context and dependencies of words in a sentence. Sentiment analysis has been used in several industries such as Marketing [START_REF] Rambocas | Online sentiment analysis in marketing research: a review[END_REF] or Finance [START_REF] Gupta | Comprehensive review of text-mining applications in finance[END_REF]. For example in marketing, sentiment analysis is used to identify customer needs which triggers different strategies to improve customer satisfaction and retention. In finance, multiple different approaches have been adopted. In [START_REF] Gupta | Sentiment Analysis for Stock Price Prediction[END_REF], the authors analyze the impact of social media opinion on different companies for stock market predictions. In credit scoring, few works have concentrated their efforts on the impact of textual data for default prediction. In [START_REF] Stevenson | The value of text for small business default prediction: A Deep Learning approach[END_REF], the authors found that a deep learning approach, based on the BERT model for default prediction using textual data, outperforms classical ML models (i.e., Logistic Regression and Random Forest). Other approaches that have been considered for the treatment of textual data for the default prediction focus on word embedding techniques to represent in a low-dimensional vector space the economic sector of the company [START_REF] Provenzano | Machine Learning approach for Credit Scoring[END_REF].

Methodology

In this section, we present our proposed framework. First, we start by presenting the data and its characteristics. Then, we describe the different strategies we propose for dealing with the text feature. We show the preprocessing pipeline and the data-splitting strategy for the training stage. Finally, we compare the performance of different state-of-the-art machine learning models for the different text feature treatment strategies.

Text Feature Treatment

The dataset contains a text feature in which the comments and opinions of the credit risk analysts are stored. The risk analysts' comments usually contain a summary of the industry trends and economic conditions that could impact the company as well as a description of the company and its activities.

The first step in the text feature pipeline is to remove stop words, punctuation, and other irrelevant information. Once the text has been preprocessed, we use an embedding algorithm to represent each word in the text as a vector in a lower dimensional space. For instance, in this work we employed two different word embedding techniques: Word2vec [START_REF] Mikolov | Efficient Estimation of Word Representations in Vector Space[END_REF] and GLoVe [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF]. The GLoVe algorithm, trained on 2B tweets, transforms the text into a 200-dimensional space. In this case, the text is encoded in a matrix G ∈ M m,300 .

In this work, we propose a different approach when treating the text feature. We evaluate the sentiment of the text generated by the risk analyst using a sentiment analysis model (i.e., FinancialBERT [START_REF] Hazourli | FinancialBERT -A Pretrained Language Model for Financial Text Mining[END_REF]). FinancialBERT is a fine-tuned BERT using large corpora of financial texts. The model categorizes the text as positive, neutral, or negative. the models.

Results

Tabular

Tab.7.2, we present the results using the approach described previously. In this work, we conducted 6 different experiences that related to the treatment of the textual feature. Tabular represents the model that has been trained just using the financial features. Tabular + Text (WE), we merged the financial features with the numerical representation generated by GLoVe and Word2Vec. In the WE + PCA experience, we performed a PCA to reduce the numerical representation from 200 and 300 respectively to 5 dimensions (i.e., these 5 dimensions represent 73.4 % of the variance). The SA represents the scenario in which we employ sentiment analysis to map the text to a categorical value.

In Tab.7.1, we show the F1-Score for the models and the different text treatment configurations. Since we compute 5 cross-validations, we have the mean F1-Score and the standard deviation. The table shows that both XGBoost and Logistic Regression models benefit from the addition of a text feature encoded using sentiment analysis.

The analysis of Tab.7.3 shows that the reason behind the low values of the F1-Score (see Tab.7.1) is the inability of the models to determine which companies will go into default. This is mainly due to the imbalanced characteristics of the dataset employed.

On the other hand, and regarding the F1-Score, we observe that Support Vector Machine and Neural Networks improve their performance by adding a significant number of features (i.e., the dimension of the word embeddings). This behavior is even more remarkable for Neural Networks.

In terms of the two embedding techniques proposed in this work, we see that there is no technique that improves the model performance for all the models. 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 0.000 ± 0.000 Table 7.3: Precision Score mean and standard deviation for the different strategies using 5-fold cross-validation. WE represent the dataset in which the text has been encoded using Word Embeddings. SA stands for Sentiment Analysis and is the dataset with the text categorically encoded using a sentiment analysis model. PCA is for the experience in which the dimensions of the numerical vector generated by the WE have been reduced using Principal Component Analysis. Chapter 8

Chapter 7. Multimodal Credit Risk Scoring

Predicting Corporate Solvency using Sentiment Analysis of Risk Analyst Textual Assessments

Introduction

Credit scoring has been the backbone of financial institutions' decision-making processes, allowing them to adequately evaluate the risk associated with providing credits to individual consumers and businesses. The recent progress in artificial intelligence has significantly improved the precision and efficiency of decision-making algorithms. These advances have provoked a waterfall of a wide variety of applications in several different industries.

In recent years, the financial industry has started to incorporate Artificial Intelligence (AI) powered algorithms and advanced data analytics in key tasks (e.g., credit scoring, pricing, etc...) ( [START_REF] Cao | AI in finance: A review[END_REF], [START_REF] Bhatore | Machine learning techniques for credit risk evaluation: a systematic literature review[END_REF]).

In the credit scoring field, the primary role of credit risk analysts is to quantitatively estimate the risk that a borrower will experiment financial difficulties afterward [START_REF] Lessmann | Benchmarking state-ofthe-art classification algorithms for credit scoring: An update of research[END_REF]. AIpowered algorithms allow financial institutions to estimate the probability of default of a company using typical financial statements. These models suffer from the complex and dynamic behavior of the economy. Credit risk analysts, in terms of data employed as an input for their decision-making process, may not be able to deal with more than several financial variables in comparison to the credit scoring algorithms, but they expand their knowledge of the situation of a concrete company by analyzing textual data such as news articles, social media or press releases. Subsequently, they write credit risk assessments in which they express their opinion about the company and whether credit should be granted to the company.

Conclusions

In this chapter, we explored the integration of credit risk text assessments made by credit analysts with a fine-tuned FinancialBERT model for sentiment analysis. Our results indicate that the model tends to overestimate the neutral class, which may be attributed to the neutral tone often employed by risk analysts in their texts. However, the extreme errors in the confusion matrix are not very high, suggesting that the model does not frequently make extreme mistakes, such as confusing positive sentiment with high-risk companies.

We further compared three different phrase-level sentiment analysis strategies and found that the one in which the weighting of each phrase grows exponentially produced the best results. Generally, the structure of a risk analysis can be split into three main parts: the presentation of the company, the financial analysis of the company, and the expert's opinion. These results are consistent with the typical structure of expert analyses, which often conclude with the expert's opinion on the solvency of a company.

The end-of-text sentiment analysis strategy, which focuses on the last paragraph, was found to be effective in determining the sentiment of the text, leading to an improvement in the results. Our findings suggest that incorporating credit analyst text assessments into a sentiment analysis model using an end-of-text approach can significantly improve the performance of credit scoring systems. Future work could further investigate the impact of the integration of this type of model with the probability of default models.