Mohamed Wiem Mkaouer

Tarek Gaber

Zaineb Chelly Dagdia

Effects of COVID-19 pandemic on computational intelligence and cybersecurity: a survey

Keywords: Coronavirus, COVID-19, computational intelligence, cybersecurity

In late December 2019, the World Health Organization (WHO) announced the outbreak of a new type of coronavirus, named the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), also known as COVID-19. The fatality of the virus has forced governments and countries to socially isolate their populations, causing a worldwide impact on the economy. Pandemic management has stressed health systems to work beyond their limits, adding more to the tragedy of losing millions of lives. As a natural response to such disasters, intelligent systems have been developed for various reasons related to virus detection, tracking, and control. The social lockdown created a record level of online platforms and applications being used to resume professional and educational activities in a virtual environment. This has triggered an unprecedented growth in cybercrime. This paper presents the effects of the pandemic on computational intelligence and cybersecurity.

Introduction

The coronavirus pandemic has been a once-in-a-lifetime phenomenon that has significantly burdened societies worldwide. The high transmissibility, life threatening, and novelty of the virus have led to a worldwide collective effort to combat the disease. Computational intelligence has been mobilised, along with health sciences, to combat the massive spread of COVID-19.

The integration of common intelligent computational models, including artificial neural networks, evolutionary computation, fuzzy systems, has been critical for the design and deployment of digital technologies that were devoted to fight the pandemic. Depending on the underlying technology, such as mobile computing, internet-of-things, big data, robotics, various dimensions of intelligent computing systems have been heavily relied on. These technologies, once augmented with the suitable intelligent techniques, have greatly lessened the damage that could have been enacted if the virus had occurred in a less technologically advanced era. The use of these technologies has aided in a range of areas such as predicting the spread of disease, controlling the spread of the disease, diagnosing patients, and developing treatments.

However, the need for an urgent response has pushed the integration and deployment of these intelligent algorithms into operational systems quickly, and into a broad range of users, without adequately performing the traditional testing procedures. Therefore, these operating systems, once augmented with intellignet algorithms (i.e., AI-enabled systems), have created various challenges for their maintainers, in terms of handling various issues related to usability, accessibility, stability, compatibility, and especially privacy. Also, the shift to the working remotely has emerged several online platforms that were vastly adopted to mimic the in-person setting everywhere. These platforms, besides being drastically operating on a larger scale than anticipated, were also being used by all types of individuals regardless of gender, age, culture, language. For instance, contact-tracing apps were designed, augmented with intelligent algorithms, developed, and shipped to production in a short cycle in comparison with traditional apps. This urgency requirement has prevented various types of important tests, such as security and privacy, to be properly performed, leading them to be more prone to vulnerabilities, and exposing their users to various risks, especially as they are being used online.

The purpose of this paper is two-fold: 1) we aim to analyze the impact of emerging technology on cybersecurity; and 2) we review the response of intelligent computing systems in the fight against the virus. We first analyze how the fast release of these applications had revealed various gaps and vulnerabilities that caused an increase in cybercrime. We foresee the absence of proper frameworks that ensure the existence of proper regulations built-in the shipped solutions. Our analysis shall provide developers and practitioners with actionable insights that need to be taken into account when developing such emerging AI-enabled systems in the future. Second, we dive into the various intelligent algorithms that were adopted in practice by these systems, as part of ensuring several tasks varying from accurately detecting the virus, into monitoring patients, and predicting the geographical and temporal propagation of the pandemic. This paper structure is as follows: Section 2 covers the impact of the pandemic on the security and privacy of users. Section 3 focuses on public health policy and Section 4 presents the medical healthcare asepct; both from a computational intelligence's applications presepective to fight the pandemic. The conlusion is given in Section 5.

Impact on cybersecurity

To respond to the impact of the pandemic, governments, healthcare providers, schools, universities and companies have used different information technology tools to mitigate the implications of lockdown constraints. Before the pandemic, mobile apps related to health (m-health apps) were usually effective tools for users to have control over their health, including calories consumption, vital signs, and activity. Such apps are found to be helpful in fighting COVID-19 [START_REF] Robert | Mobile health (m-health): Evidence-based progress or scientific retrogression[END_REF]. Namely, contacttracing apps, widely used in the last two years, were useful tools in containing COVID-19. The main idea of these apps is about collecting and processing personal information like user' current location, system settings, cell phone signals, voice and texts. With such information, users can get real-time medical advice such as alerts about COVID-19 infection hot-spots to avoid them. Many governments have adopted m-health apps applying the same idea to combat the deadly COVID-19 outbreak. Those include China, Korea, UK, and USA [START_REF] Duc | Health vs. privacy? the risk-risk tradeoff in using covid-19 contact-tracing apps[END_REF]. Although, such apps were found useful in containing the pandemic, there were a big concern about the users' privacy. More than half of US adult respondents surveyed by Pew Research Center stated that they will not install COVID-19 related health apps while 30% revealed that they will remove a mobile app which would exploit their personal data [START_REF] Boyles | Apps and privacy: More than half of app users have uninstalled or decided to not install an app due to concerns about their personal information[END_REF].

On the other hand, many online video-conferencing apps, such as Zoom, Google Meet, and Microsoft Teams, have emerged and used by people to overcome the limitations of lockdowns. Although this technology helped keeping the family, friends, and employees connected and work, they increased privacy risks. Zoom, for example, has seen a tremendous reaction as security specialists, privacy groups, and lawmakers warn that Zoom's default configurations are not safe [START_REF] Ali Khan | Ten deadly cyber security threats amid covid-19 pandemic[END_REF]. A recent consumer survey showed that apps like Google Meet, Microsoft Team, and Zoom capture more data than users realise [START_REF]Consumer Reports. It's not just zoom. google meet, microsoft teams, and[END_REF]. Such data can be analysed by AI-based applications to reveal personal information (i.e., violating users' privacy).

Due to the weak security [START_REF] Aaron | Consumer privacy during (and after) the covid-19 pandemic[END_REF] of the apps above, among many others, and the increase in the use of digital communication tools through running the businesses online (or work-from-home patterns), COVID-19 pandemic has led to an increase in the risk of cyberattacks. Responding to the lockdown, most of critical national business, such as healthcare and governmental services, had to carry on working but online. Such services would have not been equipped with right cybersecurity defence making them a good target to cyberattacks [START_REF] Wired | Hackers are targeting hospitals crippled by coronavirus[END_REF].

Responding to this threat, two big cybersecurity centres -National Cyber Security Centre (NCSC) of the United Kingdom and the Cybersecurity and Infrastructure Security Agency (CISA) of the United States -have met on April 8, 2020 and published a joint advisory committee. Among the aims of this committee are to describe how cyber-criminals and Advanced Persistent Threat (APT) groups were exploiting the current COVID-19 pandemic, and to provide how to avoid computer security threats [START_REF]Advisory: Covid-19 exploited by malicious cyber actors[END_REF]. Various vulnerabilities were mentioned in this alert, including phishing, malware, and the infiltration of communication platforms such as Zoom and Microsoft Teams. An encompassing appraisal of the full spectrum of attacks associated with the pandemic, on the other hand, is likely lacking in this report. Like other businesses at that time, the current state-of-the-art was extremely fragmented, with attacks reported by governments, the media, security organisations, and incident response teams all contributing to the overall picture. Therefore, organisations face a significant challenge in developing adequate protection and reaction measures in light of the changing environment in which they must operate.

From commerce and social connection to business and industry, the widespread adoption of digital technologies has shifted many aspects of society online, including crime. As of 2020, global cybercrime expenses are expected to rise 15% yearly over the next five years, to 10.5 trillion USD by 2025, from 3 trillion USD in 2015 [START_REF]Cybercrime to cost the world $10.5 trillion annually by 2025[END_REF]. As a result, the danger to innovation and investment is tenfold greater than the damage caused by natural disasters per year. Cybercrimes are estimated to be more profitable than all major illegal drugs combined making them more attractive for attackers. The cost estimated above include data damage and destruction, stolen money, lost productivity, theft of intellectual property, theft of personal and financial data, embezzlement, fraud, post-attack disruption to business, forensic investigation, restoration and deletion of the hacked data, and reputational harm [START_REF]Cybercrime to cost the world $10.5 trillion annually by 2025[END_REF].

Cybercrime and COVID-19

To emphasize the relation between cybercrime and COVID-19, it is important to define how a cybercrime can take place. For a cybercrime to occur, there must be three factors (fraud triangle): victim, motive, and opportunity [START_REF] Shinder | Scene of the Cybercrime[END_REF]. A victim is the target of the crime/attack, the criminal's motive is the reasons of mounting an attack such as financial or political gain, and an opportunity to perform the crime is the chance for it to happen (e.g., it can be an essential vulnerability in the system or an unprotected device). Opportunistic untargeted attacks are fairly common, as well as sophisticated targeted attacks. Opportunistic attacks target victims depending on their vulnerability to attack. Adversaries pick-up victims with known weaknesses or use social engineering to create weaknesses. Any device meant to fool a victim into becoming an attack prey is considered a hook [START_REF] Singh Lallie | Cyber security in the age of covid-19: A timeline and analysis of cyber-crime and cyber-attacks during the pandemic[END_REF]. Such a hook could exploit human distraction, time restrictions, and terror to make the attacks work [START_REF] Nurse | Cybercrime and you: How criminals attack and the human factors that they seek to exploit[END_REF]. People are more easily mislead if they are distracted or worried. Temporal restrictions also increase victims' vulnerability to mistakes and scams. There are other occurrences that have a profound and devastating influence on the entire society, such as fatalities and disasters.

Opportunistic attackers will wait for the perfect conditions to begin an attack. Such conditions include natural disasters, continuing crises, and major public events. Tysiac, in [START_REF] Tysiac | How cybercriminals prey on victims of natural disasters[END_REF], has given some examples on how cybercriminals take on victims of natural disasters. Also, from history, below are a few examples of opportunistic attacks mounted after natural disasters and public events. Natural disasters: New Orleans (USA) and its neighbourhoods were devastated by Hurricane Katrina in 2005. Shortly thereafter, residents began receiving scam emails asking for personal information in order to facilitate possible government assistance efforts, and thousands of fake websites were published claiming collecting humanitarian donations. Similar scams and attacks have been mounted in 2020 during Australia bushfires [START_REF] Elsworthy | Hundreds of bushfire donation scams circulating[END_REF]. Public events: In 2018, during the FIFA World Cup, there were numerous attempts, using spam and phishing emails, to lure people to go by offering them free tickets and gifts [START_REF] Foltýn | You have not won! a look at fake fifa world cup-themed lotteries and giveaways[END_REF]. Later, these were found to be phishing emails resulting fraud.

Given the wide range of cyberattacks mounted during natural disasters and public events as shown above, it should come as no surprise to experience similar cyberattacks during the COVID-19 epidemic. Even the scale and volume of the attacks could worse as COVID-19 affected all countries. It was evidenced that the outbreak has created widespread disruption around the world forcing people to adjust their day-to-day activities such as stopping social connection and sport exercises, and most importantly working from home depending on the Internet technologies [START_REF]NHS. 9 tips to help if you are worried about covid-19[END_REF]. Giving all these conditions which have taken place all in once, many people have been overwhelmed and got stressed and worried. Such emotional conditions would increase the vulnerability chance of individuals to be victims for attacks. Additionally, to keep running their business during the lockdown, companies were forced to create new, but maybe not well-studied, working mechanisms such as working from home. Such sudden decisions and changes could lead to changing the protection status of some company assets from high to low or weaken access control to those assets [START_REF] De Las Cuevas | Automatic rule extraction from access rules using genetic programming[END_REF]. Exploiting such weakness by cybercriminals across the world have created what is known as COVID-19 cybersecurity pandemic.

Cybersecurity pandemic

Given the massive increase of cybersecurity attacks during the pandemic, as will be highlighted below, the period from late 2019 and 2020, could be known as cybersecurity pandemic. Since the out-break of the pandemic began, numerous sources have reported a significant increase in the number of scams and cyberattacks. Cybercriminals have employed different strategies to mount attacks. For example, they exploited high-demand goods related to COVID-19 (e.g., PPE and testing kits and drugs), which have a high profitable rate or can lead to impersonation attack (i.e., one claims to a public official from WHO [START_REF] Cgi | Helping defend against a 30,000% increase in phishing attacks related to covid-19 scams[END_REF]).

As reported in [START_REF] Elsworthy | Hundreds of bushfire donation scams circulating[END_REF], on 31 December 2019 and 14 April 2020, the World Economic Forum (WEF) claimed that the pandemic resulted in a 50.1% increase in cyberattacks and an accompanying 30,000 cyberattacks that were explicitly COVID-19-related. Also, according to CGI company [START_REF] Cgi | Helping defend against a 30,000% increase in phishing attacks related to covid-19 scams[END_REF], because of COVID-19, there has been a 30% rise in cyberattacks. Between January and April 2020, the Interpol discovered a massive increase of different attacks and threats including 907,000 messages of spam, 737 incidents of malware, and 48,000 malicious URLs, all associated to COVID-19. It was also found that there was 60% rise of ransomware payout during the second quarter of 2020 comparing to that of the first quarter [START_REF][END_REF].

Another attack, ransomware was in particularly soared in April 2020 where most countries were in lockdown. Notably, this attack was mounted by a variety of previously inactive threat organisations. The average ransomware pay-out reached 178,254 USD in the second quarter of 2020 which is up 60% from the first quarter [START_REF] Davis | Covid-19 impact on ransomware, threats, healthcare cybersecurity[END_REF]. Such rise in ransomware payments may imply that cybercriminals anticipate a greater possibility of payment as a result of the pandemic's exceptional circumstances. In line with these facts, Google reported that they blocked 18 million malware and phishing emails everyday during April 2020 [START_REF] Davis | Google blocking 18m coronavirus scam emails every day[END_REF].

Most common attacks

To assess the impact of the COVID-19 on cybercrimes and cyberattacks across the world, the Interpol analysed data collected during a four-months period (January to April 2020) [START_REF][END_REF]. The analysis of these data showed that there is a massive increase of three main attacks -all related to COVID-19: 737 incidents of malware, 48,000 malicious URLs, and 907,000 spam messages. Figure 1 shows a summary of the most important results from the Interpol study which was conducted from January to April 2020 to gain insight of the COVID-19 related cybercrime posture of the Interpol members. Aligned with the results of the above studies, it was found that phishing and spam emails were the most common cyberthreats. Below, we highlight the main attacks, and how they could be mounted by the hackers using COVID-19 related information [START_REF][END_REF].

Online scams and phishing

The traditional internet frauds and phishing techniques have been reworked by hackers. Cybercriminals exploit COVID-19 information communicated through emails to trick their victims into disclosing personal or financial information, or into opening a malicious link or file, which gives them access to their target's machine. For example, attackers could use any or all the following information to compose a phishing email related to the vaccine of the COVID-19: registration of vaccines, a summary of your vaccination history, locations where the vaccination can be administered, a vaccination can be reserved in several ways, and/or requirements for vaccinations. Using such details, on Google platform only, cybercriminals managed to send around 18 million malware and phishing emails associated with COVID-19 during April 2020 [START_REF] Cgi | Helping defend against a 30,000% increase in phishing attacks related to covid-19 scams[END_REF].

Disruptive malware

This includes attacks such as ransomware and DDoS. These types of attacks are increasingly being used by cybercriminals targeting critical infrastructure. During COVID-19, healthcare facilities were a main target due to their importance at that time and for the high potential for high impact and capital profit. Several threat groups that have been relatively quiet for the previous few months stepped up their ransomware activities in the first two weeks of April 2020. As reported in [START_REF][END_REF], the majority of ransomware attackers are now able to workout how much of ransom they could ask for from their victims.

Data harvesting malware

Data stealing malware is a web-based threat that deprives victims of their personal and proprietary information with the purpose of profiting the hacked data through direct usage or underground distribution. Cybercriminals are increasingly deploying data harvesting software, such as banking trojans information stealers, spyware, and remote access trojans, to collect information about their victims. The cybercriminals attack systems using materials related to the COVID-19 context for different purposes including divert money, construct botnets, compromise networks, steal data, among many others [START_REF][END_REF].

Malicious domains

Cybercriminals are taking advantage of the increasing demand for medical supplies and information about COVID-19 by registering domain names that include keywords, such as "coronavirus" or "COVID". All sorts of harmful operations, such as C2 servers, malware distribution, and phishing, are supported by these bogus websites. Malicious registrations, such as malware and phishing, increased by 569% from February to March 2020, according to data provided to Interpol by a private sector partner, and high-risk registrations increased by 788% during the same period. Also Google reported that they managed to block 126 million COVID-19 phishing scams in just one week [START_REF][END_REF].

Misinformation

The public is being flooded with fake news and misinformation at an alarming rate. Conspiracy theories, incomplete threat assessments, and unverified information have all intensified public fear and, in some situations, made it easier for cyberattacks to be carried out. Almost a third of the countries that took part in the global survey [START_REF][END_REF] on cybercrime reported that incorrect material about COVID-19 was being circulated. One country reported 290 postings in a month, most of which contained hidden spyware [START_REF][END_REF]. Also, in this survey, it has been reported that false medical information is being sold illegally. Scams involving 'too good to be true' offers like free meals, unique bonuses, or substantial discounts at supermarkets were another source of incorrect information spread via mobile text messages.

Following the COVID-19 pandemic, statista.com [START_REF] Statista | Where do it professionals see an increase in cyber attacks and attack attempts following the covid-19 pandemic?[END_REF] conducted a survey among IT experts all over the world, and investigated whether the pandemic has left an impact on the landscape of the cyberattacks. The recent survey was done in August 18, 2021 and its main findings, as depicted in Figure 2, show an increase in cyberattacks since the COVID-19 outbreak, with the majority of the rise occurring in the area of data exfiltration and leakage. Phishing emails are also becoming more common, with half of the respondents reporting that they had received one recently.

Figure 2. Landscape of the cyberattacks after COVID-19 [START_REF] Statista | Where do it professionals see an increase in cyber attacks and attack attempts following the covid-19 pandemic?[END_REF] People all around the world are becoming increasingly reliant on the internet, which is opening up new economic prospects. However, many firms and individuals are not keeping their cyber defences up to date which may lead to privacy and security problems. The lesson learnt from surge in cyberattacks during COVID-19 would reinforce the necessity for greater public-private sector cooperation if we are to effectively combat the privacy and security threat posed by COVID-19 to our cyber space. Cybercrime is only going to get worse in the near future. Cybercriminals will continue to ramp up their activity and create more advanced and sophisticated methods of operation as a result of vulnerabilities associated with working from home and the potential financial gain that this brings. To take advantage of the public's anxiety over the outbreak, criminals will likely continue to use coronavirus as a theme in their phishing and online frauds. The economic crisis and changes in the business sector will certainly lead to an increase in Business Email Compromise schemes, creating new chances for criminal activity.

One of the main lessons learned of the COVID-19 cybersecurity threats is that the threats is global, so it needs a global effort to compact such threats for future similar events. This was started, as previously mentioned, by two big cybersecurity centres -National Cyber Security Centre (NCSC) of the United Kingdom and the Cybersecurity and Infrastructure Security Agency (CISA) of the United States -when they formed a joint advisory committee on April 8, 2020 as a response to COVID-19 related attacks. In the future, it is possible to see regulatory bodies for approving and controlling cybersecurity software as there would be a raise of the security expectations. Also, the Security-by-Design would be the main stream for developing secure software.

National cybersecurity strategies and COVID-19

Apart from the negative security and privacy impact caused by COVID-19 pandemic, it showed positive impact through digitization which enabled the continuous running of the business during lockdown. The digitization is a significant enabler of the progress in many, if not all, sectors including economy, social, healthcare and education. Recently, it was reported in [START_REF] Obiso | How tailored national cybersecurity strategies enable safe, inclusive and sustainable digital development[END_REF] that digitization is essential for accomplishing the 17 Sustainable Development Goals (SDGs) developed by the United Nations and which constitute a vital appeal to action by all nations either developed or developing ones.

The digitization of the activities of economy, social, healthcare sectors cannot achieve intended outcomes without safe and secure digital infrastructures. On the other hand, there is a rapid growth of cybersecurity threats that can affect a country's digitization process. So, there is a need for a National Cybersecurity Strategies (NCSs) for each country. Responding to this, according to Global Cybersecurity Index 2020 [START_REF]Global cybersecurity index[END_REF], many countries are developing NCSs. In the past two years, the number of nations that have adopted NCSs has increased by 40%. Nevertheless, despite recent improvements, 60% of least developed countries (LDCs) still lack strategies, and the majority of developing nations that have created NCSs failed to implement them due to a lack of financial and human resources.

To help policymakers, such as governments, to address the above concerns, a guide for developing NCSs has been developed by many international partners including ITU, Microsoft, INTERPOL, and WorldBank [START_REF]guide to developing a national cybersecurity strategy[END_REF]. Because of COVID-19, the guide included techniques and approaches on how to strengthen infrastructures resilience and service availability to disasters and pandemics. The guide presents a practical and adaptable framework to aid policymakers in developing, updating, implementing, monitoring, and assessing their strategies. Apart from the guidance and advice, the question is now how the least developed and developing countries can secure a budget to implement a national security strategy. If this cannot be achieved, such countries would not be able to use the advantages of the Internet to achieve advancement. Consequently, this would lead to the injection of more cybercriminals looking for financial gain.

Discussion of recent research

This section will cover a brief discussion of the recent literature of three important tools and mechanisms used to achieve security for the companies and organization during the pandemic.

Antivirus tools: When it comes to protecting one's computer, one of the most important steps a user may take is to install antivirus software. Iakovakis et al. [START_REF] Iakovakis | Analysis and classification of mitigation tools against cyberattacks in covid-19 era[END_REF] conducted a comparison among various antivirus tool to see which is effective to compat cyber-attacks and threats during COVID-19. They identified and used nine criteria: price, malicious URL blocking, strengths, weaknesses, phishing protection, on-demand malware scan, on-access malware scan, website rating, and behavior-based detection. The comparison and analysis showed two main points. Firstly, there are a handful antivirus software products -including Sophos, Avast, Avira, and Bitdefender Free Edition -which are available for absolutely no cost and still effective in detecting most of the viruses and blocking malicious URLs. Secondly, the most well-known antivirus tools such as McAfee, Symantec Norton, Kaspersky, Trend Micro, and Bitdefender have met all criteria but need to be updated. This study could be improved considering the privacy of the users. There are several questions which still need to be investigated and these include what type of data will be collected from users and where such date will be stored? Will the user have access to any information related to their privacy and will the collected data be shared with other third parties?

VPN: Research and Markets at Yahoo estimated that by 2026, it is anticipated that the Global Virtual Private Network (VPN) would reach 91.20 USD Billion [START_REF] Yahoo | Global virtual private network (vpn) markets report 2022[END_REF]. The global VPN market is predicted to increase from 38.56 USD billion in 2021 to 45.89 USD billion in 2022, because of the impact of the pandemic which forced companies to use VPN products to secure working from home.

One of the recent research in this area is about scaling VPN during a major disaster such as pandemics [START_REF] Joe | Safely scaling virtual private network for a major telecom company during a pandemic[END_REF]. As discussed above, the pandemic radically altered how we live and work. Companies were forced to let their employees to work from home. To protect proprietary information, monitor staff efficiency, and control shareholder profitability while satisfying customer demand, the telecommunications sector needed to scale up VPN. The main recommendation of the paper [START_REF] Joe | Safely scaling virtual private network for a major telecom company during a pandemic[END_REF] is to increase the broadband and use seamless VPN tunnels to enable connecting remote users with their company network. Although scaling up the use of VPN to support the expected increase of remote working, the study overlooked deep security analysis including attacks such as man-in-the-middle attacks. These attacks are possible to be mounted since it is possible for an adversary to take control of a VPN session and decode the data being sent and received. Public people who do not work for companies and do not use security environments may be at an increased risk of having their private information disclosed.

Multi-factor authentication: Multi-Factor Authentication (MFA) mechanism was among the best practices recommended by security specialists and governments to compact security problems during the pandemic. A recent market report by marketsandmarkets.com [START_REF]marketsandmarkets.com. Multi-factor authentication market[END_REF] showed that it is anticipated that the size of the global market for multi-factor authentication would increase from 11.1 billion USD in 2021 to 23.5 billion USD by 2026. It showed that the main reason of this jump was the impact of the pandemic where there has been a huge increase in reliance on e-businesses including banking, financial services, healthcare, and retail sector. To better secure communications and transactions of businesses, multi-factor authentication was adopted where a system requires two or more credentials to login. MFA strengthens security because even if one credential is compromised, unauthorised users cannot meet the second authentication criterion and cannot access the targeted place, device, network, or database.

To further improve the efficiency and the security of the MFA mechanism to support remote work depending on IoT and cloud technologies, Alsahlani et al. [START_REF] Fahad | Lmaas-iot: Lightweight multi-factor authentication and authorization scheme for real-time data access in iot cloud-based environment[END_REF] have proposed a Lightweight MFA and authorization scheme which can be used for real-time data access in IoT and cloud-based applications.

IoT and cloud-based technologies allow people to manage vital tasks remotely with minimal efforts, but also allow sharing and collecting sensitive information across insecure public channels which is risky. To overcome this problem, Alsahlani et al. [START_REF] Fahad | Lmaas-iot: Lightweight multi-factor authentication and authorization scheme for real-time data access in iot cloud-based environment[END_REF] proposed a lightweight scheme which can provide MFA and also authorisation for IoT and cloud-based data access. This scheme could be used in many applications including the management of large-scale systems, e.g., healthcare industry. The main limitation of this scheme is privacy. The information obtained from the users is kept on a remote server and users do neither have access to any information linked to their privacy nor they know how it will be processed.

Public health policy

Planning

Since the onset of the pandemic, computational technology has played a critical role in planning emergency protocols for governments and organizations worldwide. The results have been crucial for developing proper public health policy frameworks to control the impact of the pandemic. One of the earliest uses of computational technology was at the pandemic's epicenter in Wuhan, China. Big data analysis and artificial intelligence were used to forecast the movement of potential Covid-infected individuals who dispersed from Wuhan after the outbreak. This forecast was constructed using machine learning models, trained on movement data collected from mobile phones, mobile payment applications, and social media. This forecasting model allowed China to put in place containment measures to frame the spread. Similarly, Taiwan integrated immigration records with its centralized, real-time national health insurance database. This integrated data allowed Taiwan authorities to cross-reference records and determine individuals who have traveled to Wuhan or been in contact with someone who has been to Wuhan. These individuals could then be quarantined and tested. These procedures have drastically decreased the early spread of the virus [START_REF] Vargo | Digital technology use during covid-19 pandemic: A rapid review[END_REF].

Predictive modeling was also designed to identify geographical areas that potentially need to be restricted. These models were challenged by the need to process and extract features from heterogeneous sources of information [START_REF] Joseph T Wu | Nowcasting and forecasting the potential domestic and international spread of the 2019-ncov outbreak originating in wuhan, china: a modelling study[END_REF]. The management of personal protective equipment was also a critical challenge in the early months of the pandemic. One solution for this issue was developed by Swedish Health Services, a USA-based healthcare organization. The company developed a platform for healthcare workers to report real-time data on volumes of patients with COVID-19, personal protective equipment, staffing, ventilator usage, and other information. This platform leverages big data technologies, supported with high-performance computers, to provide an immediate service for proper monitoring of resources so that any shortfall could be quickly identified, and resource balancing for high-demand areas and those in need can be implemented.

Due to the unknown nature of the virus, there was limited amount of information about it and hence hindered the planning for virus containment. Iceland has one of the highest per-capita testing rates, and was among the first counties to put in place mobile apps to report symptoms. This symptom dataset was collected and used in conjunction with clinical and genome sequencing to reveal characteristics of the virus such as its pathology and transmissibility. The app was available in all platforms and phone devices to increase its coverage among the population. The data gathered by the app have significantly increased the awareness of the virus, and was tailored to plan its control.

Monitoring

Another significant challenge that attracted many researchers was the development of strategies for monitoring the population and identifying COVID-19 cases. Furthermore, the identification of any case automatically triggers its contact tracing. In Taiwan, as shown in Figure 3, high-performance infrared thermal cameras were set up in airports to detect individuals who have fevers so that they can be prioritized for testing. The use of thermal temperature played a central role in many other countries. In Singapore, for example, temperatures were monitored at public building entrances to identify potential COVID-19 hotspot areas. A private company in the USA also used digital thermometers to collect real-time body temperature information. The data collected were then combined with resting heart rate data collected from smartwatch applications to develop a methodology to proactively identify COVID-19 emerging outbreaks. Likewise, an interactive map of potential COVID-19 cases was created in Germany from data collected from a smartwatch application. The application gathers pulse, temperature, and sleep pattern data then evaluates these for possible viral illness.

Figure 3. Thermal imaging systems

Thermal cameras suffered from many challenges such as their ineffectiveness when simultaneously capturing temperatures of multiple individuals when they are interleaved in front of the camera (e.g., pedestrians crossing from both sides of the road). Therefore, they need a special setup to allow an alignment of the individuals under test, to avoid mass temperature screening. Besides, these systems can only capture surface skin temperature, which is typically lower than the oral measured temperature. So, they may miss few positive cases, besides requiring corrections to reduce the gap between skin and oral measured temperatures.

Another essential use of monitoring technologies was in the area of contact tracing (Figure 4). Traditionally, upon the detection of positive cases, they are interviewed by public health officials to reveal their proximities, and potential exposures. Then, they reach out to exposed individuals and request their test and self-quarantine. While this process was found to be effective with previous epidemics, COVID-19 has revealed its limitations since it is labor-intensive and time-consuming. Also, the rapid spread of the virus has created a shortage in the public health staff, hindering the timely tracing of infected individuals. Therefore, intelligent computing systems were needed to mitigate these challenges.

One of the most comprehensive forms of COVID-19 contact tracing was seen in South Korea, where the whole process was fully automated: security camera footage, facial recognition technology, bank card records, and GPS location data from vehicles and mobile phones, are all used to monitor individuals. In the event of a positive case, the monitoring data are used to ascertain the possible exposed contacts and emergency text alerts sent to them and others in the general region. Another application of contact tracing was in Singapore. The mobile phone application exchanges short-distance Bluetooth signals when individuals are close. Exposed contacts could then be identified based on the information collected by the application. Yet, contact tracing applications still have their own drawbacks. Not all apps were compatible with old mobile devices, because they did require various permissions, and advanced features that average devices did not have. That is why, the app was effective with only a small fraction of the users. For example, after the first month of its launch, the Singapore app was adopted by only 20% of the targeted individuals. As the lockdown started to be alleviated, the app adoption has reached 40% by summer 2021.

Mitigation

In the fight against COVID-19, one of the main mitigation tools has been the use of quarantine and isolation measures to isolate confirmed and potential cases from the general population. In China, quarantined individuals must fill out an app-based symptom survey and record their temperature. This information is then used to monitor an individual's movement and track the severity of their symptoms. The system then provides color codes associated with the exposure level of the individuals. Similarly, in Taiwan, home quarantine is supported by geofencing and GPS applications running on governmentissued mobile phones. When an individual breaches quarantine, a message is sent to the mobile phone, and a fine is imposed. South Korea and Iceland use a similar approach for home quarantine. However, the monitoring application runs on the individual's personal phone. In Canada, blockchain technology was used to create an app called Civitas, which associates people's official IDs with blockchain records to verify whether a person has permission to leave his/her home.

Similar to contact tracing, any mobile-based software solution faces the challenge of being easily left behind by users when they are moving. In some countries, wearable devices, such as smart watches, have been explored as an alternative to mobile phone tracking. In Hong Kong, cloud-based technology is used to support wearable risk bans that are assigned to individuals in home-quarantine. In the event of breaching a mandatory quarantine order, authorities are alerted. In Australia, quarantine violations could be punished by requiring individuals to wear tracking devices, after which any further breaches would incur a fine. Human resource demands to enforce COVID-19 measures such as quarantine and mask mandates have strained many countries worldwide. This strain has led some countries to turn to drones and other robotic solutions to supplement frontline workers. Figure 5 illustrates the range of functionalities that have been developed using drone technologies. Using AI-based drones and surveillance cameras, China supports its quarantine measure to limit gathering sizes and identify individuals not wearing masks. India company, Cyient, provided unmanned aerial drones to law enforcement in Telangana. The drones are used for maintaining COVID-19 lockdown restrictions for a wide area with less staffing. Violations spotted by the drone can be promptly forwarded to law enforcement for handling. Another advantageous use of drones is for public announcements. This practical use of drones was demonstrated in European countries such as Spain, where Madrid police used loudspeaker equipped drones to inform the public about the guidelines for the imposed state of emergency.

Drone and other robotic devices have also been used to sanitize contaminated areas to reduce human exposure. Examples of disinfecting drones have been seen in India, China, and some European countries. For instance, the Spanish military has used modified DJI-made drones, previously intended for agricultural use, to spray disinfecting chemicals over public spaces. In the robotics sphere, a Danish robotics company, UVD Robots, has developed multiple disinfection robots to be delivered to hospitals worldwide. Another example is an autonomous disinfection robot to help limit the number of hospitalacquired infections created by US-based Xenex Disinfection Services. The company was established by two John Hopkins educated epidemiologists.

Medical healthcare

Diagnosis

Given that COVID-19 was a new virus and had similarities to other diseases such as influenza and pneumonia, its detection was not easily done in the early stages of the pandemic. However, artificial intelligence research has sought to remedy the diagnosis process. Roy et al. [START_REF] Roy | Deep learning for classification and localization of covid-19 markers in point-of-care lung ultrasound[END_REF] used artificial neural networks to analyze lung ultrasonography (LUS) images to detect the presence and severity of the virus at the frame-level, video level, and pixel-level. The approach was developed using fully annotated dataset of LUS images collected from several Italian hospitals, with labels indicating the degree of disease severity at a frame-level, video level, and pixel-level (segmentation masks).

In radiology, extensive research has been done to diagnose the virus using a variety of techniques. Wang et al. [START_REF] Wang | Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images[END_REF] created COVIDNet, which is based on a convolutional neural network (CNN) trained on 13,800 images collected from 13,725 patients. COVIDNet leverages its model to detect infected patients. Similarly, Panwar et al. [START_REF] Panwar | Application of deep learning for fast detection of covid-19 in x-rays using ncovnet[END_REF] developed a deep learning-based detection model for a quick virus identification within 5 seconds using X-ray images. Wang et al. [START_REF] Wang | A fully automatic deep learning system for covid-19 diagnostic and prognostic analysis[END_REF] developed a fully automated process for the virus diagnosis and prognosis using deep learning technology using X-ray and CT scan images.

To alleviate limited x-ray training data issues, Oh et al. [START_REF] Oh | Deep learning covid-19 features on cxr using limited training data sets[END_REF] developed a neural network for COVID-19 diagnosis that is suitable for training with limited X-ray images by adopting DenseNet103 and ResNet-18. Vaid et al. [START_REF] Vaid | Deep learning covid-19 detection bias: accuracy through artificial intelligence[END_REF] also tackled the issue of limited data by developing a transfer learning approach to build a deep learning model by transferring pre-trained CNNs. Sing et al. [START_REF] Singh | Classification of covid-19 patients from chest ct images using multi-objective differential evolution-based convolutional neural networks[END_REF] tackled the issue of optimizing CNN models for detection of COVID-19 from CT-Scans by using a multi-objective differential evolution algorithm to optimize the model's hyperparameters.

Many similar diagnosis models were put into practical use in the public health system worldwide. For example, China implemented a cloud-based AI-enabled system to quickly diagnose pneumonia attributable to COIVD-19 to distinguish it from other lung diseases [START_REF] Bai | Chinese experts' consensus on the internet of things-aided diagnosis and treatment of coronavirus disease 2019 (covid-19)[END_REF]. Additionally, machine learning solutions developed in China can predict severe condition development for COIVD-19 patients. Clinicians can then use this information to guide the treatment of the patient and resource deployment.

The pandemic has also accelerated the use of telemedicine services for handling patients with and without COVID-19. In Canada, telemedicine visits for patients increased from approximately 1000 visits per day in February 2020 to 14000 per day in May 2020. Telemedicine use has allowed for infected individuals with mild and moderate symptoms to be diagnosed and treated without further exposing the public. The use of telemedicine services is also illustrated by Hospitals such as George Washington University Hospital in the USA and the state governments of Andhra Pradesh and Assam in India. George Washington University Hospital offers video consultations and live Facebook webinars to provide remote medical expertise to several people. Similarly, the state governments of Andhra Pradesh and Assam have utilized telemedicine to offer health services to potential COVID-19 infected individuals.

Treatment

Since the COVID-19 pandemic started, AI and big data analytics have played an indispensable role in fast-tracking the development process for treating COVID-19 cases. These technologies have been employed to find ways to re-purpose existing drugs and expedite the development of new medicines. For example, the German company Innoplexus AG developed an AI-supported platform to study a combination of existing drugs such as Remdesivir (an experimental antiviral developed initially to treat Ebola), Chroloquine, Tocilizumab (an immunosuppressive drug), Pegasys (used to treat Hepatitis B & C), and Clarithromycin (an antibiotic) to establish the best option for COVID-19 treatment [START_REF] Vinay Chamola | A comprehensive review of the covid-19 pandemic and the role of iot, drones, ai, blockchain, and 5g in managing its impact[END_REF].

A British start-up named Exscientia, in collaboration with other entities, has also created an AIbased solution that determines the most effective combination of compounds to treat COVID-19. The results will then be used to develop a drug for public use if it is successful in a successful clinical trial. Other potential AI-based research for re-purposing drugs to treat COVID-19 was done with Baricitinib and Afatinib (used to treat non-small cell lung cancer).

Figure 6 shows the spectrum of computational intelligence's applications in modern technologies to fight the pandemic. 1 provides a summary of COVID-19 management categories, and their corresponding technologies. Various neural network-based models are being currently used in hospitals to optimize resource allocations, in vaccine development, and in the analysis of cell-phone data to trace those who have been in contact with or proximate to individuals diagnosed with the disease. While machine learning can help make pandemic response more efficient and expedite progress toward effective treatments, privacy, equity and related human rights concerns also abound.

Neural networks are built to learn data representation for various purposes including predictions and classifications. Representation learning allows the detection and recognition of features in particular data points and allows the identification of similar data points once the model is trained. Neural networks are powerful in replacing humans and performing various tasks without the need of humans' support in extracting features. Therefore, neural networks were heavily relied on for detecting patterns that identify the virus, as soon as data is being collected. Neural networks were able to learn representations of virus propagation and the characteristic symptoms of the COVID-19 viral infection.

However, there were many challenges related to the adoption of such solutions. To develop deep neural networks, there is a need for a significant amount of data to properly learn the patterns of the virus. Due to its dynamic nature, it was extremely hard to collect sufficient amounts of data that would achieve a uniform performance across all variants that are coming from different geographical locations. That is why, as the virus was spreading across many countries, there was a collective effort of bringing together all sorts of data from various regions, to collectively build data sets and train models. One example of the very early data that were collected was X-ray images of tested positive patients chests. These images were fed into a particular type of artificial neural networks, called convolutional neural networks (CNNs). CNNs have provided promising results and emerged as one of the popular solutions. For instance, Wang et al. [START_REF] Wang | Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images[END_REF] proposed COVIDNet, CNN-based model for positive cases based on their chest X-ray images. The model was built using a set of images collected from 13,725 patients. The CNN architecture of the model uses a lightweight residual block, and a projection expansion technique, as part of enhancing the model accuracy while keeping its computational complexity low. Attention schemes were used by Han et al. [START_REF] Han | Accurate screening of covid-19 using attention-based deep 3d multiple instance learning[END_REF] to train a deep 3D multiple instance learning model (AD3D-MIL) to identify the virus from 3D chest computed tomography images. The model makes binary classification of common pneumonia, along with interpreting the results by indicating the dominant features driving the binary decision.

As a summary, neural networks have been used to tackle the problem of symptoms characterization. This is primarily due to the explosion of medical images of positive cases. As more varients of the virus are appearing, more variants of the models themselves are being re-enforced. For instance, recurrent neural network models (RNNs) have surged as a reliable alternative for time series prediction. With the increase in the number of cured patients, datasets of treatment records are now available for training as time series. Moreover, the adoption of IoT technologies for smart care, is generating massive amounts of data that can be used to enhance the performance of current models and refine them to also detect current and future variants of the virus.

The blockchain technology can bring another layer of protection to data reliability. This can be useful as it can handle the heterogeneous nature of the data being collected from multiple sources, under various formats [START_REF] Dinh C Nguyen | Blockchain for 5g and beyond networks: A state of the art survey[END_REF]. It also provides another layer of protection against data alteration. Because of the distributed nature of blockchain, it is more reliable when it comes to individual points of failure. Blockchain represents an opportunity to process the data in a decentralized fashion, while ensuring its reliability [START_REF] Sin | Reliability analysis for blockchain oracles[END_REF]. Generally speaking, emerging technologies, such as the blockchain, provide solutions of data privacy, lack of information, communications technology infrastructure, and monetary contrast. Governments need to establish needed infrastructures and its corresponding regulations to protect their citizens. The lessons learned from this pandemic will benefit us in all facets of healthcare, including the potential next crisis.

Conclusions

Computational intelligence has been heavily solicited as a response to the massive outbreak of COVID-19. Various researchers have been developing intelligent methods, discussed in this paper, that High cost can be incorporated into AI-enabled systems for combating the pandemic. Yet, the widespread usage of online resources (e.g., [START_REF] Zaineb | Effects of covid-19 pandemic on education and society[END_REF]) as means to pursue industrial and educational functions has brought its own challenges in terms of privacy and security. For instance, this paper has shown how such an online setting has significantly increased cybercrime.

Despite the large number of studies proposing various intelligent computing systems, their adoption in practice was challenged by various constraints that scientists and scholars typically do not address in their research papers. For instance, there is a strong assumption that the execution environment is capable of deploying these models and providing the necessary hardware and software requirements, which has been found to be nearly impractical, since many of these systems were lightweight mobile apps and wearable IoT devices. Furthermore, many of these models were not properly documented for software developers to acknowledge how exactly they should be deployed and used in real-world settings. Another interesting observation across the discussed papers, is that the models evaluation was theoretical, and relies on traditional validation metrics, such as precision, recall, and AUC. While there is no discussion on how these highly performing models would cost the manufacturers and software developers. For example, many of the trained deep learning models are found to be expensive in terms of energy consumption. Thus, shipping them into mobile and wearable devices would be challenging.

Another interesting insight reveals how these models were trained on specific datasets which may become quickly obsolete as new variants of the virus keep appearing. So there is a need for planning to reinforce the learning of these models, but many of their corresponding papers do not tackle this problem, leaving the assumption that these models are parameterized in an ad-hoc fashion. In such a scenario, not only the model can become quickly deprecated, its update would be unreasonable for software maintainers, with no prior knowledge. The lack of expertise on how models should be pipelined in the software update lifecycle would eventually cause a waste of development and maintenance effort, and therefore needs to be addressed by future studies.

Figure 1 .

 1 Figure 1. Most common attacks from January to April 2020 of Interpol countries[START_REF] Cgi | Helping defend against a 30,000% increase in phishing attacks related to covid-19 scams[END_REF]

Figure 4 .

 4 Figure 4. Contact tracing through wearable devices

Figure 5 .

 5 Figure 5. Drones used for COVID-19 impact management

Figure 6 .

 6 Figure 6. Applications of computational intelligence

Table 1 .

 1 Summary of used technologies

		Description	Digital Technology Example Countries
	Planning	Projecting spread & impact of COVID-	AI	China
		19 to guide healthcare policy
			Big Data	Taiwan
			Mobile Technology USA
				Iceland
	Monitoring Observing the population for potential	Mobile Technology USA
		COVID-19 cases
			Digital thermometer South Korea
			Big Data	Singapore
			Wearables
			GPS
			Bluetooth
	Mitigation Measures for controlling the spread of	AI	South Korea
		COVID-19
			Mobile Technology China
			Big Data	Australia
			Wearables
			Cloud Computing
			IoT
			Drones
	Diagnosis	Developing methods detection of	AI	Canada
		COVID-19 cases
			Big Data	USA
			Cloud Computing	India
			Telemedicine
	Treatment Developing medical treatments of	AI	UK
		COVID-19 cases
			Big Data	Germany

Acknowledgments

This study has received a public grant through the national program "Programme d'Investissements d'Avenir (PIA)" under the reference ANR-18-RHUS-0004. This work is part of the Federation Hospitalo-Universitaire (FHU) Saclay and Paris Seine Nord Endeavour to PerSonalize Interventions for Sepsis (SEPSIS). This work is also supported by ANR PIA funding: ANR-20-IDEES-0002.


É Gennet

G Chassang

M Azzouz

E Rial-Sebbag

Ensuring ethical and legal compliance of cutting-edge DNA sequencing technologies. Practical considerations from H2020 project EASI-Genomics

Apply now!

EASI Genomics has recently launched an extraordinary call for Covid-19 related projects which is opened until 30 June 2020 and will benefit from an accelerated selection process. This call will focus on wholegenome (short-and/or long-read), whole exome and RNA sequencing of human patients from lowrisk populations with severe forms of SARS-Cov-2 infection that have experienced adverse reactions or fatal outcomes For more information: https://www.easigenomics.eu/access/calls/ covid-19-call

EASI Genomics in one sentence

The European Advanced infraStructure for Innovative Genomics (EASI-Genomics) is unifying the major European genomic centers providing free external access to cuttingedge DNA sequencing technologies (human, animal, vegetal) to researchers from both academia and industry, within or outside of Europe.

Values shared by EASI Genomics partners

Explaining in general terms the characteristics, objectives and ethicsby-design approach of EASI Genomics, the Memorandum of Understanding (MoU) has to be signed by EASI Genomics members, sequencing facilities as well as users who have successfully applied and obtained transnational access. In this regard, the MoU is an important, constitutional and structuring act for EASI-Genomics.

What have been the most challenging topics so far?

Applicability of Access and Benefit Sharing (ABS) Regulation to ancient DNA?

While different views were expressed among the experts, it appeared that this question was a grey area of law on which only little literature is available.

Safeguards for CRISPR CAS9 engineered biomaterials?

There does not seem to be a real international or European consensus about the rules on cross-border exchange of such biomaterials regarding any biosafety and biosecurity requirements.

How does EASI Genomics ensure ethical and legal compliance?

Interdisciplinary ethics check by two reviewers on projects that have been successfully evaluated by the Transational Access Committee

  . The goal is to verify and document ethical and legal compliance of research projects by collecting administrative authorisations. EASI Genomics is especially vigilant on : • Compliance with ethics rules for research with humans • Compliance with ethics rules for research with animals • Compliance with personal data protection rules • Compliance with the Nagoya Protocol • Publication of resulting knowledge in open access • Respect of the FAIR principles

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 824110


Malika Smail

Md Kamrul
email: kamrul.islam@loria.fr

Malika Smail-Tabbone
email: malika.smail-tabbone@loria.fr

From competition to collaboration: Ensembling similarity-based heuristics for supervised link prediction in biological graphs

Keywords: Biological graphs, Link prediction, Similarity-based heuristics, Supervised learning

published or not. The documents may come

From competition to collaboration: Ensembling similarity-based heuristics for supervised link prediction in biological graphs

Introduction

Many complex biological systems can be well-represented with graphs where a node represents a biological entity (e.g. protein, gene, etc.) and a link represents the interaction between two entities. Most real-world biological graphs are incomplete in nature. For example, 99.7% of the molecular interactions in human cells are still not known [START_REF] Stumpf | Estimating the size of the human interactome[END_REF]. The links in biological graphs must be validated by field and/or laboratory experiments, which are expensive and time consuming. Researchers have developed link prediction methods to compute the plausibility of a link between two unconnected nodes in a graph to avoid the blind checking of all possible interactions. Formally, link prediction is the task of predicting the likelihood of a link between two nodes based on available topological/attribute information of a graph [START_REF] Xu | Link prediction based on path entropy[END_REF]. Link prediction methods help us toward a deep understanding of the structure, evolution, and functions of biological graphs [START_REF] Shen | Reconstructing propagation networks with natural diversity and identifying hidden sources[END_REF].

Similarity-based methods are the simplest and unsupervised methods of link prediction in biological graphs, which define the proximity of a link by the similarity between its end nodes. The great advantage of these methods is their interpretibility which is essential for any biological system [START_REF] Zhou | Experimental analyses on 2-hop-based and 3-hop-based link prediction algorithms[END_REF]. However, each of the similarity-based methods performs well only in some particular graphs and no one wins in all graphs. These methods necessitate manually formulating various heuristics based on prior beliefs or extensive knowledge of various biological graphs. The lack of universal applicability of similarity-based methods motivates researchers to study machine learning methods to automatically learn the heuristics from a graph. To learn the appropriate heuristics automatically from a graph, researchers have developed embedding-based methods which represent nodes, edges, graphs in low dimensional vector space [START_REF] Cui | A survey on network embedding[END_REF]. The embeddingbased method has become a popular link prediction tool in graphs over the last decade. These methods show impressive link prediction performance in most of the graphs. The downside of embedding-based methods is that they seriously suffer from the well-known 'black-box' problem. As the link decisions in biological graphs are critical, a link prediction method should be sufficiently interpretable to achieve trust among stakeholders [START_REF] Gerke | Ethical and legal challenges of artificial intelligence-driven healthcare[END_REF]. The requirement for link prediction methods to be interpretable may limit the use of embedding-based methods in real-world biological systems. Researchers are still working on opening the 'black-box' of embedding-based methods [START_REF] Faber | Contrastive Graph Neural Network Explanation[END_REF][START_REF] Yuan | On explainability of graph neural networks via subgraph explorations[END_REF]. Another group of link prediction methods is developed based on traditional supervised learning-based methods. These methods extract features from a graph and train a traditional classifier for the link prediction task [START_REF] Cukierski | Graph-based features for supervised link prediction[END_REF][START_REF] Hasan | Link prediction using supervised learning[END_REF][START_REF] Berton | Link prediction in graph construction for supervised and semi-supervised learning[END_REF][START_REF] Benchettara | A supervised machine learning link prediction approach for academic collaboration recommendation[END_REF][START_REF] Ahmed | A supervised learning approach to link prediction in Twitter[END_REF][START_REF] Shibata | Link prediction in citation networks[END_REF][START_REF] Kumari | Supervised link prediction using structured-based feature extraction in social network[END_REF]. These methods are nearly as performant as embedding-based methods and as interpretable as similarity-based methods in many biological graphs. These methods describe the link prediction problem as a link classification problem with two classes: existence and absence of a link. In this paper, we intend to investigate whether the existing similarity-based heuristics collaboratively improve the link prediction performance in biological graphs. We study similarity-based heuristics for feature extraction and utilize the features in supervised learning-based classifiers for link prediction in biological graphs. We find that this is not the first attempt to study supervised learning methods to link prediction problem in graphs. But there are important differences between past works [START_REF] Hasan | Link prediction using supervised learning[END_REF][START_REF] Liben-Nowell | The link-prediction problem for social networks[END_REF][START_REF] De Sá | Supervised link prediction in weighted networks[END_REF]] and this study. The existing methods mostly focus on node attributes for extracting features which are application dependent. However, node attributes are not available in many real-world biological graphs. In contrast, our supervised learning-based method is developed based on only the topological features (similarity-based heuristics). Kumari et al [START_REF] Kumari | Supervised link prediction using structured-based feature extraction in social network[END_REF] studied a few local (four) and global (three) similarity heuristics for supervised link predictions, which is the closest work in the literature to our study. However, for large graphs, global methods are not the best option as they are computationally expensive [START_REF] Martínez | A survey of link prediction in complex networks[END_REF]. In this study, we enrich the feature set by including fourteen local similarity-based heuristics. In addition, we extract few other topological features of nodes and derive link-based features based on end node features. We study these features in supervised machine learning methods for link prediction in biological graphs. We see that supervised learning methods show comparable prediction results in many of the biological graphs. We also demonstrate the feature importance in different datasets for different supervised learning-based methods.

Similarity-based link prediction

Link prediction is the task of discovering or inferring a set of non-existing links in a graph based on the current snapshot of the graph. Similarity-based is the simplest category of link prediction methods, which is formulated based on the assumption that two nodes interact if they are similar in a graph [START_REF] Martínez | A survey of link prediction in complex networks[END_REF]. Generally, these methods compute similarity scores of non-existent links, sort the links in decreasing order of their scores and top-L links are predicted as potential existent links. Defining the similarity is a crucial and non-trivial task which differs from graphs to graphs [START_REF] Martínez | A survey of link prediction in complex networks[END_REF]. Consequently, numerous similarity-based methods exist in the literature. These methods are broadly categorized into three categories: local, global and quasi-local methods. Local methods are developed based on local topological or neighbourhood information, whereas global methods use the global topological information of graphs to define similarity functions [START_REF] Martínez | A survey of link prediction in complex networks[END_REF]. Quasilocal methods consider the neighbourhood up-to a predefined hop for defining the similarity function. The high computational time of global methods motivates us to study only local and quasi-local methods. We study fourteen well-known local similarity-based methods for link prediction in graphs, thirteen of which are summarized in Table . 1 local and one quasi-local. We summarize the similaritybased methods and the rest one (Preferential Attachment (PA)) in Table . 2 with basic principles and the definition of similarity functions.

Methodology

In a broader sense, we consider the similarity-based heuristics as individual features to generate the feature set for a supervised learning-based classifier. We describe each of the steps in Sections 2.1-2.3.

Feature extraction

The most crucial task of a supervised learning-based classifier is to define an appropriate feature set [START_REF] Hasan | Link prediction using supervised learning[END_REF]. Given a graph and a train set of links, we extract structural features for the train links. When extracting the features of a link, the link is temporarily removed from the graph and re-connected after feature extraction to ensure that the extracted features are not biased by the existence of the train link. We are motivated to use only topological features for defining our feature set as they exist in all kinds of graphs. Our feature set contains twenty topological features which are broadly categorized into two categories: similarity-based and derived link features (Fig. 1).

Table 1: Summary of similarity-based methods. Each method is considered as an individual link feature. S(x, y) is the similarity function between two end nodes x and y. Γ x and Γ x denote the neighbour sets of nodes x and y respectively. A is the adjacency matrix and λ is a free parameter.

Method Principle Similarity-function

Common

Neighbours (CN) [START_REF] Lorrain | Structural equivalence of individuals in social networks[END_REF] Two nodes are more likely to be linked if they have more neighbours in common.

CN (x, y) = |Γ x ∩ Γ y|
Adamic-Adar (AA) [START_REF] Adamic | Friends and neighbors on the web[END_REF] A variant of CN in which each common neighbour is penalized logarithmically by its degree.

AA(x, y) = z∈Γ x∩Γ y 1 log|Γ z|
Resource Allocation (RA) [START_REF] Zhou | Predicting missing links via local information[END_REF] Based on the resource allocation mechanism, the high degree common neighbours will be penalized even more.

RA(x, y)

= z∈Γ x∩Γ y 1 |Γ z|
Jaccard Index(JA) [START_REF] Jaccard | Étude comparative de la distribution florale dans une portion des Alpes et des Jura[END_REF] The score is punished for each noncommon neighbour in the normalization of CN.

JA(x, y)

= |Γ x∩Γ y| |Γ x∪Γ y|

Salton

Index(SA) [START_REF] Salton | Introduction to modern information retrieval[END_REF] The cosine similarity between adjacency vectors for a pair of nodes is used to compute the link probability.

SA(x, y)

= |Γ x∩Γ y| √ |Γ x|×|Γ y|
Sørensen Index(SO) [START_REF] Sorensen | A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons[END_REF] The overall fraction of common neighbours from a local perspective is what the link prediction is described as.

SO(x, y) = 2×|Γ x∩Γ y| |Γ x|+|Γ y|

Hub Promoted Index (HPI) [START_REF] Ravasz | Hierarchical organization of modularity in metabolic networks[END_REF] Link establishment between high-degree nodes and hubs is encouraged.

HP I(x, y) = |Γ x∩Γ y| max(|Γ x|,|Γ y|)
Hub Depressed Index (HDI) [START_REF] Ravasz | Hierarchical organization of modularity in metabolic networks[END_REF] Link establishment between low-degree nodes and hubs is encouraged.

HDI(x, y) =

|Γ x∩Γ y| min(|Γ x|,|Γ y|)

Local

Leicht-Holme-Newman (LLHN) [START_REF] Leicht | Vertex similarity in networks[END_REF] The real and expected number of shared neighbours are used to define the similarity between two nodes.

LLHN (x, y) = |Γ x∩Γ y| |Γ x|×|Γ y|

Cannistrai-Alanis-Ravai (CAR) [START_REF] Cannistraci | From link-prediction in brain connectomes and protein interactomes to the local-community-paradigm in complex networks[END_REF] In measuring the similarity score between two end nodes, level-2 linkages are combined with shared neighbourhood information.

CAR(x, y) =

z∈Γ x∩Γ y

+

|Γ x∩Γ y∩Γ z| 2

Clustering

Coefficient-based Link Prediction (CCLP) [START_REF] Wu | Link prediction with node clustering coefficient[END_REF] The influence of each shared neighbour is quantified by using the node's local clustering coefficient.

CCLP (x, y) = z∈Γ x∩Γ y

CCz

Node and Link Clustering(NLC) [START_REF] Wu | Predicting top-L missing links with node and link clustering information in large-scale networks[END_REF] The contribution of each common neighbor is quantified by using the node's node and link clustering coefficients.

N LC(x, y) = z∈Γ x∩Γ y CN (x,z) |Γ z|-1 × CCz + CN (x,z) |Γ z|-1 × CCz Local Path
Index(LPI) [START_REF] Lü | Similarity index based on local paths for link prediction of complex networks[END_REF] Similarity is calculated using the second and third order paths between the end nodes.

LP I(x, y)

= [A 2 + λA 3 ]x,y

Feature

Principle Link feature function Preferential Attachment (PA) [START_REF] Barabási | Emergence of scaling in random networks[END_REF] Based on the rich-get-richer principle, in which the link likelihood between two high-degree nodes is greater than that between two low-degree nodes.

P A(x, y) = |Γ x| × |Γ y|
Pager Rank (PR) [START_REF] Langville | A survey of eigenvector methods for web information retrieval[END_REF] PageRank computes a ranking of the nodes based on the structure of the links.

P R(x, y) = P R(x) + P R(y)

Clustering Coefficient (CC) [START_REF] Onnela | Intensity and coherence of motifs in weighted complex networks[END_REF] The clustering coefficient of a node is the fraction of possible triangles through that node that exist

CC(x, y) = CC(x) + CC(y)
Degree Centrality (DC) [START_REF]Encyclopedia of systems biology[END_REF] The degree centrality for a node is the fraction of nodes it is connected to.

DC(x, y) = DC(x) + DC(y)

Eigen vector centrality (EVC) [START_REF] Bonacich | Power and centrality: A family of measures[END_REF] Eigenvector centrality computes the centrality for a node based on the centrality of its neighbors.

EV C(x, y) = EV C(x) + EV C(y)

Closeness Centrality(CLC) [START_REF] Freeman | Centrality in networks: I. conceptual clarifications[END_REF] Closeness centrality of a node is the reciprocal of the average shortest path length to other reachable nodes.

CLC(x, y) = CLC(x) + CLC(y)

Vote Rank Centrality(VRC) [START_REF] Zhang | Identifying a set of influential spreaders in complex networks[END_REF] Ranking of the nodes based on a voting scheme where a node casts votes to its neighbours. A node with the highest votes has the best (lowest) ranking.

V RC(x, y) = 1 V RC(x) + 1 V RC(y)
Similarity-based link features : We define the link-based features as the features which are related to the common topological information of end-nodes of a link. We use thirteen existing similarity-based heuristics as link-based features, which are summarized in Table 1. For instance, the number of common neighbours of end nodes of a link is used as the common neighbour(CN) feature.

Derived link features : Few link-based features are derived from the individual features of the link's end nodes. We summarized six derived features in Table 2. These features are related to the topological information of individual nodes only. For example, the degree of end nodes is multiplied in Preferential Attachment (PA) to define the similarity score. Note that the link features in Table 2 except PA are not directly defined in the literature. We derive the link features based on the end node feature. To compute the link feature, features of end nodes are simply added except PA. As the voterank centrality computes low ranks for high-influencing nodes in a graph, the reciprocals of the voterank scores of end nodes are summed to define the voterank centrality feature.

Feature scaling

In general, the magnitude scale for different features in different graphs varies [START_REF] Islam | Appraisal study of similaritybased and embedding-based link prediction methods on graphs[END_REF][START_REF] Islam | An experimental evaluation of similarity-based and embedding-based link prediction methods on graphs[END_REF]. Supervised learning-based methods are easily affected by the non-uniform scaling as there is a high chance that features with higher magnitude play a more decisive role during the training of a classifier. But, it is not desirable for the classifier to be biased towards one particular feature. Hence, we normalize each feature in the range of 0-1.

Classifier training and link prediction

For the link prediction task, we train a traditional supervised machine learning classifier to classify a link into either existent or non-existent classes. There exist many classifiers in the literature which perform better than others in some particular datasets. In this paper, we study three traditional classifiers: Support Vector Machine(SVM) with RBF kernel, Decision Tree, and Logistic Regression. We extract the features of the test links and classify them into existent or non-existent classes using a trained classifier to evaluate the link prediction performance.

Experiments

The baselines

To evaluate the prediction performance of supervised learning methods, we consider two categories of link prediction methods: similarity-based and embeddingbased methods.

For the similarity-based category, we consider all the heuristics in Table 1 in Table 2. For the embedding-based methods, we choose two popular methods: Node2Vec [START_REF] Grover | node2vec: Scalable feature learning for networks[END_REF] and SEAL [START_REF] Zhang | Link prediction based on graph neural networks[END_REF]. We shortly describe Node2Vec and SEAL methods. For more details, we refer to the original papers. Node2Vec [START_REF] Grover | node2vec: Scalable feature learning for networks[END_REF] is a classical skip-gram model-based graph embedding method which learns node embeddings by optimizing a neighbourhood preserving objective function. It makes an interpolation between BFS(Breadth First Search) and DFS(Depth First Search) to define a 2 nd order random walk. A fixed size neighbourhood is sampled using the 2 nd order random walk and fed into the well-known skip-gram model [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF] to learn the node embedding. The link embedding is then computed as the Hadamard product of the end node embeddings. A logistic regressionbased classifier is then trained for the link prediction task. SEAL, the second embedding-based approach, is based on neural networks (NN). Learning from Sub-graphs, Embeddings and Attributes (SEAL) utilizes the latent and explicit features of end nodes and structural information of the graph to learn the link embedding. SEAL starts with extracting a h-hop neighbouring subgraph and node labeling by a double radius node labeling (DRNL) algorithm. In the second step, the labelled sub-graph is then used to generate the structural encoding. The link embedding is the concatenation of structural encoding, pre-computed latent encoding and explicit feature encoding. In the final step, a neural network(NN) is trained for link prediction task.

Experimental datasets

In this study, we focus on only biological graphs. For evaluating performance, we collect six biological graphs from the Network Repository1 . Table 3 summarizes the topological statistics and descriptions of the graph datasets. The link prediction performance is evaluated using a random sampling validation protocol [START_REF] Zhang | Link prediction based on graph neural networks[END_REF][START_REF] Islam | Appraisal study of similaritybased and embedding-based link prediction methods on graphs[END_REF][START_REF] Islam | An experimental evaluation of similarity-based and embedding-based link prediction methods on graphs[END_REF]. For a graph dataset, train and test sets are prepared by splitting the existent links. The train set consists of 90% existent and an equal number of non-existent links. The test set contains the remaining 10% existent and equal number of non-existent links. To prepare five train and five test sets for each graph, we repeat the link splitting operation five times independently. The datasets are available in a GitLab repository2 .

Evaluation metrics

The link prediction problem is considered as a binary classification problem [START_REF] Kumar | Link prediction techniques, applications, and performance: A survey[END_REF]. A traditional classifier, in general, learns a threshold to classify links as existent or non-existent. However, for similarity-based link classification methods, we find no standard approach for computing the threshold. The threshold is calculated in an optimistic manner. We first normalize the link scores to a range of 0-1 and then use the normalized scores to compute a ROC curve. The curve gives the true positiverate (TPR) and false positive rate (FPR) for different score threshold settings.The threshold point with the highest [TPR+(1-FPR)] is computed as the threshold as we want to maximize TPR as well as minimize FPR. We classify links based on this threshold. A link with a score >= threshold is classified as existent and non-existent otherwise. Based on the true and predicted classes of links, we define four metrics: true positive (TP), true negative (TN), false positive (FP), and false negative (FN). TP is the number of existent links predicted to be existent, TN is the number of non-existent links predicted to be non-existent, FP is the number of non-existent links predicted to be existent, and FN is the number of existent links predicted to be non-existent links. We compute the following three well-known metrics using these four metrics.

Recall = T P T P + F N (1)
P recision = T P T P + F P

(2)

F 1 = 2 × P recision × Recall P recision + Recall (3)

Results and discussion

In this section, we describe the prediction performance of supervised learningbased methods on six biological graphs. We also illustrate the importance of the features in graphs.

Prediction performance

The prediction performance is computed for all methods over all the five sets for each graph, and the average scores are recorded. We do not include the standard deviation results as the values are very low in all the experiments. The precision, recall and F1 scores are tabulated Table 4, where the best two similarity-based methods are denoted with Sim 1 and Sim 2 . We compute the precision scores of similarity-based methods in a optimistic way. The precision scores of similarity-based methods (best and second best) are very high and highest among all the methods in all the graphs, as shown in the table. This demonstrates the ability of similarity-based methods to predict high-quality links. However, the recall scores are low, implying that these methods identify the majority of existing test links as non-existent. As a result, the F1 score for similarity-based methods is very low. We also see that, as expected, the two best-performing similarity-based methods differ for different datasets. Among the supervised learning methods (SVM, DT, LR), DT shows the worst prediction results, but it is still much better than similarity-based methods. The other two classifiers have similar performance scores. The performance of the other two classifiers in terms of prediction scores is impressive. Yet in many graphs, supervised learning-based classifiers show superior prediction performance than embedding-based methods. Relating the performance to graph properties, we see that traditional classifiers outperform embedding-based methods in dense Fig. 2: Feature importance in HS-HT graph by Logistic regression classifier python-based machine learning tool3 . When a feature is unavailable, the coefficient is calculated by looking at how much the score (accuracy) drops [START_REF] Breiman | Random forests[END_REF]. The higher the coefficient, the higher the importance of the feature. In Fig. 2, we demonstrate the feature importance in the HS-HT biological graph in the logistic regression (LR) classifier to investigate how the importance of features differs in different sets of the same biological graph. In the LR classifier for the HS-HT biological graph, four features dominate. The dominance of multiple heuristics or features in a graph shows that heuristics that work collaboratively perform bet-ter than heuristics that work alone. We can also find that the feature importance coefficients in all five sets in the HS-HT graph are substantially identical. We further investigate the importance score of features in three classifiers (SVM, DT, LR) for three different datasets. We evaluate the importance score of features for only one set for each graph. We see that different classifiers give different importance coefficients to different features in different datasets. In DM-HT dataset, all the classifiers compute high coefficient for LPI feature and they have close prediction performance (in Table 4). In the Celegans dataset, the HPI feature dominates in SVM and LR classifiers whereas LPI dominates in the DT classifier. In the Celegans dataset, SVM and LR outperform DT in terms of prediction (in Table 4), demonstrating that LR and SVM compute feature importance scores more correctly. Surprisingly, we see that DT has a tendency to give more importance to the LPI feature in these three datasets.

Conclusion

Do similarity-based heuristics compete or collaborate for link prediction task in graphs? In this article, we study this question. We study fourteen similaritybased heuristics in six biological graph from three different organisms. As expected, we observe they perform well only in some particular biological graphs and no one wins in all graphs. Rather than using them as standalone link prediction methods, we consider them as features for supervised learning methods. In addition, we derive six link features based on the node's topological information. Based on the twenty features, we train three traditional supervised learning methods: SVM, DT and LR-based classifiers. We see that the similarity-based heuristics collaboratively improve link prediction performance remarkably, even outperforming embedding-based methods in some graphs.

We propose three future dimensions of this study. Firstly, studying collaboration of similarity-based heuristics in large scale biological as well as social graphs could be a potential future work as the graphs in the current study are small/medium in size. Secondly, exploring some other heuristics might improve prediction performance in sparse graphs. The final future research could be studying other classifiers like Random Forest, AdaBoost, K-Neighbors for the link prediction task in graphs.

Fig. 1 :

 1 Fig. 1: Feature set for supervised learning

Fig. 3 :

 3 Fig. 3: Feature importance in different datasets by different supervised methods: (a)-(c) in Celegans, (d)-(f) in Diseasome, (g)-(i) in DM-HT

Table 2 :

 2 Summary of derived link features: The derived link feature function S(x, y) is defined based on end nodes features.

Table 3 :

 3 The graph datasets: number of nodes(|V|), links(|E|), average node degree (NDeg), average clustering coefficient (CC), and description.

	Graph	Organism |V|	|E|	NDeg CC Description
	CE-GT [43] Worm	924	3239 7.01	0.605 Nodes: Genes in C. elegans
						Links: Gene functional associations in C.elegans
	CE-HT [43] Worm	2617 2985 2.28	0.008 Nodes: Proteins in C. elegans
						Links: High-throughput protein-protein interac-
						tions
	Celegans	Worm	453	2040 9.01	0.647 Nodes: Substrates in Caenorhabditis elegans
	[43]
						Links: Metabolic reactions between substrates
						in C.elegans
	CE-LC [44] Worm	1387 1648 2.37	0.076 Nodes: Proteins in C.elegans worm
						Links: Small/medium-scale protein-protein in-
						teractions (compiled from protein-protein inter-
						action data bases)
	Diseasome	Human	516	1188 4.61	0.636 Nodes: Known genetic disorders in H.sapiens
	[45]
						Links: Connections between pair of disorders
						when they share minimum one gene.
	DM-HT [45] Fly	2989 4660 3.12	0.009 Nodes: Proteins in D.melan-ogaster fly
						Links: High-throughput protein-protein interac-
						tions
	DM-LC [44] Fly	658	1129 3.43	0.105 Nodes: Proteins in D.melanogaster fly
						Links: Small/medium-scale protein-protein in-
						teractions (compiled from protein-protein inter-
						action data bases)
	HS-HT [44] Human	2570 13691 10.65 0.169 Nodes: Proteins in human
						Links: Protein-protein interactions in human
						protein network
	SC-LC [44] Yeast	2004 20452 20.41 0.168 Nodes: Proteins in S.cerevisiae yeast
						Links: Small/medium-scale protein-protein in-
						teractions in yeast network
	Yeast [44]	Yeast	2114 2277 2.15	0.059 Nodes: Proteins in S.cerevisiae yeast
						Links: Protein-protein interactions in yeast net-
						work

Table 4 :

 4 Performance metrics: The dataset-wise best and second best precision, recall and F1 scores are indicated in bold and underline. The best and second best similarity-based methods are denoted with Sim 1 andSim 2 respectively. For Sim 1 and Sim 2 methods, the methods are specified and the performance scores are given in (). This is intuitive as the majority of the studied similarity-based heuristics are based on common neighbours (see Table.1). The performance scores of traditional classifiers are worse in the sparse graphs (CE-HT, CE-LC, Yeast), where embedding-based methods show better performance scores.Feature importance In this section, we investigate the influence of each feature in a classifier for the link prediction task. To compute the feature importance coefficient, we use the Permutation importance module from the sklearn

	Datasets Metric Sim 1	Sim 2	N2V	SEAL SVM DT	LR
		Precision NLC (0.960) JA (0.896)	0.707 0.842 0.842 0.828 0.901
	CE-GT Recall	NLC (0.039) JA (0.042)	0.707 0.931 0.827 0.776 0.900
		F1	NLC (0.075) JA (0.078)	0.707 0.885 0.834 0.801 0.901
		Precision RA (0.996) AA (0.996) 0.596 0.705 0.753 0.745 0.752
	CE-HT Recall	RA (0.001) AA (0.001) 0.593 0.791 0.529 0.519 0.510
		F1	RA (0.002) AA (0.002) 0.594 0.745 0.622 0.612 0.608
		Precision RA (0.938) CCLP (0.932) 0.778 0.806 0.899 0.850 0.907
	Celegans Recall	RA (0.042) CCLP (0.041) 0.777 0.888 0.899 0.830 0.906
		F1	RA (0.08)	CCLP (0.079) 0.778 0.845 0.899 0.840 0.906
		Precision AA (0.969)	RA (0.969)	0.658 0.763 0.715 0.763 0.789
	CE-LC Recall	AA(0.009)	RA(0.009)	0.647 0.794 0.620 0.584 0.673
		F1	AA(0.028)	RA(0.028)	0.652 0.778 0.664 0.662 0.726
		Precision NLC (0.991) AA (0.988) 0.757 0.914 0.926 0.800 0.927
	Diseasome Recall	NLC (0.035) AA (0.040) 0.756 0.896 0.919 0.692 0.920
		F1	NLC (0.067) AA (0.078) 0.756 0.905 0.922 0.742 0.924
		Precision CCLP (0.999) NLC (0.998) 0.712 0.720 0.780 0.796 0.770
	DM-HT Recall	CCLP (0.001) NLC (0.001) 0.704 0.703 0.657 0.661 0.644
		F1	CCLP (0.002) NLC (0.002) 0.708 0.712 0.714 0.722 0.701
		Precision PA (0.979)	CCLP(0.944)	0.696 0.790 0.829 0.828 0.812
	DM-LC Recall	PA(0.02)	CCLP(0.007)	0.688 0.835 0.771 0.770 0.777
		F1	PA(0.039)	CCLP(0.014)	0.692 0.812 0.799 0.798 0.794
		Precision NLC (0.954) CCLP (0.949) 0.797 0.854 0.861 0.847 0.861
	HS-HT Recall	NLC (0.031) CCLP (0.031) 0.794 0.815 0.840 0.791 0.848
		F1	NLC (0.060) CCLP (0.061) 0.796 0.834 0.850 0.818 0.854
		Precision NLC (0.893) AA (0.873)	0.772 0.784 0.868 0.850 0.853
	SC-LC Recall	NLC (0.035)	AA (0.036)	0.770 0.815 0.849 0.810 0.844
		F1	NLC (0.067)	AA (0.068)	0.771 0.799 0.859 0.829 0.849
		Precision CCLP (0.971) RA (0.967)	0.699 0.705 0.753 0.746 0.755
	Yeast	Recall	CCLP (0.006) RA (0.008)	0.699 0.726 0.567 0.551 0.598
		F1	CCLP (0.012) RA (0.015)	0.699 0.716 0.647 0.634 0.668
	graphs.

https://networkrepository.com/bio.php

https://gitlab.inria.fr/kislam/supervised-lp

https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation importance.html