Nikolaos L Tsakiridis
email: tsakirin@ece.auth.gr

Themistoklis Diamantopoulos

Andreas L Symeonidis

John B Theocharis
email: theochar@eng.auth.gr

Athanasios Iossifides
email: aiosifidis@el.teithe.gr

Periklis Chatzimisios

George Pratos
email: pratos@indinf.gr

Dimitris Kouvas

Versatile Internet of Things for Agriculture: An eXplainable AI Approach

Keywords: Precision Irrigation, Internet of Things, eXplainable AI

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The number of devices connected to the Internet, known as IoT (Internet of Things) devices, has been continuously growing. Indicatively, in 2008 their number exceeded the global population, while in 2019 there were more than 26 billion connected devices 5 . This number is expected to reach 75 billion by 2025 6 . Apart from home/consumer use, IoT today is largely used in different industrial sectors, including manufacturing, security, transportation, and agriculture [START_REF] Sharma | The History, Present and Future with IoT[END_REF].

As the global population is expected to rise 7 , there is a demand for augmenting food production coupled with increased use of arable land water resources [START_REF] Fao | The State of Food Security and Nutrition in the World 2019. Safeguarding against economic slowdowns and downturns[END_REF]. The challenge of achieving sustainable agriculture and increasing food production using technologies such as Artificial Intelligence (AI) and IoT has become evident ever since the beginning of the 21st century, when Zhang et al. [START_REF] Zhang | Precision Agriculture -A Worldwide Overview[END_REF] coined the term precision agriculture. The goal of this area is to maximize production while mitigating the relevant costs, i.e. minimizing water/energy consumption and optimizing the use of pest-control/growth chemicals.

Research in precision agriculture is broad, focusing on all three layers of IoT: the perception layer (i.e. sensors that are placed at fields), the network layer (i.e. communication protocols between sensors and applications), and the application layer (i.e. data storage and decision-making) [START_REF] Tzounis | Internet of Things in agriculture, recent advances and future challenges[END_REF]. Several challenges still exist: the data transfer cost, the need for uninterrupted power supply (that undermines their applicability), the lack of efficient and versatile data storage, and most importantly the need for automated control systems that are smart and can support precision agriculture with minimal human intervention.

The advent of increasingly opaque decision systems, such as deep and ensemble learning techniques, highlights a key issue: when decisions derived from such systems affect the livelihood of humans, as in agriculture, there is an increasing need to understand how the AI methods derive them. The consideration of interpretability as an additional design driver can improve the models by: i) facilitating the robustness by highlighting the potential perturbations, ii) ensuring impartiality in decision-making, and iii) guaranteeing that an underlying truthful causality (linking the cause with the effect) exists in model reasoning.

In this work, we present a system overcoming the aforementioned limitations. Our system, named Vital, employs IoT devices with low installation, use, and maintenance costs, capable of supporting open-field irrigation by measuring all relevant metrics (temperature, humidity, etc.). Furthermore, our system employs a distributed and scalable data management infrastructure, and a smart fuzzy rule-based system that fully automates the irrigation of open fields. Our system lies in the broad category of eXplainable AI (XAI) [START_REF] Fernandez | Evolutionary fuzzy systems for explainable artificial intelligence: Why, when, what for, and where to[END_REF][START_REF] Arrieta | Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI[END_REF] systems, therefore it produces interpretable high-efficiency results, while enabling humans to inspect the model reasoning and thus understand, trust, and manage the AI technologies.

Background and Related Work

IoT has already been applied in several areas relevant to agriculture [START_REF] Tzounis | Internet of Things in agriculture, recent advances and future challenges[END_REF]. In this section we focus on IoT for precision agriculture and specifically on what is referred to as open-field agriculture. Open-field deployments typically comprise sensors that measure climate conditions as well as soil sensors, and their main purpose is to optimize irrigation and enhance crop production.

One of the first systems to support such functionality [START_REF] Khriji | Precision Irrigation based on Wireless Sensor Network[END_REF] is based on three nodes: one measuring soil moisture and temperature, another for weather parameters (air temperature and humidity, wind speed, and luminosity), and one for irrigation control. The first two nodes send data to a base station and the user can manually activate the valve connected to the third node to irrigate the field. A similar system was proposed in [START_REF] Shuwen | Study on farmland irrigation remote monitoring system based on zigbee[END_REF], which also monitors these metrics and further integrates a solar-power node to reduce the energy needs of the system. Project SWAMP [START_REF] Kamienski | SWAMP: an IoT-based Smart Water Management Platform for Precision Irrigation in Agriculture[END_REF] takes water control one step further, by integrating not only water consumption but also water reserve and distribution metrics. All of these systems, however, are not automated, so an expert agronomist has to continuously monitor them and take any decisions concerning irrigation.

An interesting alternative proposed in [START_REF] Muangprathub | IoT and agriculture data analysis for smart farm[END_REF] employs a set of rules to irrigate automatically based on soil moisture. These are set by the administrator (usually an agronomist) based on past data, while the farmer may override them and manually irrigate. A similar algorithm based on sensor and weather data is proposed in [START_REF] Mohanraj | Field Monitoring and Automation Using IOT in Agriculture Domain[END_REF]. Though useful, these approaches still require expert monitoring. A more advanced method is proposed in [START_REF] Goap | An IoT based smart irrigation management system using Machine learning and open source technologies[END_REF], employing Support Vector Regression to predict soil moisture for a range of upcoming days, which is then used to determine whether and when irrigation should start. The main drawback in this case lies with interpretability; the inability to review the underlying model is a risky practice, considering errors may prove critical for the crop.

As a result, several research efforts have been directed towards fuzzy rulebased systems (FRBSs), which are interpretable and generalize well in cases when data are derived from sensors and thus may be too fine-grained. The involvement of fuzzy logic is justified by the following: i) expert knowledge is fuzzy and not precise and can thus be easier integrated, ii) data from sensors in the real world are noisy and are reported with a degree of fuzziness, iii) fuzzy consequents enable a smoother output response than traditional crisp systems [START_REF] Arrieta | Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI[END_REF][START_REF] Fernandez | Evolutionary fuzzy systems for explainable artificial intelligence: Why, when, what for, and where to[END_REF]. One of the first approaches [START_REF] Gao | An intelligent irrigation system based on wireless sensor network and fuzzy control[END_REF] includes a fuzzy logic model that receives as input soil humidity values from a sensor and determines the irrigation duration using four linguistic variables: zero, short-term, middle-term, and large-time. A more complex model is proposed in [START_REF] Yadav | Fuzzy Based Smart Farming using Wireless Sensor Network[END_REF], which is based on temperature (low, moderate, high), soil moisture (dry, moderate, wet), and humidity (less, medium, high) and results in a set of rules capable of determining pump operation and its duration. In [START_REF] Mohapatra | Neural Network Pattern Classification and Weather Dependent Fuzzy Logic Model for Irrigation Control in WSN Based Precision Agriculture[END_REF] five triangular membership functions are used as input corresponding to sunlight intensity, wind speed, humidity, temperature, and soil moisture, while the output indicates the necessary irrigation quantity. Similarly, the model proposed in [START_REF] Kokkonis | A Smart IoT Fuzzy Irrigation System[END_REF] uses Gaussian membership functions and receives input from ten sensors, two air sensors (for humidity and temperature) and eight ground moisture sensors placed in different parts of the area, to provide more accurate recommendations.

Although the aforementioned systems are quite effective in certain scenarios, they are usually not applicable to multiple generic precision irrigation scenarios. They usually require human intervention in order to adapt to different fields and they are largely based on specific sensors that may not be available in all cases (e.g. certain systems may require a very large number of sensors [START_REF] Kokkonis | A Smart IoT Fuzzy Irrigation System[END_REF]).

In this work, we propose a flexible IoT system operating on a set of sensors that is upgradeable and further employs a data management infrastructure able to handle large amounts of values from these sensors. We design an interpretable FRBS which functions automatically, thus not requiring any input from the farmer/agronomist. As mentioned, our system can be classified into the XAI domain. Within this domain, FRBSs have demonstrated their potential to attain both interpretability and accuracy in a variety of application areas [START_REF] Fernández | A View on Fuzzy Systems for Big Data: Progress and Opportunities[END_REF][START_REF] Tsakiridis | An evolutionary fuzzy rulebased system applied to the prediction of soil organic carbon from soil spectral libraries[END_REF][START_REF] Tsakiridis | Using interpretable fuzzy rule-based models for the estimation of soil organic carbon from VNIR/SWIR spectra and soil texture[END_REF].

Vital: An IoT System for Precision Agriculture

The Vital system comprises the following components (depicted in Figure 1):

-The slave devices (end nodes) connected physically with the devices (e.g. weather stations, sensors, valves etc.); -The IoT gateway that wirelessly connects to multiple slave devices and exchanges data and other information; -The IoT platform which collects the data from multiple gateways, transmitted through 3G networks, and acts as the data focal point; -The web application acting as the front-end of the system and relaying the output of the rule-based system via SMS to the relevant gateway; and -The rule-based system which reads the data stored in the IoT platform and collected by the web application, and infers data-driven decisions.

SMS alarms and commands

IoT Platform

3G data transmission

Rule-based System

SMS commands

End node (slave device)

Web Application

Wireless Slaves

The wireless slaves are ultra low-power, battery-powered end nodes in the network. The sensor nodes are frequently synchronized by the gateway in order to achieve real-time data collection with a fixed update rate of 5 minutes. Two types of slave devices were considered, namely ADS-200 and ADS-210. ADS-200 is the end-node used to receive input data from sensors, while ADS-210 is used to drive 12V latching valves and relays. Their D-size Lithium Thionyl battery can provide autonomous operation for over 10 years. ADS-200 incorporates 1 configurable input (digital, pulse counter or analog), and multiple excitation options for powering transducers. The device supports acquisition of up to 16 measurement channels, based on the popular SDI-12 communication protocol and 8 channels, based on the MODBUS protocol. ADS-210 on the other hand, incorporates 2 digital inputs, 1 with counting capability, 1 analog input, and a full bridge output for driving 12V latching valves and relays.

IoT Gateway

The IoT gateway (ADU-700) communicates wirelessly in license-free bands with the slaves and transmits their data to the IoT platform. The gateways can both receive the data (telemetry) from the slaves (upstream component) and relay the commands received through SMS text messages (downstream component), to perform specific tasks, like opening or closing a valve. Able to withstand ambient conditions, they are low-power IP66 certified and capable of securely connecting to multiple slave devices. The wireless network coverage reaches a radius of 1-6 km (line of sight), enabling the use of one gateway across multiple fields. The network uses a proprietary protocol with bidirectional communications. Modes of operation include the autonomous battery operation and an external power supply operation with uninterruptible transition to battery operation during power outage. The ADU-700 incorporates a 3G, GSM/GPRS modem and supports periodical data transmission via FTP and alerting via SMS. The data are transmitted to the IoT platform at regular intervals, configurable by the user.

IoT Platform

The IoT platform is the central datahub, i.e. where all telemetry values are securely stored. In this implementation, the Cenote platform is used [START_REF] Chatzidimitriou | Cenote: A Big Data Management and Analytics Infrastructure for the Web of Things[END_REF]. Cenote is a Big Data Management System (BDMS) with analytics capabilities for the Web of Things, providing (near) real-time analytics capacities to event streams stemming from heterogeneous sources of information. Moreover, it is an open source platform following a component-based development approach. Deployed in a distributed and scalable manner, it acts as a general-scope, out-of-the-shelf, BDMS, supporting analytics out-of-the-box in many scenarios that involve event stream processing. Its overall architecture is presented in Figure 2.

Web Application

The web front-end of the platform was developed in Javascript and is an interface to visualize telemetry data. It further has an API connecting to an external SMS service, enabling the platform to task a gateway with actions via SMS commands (e.g. for irrigation this concerns the opening of a valve for a specified number of minutes). Moreover, it enables the management of data (e.g. translating gateway ids to actual fields, using proper units, etc.), manages the access of users, and enables users to inspect and tweak the rule-based system. This latter component is very crucial for XAI; by enabling users to visualize the inner manifestations of the rule-based system the model is transparent and thus trustworthy.

Rule-based System

Fuzzy systems are an important tool in the domains of Computational Intelligence and Soft Computing, having been applied to a plethora of different tasks.

Most commonly, a model structure in the form of an FRBS is considered. These systems have displayed their superiority to classical decision trees and crisp rule-based systems and their robustness in different AI applications including classification [START_REF] Tsakiridis | DECO3R: A Differential Evolutionbased algorithm for generating compact Fuzzy Rule-based Classification Systems[END_REF][START_REF] Antonelli | Multiobjective evolutionary optimization of type-2 fuzzy rule-based systems for financial data classification[END_REF][START_REF] Elkano | CFM-BD: A distributed rule induction algorithm for building compact fuzzy models in big data classification problems[END_REF], big data analysis [START_REF] Tsakiridis | An evolutionary fuzzy rule-based system applied to real-world Big Data -The GEO-CRADLE and LUCAS soil spectral libraries[END_REF][START_REF] Elkano | CFM-BD: A distributed rule induction algorithm for building compact fuzzy models in big data classification problems[END_REF], and regression [START_REF] Tsakiridis | DECO3RUM: A Differential Evolution learning approach for generating compact Mamdani fuzzy rule-based models[END_REF][START_REF] Rodríguez-Fdez | S-FRULER: Scalable fuzzy rule learning through evolution for regression[END_REF]. FRBSs allow for the definition of simple and verbally formulated rules over imprecise domains which can be combined to generate precise yet understandable results. They are an extension to classical rule-based systems, since they involve "IF-THEN" type of rules, but the antecedents and consequents comprising each rule are composed of fuzzy logic statements, instead of classical ones. The use of fuzzy sets with linguistic labels is particularly noteworthy, as the output system has a significant interpretability degree for the expert to understand the reasoning mechanisms of the model and the inner details of the problem characteristics. These types of systems are called Mamdani-type FRBSs [START_REF] Cordón | A historical review of evolutionary learning methods for Mamdani-type fuzzy rule-based systems: Designing interpretable genetic fuzzy systems[END_REF] and have two main components (illustrated in Figure 3): i) the fuzzy inference system, which implements the fuzzy reasoning process to be applied on the input to infer the output, and ii) the fuzzy knowledge base (KB), representing the knowledge about the problem being solved. The KB contains the fuzzy rules composed of linguistic variables that take values from a descriptive term set with a real-world meaning. This distinction between the fuzzy rule structures and their meaning allows us to define two different sub-components, namely the fuzzy rule base (RB) containing the collection of fuzzy rules, and the data base (DB) containing the membership functions of the fuzzy partitions associated to the linguistic variables.

The Mamdani-type FRBS structure demonstrates several features that are of particular importance for the application presented herein: i) firstly, it provides a natural framework to include expert knowledge in the form of linguistic fuzzy rules; ii) secondly, the fuzzy inference mechanism makes full use of the power of fuzzy logic-based reasoning; and iii) thirdly, it composes a highly flexible means to formulate knowledge, while at the same time it remains interpretable.

More formally, given a dataset

E = {x p 1 , . . . , x p N , y p } Q p=1
, which is composed of Q patterns, with each pattern consisting of an N -dimensional predictor x and one response (or target) variable y, the FRBS models the input-output relationship using a total of R fuzzy rules of the form:

R k : IF x 1 is Ãk 1 AND . . . AND x N is Ãk N THEN y is B k with CF k
where R k is the k-th fuzzy rule, Ãk i is the set of interpretable linguistic terms for feature i, B k is the linguistic term for the output, and CF k is the certainty factor (or degree of importance) for B k .

The input space and the output variable are partitioned using uniformly distributed triangular membership functions into ℓ in and ℓ out fuzzy sets respectively. Thus, they constitute a strong (or proper) fuzzy partition [START_REF] Klir | Fuzzy Sets and Fuzzy Logic: Theory and Applications[END_REF], implying that in a given sample space X :

∀x : ℓ ∑ q=1 µ L q (x) = 1 (1)
where L q is the q-th linguistic fuzzy set of space X . The DNF (disjunctive normal form) approach [START_REF] Cordón | A historical review of evolutionary learning methods for Mamdani-type fuzzy rule-based systems: Designing interpretable genetic fuzzy systems[END_REF] is used, where each input variable x i of R k takes a set of consecutive linguistic terms as a value Ãk i = {L 1 i or . . . or L ℓin i }, joined by the disjunctive operator OR. Figure 4 depicts the DNF approach in a strong fuzzy partition using triangular membership functions for a given space X .

The matching degree between a pattern p and a rule R k is: where µ L q i (x p i ) is the membership grade of each linguistic term, while ∧ and ∨ denote the AND and OR operators, respectively. The AND operator is implemented using the minimum operator, whereas the OR operator is implemented using the bounded sum, defined as boundedSum(a, b) = min (1, a + b). Given an input vector x p , the output of the system is calculated in the following way:

µ k A (p) = N ∧ i=1 Ãk i = N ∧ i=1 { ℓin ∨ q=1 µ L q i (x p i ) } (2)
ŷp = R ∑ k=1 {µ A (p) • CF k • CoG(B k )}/{ R ∑ k=1 µ k A (p) • CF k } (3)
The DB parameters were ℓ in = 5 (i.e. five uniformly distributed triangular sets for the input variables) and ℓ out = 7. The larger granularity in the output space allows for more fine-grained irrigation control. The learning of the RB through the dataset takes place by following the learning algorithm presented in [START_REF] Tsakiridis | DECO3RUM: A Differential Evolution learning approach for generating compact Mamdani fuzzy rule-based models[END_REF].

The rule-based system is continuously operating and monitoring the raw telemetry; once new data are received the rules are checked. If the conditions are met (i.e. the proper rules are activated by the input) the relevant action is determined from the output of the rule base and sent to the gateway as an SMS command. If no rules are activated, then no irrigation command is transmitted.

Pilot Cases

To demonstrate the efficacy of our approach we considered two independent pilot cases which are presented below.

Integration with Existing Telemetry Network

In this pilot case, the goal was to demonstrate the potential of the proposed system to integrate data of existing (legacy) network stations. This demonstrates the suitability of this work to replace legacy technologies that were installed but were either malfunctioning or underfunctioning or had no web-based integration capabilities. Accordingly, a set of environmental sensors was selected that was operating in lake Koronia, located in the Mygdonian basin of the Region of Central Macedonia in Northern Greece (latitude/longitude of 40.68/23.15). The environmental sensors record the ambient temperature and relativity humidity, as well as the level of the lake in discrete and selected positions. The sensors were connected via the SDI-12 interface using three separate slave devices, while a single gateway acted as the local data collector. The integration was successful, indicating that our system can integrate and/or replace legacy installations.

Precision Irrigation

In this pilot application we focused on a pilot area of young olive trees within the premises of the Farm School the Aristotle University of Thessaloniki in Northern Greece (at a latitude/longitude of 40.54033/23.00083), with the field covering an area of 12 acres and consisting of 1780 individual trees, using a drip irrigation system. The field was then divided into 4 equal areas (hereafter referred to as plots), each controlled individually by a separate irrigation valve. One of the plots was used as control (i.e. operated manually by an expert), another was operated by the expert using remote commands (i.e. via SMS commands), while the other two were irrigated by the automated system. An all-in-one weather station (ATMOS 41) was placed in the center of the field to supply the meteorological data (i.e. solar irradiance, precipitation, wind speed and direction, temperature and humidity). In the three experimental plots a GS3 sensor by Decagon was installed in the soil that measures the soil's temperature, conductivity and dielectric permittivity. All sensors and irrigation valves were connected to the slave devices that relayed the telemetry to the gateway and thence it was transmitted to the Cenote platform over TLS. The raw telemetry data were checked for abnormal values (e.g. if the data were below or above predefined minima and maxima values) before they were stored to the database.

In addition to the raw telemetry values, the soil moisture content (calculated via the Topp equation [START_REF] Topp | Electromagnetic determination of soil water content: Measurements in coaxial transmission lines[END_REF] using as input the soil's dielectric permittivity) and the reference crop evapotranspiration ETo (according to the FAO Penman-Monteith equation [START_REF] Allen | Crop evapotranspiration-Guidelines for computing crop water requirements-FAO Irrigation and drainage paper 56[END_REF]) were calculated on-the-fly and served through the webbased platform to the FRBS as inputs. Both soil moisture content and ETo were potential inputs and could be taken into account by the FRBS in order to infer the best time to irrigate as well as the amount of irrigation necessary.

In the two plots controlled by the FRBS two independent courses of actions were followed. In both cases, the fuzzy KB was preselected by an expert (i.e. an agronomist) using the linguistic description of the input variables. This was done to demonstrate the ability of fuzzy systems to incorporate the expert knowledge in a straightforward way, enabling people not familiar with machine learning models to effortlessly develop an AI-based model. In one of these two plots however, the rule base was allowed to evolve to automatically tune the components of the fuzzy KB. This was deemed important as the irrigation mode of operation represents an open-loop type of control, where a decision to irrigate is sent to the gateway and thence to the relevant slave device and valve, without immediate feedback -feedback is received once the gateway synchronizes its data with the IoT platform at its predetermined time interval. The FRBS can then use this input-output relationship from prior actions and improve its structure and components by tweaking the necessary parameters, in accordance to the delayed feedback it receives, so that it can improve in the future. Additionally, and because the system automatically redefines its components to address the needs, the rule base can accordingly be more robust to perturbations and changes (e.g. in ambient conditions). Hence, this smart AI-driven system can be more precise in the management of irrigation and thus ipso facto help conserve water and resources. This is demonstrated in Figure 5 for the month of January (which was dry and had the most irrigation needs). The crop needs are the calculated ETo minus the precipitation; the FRBS-auto is the closest to the theoretical values with an RMSE of 0.63 mm, compared to 2.10 mm for FRBS-expert and 6.47 mm for control, which was the most wasteful of all.

Conclusion

In this paper, we have presented Vital, a system that employs IoT devices in order to monitor sensor values of fields and automate irrigation. Vital integrates a set of IoT devices (sensors) with low installation and maintenance costs, as well as a robust distributed data management infrastructure. Furthermore, we design a fuzzy-rule based system that effectively takes decisions about the irrigation of the fields. Compared to other open-field agriculture systems, our system encompasses an eXplainable AI approach, thus ensuring maximum efficiency as well as increased interpretability, by enabling humans to inspect the model, which is highly desirable in such critical applications.

Concerning future work, one may consider further adapting our system by adding more types of sensors (e.g. for digital imaging) and/or even attaching other types of valves apart from irrigation (e.g. for pesticide control). Finally, concerning the smart system, we plan to evaluate it against scenarios with missing data and assess whether it can be used effectively in such cases, as well as perform comparisons against other approaches in this area.

Fig. 1 .

 1 Fig. 1. Overview of the Vital System.

Fig. 2 .

 2 Fig. 2. Overview of the Cenote IoT platform depicting its components.

Fig. 3 .

 3 Fig. 3. The architecture of a Mamdani FRBS.

Fig. 4 .

 4 Fig. 4.A strong fuzzy partition of the space X using triangular membership functions. Highlighted is the fuzzy set à = {L2 or L3} following the DNF approach.

Fig. 5 .

 5 Fig. 5. Comparison among the different irrigation techniques for January of 2020.

https://safeatlast.co/blog/iot-statistics/

https://www.statista.com/statistics/802690/worldwide-connected-devices-byaccess-technology/

https://www.un.org/development/desa/publications/world-population-prospects-2019-highlights.html

Acknowledgments

This research has been co-financed by the European Regional Development Fund of the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RE-SEARCH -CREATE -INNOVATE (project code: T1EDK-02296).


Eleni Adamopoulou

Lefteris Moussiades

An Overview of Chatbot Technology

Keywords: Chatbot, Chatbot Architecture, Artificial Intelligence, Machine learning, NLU

The use of chatbots evolved rapidly in numerous fields in recent years, including Marketing, Supporting Systems, Education, Health Care, Cultural Heritage, and Entertainment. In this paper, we first present a historical overview of the evolution of the international community's interest in chatbots. Next, we discuss the motivations that drive the use of chatbots, and we clarify chatbots' usefulness in a variety of areas. Moreover, we highlight the impact of social stereotypes on chatbots design. After clarifying necessary technological concepts, we move on to a chatbot classification based on various criteria, such as the area of knowledge they refer to, the need they serve and others. Furthermore, we present the general architecture of modern chatbots while also mentioning the main platforms for their creation. Our engagement with the subject so far, reassures us of the prospects of chatbots and encourages us to study them in greater extent and depth.

Introduction

Artificial Intelligence (ΑΙ) increasingly integrates our daily lives with the creation and analysis of intelligent software and hardware, called intelligent agents. Intelligent agents can do a variety of tasks ranging from labor work to sophisticated operations. A chatbot is a typical example of an AI system and one of the most elementary and widespread examples of intelligent Human-Computer Interaction (HCI) [START_REF] Bansal | A Review Paper on Human Computer Interaction[END_REF]. It is a computer program, which responds like a smart entity when conversed with through text or voice and understands one or more human languages by Natural Language Processing (NLP) [START_REF] Khanna | A Study of Today's A.I. through Chatbots and Rediscovery of Machine Intelligence[END_REF]. In the lexicon, a chatbot is defined as "A computer program designed to simulate conversation with human users, especially over the Internet" [START_REF]chatbot | Definition of chatbot in English by Lexico Dictionaries[END_REF]. Chatbots are also known as smart bots, interactive agents, digital assistants, or artificial conversation entities. Chatbots can mimic human conversation and entertain users but they are not built only for this. They are useful in applications such as education, information retrieval, business, and e-commerce [START_REF] Abu Shawar | Chatbots: are they really useful?[END_REF]. They became so popular because there are many advantages of chatbots for users and developers too. Most implementations are platform-independent and instantly available to users without needed installations. Con-tact to the chatbot is spread through a user's social graph without leaving the messaging app the chatbot lives in, which provides and guarantees the user's identity. Moreover, payment services are integrated into the messaging system and can be used safely and reliably and a notification system re-engages inactive users. Chatbots are integrated with group conversations or shared just like any other contact, while multiple conversations can be carried forward in parallel. Knowledge in the use of one chatbot is easily transferred to the usage of other chatbots, and there are limited data requirements. Communication reliability, fast and uncomplicated development iterations, lack of version fragmentation, and limited design efforts for the interface are some of the advantages for developers too [START_REF] Klopfenstein | The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms[END_REF].

The rest of the paper is organized as follows. In section 2, we briefly present the history of chatbots and highlight the growing interest of the research community. In section 3, some issues about the association with chatbots are discussed, while in section 4, essential concepts relevant to chatbot technology are described. Next, in section 5, we present a classification of existing chatbots while in section 6, we present the underlying chatbot architecture and the leading platforms for their development. Finally, section 7 reports conclusions and highlights further research topics.

History

Alan Turing in 1950 proposed the Turing Test ("Can machines think?"), and it was at that time that the idea of a chatbot was popularized [START_REF] Turing | [END_REF]. The first known chatbot was Eliza, developed in 1966, whose purpose was to act as a psychotherapist returning the user utterances in a question form [START_REF] Weizenbaum | ELIZA-a Computer Program for the Study of Natural Language Communication Between Man and Machine[END_REF]. It used simple pattern matching [START_REF] Brandtzaeg | Why People Use Chatbots[END_REF] and a template-based response mechanism. Its conversational ability was not good, but it was enough to confuse people at a time when they were not used to interacting with computers and give them the impetus to start developing other chatbots [START_REF] Klopfenstein | The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms[END_REF]. An improvement over ELIZA was a chatbot with a personality named PARRY developed in 1972 [START_REF] Colby | Artificial Paranoia[END_REF]. In 1995, the chatbot ALICE was developed which won the Loebner Prize, an annual Turing Test, in years 2000, 2001, and 2004. It was the first computer to gain the rank of the "most human computer" [START_REF] Wallace | The Anatomy of A[END_REF]. ALICE relies on a simple patternmatching algorithm with the underlying intelligence based on the Artificial Intelligence Markup Language (AIML) [START_REF] Marietto | Artificial Intelligence MArkup Language: A Brief Tutorial[END_REF], which makes it possible for developers to define the building blocks of the chatbot knowledge [START_REF] Wallace | The Anatomy of A[END_REF]. Chatbots, like SmarterChild [START_REF] Molnár | The Role of Chatbots in Formal Education[END_REF] As shown in Fig. 1 according to Scopus [18], there was a rapid growth of interest in chatbots especially after the year 2016. Many chatbots were developed for industrial solutions while there is a wide range of less famous chatbots relevant to research and their applications [START_REF] Colace | Chatbot for e-learning: A case of study[END_REF].

Associate with Chatbots

Why do users use chatbots? Chatbots seem to hold tremendous promise for providing users with quick and convenient support responding specifically to their questions. The most frequent motivation for chatbot users is considered to be productivity, while other motives are entertainment, social factors, and contact with novelty. However, to balance the motivations mentioned above, a chatbot should be built in a way that acts as a tool, a toy, and a friend at the same time [START_REF] Brandtzaeg | Why People Use Chatbots[END_REF].

The reduction in customer service costs and the ability to handle many users at a time are some of the reasons why chatbots have become so popular in business groups [START_REF] Ranoliya | Chatbot for university related FAQs[END_REF]. Chatbots are no longer seen as mere assistants, and their way of interacting brings them closer to users as friendly companions [START_REF] Costa | Conversing with Personal Digital Assistants: on Gender and Artificial Intelligence[END_REF]. According to a study, social media user requests on chatbots for customer service are emotional and informational, with the first category rate being more than 40% and with users not intending to take specific information [START_REF] Xu | A New Chatbot for Customer Service on Social Media[END_REF]. Machine learning is what gives the capability to customer service chatbots for sentiment detection and also the ability to relate to customers emotionally as human operators do [START_REF] Følstad | What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study[END_REF].

Concerning the user's trust in chatbots, it depends on factors relative to the chatbot itself, like how much it responds like a human, how it is self-presented, and how much professional its appearance is. Nevertheless, it depends also on factors relative to its service contexts, like the brand of the chatbot host, privacy and security in the chatbot, and other risk issues about the topic of the request [START_REF] Wallace | The Anatomy of A[END_REF]. Human-likeness can be suggested by using human figures (visual cues), human-associated names, or iden-tity (identity cues) and mimicking of human languages (conversational cues) [START_REF] Go | Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions[END_REF]. It has already been studied the influence of personification and interactivity in people's disclosures around sensitive topics, such as psychological stressors [START_REF] Sannon | How Personification and Interactivity Influence Stress-Related Disclosures to Conversational Agents[END_REF]. Important to mention is that chatbots still lack empathy understanding meaning and that they are not as capable as humans of understanding conversational undertones. Though progress has been made in this field, and soon machines will not only be able to understand what somebody is saying but also what is the feeling of what he is saying [START_REF] Fernandes | NLP, NLU, NLG and how Chatbots work[END_REF].

However, a biased view of gender is revealed, as most of the chatbots perform tasks that echo historically feminine roles and articulate these features with stereotypical behaviors. Accordingly, general or specialized chatbots automate work that is coded as female, given that they mainly operate in service or assistance related contexts, acting as personal assistants or secretaries [START_REF] Costa | Conversing with Personal Digital Assistants: on Gender and Artificial Intelligence[END_REF].

Soon we will live in a world where conversational partners will be humans or chatbots, and in many cases, we will not know and will not care what our conversational partner will be [START_REF] Dale | The return of the chatbots[END_REF].

Essential Concepts

Below are some fundamental concepts related to chatbot technology. Pattern Matching is predicated on representative stimulus-response blocks. A sentence (stimuli) is entered, and output (response) is created consistent with the user input [START_REF] Marietto | Artificial Intelligence MArkup Language: A Brief Tutorial[END_REF]. Eliza and ALICE were the first chatbots developed using pattern recognition algorithms. The disadvantage of this approach is that the responses are entirely predictable, repetitive, and lack the human touch. Also, there is no storage of past responses, which can lead to looping conversations [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF].

The Artificial Intelligence Markup Language (AIML) was created from 1995 to 2000, and it is based on the concepts of Pattern Recognition or Pattern Matching technique. It is applied to natural language modeling for the dialogue between humans and chatbots that follow the stimulus-response approach. It is an XML-based markup language and it is tag-based. As shown in Fig. 2, AIML is based on basic units of dialogue called categories (tag <category>) which are formed by user input patterns (tag <pattern>) and chatbot responses (tag <template>) [START_REF] Marietto | Artificial Intelligence MArkup Language: A Brief Tutorial[END_REF].

<aiml version="1.0.1" encoding="UTF-8"?> <category> <pattern> My name is * and I am * years old </pattern> <template> Hello <star/>. I am also <star index="2"/> years old!</template> </category> </aiml> Fig. 2

. Example of AIML code

Latent Semantic Analysis (LSA) may be used together with AIML for the development of chatbots. It is used to discover likenesses between words as vector representation [START_REF] Akma | Review of Chatbots Design Techniques[END_REF]. Template-based questions like greetings and general questions can be answered using AIML while other unanswered questions use LSA to give replies [START_REF]An e-business chatbot using AIML and LSA -Semantic Scholar[END_REF].

Chatscript, being the successor of the AIML language, is an expert system, which consists of an open-source scripting language and the engine that runs it. It is comprised of rules which are associated with topics, finding the best item that matches the user query string and executing a rule in that topic. Chatscript also includes long-term memory in the form of \$ variables which can be used to store specific user information like the name or age of the user. It is also case-sensitive, widening the possible responses that can be given to the same user input based on the intended emotion, as uppercase is typically used in conversations to indicate emphasis [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF].

RiveScript is a plain text, line-based scripting language for the development of chatbots and other conversational entities. It is open-source with available interfaces for Go, Java, JavaScript, Perl, and Python [31].

Natural Language Processing (NLP), an area of artificial intelligence, explores the manipulation of natural language text or speech by computers. Knowledge of the understanding and use of human language is gathered to develop techniques that will make computers understand and manipulate natural expressions to perform desired tasks [START_REF] Jung | Semantic vector learning for natural language understanding[END_REF]. Most NLP techniques are based on machine learning.

Natural Language Understanding (NLU) is at the core of any NLP task. It is a technique to implement natural user interfaces such as a chatbot. NLU aims to extract context and meanings from natural language user inputs, which may be unstructured and respond appropriately according to user intention [START_REF] Jung | Semantic vector learning for natural language understanding[END_REF]. It identifies user intent and extracts domain-specific entities. More specifically, an intent represents a mapping between what a user says and what action should be taken by the chatbot. Actions correspond to the steps the chatbot will take when specific intents are triggered by user inputs and may have parameters for specifying detailed information about it [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF]. Intent detection is typically formulated as sentence classification in which single or multiple intent labels are predicted for each sentence [START_REF] Jung | Semantic vector learning for natural language understanding[END_REF].

An entity is a tool for extracting parameter values from natural language inputs. For example, consider the sentence "What is the weather in Greece?". The user intent is to learn the weather forecast. The entity value is Greece. Therefore, the user asks for the weather forecast in Greece [START_REF] Canonico | A Comparison and Critique of Natural Language Understanding Tools[END_REF]. Entities can be either system-defined or developer-defined. For example, the system entity @sys.date corresponds to standard date references like 10 August 2019 or the 10th of August [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF]. Domain entity extraction usually referred to as a slot-filling problem, is formulated as a sequential tagging problem where parts of a sentence are extracted and tagged with domain entities [START_REF] Jung | Semantic vector learning for natural language understanding[END_REF].

Finally, contexts are strings that store the context of the object the user is referring to or talking about. For example, a user might refer to a previously defined object in his following sentence. A user may input "Switch on the fan." Here the context to be saved is the fan so that when a user says, "Switch it off" as the next input, the intent "switch off" may be invoked on the context "fan" [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF].

Types of Chatbots

Chatbots can be classified using different parameters: the knowledge domain, the service provided, the goals, the input processing and response generation method, the human-aid, and the build method.

Classification based on the knowledge domain considers the knowledge a chatbot can access or the amount of data it is trained upon. Open domain chatbots can talk about general topics and respond appropriately, while closed domain chatbots are focused on a particular knowledge domain and might fail to respond to other questions [START_REF] Nimavat | Chatbots: An Overview Types, Architecture, Tools and Future Possibilities[END_REF].

Classification based on the service provided considers the sentimental proximity of the chatbot to the user, the amount of intimate interaction that takes place, and it is also dependent upon the task the chatbot is performing. Interpersonal chatbots lie in the domain of communication and provide services such as Restaurant booking, Flight booking, and FAQ bots. They are not companions of the user, but they get information and pass them on to the user. They can have a personality, can be friendly, and will probably remember information about the user, but they are not obliged or expected to do so. Intrapersonal chatbots exist within the personal domain of the user, such as chat apps like Messenger, Slack, and WhatsApp. They are companions to the user and understand the user like a human does. Inter-agent chatbots become omnipresent while all chatbots will require some inter-chatbot communication possibilities. The need for protocols for inter-chatbot communication has already emerged. Alexa-Cortana integration is an example of inter-agent communication [START_REF] Nimavat | Chatbots: An Overview Types, Architecture, Tools and Future Possibilities[END_REF].

Classification based on the goals considers the primary goal chatbots aim to achieve. Informative chatbots are designed to provide the user with information that is stored beforehand or is available from a fixed source, like FAQ chatbots. Chatbased/Conversational chatbots talk to the user, like another human being, and their goal is to respond correctly to the sentence they have been given. Task-based chatbots perform a specific task such as booking a flight or helping somebody. These chatbots are intelligent in the context of asking for information and understanding the user's input. Restaurant booking bots and FAQ chatbots are examples of Task-based chatbots [START_REF] Nimavat | Chatbots: An Overview Types, Architecture, Tools and Future Possibilities[END_REF] [START_REF] Kucherbaev | Human-Aided Bots[END_REF].

Classification based on the input processing and response generation method takes into account the method of processing inputs and generating responses. There are three models used to produce the appropriate responses: rule-based model, retrieval-based model, and generative model [START_REF] Hien | Intelligent Assistants in Higher-Education Environments: The FIT-EBot, a Chatbot for Administrative and Learning Support[END_REF].

Rule-based model chatbots are the type of architecture which most of the first chatbots have been built with, like numerous online chatbots. They choose the system response based on a fixed predefined set of rules, based on recognizing the lexical form of the input text without creating any new text answers. The knowledge used in the chatbot is humanly hand-coded and is organized and presented with conversational patterns [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF]. A more comprehensive rule database allows the chatbot to reply to more types of user input. However, this type of model is not robust to spelling and grammatical mistakes in user input. Most existing research on rule-based chatbots studies response selection for single-turn conversation, which only considers the last input message. In more human-like chatbots, multi-turn response selection takes into consideration previous parts of the conversation to select a response relevant to the whole conversation context [START_REF] Wu | Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots[END_REF].

A little different from the rule-based model is the retrieval-based model, which offers more flexibility as it queries and analyzes available resources using APIs [START_REF] Hien | Intelligent Assistants in Higher-Education Environments: The FIT-EBot, a Chatbot for Administrative and Learning Support[END_REF].

A retrieval-based chatbot retrieves some response candidates from an index before it applies the matching approach to the response selection [START_REF] Wu | Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots[END_REF].

The generative model generates answers in a better way than the other three models, based on current and previous user messages. These chatbots are more humanlike and use machine learning algorithms and deep learning techniques. However, there are difficulties in building and training them [START_REF] Hien | Intelligent Assistants in Higher-Education Environments: The FIT-EBot, a Chatbot for Administrative and Learning Support[END_REF].

Another classification for chatbots considers the amount of human-aid in their components. Human-aided chatbots utilize human computation in at least one element from the chatbot. Crowd workers, freelancers, or full-time employees can embody their intelligence in the chatbot logic to fill the gaps caused by limitations of fully automated chatbots. While human computation, compared to rule-based algorithms and machine learning, provides more flexibility and robustness, still, it cannot process a given piece of information as fast as a machine, which makes it hard to scale to more user requests [START_REF] Kucherbaev | Human-Aided Bots[END_REF].

Chatbots can also be classified according to the permissions provided by their development platform. Development platforms can be of open-source, such as RASA, or can be of proprietary code such as development platforms typically offered by large companies such as Google or IBM. Open-source platforms provide the chatbot designer with the ability to intervene in most aspects of implementation. Closed platforms, typically act as black boxes, which may be a significant disadvantage depending on the project requirements. However, access to state-of-the-art technologies may be considered more immediate for large companies. Moreover, one may assume that chatbots developed based on large companies' platforms may be benefited by a large amount of data that these companies collect.

Of course, chatbots do not exclusively belong to one category or another, but these categories exist in each chatbot in varying proportions.

Design and Development

The design and development of a chatbot involve a variety of techniques [START_REF] Akma | Review of Chatbots Design Techniques[END_REF]. Understanding what the chatbot will offer and what category falls into helps developers pick the algorithms or platforms and tools to build it. At the same time, it also helps the end-users understand what to expect [START_REF] Nimavat | Chatbots: An Overview Types, Architecture, Tools and Future Possibilities[END_REF].

The requirements for designing a chatbot include accurate knowledge representation, an answer generation strategy, and a set of predefined neutral answers to reply when user utterance is not understood [53]. The first step in designing any system is to divide it into constituent parts according to a standard so that a modular development approach can be followed [START_REF] Ramesh | A Survey of Design Techniques for Conversational Agents[END_REF]. In Fig. 3, a general chatbot architecture is introduced. The process starts with a user's request, for example, "What is the meaning of environment?", to the chatbot using a messenger app like Facebook, Slack, WhatsApp, WeChat or Skype, or an app using text or speech input like Amazon Echo [START_REF] Zumstein | Chatbots -An Interactive Technology for Personalized Communication, Transactions and Services[END_REF].

After the chatbot receives the user request, the Language Understanding Component parses it to infer the user's intention and the associated information (intent: "translate," entities: [word: "environment"]) [START_REF] Kucherbaev | Human-Aided Bots[END_REF].

Once a chatbot reaches the best interpretation it can, it must determine how to proceed [START_REF] Fern | es: The Best Open Source Chatbot Platforms in 2019[END_REF]. It can act upon the new information directly, remember whatever it has understood and wait to see what happens next, require more context information or ask for clarification. When the request is understood, action execution and information retrieval take place. The chatbot performs the requested actions or retrieves the data of interest from its data sources, which may be a database, known as the Knowledge Base of the chatbot, or external resources that are accessed through an API call [START_REF] Kucherbaev | Human-Aided Bots[END_REF].

Upon retrieval, the Response Generation Component uses Natural Language Generation (NLG) [START_REF] Singh | Open source NLG systems: A survey with a vision to design a true NLG system[END_REF] to prepare a natural language human-like response to the user based on the intent and context information returned from the user message analysis component. The appropriate responses are produced by one of the three models mentioned in section 5 of the paper: rule-based, retrieval based, and generative model [START_REF] Hien | Intelligent Assistants in Higher-Education Environments: The FIT-EBot, a Chatbot for Administrative and Learning Support[END_REF].

A Dialogue Management Component keeps and updates the context of a conversation which is the current intent, identified entities, or missing entities required to fulfill user requests. Moreover, it requests missing information, processes clarifications by users, and asks follow-up questions. For example, the chatbot may respond: "Would you like to tell me as well an example sentence with the word environment?" [START_REF] Kucherbaev | Human-Aided Bots[END_REF].

Many commercial and open-source options are available for the development of a chatbot. The number of chatbot-related technologies is already overwhelming and growing each day [START_REF] Nayyar | Chatbots and the Open Source Tools You Can Use to Develop Them[END_REF]. Chatbots are developed in two ways: using any programming language like Java, Clojure, Python, C++, PHP, Ruby, and Lisp or using state-of-theart platforms. At this time, we are distinguishing six leading NLU cloud platforms that developers can use to create applications able to understand natural languages: Google's DialogFlow [START_REF] Dialogflow | [END_REF], Facebook's wit.ai [START_REF] Wit | [END_REF], Microsoft LUIS [START_REF] Luis | Language Understanding) -Cognitive Services -Microsoft Azure[END_REF], IBM Watson Conversation [17], Amazon Lex [START_REF]Amazon Lex -Build Conversation Bots[END_REF], and SAP Conversation AI [47]. All these platforms are supported by machine learning. They share some standard functionality (they are cloud-based, they support various programming and natural languages) but differ significantly in other aspects [START_REF] Canonico | A Comparison and Critique of Natural Language Understanding Tools[END_REF]. Other known chatbot development platforms are RASA [48], Botsify [49], Chatfuel [START_REF] Chatfuel | [END_REF], Manychat [START_REF]ManyChat -Chat Marketing Made Easy[END_REF], Flow XO [START_REF]AI Online Chatbot Software, Live Chat on Websites[END_REF], Chatterbot [53], Pandorabots [START_REF]Pandorabots: Home[END_REF], Botkit [START_REF]Botkit: Building Blocks for Building Bots[END_REF], and Botlytics [56].

Conclusions

Minimal human interference in the use of devices is the goal of our world of technology. Chatbots can reach out to a broad audience on messaging apps and be more effective than humans are. At the same time, they may develop into a capable information-gathering tool. They provide significant savings in the operation of customer service departments. With further development of AI and machine learning, somebody may not be capable of understanding whether he talks to a chatbot or a real-life agent. We consider that this research provides useful information about the basic principles of chatbots. Users and developers can have a more precise understanding of chatbots and get the ability to use and create them appropriately for the purpose they aim to operate.

Further work of this research would be exploring in detail existing chatbot platforms and compare them. It would also be interesting to examine the degree of ingenuity and functionality of current chatbots. Some ethical issues relative to chatbots would be worth studying like abuse and deception, as people, on some occasions, believe they talk to real humans while they are talking to chatbots.

Fig. 1 .

 1 Fig. 1. Search results in Scopus by year for "chatbot" or "conversation agent" or "conversational interface" as keywords from 2000 to 2019.

Fig. 3 .

 3 Fig. 3. General Chatbot architecture

Acknowledgments. This work is partially supported by the MPhil program "Advanced Technologies in Informatics and Computers", hosted by the Department of Computer Science, International Hellenic University.


Seemab Latif
email: seemab.latif@seecs.edu.pk

Sarmad Bashir

Mir Muntasar

Ali Agha

Rabia Latif
email: rlatif@psu.edu.sa

Backward-Forward Sequence Generative Network for Multiple Lexical Constraints

Keywords: Recurrent neural networks, natural language generation, language models, lexical constraints, word embedding

Advancements in Long Short Term Memory (LSTM) Networks have shown remarkable success in various Natural Language Generation (NLG) tasks. However, generating sequence from pre-specified lexical constraints is a new, challenging and less researched area in NLG. Lexical constraints take the form of words in the language model's output to create fluent and meaningful sequences. Furthermore, most of the previous approaches cater this problem by allowing the inclusion of prespecified lexical constraints during the decoding process, which increases the decoding complexity exponentially or linearly with the number of constraints. Moreover, some of the previous approaches can only deal with single constraint. Additionally, most of the previous approaches only deal with single constraints. In this paper, we propose a novel neural probabilistic architecture based on backward-forward language model and word embedding substitution method that can cater multiple lexical constraints for generating quality sequences. Experiments shows that our proposed architecture outperforms previous methods in terms of intrinsic evaluation.

Introduction

Recently, Recurrent Neural Networks (RNNs) and their variants such as Long Short Term Memory Networks (LSTMs) and Gated Recurrent Units (GRUs) based language models have shown promising results in generating high quality text sequences, especially when the input and output are of variable length. RNN based Language Models (LM) have the ability to capture the sequential nature of language, be it for words, characters or whole sentences. This allows them to outperform other language models in sequence prediction and classification tasks. To learn the distributed representation of data efficiently by RNNs, multiple methods have been proposed such as word embeddings. It mainly include Continuous Bag-Of-Words (CBOW) and Skip-Gram (SG) model [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF].

CBOW model predicts the word as vector at a current time step, given preceding and proceeding context word vectors. The SG model is opposite in approach to predict the representation of target word vector, but same in the architecture.

Existing methods to incorporate constraints in the output sentences or generating lexical constrained sentences have multiple limitations. [START_REF] Mou | Backward and forward language modeling for constrained sentence generation[END_REF] proposed variants of backward-forward generation approach which can not handle Out-of-Vocabulary (OOV) words and only generate sentences with single lexical constraint. Similarly, [START_REF] Liu | Bfgan: Backward and forward generative adversarial networks for lexically constrained sentence generation[END_REF] proposed a synchronous training approach to generate lexical constrained sequences with Generative Adversarial Networks (GANs). Moreover, various lexical constrained decoding methods have been proposed for constrained sequence generation through the extension of beam search to allow the inclusion of constraints [START_REF] Anderson | Guided open vocabulary image captioning with constrained beam search[END_REF][START_REF] Hokamp | Lexically constrained decoding for sequence generation using grid beam search[END_REF]. Such lexical constrained decoding methods do not examine what specific words need to be included at the start of generation, but try to force specific words at each time step during the generation process at a cost of high computational complexity [START_REF] Post | Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[END_REF].

The remainder of this paper is organized as follows. We review the related work in Section 2. Section 3 describes our proposed architecture and section 4 explains the dataset, experimental setup, comparison models and evaluation criteria. Section 5 gives in detail result analysis, finding and discussions about future directions. Finally, Section 6 concludes the paper.

Literature Review

In general, the purpose of LM is to capture the regularities of a language as well as its morphological and distributional properties. LM aims to compute the probability of a word sequence in order to estimate the maximum likelihood of an upcoming word to be predicted in the sequence. LM learns the distributed representation of words to interpret semantic and syntactic relations between the sequence of words. In past, RNN has shown progressive success in language modeling over traditional methods based on statistical counts. The ability of RNN Language Model (RNNLM) to learn long term contextual dependency and capturing inherited sequential nature of language makes it better than other traditional methods [START_REF] Mikolov | Recurrent neural network based language model[END_REF]. Particularly in sentence generation task, RNNLM performed well because of its capability of learning highly complicated structures of language. RNNLM makes Maximum A Posteriori (MAP) estimation for predicting words in a sentence [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF].

Mou et at. first proposed multiple variants of Back-ward and Forward (B/F) language models based on GRUs for constrained sentence generation [START_REF] Mou | Backward and forward language modeling for constrained sentence generation[END_REF]. For training the B/F language models, sentences were split by choosing a word randomly. This resulted in the positional information of words getting smoothed out while generating sentences, and thus they lose the positional information of the word. This method of choosing a split word badly influences the joint probability estimation of a sentence.

Liu et al. proposed an algorithmic framework dubbed as Backward and Forward Generative Adversarial Networks (BFGAN) for constrained sentence gen-eration [START_REF] Liu | Bfgan: Backward and forward generative adversarial networks for lexically constrained sentence generation[END_REF]. BFGAN constitutes three modules; a discriminator, LSTM based backward and a forward generator with attention mechanism. The purpose of discriminator is to distinguish the real sentences from constrained sentences generated by machine and to guide the joint training of both backward and forward generators by assigning them reward signals. The backward generator takes lexical constraint as an input, which can be a word, phrase or fragment and generate the first half of the sentence backwards. The Forward generator takes the input half sentence generated by backward generator to complete the sentence with the aim of fooling the discriminator. The sentences prepared for training of backward generator relies on random splitting of sentences and the proposed framework can tackle single lexical constrained sentence generation.

Another line of work tackles the problem of constrained sentence generation by sampling the sentences from search space. Su et al. proposed a Gibbs sampling method based on Markov Chain Monte Carlo (MCMC) method for decoding constrained sentences [START_REF] Su | Incorporating discriminator in sentence generation: a gibbs sampling method[END_REF]. The proposed approach consists of a discriminator and a pure language model conditioned on a bi-directional RNN. Introducing discriminator in the proposed method caters the job for calculating probability of a sentence satisfying the constraints. Gibbs method samples the set of random variables x 1 ...n from a joint distribution, which takes the form of words to make a sentence. The shortcoming of Gibbs sampling is that it cannot change the length of sentences and hence not able to solve complicated tasks like directly generating sentences from constraints established in advance. Miao et al. extends Gibbs sampling by introducing Metropolis-Hastings for Constrained Sentence Generation (CGMH) [START_REF] Miao | Cgmh: Constrained sentence generation by metropolis-hastings sampling[END_REF]. The proposed method directly samples from the sentence space by defining local operations in the sentence space such as word replacement, insertion and deletion.

Hokamp et al. proposed Grid Beam Search (GBS) algorithm, an extension of beam search, for incorporating specified lexical constraints in the output sequences [START_REF] Hokamp | Lexically constrained decoding for sequence generation using grid beam search[END_REF]. In Neural Machine Translation (NMT) task, the proposed algorithm ensures that all specified constraints must meet the hypothesis before they can be considered to be completed. To generalize image caption generative models for out-of-domain images constituting novel scenes or objects, Anderson et al. proposed a Constrained Beam Search (CBS) decoding method, which utilizes Finite-State Machine (FSM) [START_REF] Anderson | Guided open vocabulary image captioning with constrained beam search[END_REF]. The proposed search algorithm is capable of forcing certain image tags over resulting output sequences by recognizing valid sequences with FSM.

Table 1 summarizes techniques for generating constrained sequences. It is evident that many of the architectures are designed for specific scenarios and have high computational complexity. Due to performance gaps and inability to handle multiple constraints efficiently, a method need to be addressed. Therefore, we have proposed a neural probabilistic Backward-Forward architecture that can generate high quality sequences, with word embedding substitution method to satisfy multiple constraints.

General Model

To begin with, we state the problem of constrained sequence generation as follows: given the constraint(s) c as input, the proposed B/F LM needs to generate a fluent sequence s = w 1 , • • •, w v , • • •, w m maximizing the conditional probability p(s|c). For this purpose, we need to select a split word in a sequence s to train the proposed B/F LM. As a sequence provides us an expression, the Parts-Of-Speech (POS) verb plays a vital role in placing the subject of a sequence into motion and offers more clarification about sequence. In this section, we first discuss the general seq2seq model for generation of sequences. After that, we discuss our proposed architecture to deal with constrained sequence generation. Conventionally, RNNLMs for text generation are trained to maximize the likelihood of a word w t or character c t at time step t while given the context of previous observations in the sequence. This type of learning technique for generating sequences is known as teacher forcing [START_REF] Dai | A comparison of mlp, rnn and esn in determining harmonic contributions from nonlinear loads[END_REF]. In such learning technique, input to the recurrent neural probabilistic language model is of fixed size. The training purpose is to predict only next token until a special stop sign is generated or specific constraint is satisfied in a sequence given the context of previous observations.

In traditional seq2seq models we cannot satisfy lexical constraints, where disintegrating joint probability of a sentence y = y 1 , y 2 • • •y m for given input sentence "we all are drinking apple juice" Thus, the output sentence y is predicted from y 1 to y m in sequence either by a greedy or beam decoder. Such decomposition is because of natural language's sequential nature.

x = x 1 , x 2 • • •x n is given by p(y|x) = m i=1 p(y i |y 1 • • •y i-1 , x) (1)

Proposed Architecture

Our proposed approach consists of a neural probabilistic architecture that is an ensemble of two LSTM based B/F LM for generating lexical constrained sequences, which captures the statistical properties of text sequences effectively. In order to generate the coherent sequences from given multiple constraints as input, we first generate the sequence from verb constraint w v through B/F LM, and then we satisfy the other given constraints by word embedding substitution method during the inference process. The predicted verb v splits the sequence into two sub-sequences as:

Backward Sequence = w v-1 , w v-2 , • • •, w 1 Forward Sequence = w v+1 , w v+2 , • • •, w m If m denotes the length of words in a sequence s i.e. s = w 1 , • • •, w v , • • •, w m ,
then the joint conditional probability of remaining m words, given lexical constraint w v and training parameters θ can be calculated as:

p(s) = p(s|w v ; θ) = p bw θ (s <v |w v ) . p f w θ (s >v |s 1 : w v ) (2)
Where p bw θ and p fw θ depict the probabilities of generated sub-sequences by backward and forward language models. The sub-sequences are generated asynchronously i.e. we first generate the half sequence s <v in reverse order given verb constraint w v , then generate the other half sequence s >v conditioned on backward sequence s 1 : w v . Therefore, following the spirit of ensemble models that are trained separately, joint probability factors in equation 2 becomes

p bw θ (s <v |w v ) = wv-1 j=1 p bw θ (w v-j |w v , • • •, w v-j+1 ) (3)
Where 1 ≤ j ≤ v -1. Backward LM decodes the output in reverse order from w v-1 , w v-2 to w 1 , which is reversed again to input forward language model for decoding the complete sequence. Consequently,

p f w θ (s >v |s 1:v ) = m-wv j=1 p f w θ (w v+j |w 1 , • • •, w v+j-1 ) (4)
Here 1 ≤ j ≤ m -v. As the output order of sub-sequence generated by backward LM is reversed again to decode the entire sequence from forward language model, therefore

s 1:v is equal to w 1 , • • •, w v .
For learning the sequences, we used LSTM networks in proposed architecture. The LSTM networks have the capability of capturing sequential data effectively where the network transforms a sequence of given input word vectors x = x 1 , • • •, x n into the sequence of hidden states h = h 1 , • • •, h t by maintaining a history of inputs at each hidden state. The LSTM cell depends on gating mechanism for information processing. LSTM network's hidden state h at time step t is dependent on the previous state h t-1 and current input x t word vectors. Particularly, in our scenario for generating variable length text sequences, the probability of an output word w out from both language models calculated as:

p bw θ (w v-t |h t ) = Sof tmax(w bw out h t + b out ) (5)
p f w θ (w v+t |h t ) = Sof tmax(w f w out h t + b out ) (6)
Where w bw out and w f w out are shared across all time steps in their respective LSTM models, which projects the hidden state vector h t into a fixed same size vector as target vocabulary in order to generate a sequence of outputs y t = w v-t , • • •, w 1 for backward language model and y t = w v+t , • • •, w m for forward language model. The softmax function is in the final layer of LSTM network, applied to each word vector for calculating the probability distribution over vocabulary of distinct word vectors.

Word Embedding Substitution

In order to satisfy the given lexical constraints c other than verb constraint w v , we have used a lexical substitution method based on word embedding substitution. SG model embeds both target words and their context in the same dimensional space. In this space, the vector representations of words are drawn closer together when they co-occur more frequently in a learning corpus. Thus, Cosine distance between them can be viewed as target-to-target distributional similarity measure. Our method relies on a natural assumption that a good lexical constraint substitution for a target word w instance in a generated sequence s = w 1 , • • •, w v , • • •, w m needs to be consistent with the given sequence and lexically similar to the target word w instance. During inference, we find the cosine similarity [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] of given input constraint c with every word w in a sequence s generated by the proposed B/F LM. After that, we replace the constraint c with the closest matching (least cosine distance) word w in a sequence s. Step 3 of Figure 1 illustrates the concept. For this purpose, we have created word embedding vectorization from FastText.

Experiments

In this section, we introduced our experimental designs, containing the preparation of dataset for training and testing, experimental configuration, comparison architectures and evaluation criteria.

Dataset

There are many benchmark datasets for evaluating pure LM consisting of seq2seq networks for text classification and generative models, but specifically there is no such benchmark corpus for evaluation of constrained sequence generation based on statistical language models. As far, we have used Stanford Natural Language Inference (SNLI) [START_REF] Bowman | A large annotated corpus for learning natural language inference[END_REF] dataset for evaluation and training of proposed architecture. As we target the domain of generating sequences from lexical constraints, we extracted unlabeled sequences within range of minimum 3 and maximum 25 tokens, resulting in 451k sequences for training of proposed architecture. The proposed architecture ensemble backward-forward LM, therefore, to prepare training sequences for backward LM, following steps have been carried out:

-Annotate the tokens with their lexical categories using POS Tagging.

-Split the sentences on verb category instead of random splitting.

-Sentences with more than one verb are broken up into multiple sequences.

-After splitting the sequence on verb category, invert the half sequences.

For the forward language model, the dataset contains complete sequences for training the network. Here, it should be noted that backward language model requires only half sequences till verb token for training the network.

Word Vectorization

We follow the work of bojanowski et. al [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] to create dense representations of words in dataset. A word vector is represented by augmenting the character n-grams appearing in the word, where the scoring function s takes into consideration the internal structure information of words, which is ignored by conventional skip-gram models [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]. The proposed model represents each word w as a bag of character n-gram, where adding special boundary symbols < and > at the beginning and end of words for distinguishing prefixes and suffixes from other character sequences. In addition to character n-grams of word w, the word w is also included in its set of n-grams for learning representation of each word. For example, taking the word 'apple and let n = 3, it will be represented by the character n-grams as < app, ppl, ple > and the special sequence < apple >.

Let a dictionary of n-grams with size G. Given a word w where L w ⊂ 1, ...G is the set of n-grams appearing in word w. Vector z g represents the each n-gram g, therefore a word w is represented by the sum of vectors of its n-gram g. In this regard, scoring function of word w with surrounding set of word indices c is calculated by:

s(w, c) = g∈Lw z T g v c (7)
This extension of skip-gram model for creating word embedding allow the sharing of word vector representations across all words, thus enabling the reliable representational learning of rare or Out-Of-Vocabulary (OOV) words.

We have used extension of FastText's SG model to learn such data representations for both backward and forward language model given their respective data sets. In order to train the FastText model, the word embedding dimension set to 300. Min count value set to 2, which represents that all the word frequencies lower than 2 were ignored while learning the word representations. Window size set to 5, defining the maximum distance between a current and predicted word within a sequence. Workers parameter set to 16, explaining the worker threads for faster training of FastText SG model. Epochs value set to 30 iteration, over the whole data set.

Experimental Configuration

We performed different experiments on test set to get the most optimal hyperparameters and evaluate change in performance of the model. Table 2 shows the different experimental configurations and change in performance w.r.t perplexity metric. In the proposed architecture, we get the best results by employing 2-layers of LSTM in both backward and forward language model. Both the LSTM networks were trained with Adam algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] for stochastic optimization of networks. During training, the parameters were adjusted using Adam optimizer for minimizing the training loss function, also known as misclassification rate. For calculating optimization score, we used categorical cross entropy loss function between the actual y and predicted ŷ word probability distribution [START_REF] De Boer | A tutorial on the crossentropy method[END_REF]. In target of accurately capturing the regularities by the neural networks and preventing overfitting, we appended drop-out layer after every LSTM layer in both the networks. The idea of drop-out layer is to randomly drop units with their connections while training, thus preventing units from co-adapting too much. Dropping units significantly leads to major improvements than other regularization methods [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF].s The epochs value was set to 50 and mini batch size was set to 128 in both the networks. Both the Backward and Forward models are trained on NVIDIA GTX 1080 Ti GPU. The LSTM based networks are developed in keras. Training took about 17 hours approx. per model with this implementation and optimal hyperparameter configuration.

Comparison and Evaluation Metrics

We compared our proposed methodology with state-of-the art sampling method CGMH [START_REF] Miao | Cgmh: Constrained sentence generation by metropolis-hastings sampling[END_REF] for satisfying multiple constraints in a sequence. We also evaluated our methodology of verb based split generation with different variants [START_REF] Mou | Backward and forward language modeling for constrained sentence generation[END_REF], which can only handle single lexical constraint. We have used intrinsic evaluation metric that allows to determine the quality of a LM without being associated or embedded to a particular application. The most conventional intrinsic evaluation metric is perplexity (PPL). PPL of a language model given a test set w = w 1 , w 2 , ...w m is the inverse probability of w where the probability is normalized by the number of words

P P L(w) = m m i=1 1 P (w i , • • •, w i-1 ) (8)

Results and Discussions

For intrinsic evaluation of our proposed methodology, we first make comparisons with variants such as separate B/F and asynchronous B/F language models proposed by [START_REF] Mou | Backward and forward language modeling for constrained sentence generation[END_REF]. As mentioned earlier, in our proposed methodology the given word is verb constraint w v through which we decode complete sequence whereas in variants of B/F, the complete sequence is decoded by random split word. We calculated PPL with both verb and random constraint as input to decode the complete sequences. Table 2 represents the comparison in terms of PPL, where the higher probability of a sequence results in the lower of perplexity, which is better. Separate B/F variant yields worse sequences with huge perplexity score because both the B/F LM were enforced to output separately with the input constraint and concatenated after decoding of sequences. This is due to the fact that forward LM does not have the context of half sequence decoded by backward LM. Our proposed approach is more similar to asynchronous B/F LM, but technically very different as we are satisfying multiple constraints while asynchronous approach can deal with only single constraint. The results clearly shows that decoding a sequence on specific verb constraint can make use of the positional information of words in a sequence, that is smoothed out when we generate a sequence with random constraint. Table 3 shows the comparison of our proposed approach for catering multiple constraints with CGMH [START_REF] Miao | Cgmh: Constrained sentence generation by metropolis-hastings sampling[END_REF]. Our proposed approach shows lower perplexity than CGMH sampling method for sentence generation through keywords/constraints 1 to 3, while with 4 constraints as input CGMH shows slightly better result than our approach of generating sequence with verb constraint and during inference replacing the words in sequence with closest embedding similarity. The decoding complexity of CGMH increases linearly with the number of constraints, while there is no such factor in our approach for catering multiple constraints. There is always a trade-off between fluency of sequence and decoding complexity. In practice, the downside of CGMH sampling methods is that we are not sure of which sampling step size is best for proposal distribution.

Discussion

To validate our proposed architecture of generating sequence, we performed a series of experiments. Results of intrinsic evaluation confirms that our proposed approach for sequence generation given constraint(s) outperforms previous methods. Splitting and generating a sequence on verb constraint makes use of positional information, which is smoothed out in breaking down a sequence with random word. We observe that decoding a sequence given random word as input in proposed B/F LM even performs better when the backward LM is trained over half sequences till verb. Moreover, in future we would like to explore about the constraint-to-target context similarity, indicating their syntagmatic compatibility for improving the word embedding substitution method. Introducing attention mechanism as context vectors for constraints would be an interesting side in the proposed architecture.

Conclusion

In this paper, we have proposed a novel method, dubbed Neural Probabilistic Backward-Forward language model and word embedding substitution method to address the issue of lexical constrained sequence generation. Our proposed system can generate constrained sequences given multiple lexical constraints as input. To the best of our knowledge, this is the first time that multiple constraints have been handled through LSTM based backward-forward LM and word embedding substitution of the sequences. The proposed method contains a backward language model based on LSTM network, which learns the half representation of a sentence until the verb splitting word and forward language model constitute LSTM Network, learning the complete representation of a sequence. Moreover, word embedding substitution method satisfy other constraints by substituting the target word in the sequence with given constraints based on similar context in an embedding space.

Fig. 1 :

 1 Fig. 1: An illustration of proposed System Architecture

Table 1 :

 1 Comparison of different Constrained Sequence Generation techniques.

		Multiple Computational Decoding Decoder Target
		Constraints	Time	Complexity		Domain
	Mou et al. [13]	x	low	O(N k)	-	Research
						Titles
	Anderson et al. [1]		high	O(N k2 C )	CBS Image
						Captioning
	Su et al. [16]		high	O(N + dN M ) GSM Product
						Sentiments
	Liu et al. [8]	x	-	-	Beam Product
					Search Reviews
	Hokamp et al. [6]		high	O(N k2C)	GBS NMT
	Post et al.[14]		low	O(N k)	DBA NMT
	Miao et al. [9]		high	O(N + dN M ) MH Generic
	Proposed Technique		low	O(N k)	Greedy Generic
					Search

Table 2 :

 2 Hyper-parameter tuning and model performance

	LSTM layers Hidden Units LR drop-out PPL Score
	1	256	0.01	0.2	35.48
	1	512	0.001	0.3	33.15
	2	256	0.01	0.2	27.48
	2	512	0.001 0.3	24.20

Table 2 :

 2 Intrinsic Evaluation

Table 3

 3

	: CGMH vs Proposed