Sihem Amer-Yahia

From Large Language Models to Databases and Back A discussion on research and education

LLMS AND DATABASES

In recent years, large language models (LLMs) have garnered increasing attention from both academia and industry due to their potential to facilitate natural language processing (NLP) and generate high-quality text. Despite their benefits, however, the use of LLMs is raising concerns about the reliability of knowledge extraction. The combination of database research and data science has advanced the state of the art in solving real-world problems, such as merchandise recommendation and hazard prevention. In this discussion, we explore the challenges and opportunities related to LLMs in database and data science research and education.

LLMs for database research. LLMs have proven to be highly useful for language writing, as they can identify and correct grammar errors. Additionally, LLMs can serve as a valuable resource for knowledge acquisition and analysis. However, the accuracy of extracted knowledge is not guaranteed, and bias can be introduced, leading to potential inaccuracies in the data analysis process.

LLMs can be highly effective in data preparation and labeling tasks, such as text mining, text parsing, keyword extraction, and sentiment analysis. It also has great potential to improve feature extraction, selection, and parameter tuning.

Database research for LLMs. Database research can support LLM development from data cleaning and preprocessing to training and optimization. For instance, domain-specific knowledge can be incorporated into training data to create more accurate and reliable models. Furthermore, database research can be used to optimize prompt engineering to improve the effectiveness of LLM.

LLMs for education. LLMs can be used to reform database education and address the challenges and concerns related to their use. LLMs can provide students with a wealth of knowledge and practical skills, such as techniques for handling dirty data. However, care must be taken to ensure that the information and programming styles provided by LLMs are accurate and free from bias or misinformation. As such, it is crucial to consider how to detect plagiarism when people use LLMs to generate scientific articles, papers, or assignments. We conclude that, although there are challenges associated with the use of LLMs in database research and education, these can be addressed through careful research and thoughtful integration of LLMs into the data science curriculum.

ChatGPT, today's famous LLM. A Generative Pre-Trained Transformer (GPT) is a language model relying on deep learning that is designed to take a text-based input and generate a natural human-like text. Chat Generative Pre-trained Transformer (Chat-GPT) is a chatbot released by OpenAI in November 2022 and is built on top of OpenAI's GPT-3.5 large language model (LLM). ChatGTP has been trained on a large body of text from a variety of sources in 2020 and can write any form of text such as essays, poems, paragraphs and computer programs. It is able to understand and generate human-like natural language with a high level of accuracy and fluency. A new version based on GPT-4 was released on March 2023 and is available for paid subscribers on a limited basis.

While ChatGPT can help people with a lot of well-suited tasks such translation of foreign languages, summarization of a text and generation of human-like conversational responses, it has several drawbacks. Since large language models perform the task of predicting the next word in a series of words by replicating common patterns of trained texts, it produces the output text without any concern of originality, plagiarism and privacy. Furthermore, since the training text data is derived from publicly available data before 2021, it cannot provide accurate information in a timely manner and may not know the most up-to-date information. Moreover, ChatGPT does not seem to be yet able to perform well complicated mathematical calculations or high-level problem-solving tasks. Since its outputs may contain false or outdated information, we should carefully evaluate the outputs of ChatGPT and use them cautiously.

PROS AND CONS OF LLMS FOR RESEARCH AND EDUCATION

LLMs for Data Science Research. Calculators and word processors are useful tools for people that allow not to worry about complex arithmetic calculations and incorrect spellings/grammars as well as citation labels of their writing, respectively. The positive implications of using both tools are that people can focus and concentrate more on the content of their work without worrying about inaccurate calculations or spelling/grammatical errors. Similarly, data scientists can utilize ChatGPT for their data science research to focus more on high-level creative thinking including getting the big picture, original idea generation, analytical thinking and problem solving. For example, they can utilize it to summarize the texts about related works, learn about a particular research topic, brainstorm about research directions for their new project and improve their writing skills of technical papers. Chat-GPT can also assist non-native data scientists in understanding, interpreting and writing English texts. On the other hand, they can use ChatGPT to produce a high-quality code with explanations and improve their coding skills. It can even help a data scientist rewrite his old code in a programming language to an equivalent code in another programming language. Note that an important skill required for data scientists is the ability to produce a good quality of code. Thus, instead of spending time in learning how to code or producing code for data analysis, data scientists can concentrating more on their research by utilizing ChatGPT.

While more training data is likely to produce a more accurate model [START_REF] Banko | Scaling to very very large corpora for natural language disambiguation[END_REF], there are many applications such as named entity recognition [START_REF] Wang | Meta self-training for few-shot neural sequence labeling[END_REF], relation extraction [START_REF] Jung | Dual supervision framework for relation extraction with distant supervision and human annotation[END_REF] and image classification [START_REF] Zhang | Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling[END_REF] where producing a large-scale training data by manual labeling is expensive and time-consuming. To quickly obtain a large-scale training data with low cost, one approach is to use weak supervision that automatically annotates unlabeled data by heuristic rules or machine learning models. For example, one of the most popular techniques for weak supervision is distant supervision that utilizes external knowledge bases to produce weak labels [START_REF] Liang | BOND: bert-assisted open-domain named entity recognition with distant supervision[END_REF]. We can also utilize ChatGPT as an alternative method for weak supervision. For instance, ChatGPT was recently investigated to augment training data for few-shot classification by rephrasing each sentence in the training data into multiple similar sentences [START_REF] Dai | Auggpt: Leveraging chatgpt for text data augmentation[END_REF].

Data Science Research for LLMs. Since an LLM model is only as knowledgeable as the training texts that have been provided for learning, its knowledge is limited according to the training data and it may become unfair by absorbing the biases from the training data. Furthermore, it lacks the capability of ethical thinking too. Thus, developing learning techniques to overcome such handicaps of LLM models will be very helpful to LLMs. Publicly available text data on the Web has a lot of sensitive information and training LLMs with public data can thus disclose sensitive and private information of people. On the other hand, as users input more data with conversations into ChatGPT, it may potentially leak the sensitive information to other users of ChatGPT. Thus, developing the privacy preserving schemes, such as the differential privacy, with high utility for training LLM modles will help LLMs to protect the privacy of individuals.

LLMs for Computer Science and Data Science Education. ChatGPT can be a useful tool for both disciplines and we need to reform the curricular to include ChatGPT. In [START_REF] Hazzan | Chatgpt in computer science education[END_REF], opportunities of utilizing ChatGPT are addressed and several ChatGPT-based tasks are suggested for computer science education. For instance, teachers can ask students to generate a code for a given problem, and then explain, analyze and improve the code. In addition, we can also ask students to write their own code for the same problem and find the similarities as well as differences between two codes.

Students can utilize ChatGPT to summarize/understand/learn the texts about existing works for a particular research topic, enhance their coding as well as debugging skills to generate a high quality code, brainstorm about research topics and improve their writing skills of technical papers. Thus, students can utilize Chat-GPT to focus more on high-level creative thinking including getting the big picture, original idea generation, analytical thinking and the detailed steps of their methods to solve a given problem. To do so, since the outputs of ChatGPT may contain false or outdated information, students should learn how to use ChatGPT effectively and cautiously.

While there are many advantages of including ChatGPT in the curriculum, students who consistently depend on ChatGPT may lose or cannot improve their skills of summarizing the texts, searching for relevant materials about a particular topic and writing technical papers by themselves. Furthermore, while ChatGPT is proficient in generating fluent text, it may produce the contents with lack of clarity as well as originality. Moreover, the outputs of ChatGPT may contain false or outdated information, and may even present a plagiarized writing from another source without citing properly. Thus, we need to provide precise guidelines of using ChatGPT to students so that they can learn how to use ChatGPT effectively and utilize the ChatGPT outputs with caution.

Is ChatGPT charming? Have you ever watched the movie "Cyrano"? It is a story about a man who sent love letters to a woman that were actually written by another man with a good skill of writing romantic love letters to a woman. When you start chatting with someone for the first time on an online dating site or dating app, if you feel too attracted to the person, watch out -you may actually be talking with ChatGPT!

WHAT CAN AND CANNOT LLM DO FOR DATABASES?

As we all know, LLMs typically report probabilistic results but cannot be used to report fully deterministic results [START_REF] Ghahramani | Probabilistic machine learning and artificial intelligence[END_REF][START_REF] Patrick | Probabilistic machine learning: An introduction[END_REF]. Therefore, LLM can be leveraged to handle inexact data/query processing problems that can tolerate approximate results, e.g., approximate query processing and data integration. However, it is hard to use LLMs to support exact data/query processing components, e.g., query answering and query rewriting. In the following, we discuss how to use LLMs to support exact and inexact data/query processing.

LLMs for Database Research. For most of database problems, e.g., data discovery, data cleansing, and data integration, the users are satisfied with approximate results. The optimization goal is to improve the generalizability, efficiency and quality. Intuitively, we can utilize LLMs to improve the generalizability. However, several challenges arise. The first challenge is automatic prompt engineering that automatically generates appropriate prompts to guide LLMs to find correct answers. LLMs have limitations on the number of token constraints and long latency, and the automatic prompt engineering tool should optimize these two factors. The second challenge is how to integrate domain knowledge into LLMs. Current LLMs use open Web corpus to pretrain a large language model and thus can well support data cleaning and integration on Web data but cannot effectively support vertical domains that are absent on open Web. Hence, the challenge is to fine-tune LLMs to support domain knowledge or use prompt engineering to teach LLMs to do this. The third challenge is how to combine LLMs and existing data science tools, as it is expensive to call LLMs and it is beneficial to utilize some existing tools to reduce the cost. For example, there are many good database tools, e.g., blocking tools and entity-matching tools, and we can design tool learning that enable LLMs to call effective tools to reduce the cost.

Database Research for LLMs. The theory and model architecture of LLMs are almost mature, and the researchers and scientists that are working on LLMs focus on providing high-quality data to train LLMs, e.g., discovering data, cleaning data and integrating data. There exists a plethora of tools for the above database tasks that can be used to prepare the data on which LLMs are trained. A challenge is how to make a win-win loop between database systems and LLMs, which uses database techniques to provide high-quality of LLMs and uses LLMs to optimize the database tools.

Querying data with natural language. An interesting application is Text-to-SQL, which converts natural language queries into SQL statements. This has been a long-studied research problem. The state-of-the-art is currently a work presented at AAAI 2023 [START_REF] Li | RESDSQL: Decoupling schema linking and skeleton parsing for text-to-SQL[END_REF], which achieved an accuracy of 79.9% using a seq2seq pre-trained language model. With continuous prompts, ChatGPT enables users to interactively refine the generated SQL queries and could further improve their accuracy. This unique feature presents a potential opportunity to integrate LLMs with existing Text-to-SQL techniques to generate more precise SQL statements.

Additionally, ChatGPT allows users to query a dataset with natural language. This could eliminate the need for SQL queries and increase the efficiency of data retrieval and analysis for certain applications, which poses the question of whether SQL remains necessary or if it is possible to translate text into a query evaluation plan for database result evaluation. The integration of LLMs with database techniques has the potential to open up new opportunities for research and advancement in the field of data science.

LLMs for Logical Query Optimization. Logical query optimization (e.g., query rewriting) aims to translate a query plan to an optimized query plan (possibly with a lower cost but without guarantee). It seems that LLMs can be used to support this problem. But the key challenge is that the translated query plan should be exactly equivalent to the original plan. Therefore, there are several possible solutions. The first uses LLMs to obtain an optimized query plan and then verifies the equivalence using existing techniques (and then keeps the equivalent query and drops the in-equivalent one). The second uses LLMs to optimize the using of query rewriting rules, including discovering new query-rewrite rules and judiciously using the rules (including whether to use a rule and to determine the order of using different rules).

LLMs for Physical Query Optimization. Different from logical query optimization, physical query optimization should utilize the physical database statistics and it is vital to provide these information to LLMs. However, the current LLMs cannot effectively support numerical values. Two challenges arise in this context. The first is to fine-tune the LLMs that enables LLMs to support numerical statistics. The second is to embed database statistics into the prompt to facilitate that LLMs can use such information to get an optimized physical plan. There are also some other similar problems that should utilize physical statistics, e.g., knob tuning, index/view advisor, and query diagnosis.

Database Tuning for LLMs. Database knob tuning [START_REF] Duan | Tuning database configuration parameters with ituned[END_REF] is important to achieve high performance of database systems. Traditionally, database administrators (DBAs) tune database systems. Since the number of knobs in database systems increases as database systems become more sophisticated, it is difficult for DBAs to tune the database systems by considering all possible values of knobs [START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF].

To overcome the drawbacks, auto-tuning methods were proposed to find an optimal configuration of database systems without human intervention [START_REF] Duan | Tuning database configuration parameters with ituned[END_REF][START_REF] Van Aken | Automatic database management system tuning through large-scale machine learning[END_REF]. Database knob tuning techniques developed for database systems can be useful to find an optimal configuration of an LLM for applications.

LLMs for Database Storage and Transactions. Both database storage and transactions are deterministic and we are pessimistic that LLMs cannot be used to optimize them.

LLMS (BADLY) NEED INTEGRATED DATA AND REASONING

LLMs and data integration To understand the differences between LLMs and databases, let us compare them with the process of integrating heterogeneous data sources. Data integration is a longstanding research problem in data management [START_REF]Schema Matching and Mapping[END_REF][START_REF] Stonebraker | Data integration: The current status and the way forward[END_REF]. Schema mapping, data deduplication, schema and data fusion are all tasks that involve considering up-to-date data sources, as well as personal and proprietary data. By leveraging the inherent semantics of mappings (correspondences between queries or views on different data sources), these tasks do not need re-training on large corpuses of data and can easily capture new incoming data. As recently argued in a vision paper, these data of different nature are not considered so far in the LLM [START_REF] Alon | Learnings from data integration for augmented language models[END_REF]. Moreover, integrated data can easily cater for privacy constraints and become trustworthy, for instance by blending mappings with policy views [START_REF] Bonifati | Exchanging data under policy views[END_REF] or by having humans as first-class citizens in the data integration process [START_REF] Arioua | User-guided repairing of inconsistent knowledge bases[END_REF][START_REF] Bonifati | Interactive mapping specification with exemplar tuples[END_REF][START_REF] Chiticariu | Debugging schema mappings with routes[END_REF]. Provenance and lineage information, in particular why, how and where provenance characterizing query results [START_REF] Cheney | Provenance in databases: Why, how, and where. Found[END_REF], could serve the need of filling an existing gap of large language models, missing the key capability of locating the sources of information.

But what is missing in LLMs to take advantage of databases and integrated data sources? The following is a non-exhaustive list of missing features (and the reader is invited to add more):

-Need to retain provenance and schema information as well as other kinds of metadata, which is not merely raw data and should not be unified with raw data; -Need to process and compute provenance throughout the learning process and be able to annotate the results with provenance information (and, thus, citation sources); -Need to capture privacy constraints and privacy policies in the data acquisition and data fusion process.

LLMs and Graphs. Graphs are a great source of knowledge, which is typically curated by humans and, as such, can be seen as high-quality and trustworthy integrated information. Examples of such highly curated graphs are Wikidata and DBPedia, that are typically used by search engines [START_REF] Weikum | Knowledge graphs 2021: A data odyssey[END_REF], whereas LLM are trained on large text corpora, such as Wikipedia, books, news and open datasets. Knowledge graphs such as DBPedia and Wikidata can be navigated and queried by leveraging query endpoints. Queries collected at the endpoints allow to understand what the users search within the knowledge graphs and thus indirectly to characterize the underlying structure of the data.

To illustrate the difference between semantics in graph databases and language models, we choose to confront a query from the DBPedia graph query logs with question answering in ChatGPT. Figure 1: The Henry VIII query, a 7-clique containing one constant and six variables. All edges between Henry VIII and the variables are labeled "dbpedia-owl:spouse" and all edges between variables are labeled with the property path "!dbpedia-owl:sameAs" [START_REF] Bonifati | An analytical study of large SPARQL query logs[END_REF].

Figure 1 shows the shape of a 7-node clique query involving Henry The VIII to his 6 wifes where all edges between Henry The VII and his wifes are labeled as "dbpedia-owl:spouse" and the edges between each pair of wifes are labeled with a property path 1 , namely "!dbpedia-owl:same-As" (to retrieve his distinct wifes).

Figure 2 shows question answering on ChatGPT on the possible relationships (in terms of RDF properties) between Henry The VIII and his 6 wives. It shows the RDF syntax of the "spouse" relationship but it does substantiate the result with the information about the source from which this information has been retrieved. ChatGPT cannot access databases such as DBPedia or Wikidata at the time being and, as text-based AI model is unable to draw diagrams on query shapes such as the one in Figure 1.

LLMs and Reasoning. Contrarily to LLMs, graphs enable symbolic reasoning, when logic-based existential rules and path queries are used to augment existing knowledge graphs with additional inferred data [START_REF] Carral | Chasing sets: How to use existential rules for expressive reasoning[END_REF]. Moreover, database queries including graph queries return certain answers [START_REF] Bonifati | Querying Graphs[END_REF], whereas output of LLM is highly uncertain depending on the frequency of values in the training data.

Combining results from LLMs with graphs could improve the results of LLMs by unifying machine reasoning with symbolic and logic-based reasoning [START_REF] Sherif Sakr | The future is big graphs: a community view on graph processing systems[END_REF]. The knowledge graph lifecycle with maintenance operations, update propagation and error fixing is nonexistent for LLMs. On the other hand, the outputs of LLM could be used to enrich knowledge graphs for question answering tasks by leveraging graph attention networks [START_REF] Yasunaga | QA-GNN: reasoning with language models and knowledge graphs for question answering[END_REF].

Concluding, we cannot disagree with Gary Markus' blog 'Hoping for the Best as AI Evolves' [START_REF] Marcus | Hoping for the best as ai evolves[END_REF]: "Large language models lack mechanisms for verifying truth; we need to find new ways to integrate them with the tools of classical AI, such as databases, Webs of knowledge, and reasoning. ". 1 A property path is a path with a regular expression allowing to navigate graph data.

LLMS AS A RESEARCH ASSISTANT

Helping with Scientific Writing. LLMs like ChatGPT can help with scientific writing in several ways, such as proofreading, rewriting, summarization, and even suggesting titles for research papers. It can also help improve the language to better communicate research ideas and results, so as to facilitate research collaborations. However, the key challenge lies in creating effective prompts that generate high-quality responses. Below are some examples:

• Revise the following paragraph from the introduction of a Computer Science academic paper so the citations are kept and the text has a clear sentence structure. • Here is the abstract of a paper. Suggest five creative titles.

• Write a 1-page sensational press release for this research.

With well-crafted prompts, ChatGPT can deliver results that rival those of paid editing services.

Assisting with Data Analytics. LLMs can assist with data creation. It can generate data based on input parameters, which can be useful in situations where large amounts of synthetic data are needed for research. Additionally, LLMs can help generate code for data analytics tasks [START_REF] Datacamp | A guide to using ChatGPT for data science projects[END_REF]. One such task is exploratory data analysis (EDA). For example, suppose we have a loan dataset that we want to perform EDA on. We can simply prompt ChatGPT with a request to write Python code to load and perform EDA on the loan dataset, and it will provide us with a code that we can use to analyze and visualize the data. ChatGPT can also assist with other data analytics tasks such as data cleaning and preprocessing, feature engineering, hyperparameter tuning, and model selection and evaluation. It can help save time and effort by automating some of these tedious tasks and focus on more complex aspects of data analytics.

Limitation of Hallucination in LLMs. Despite their many benefits, LLMs do have some limitations that need to be considered. One of these limitations is their potential to generate incorrect Figure 3: Overview of AIED research, focusing on the examination of underlying data and applications [START_REF] Khosravi | Data management of ai-powered education technologies: Challenges and opportunities[END_REF]. content that appears plausible, known as "hallucination". This is particularly relevant in research paper writing, where ChatGPT may suggest non-existing references or provide inaccurate information. For example, when asked to recommend a paper on the topic of data science authored by Lei Chen, ChatGPT suggested a paper titled "Crowdsourced Data Management: A Survey" authored by Lei Chen, Reynold Cheng, Silviu Maniu, and Wang-Chien Lee and published in IEEE Transactions on Knowledge and Data Engineering, vol. 25, no. 9, pp. 1959-1977, September 2013. However, although this paper title does exist, it was not authored by Lei, rather by Guoliang et al. and published in 2016 (see [START_REF] Li | Crowdsourced data management: A survey[END_REF]). In fact, Lei, Sihem, and Anand did author a survey paper on a related topic but their title is different (see [START_REF] Inasu | A survey of general-purpose crowdsourcing techniques[END_REF]). To address this issue, one possible solution is to integrate ChatGPT with a knowledge graph and ground truth facts, which can help verify the accuracy of the generated information by crossreferencing it with existing data.

Incorporating External Data into LLMs. While LLMs have access to a vast amount of data, their knowledge is still limited by the data they have been trained on. There are some recent efforts to incorporate external data into LLMs through the use of prompts. However, LLMs may have a limitation on the length of input they can process, which can impact their ability to understand complex or lengthy inputs. For example, the GPT-4 base version allows up to 8,192 tokens, which may not be sufficient for processing longer texts. One approach to overcome this limitation, known as chunking, is to split the longer text into smaller segments and process each segment separately [START_REF] Langchain | LangChain Chat[END_REF]. It can be combined with traditional data science techniques such as embedding and indexing to improve the model's performance on longer inputs. Yet, chunking can also introduce challenges such as maintaining coherence and consistency between segments, which may require further research.

Ethical and legal issues. Ethical concerns such as bias, plagiarism, and data privacy and security are also significant issues when using LLMs. LLMs may be biased towards the data they were trained on, which can lead to unfair or inaccurate results. Additionally, there is a risk of privacy breaches if the input contains sensitive information. Moreover, legal and copyright issues must also be considered when using LLMs. If the model generates copyrighted material without proper licensing, it could lead to legal repercussions. Thus, it is essential to responsibly and ethically use LLMs by thoroughly vetting and verifying generated content before using or publishing it. The new ACM policy requires disclosure of the use of generative AI tools for content generation in published work, with specific details regarding their usage provided in the acknowledgments section or elsewhere in the work [1].

LLMS AND EDUCATION

One area that is receiving both scientific and media coverage these days is the impact of LLMs on education. Several concerns have been raised about students using ChatGPT to complete tests, ChatGPT passing bar exams and professors using it to devise quizzes. As scientists, we focus on discussing LLMs in teaching and LLMs for doing research on education, that we refer to as LLM4ED.

Teaching LLMs. First of all, we need to teach LLMs just like other models. This will contribute to demystifying them and to raising awareness about their lack of transparency as well as their benefits and pitfalls. We also need to encourage our students to treat LLMs just like other recommendation engines. Their prediction accuracy should be tested keeping in mind that the best paper award at RecSys in 2019 showed that KNN outperformed 6 Deep Learning recommendation methods on MovieLens data [START_REF] Ferrari Dacrema | Are we really making much progress? A worrying analysis of recent neural recommendation approaches[END_REF]. These includes Collaborative Variational Autoencoder and and Neural Collaborative Filtering methods. Students need to learn to build on top of LLMs and treat them just like other models. In recent work, we built a meta-recommender that learns the best algorithm to apply given a (user,dataset) or a (user,question) pair [START_REF] Bouarour | How useful is metarecommendation? an empirical investigation[END_REF]. This approach could integrate LLMs as a recommendation option.

LLM4ED. There are many challenges and opportunities of database research in education [START_REF] Amer-Yahia | Towards ai-powered data-driven education[END_REF][START_REF] Khosravi | Data management of ai-powered education technologies: Challenges and opportunities[END_REF]. An LLM could be modeled as a learner or to support learners, teachers or administrators.

LLMs as Learners. This would require to model learners' data and behavior. Figure 3 represents data about learners and learning artefacts that can be found in most education systems. This data is highly diverse. Student and curriculum records capture individual learners' records such as their demographics which are usually provided by learners at registration time as well as information on learning material such as artefacts, and assessment and outcome requirements. Learning records capture data on learners' achievements such as grades and assessment outcomes. Learning logs record learners' engagement with artefacts, feedback to learners and collaboration. This would encourage students to see LLMs as a peer from which to learn and to criticize.

LLMs for Learners, Teachers and Administrators. This opens new opportunities for LLM-in-the-loop research in education. Figure 3 summarizes some research directions. For instance, in the context of collaborative learning, team formation algorithms could be revisited to consider LLMs as team members. In the context of developing an intelligent teaching assistant, LLMs are already in use for grading students which raises the question of accuracy and fairness. They are also used help teachers create content. In both grading and content creation, ensuring teachers' agency will allow them to guide the process and override automatic decisions. LLMs can also be used for admission support and student dropout prediction analytics. A particular point of attention in all these applications is the study of bias in ranking (student ranking) and in classification (student dropout prediction). Adding provenance to LLMs to better identify their sources of flaws would also address some of these concerns.

Figure 2 :

 2 Figure 2: Snapshot of question answering on Henry The VIII's wives on ChatGPT.


Luyin Hu
email: lu-yin.hu@polyu.edu.hk

David Navarro-Alarcon

Andrea Cherubini
email: andrea.cherubini@lirmm.fr

Mengying Li
email: mengying.li@polyu.edu.hk

Lu Li

On Radiation-Based Thermal Servoing: New Models, Controls and Experiments

Keywords: Thermoception, visual servoing, sensor-based control, robotic manipulation, adaptive control. I

In this paper, we introduce a new sensor-based control method that regulates (by means of robot motion) the temperature of objects that are subject to a radiative heat source. This valuable sensorimotor capability is needed in many industrial, dermatology and field robot applications, and it is an essential component for creating machines with advanced thermo-motor intelligence. To this end, we derive a geometricthermal-motor model which describes the relation between the robot's active configuration and the produced dynamic thermal response. We then use the model to guide the design of two new thermal servoing controllers (one model-based and one adaptive), and analyze their stability with Lyapunov theory. To validate our method, we report a detailed experimental study with a robotic manipulator conducting autonomous thermal servoing tasks. We show that the temperature of multiple objects with unknown thermophysical properties attached to the same end-effector can be effectively regulated by controlled robot motion. Although thermal sensing is a mature technology in many industrial thermal engineering applications, its use as a feedback signal for robot control has not been sufficiently studied in the literature. To the best of our knowledge, this is the first time that temperature regulation is formulated as a motion control problem for robots.

tion of a geometric-thermal-motor (GTM) model 1 describing the relation between the robot's motion and the consequent thermal response, and (b) the development of a sensor-based strategy (that relies on the thermal interaction matrix) 2 to autonomously impose a desired heat profile onto the surface of interest. Note that unlike other perception modalities for robot control (e.g. vision [START_REF] Navarro-Alarcon | Automatic 3D manipulation of soft objects by robotic arms with adaptive deformation model[END_REF], proximity [START_REF] Cherubini | Visual navigation of a mobile robot with laser-based collision avoidance[END_REF], touch [START_REF] Li | A review of tactile information: Perception and action through touch[END_REF], audition [START_REF] Magassouba | Audio-based robot control from interchannel level difference and absolute sound energy[END_REF] and even smell [START_REF] Rahbar | A 3-d bio-inspired odor source localization and its validation in realistic environmental conditions[END_REF]), thermoception has not been fully formalized in the literature as a bona fide feedback signal for motion control. In the robotics community, we still lack the framework to fully exploit it. Up to now, the overwhelming use of thermoception in robotics has been to monitor processes (e.g., image-based visual servoing with thermal cameras [START_REF] Dena | Image based visual servoing for landmine detection using quadrotors[END_REF]), but not to establish explicit thermal servo-loops [START_REF] Gade | Thermal cameras and applications: A survey[END_REF], which are needed to accurately control temperature. Our aim in this paper is to develop the necessary framework that enables the design of thermal servoing controls with radiative heat sources.

Although thermal sensing is a mature technology and has a rich history in the automation of many tasks (see e.g. [START_REF] Benli | Thermal multisensor fusion for collaborative robotics[END_REF]- [START_REF] Lai | Validation of size estimation of debonds in external wall's composite finishes via passive infrared thermography and a gradient algorithm[END_REF]), its use as a feedback signal for robot control has not been sufficiently studied in the literature [START_REF] Cherubini | Sensor-based control for humanrobot collaboration: Fundamentals, challenges and opportunities[END_REF], where only a few works have addressed this challenging servo-control problem. Some representative works that deal with explicit thermal control include: [START_REF] Binrui | Temperature control of PEM fuel cell stack application on robot using fuzzy incremental PID[END_REF], where a fuzzy controller is developed to regulate the temperature of a fuel cell actuator; [START_REF] Ho | Towards a MRI-compatible meso-scale smaactuated robot using pwm control[END_REF], where the influence of temperature in the deformation behavior of a surgical robot is investigated, and an explicit thermal regulator is designed; [START_REF] Backer | Temperature control of robotic friction stir welding using the thermoelectric effect[END_REF], where a control method is designed to maintain a constant tool temperature by adjusting the spindle speed in a stir friction welding robot. However, in these types of methods, temperature control is achieved by directly modulating the power of the heat-generating components. This approach is not suitable when considering external heat sources, e.g. wildfires [START_REF] Kim | Real-time probabilistic classification of fire and smoke using thermal imagery for intelligent firefighting robot[END_REF] and sunlight [START_REF] Cammarata | Optimized design of a large-workspace 2-dof parallel robot for solar tracking systems[END_REF], or when the source's power should not be varied, e.g. in cosmetic procedures [START_REF] Muddassir | Robotics meets cosmetic dermatology: Development of a novel visionguided system for skin photo-rejuvenation[END_REF].

A different strategy is to use sensor-based control, i.e. to dynamically change the source-object geometric configuration to achieve a desired thermal response (similar to what many organisms do [START_REF] Mckemy | Temperature sensing across species[END_REF]). This can be easily done by rigid robots, since their basic function is motion control. Such approach demands the development of appropriate models that can effectively capture the system's GTM relations. This idea has been partially demonstrated in [START_REF] Mlynek | Optimization of a heat radiation intensity and temperature field on the mould surface[END_REF], where the optimal fixed location of multiple radiating heaters in a process is automatically calculated to evenly imprint a desired thermal profile onto a surface. Yet, the heater is static and the method requires exact knowledge of all thermodynamic parameters (which are generally unknown). The proposed approach has also the potential to be used e.g. in fire-fighting [START_REF] Imdoukh | Semi-autonomous indoor firefighting uav[END_REF] or volcano exploration robots [START_REF] Muscato | Volcanic environments: Robots for exploration and measurement[END_REF] to calculate optimal trajectories that avoid overheating or damaging the robot's components.

The dynamic coupling between temperature and motion may seem unintuitive for humans [START_REF] Fu | Review on modeling heat transfer and thermoregulatory responses in human body[END_REF], while many organisms extensively exploit these relations [START_REF] Tsai | Physical and behavioral adaptations to prevent overheating of the living wings of butterflies[END_REF]. However, these advanced thermoception-based capabilities have not yet been fully incorporated in robot control, a discipline with good track record of borrowing inspiration from nature [START_REF] Arechavaleta | An optimality principle governing human walking[END_REF], but which seems to be lagging in this direction.

As a feasible solution to the above-mentioned issues, in this paper, we present a rigorous formulation for robot thermal servoing with radiative sources. The main contributions are:

• We develop an efficient algorithm for computing in realtime the radiation-based thermal interaction matrix which relates robot velocity and object temperature rate. • We present a novel robot control method for automatically regulating the temperature of grasped objects. • We report experiments to validate the proposed theory. To the best of the authors' knowledge, this is the first time that temperature regulation has been formulated in the literature as a robot servoing problem. The proposed approach could advance the development of multimodal robot controllers.

The rest of the paper is as follows: Section II presents the mathematical models; Section III derives the controller; Section IV reports results; Section V gives final conclusions.

II. MATHEMATICAL MODELING A. Heat Transfer Model

Throughout this manuscript, we denote all column vectors by small bold letters, e.g. v ∈ R n×1 , and matrices by capital bold letters, e.g. M ∈ R m×n .

In the following sections, we introduce basic thermodynamic concepts (we refer the reader to [START_REF] Bergman | Fundamentals of heat and mass transfer[END_REF], [START_REF] Modest | Radiative heat transfer[END_REF]) that are needed for developing the system's GTM model. To this end, consider a robot manipulator with end-effector pose denoted by x ∈ R n . The robot rigidly grasps (through an adiabatic layer) a planar "small-enough" object (such that its temperature can be fairly approximated by a single sensing point), whose surface temperature is to be controlled by changing the relative pose to a heat source. The heat transfer model is composed of three main parts (depicted in Fig. 2): (i) heat source, (ii) heat collector (i.e. the object), and (iii) surrounding environment. Thermophysical parameters of different parts are denoted by the same symbol but with different subscripts.

We denote the (constant) temperature of the heat source and the (varying) temperature of the object by T 1 and T 2 , respectively. We assume both temperatures to be spatially uniform during the heat transfer process 3 . The environment temperature (assumed to be constant) is denoted by T 3 .

Remark 1. In this paper, we use the subscripts i = 1, 2, 3 to denote the thermophysical parameters of the heat source, the object and the environment, respectively (a convention followed by many works dealing with heat transfer).

Heat transfer occurs amongst the three parts whenever T 1 , T 2 , T 3 have different values. The direction of heat transfer is always from a high temperature part to a low temperature part. We denote the net energy transfer rate to the object by Q 2 , where a positive value indicates energy inflow. We introduce q 2 = Q 2 /A 2 to represent the surface's net heat flux and v = dT 2 / dt to describe the temporal change of the measured temperature T 2 . According to the energy conservation laws, these quantities satisfy the relation:

v = Q 2 /(m 2 c 2 ) (1)
where m 2 denotes object's mass and c 2 denotes the material's specific heat. To synthesize a thermal servoing controller, it is useful to find an expression of the following form:

v = f (x, T 2 ) (2)
which describes the thermal-geometric relation between the robot configuration and the temperature rate.

B. Radiation Exchange Between Planar Surfaces

In this subsection, we show how to calculate Q 2 between planar surfaces when thermophysical properties are known. According to different mechanisms involved in the heat transfer processes [START_REF] Lienhard | A heat transfer textbook[END_REF], the object's net heat flux q 2 satisfies the expression q 2 = q rad + q conv + q cond , for radiative q rad , convective q conv and conductive q cond fluxes. In our case of study, thermal radiation is the dominant heat transfer mode; Note that q cond and q conv are negligible since the object is grasped through an adiabatic layer and the source's temperature is much higher than those of the object and environment.

Assumptions 1. We assume that the following conditions are satisfied during the task (see Fig. 2):

1) All surfaces have uniform thermophysical properties.

2) All surfaces are gray, i.e. they are diffuse emitters with equal emittance and absorptance.

3) The environment/room is modeled as a black body (i.e. ε 3 = α 3 = 1, see the variables' definition below).

The net energy transfer rate Q 2 has the following form:

Q 2 = A 2 q rad = A 2 (α 2 G 2 -E 2 ) (3)
where A 2 denotes the object's surface area, G 2 the radiative flux incident at the surface, α 2 ∈ [0, 1] the object's absorptance 3 This assumption simplifies transient heat conduction problems with the lumped capacitance method. It is generally valid for objects with high thermal conductivity, small characteristic length, and subject to moderate heat inflow/outflow. A quantitative valid condition of the assumption is the Biot number criterion, which is introduced in [27, Chapter 5]. For cases where the assumption is not valid, the object could be divided into different regions according to valid conditions, and multiple sensors could be used to obtain temperature feedback

Heat Source T1 Surrounding T3 Object T2 E 2 G 2 ρ 2 G 2 q conv Adiabatic q rad q cond J 1 J 3
Layer Fig. 2. Representation of the heat transfer model. A part of the object surface is magnified to show the various heat transfer processes.

and E 2 the heat flux emitted by a surface (i.e. emissive power) which is approximated using the Stefan-Bolzmann law [27]

E 2 = ε 2 σT 4 2 (4)
with ε 2 ∈ [0, 1] the material's emittance, and σ is the Stefan-Bolzmann constant. Note that for an opaque surface (i.e. with zero transmittance), its reflectance ρ i and absorptance α i satisfy ρ i +α i = 1. Radiosity is defined as J i = E i + ρ i G i , and since for our heat source E 1 ≫ ρ 1 G 1 , we can fairly approximate it as

J 1 ≈ E 1 .
Remark 2. The view factor F ij represents the fraction of J i that is incident on surface j. The view factor depends on the end-effector configuration, i.e. F ij = F ij (x). Thus, its calculation is essential for deriving the geometric-thermalmotor model. The detailed derivation of F ij for different cases and configurations is presented in later sections. Here, we assume F ij in known and only focus on the derivation of Q 2 .

The radiation incident to a surface is the summation of the corresponding portion of radiation coming from other surfaces. Thus, G i can be calculated from the expression:

A 2 G 2 = 3 j=1 F j2 A j J j = F 12 A 1 J 1 +F 22 A 2 J 2 +F 32 A 3 J 3 (5)
where by using the reciprocity relation A i F ij = A j F ji , the summation rule N j=1 F ij = 1, and the planar surfaces' property F ii = 0 (see [START_REF] Bergman | Fundamentals of heat and mass transfer[END_REF]), we can simplify (5) into:

A 2 G 2 = F 21 A 2 J 1 + F 23 A 2 J 3 = F 21 A 2 E 1 + (1 -F 21 )A 2 E 3 (6)
Substitution of (6) into (3) yields:

Q 2 = A 2 α 2 (E 1 -E 3 )F 21 + A 2 α 2 E 3 -A 2 E 2 (7)
which we substitute alongside (4) into (1) to obtain the following key expression for the object's temperature rate:

v = λ 1 F 21 -λ 2 T 4 2 + λ 3 (8)
for constant scalar parameters λ 1 , λ 2 , and λ 3 satisfying

λ 1 = A 2 α 2 σ(ε 1 T 4 1 -T 4 3 ) m 2 c 2 , λ 2 = A 2 ε 2 σ m 2 c 2 , λ 3 = A 2 α 2 σT 4 3 m 2 c 2 (9) i 1 j 1 k 1 i 2 k 2 j 2 θ x θ z θ y c 1 c 2 ω 1 s 1 s 2 s n 2 n 1 θ 1 θ 2

Heat Source

Object

A 1 d A2 d
r Fig. 3. Geometry of the view factor between two elementary surfaces.

C. View Factor Analytical Definition

In this section, we provide the general expression of F 21 , which we will instantiate (in the following sections) for various configurations. To this end, consider the elementary areas dA 1 and dA 2 on the source and object surfaces, respectively. These areas are separated by a length r that forms polar angles θ 1 and θ 2 (see Fig. 3). The definition of the view factor is:

F 21 = 1 A 2 A2 A1 cos θ 2 cos θ 1 πr 2 dA 2 dA 1 (10)
The solution of ( 10) is usually complicated to derive. A variety of methods [START_REF] Vujičić | View factor calculation using the monte carlo method and numerical sensitivity[END_REF]- [START_REF] Vujičić | Numerical sensitivity and view factor calculation using the monte carlo method[END_REF] have been proposed to calculate it. Here, we use the method in [START_REF] Sparrow | Radiation heat transfer[END_REF], which converts the double surface integrals into double contour integrals as follows:

F 21 = 1 2πA 2 Γ1 Γ2 ln s ds 2 • ds 1 , (11)
where Γ i denotes the contour of the ith surface, s i the position vector of an arbitrary point on boundary Γ i , and s = s 2 -s 1 the distance between two contour points. The advantage of using this approach is its efficient computation time [START_REF] Rao | Efficient evaluation of diffuse view factors for radiation[END_REF].

D. Thermal Servoing with Parallel Circular Surfaces

In this section, we derive the thermal servoing model for two parallel source-object surfaces. To this end, we denote the surfaces' center and radius by c i and r i , respectively. The origin of the coordinate system i 1 j 1 k 1 is set at c 1 , with a unit basis vector k 1 along the normal n 1 , and a unit basis vector i 1 perpendicular to the ground. We define i 2 j 2 k 2 as the translation of i 1 j 1 k 1 , with origin at c 2 . The scalars ω i denote the angle between i i and s i . We set the frames' centers at c 1 = [0, 0, 0] ⊺ and c 2 = [p 1 , p 2 , p 3 ] ⊺ , with respect to i 1 j 1 k 1 . The parametric position vectors s i are then computed as:

s 1 = r 1 cos ω 1 r 1 sin ω 1 0 ⊺ s 2 = r 2 cos ω 2 + p 1 r 2 sin ω 2 + p 2 p 3 ⊺ . ( 12
)
Their differential changes satisfy the following relations: Then, the distance s = s 2s 1 can be derived as:

ds 1 = -r 1 sin ω 1 dω 1 r 1 cos ω 1 dω 1 0 ⊺ ds 2 = -r 2 sin ω 2 dω 2 r 2 cos ω 2 dω 2 0 ⊺ ds 1 • ds 2 = r 1 r 2 cos(ω 1 -ω 2 ) dω 1 dω 2 . (13)
s = s(p 1 , p 2 , p 3 , ω 1 , ω 2 ) = p 1 2 + p 2 2 + p 3 2 + 2p 1 (r 2 cos ω 2 -r 1 cos ω 1 ) + 2p 2 (r 2 sin ω 2 -r 1 sin ω 1 ) +r 1 2 + r 2 2 -2r 1 r 2 cos(ω 2 -ω 1 ) 1 2 . ( 14
)
By substituting ( 13)-( 14) into [START_REF] He | Eddy current volume heating thermography and phase analysis for imaging characterization of interface delamination in cfrp[END_REF], F 21 can be calculated via:

F 21 = 2π 0 0 2π r 1 r 2 2πA 2 cos (ω 1 -ω 2 ) ln s(x, ω 1 , ω 2 ) dω 2 dω 1
(15) where we define the end-effector position as x = [p 1 , p 2 , p 3 ] ⊺ . By injecting [START_REF] Ho | Towards a MRI-compatible meso-scale smaactuated robot using pwm control[END_REF] into [START_REF] Gade | Thermal cameras and applications: A survey[END_REF], we can finally obtain the system's thermal-geometric relation:

v = f (x, T 2 ) = λ 1 F 21 -λ 2 T 2 4 + λ 3 (16)
where f (•) is the function in [START_REF] Navarro-Alarcon | Automatic 3D manipulation of soft objects by robotic arms with adaptive deformation model[END_REF]. By differentiating [START_REF] Backer | Temperature control of robotic friction stir welding using the thermoelectric effect[END_REF], we obtain the following key dynamic model:

v = l • u -4λ 2 T 2 3 v (17)
where l = λ 1

∂F21 ∂x

⊺ denotes the interaction/Jacobian matrix (vector, in this example), and u = ẋ ∈ R n is the robot's Cartesian velocity. The above expression is used for designing control laws for ẋ in the following sections. By using Leibniz integral rule [START_REF] Flanders | Differentiation under the integral sign[END_REF], the interaction matrix can be expressed as:

l =         2π 0 0 2π h(p 1 + r 2 cos ω 2 -r 1 cos ω 1 ) dω 2 dω 1 2π 0 0 2π h(p 2 + r 2 sin ω 2 -r 1 sin ω 1 ) dω 2 dω 1 2π 0 0 2π hp 3 dω 2 dω 1         (18)
with the scalar h defined as:

h = λ 1 r 1 r 2 cos(ω 1 -ω 2 )/(2πA 2 s 2 ) ( 19
)
Since it is hard to analytically compute the double integrals, we use numerical methods to approximate l in real-time.

E. Circular Surfaces in Arbitrary Configurations

In this section, we extend the parallel surfaces problem to a 6-DOF scenario, where the end-effector configuration is now defined as

x = [p 1 , p 2 , p 3 , θ x , θ y , θ z ]
⊺ , for θ i as the angles around the object's coordinate system (see Fig. 3). We denote by R the 3D rotation matrix corresponding to this relative orientation. Note that in some configurations of this nonparallel case, radiation from a source's region cannot reach the front side of object's surface (hence, will not contribute to the heat inflow). We refer to this problem as self-obstruction.

To model this situation, let us denote the object plane as D 1 D 2 c 2 , for D 1 and D 2 as the intersections with the bounded source plane. This setup is depicted in Fig. 4, where the heat source is divided into two surfaces: the red surface composed 4

of > D 1 D 5 D 2 D 2 D 1 and the black surface composed of > D 2 D 6 D 1 D 1 D 2 .
The black surface only "sees" the object's backside (i.e. the robot's gripper), thus, is omitted from the following calculation of F 21 :

F 21 = 1 2πA 2 Γa Γ2 ln s a ds 2 ds a + Γ l Γ2
ln s l ds 2 ds l [START_REF] Mckemy | Temperature sensing across species[END_REF] where Γ a denotes the arc

> D 1 D 5 D 2 and Γ l the line D 1 D 2 .
To derive its respective position vectors s a and s l , we compute the vector

n 2 = [n 1 2 , n 2 2 , n 3 2 ]
⊺ normal to the object plane as:

n 2 = R 0 0 -1 ⊺ (21)
whose plane equation satisfies:

n 1 2 (x -p 1 ) + n 2 2 (y -p 2 ) + n 3 2 (z -p 3 ) = 0 (22)
To find the intersection with the plane, we use ( 21) and substitute x = r 1 cos ϕ, y = r 1 sin ϕ and z = 0 into [START_REF] Imdoukh | Semi-autonomous indoor firefighting uav[END_REF], for ϕ as a variable angle. Self-obstruction occurs when there exist two solutions ϕ 1 and ϕ 2 ; After some algebraic operations, the arc and line parametric vectors are obtained from:

s a =   r 1 cos ϕ r 1 sin ϕ 0   , s l =   x l k l (x l -r 1 cos ϕ 2 ) + r 1 sin ϕ 2 0   (23) for a distance range x l ∈ [r 1 cos ϕ 2 , r 1 cos ϕ 1 ], an angle range ϕ ∈ [ϕ 1 , ϕ 2 ]
, and a slope k l = sin ϕ2-sin ϕ1 cos ϕ2-cos ϕ1 of the line D 1 D 2 . The parametric vector on the object's contour is computed as:

s 2 = R r 2 cos ω 2 r 2 sin ω 2 0 ⊺ + p 1 p 2 p 3 ⊺ (24)
As with the parallel surface case, we compute the arc and line distances s a = s 2 -s a and s l = s 2 -s l . The 6-DOF view factor for the self-obstruction case is as follows:

F 21 = 1 2πA 2 ϕ2 ϕ1 0 2π ln s a dω 1 dϕ + 1 2πA 2 r1 cos ϕ1 r1 cos ϕ2 0 2π ln s l dω 1 dx l . (25)
With this expression, we can derive a similar GTM model

v = l • u -4λ 2 T 2 3 v
, where u = ẋ ∈ R 6 and the interaction matrix is l = λ 1 ∂F21 ∂x ⊺ ∈ R 6 . For this 6-DOF case, we use the following numerical differentiation method to approximate l:

l = λ 1    (F 21 (p 1 + dp 1 , p 2 , ..., θ z ) -F 21 (x)) /dp 1 . . . (F 21 (p 1 , p 2 , . . . , θ z + dθ z ) -F 21 (x)) /dθ z    (26)
Parallel programming techniques can be applied to achieve real-time capabilities, where every element of l is simultaneously calculated by an independent process.

Remark 3. When the object plane and the bounded source plane do not intercept, we classify it either as a common (nonself-obstruction) case or as a complete self-obstruction case. This classification can be done by checking whether the object plane intercepts the bounded source plane. For the common case, the entire contour of the heat source will be used to calculate F 21 (in this situation, we use [START_REF] He | Eddy current volume heating thermography and phase analysis for imaging characterization of interface delamination in cfrp[END_REF] for calculation). For the complete self-obstruction case, we set F 21 = 0.

F. Thermal Servoing Model with Multiple Objects

Here, we consider the case where the robot rigidly manipulates N "small" objects in space (for N ≤ 3) with its end-effector, and regulates the temperature of each sensing point. The local range of feasible temperatures for each object is constrained by the geometry of the view factors and the rigid inter-object kinematics. For this situation, we assume that heat exchange amongst the objects is negligible, therefore, the derivation of the N -object interaction matrix L ∈ R N ×n (where n is the number of DOF of the robot) is analogous to the previous sections and is simply constructed with N vectors l i as follows:

L = l 1 • • • l N ⊺ . ( 27
)
To effectively control each feedback temperature, the number of robot DOF must satisfy N ≤ n. For this multi-object system, we construct the following structures:

τ = T 1 2 • • • T N 2 ⊺ ∈ R N (28)
T = diag (T 1 2 ) 3 , • • • , (T N 2 ) 3 ∈ R N ×N (29) v = v 1 • • • v N ⊺ ∈ R N (30)
The constant thermophysical parameters λ 2 are defined for ith object as λ i 2 , and are grouped into the constant matrix:

Λ = diag(λ 1 2 , • • • , λ N 2 ) ∈ R N ×N (31)
With all these terms, the geometric-thermal-motor model can be extended to a multi-object case:

v = Lu -4TΛv (32)

G. Irregularly Shaped Surfaces

In practical applications, contours are typically irregular, thus, have no simple parametric form. Depending on the requirements of the application, we can use the following two strategies to calculate the view factor (which is needed to derive the thermal interaction matrix L): (i) If the computation power of the robot is limited (e.g. field robots), we recommend to use a simple shape (e.g. circles and rectangles) to approximate the real contour; (ii) If an accurate calculation of the view factor is required for conducting numerical simulation and analysis (as will be discussed in Sec. IV-F), methods such as truncated Fourier series [START_REF] Navarro-Alarcon | Fourier-based shape servoing: a new feedback method to actively deform soft objects into desired 2-d image contours[END_REF] could be used to parameterize the irregular contour (we verified this approach on an Intel i7-9750H CPU using 5 harmonics and obtained an estimation error smaller than 0.2% with a computation time of 0.15 s).

III. CONTROLLER DESIGN

A. Model-Based Controller

To regulate the feedback temperature of N objects5 , we design the following velocity controller (see Fig. 5):

u = L + (-Dv -K∆τ + 4TΛv) (33)
where

L + = L ⊺ (LL ⊺ ) -1
is the right pseudoinverse of L, and D > 0 and K > 0 are control gains. Note the analogy with visual servoing, which relies on the model of the first derivative of the task error, ė = Lu, to regulate e to 0 via: u = -L + Ke, which enforces the closed-loop system ė = -Ke. Here, we rely on the second derivative of the task error, ë = Lu + L ′ ė (viz. system [START_REF] Mishra | View factor calculation in the 2-d geometries using the collapsed dimension method[END_REF] with e = ∆τ , v = ė and L ′ = -4TΛ) and regulate e to 0 via: [START_REF] Vujičić | Numerical sensitivity and view factor calculation using the monte carlo method[END_REF]). The closed-loop system becomes ë = -Ke -D ė, which is stable, as we will show hereby.

u = L + (-Ke -L ′ ė - D ė) (viz. controller
Proposition 1. Consider that thermodynamic parameters in [START_REF] Mishra | View factor calculation in the 2-d geometries using the collapsed dimension method[END_REF] are accurately known. For this situation, the control input (33) enforces a stable closed-loop system which asymptotically minimizes ∆τ .

Proof: Substituting (33) into the nonlinear dynamic system [START_REF] Mishra | View factor calculation in the 2-d geometries using the collapsed dimension method[END_REF], yields the following closed-loop system:

v = -Dv -K∆τ . ( 34
)
Consider the quadratic Lyapunov function

Q(v, ∆τ ) = 1 2 v 2 + 1 2 K ∆τ 2 (35)
whose time derivative along trajectories of (34) yields

Q(v, ∆τ ) = v ⊺ v + K∆τ ⊺ v = -D v 2 (36)
which shows that the energy function is non-increasing, i.e. Q ≤ 0, thus, the closed-loop system is stable. By applying the Krasovskii-LaSalle principle, the asymptotic minimization of ∆τ can be proved.

Remark 4. In our proposed method, the terms T and τ in the controller (33) can be directly obtained from real-time sensor measurements. Yet, to implement the variable v, we use a rate estimation algorithm based on polynomial fitting with sliding windows [START_REF] Titchener | On the calculation of boundary-layer parameters from discrete data[END_REF].

B. Adaptive Controller

In the above model-based controller, we assume that the object's thermophysical properties are exactly known in advanced. However, due to uncertainties in the material's conditions, it is hard to know these values in practice. To deal with this issue, we propose an adaptive control strategy that does not require knowledge of the true parameters. To this end, we start by introducing the unknown parameters a 1 = 1 λ1 and a 2 = λ2 λ1 , which are well-defined since λ 1 > 0; We use the superscripts a i 1 and a i 2 to distinguish them between different objects. With these parameters, we construct the following constant vector a 1,2 ∈ R N and matrix A 1,2 ∈ R N ×N :

a 1 = a 1 1 • • • a N 1 ⊺ , A 1 = diag(a 1 ) > 0, a 2 = a 1 2 • • • a N 2 ⊺ , A 2 = diag(a 2 ) > 0. ( 37
)
By applying the dynamic expression [START_REF] Kim | Real-time probabilistic classification of fire and smoke using thermal imagery for intelligent firefighting robot[END_REF] to the multi-object case and dividing it by λ i 1 for each ith object, we obtain:

A 1 v + 4TA 2 v = Ju (38)
for an new interaction matrix

J = l 1 /λ 1 1 • • • l N /λ N 1 ⊺ ∈ R N ×6 (39)
which is independent from the unknown thermophysical parameters 6 and is entirely computed with the N gradients of the view factors. To design the adaptive controller, it is useful to introduce the combined thermal error vector

ζ = [ζ 1 , . . . , ζ N ] ⊺ ∈ R N defined as: ζ = ∆ τ + µ∆τ = v + µ∆τ (40)
for µ > 0 a feedback gain. To control the N object temperatures, we design the following velocity input (see Fig. 6):

u = J + (-µ A 1 v -Kζ + 4T A 2 v) (41)
where the elements of the adaptive diagonal matrices A i = diag( a i ) ∈ R N ×N are computed with the update rules:

˙ a 1 = γ 1 µ v 1 ζ 1 . . . v N ζ N ⊺ ∈ R N (42) ˙ a 2 = -4γ 2 v 1 ζ 1 (T 1 2 ) 3 . . . v N ζ N (T N 2 ) 3 ⊺ ∈ R N (43)
where scalars γ i > 0 are used for tuning the algorithm's rate.

Proposition 2. The adaptive controller [START_REF] Navarro-Alarcon | A Lyapunov-stable adaptive method to approximate sensorimotor models for sensor-based control[END_REF] with update rules (42)-( 43) guarantees a bounded estimation of the unknown parameters a 1 and a 2 , and the asymptotic minimization of the thermal error ∆τ .

Proof: Substitution of ( 41) into (38) yields:

A 1 v + 4TA 2 v = -µ A 1 v -Kζ + 4T A 2 v (44)
By adding µA 1 v to both sides of [START_REF] Endres | A simplicial homology algorithm for lipschitz optimisation[END_REF], noting that ζ = v+µv, and performing some algebraic operations we can obtain:

A 1 ζ + Kζ = -µ A 1 v + 4T A 2 v. ( 45
)
6 An analogous result to the depth-independent interaction matrix in [START_REF] Liu | Uncalibrated visual servoing of robots using a depth-independent interaction matrix[END_REF] for error matrices

A i = A i -A i = diag( a i ), with error vectors a i = a i -a i .
To analyze the stability of the closedloop dynamical system ( 42)-( 43) and ( 45), we introduce the following Lyapunov function:

H(ζ, a 1 , a 2 ) = 1 2 ζ ⊺ A 1 ζ + 1 2γ 1 a 1 2 + 1 2γ 2 a 2 2 (46)
whose time derivative along ( 42)-( 43) and ( 45) yields

Ḣ(ζ, a 1 , a 2 ) = ζ ⊺ A 1 ζ + 1 γ1 ˙ a ⊺ 1 a 1 + 1 γ2 ˙ a ⊺ 2 a 2 = -K ζ 2 (47)
which shows that the energy function is non-increasing, i.e. Ḣ ≤ 0, thus, the parameter estimation errors a i are bounded. Asymptotic stability of ∆τ directly follows by applying the Krasovskii-LaSalle principle.

C. Target Feasibility

In previous sections, we proved that ∆τ can be asymptotically minimized by two automatic controllers. However, it is not guaranteed that such error can be enforced to zero. Failure cases are caused by the choice of unfeasible target temperatures: Intuitively, if targets are set to too high/low, they might be physically unachievable; In addition, for objects fixed to the same end-effector, the difference range between their target temperatures is constrained by the fixed distance between the objects. In this section, we analyze two necessary but not sufficient conditions to ensure the feasibility of the targets. Failure experiments are analyzed in Sec. IV-F.

Consider a simple case with two objects, object 1 and object 2, fixed to the end-effector (the extension to N object is straightforward). For one of the objects, recall the thermalgeometric relation [START_REF] Backer | Temperature control of robotic friction stir welding using the thermoelectric effect[END_REF] and rewrite it as follows:

v = -λ 2 T 2 4 + λ 1 F 21 (x o ) + λ 3 (48)
where x o denotes an object configuration. Let us assume there exists a temperature T 2 = T v0 that makes the rate v = 0. As temperature is always non-negative, T v0 can be solved as:

T v0 (F 21 ) = ((λ 1 F 21 + λ 3 )/λ 2 ) 1 4 (49)
Since the parameters λ i > 0 are all positive and F 21 ∈ [0, 1), T v0 always exists. T v0 represents the steady state temperature at x o . Note that T v0 is a function of F 21 and that ∂T v0 /∂F 21 > 0 is always positive. Thus, the minimum value of T v0 is determined when F 21 = 0 as:

min(T v0 ) = (λ 3 /λ 2 ) 1 4 = (α 2 T 4 3 /ε 2 ) 1 4 (50)
According to Kirchhoff's law of thermal radiation [START_REF] Bergman | Fundamentals of heat and mass transfer[END_REF], at thermodynamic equilibrium, α 2 = ε 2 . Thus, the minimum is:

min(T v0 ) = T 3 (51)
When F 21 (x o ) → 1, the maximum value of T v0 approaches:

max(T v0 ) → ((λ 1 + λ 3 )/λ 2 ) 1 4 = (α 2 ε 1 T 4 1 /ε 2 ) 1 4 = ε 1 4
1 T 1 (52) From ( 51)-(52), we derive the first boundary value condition:

T 1 * , T 2 * ∈ [T 3 , ε 1 4 1 T 1 ) (53)
Now we discuss the limitation of the difference between target temperatures |δT * | = |T 1 * -T 2 * |. We denote the

χ 1 (F 21 ) = λ 1 1 F 21 + λ 1 3 λ 1 2 1 4 , χ 2 (F 21 ) = λ 2 1 F 21 + λ 2 3 λ 2 2 1 4
(55) where λ 1 i and λ 2 i are the thermophysical parameters of the two objects. Note that for the continuous function ∆T v0 (x + ∆x 1 , x + ∆x 2 ), where x ∈ W for W as the bounded workspace and ∆x j as constant vectors, there must exist a minimum value min(∆T v0 ) = ∆T v0 (x min ) and a maximum value max(∆T v0 ) = ∆T v0 (x max ) which encompass all possible values of ∆T v0 , where x min and x max are the endeffector configurations corresponding to the two extreme cases. The second condition for feasible target temperatures is:

δT * ∈ [min(∆T v0 ), max(∆T v0 )] (56)
A numerical (geometric) interpretation of x min and x max will be discussed in Section IV-F.

IV. RESULTS

A. Experimental Setup

We conducted a series of experiments on a 4-DOF robot (3 translations and 1 rotation) to evaluate the proposed method. Fig. 7 shows the robot, whose end-effector is replaced by a 3D printed connector fixed to an aluminum holder. The objects are attached to the holder through an adiabatic layer to minimize heat conduction. We prepared three different kinds of objects for temperature control experiments (see Fig. 7): An aluminum circular sheet with 1.5 cm radius and 3 mm thickness; A bunny-shaped object with 1 mm thickness, 3D printed using polylactic acid (PLA) material with 30% infill density; A handshaped sheet with 1 mm thickness, also 3D printed using PLA but with 50% infill density. We approximate the aluminum sheet's and the heat source's thermophysical properties via standard tables [START_REF] Bergman | Fundamentals of heat and mass transfer[END_REF]. The object's emittance, absorptance, specific heat, and density are 0.04, 0.04, 903 J • K -1 • kg -1 , and 2702 kg/m 3 , respectively. The source's emittance and absorptance are estimated as 0.25 and 0.25.

For the two 3D printed objects, different infill densities, colors, and uncertain surface conditions make their thermophysical properties hard to be estimated. Thus, we only consider the aluminum sheet for the experiments with the model-based controller (adaptive control is used for the other objects). A radiative heating platform with adjustable temperature output is used as the heat source. The (indoor) environment temperature is assumed to be constant at 23 °C.

To obtain the feedback temperatures, we attach a PT100 platinum thermistor with 0.3 °C accuracy and 0.1 °C precision to each object. The raw data obtained by thermistors is processed by a current-temperature transformation module and sent to a Linux-based control computer as the feedback signal. The motion command is calculated by the computer program and sent to the robot under a position-stepping mode. At the beginning of the experiments, we use an RGB camera and three ArUco markers [START_REF] Garrido-Jurado | Automatic generation and detection of highly reliable fiducial markers under occlusion[END_REF] to calibrate the configuration between the heat source and the end-effector (see Fig. 8).

B. Experiments with the Model-Based Controller

We conduct a series of thermal servoing experiments to evaluate our proposed control methodology (see Fig. 9 for a representative experiment). Here, we first evaluate the performance of the model-based controller with aluminum objects (whose properties are approximately known); The experiments are conducted with a source's temperature of 200 °C. We denote the scalar temperature error by ∆T = T 2 -T * . By using the controller (33), we enforce a closed-loop heat transfer system that resembles a mass-spring-damper system. Therefore, the values of the stiffness/damping-like gains K and D can be used to specify the system's performance. Fig. 10 (a) demonstrates the effect of the gain K on the thermal response. For that, we set D = 0.2 and T * = 50 °C and conduct three experiments with different K values. These show that when K = 0.005 (red curve), the error ∆T asymptotically decreases to zero with a relatively slow speed; When K = 0.05 (blue curve), ∆T decreases faster and a small overshoot occurs; When K = 0.5 (black curve), ∆T oscillates near zero with an approximate 3 °C amplitude (which demonstrates the controller's capability to deal with overheating). This results shows how the closed-loop system varies from over-damped to under-damped. Thus, the gains should be specified according to the desired thermodynamic performance. We further conducted experiments with the same gains (K = 0.05, D = 0.2) but with different targets T * and found a consistent response (see Fig. 10 (b)).

Model-based experiments were also conducted to independently regulate the temperatures of two aluminum objects, shown in Fig. 7 (c). We designed 4 experiments with different targets τ * (measured in °C). Fig. 11 ⊺ in (a), (b), (c), and (d), respectively. For the first three experiments where the differences between the target temperatures |T * 1 -T * 2 | are small (or null), the thermal error ∆τ can be asymptotically minimized to zero. However, when |T * 1 -T * 2 | is large, as in Fig. 11 (d), the two temperatures cannot be accurately controlled. This failure case can be explained by the second condition for feasible targets discussed in Section III-C.

C. Experiments with the Adaptive Controller

We designed a series of experiments to evaluate the performance of the proposed adaptive controller. For that, we consider with three different objects (see Fig. 7 (d)) with unknown thermophysical properties and irregular shapes. To compute the interaction matrix, we use truncated Fourier series with 5 harmonics terms; This approach provides a fast calculation time with a "good enough" shape approximation. The controller's gains are set to µ = 0.05 and K = 0.15. To initialize the parameters a i (0) at the time instance t = 0, we use (for the "hand" and "bunny" objects) the constant values calculated for the aluminum object in the previous modelbased controller, i.e. a i (0) = a i ; For the circular object, we simply initialize a i (0) with random values.

In this study, we report eight temperature control experiments with different targets, objects and source conditions. Figure 12 shows the evolution of the individual thermal errors ∆T i . For ease of presentation, we name these eight experiments as exp 1, . . . , exp 8, and denote the corresponding target temperature for each experiment by τ * 1 , . . . , τ * 8 . In exp 1, we set the three target temperatures to the same value. In exp 2exp 4, we only set two targets to the same value. In exp 5 and exp 6, set all targets to different values, with a non-uniform thermal separated in exp 5 and a uniform one in exp 6. In exp 7 and exp 8, all targets are set to the same value, but with different heat source conditions. The source temperature T 1 is set to 200 °C in exp 1exp 6, to 300 °C in exp 7, and varies from 200 °C to 300 °C in exp 8.

In all these experiments with all these different conditions, the magnitude of the temperature error τ asymptotically decreases to zero. Yet, failure control experiments do happen and are reported and discussed in Section IV-F). The results experimentally confirm that (for feasible target temperatures) the adaptive method is able to independently regulate temperatures of various objects with different shapes and materials, without exact knowledge of their thermophysical properties or the source's/environment's temperatures. Fig. 13 depicts the performed object trajectories during the experiments in Fig. 12. The boundary of the circular heat source is depicted as a black circle (and ellipse). The color of a trajectory point represents the feedback temperature at that position; Variation from blue to red corresponds to a change from "low" to "high". For clarity, we depict two sets of trajectory visualizations from different viewing angles: For Fig. 13 (a 1 ), (b 1 ), . . . , (h 1 ), the trajectories are viewed ink 1 direction; For Fig. 13 (a 2 ), (b 2 ), . . . , (h 2 ), the trajectories are viewed in i 1 direction.

From these trajectory visualizations, we can see that when target temperatures are set to different values, the object with a higher target temperature usually reaches a position that is closer to the center of the heat source; This situation will be further discussed in the Section IV-E. For the case when target temperatures are set to the same value, the final position of the circular aluminum sheet is always closer to the center of the heat source. This phenomenon could be explained by the fact that absorptance of a metal is usually much smaller than the absorptance of non-metallic materials (e.g. PLA) [START_REF] Lienhard | A heat transfer textbook[END_REF]. It is also worth noting how the controller can cope with sudden changes in the source's temperature (see Fig. 12(h)).

D. Visual Recalibration of Uncertain Interaction Matrices

In this section, we report an integrated visual-thermal experiment, where the adaptive controller is combined with an online ArUco tracking algorithm to achieve temperature regulation when the source's location is uncertain. This geometric information is essential to compute the thermal interaction matrix L.

Here, we study the case where the robot is fixed and the heat source is manually moved. We track marker 2 attached to the source to obtain its configuration. The target temperature vector is set to τ = [40, 40, 40] ⊺ °C. Fig. 14 shows: (a) The initially calibrated set up, (b) the manual movement applied to the source, (c) the detected marker 2 when the source is moving, and (d) the evolution of the individual temperature errors. These results show that by continuously updating the source-object pose, the control of the individual temperature errors is not significantly affected and that ∆τ can still be asymptotically minimized. This experiment demonstrates how our new thermal servoing method can be combined with other traditional methods (vision-based controls in this case) to extend the sensorimotor capabilities of a robot [START_REF] Navarro-Alarcon | A Lyapunov-stable adaptive method to approximate sensorimotor models for sensor-based control[END_REF].

E. View Factor Visualization

In previous sections, we designed thermal controllers based on derived heat transfer models. However, the models that relate F 21 and x are generally complex. Therefore, part of the controlled system behaves as a "black box" to the user. To investigate these aspects, in this section, we introduce the visualization of the view factor F 21 with respect to the end-effector configuration x as a useful tool for analyzing radiation-based thermal servoing problems.

As an example, we take the "circular surfaces in arbitrary configurations" case discussed in Section II-E. We implement the controlled variable method [START_REF] Williams | Research methods[END_REF] to split the 6-DOF pose x into two subsets: One where the translation coordinates p 1 , p 2 , and p 3 (measured in cm) are the controlled variables, and another where the rotation coordinates θ x , θ y , θ z (measured in degrees) are controlled variables. In the translation subset, rotations are set to constant values of θ i = 0, then, we compute F 21 for points in a selected working range of controlled variables p 1 , p 2 ∈ [-20, 20], p 3 ∈ [0, 30] with a step of 1 (with 48, 000 points in total). In the rotation subset, translations are similarly set to constant values p 1,2 = 0 and p 3 = 5 cm; then, points in the range of θ x , θ y , θ z ∈ [-90, 90] are computed with an incremental step of 2 (i.e., 729, 000 points in total).

We use the isosurface visualization tool provided by Plotly [START_REF] Inc | Collaborative data science[END_REF] to visualize the data. The translation and rotations subsets are shown in Fig. 15, where 3-DOF end-effector configurations are represented by points in space, and the view factor values are represented by isosurfaces with different colors (the isosurfaces are formed by points which have the same or very close values of F 21 ). This visualization method is inspired by the approximation of the interaction matrix in [START_REF] Arechavaleta | An optimality principle governing human walking[END_REF], which reveals that L is positive proportional to the directional derivative of F 21 (x) along x as L ⊺ = λ 1 ∇ x F 21 (x). According to the definition of isosurface, the surface normal of every point on the surface also points in ∇ x F 21 (x) direction. In addition, the interval distance between isosurfaces with an equal value difference (also called "isosurface interval") reveals the magnitudes of the elements of ∇ x F 21 (x); A larger distance represents a smaller magnitude.

As an example, let us analyze the translation subset shown in Fig. 15 (a). For this single-object scenario, the normal vector at a point on the isosurface indicates the direction of the endeffector movement (as computed from the thermal controls ( 33), ( 41)) at that point. There are some characteristics of these isosurfaces that can be intuitively deducted from the setup, e.g., the symmetric spatial distribution of F 21 (due to the circular shape of the heat source), and the proportionality of values of F 21 with respect to the source-object separation.

However, the visualization provides two useful pieces of information. First, that the centers of the incomplete spherical isosurfaces shift upwards when F 21 decreases, which means that at some points, movement in the k 1 direction will cause a decrease of F 21 (which seems counter-intuitive). See e.g. c 2 on the F 21 = 0.1 isosurface in Fig. 15 (a), which shows that in that configuration, the end-effector needs to move backwards along the k 1 direction to heat up faster. Second, the isosurface intervals at regions that are farther from the heat source center are comparatively larger, which indicates that the end-effector will move comparatively faster in those regions. Similarly, Fig. 15 (b) shows the (much simpler) case where angles are varied at a fixed position.

F. Unfeasible Thermal Targets

In Section III-C, we discussed two necessary but not sufficient conditions for feasible targets. When one of the two conditions is not fulfilled, the temperature error cannot be minimized to zero. This section reports and analyzes two failed experiments with the proposed adaptive controller where the target temperatures are set to τ * = [80, 80, 80] ⊺ and τ * = [70, [START_REF] Rao | Efficient evaluation of diffuse view factors for radiation[END_REF][START_REF] Rao | Efficient evaluation of diffuse view factors for radiation[END_REF] ⊺ °C; The temperature errors of each coordinate ∆T i are depicted in Fig. 16 (a),(c). The evolution of the error ∆τ for the two experiments is shown in Fig. 16 (b),(d). In this experimental study, we found that when all the individual target temperatures are set relatively "high" (80 °C in this example), its corresponding errors converge to a local minimum. Also, when the difference between target temperatures is too large, one of the objects might more closely reach its target, while the other will present steady-state errors.

In Section III-C, we prove that the steady-state temperature of an object heated by a radiative source is directly propor- tional to F 21 . Thus, the geometry of view factor isosurfaces is a useful tool for analyzing the such reachability conditions. Here, we discuss a simple but representative case where two aluminum circular sheets with radius r o1 = 1.5 cm, r o2 = 4.5 cm are attached to the end-effector at o 1 and o 2 (see Fig. By using the expression (49) and assuming that the thermophysical properties are the same as mentioned in Section IV-A, the view factor values corresponding to steady-state temperatures 30 °C, 40 °C, 50 °C are calculated as 0.12, 0.37, and 0.65. According to this one-to-one correspondence between the isosurface and the steady-state temperature, to automatically reach the target temperature T * can be geometrically interpreted as positioning the object center over the isosurface that corresponds to T * . Similarly, determining the feasibility of target temperatures T * 1 and T * 2 of two objects attached to the same end-effector is identical to finding whether there exists an end-effector pose that places both objects onto their "desired isosurfaces".

An example is shown in Fig. 18, where we denote the steady-state temperatures of objects 1 and 2 by T 1 ss and T 2 ss . Fig. 18 (a)-(b) show the steady-state temperature isosurfaces of objects 1 and 2 where T 1 ss = T 12 ss = 30 °C, 40 °C, 50 °C. Since the two objects are circular plates with different radii, the shapes of their isosurfaces are slightly different; We use red and blue color to differentiate them, and are jointly depicted in the same coordinate system in Fig. 18 (c).

Fig. 18 (d), (e), (f) depict different combinations of target temperatures T 1 * , T 2 * and their corresponding isosurfaces. These figures graphically demonstrate how for thermal targets T * 1 = T * 2 = 30 °C (depicted in Fig. 18 (d)), and T * 1 = 30 °C, T * 2 = 40 °C (depicted in Fig. 18 (e)), the end-effector can position the objects into their desired final isosurfaces (corresponding to their target steady-state temperatures); The initial position of this trajectory is colored in blue, the final in red. However, for the case where T * 1 = 50 °C and T * 2 = 30 °C, Fig. 18 (f) shows that the minimum distance between the two isosurfaces is larger than l e1 + l e2 ; Thus, targets T * 1 and T * 2 are not feasible. Similarly, if l e1 + l e2 is larger than the maximum distance between two target isosurfaces, that combination of T * 1 and T * 2 is also unfeasible.

In addition to the aforementioned geometric explanation, we conduct an analysis of the characteristics of the entire feasible temperature space Θ temp , which is defined as the collection of all sets of steady-state temperature T ss (x) = [T 1 ss (x 1 obj ), . . . , T n ss (x n obj )], T ss (x) ∈ Θ temp of N objects attached to the same end effector, where x is the end-effector configuration, x n obj , n = 1, . . . , N is the object configuration, and T n ss (x n obj ) is the steady-state temperature of an object when its center is at x n obj . Since there is a fixed geometric relationship between x and x n obj , values of T n ss (x n obj ) can be calculated according to (49).

Here, we take the set-up in Fig. 19 (b) where 2 identical circular aluminium objects are attached to a 3-DOF endeffector with an identical distance l e as a case of study. We uniformly sample a discrete end-effector configuration space where p 1 , p 2 ∈ [-30 cm, 30 cm], p 3 = 1 cm with a step of 0.2 cm (90,000 points) and calculate the view factor values. Accordingly, T ss (x) ∈ R 2 , and Θ temp ∈ R 90000×2 can be obtained. For each T ss (x) in a feasible temperature space Θ temp , we depict it as a 2D point such as in Fig. 19 (a). Although x is uniformly sampled, the distribution of T ss (x) is not uniform, which is caused by the non-linear thermalgeometric coupling. By changing the values of l e , p 3 , several representative combinations are also depicted in Fig. 19 (c), (d), (e). We analyze these figures and discover the following thermal-geometrical coupling characteristics: Consider Fig. 19 (c),(d), if the end-effector moves away from the heat source (p 3 increases), the area of feasible temperature space Θ temp shrinks, which means the range of feasible temperature is smaller. However, the density of points increases; It indicates the robot motion induces a smaller change on heat transfer to the object, which to some extent increases the accuracy of temperature control when p 3 increases. Secondly, refer to Fig. 19 (e), we discover the geometric coupling between objects (l e in this case) also affects Θ temp . If objects are close (e.g. l e = 1), and the difference between the target temperature of the objects is large, these combinations of target temperature are not feasible (see the blank area e 1 , e 2 ); If objects are far away (e.g. l e = 9), controlling two objects to reach high temperature simultaneously is not feasible (see the blank area e 3 ).

For cases where N > 3, visualization is not practical, and some advanced data analysis is required (which is beyond the scope of this paper). The analysis of the feasible temperature space reveals the physical essence of the thermalgeometric coupling, and could shed some light on practical thermal servoing system design. Nevertheless, it takes hours of computation even with parallel-computing to obtain the required data. To quickly verify the feasibility of a specific set of target temperature, we find reformulating the problem from an optimization perspective is more effective.

Consider N objects attached to the same end-effector, the steady-state temperature of each object is denoted by T n ss (x n obj ), n ∈ {1, 2, . . . , N }. We denote the target temperature of each object by T n * , then the target feasibility problem where W is the robot working space. If the global minimum of c(x) equals to 0, the set of target temperature is feasible. We use a simple homology global optimization [START_REF] Endres | A simplicial homology algorithm for lipschitz optimisation[END_REF] algorithm, which is implemented in SciPy library, to conduct a verification of the method's feasibility. It turns out that the global minimum of c(x) can be found effectively (in less than 1 minute) for randomly selected target temperatures and geometric relationships between objects. In general, thermophysical properties, view factors, and fixed spatial relationships between objects are the main three factors that determine the feasibility of thermal targets. The geometric interpretation of the feasibility problem might also be useful for path planning-like algorithms dealing with thermal servoing problems. The accompanying multimedia file demonstrates the performance of our method with multiple experiments, including overheat, unfeasible targets, single/multiple objects, and view factor visualizations.

V. CONCLUSION

In this paper, we present a new robotic temperature control technique based on heat radiation to automatically regulate temperatures of multiple objects. For that, we provide a comprehensive formulation of different scenarios of thermal servoing problems. Two asymptotically stable controllers, one model-based and one adaptive, are designed and validated by a series of experiments where temperatures of three different objects are independently regulated. We also discussed potential applications of the isosurface visualization, such as analyzing the geometry of the seemingly invisible heat transfer process.

The key concept of the proposed method is to exploit the geometric-thermal-motor relations between the heat source and the surface for automatically computing motion controls. This advanced feedback control capability is needed to improve the performance of many economically-important applications. However, from the point of view of generality, the proposed algorithm has many limitations, since we only consider cases where heat radiation is dominant. For cases where other heat transfer modes are dominant or comparable (objects in contact with non-adiabatic surfaces, electrical equipment cooled by high speed air flow, human skin treated by laser thermal excitation, food heated up in a pan, etc.), different heat transfer models need to be analyzed. Another possible solution is to implement model-free control algorithms that primarily rely on collected data instead of analytical models (which we formulated based on fundamental physical principles).

For future work, we would like to integrate thermal servoing with existing visual and proximity servoing algorithms; This multimodal perceptual and control capability is essential for developing advanced robotic temperature control systems in complex scenarios, such as service tasks in human environments and and intelligent industrial manufacturing. Our team is currently working towards developing algorithms which simultaneously consider the three basic modes of heat transfer. For this situation, thermal images (which provide detailed temperature profiles of an object surface) may be used as a sensing system. Sensor-based feedback control is certainly not the only way to achieve robot temperature regulation. Formulating the problem from an optimization perspective could also be interesting. For scenarios where multiple heaters attached to robots are used to heat up a large object, the coverage problem [START_REF] Santos | Coverage control for multi-robot teams with heterogeneous sensing capabilities using limited communications[END_REF]

Fig. 1 .

 1 Fig. 1. Creatures and robots with thermo-motor intelligence: When exposed to the sun, butterflies adjust their wings configuration to control their temperature (a); robotic systems with thermal servoing algorithms can be used for firefighting (b), volcano exploration (c) and industrial applications (d).

Fig. 4 .

 4 Fig. 4. Conceptual representation of a self-obstruction case.

  Problem statement. Given a constant temperature reference vector τ * = [T * 1 , . . . , T * N ] ⊺ ∈ R N , design a velocitybased motion controller u that asymptotically minimizes the feedback error ∆τ = ττ * for all N objects.

Fig. 5 .

 5 Fig. 5. Schematic representation of the model-based controller.

Fig. 6 .

 6 Fig. 6. Schematic representation of the adaptive controller.

Fig. 7 .

 7 Fig. 7. Experimental setup for our radiation-based thermal servoing tests.

Fig. 8 .

 8 Fig. 8. Geometric calibration using ArUco markers before the experiments.

Fig. 9 .Fig. 10 .

 910 Fig. 9. Snapshots of a representative thermal servoing experiment: (a) Initial position, (b)-(c) transient motion, and (d) steady-state configuration.

Fig. 11 .

 11 Fig. 11.Evolution of the temperature errors (∆T 1 , ∆T 2 ) using two aluminum objects with the model-based controller.

  depicts the minimization of the thermal errors for these 4 experiments, with target temperatures defined as τ * = [50, 40] ⊺ , τ * = [60, 40] ⊺ , τ * = [50, 50] ⊺ , and τ * = [80, 40]

Fig. 12 .Fig. 13 .

 1213 Fig. 12. Evolution of the temperature errors of the three objects in the 8 experiments with the adaptive controller, ∆T 1 , ∆T 2 , and ∆T 3 (measured in °C). The target temperatures are set as: τ * 1 = [40 40 40] ⊺ , τ * 2 = [50 40 40] ⊺ , τ * 3 = [50 50 35] ⊺ , τ * 4 = [60 60 30] ⊺ , τ * 5 = [50 45 35] ⊺ , τ * 6 = [40 45 50] ⊺ , τ * 7 = [45 45 45] ⊺ , τ * 8 = [45 45 45] ⊺ .

Fig. 14 .

 14 Fig. 14. Experiments with the adaptive controller with disturbances.

Fig. 15 .

 15 Fig. 15. Isosurfaces visualization of the two view factor subsets: (a) Translation subset and (b) rotation subset.

Fig. 16 .

 16 Fig. 16. Evolution of the temperature errors with unfeasible target temperatures: (top) τ * = [80, 80, 80] ⊺ °C, and (bottom) τ * = [70, 35, 35] ⊺ .

17

 17

  ) and heated by a source with T 1 = 200 °C. The center of the end-effector is at o e , and l e1 = l e2 = 2 cm are the distances between the centers o 1 and o 2 and the end-effector o e . The view factors of the objects are calculated based on the same setup as in previous sections. We use the visualization method where three translations are the controlled variables, for a parallel object and source surfaces.

Fig. 17 .Fig. 18 .

 1718 Fig. 17. Conceptual illustration of two objects fixed to an end-effector for analyzing unfeasible target temperatures.

Fig. 19 .

 19 Fig. 19. Visualization of the target temperature space Θtemp.

  in multi-agent systems represents a feasible approach. The development of efficient algorithms to estimate in real-time the geometry of the view factors (and hence the target's feasibility) is still an open research problem. We encourage interested readers to work along these open research directions.

The GTM model is analogous to the geometric-image-motor model used in visual servoing to control the robot's motion[START_REF] Bechlioulis | Robust image-based visual servoing with prescribed performance under field of view constraints[END_REF]. Its derivation relies on thermophysical principles (to be introduced in Sec. II), which correspond to the role of a camera model in the visual servoing formulation.

Similar to the interaction (Jacobian) matrix of servoing problems, a thermal interaction matrix relates the heat energy inflow/outflow towards/from the object of interest (that causes a temperature change) with the robot motion.

The symbol > abc denotes the arc that passes through the points a, b and c.

Throughout this paper, we consider that N ≤ 3 ≤ n. The method's extension to N > n > 3 is straightforward (it only requires to use the left pseudoinverse). However, asymptotic stability cannot be ensured.

This work is supported in part by the Key-Area Research and Development Program of Guangdong Province 2020 under project 76, in part by the Research Grants Council of Hong Kong under grants 14203917 and 15212721, in part by PROCORE-France/Hong Kong Joint Research Scheme sponsored by the RGC and the Consulate General of France in Hong Kong under grant F-PolyU503/18, in part by the Jiangsu Industrial Technology Research Institute Collaborative Funding Scheme under grant 43-ZG9V, and in part by The Hong Kong Polytechnic University under grants ZZHJ and YBYT.