Edmond Menya
email: emenya@strathmore.edu

Mathieu Roche
email: mathieu.roche@cirad.fr

Roberto Interdonato
email: roberto.interdonato@cirad.fr

Dickson Owuor
email: dowuor@strathmore.edu

Enriching Epidemiological Thematic Features For Disease Surveillance Corpora Classification

Keywords: Epidemiology Intelligence, Thematic Epidemiology Features, Disease Surveillance, Text Mining, Corpus Classification

We present EpidBioBERT, a biosurveillance epidemiological document tagger for disease surveillance over PADI-Web system. Our model is trained on PADI-Web corpus which contains news articles on Animal Diseases Outbreak extracted from the web. We train a classifier to discriminate between relevant and irrelevant documents based on their epidemiological thematic feature content in preparation for further epidemiology information extraction. Our approach proposes a new way to perform epidemiological document classification by enriching epidemiological thematic features namely disease, host, location and date, which are used as inputs to our epidemiological document classifier. We adopt a pre-trained biomedical language model with a novel fine tuning approach that enriches these epidemiological thematic features. We find these thematic features rich enough to improve epidemiological document classification over a smaller data set than initially used in PADI-Web classifier. This improves the classifiers ability to avoid false positive alerts on disease surveillance systems. To further understand information encoded in EpidBioBERT, we experiment the impact of each epidemiology thematic feature on the classifier under ablation studies. We compare our biomedical pre-trained approach with a general language model based model finding that thematic feature embeddings pre-trained on general English documents are not rich enough for epidemiology classification task. Our model achieves an F1-score of 95.5% over an unseen test set, with an improvement of +5.5 points on F1-Score on the PADI-Web classifier with nearly half the training data set.

Introduction

In recent years, with the rise of infectious disease outbreaks, more focus has been put on epidemic intelligence and biosurveillance intelligent systems. These epidemiological surveillance systems have gained track within the natural language processing community. Such systems are able to monitor official intergovernmental digital sources as well as unofficial sources such as unstructured web news articles for early detection and reporting of existing, reemerging and novel disease outbreaks [START_REF] Woodall | Global surveillance of emerging diseases: the promed-mail perspective[END_REF][START_REF] Arsevska | Web monitoring of emerging animal infectious diseases integrated in the french animal health epidemic intelligence system[END_REF][START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF]. Early disease surveillance systems generally used an indicator based approach that uses formal rule systems to monitor relevant official sources [START_REF] Paquet | Epidemic intelligence: a new framework for strengthening disease surveillance in europe[END_REF]. Current surveillance systems such as ProMED, HealthMap and the Platform for Automated extraction of Animal Disease Information from the web PADI-Web, use an event based approach with multiple different corpora and language sources [START_REF] Woodall | Global surveillance of emerging diseases: the promed-mail perspective[END_REF][START_REF] Brownstein | Healthmap: the development of automated real-time internet surveillance for epidemic intelligence[END_REF][START_REF] Arsevska | Web monitoring of emerging animal infectious diseases integrated in the french animal health epidemic intelligence system[END_REF]. PADI-Web is an event based biosurveillance system developed for the French Epidemic Intelligence System (FEIS) focused on monitoring online news sources for detection and alerting of existing and emerging infectious animal diseases (Valentin et al., 2020a;[START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF]. PADI-Web 1.0 by [START_REF] Arsevska | Web monitoring of emerging animal infectious diseases integrated in the french animal health epidemic intelligence system[END_REF] used a keyword-based classification approach where corpora are classified based on existence of one or more preset list of disease outbreak-related keywords in the document. This classification process is then further enhanced in PADI-Web 2.0 [START_REF] Valentin | PADI-web: An Event-Based Surveillance System for Detecting, Classifying and Processing Online News[END_REF] by incorporating a multilingual module and machine learning techniques based on bag-of-words and Term Frequency -Inverse Document Frequency (TF-IDF) approaches [START_REF] Luhn | A statistical approach to mechanized encoding and searching of literary information[END_REF][START_REF] Jones | A statistical interpretation of term specificity and its application in retrieval[END_REF]. Later PADI-Web 3.0, [START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF], proposes a finegrained classification of sentences in order to identify specific classes (e.g. Descriptive epidemiology, Preventive and control measures, Economic and political consequences, etc.).

Even though epidemic intelligence has grown with the introduction of event based epidemiology surveillance systems, major challenges include their reliance on labeled data sets for supervised learning training. Labeling such data is relatively costly and time consuming. In order to train an epidemiology document classifier, human experts have to manually label unstructured news articles as either relevant, related or irrelevant for further disease surveillance processing [START_REF] Paquet | Epidemic intelligence: a new framework for strengthening disease surveillance in europe[END_REF]. Relevant corpora are those news articles that describe an infectious animal disease outbreak event, irrelevant corpora are those which are not related to dis-ease outbreak, and related corpora contain a different subject which relates to disease outbreaks [START_REF] Arsevska | Web monitoring of emerging animal infectious diseases integrated in the french animal health epidemic intelligence system[END_REF]. Beyond machine learning approaches of building language understanding models, there has been great advances in deep learning techniques based on word embeddings and language models. Word Embeddings represent the semantic meaning of lexicon encoded as vectors. Various successful models have been developed for this task.

This begun with context-insensitive embeddings; Word2Vec proposed by [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF], Global Vectors for Word Representation (GloVe) Embeddings by [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF]FastText (Bojanowski et al., 2016). The works of [START_REF] Peters | Deep contextualized word representations[END_REF], introduced models of encoding contextual knowledge into word embeddings, models which have now been far improved through the use of Transformer networks in place of both Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) Networks [START_REF] Vaswani | Attention is all you need[END_REF]. This study aimed at developing a new thematic embedding based approach for epidemiological corpus classification over tagged news article sources. The classifier improves over current keyword based and machine learning based approaches. Our epidemiology document classifier learns rich thematic embeddings to discriminates between relevant and irrelevant news sources for further disease surveillance in PADI-Web system. The contributions of this study are as follows:

• Our study proposes a new way to improve epidemiological document classification by enriching epidemiological thematic features mainly disease, host, location and date contained in animal disease news articles that forms our train set corpus. We achieve this by proposing Epid-BioBERT model whose architecture is two staged; pre-trained on BioBERT [START_REF] Lee | BioBERT: a pre-trained biomedical language representation model for biomedical text mining[END_REF] language model that we use to learn thematic embeddings for our features and fine-tuned to learn a classifier model that discriminates between relevant and irrelevant news corpora for epidemic intelligence tasks.

• We show that fine tuning a biomedical language model improves epidemiological corpus classifier more significantly than fine tuning a general purpose pre-trained language model such as BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF].

• We experiment the impact of each individual thematic feature in the overall epidemiological classifier showing that both host and disease thematic features carry vital information on corpus relevance for epidemic intelligence.

• Our model achieves State-Of-The-Art performance with a smaller data set without using thesaurii to enrich thematic features1 .

• We improve over false positive alerts in epidemiology surveillance brought about by missclassifying news articles that mention disease-free countries and those describing aftermaths of a disease outbreak.

The rest of the paper is organized as follows: Section 2 outlines related works majorly reviewing PADI-Web document classifier and our major contributions; Section 3 introduces our model its architecture and pipeline; Section 4 discusses empirical results from experiments comparing the baselines with our model; Section 5 presents ablation studies findings on key model and data aspects and Section 6 summarizes the entire paper.

Background

PADI-Web Relevance Classification. This work is majorly an extension of the epidemiology document classification by [START_REF] Arsevska | Web monitoring of emerging animal infectious diseases integrated in the french animal health epidemic intelligence system[END_REF][START_REF] Valentin | PADI-web: An Event-Based Surveillance System for Detecting, Classifying and Processing Online News[END_REF] and [START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF]. The PADI-Web classifier takes two approaches. Firstly disease corpus is tagged using predetermined list of disease outbreak related keywords. In the second approach, a more accurate supervised binary relevance classifier that takes manually tagged news articles encoded using TF-IDF as inputs is developed. A bag-of-words representation combined with machine learning techniques are implemented in order to identify relevant and irrelevant documents. As an extension to improve document classification, PADI-Web further incorporates a more finegrained topic classification task to refine article relevance. This helps set priority for news articles that declare disease outbreak as opposed to those that describe outbreak consequences or aftermaths such as the economic effects caused by infectious disease outbreak.

A more recent approach focuses on a fine-grained classification of sentences dealing with animal disease surveillance based on Naive Bayes, Random Forest and Surpoort Vector Machines (SVM) algorithms [START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF]. Our approach proposes a pre-trained BioBERT language model to enrich such epidemiological information to extend this work. These learnt epidemiological thematic embeddings form the thematic features that are fed into our fine tuned model. BioMedical Language Model Fine Tuning. As demonstrated by Broscheit (2020) large pre-trained language models such as ELMo and BERT contain entity linking knowledge within their trained architectures [START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF]. Other research has looked into such large pre-trained models for specific domains. For instance, in the Biomedical domain, [START_REF] Lee | BioBERT: a pre-trained biomedical language representation model for biomedical text mining[END_REF] introduced BioBERT, an original BERT architecture trained over biomedical corpora; other biomedical language models such as ClinincalBERT and BioELECTRA have been proposed in recent years [START_REF] Huang | Clinicalbert: Modeling clinical notes and predicting hospital readmission[END_REF][START_REF] Kanakarajan | BioELECTRA:pretrained biomedical text encoder using discriminators[END_REF]. Original ways to fine tune such

Embeddings for Thematic Sentences

Add and Norm

Pointwise Feed-Forward

Add and Norm

Multi-Head Self Attention ...

... E 1 E M Thematic Tokens ⊕ Positional Embeddings [CLS]T hemT ok 1 1 , .., T hemT ok N M [EOS] σ(z ij ) = e z ij j e z j W T * X + B T Drop Out Norm Max Pool {c 1 , c 2 } T hemEmb 1 1 , .., T hemEmb N M Figure 1:
EpidBioBERT Transformer Architecture with fine tuned deep layers on top of pretrained BioBERT.

[CLS]T hemT ok 1 1 , .., T hemT ok N M [EOS] are the N thematic feature tokens from M sentences in the annotated train corpus that are inputs to the model. [CLS] and [EOS] are the tokenizers' tags for start and end of sentence respectively. A probability distribution over document classes relevant and irrelevant represented as {c 1 , c 2 } are the output labels. large pre-trained models can then be defined, in order to improve their effectiveness in various biomedical related NLP tasks, e.g., epidemiological surveillance document classification. Recently, [START_REF] Ruder | Recent Advances in Language Model Fine-tuning[END_REF] has investigated different fine tuning techniques to improve such domain specific tasks. Our technique uses the behavioural fine tuning approach over PADI-Web dataset, with the aim of enriching thematic features.

Behavioural Fine Tuning. This technique, as demonstrated by [START_REF] Ruder | Recent Advances in Language Model Fine-tuning[END_REF], adopts an intermediate task approach in its training pipeline. The pre-trained language model is first trained end-to-end on the intermediate task in order to improve over a related specific task. The work in [START_REF] Phang | Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks[END_REF] shows that this approach is beneficial for the downstream tasks that depend on language understanding. [START_REF] Sun | How to fine-tune bert for text classification?[END_REF] showed how pre-trained language models benefits from few shot learning for tasks that have smaller training data sets. In this study, we extend PADI-Web classifier using a behavioural few shot learning approach. In our approach, BioBERT language model is first fine tuned for our epidemiology task, then used to enrich epidemiology thematic features with a focus of improving disease corpora classification.

EpidBioBERT

In this section, we introduce our epidemiological corpus classifier model (EpidBioBERT). EpidBioBERT adopts a two step framework transfer learning approach using a pre-trained Biomedical Language Model followed by fine tuning to improve epidemiological document classification in PADI-Web disease surveillance system.

Model Architecture

EpidBioBERT model architecture uses a base BioMedical Language model (i.e., BioBERT) with fine tuned disease surveillance deep layers above its architecture as described in Fig. 1. BioBERT is a deep self-attention model pretrained over biomedical corpora achieving State-of-The-Art levels in biomedical text mining on BLURB leader-board tasks [START_REF] Gu | Domain-specific language model pretraining for biomedical natural language processing[END_REF]. Three pretrained versions of BioBERT are introduced in Lee et al.

(2019), namely BioBERT(+PubMed), BioBERT(+PMC) and BioBERT(+PubMed+PMC). These versions differ in architecture size since they are pretrained on different datasets. Our architecture is based on the BioBERT(+PubMed) model [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Lee | BioBERT: a pre-trained biomedical language representation model for biomedical text mining[END_REF].

[CLS]

T hemT ok In related research, [START_REF] Broscheit | Investigating entity knowledge in BERT with simple neural end-to-end entity linking[END_REF] fine tuned BERT for named entity recognition and showed that different layers of BERT contain different encoded information. For instance, the lower layers of BERT are rich in named entities information which are related to our epidemiological thematic features. In our approach, we first fine tune the whole BioBERT model by unfreezing all the weights and using the last state of the pretrained optimizer to train end to end on PADI-Web corpus. The second fine tuning stage tunes for classification similar to the baselines. We adopt a cross entropy loss objective function over the two target classes. Our model learns to maximize probability of the correct classes. The output is a document tag with a relevant/irrelevant label.

1 1 ... T hemT ok 1 N [EOS] T hemT ok 2 1 ... T hemT ok 2 N [EOS] ⃗ w [1]1 * 768 1 ... ⃗ w [1]1 * 768 128 ⃗ w [2]1 * 768 1 ... ⃗ w [2]1 * 768 128

Model Definition

Epidemiological document relevance classification task takes in a set of N disease outbreak news articles which we denote as D = {d 1 , ..., d N }. The task can formally be defined as: given a disease outbreak news article d j ∈ D which contains n epidemiological thematic features denoted F = {f 1 , ..., f n }, output a probability distribution classifying the article as either of the document classes C = {c 1 , c 2 } where c 1 = relevant, c 2 = irrelevant for epidemiological surveillance. Our model learns to maximize the probability p(c i |d j ) where c i ∈ C and d j ∈ D by minimizing the models' objective function:

L = 1 N b |C| i |N b | j -{y ij * ln σ(z ij )}
Where b is the batch size set as hypeparameter described in 4.2 and y ij is the true label vector for training input corpus d j signifying its true class label c i . z ij is the output from our last linear layer such that z j = W T * X + B T where W T is the layers' weight matrix that is multiplied with the epidemiology thematic features embedding matrix X learnt from the pre-trained model and added to the bias matrix B T . z ij is then softmaxed such that σ(z ij ) = e z ij j e z j followed by taking of natural logs to get predicted output vector ŷij = p(c|d j ), for c ∈ {c 1 , c 2 }. We achieve EpidBioBERT by adopting a deep pretrained transformer based model and deep fine tuned network layers as explained in Fig. 1.

EpidBioBERT Training Data

Our training corpora is derived from PADI-Web dataset which is supplied in JSON format [START_REF] Rabatel | PADI-web corpus: news manually labeled[END_REF]. We prepare our corpus from the original data set, by running a cascade of regex rules to extract epidemiology thematic features,their annotated labels and document gold labels.

Our regex cascade rules find 180 news articles (i.e., about 35%) tagged as relevant and 350 (i.e., about 65%) tagged as irrelevant. The articles contain epidemiological entities manually labeled by human experts. Each token considered as an epidemiological entity candidate is originally labelled as either location for location of disease outbreak, date for date of disease outbreak, number for the number of reported cases, disease for type of disease experienced in the outbreak, and host for the disease carrier species. We form our epidemiological thematic features by selecting disease, host, date and symptoms to enrich our classification, the number epidemiology feature is dropped.

In addition, these epidemiological entity candidate are labelled in PADI-Web data as, "correct", "partial" or "incorrect" with correct label signifying relevant candidate features for information extraction task while incorrect signifying irrelevant candidate features [START_REF] Arsevska | Web monitoring of emerging animal infectious diseases integrated in the french animal health epidemic intelligence system[END_REF]. For example having a date candidate feature that does not fit the date of disease event occurrence would have that date feature labelled irrelevant. There are 6K thematic features with 66% relevant, 20% irrelevant and 14% partial relevant as captured in Fig. 3. The resulting data set contains both relevant and irrelevant documents, which in turn contain epidemiology thematic features. For experimentation purposes, we use 60% of the documents for training, 20% as Held-Out corpus for hyperparameter tuning and the remaining 20% for Model performance Evaluation. We cross validate these splits for different K-Folds under Section 5.4, in order to understand how our approach generalizes to train datas' true distribution as the data set size increases and instances change.

Epidemiology Thematic Feature Engineering

To enable our model and baselines to take in significant inputs, the annotated corpus detailed in 3.3 is first preprocessed using NLTK library. The train corpus is first preprocessed by sentence segmentation to extract sentences from running text. This process is followed by tokenization to separate words from running sentences forming tokens while maintaining their annotated labels.

Thematic tokens for each document are then selected as per their annotated labels captured in Fig. 3. These thematic labeled tokens form our model inputs as thematic features. The thematic features are then vectorized using various strategies, depending on the baseline model (as described below), to form thematic embeddings having each thematic feature represented by a vector ⃗ w ∈ R 1 * |V | where |V | is the size of the vocabulary set.

For the bag-of-word approach, we initially set |V | = 30K and represent every thematic feature with a one hot vector ⃗ w such that ⃗ w ∈ R 1 * |30K| . Since this vector is sparse, we use truncated Singular Value Decomposition (truncated-SVD) setting the number of critical values to 300 to convert the thematic embeddings to dense vectors of dimension ⃗ w |1 * 300| .

For the TF-IDF approach, we also set |V | = 30K thus having thematic embeddings with the same dimension as initial bag-of-words approach. However, since these embeddings are not sparse, we use Principal Component Analysis (PCA) setting number of principal components to 300 for dimensionality reduction thus we end up with thematic embeddings of dimension ⃗ w |1 * 300| . For the pretrained approach, we use pretrained GloVe embeddings [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF] of the same 1 * 300 dimension for our thematic features.

For our transfer learning approach, we first tokenize using the pretrained BioBERT word piece tokenizer, then learn thematic feature embeddings from pre-trained BioBERT transformer encoder architecture as reported in Fig.

2.

Each document in the data set is embedded by a tensor of dimension (no of sentences, sequence len, |V |) where no of sentences represents sentence count in the document, having set sequence len = 128 initially and |V | = 768 from the pretrained architecture. Each thematic feature is embedded by a ⃗ w such that ⃗ w ∈ R 1 * |768| . Furthermore, we experiment with different embedding dimensions to see how this improves our epidemiology corpora classification task in Section 5.

For the baseline model, words contained in a given news article form the feature set for each document, with unique words forming the Vocabulary set |V |. We experiment with all the above embeddings beginning with One Hot Encoded thematic vectors for the bag-of-words Approach. On the Transfer learning approach, pre-trained thematic embeddings form the features for each document. We experiment using pretrained GloVe embeddings. In the deep learning approach, we adopt pre-trained BioBert thematic embeddings after which we compare classification results presented in Section 4.

Experiments

In this section we compare and report EpidBioBERT results with machine learning classifiers recently used by [START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF] over PADI-Web system.

Baselines

We first experiment with the bag-of-words approach having One Hot encoded (OHE) epidemiology thematic feature embeddings. We train an SVM classifier model boosted with Gaussian Kernel (SVM+OHE) with the thematic embeddings as input features and the document class labels relevant and irrelevant as targets. We then experiment beyond the bag-of-words approach using both TF-IDF (SVM+TF-IDF) and pre-trained GloVe thematic embeddings (SVM+GloVe) over the same SVM model. We then experiment further by training an LSTM classifier model with the GloVe thematic embeddings first by freezing the embedding layer (LSTM+GloVe f rozen ) and then by training an end to end classifier (LSTM+GloVe unf rozen ). We also train a similar Bidirectional LSTM model (Bi-LSTM+GloVe unf rozen ) and compare performance of all these baselines in Table 1.

Hyperparameters

We fine tune BioBERT(+PubMed) model [START_REF] Lee | BioBERT: a pre-trained biomedical language representation model for biomedical text mining[END_REF] with hidden embedding size of 768, 12 Attention Heads and 12 Transformer blocks [START_REF] Vaswani | Attention is all you need[END_REF]. We set a small batch size of 16 and a sequence length of 128 and experiment with 50 epochs. For our higher fine tuned layers, we experiment with dropout rates of 0.2,0.3 and 0.4 to control model overfitting. We adopt Adam optimizer with decoupled weight decay (AdamW) [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] with β 1 = 0.9 and β 2 = 0.999, we set ϵ =1e-8 and weight decay = 0.01. We also set small initial learning rates of 1e-5 and 2e-5 with a higher number of epochs to favour our fine tuning approach [START_REF] Ruder | Recent Advances in Language Model Fine-tuning[END_REF]. We then evaluate and save the best model over the held out corpus.

Model

F

Results and Discussions

In this section we report and discuss baseline performance as compared to EpidBioBERT.

Performance Metrics

We evaluate our model by comparing it with all the baseline models. We compute Precision, Recall, F1 Score and Accuracy performances for all classifiers.

The models are evaluated on their effectiveness to classify documents as either relevant or irrelevant, by comparing model-predicted versus true document labels as explained in Section 3.2 over an unseen test set.

Results

In Table 1, we compare and report EpidBioBERT results with machine learning classifiers recently used by [START_REF] Valentin | Padi-web 3.0: A new framework for extracting and disseminating fine-grained information from the news for animal disease surveillance[END_REF] over PADI-Web system. The difference between SVM+OHE and SVM+TF-IDF is that the latter takes TF-IDF thematic embeddings as inputs while the former takes one hot thematic embeddings as input features. We also train LSTM based models with GloVe thematic features as inputs to enhance baseline performance. We freeze the GloVe pretrained thematic embeddings in LSTM+GloVe f rozen model but train end-to-end in LSTM+GloVe unf rozen .

Discussions

Results recorded on both SVM+OHE and SVM+TF-IDF baseline models show that they commit a lot of false negative errors leading to low recall scores. This is attributed to the low quality of both one hotted and TF-IDF epidemiology thematic embeddings. Such embeddings encode limited thematic information required for correct discrimination between relevant and irrelevant documents by the epidemiological classifier. We note a recall score improvement between 33 and 38 points when we use pre-trained GloVe embeddings on the same SVM baseline model.

Even though pre-trained thematic feature embeddings significantly improves recall and overall F1 scores we note that this largely depends on the nature of pretraining. Epidemiological thematic embeddings pretrained on general English corpora prove not rich enough for robust epidemiology feature classification task (we study this further in Section 5.1 by comparing BERT to BioBERT as a base pretrained model for our classification task). This is the case with 5 to 12 points accuracy drop in SVM+GloVe model given the nature of GloVe embeddings [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF]. We attribute this largely due to the fact that the number of Out of Vocabulary (OOV) thematic tokens unseen during pre-training are high since the thematic embeddings are not pretrained on Bio Medical data. This causes features like diseases, host and location to largely fall under OOV. In Section 5.3 we investigate if this can be improved with better generalization and higher dimensional thematic embeddings.

LSTM based baselines improve performance using GloVe thematic embedding features as inputs.

Bi-LSTM+GloVe unf rozen achieves the best baseline scores. We attribute this first to the nature of end-toend training that allows the model to learn epidemiology thematic feature embeddings from the PADI-Web train set data, and second to the Bi-directional [START_REF] Melamud | context2vec: Learning generic context embedding with bidirectional lstm[END_REF] nature of the Bi-LSTM model that enables it to learn both left and right contexts of thematic embeddings enriching them further than unidirectional LSTM+GloVe architecture.

EpidBioBERT outperforms Bi-LSTM+GloVe unf rozen on precision, recall, F1 and accuracy scores. This we attribute to a number of key architectural benefits that EpidBioBERT has over all the baselines. First the underlying pre-trained model BioBERT uses transformer [START_REF] Vaswani | Attention is all you need[END_REF] architecture which has better surrounding-features context management Attention technique that surpasses LSTM architectures. Secondly, BioBERT is based on BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] which is a bidirectional architecture thus maintaining the bidirectionality advantages aforementioned. Thirdly, we fine tune BioBERT(+PubMed) that is pretrained on biomedical data, this enriches our epidemiological thematic features beyond capabilities of a general purpose language model. Lastly, EpidBioBERT novel fine tuning technique uses network layers and hyperparameters that favour few shot learning that coun- ters our small and class imbalanced train set [START_REF] Sun | How to fine-tune bert for text classification?[END_REF][START_REF] Ruder | Recent Advances in Language Model Fine-tuning[END_REF]. We study further the effect of small train and test sets in EpidBioBERT and selected baselines in Section 5.4 by use of cross validated experiments.

Discussion: Ablation Studies

In this section we present the findings on ablation experiments, carried out to understand the independent effects of quality of thematic features and data size on our model.

Effect of Biomedical Pretraining

To test if biomedical pretraining improves our epidemiological document classification task, we evaluate our model alongside a BERT based model pretrained on general English corpora. To set a fair comparison, we fine tune distill-bert model [START_REF] Sanh | Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter[END_REF] for our classification task setting hyperparameters to be the same as those used in EpidBioBERT described in 4.2.

We present this experiment results in Fig. 4 showing the training loss over 50 epochs. We report on the training loss metric observing that the loss decreases much faster and more significantly for EpidBioBERT as compared to the BERT based model. This shows that BioBERT significantly improves training of our epidemiological classifier largely because BioBERT is pretrained on biomedical corpora thus enriching epidemiological thematic features as opposed to BERT.

Effect of Thematic Feature Enrichment

To understand how much impact each thematic feature has in our epidemiology corpus classifier model, we experiment by running EpidBioBERT model on different data sets each having one epidemiology thematic feature dropped. We focus on four thematic features namely Disease, Host, Location and Date and prepare four train, validation and test sets. We present results in Table 2 showing the F1-Score, precision, recall and accuracy drops as compared to EpidBioBERT model results in Table 1.

The Host thematic feature causes the highest drop in classifier accuracy and F-Score of -9. We find this interesting since the distribution of thematic features in the data follows a different distribution order with Disease, Location, Host and Date, in that order, having high representations as presented in Fig. 3.

Effect of High Dimension Thematic Embeddings on Baselines

We explore the effects of having deeper embeddings as inputs to our model. High dimensional word embeddings can be thought as being rich since they contain more dimensions to carry context information. We start by training different versions of SVM+OHE model with increasing OHE embedding dimensions and report results in Table 4. We find that for the bag-of-words based approach, increasing embedding dimension alone does not necessarily improve performance as seen between the baseline models SVM+OHE 50dims , SVM+OHE 100dims and SVM+OHE 300dims . However, beyond the bag-ofwords approach performance can slightly be improved in this way as shown between LSTM+TF-IDF 100dims LSTM+TF-IDF 300dims . We achieve a competitive classifier using LSTM+GloVe 300dims .

EpidBioBERT+CV [2,3,4,5,6,7,8,[START_REF]Language Resource References[END_REF]10] and run our model for 5 epochs using the same architecture and hyperparameters of EpidBioBERT from 2-fold up to 10-fold cross validations. To compare cross validation effect, we also train the cross validated SVM-based baseline versions (SVM+OHE+CV, SVM+GloVe+CV, SVM+TF-IDF+CV) over the same values of K and present average F-Score per fold measures over an unseen test set in Fig. 5. The results show that higher folds of cross validation improves EpidBioBERT though with a small margin. We report EpidBioBERT+CV standard deviations over different folds in Table 3. The results show the slight variation of both accuracy and F-measure when the data is changed from fold to fold. However we observe that the SVM baselines SVM+OHE+CV and SVM+TF-IDF+CV improve beyond SVM+GloVe+CV for higher K-Fold values. This highlights the ability of EpidBioBERT to learn a lot from a small epidemiology data set as compared to the baseline models as demonstrated by smaller deviations that uniformly holds in all

Conclusion and Future Work

This paper presents an epidemiological document tagger EpidBioBERT over the infectious animal disease biosurveillance system; PADI-Web. Our contribution improves the classification accuracy by enriching epidemiological thematic features in disease news articles using transfer learning approach on knowledge contained in BioBERT. We found out that biomedical language models contains encoded knowledge for enriching and improving disease surveillance systems. We however also found out that there is limited annotated news corpora for this task. As future work, we propose an unsupervised pipeline that can take in unlabelled news articles which are much available. To further improve epidemiological relevance corpus classification, we propose an integration of semantic knowledge from relevant sources such as Medical Subject Headings (Mesh) to further enrich thematic features. We also propose to research further on individual impact of thematic features in disease surveillance systems.

Acknowledgements

This study was partially funded by EU grant 874850 MOOD and is catalogued as MOOD032. The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the views of the European Commission.

Figure 2 :

 2 Figure 2: EpidBioBERT input representation. The Thematic embeddings ⃗ w are enriched by BioBERT embeddings from word piece tokenized annotated tokens from training copora.

Figure 3 :

 3 Figure 3: Epidemiology Thematic Feature Distribution in PADI-Web Dataset.

Figure 4 :

 4 Figure 4: Train Loss scores of EpidBioBERT compared to BERT-FT for epidemiology document classification.

  4 and -8 respectively. Training the classifier without Date thematic information reduces models recall value increasing false negative classification error. Location, Disease and Host thematic features roughly contain the same information to equally influence recall measure meaning less false negatives. The results signifies that both Host and Disease are the key thematic features influencing our classifier followed by Date and lastly Location.

Figure

  Figure 5: Cross Validated Models

Table 1 :

 1 1 Score P recision Recall Accuracy Performance of Our Model in PADI-Web Epidemiology Feature Extraction. One hot encoded based model are denoted OHE. Best scores are in bold .

	Baselines
	SVM+OHE	0.29	1.0	0.17	70.0
	SVM+TF-IDF	0.35	0.83	0.22	77.12
	SVM+GloVe	0.51	0.65	0.55	65.34
	LSTM+GloVe f rozen	0.84	0.84	0.85	86.13
	LSTM+GloVe unf rozen	0.85	0.85	0.85	87.12
	Bi-LSTM+GloVe unf rozen	0.86	0.89	0.85	88.11
	Ours
	EpidBioBERT	0.95	0.97	0.94	95.8

Table 2 :

 2 Impact of each Thematic Feature on our classifier performance as captured during EpidBioBERT training and testing. Features with the most information have cause highest drop in both F-Score and Accuracy as shown in bold.

	Thematic Feature F 1 ScoreDrop Precision Drop Recall Drop Accuracy Drop
	Date	-4	-3	-5	-4.8
	Location	-1	-6	+2	-2
	Host	-8	-18	+2	-9.4
	Disease	-5	-12	+2	-5.8
	All Features	0.0	0.0	0.0	0.0

Table 3 :

 3 folds Avrg Acc Acc Std Dev Avrg F-Score F-Score Std Dev EpidBioBERT+CV K-Folds Test-set Average Accuracy, F1 Scores and their Standard Deviations showing folds 2 upto 7. The model demonstrates a small deviation uniformly among the data folds. The best scores are in bold having considered ones with smaller deviations.

	2		0.6670	±0.0465	0.8083	±0.0583
	3		0.8024	±0.0534	0.8685	±0.0464
	4		0.7131	±0.028	0.8680	±0.0542
	5		0.8492	±0.0398	0.8239	±0.0393
	6		0.8139	±0.0368	0.8167	±0.0423
	7		0.8447	±0.0352	0.8784	±0.0513
	Model	F 1 Score Accuracy
	OHE Thematic Features
	SVM+OHE 50dims	0.34	0.71
	SVM+OHE 100dims	0.31	0.62
	SVM+OHE 300dims	0.29	0.70
	TF-IDF Thematic Features
	SVM+TF-IDF 50dims	0.30	0.71
	LSTM+TF-IDF 100dims	0.31	0.72
	LSTM+TF-IDF 300dims	0.35	0.77
	GloVe Thematic Features
	SVM+GloVe 300dims	0.64	0.69
	LSTM+GloVe 300dims	0.74	0.75

Table 4 :

 4 Effect of Increasing Thematic Embedding Dimensions5.4. Effect of Epidemiology Data SizeTo understand how increase in training data size would impact EpidBioBERT model, we train the cross validated model version (EpidBioBERT+CV) over a series of K-Fold cross validated data set. We set the range values of K =

https://github.com/menya-edmond/EpidBioBERT


Zacharie Guillaume
email: zacharie.guillaume@devinci.fr

Théo Richalet
email: theo.richalet@edu.devinci.fr

Marc Teyssier
email: marc.teyssier@devinci.fr

Sylvain Pauchet
email: sylvain.pauchet@enac.fr

Jérémie Garcia
email: jeremie.garcia@enac.fr

Benoit Roman
email: benoit.roman@espci.fr

Make-A-Morph: Exploring the design space of inflatable devices made from planar fabric

Keywords: Inflatable, Tangible Interface, Shape Changing, Computer Assisted Design (CAD), Digital Fabrication ETIS: European Tangible Interaction Studio, ETIS'22, Nov 7 -

Developing inflatable devices from planar fabric is a new versatile fabrication process that allows the development of complex geometric shapes with a beneficial mass to robustness ratio. However, designing and fabricating with this matter is complex, and the existing design primitives for shape change can constrain designers' creativity. We present a pipeline that allows users and designers to explore and compose with various shape-change primitives. To this extent, we rely on digital simulation combined with a simple digital fabrication tool. This pipeline allows to explore and visualize deformation and develop new application cases for shape-changing interfaces. We propose a workshop around manipulating these tools to foster discussion between designers and researchers around the future of shape-changing interface fabrication.

Introduction

Previous works on shape-changing interfaces have demonstrated that inflatable devices made from planar fabric are doted of various advantageous features, from robustness [START_REF] Sareen | Printflatables: printing human-scale, functional and dynamic inflatable objects[END_REF] to versatility [START_REF] Ou | aeromorph-heatsealing inflatable shape-change materials for interaction design[END_REF][START_REF] Siéfert | Programming stiff inflatable shells from planar patterned fabrics[END_REF]. The principle in those examples is to seal (most of the time using heat-sealing technology) two sheets of airproof fabric in a specific pattern to obtain a specific deformation when inflated. In aeronautics, where heavy mechanical devices are traditionally used [START_REF] Cadogan | Inflatable and rigidizable wings for unmanned aerial vehicles[END_REF], portability can be used as an advantage: deflated devices can be rolled up and are not submitted to breakage as other extra-light materials such as Styrofoam or aerogel silica can. It is also relevant in interaction since shape-changing interfaces can offer new dimensions for tangible interaction by providing dedicated form constraints or specific feedback.

Programming the controlled deformation of an inflatable shape-changing interface remains challenging. As of now, the design of such devices is complex, and the design primitives defined in previous works [START_REF] Ou | aeromorph-heatsealing inflatable shape-change materials for interaction design[END_REF][START_REF] Siéfert | Programming stiff inflatable shells from planar patterned fabrics[END_REF] exploring shape-changing behavior are either created with vast knowledge in mathematics and material science or rely on a long trial-and-error process. Furthermore, the fabrication is often performed manually, involves iterations, and is challenging to reproduce. We developed a pipeline that addresses these challenges. It relies on a modeling software simulating the result of any given sealing pattern and generating the files necessary to realize it depending on the chosen fabrication method. The fabrication can be done with equipment accessible in most maker spaces, such as soldering iron or laser plotter, as presented in [START_REF] Ou | aeromorph-heatsealing inflatable shape-change materials for interaction design[END_REF]. Moreover, it does not require complex assembly like screws or nails and uses off-the-shelves or up-cycled materials from nylon fabric to chips bag [START_REF] Choi | Therms-up!: Diy inflatables and interactive materials by upcycling wasted thermoplastic bags[END_REF].

We propose to conduct a workshop that explores this technique. We will rely on a set of computational tools that enable users to experiment with a set of primitives and allow easy fabrication. This exploration allows the creation of custom patterns and variations in a free-form workspace and simulates the resulting deformation.

We argue that manual fabrication is a great way to be familiar with the design opportunities, constraints, and capabilities of a process or a fabrication method. The users will manipulate and play with the identified primitives on pre-sealed pocked (Figure 1). Through this workshop, we aim to understand better this technique's opportunities and limitations and foster discussion among the HCI researchers.

Workshop

This workshop is part of a more extensive study aiming at developing new tools for creating shape-changing devices. It intends to deepen our comprehension of the previous results con-cerning the limitation of the fabrication technique, the process, and the expected outcome and to gather valuable feedback from the HCI community.

In this workshop, we will explore shapes and functions developed by novice users in shapechange. As such, the goal is to:

• have finished shape • having feature oriented device • share knowledge in fabrication process among participants • explore and discuss a new tool with participants

Steps

The Workshop will be articulated in the following phases:

Participant

This workshop aims to host between 8 to 15 participants to cover comprehensive application cases while offering each one the possibility to experiment with the technique manually. No knowledge in fabrication processes is needed. Depending on attendance, the participant will be paired in groups of two to three.

Apparatus

Digital tools and materials used will be brought by the workshop organizer. It consists of soldering irons for the hands-on manipulation time and a custom-built machine to draw more accurate shapes and materials in the form of:

• Pre-made pocket to explore manually the design primitives • various material from nylon fabric TPU coated to mylar and other up-cycled materials from diverse sources.

Participants are required to bring their computer with a running rhino 7 software [START_REF] Mcneel | Rhino 3d modeling software[END_REF] the free demo version is sufficient.

Results

The thirteen participants were divided into four groups of three to four. They were introduced to the design principle behind inflatable and left free to discuss what kind of device they wanted to develop during the workshop.

Intention definition

Each group came up with ideas exploiting different particularities of inflatable devices.

• One active scarecrow monster: a device that deploys violently to scare birds or other incommodities using inflation-induced movement. • One inflatable cushion: an inflatable cushion that can deploy everywhere, taking advantage of the device's two states (inflated and deflated). • One breathing/distress aid: the group used the slowness of the airflow as an advantage to press and relax while the user feel the air pass through a tight seam from one pouch to another. • Exploration of the textures changes through inflation. The group focused on the change in shape and feel of the material when inflated.

Digital and physical prototyping

The participants reported an interest in simulating and visualizing their prototype on a computer however it was pulled back by the work-in-progress state of the software.

On the other hand, the physical manipulation did not come as a difficulty for the participants, who rapidly used their newly developed knowledge of the material to realize the prototype of their intended design.

Testing

At the same time as the physical prototyping part, the participants took on themselves to try and test their prototype, evaluating the success of their first intention. They graded their results on an average of three and a half on five, with nine out of ten evaluating it at three or higher.

Future works

Previous workshops were preliminary works oriented toward the field of aeronautics. In contrast, this one aims to extend it to a larger audience and explore the synergy possible with other fields. The goal was to grasp opportunities for HCI tangible interaction. Results of these workshops may have relevance in future paper publications.

Figure 1 :

 1 Figure 1: Figure 1: Pre-sealed pocket used to start the exploration with a soldering iron

•

  Introduction to the workshop, the design principle behind baromorphic object and problem statement (20 min) • Group creation (ensure mixed background) and theme choice (10 min) • Hands on manipulation, application of the design principle and first iterations (45 min) • First feedback break and discussion (10 min) • Introduction to the software (10 min) • Conception and Iteration through the software (45 min) • Second feedback break and discussion (10 min) • Presentation of our pipeline to produce inflatables from start to finish (10 min) • Fabrication and test of each group concept (60 min) • Feedback and open talk about possible future application of the technology (20 min)

Figure 2 :

 2 Figure 2: Prototyping of the design ideas

Figure 3 :

 3 Figure 3: Participants testing their design ideas


Xiaojiao Yuan

Neus Sunyer-Pons

Aleix Terrado

José Luis León

Georges Hadziioannou

Eric Cloutet

Katherine Villa

3D-Printed Organic Conjugated Trimer for Visible-Light-Driven Photocatalytic Applications

Keywords: Organic semiconductors, Conjugated trimer, 3D-printing, Photocatalysis, Hydrogen generation

Small molecules organic semiconductors (SMOS) have emerged as a new class of photocatalysts that exhibit visible light absorption, tunable band gap, good dispersion, and solubility. However, the recovery and reusability of such SMOS in multiple photocatalytic reactions is challenging. In this work, we report a 3D-printed hierarchical porous structure based on an organic conjugated trimer (4,7-Bis(thieno[3,4-b] [1,4] dioxin-5-yl) benzo[c] [1,2,5] thiadiazol, EBE). Upon manufacturing, the photophysical and chemical properties of the organic semiconductor were maintained. Interestingly, the 3D-printed EBE photocatalyst showed a longer lifetime (11.7 ns) compared to the powder state EBE, indicating microenvironment effect of solvent (acetone) and better dispersion, as well as reduced intermolecular 𝜋-𝜋 stacking, resulting in improved separation of charge carriers. As a proof-of-concept, we evaluated the photocatalytic activity of the 3D-printed EBE catalyst for water treatment and hydrogen production under sun-like irradiation. The resulting degradation efficiencies and hydrogen generation rates were higher than the ones reported for the state-of-the-art 3D-printed photocatalytic structures based on inorganic semiconductors. The photocatalytic mechanism was further investigated, and the results suggested that hydroxyl radicals (HO ⦁ ) are the main radicals responsible for the degradation of an organic pollutant. Besides, the reusability of the EBE-3D photocatalyst was demonstrated in up to 5 recycling uses. Overall, these results provide the great potential of a novel 3Dprinted organic conjugated trimer for photocatalysis applications.

INTRODUCTION

Solar-driven photocatalysis has the potential to alleviate current environmental pollution and energy shortage by using a simple setup, water, sunlight, and efficient photocatalytic materials.

Recently, organic π-conjugated polymers (CPs) have been widely investigated as promising alternatives to traditional inorganic semiconductors for photocatalysis, due to their tunable optoelectronic properties, low cost, and high chemical stability. [1][2][3][4][5][6][7] However, poor dispersion and solubility in reaction media limit their overall practical applications.

Small molecules organic semiconductors (SMOS) possess distinct advantages compared with CPs, such as well-defined chemical structures with tunable structures, easy purification, good solubility, and no batch-to-batch variations. Owing to these unique properties, SMOS have been widely explored in organic photovoltaics, organic light-emitting diodes (OLEDs), solar cells, and organic thin-film transistors. [8][9][10][11][12][13] However, their use in photocatalytic applications is still rare. A recent work reported that conjugated thiophene-based trimers (Th-BT-Th) with donor-acceptor-donor (D-A-D) units can potentially be used for photocatalytic intermolecular C-H functionalization under visible light irradiation. The efficiency of Th-BT-Th was comparable to the state-of-art transition metal-based photocatalysts, such as [Ru(bpy)3]Cl2. 14 Therefore, this work opened the door for a new class of metalfree and visible light-active catalysts, based on small molecule organic semiconductors, for photochemical organic reactions. Besides, C. M. Aitchison et al. reported that different oligomers with D-D, A-A, D-A-D, and A-A-A present comparable photocatalytic H2 production rates to the corresponding polymer. Thus, it demonstrated that high molar masses are not required for displaying high activity, while backbone planarity, bridgehead atom, and chain length have a greater impact on the photocatalytic property. 15 Despite the superior advantages of SMOS compared to CPs and inorganic semiconductors, they require prior dissolution in organic solvents before the photocatalytic reaction, which complicates their overall recovery and reusability. 16,17 . Moreover, not all the powder dispersion, especially organic semiconductors, is equally exposed to the incident solar light due to the aggregation of powder and the limitations of the reactor design.

The merging of photocatalysis with cutting-edge technologies, such as three-dimensional (3D) printing, enables the design of supported photocatalysts with multiple shapes and versatile structures. Different 3D-printed photocatalysts have been reported, showing excellent photocatalytic activities for the degradation of organic pollutants and hydrogen generation. [18][19][20][21][22] The improved photocatalytic activity has been mainly attributed to better light harvesting by the hierarchical structure and the high surface area associated with the porous photocatalytic network. Therefore, 3D printing facilitates the manufacturing of photoactive materials to develop robust photocatalytic systems with easy reusability without requiring any additional filtering. 20,[23][24][25] Up to date, inorganic semiconductors have been mainly used for designing 3D-printed photocatalysts with applications in energy and environmental remediation. 26 However, the lack of solubility of such materials in organic solvents results in an inhomogeneous dispersion of the photocatalytic nanoparticles in the thermoplastics that complicates the whole fabrication and extrusion process. Here, we present, for the first time, the design of a 3D-printed hierarchical porous structure, containing an organic photoactive molecule, based on a conjugated trimer with a D-A-D configuration (EBE), for the photocatalytic degradation of persistent organic pollutants and hydrogen generation.

Because of the high solubility of the EBE molecule in organic solvents, the resulting filaments showed good malleability and homogeneity similar to the ones without photocatalysts. The characterization of the 3D-printed structure showed that the photophysical and chemical properties of the organic semiconductor were maintained upon the 3D printing processing. Moreover, despite the low amount of the photocatalyst (0.05 %), the 3D-printed EBE (EBE-3D) structure showed good photocatalytic efficiencies for the removal of methylene blue and 4-nitrophenol pollutants and hydrogen generation under sun-like irradiation. Therefore, this work opens new horizons for practical photocatalytic applications of 3D-printed organic semiconductors in the fields of environmental remediation and solar fuels.

EXPERIMENTAL SECTION

Reagents. 4,7-Dibromo-2,1,3-benzothiadiazole (95%), potassium carbonate (K2CO3, ≥99.99%), PEPPSI TM -IPr (98%), toluene (anhydrous, 99.8%), methanol (anhydrous, 99.8%), were purchased from Sigma-Aldrich and 3,4-Ethylenedioxythiophene-2-boronic acid pinacol ester was purchased from TCl. All the reagents were used as received. Deionized water (Milli-Q, 18.6 MΩ) was used throughout all experiments. Synthesis of EBE. EBE was synthesized via a Suzuki coupling reaction. Briefly, 4,7-Dibromo-2,1,3benzothiadiazole (400 mg) and 3,4-Ethylenedioxythiophene-2-boronic acid pinacol ester (800 mg) were dissolved in toluene/methanol (120:160 ml) in a flame-dried round-bottom flask. Argon was bubbled for 15 minutes in the solution to deaerate it. Then, K2CO3 (1.2 g) was added to the reaction in one portion and the mixture was heated to 40°C and stirred for 10 min. Then PEPPSI TM -IPr catalyst (46 mg) was added under argon flow and the flask was sealed. The temperature was increased to 75°C and the mixture was stirred for 4h. The reaction mixture was filtered, and the solid precipitate was washed with toluene and methanol and dried under a vacuum (Figure S1). 1 3D printing of the EBE molecule. The 3D EBE structure was fabricated by fused deposition modeling (FDM), due to its low cost and good materials compatibility. The initial step includes the fabrication of the filament by mixing the photocatalyst with melted thermoplastic pellets, followed by an extrusion/heating process. For this purpose, 30 g of acrylonitrile butadiene styrene (ABS, Smart materials pellets©) and 15 mg of EBE catalyst were dissolved in 120 mL of acetone. The solution was sonicated for 3h until a homogeneous mixture was obtained. Then, the ABS-EBE solution was heated at 80 ºC to evaporate the acetone and obtain a solid orange film. The film was cut into pieces to obtain ABS-EBE shavings to make the filament. A Felfil-Evo© extruder and Felfil-Spooler© were used to extrude the filament at 180 ºC with a 1.7 mm diameter. EBE catalyst was 3D-printed by using a Creality3D Ender 3 V2 3D Printer and the software Ultimaker Cura©. The 3D structure was designed with FreeCad© and Solid Works© software (Figure S2).

Characterization of the 3D-printed EBE structure and EBE powder. 1 H and 13 C NMR spectra were recorded at ambient temperature with a 400 MHz Bruker Avance Spectrometer with a 5 mm BBFO probe in the appropriate deuterated solvent. The contents of residual Pd in trimers were measured by inductively coupled plasma-mass spectrometry (ICP-MS). The morphology of the EBE-3D was investigated by scanning electron microscope Quanta 600 FEI (Thermofisher) and the EDX detector from Oxford instruments. Absorption spectra were collected by using a UV-vis spectrometer (Agilent Cary 60) with a wavelength range of 350~800 nm. Fluorimeter (Fluorolog-QM, Horiba) was used to measure the emission fluorescence spectra and the lifetime of the samples was detected by the Lifetime TCSPC configuration, under excitation from a NanoLED. The pulse (instrument response) was determined by measuring the light scatted by a Ludox suspension. The lifetime was calculated by fitting the decays with a monoexponential model: I(t) = I0 * 𝑒 -𝑡 𝜏 , where t is the time and I0 is the intensity at time zero. FTIR-ATR TR0 was used to investigate the functional groups present in the 3Dprinted EBE sample. Raman spectra were collected by using an FTIR-FTRaman Thermo Nicolet iS50.

X-ray diffraction (XRD) spectra of 3D EBE structure and EBE-pellet were obtained with a Bruker AXS D8-Discover diffractometer (40 kV and 40mA). Thermal analysis was conducted by using the Thermogravimetric balance (Mettler Toledo, TGA/SDTA851). X-ray photoelectron spectroscopy (XPS) experiments were performed in a PHI 5500 Multitechnique System (from Physical Electronics) with a monochromatic X-ray source (Aluminium Kalfa line of 1486.6 eV energy and 350 W), placed perpendicular to the analyzer axis and calibrated using the 3d5/2 line of Ag with a full width at half maximum (FWHM) of 0.8 eV.

Photocatalytic Test. The photocatalytic activity of the 3D-printed EBE structure was assessed for the degradation of organic pollutants and the generation of hydrogen. For this purpose, methylene blue (MB, 50 ppm) and 4-nitrophenol (4-NP, 50 ppm) were used as model pollutants. Before irradiation, the 3D-printed EBE catalyst was placed in the respective pollutant solution and kept in dark for 30 minutes to ensure adsorption-desorption equilibrium. Specifically, for the photocatalytic degradation, the 3D-printed EBE cube was placed inside 25 mL of MB or 4-NP solutions under sun-like light irradiation (Ultra-vitalux 230V 300W, OSRAM). Then, 1 mL of an MB or 4-NP solution was taken out every 30 min and 1 hour, respectively. UV-vis spectrometer was used to record the absorption of pollutants. The degradation rate of pollutants is expressed as (C0-C)/C0 *100% (C0-initial concentration; C-the concentration of taken samples). For hydrogen production, the 3D-printed catalyst was placed in 20 mL of N2 degassed (30 min) aqueous solution with 25 vol% of methanol as a hole scavenger (the total volume of the reactor was 50 mL). The sample was exposed under solar simulator (AM1.5) irradiation. Before turning on the light, the reactor was degassed by flowing N2 for 1h to remove the O2 in the system. A single-quadrupole mass spectrometry (MS, Omnistar TM GSD 320) connected to the reactor outlet was used to determine the amount of hydrogen generated online.

Evaluation of the generation of radicals. The photogenerated hydroxyl radicals (HO ⦁ ) by EBE-3D structure was examined by using terephthalic acid (THA) as probe molecule. EBE-3D in THA (0.125 mM) solution was irradiated under sun-like light irradiation and then, 2 mL of solution was taken at 30 min, 1h and 2h, respectively. The fluorescence intensity of these solutions was obtained by Fluorimeter (Fluorolog-QM, Horiba). In addition, the photocatalytic activity of EBE-3D for MB decomposition was evaluated in the presence of different radical scavengers: 0.1M 2-propanol (0.5 mL, hydroxyl radical scavenger) and 0.1 M benzoquinone (0.5mL, superoxide radical scavenger).

RESULTS AND DISCUSSION

The conjugated trimer molecule (EBE) was synthesized by Pd (0)-catalyzed Suzuki-Miyaura crosscoupling reactions. The 1 H HMR and 13 C NMR characterizations are described in the Supporting Information, SI (Figure S3-S4). Here, 3,4-ethylenedioxythiophene (E) is used as a donor group due to its strong electron-donor properties with self-organization ability associated with non-covalent sulfuroxygen interactions. 27 Benzothiadiazole (B) is a classical electron-deficient acceptor unit that is widely used in polymer solar cells. 28 The preparation process of EBE-3D is shown in Figure 1 and the 3D printing settings are specified in Table S1. It involves the melting of ABS pellets in combination with EBE in acetone. The resulting suspension was maintained at 80 ºC until a homogeneous mixture was obtained, followed by solvent evaporation (Figure 1a). The obtained orange thin film was cut into pieces and extruded to create the starting filaments (Figure 1b), For potential real-world photocatalytic applications, it is necessary to develop visible-light responsive materials that enable a strong solar light absorption capacity and high efficiencies. Figure 3a shows the normalized UV-vis absorption and emission spectra of the 3D-printed catalyst. It presents a strong absorption in the visible range, from 400 nm to 600 nm, with a maximum absorption band at λmax = 486 nm and a strong orange fluorescence at λex = 486 nm (~601 nm), which are consistent with the optical properties of EBE-powder (Figure S5). The Stokes shift (ca. 115 nm) suggests intramolecular charge transfer character in the single excited state of the 3D-printed EBE catalyst.

Moreover, a single exponential function decay model of EBE powder and EBE-3D are exhibited in Figure 3b. Both EBE-powder and EBE-3D present long-lived emission decay on ns time scales. The average emissive lifetimes of the EBE-powder and EBE-3D are 1.4 ns and 11.7 ns, respectively. The relatively longer lifetime shown by the 3D-printed EBE structure in comparison with the EBE powder might be attributed to the microenvironment effect of solvent (acetone) and a strong intermolecular 𝜋-𝜋 stacking which effectively quench the fluorescence in the powder state. 29 Such a long lifetime of the photogenerated charge carriers allows electron-hole pairs to have more opportunities to participate in photocatalytic redox reactions before recombination. 30,31 These results imply that 3D printing of organic small molecules is an efficient way to extend the lifetime of charge carriers in comparison to the solid powder state.

The FTIR spectrum of the 3D structure evidence the presence of small peaks of C-O-C (1158 cm - 1 ), C-O (1026 cm -1 ), and C-S (750 cm -1 ), which corresponds to the structure of the EBE organic molecule. A strong C≡N stretching band at 2237 cm -1 is assigned to the acrylonitrile (Figure 3c of EBE-pellet (EBE film before extrusion) and EBE-3D. The peak located at ~1000 cm -1 is ascribed to the C-H (R-CH-CH2) vinyl groups of polyacrylonitrile and polybutadiene. The small peaks from EBE can be also observed in EBE-pellet and EBE-3D. There is not much difference between EBEpellet and EBE-3D, which suggests that 3D printing EBE does not alter the chemical composition of the EBE.

The thermal property of EBE-based samples was studied by the thermogravimetric analysis (TGA). EBE-3D shows good thermal stability with a decomposition temperature (100% weight loss) of 490 °C in nitrogen, which exhibits better thermal stability than EBE-powder (65% weight loss at 383 °C) (Figure 3d). This thermal enhancement might be related to the presence of the ABS polymer on the 3D-printed structure. XRD patterns of EBE-powder, EBE-pellet, EBE-3D are displayed in

Photocatalytic applications

The photocatalytic process mainly includes the light absorption, combination and separation of photogenerated carriers, and the redox reactions on the catalyst surface with the participation of excitons. The photocatalytic performance of EBE-3D with only 0.05% catalyst content (mass ratio EBE/ABS = 0.05%) was investigated by degradation of organic pollutant methylene blue (initial concentration C0 = 50 ppm) and 4-nitrophenol (C0 = 50 ppm) and hydrogen generation from water. As shown in Figure 4a, the degradation removal efficiency of MB is about 65% after 3h of sun-like light irradiation. The semi-logarithmic plot indicates the photocatalytic degradation of MB confirms to the first-order kinetic model (Figure S9). 4-NP presents a strong characteristic peak at 317 nm along with a weak peak at 400 nm (Figure 4b). In the presence of catalyst EBE-3D, the intensity of peaks of 4-NP reduced with temporal evolution. The degradation rate of 4-NP is around 27% after 3h and the regression curve of the nature logarithm of concentration versus time is also linear, suggesting the degradation of 4-NP over EBE-3D can be described as a first-order kinetic model (Figure S10). The stability of the EBE-3D is obtained through cyclic testing. After 5 cycles, the degradation rate of MB was maintained, which indicates the good stability of our EBE-3D catalyst (Figure 4c and Figure S11). In addition, the photocatalytic activity of cut EBE pellet were compared with EBE-3D, which showed lower activity for MB and 4-NP degradation (Figure S12).

Hydrogen is a clean and environmentally friendly source to solve the energy crisis and environmental contamination. Photocatalytic hydrogen generation through a water-splitting process has attracted much attention. Therefore, the performance of EBE-3D for photocatalytic hydrogen production was also investigated in a methanol-water solution. Methanol was used as a hole scavenger that reacts with the holes to produce CO2 (CH3OH→HCHO→HCOOH→CO2) 35 (insert figure in Figure 4d). To determine the H2 is from photocatalytic reaction, additional controlled experiments were carried out including i) performing the photocatalysis in the absence of an EBE-3D catalyst and ii) under dark conditions. No H2 was detected under both conditions suggesting the H2 is produced by photocatalytic reduction reaction. As shown in Figure 4d, hydrogen production is increasing with time.

After 7 h of reaction, the rate of hydrogen production was 5.4 µmol min -1 . During photocatalytic process, the photogenerated electron-hole pairs react with the adsorbed molecules to generate various redox free radicals, which will further oxidize organic pollutants into CO2 and water. Among these active free radicals, hydroxyl radicals (˙OH) and superoxide radicals (O - 2 ) on the oxidation of organic pollutants (such as MB and 4-NP) (insert figure in Figure 4b). Hydroxylation of aromatic compounds has been used to detect hydroxyl radicals. Terephthalate (THA) is a particularly sensitive probe of ˙OH. 36 Due to the symmetry of THA (non-fluorescent), only one monohydroxylate isomer (THA-OH, fluorescent) is produced when reacting with hydroxyl radicals (see Figure5b). Figure 5a shows the increase in fluorescent intensity of THA-OH as a function of time, which indicates that ˙OH radicals are generated as the photocatalytic reaction proceeds. When 2-propanol (as a ˙OH scavenger) was added to the photocatalytic reaction, the degradation rate of MB was significantly decreased, while when benzoquinone (as superoxide radical scavenger) was added, the activity was close to that without adding any sacrificial agent (Figure 5c-d). These results suggested that the photocatalytic degradation of MB by 3D-EBE is mainly caused by ˙OH radicals. A comparison of the photocatalytic performance of main 3D-printed photocatalysts reported in the literature is included in Table 1. The photocatalyst catalyst content (0.05%) present in the 3Dprinted EBE structure is the lowest reported so far. Moreover, compared to 3D-printed inorganic photocatalysts, our system showed better photocatalytic performances for pollutant removal and hydrogen generation under visible light irradiation. For instance, the photogenerated H2 rate by our 3D-printed EBE photocatalyst was 2x higher (5.4 µmol min -1 ) than that obtained with TiO2/Au (2.2 µmol min -1 ). Therefore, these results demonstrate the potential of 3D-printed EBE-based molecule for potential scalable photocatalytic processes under solar light irradiation.

CONCLUSIONS

In summary, an organic conjugated small molecule, EBE trimer, was successfully 3D-printed for the first time. The excellent solubility of the EBE trimer in organic solvents facilitated the whole extrusion and 3D printing process, resulting in resistant photocatalytic-based filaments similar to bare thermoplastics. Interestingly, the 3D-printed EBE structure exhibited a longer lifetime of photogenerated carriers in comparison to EBE in a powder state due to the microenvironmental effect from the solvent and the less aggregation of EBE compare with EBE-powder. As a proof-of-concept, the photocatalytic performance of EBE-3D was investigated for the degradation of organic pollutants (MB and 4-NP) and hydrogen production, showing competitive photocatalytic rates under sun-like irradiation with only 0.05% catalyst content. Moreover, the stability of the 3D-printed photocatalyst was demonstrated up to 5 cycles. Therefore, this work opens new avenues for the fabrication of immobilized organic small molecules by 3D printing technology, enabling the design of hierarchical and porous structures with high efficiencies in the fields of environmental remediation and energy applications.

ASSOCIATED CONTENT Supporting Information.

The Supporting Information is available free of charge on the Wiley Publications website.

Figure 1 .Figure 2 .

 12 Figure 1. Schematic process for the fabrication of EBE-3D structures. (a) chemical structures of the EBE and ABS; (b) Filament of EBE-ABS; (c) EBE-3D.

  ). The Raman spectra of samples are shown in Figure S6. Characteristic ABS bands appear in both spectra

Figure S7 . 34 200Figure 3 .

 S7343 Figure S7. The broad peak at 2𝜃 = 20 ∘ is attributed to ABS. Due to the low content of EBE, we could not observe distinct EBE characteristic peaks in EBE-pellet and/or EBE-3D (Figure S7). XPS survey spectra of the samples indicate the presence of C, N, O elements. Pd was only observed in EBE-powder, while in EBE-3D, there is no signal peaks of Pd, mainly because the content of EBE is too low compared to ABS in the EBE-3D sample (Figure 3e). Two sets of Pd 3d XPS peaks can be assigned to metallic Pd 0 (Figure S8). The high-resolution C 1s spectra of EBE-3D are shown in Figure 3f. The peaks centered at 284.5 eV and 286.1 eV are assigned to C-H/C-C/C=C and C=N/C-O from ABS,respectively. The small peak located at 285.4 eV is attributed to O-C=O from EBE, which also indicates the small amount of EBE in EBE-3D.[32][33][34]

Figure 4 .

 4 Figure 4. Photocatalytic performance of 3D-printed EBE catalyst for degradation of organic pollutants and hydrogen generation. (a) UV-vis absorption spectra of methylene blue and the degradation rate (insert figure) of EBE-3D; (b) The degradation rate of MB after 5 cycles of reuse and the proposed photocatalytic mechanism for the degradation (insert figure); (c) UV-vis absorption spectra of 4nitrophenol and the degradation rate (insert figure) of EBE-3D; (d) Time courses of hydrogen evolution from methanol/H2O (vol.% = 1:3) solution and proposed photocatalytic mechanism for hydrogen production (insert figure) and upper insert image shows the activity at 2 h and 5 h, respectively.

⦁ 2 -

 2 photogenerated ˙OH (H2O+ h + → ˙OH) and O .- 2 (O2 + e -→ O .- 2 )on the oxidation of organic pollutants (such as MB and 4-NP) (insert figure in Figure4b). Hydroxylation of aromatic compounds has been

Figure 5 .

 5 Figure 5. (a) Fluorescence spectra of THA-OH with the evolution of photocatalytic reaction time; (b)Hydroxyl radicals react with THA to yield fluorescent THA-OH; (c) UV-Vis absorption of MB in the presence EBE-3D with additional 0.1 M 2-propanol and 0.1 M benzoquinone after 3 h sun-like lamp irradiation; (d) Photodegradation rate of MB in the presence of different scavengers.

  PhotodegradationRate (%)

Table 1 .

 1 A comparison of 3D-printed photocatalysts for MB degradation and hydrogen generation

	3D-printing	Mass Ratio	Light source	Degradation	H2 production	Ref.
	Samples	(Catalysts ratio)		rate (MB)
	LDPE-TiO2	1%	UV (300-380 nm)	14% (2h)	-	19
	ZnO-NF	50%: 50%	UV-Vis	100% (2h)	-	20
	Au/TiO2	66%	UV LEDs (λ = 365±5 nm)	-	2.2 µmol min -1	21
	Al2O3/TiO2/Pt/C3DP	65%	UV-Vis (500 W Xe/Hg lamp)	-	101 µmol g -1 h -1	22
	TNT (857 µm)	50%	UV LEDs (λ = 365±5 nm)	K=0.0501 min -1	-	24
	EBE-3D	0.05%	UV-Vis (300 W Xe lamp)	65% (3h)	5.4 µmol min -1	This
						work

Additional results on 3D design details; 1 H and 13

Author Contributions

The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.

Notes

The authors declare no competing financial interest. ACKNOWLEDGMENT X.Y. and K. V. acknowledges the support from the "la Caixa" Foundation (ID 100010434) with the fellowship code LCF/BQ/PI21/11830017.