Diane Carole

Tala Metalom

Jean Louis Fendji
email: lfendji@gmail.com

Kedieng Ebongue

Blaise Omer Yenke

Jean Louis

Fendji Kedieng Ebongue

Towards to a Direct Speech to Speech for Endangered Languages in Africa

Keywords: Direct Speech to Speech, Endangered Language, Speech Translation

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I INTRODUCTION

Direct speech to speech translation (DS2ST) has been the subject of several research projects in recent years. Kano et al in [START_REF] Kano | Transformer-Based Direct Speech-To-Speech Translation with Transcoder[END_REF] claimed that the traditional cascading method of translating a source language voice to a target language voice, requiring going through automatic speech recognition and then speech synthesis, increases the error rate, due to the error in each step. This error would be minimized by going through direct translation. As a result, some researchers developed models based on sequence to sequence [START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF][START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF], while others worked on the construction of datasets for direct speech to speech translation [START_REF] Zhang | BSTC: A Large-Scale Chinese-English Speech Translation Dataset[END_REF][START_REF] Cattoni | MuST-C: A Multilingual Corpus for End-to-End Speech Translation[END_REF][START_REF] Godard | A Very Low Resource Language Speech Corpus for Computational Language Documentation Experiments[END_REF][START_REF] Beilharz | LibriVoxDeEn: A Corpus for German-to-English Speech Translation and German Speech Recognition[END_REF]. Some others worked on research on the construction of direct speech to speech translation systems [START_REF] Tjandra | Speech-to-speech translation between untranscribed unknown languages[END_REF][START_REF] Lee | Textless Speech-to-Speech Translation on Real Data[END_REF]. The ultimate goal is the translation of speech from a source language into a corresponding speech in a target language. But so far, the work done in this area focuses on written languages such as Chinese-English [START_REF] Zhang | BSTC: A Large-Scale Chinese-English Speech Translation Dataset[END_REF], German-English [START_REF] Beilharz | LibriVoxDeEn: A Corpus for German-to-English Speech Translation and German Speech Recognition[END_REF], Spanish-English [START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF][START_REF] Zhang | BSTC: A Large-Scale Chinese-English Speech Translation Dataset[END_REF][START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF][START_REF] Lee | Direct Speech-to-Speech Translation with Discrete Units[END_REF]. A direct speech to speech model has been developed for unwritten languages [START_REF] Zhang | UWSpeech: Speech to Speech Translation for Unwritten Languages[END_REF], but the authors carried out the experiments using again well-resourced languages. Therefore, there is a need to design datasets for low-resourced or unwritten languages that can be used for direct speech to speech to evaluate the performance of this approach on low-resourced or real unwritten languages.These datasets could also be used to make some applications voice-based and interactive, allowing illiterate populations working in a field such as agriculture [START_REF] Fountsop | Deep Learning Models Compression for Agricultural Plants[END_REF] to improve their performance. The design of such datasets requires a suitable methodology. In the second section of this paper, we present a linguistic context to show the fragility of unwritten languages and the danger of a complete extinction. In the third section we briefly discuss automatic speech recognition mechanisms and the required resources. Then we show how direct speech to speech translation as been applied on "unwritten languages" by quoting authors who have developed models for these languages. We expose the problem of constructing datasets for unwritten languages in section five, before giving a dataset construction method for unwritten languages. Finally, we propose architectures for the construction of DS2ST models. The first architecture focuses on the case where the source language is written and the target language is unwritten, while the second architecture tackles the case where the source and target languages are both unwritten.

II LINGUISTIC CONTEXT

Language is the first means of communication of a population. It is also a crucial element of the identity of people. One of the best ways to promote or preserve a language is to write related documents such as dictionaries or books. Thanks to globalization and innovative methods of information and communication technologies, the development of languages could be effective. This development involves: the use of documents designed for these languages; and/or the design and deployment of applications that can help illiterate populations to master this language in their activities. The world counts 7139 languages according to The Ethnologist1 . Africa in particular has a linguistic richness representing the cultural, socio-cultural heritage and identity of many ethnic groups. Indeed, 30% of the world's languages, i.e. 21442 , are spoken in Africa by at least 887 million people. Among the world's languages, 42.27%3 are endangered, and about 308 African languages are "highly threatened" to disappear [START_REF] Batibo | Language Decline and Death in Africa: Causes, Consequences and Challenges[END_REF] (14.36% of all existing languages on the continent). More severe, at least 201 African languages are known to have become extinct [START_REF]Africa's Endangered Languages: Documentary and Theoretical Approaches[END_REF], not to mention the many otherless threatened but nevertheless vulnerable languages. This shows that the situation of languages disappearing in sub-Saharan Africa is more serious than we perceive. These languages that fade over time are mostly absent in cyberspace because they are unwritten or under-resourced. This disparity is also explained by military conquests with genocides, numerical weakness, socio-economic domination and many others. The disappearance of a language leads to the disappearance of part of the heritage of humanity in the sense that a language embodies a way of conveying knowledge. To reduce the linguistic divide, i.e. the possibility of the disappearance of certain African languages, one approach is to build automatic speech recognition systems for these languages.

III GENERAL INFORMATION ON AUTOMATIC SPEECH RECOGNITION SYS-TEMS

Automatic speech recognition (ASR) is a technique for analyzing the human voice captured by means of a microphone to transcribe it into a text [START_REF] Besacier | Automatic Speech Recognition for Under-Resourced Languages: A Survey[END_REF]. Speech synthesis is the transcription of a text into audio. Speech recognition and speech synthesis are speech processing techniques. These techniques make it possible to create human-machine voice interfaces where part of the interaction is done by voice. The construction of a speech recognition system involves modeling, a step that requires several resources. The diagram in Figure 1 shows the set of resources required to build an ASR system and the role of each resource in that system. The word spoken is compared with those of the audio signal corpus, the comparison consists in subtracting the shades of gray of the pixels of the spoken word from those of the words of the corpus and repeating this for each row and column. We will be able to find the most similar signal. The voice signal is compared to the words in the pronunciation dictionary through acoustic modeling where the acoustic model gives for each sequence of words, the possible pronunciation or pronunciations with their probabilities. The recognition algorithm makes it possible to choose the most similar word, calculating the rate of similarity between the word pronounced and the various references. The language model gives the probability of each sequence of words in the target language. The construction of an ASR requires linguistic resources such as good quality pronunciation dictionaries, textual and audio corpora, as well as vocabularies [START_REF] Fendji | Automatic Speech Recognition using limited vocabulary: A survey[END_REF]. However, languages that tend to disappear are unwritten and poorly endowed, which is a limit in the construction of ASRs for these languages. Recent works report on a new method of recognition: direct speech to speech recognition.

IV DIRECT SPEECH TO SPEECH RECOGNITION

Background

Direct speech-to-speech translation (S2ST) translates a word from a given language into a word from another language without relying on the generation of intermediate texts [START_REF] Lee | Direct Speech-to-Speech Translation with Discrete Units[END_REF]. Several works have been done on the design of direct speech to speech translation models.

The main elements of a DS2ST model are:

• Network LSTM (Long Short Term Memory): It is a recurrent neural network (RNN), which can learn long-term dependencies. It is based on the fact that long-term knowledge recall is a natural behavior of people. Many factors such as the dataset, the complexity of the model and the learning time taken by the LSTM network influence the translation process [START_REF] Sinan | Development of Output Correction Methodology for Long Short Term Memory-Based Speech Recognition[END_REF]. • Encoder: It is a component of the Encoder-Decoder architecture, it takes as input a source utterance of variable length and returns a vector of fixed length. The most common encoders used by researchers are the Variational AutoEncoder (VAE) [START_REF] Tjandra | Speech-to-speech translation between untranscribed unknown languages[END_REF][START_REF] Zhang | UWSpeech: Speech to Speech Translation for Unwritten Languages[END_REF], to transform source speech into discrete elements, forming a sequence-to-sequence model. In papers [START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF][START_REF] Lee | Direct Speech-to-Speech Translation with Discrete Units[END_REF][START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF], authors use either a 5-layer [START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF], or a 8-layer [START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF][START_REF] Lee | Direct Speech-to-Speech Translation with Discrete Units[END_REF] LSTM encoder to map the input spectrogram features into a fixed number of channels. The first layers represent the source content and the deep layers learn to encode the target content information. • Decoder: It predicts the output sequence using the vector produced by the encoder. The LSTM decoder is used in several DS2ST works [START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF][START_REF] Lee | Direct Speech-to-Speech Translation with Discrete Units[END_REF][START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF]. The 4 or 6 layer LSTM decoders give good performances [START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF][START_REF] Sinan | Development of Output Correction Methodology for Long Short Term Memory-Based Speech Recognition[END_REF]. • Vocoder: This component converts the spectrogram into a waveform file that can be understood by a human. Authors in [START_REF] Lavie | JANUS-III: Speech-to-speech translation in multiple languages[END_REF] describe the HiFi-GAN neural vocoder based on a generative adversarial network framework. HifiGAN is a neural vocoder based on a generative adversarial network framework. A discriminator composed of sub-discriminators is used by the model during training. Each sub-discriminator performs processing on a specific periodic part of a raw waveform. This vocoder is modified [START_REF] Kong | HiFiGAN: Generative adversarial networks for efficient and high fidelity speech synthesis[END_REF] to convert the units into a waveform. [START_REF] Jang | UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation[END_REF] has set up UnivNet, a neural vocoder that synthesizes highfidelity waveforms in real time.

Existing DS2ST models

In 2019, Jia et al [START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF] Figure 2, shows the architecture of the Direct S2ST model for translatotron described in [START_REF] Jia | Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model[END_REF]. The model aims to translate Spanish into English, it takes as input a Spanish spectogram obtained after Fast Fourrier extraction. The model provides as output an English voice (waveform). The Bi-Directional Long Short Term Memory (BLSTM) encoder receives the input file and returns 8 layers which are concatenated and the result is passed through a multihead Attention. The result of the latter is sent back to the spectogram decoder which will calculate a linear frequency spectrogram output. The Vocoder receives this output and converts it into a waveform that we can hear. To have a waveform output with a voice similar to the source file, and to maintain the voice characteristics, a complex trained network also called encoder is used. The block made up of LSTM decoders allows to perform the auxiliary tasks added to obtain better results.

Chen Zhang et al. [START_REF] Zhang | UWSpeech: Speech to Speech Translation for Unwritten Languages[END_REF] described a UWSpeech direct speech to speech translation model for unwritten languages. Figure 3 describes the drive and inference pipeline of this model. UWSpeech uses the method named XL-VAE (Cross-Lingual Variational AutoEncoder) based on the improvement of the VQ-VAE method (Vector-Quantized Variational AutoEncoder) so that it is able to recognize speech in several languages. UWSpeech consists of a converter that transforms unwritten target speech into discrete tokens.

The translator in this model translates the speech of the source language into discrete tokens of the target language. Finally, the inverter converts the discrete tokens translated into unwritten target speech. For good accuracy, XLVAE improves the discretization (transformation of speech into a discrete token, using the converter) and reconstruction (synthesis of discrete token speech, using the inverter) capabilities of the VQ-VAE method. The tests in UWSpeech are done with written languages,namely Spanish and English. This work does not really integrate unwritten languages. Consequently, it would be interesting to show how to design datasets of unwritten languages for the construction of direct speech to speech models in order to build models and perform tests on unwritten languages. There is still a need to go through transcripts.

[12] Conversion and translation of the source language into discrete units of the target language using the translator, then synthesis of the target speech from the discrete units in an inventer.

Fisher

Spanish-English dataset. UWSpeech achieves 17.33 BLEU points about 10 points higher than VQ-VAE.

Good performance. Tests are still done using written languages.

[6] The model consists of the attention, a vocoder, a decoder and an encoder stack. The attentionbased sequence generates the spectrograms, the vocoder converts the target spectrograms into waves. The encoder stack maps the input features of the 80-channel log-mel spectrogram into hidden states that pass through an attention-based alignment mechanism to condition a decoder that predicts 1025-dimensional log-mel spectrogram frames corresponding to the translated speech. V DIRECT SPEECH TO SPEECH FOR THE LANGUAGES OF SUB-SAHARAN AFRICA

Case of the Fulfulde language

Fulfulde is one of the most common African languages in sub-Saharan Africa. There are about fifteen countries where the language is spoken, most often by a very strong national community. Despite the fact that Fulfulde is an under-rejuvenated language, researchers in language processing have observed and noted differences with the French language on the phonetic, morphological, phonological and prosodic levels.

Phonetics, phonology and transcription

The phonological system of Fulfulde has 28 consonant phonemes b, c, d, f, g, h, j, k, l, m, n, p, r, s, t, v, w, z, y,b,d, γ, η, nb, nd, ng, nj, ny and 5 vowels a, i, o, e and u [START_REF] Tourneux | Peuhl Dictionary of Agriculture and Nature (Diamaré, Cameroon): followed by a French-Fulfulde index[END_REF]. Generally, the letters a, b, d, f, i, k, l, m, n, o, p, r, t, w are written and pronounced in the same way as in French [START_REF] Tourneux | Fulfulde-French/French-Fulfulde Dictionary (Fulani Dialect[of] Diamaré, Cameroon) Illustrations by Christian Seignobos[END_REF]. Fulfulde and French have eight letters that are written the same way, but are not always pronounced in the same way. These are: c, e, g, h, j, s, u, y. The letters b,d,γ and η are specific to fulfulde. Double letters are pronounced hard when it comes to consonants and long for vowels. In Fulfulde, the demonstrative adjective precedes the common noun it determines and agrees in nominal class with it. Its phonetic form may vary depending on the situation of the reference.

Morphology

Fulfulde has specific morphological rules different from those of French and English. For instance, the proper names of people, animals and places do not bear specific morphological marks. They are often used in the sentence without class marker (determinant). Simple common nouns are morphologically composed of a radical and a class suffix, called a nominal class index or marker. This suffix marker is used to form the determinant of the name. It also indicates, by its nature, whether the noun is singular or plural, whether it expresses the dimensional class of the diminutive or augmentative. The demonstrative adjective precedes the common noun it determines and agrees in nominal class with it.

Prosody

In Fulfulde, a sound usually corresponds to a letter. Any letter that is written shall be pronounced separately and distinctly. There are no fixed rules for the transition from a written language such as French or English to Fulfulde, which makes translation and the construction of datasets difficult.

Like Fulfulde, most of the languages of Sub-Saharan Africa have specificities that do not allow them to be easily understood, hence the particularity of the dataset construction program for these languages .

Constructing datasets of unwritten languages for direct speech to speech

The construction of the dataset for speech to speech translation consists of recordings of pairs of audio segments, the first audio corresponding to the source audio (here a written language) and the second audio is the target audio (unwritten language) [START_REF] Zhang | BSTC: A Large-Scale Chinese-English Speech Translation Dataset[END_REF]. There are two ways to create datasets: speech translation where data is generated based on complete audio recordings or their transcripts and simultaneous translation where data is transcribed in real time by human interpretation. For the construction of datasets using speech translation, we need the source audio and their transcriptions into text. Simultaneous translation would be the best option for building direct speech to speech datasets of unwritten languages (source and target), If the source language is a written language, one can combine the two methods. The steps that we can remember for the construction of the datasets of direct speech to speech translation are as follows. It is noted here that the source language is written and the target language is unwritten. Figure 4 gives ataset construction steps for direct speech to speech for unwritten languages. 1. Search and download data: The first step is to download audio or text data of source language, dictionaries, or HTML files that contain manual transcriptions of the audio data. During this step, audio recordings are produced for the target language and also for the source language when the latter is also unwritten. When the source language is written, steps 2 and 3 are performed. 2. Segmentation: Cutting transcribed texts into sentences and utterances. 3. Text to target audio:This step consist to obtain translations. The texts in the source language from the segmentation phase are transcribed into audio of the target language. This transcription is done by experienced people who master the source and target languages.

To achieve this goal, we must go through [START_REF] Post | Improved speech-to-text translation with the Fisher and Callhome Spanish-English speech translation corpus[END_REF] • Crowdsourced translations: collection of crowd-sourced translations of the collected data.

• Selection of preferred translators: Select successful translators based on the criteria described by Zaidan in [START_REF] Zaidan | Crowdsourcing translation: Professional quality from non-professionals[END_REF] For this step, a specialist in the target language will be called in to listen to the various translators' results and then select the best translators. • Complete translations: The selected translators will translate the collected data . 4. Filtering: The next step is to create a YAML file containing the recording time of sentences, words and utterances. During this phase, we eliminate noisy segments, long silences, words that do not match. 5. Collecting speech output: After obtaining the collection of translations, a split of the data must be made into training, development and test sets. This strategy will allow to verify the efficiency of the currently developed model.

Construction of the DS2ST model for weakly resourced languages

The construction of the model can be done according to whether the source language is written or not.

Case where the source language is written

The source language being written, it is possible to get transcripts. Automatic Speech Recognition (ASR) and Text to Speech Translation (T2ST) will be used. The ASR is used for the recognition of the source speech, while the T2ST is used for the translation of the text from the written language to the unwritten target speech. For T2ST, the model will be described so that as input, we have the source text of variable length which passes through an LSTM encoder as developed in [START_REF] Weiss | Sequence-to-Sequence Models Can Directly Translate Foreign Speech[END_REF]. However, an encoder of 6 layers is considered to improve the performance. The encoder transforms the text taken as input by a source state of fixed form. The state passes through a decoder to produce a spectogram of the speech in the target language. Finally the spectogram passes through the vocoder to produce the target speech. Figure 5 shows the architecture of this model.

Case where the source language is unwritten

In the case where the source and target speech are unwritten, there will be no intermediate texts.

For this, we will use a VQ-VAE encoder that takes as input the spectogram of the source speech. This encoder will generate a discrete representation given the log-mel filter input obtained by the source speech passed through the feature extraction. We will then train a sequence to sequence (Seq2Seq) model with two 6-layer LSTM decoders. The first decoder focus on the recognition of the source speech into its corresponding discrete tokens. This decoder is part of the auxiliary task as proposed in [START_REF] Lee | Direct Speech-to-Speech Translation with Discrete Units[END_REF]. Adding auxiliary tasks to a translation model regularizes the encoder and helps the model learn attention that significantly improve performance. The other decoder for the translation of the source speech into discrete tokens of the target language. The architecture for this model is given in figure 6 VI CONCLUSION AND OUTLOOK

Conclusion

This work aimed to shed light of on the possibility of using Direct Speech to Speech to help reducing the rate of extinction of endangered languages, particularly in Africa. For this aim, there is an urgent need to develop datasets for unwritten languages and to test the models developed for these languages, since authors who developed direct speech to speech model for unwritten languages only ran tests using written languages. Although the paper focused on Fulfulde, it paved the way for other similar languages.

Outlook

In future works, a dataset of French words for the source language and Fulfulde words for the target language, to perform tests on the Direct Speech to Speech system for unwritten languages developed in [START_REF] Zhang | UWSpeech: Speech to Speech Translation for Unwritten Languages[END_REF]. Additionally, a DS2ST system based on the discrete unit for translation from French source text to Fulfulde speech will be built. Finally relevant particularities of Direct Speech to Speech translation for unwritten language will be exposed.

Figure 1 :

 1 Figure 1: Construction of ASR system and required resources.

  developed a Direct speech to speech translation model and performed tests for Spanish-English translation. This model is not as efficient as the ASR (Automatic Speech Recognition) or MT (Machine Translation) or TTS translation (Text To Speech translation) models. But the authors specify that it thus opens up a new line of research. Sequence to sequence models of voice conversion were the basis for the realization of this model, the goal being to recreate a statement in the voice of another person.

Figure 2 :

 2 Figure 2: Model Architecture implemented in Google's Translatotron [6].

Figure 3 :

 3 Figure 3: The training and inference pipeline of UWSpeech [12].

Figure 4 :

 4 Figure 4: dataset construction steps for direct speech to speech for unwritten languages .

Figure 5 :

 5 Figure 5: Architecture of model for translation Written-Unwritten language.

Figure 6 :

 6 Figure 6: Architecture of model for translation Unwritten-Unwritten language .

  has been a Senior Lecturer with the Computer Engineering Department, University Technology Institute, At the University of Ngaoundere. He is the author of two book's chapters and more than 20 publications. He has been working for more than 10 years on Network modeling and optimization. His current research interests include Machine Learning, Deep Learning, NLP, Speech recognition, Optimization, AI in Agriculture and Education. Blaise O. Yenke (Member, IEEE) received the Ph.D. degree in international joint supervision from the University of Yaounde 1, Cameroon, and the University of Grenoble, France, in 2010. He is an Associate Professor and a Researcher in computer engineering. He is also the Head of the Department of Computer Engineering, University Institute of Technology of Ngaoundere, Cameroon. His current research interests include distributed systems, high-performance computing, fault tolerance, network modeling, simulation, sensor networks design, and sensor's architecture.

Table 1 :

 1 Summary of works on Direct Speech to Speech Translation.

	Disadvantages
	Advantages	Tests were	done for the	translation	of several	languages and	by using two	metric scores	we can see	that the model	is better.
	Results	Test done for specific languages	shows that BLEU and ME-	TEOR score of transcoder are	better than Google Transformer,	Google RNN, Cascade RNN and	cascade transformer.
	Experimentals data	Basic Travel Ex-	pression Cor-	pus (BTEC),	authors choose	English-to-Spanish,	Japanese-to-Korean,	English-to-Japanese	and Japanese-to-	English.
	Papers Methodology	[16] Training of an attention-based encoder-decoder	component. Use of transcoders to connect the re-	sulting components. One transcoder that transfers	attentional context information from hidden acous-	tic representations to hidden linguistic representa-	tions of the target language, and a second that do	the reverse work. Using a source language speech	encoder and three decoders that predict source and	target language text transcripts.

https://www.ethnologue.com/guides/how-many-languages

https://www.ethnologue.com/guides/continents-most-indigenous-languages

https://www.ethnologue.com/guides/how-many-languages-endangered


Byungsoo Kim

Xingchang Huang

Laura Wuelfroth

Jingwei Tang

Guillaume Cordonnier

Markus Gross

Barbara Solenthaler

Deep Reconstruction of 3D Smoke Densities from Artist Sketches

Keywords:

A Figure 1: Our method reconstructs volumetric density fields from sketch inputs using an updater CNN. The efficient reconstruction at test time enables interactive authoring and editing of arbitrary smoke shapes. From left to right we show results for the smoke jet, puppy and dissolving character examples.

Introduction

Digital content creation requires a constant and iterative interplay between artists and the generated data. At the core of these workflows, digital artists first manually generate concept sketches in the pre-visualization stage, which are then transferred into 3D models. Creating content in this manner requires a significant and costly authoring effort. To bridge the gap between early-stage prototyping and visual realization, sketch-based modeling methods have been developed that can reconstruct 3D geometries from 2D input sketches.

Many existing approaches target a particular class of objects and leverage domain-specific geometric constraints. Accordingly, they make use of properties such as symmetry, smoothness, or adjacency [EBC * 15, MHZ * 15]. To reconstruct non-flat silhouettes or higher-frequency information, various sketching paradigms have been explored to inject curvature variation across surfaces or sharp features [JHR * 15, LPL * 17, LSGV18]. Particularly challenging is the integration of domain knowledge for creating volumetric density fields, such that intrinsic flow details are captured. Existing sketch-based cloud modeling techniques circumvent this by adding noise to the shape boundary [SBRS10] or by deferring the synthe-sis of high-frequency details to the rendering stage [WBC08], at the cost of reduced control.

We address this limitation by integrating low-and mid-frequency flow structures already in the modeling step. We focus on the general class of turbulent density volumes, including physically plausible smoke keyframes and artistic smoke creatures, and target the method to early-stage prototyping of effects and communication of ideas. We impose a shading-based sketching style that follows the principles of effects drawing depicted in [Gil12]. Our datadriven method then infers 3D densities from the sketch inputs in real-time (Figure 1 and Figure 2), allowing on-the-fly testing and editing. Such a sketch-based reconstruction has to handle a set of unique challenges, as the problem is ill-posed due to the sparsity of sketch data and the ambiguity when inferring another dimension [DAI * 18, WCPM18, LPL * 18]. Previous work indicates that the use of convolutional neural networks is especially powerful in such tasks [DAI * 18, LSS * 19, MST * 20].

In our work, we present a neural network architecture that keeps the artist in the loop. The method uses an iterative strategy [DAI * 18, LTJ18], which alternately refines the shape for each input viewpoint (e.g., front and side). This design choice further allows an artist to dynamically add sketches from further viewpoints if needed, and hence enables fine-grained control over the final shape. As opposed to previous CNN-based reconstruction methods (such as [DAI * 18]), our method can reconstruct richer flow details and is more robust to variations of the input. This is attributed to the following key contributions that are essential for the robust reconstruction of smoke volumes:

• A real-time, iterative refinement method that can handle arbitrary viewpoints and preserves previous views in the reconstruction. • A differentiable sketch generator for smoke volumes that mimics the principles of effects drawing, and corresponding loss function. • A data augmentation technique accounting for sketch style variations that significantly increases the robustness of the reconstruction. • A post-processing network based on multi-pass GAN [WXCT19] that extends the sketched low-and midfrequency details with fine structures.

We evaluate the method on synthetically generated sketches, validate the quality of the results and ease of use by accompanying user studies and comparisons with previous work, and demonstrate the following demo applications:

• 3D results reconstructed from artist sketches.

• Real-time authoring sessions of artists displaying the ease-of-use creation and editing of volumetric density fields, which demonstrates a gentle learning curve even for novices and non-artists. • Density keyframe generation for smoke animation control.

Related Work

We focus our discussion on deep learning based 3D modeling from sketches. For a more general overview of sketch-based systems we refer the interested reader to [START_REF] Cordier | Sketchbased modeling[END_REF][START_REF] Kazmi | A survey of sketch based modeling systems[END_REF]. trained a CNN to predict occupancy in a voxel grid based on a single or multiple contour drawings as input. The method first generates an initial reconstruction with a single-view network, and then uses a so-called updater network to iteratively refine the prediction as new drawings from additional views are provided. We follow their idea of using subsequent refinement steps, but propose significant changes to the network components, loss functions and training process for increased robustness and multi-view consistency. * 18] use a CNN to infer the depth and normal maps representing the surface of an object. To reduce ambiguity, the network additionally considers the flow field of the surface to generate a confidence map. The input to the network consists of sketches, silhouette mask, and optional depth sample points and curvature hints. Depth and normal maps were also used to reconstruct a dense point cloud from sketches [LGK * 17]. The decoder captures the object's surface from several viewpoints, which are then fused into a single 3D point cloud through optimization. To eliminate the need of well-labeled hand-drawn sketch data during the training process and hence allow sketch style variations, Wang et al. [START_REF] Wang | Unsupervised learning of 3d model reconstruction from hand-drawn sketches[END_REF] introduced an unsupervised learning model. A shape representation was also learned in a fully self-supervised system in [START_REF] Smirnov | Learning manifold patch-based representations of man-made shapes[END_REF] to prevent non-manifold artifacts in the parametric shapes.

Li et al. [LPL

A deep learning based sketching system has also been used for 3D face and caricature modeling [START_REF] Han | Deepsketch2face: A deep learning based sketching system for 3d face and caricature modeling[END_REF]. 2D lines representing the contours of facial features represent the input to a CNN. An initial sketching mode is followed by sketch and gesture based refinement steps. Deep learning was also used for an interactive modeling tool of 3D hair from 2D sketches [SZF * 20]. Hair contour and a few strokes indicating the hair growing direction are used by a first network to generate a 2D hair orientation field, which is then processed by a second network to output a 3D vector field. For sketch-based garment design, a joint latent shape space is learnt across different modalities representing the draped garment, body shape parameters and garment parameters [START_REF] Wang | Learning a shared shape space for multimodal garment design[END_REF].

Noteworthy is also the non-photorealistic rendering system Syn-Draw that facilitates the generation of synthetic drawings to train sketch-based modeling systems [START_REF] Wailly | Line rendering of 3d meshes for data-driven sketch-based modeling[END_REF], and the work of Zhong et al. [ZGZS20] that identified the main differences between sketchand image-based modeling with respect to style variance, imprecise perspective and sparsity.

The above discussed CNN-based 3D shape reconstruction methods fail by design to reconstruct realistic flow details, as they target different classes of objects compared to our work. In contrast, our reconstruction is driven by fluid simulation data that captures turbulent structures realistically and maps them to the sketched cues.

Modeling density volumes. To our knowledge, no method exists that generates 3D reconstructions of density volumes with realistic flow structures from 2D artist sketches. Classical, non-learning based approaches typically require a mesh or surface as input and fill the volume with density values or particles. Turbulent structures can be mimicked by adding noise to the boundary particles [SBRS10]. The lack of physical plausibility is manifested in the results, which we address by driving our reconstruction with simulated flow data. Clouds have also been authored by a skeleton approach to model the surface shape and by adding details during rendering [WBC08]. This is opposed to our method, where such details are included in the 3D representation. While it would be possible to couple the above mentioned methods with a fluid solver to achieve physically realistic structures, this would inherently prevent interactive frame rates during authoring and restrict the tool to artists with expert knowledge in animation and effects.

Less related to our work but noteworthy are reconstruction methods developed in fluid animation. 3D fluid densities and motion have been reconstructed from image sequences [EHT18, EUT19, FST21, QLWQ21] and 3D smoke has been stylized based on 2D example images [KAGS19, KAGS20]. Sketches have only been used in form of strokes on selected keyframes to control the motion of liquids [PHT * 13]. Local editing is computed with an efficient optimization and propagated spatially and temporally. Sketched strokes were also used in interactive environments to define shapes and connections of fluid circuits primarily applied in medicine [ZIH * 11] and for 2D flow field design using a generative adversarial network [HXF * 19].

Overview

The design principles of effects drawing, and in particular of smoke, provide us a solid foundation for the design of an algorithm that progressively refines a 3D density volume at arbitrary viewpoints based on artist sketches.

Fluid Sketch Principles

Sketching fluid objects, such as smoke, requires a specific style and workflow. The first step towards a tool designed to interpret such drawings is to understand their specificity. We analyzed smoke drawings from multiple sources, such as the book of Gilland [Gil12], and summarize the findings that have driven the development of our tool and that explain the differences to drawings of solid objects.

Strokes. Common to many sketch-based modeling techniques, the input to our method is a user-drawn sketch. Even with modern digital tools, the workflow is still close to pen and paper drawing. Our style is thus made of several strokes with varying shades of grey and thickness (both properties respectively linked to the pressure and inclination of the pencil or piece of chalk). The main dark strokes express the silhouette and the contours of additional inner details.

Generality. Key to the plausibility of natural systems is the chaotic nature, which forbids any kind of problem simplification with structures, symmetries, hard edges or parallel strokes. Therefore, our method does not have to carefully reproduce such geometric features, but cannot either rely on them to better reconstruct the object, as commonly done by previous work [XCS * 14].

Simplicity. Intrinsic to effects drawings are natural design principles. Instead of filling the drawing with random details, every detail represents a pattern of energy. Fluid energy propagates through different scales, the so-called Kolmogorov cascade, which results in turbulence patterns that can visually be clearly distinguished. It is often advised to keep the drawing simple and to focus on a few frequencies rather than drawing all intricate details -not least to reduce the sketching time that would otherwise grow exponentially.

Our sketch style follows this principle, which at the same time is also beneficial for machine learning.

Volume. Inner contours are too sparse to fully express the many variations of the curvature of smoke objects. Additional shading is used by artists to encode this information, which also gives a better intuition of the relative placement of smoke structures. To comply with the simplicity of the strokes guideline, we impose a two-color toon shading with a light source placed such that on average 2/3 of the smoke is in the light. We see this requirement for shading as a trade-off between a too simplistic sketch that would fail to capture the intricate details of the smoke and a more complex representation that might prevent novice users from using our tool. We show in Section 8.1 that this additional color information, even painted carelessly and unrealistically, provides a powerful control hint to solve curvature and depth ambiguities in the reconstructed density.

Pipeline

One major challenge of sketch-based modeling lies in the depth ambiguity and occlusions that cannot be captured from a single viewpoint. We follow [DAI * 18] and allow a user to progressively refine sketches at different angles and hence iteratively update and relocate smoke structures. Our pipeline is depicted in Figure 3. The core of our method is an updater network (Section 4.2), which takes as input a 3D voxel grid containing smoke densities and a sketch of the desired output from the target view. The output is a corrected volume that respects two key properties:

Accuracy The output density volume seen from front view should correspond to the input sketch. Stability The density parts that are not visible from the front view should be preserved to ensure multi-view continuity.

The initial density can be user-provided, but we also propose a simple method to create an initial guess from two sketches in canonical views (Initial Volume Modeling, or IVM, Section 4.1). We eliminate the need for artists to sketch high-frequency flow structures by using a variation of multi-pass GAN [WXCT19] that synthesizes details in a post-processing step (Section 4.4).

We use a semi-supervised training strategy (Figure 3, bottom), designed to enforce the two properties of the updater detailed above. The key ingredient is a new synthetic sketcher for smoke volumes (Section 5), aiming at producing sketches that are compliant with the guidelines of Section 3.1. It has two main functions: First, to automatically annotate a data set of smoke models (Section 6). Second, the differentiability property of the sketcher allows us to incorporate it in the loss functions. This enforces the accuracy property, which we reformulate as the minimization of the difference between the input sketch and the synthetic sketch of the output. Generalizing this to all previous viewpoints leads to the rollback loss (Section 4.3), which enforces the stability property of the updater.

2D Sketch to 3D Density Prediction

Sketches are the primary control tool for the artist during the authoring process. They represent the input to our convolutional neural network (CNN) that computes the corresponding volumetric density fields. To allow for multi-view sketching, we use a refinement strategy: the volume is rotated to face the sketch viewpoint and is then corrected by the network.

Initial Volume Modeling

The initial volume modeling (IVM) computes an initial guess for the density field d0 from the provided input sketches s, i.e. d0 = V(s 1 , .., sn). For each sketch, we first extract the contours and foreground with thresholding and filling operations (note that our sketching style imposes a different color for the background). From the inputs of the different viewpoints, we compute a visual hull and an initial guess of the density distribution. We calculate the intersection of these volumes from the different views, blend them and apply Gaussian blur to smooth the boundaries. The output represents the initial guess for the subsequent optimization steps, i.e., the input to the neural network. The initial volumes are depicted in Figure 3 and Figure 4. Note that IVM requires at least two viewpoints (possibly duplicates of the front view). In all our illustrations and most examples we use two input sketches indicating the front and left views, i.e., d0 = V(s f , s l ). In practice, arbitrary additional views can be added if needed.

Updater CNN

We adopt an updater CNN U that alternately optimizes information from the different input views. We refer to each of these iterations as pass p, associated with a camera orientation θp.

Each pass p takes as input an artist sketch sp and a previously reconstructed density field dθp p-1 = T ( dθp-1 p-1 ). Note that we need to rotate (i.e., T (.)) the output of pass p -1 computed at an angle θ p-1 to obtain the input of the next pass p at an angle θp. The reconstructed density after p passes, seen from an angle θp, is then given as dθp p = U( dθp p-1 , sp). In the first iteration, the predicted density field corresponds to the initial density dθ0 0 = d0 computed with IVM and no rotation is performed dθ1 0 = dθ0 0 . During training, we pick a random view from one of the 24 canonical views in each pass, as multi-pass/view training results in higher robustness for the reconstruction when the density field is optimized from multi-ple viewpoints. An overview of the training pipeline is shown in Figure 3.

Loss Functions

In each pass, we calculate view-dependent losses (density loss, sketch loss, depth variation loss) and a loss over all preceding passes (rollback loss) for preserving reconstructions of previous input views.

Density Loss. We define our density loss L d in terms of L1 loss on the reconstructed field, similar to [KAT * 19]:

L d ( d, d) = || d -d|| 1 , (1)
where d refers to the ground truth density field and d is the reconstructed field from our updater network.

Sketch

Loss. An important difference to previous work is the integration of a differentiable sketcher R (Section 5) into the network, which allows us to generate a target sketch sp = R(dp) and a sketch of the reconstructed density ŝp = R( dp) at each pass p during training. We use this to minimize the distance to the input sketch with a sketch loss Ls (Figure 5, left) given as

Ls( ŝ, s) = || ŝ -s|| 1 , (2)
where s is the input sketch and ŝ is the rendered sketch from the reconstructed density d.

Depth Variation Loss. As the sketch loss only focuses on the rendering of the 3D volume, there is no constraint in the depth direction during training. Although the sketch shading encodes some depth information, artifacts in depth are still noticeable even with the density loss (Figure 20). We, therefore, use the total variation loss in the depth direction as a regularizer to enforce smoothness:

Ltv( d) = || ∂ d ∂z || 1 . ( 3
)
Full Objective. We define an auxiliary loss Laux combining sketch loss and total variation loss in depth as following:

Laux( d, d) = wsLs(R( d), R(d)) + wtvLtv( d), (4)
where ws and wtv are weights for the sketch loss and total variation loss set to 0.03 and 0.1, respectively. We can then define a pass loss Lp as

Lp = pL d dθp p , d θp +Laux dθp p , d θp +w rb p-1 ∑ n=1
Laux dθn p , d θn .

(5) We refer to the last term as the rollback loss (Figure 5, right) weighted by w rb . The rollback loss compares the synthetic sketches of the generated density and the one of the ground truth from all previously reconstructed viewpoints. This enforces the updater to preserve the density in the areas unseen by the current view as much as possible, which would otherwise be degraded. The rollback weight defines how much the updater network preserves the reconstructions of previous viewpoints. While at the beginning of the optimization a large degree of freedom is necessary and hence a lower value for w rb is advantageous, a larger value is important in later steps to preserve structures reconstructed from other viewpoints. Therefore, we progressively increase w rb from 1 to 100 during the first epoch.

The total loss is then defined as

L total = ∑ p Lp ∑ p p . (6)

Post-processing Network

To alleviate the need for an artist to sketch small-scale structures, we use a second network to synthesize flow details onto the reconstructed density. We use the architecture of tempoGAN [XFCT18] but restrict the model to the spatial discriminator. For the training, we first add the ground truth velocity to the input data, which during inference is the output of the density prediction dp. Similar to multi-pass GAN [WXCT19] we train the network in two passes. In

Differentiable sketcher

Our sketcher aims to generate a stylized representation of a smoke keyframe that respects the guidelines observed in Section 3.1.

Previous techniques typically extract contours from meshes [START_REF] Decarlo | Suggestive contours for conveying shape[END_REF] (inset image, left) and cannot be directly applied to amorphous shapes such as smoke densities. Although isosurfaces can be extracted from the density grid and converted into a mesh, the choice of the isovalue is not obvious and typically low-density details are lost in the process. Therefore, for cartoon rendering of smoke, particles have been used and traced through the underlying fluid grid, and then rendered as textured billboards without [SMC04] and with [MF06] shading and shadow effects.

In contrast to using tracer particles, we work directly with volumetric data and adapt the mesh contour extraction to amorphous smoke shapes (inset image, right). Our method is inspired by the absorption in smoke rendering and estimates a normal map. We average normals for a range of density isosurfaces at the visible 'boundary' of the smoke (Equation (8)). At each position p in the volume, the normal of the isosurface is the gradient of the density field: ∇d(p). We integrate the normals along the viewing rays while weighting them such that the normals closer to the boundary have higher importance. For a ray r, the normal nr is given as

nr = ∞ 0 ∇d(r(s)) w(r(s)) ds, ( 7
)
where s is the distance to the camera and w the weighting factor.

A possible choice for the weights is to use the transmittance of the smoke. Computing the averaged normal would then be equivalent to the rendering of the smoke, where the density gradient is set as the light emission of each of the voxels. We found, however, that this strategy results in too smooth variations in normals, leading to blurred sketches. Therefore, we choose a stricter filtering strategy: Let τ(s) be the accumulated density along the ray: τ(s) = s 0 d(t) dt. We choose our weight as

w(s) = ξ 2 • τ(s) e -ξ•τ(s) , (8)
where ξ = 5 is a scale coefficient. This ensures that a small band of smoke close to the boundary contributes to the final normal while discarding the low-density voxels that are first encountered by the ray and which produce noisy normals.

We make two assumptions to discretize Equation 7. First, we assume that the gradient of the density is constant between two voxels , and second, that τ varies linearly in a small neighborhood. The normal can then be defined as

nr = i=n-1 ∑ i=0 ∇d(i) + ∇d(i + 1) 2 i+1 i w(τ(s)) ds + n∞ = i=n-1 ∑ i=0 ∇d(i) + ∇d(i + 1) 2 [-(ξ • τ + 1) • e -ξ•τ ] τ(i+1) τ(i) τ(i + 1) -τ(i) + n∞, ( 9
)
where n is the number of cells along the ray (here it is assumed to have a cell size of 1, without loss of generality). Similarly, n∞ is the background normal, that we set as opposed to the view direction v:

n∞ = -v ∞ n w(τ(s)) ds = -v (ξ • τ(n) + 1) • e -ξ•τ(n) . (10)
The integral appearing in Equation 10 is also used as a mask that indicates if the ray intersected the smoke.

We normalize nr, which is then used to extract the contours c as:

c = max(0, min(-nr • v, δ))/δ, ( 11
)
where δ is a threshold set to 0.8. We then add a two-colors toon shade t to provide more information about the volumes and we combine all to compute the sketch s as

s = (1 -c) • c + c • t. ( 12
)
The individual steps of the sketcher R are visualized in Figure 7.

Training Data Generation

The generality of our data set (data set and code will be released after acceptance) is key for robust and high-quality reconstruction of diverse scenes sketched by artists, that can include both physically inspired and non-physical shapes and motions of smoke. Therefore, we build a new data set consisting of smoke densities, which captures a large variety of volumetric shapes. We also augment the output of our synthetic sketcher with several strategies to increase the robustness of our network with respect to small variations in sketching styles.

Simulation Training Data

Our simulation data set consists of 39, 380 density fields (from 1641 simulations) used for training and 600 density fields (25 simulations) for validation. Example snapshots from the training set are shown in Figure 9 and in the Supplementary Information (SI). Each simulation contains 20 frames without source and 30 frames with source computed at a resolution of 129 3 . We run the simulation in Houdini with randomly changing temperature diffusion factor, cooling rate, viscosity, buoyancy strength, and direction, as well as sharpening and turbulence factors.

For simulations without a source, the pre-generated source is set as the initial density of the simulation. For the ones with source, we selected one of the regenerated sources and combined it with a max operator on the ongoing simulation. The number of sources for each simulation are [1,2,3,4] with probability [10,5,2,1]/18. The shapes of the initial sources are generated by computing a random shape from a set of union or difference of cube, cylinder, and sphere, and setting the resulting shape as the initial density of a fluid simulation that we run for 10 frames. We added this supplementary operation to remove sharp edges and corners found in the initial shapes and to capture a more diverse range of densities than the original binary ones.

Based on the density histogram, we find that both simulation strategies are complementary to each other. No source data are dominant by small density values, while with source data higher density values can be observed. In this way, our neural network model is not biased to learn to output small or large values only. It also helps to increase variation in the data set so that we can handle various unseen data, such as simulation with and without sourcing.

Sketch Training Data and Augmentation

The synthetic sketcher allows us to inexpensively build a large data set of smoke simulations coupled with sketches. We used sketch sizes of 258 2 . Using a larger size compared to the density fields improves quality and is also more artist-friendly. To increase the robustness against variations in artist style, we augmented the sketches used in the training to cover different values for brightness, contrast, contour strength, toon shading color, blurring, and slurring [SSISI16] in x and y directions as noise addition, and variations in the light direction in x and y (see SI). Figure 8 illustrates the different properties.

Implementation and Synthetic Results

We first provide additional details on the implementation and then evaluate our reconstructions on the training and validation data set and on synthetic scenes.

Implementation and Performance

We implemented our system in PyTorch and used the Adam optimizer [KB15] with a learning rate of 0.0001. Our network follows Our model converges in 10 epochs (see SI), and the training time with our data set takes up to 3 days using 4 NVIDIA GTX 1080 GPUs. At test time, the IVM computation with front and side input sketches takes ∼ 6.5ms per frame on a single GPU. Similarly, the update step of our trained updater CNN takes ∼ 3ms per frame. With refinements on front and left views (fl), the total inference time is ∼ 12.5ms per frame and hence our method is able to reconstruct 3D densities in real-time.

The post-processing model is trained with 20 epochs and takes up to 8 hours to train the first pass and another 8 hours for the second pass using an NVIDIA RTX 3080 GPU. At test time the network takes ∼ 2s per frame for the first pass and ∼ 0.5s for the second pass.

Reconstruction of Training and Validation Data

Figure 9 shows the front view result of a reconstruction of the training data set ('seen', top row) and validation set ('unseen', bottom row). From left to right we show the sketch of the ground truth (input sketch), the sketch of the reconstructed density, the reconstructed density (without post-processing), and the ground truth density. It can be seen that only minimal differences are visible between the input sketch and the sketch of the output. The reconstructed density is -as expected -smoother than the ground truth. This is because the sketch lacks complete information about the original density. This implies the ambiguity of several possible densities corresponding to a given sketch. While an alternative could be to use a sketch style that also encodes fine flow structures, we preferred to follow the sketching principles outlined in Section 3.1 and, if desired, to synthesize high-frequency details in the postprocessing.

Reconstruction of Synthetic Scenes

We show that our method generalizes well to various scenes by applying it to inputs from physically-based simulations and selected frames from animation sequences including a character, cloud, and puppy animation. For these examples, we generated synthetic sketch inputs by applying our sketcher to the given density field. It is important to note that the given density cannot be directly compared to the reconstructed density: our method infers a density such that its sketch is close to the input sketch.

Simulated Smoke

We used a selected frame of the smoke jet data set of [KAGS19] and applied our sketcher to the existing density field to generate sketch input keyframes for front and left views. Figure 10 shows the synthetically generated sketch from the given density (input to our method), the sketch of the reconstructed density field, and the reconstructed density without and with post-processing. The reconstructed density closely matches the footprint of the input sketch.

We measure the quality of the inner details by comparing the synthetic sketches of the input with the one of our output, where we see that the shading and inner contours are well captured.

Animation Data

To evaluate the model on unseen (non-smoke) shapes, we used selected keyframes of animation sequences as input and show the corresponding reconstructions in Figure 11

Arbitrary Viewpoints

In most of our examples, we show results where the sketches are prescribed from canonical viewpoints (front and left views). We believe that this is one of the most complicated cases for density reconstruction because no information is shared between the different views. But we also show that our method can be applied efficiently to sketches from arbitrary viewpoints. The main -classical -issue is that the corners of the domain are cropped during rotation. We chose not to increase the size of the volume, but instead we reduce the domain to a sphere inscribed inside the 129 3 cube. A quick computation shows that proportionally less volume is wasted with this strategy. Figure 12 shows an example where the puppy model was reconstructed from sketches drawn from arbitrary viewpoints. The last row demonstrates again the effect of the neural post-processing to synthesize turbulence details.

Application Demos

We demonstrate our approach on three real-world applications. We first present 3D smoke results generated by artists, introduce our in- teractive authoring tool for smoke keyframe prototyping and show how the 3D models could be used in animation.

Results on Artist Sketches

We used sketches drawn by multiple different artists to evaluate our method at test time. We instructed the artists to sketch keyframes of smoke or an imaginary smoke-like object. For each example, we requested a front and side view sketch. We provided specific instructions for image size (256x256), light direction (front-right, slightly up), shading (toon, white, and grey), colors (grey-black lines, white background), and line width and style (up to 5px with Gaussian falloff).

Figure 14 depicts different keyframes drawn by artists and the corresponding reconstructions, without (left) and with (right) postprocessing. The scenes illustrate a dissolving character that transforms into a rising smoke cloud, a shape of a lion that transforms into a rhino, and a smoke plume that transforms into a bird-like object. A sequence depicting the dissolving character can be seen in Figure 13. Further sequences, as well as sketches and renderings from other viewpoints, are provided in the SI.

Our method works surprisingly well for such highly nonphysical examples. The reconstructed density and sketches thereof closely match the input sketches from the artists. Note that the reconstruction is also robust to variations from the prescribed sketch style and can handle inconsistencies of shape and shading between the different views.

We observed that artists took advantage of our sketch style to draw in a coarse-to-fine manner (Figure 15). First, they specify the silhouette, giving a global sense of the resulting volume. They subsequently refine with sketches detailing the inner contours, which result in interior volumes where the direction of curvature or relative depth is not the one desired by the artists. Fortunately, this can be effectively corrected by a last stroke with shade information. Interestingly, our system proved robust to unrealistic shading, and even non-artists intuitively used this tool as an extra hint for the network to solve ambiguities in the reconstruction.

Real-time Authoring in Houdini

We have implemented a Houdini plugin for interactive smoke field authoring. For two selected authoring sessions with different artists, we show a screen capture, the generated sketch, and final rendering in Figure 16 and Figure 2. It took the untrained artists only a couple of minutes to create, iteratively edit the shape, and render the final result. Noteworthy is that our sketcher provides the artist with an estimate of sketches as guidance from unseen viewpoints. The accompanying video shows additional examples by novice users. It can be seen that the sketched front and side view inputs do not perfectly correspond and that some artists even varied the sketch style or introduced imperfections such as gaps between strokes. Even in such challenging cases, our method could reliably infer the 3D shape. In Figure 17 we used an image from [Gil12] as input to our tool. We duplicated the front view, edited the resulting side view, computed the 3D reconstruction, and rendered the result with a liquid texture.

Keyframe Control of Smoke Simulations

A further application of our method is the authoring of density volumes that can then be used as target keyframes in fluid simulation control methods. We use the most recent approach for keyframe control [TACS21] that employs a differentiable fluid solver to compute derivatives of the objective indicating target density matching quality with respect to control parameters, to generate animations that match the targets. We take the reconstructed densities as target keyframes from the dancing character (Figure 11), top row) and dissolving character (Figure 14, top row) scenes, and use the earliest frame as the initial state and the later two frames as targets. Selected frames from the keyframe control animation can be seen in Figure 18. The resulting animation sequences show that the simulation matches the target density keyframes (inset images) while generating intrinsic flow structures and filaments due to the dynamics computed with the underlying physics solver (see also the accompanying video).

Evaluation

We compare our method to previous work and evaluate the results in a user study with viewers. We additionally discuss the results of a user study with artists to answer questions related to our authoring tool and sketch style. Lastly, we evaluate the design choices of our network and method in an ablation study.

Comparison with Previous Work

We compare our approach (without post-processing) to a previous CNN-based shape reconstruction method [DAI * 18] that predicts occupancy in a voxel grid based on a single or multiple contour drawings as input. Similar to our method, an updater network is used to iteratively refine the result. Since the previous work targets solid geometry reconstruction, we retrained the model on contour drawings of our data set with the density loss (Equation (1)) to make the results comparable. Figure 19 shows the superiority of our method in terms of reconstruction quality of outer and inner structures, which is attributed to the network design and differentiable sketcher. The same figure shows the results of a classical, non-learningbased approach for density authoring. The cloud modeling method [SBRS10] first computes a surface mesh, which is then filled with particles. By adding noise to the particles, finer structures are generated. The results indicate that our approach can reconstruct flow details more realistically and accordingly suggest the advantage of driving the modeling by simulated flow structures rather than noise.

User Study

Evaluation with Viewers

We conducted a user study with 53 viewers to evaluate how they perceive our results in comparison to previous work (A) [DAI * 18] and (B) [SBRS10] (Section 9.1). For 8 different scenes, we showed the rendered images of the three methods side-by-side in random order. Table 1 summarizes the statistics for the three scenes shown in Figure 19. We asked the users to rate the image with the best overall similarity to the input sketch, outer contours, inner contours, shading (normal information), and similarity to real-world smoke. We choose that specific order for the questions to minimize the bias induced in the subsequent analyses. We observed another bias in the appearance of the results, especially of method (B), for which we rescaled the densities so that they resemble ours. The remaining difference in appearance comes from the structure of the details and the thicker boundaries inherent to method (B). In all questions our method clearly outperforms the previous work, except for one The results indicate that although Scalarflow exhibits smoke volumes of extremely good quality, it does not embed enough variability to reconstruct arbitrary scenes.

Discussion and Conclusion

We presented a method for reconstructing 3D density fields from 2D artist sketches. Our technique fills the currently existing gap between hand-drawn sketches and 3D pre-visualization, allowing an artist for the first time to prototype interactively 3D smoke volumes for testing and communicating ideas. Accordingly, our method targets the early design stage of 3D smoke rather than high-end production. We proposed an updater CNN architecture for optimizing density fields from arbitrary viewpoints, synergistically combined with a differentiable sketch renderer in an end-to-end training. The loss functions are specifically designed for the sketch-to-density problem. The training data set was carefully designed for general applicability, demonstrated by applying the method to diverse density volumes ranging from physics simulations, captured realworld flows, procedural clouds, character animation, and highly non-physical artist sketches.

During the algorithm design, we had in mind that the method must be efficient at test time to enable interactive prototyping, editing, and pre-visualization of smoke animations. This is a prerequisite for future integration into workflows and tools used by artists. Our total inference time per frame is 12.5ms and hence enables real-time previews during sketching. We have implemented a Houdini plugin and demonstrated in interactive authoring sessions that a volumetric smoke shape can be authored by an untrained artist in only a couple of minutes.

We decided to use a high-level sketch style inspired by effects drawing principles, which is easy to use and that allows rapid prototyping of ideas. A general problem of sketch-based modeling is that the simplified representation used in the sketches introduces ambiguities since several possible density fields correspond to a given input sketch. This limitation is especially pronounced when comparing the reconstructed densities with their ground truth counterparts, showing that the reconstructions are smoother than the original volumetric field. The neural post-processing step alleviates this to some extent; it synthesizes details that are visually important but, on the other hand, are physically incorrect (and hence deviate from the ground truth density fields).

Another limitation of our work is that arbitrary drawing styles of artists are not supported. We rely on the fact that artists follow our design principles for the sketching style, which is, for example, using clean contours and two-color toon shading. Due to our sketch augmentation strategy during training, we get a robust reconstruction even if an artist's style slightly deviates from the prescribed one, such as using different contour widths, brightness, or light direction. Since imposing a certain sketch style may hinder creative freedom, future work may address possibilities to relax this constraint, e.g., using a pre-processing network that converts any sketch to our style.

We have evaluated the artists' experience with our method and their perception of its usefulness in production pipelines. The artists were very enthusiastic about the overall idea, the result they achieved, and the potential impact on industrial workflows. They especially liked the idea to use it as a tool to communicate early ideas with the client or art director. As the main drawback the prescribed style and sketching with multiple viewpoints was mentioned. While the sketch style is a limitation of our method, all artists responded positively to the simple use of the system and the robustness of the reconstructions, which can be accounted to the prescribed style.

We have further demonstrated a potential application beyond density keyframe authoring. We used our reconstructed densities as target shapes to guide a physics simulation with the most recent fluid control technique. The resulting animation sequences are smooth and generate realistic flow patterns due to the underlying dynamics.

For production quality authoring higher resolution density fields are needed. Our method does not readily extend to higher resolutions because of memory requirements during training, especially because of our recursive rollback loss. Enabling scalability of the method to higher-resolution settings, for example by using patchbased approaches, could be an interesting avenue for future research.

AFigure 2 :

 2 Figure 2: Screen capture of an interactive authoring session of a volcanic plume, and the corresponding sketch and rendered result.

Figure 3 :

 3 Figure 3: Overview of our method. At inference time, the updater network U takes as input a sketch and a density volume (initialized by IVM) and computes a residual to correct the volume. By rotating it to arbitrary viewpoints the result can be iteratively refined. A subsequent post-processing GAN synthesizes additional fine structures. At training time, our sketcher generates drawings of our smoke data set. The sketcher is also a key component of the unsupervised sketch and roll-back losses.

Figure 4 :

 4 Figure 4: From two input sketches depicting the front and left views, we compute the initial volume and then alternately optimize the front (f) and left (l) view density reconstructions with an updater CNN. The intermediate density fields and corresponding sketches illustrate the convergence.

Figure 4

 4 Figure4shows the output of the individual passes at test time for an example where two sketches are provided. The updater CNN alternately uses the front and left view sketch in the optimization. In this example, a total of 4 passes are computed (front -left -front -left). The intermediate density reconstructions and corresponding sketch representations illustrate the convergence after multiple passes.The concept of using an updater network is similar to Delanoy et al. [DAI * 18], but there are significant differences in the network components, loss functions, and training process that impact reconstruction quality and robustness of predictions. We train the network to predict the residual (d θpdθp-1 p ) [HZRS16] for correction instead of predicting the density field directly. Due to the ambiguity of line drawings, [DAI * 18] focuses on a few informative viewpoints for training. Although we also use canonical viewpoints, our differentiable sketcher allows not only less ambiguity on depth with its shading, but also easy extension to arbitrary viewpoints on the fly, without any loss of generality.

Figure 5 :

 5 Figure 5: In addition to the density loss, our training process uses the sketch loss (left), which compares the input sketch sp to a synthetic sketch created from the output density ŝp. The rollback loss (right) further rotates the output density back to face the previous viewpoints to match the corresponding target sketches. In practice, this enforces that the unseen details are not changed by the updater.

Figure 6 :

 6 Figure 6: Details can be synthesized with multi-pass GAN onto a density field (left) and controlled by the curl-noise parameters (middle, right).

Figure 7 :

 7 Figure 7: The differentiable sketcher computes normals, contours, and toon shading (from left to right), and combines them into the final sketch (right).

Figure 8 :

 8 Figure 8: Sketches are augmented to consider variations in sketching styles to increase robustness of the reconstruction.

Figure 9 :

 9 Figure 9: Density reconstructions of seen (top) and unseen (bottom) examples of our data set. From left to right: input sketch, sketch of the reconstructed density, reconstructed density, ground truth density.

Figure 10 :

 10 Figure 10: Result using a physics-based simulation data set (smoke jet) of [KAGS19] showing front and left views. From left to right: input sketch, sketch of the reconstructed density, reconstructed density, post-processed result.

Figure 11 :

 11 Figure 11: Results using keyframes of three unseen animation sequences: dancing character, clouds, and puppy. From left to right: input sketch, sketch of reconstructed density, reconstructed density, post-processed result.

Figure 12 :

 12 Figure 12: Our method can seamlessly handle progressive generation from arbitrary viewpoints despite training with canonical views. From top to bottom: input sketches, reconstructed densities, and post-processed densities.

A

Figure 13 :Figure 14 :

 1314 Figure13: A sequence depicting a dissolving character sketched by an artist and reconstructed with our method. Our efficient sketch-todensity computation is especially useful for rapid prototyping and communication of ideas.

A

Figure 15 :AFigure 16 :

 1516 Figure 15: Our sketch style naturally guides the users to a globallocal approach, where the silhouette (left) defines the boundaries of the objects, inner contours (middle) specifies some details, and shading provides an extra level of control on the direction of curvature.

Figure 22 :

 22 Figure 22: Using sketch variations in the training improves the robustness of the reconstruction quality. We reduced the brightness and contour width of the sketch of the character example (top row) and evaluated the bird scene that deviates from the prescribed sketch style (bottom row). From left to right: sketch input, results after training with sketch augmentation, results without augmentation.

Figure 23 :

 23 Figure 23: 3D reconstructions computed with a model trained on our data set (top) and Scalarflow (bottom), highlighting the importance of using a well-designed training data set for generic applications.

© 2022 The Author(s) Computer Graphics Forum © 2022 The Eurographics Association and John Wiley & Sons Ltd.

Acknowledgement

The authors would like to thank Maurizio Nitti (bird sketches and puppy animation), Isabelle Hock (dissolving character), Fraser Rothnie (rhino) for their artistic contributions and Daniel Dorda, Emilie Yu, Capucine Nghiem for testing our tool. The work was supported by the Swiss National Science Foundation under Grant No.: 200021_168997.

where the users found that method (A) better resembles real smoke (Figure 19, top row). Table 1: User study statistics for the three scenes of Figure 19.

We asked 53 participants to rate the best reconstruction (Ours, A [DAI * 18], B [SBRS10]) with respect to overall similarity, outer contours, inner contours, shading (normals), and similarity to real smoke structures.

Evaluation with Users

We have evaluated the authoring tool with 14 users (7 non-artists, 5 hobbyists, 2 professional artists), who either tested the authoring tool or contributed with sketches to our work. All users have at least basic experience with image editing and 3D modeling tools. They read our instructions on the sketch style (see SI) and were able to adopt the style already in their very first design tests.

100% found that the method is targeting artists that need to prototype and communicate ideas, while 57% and 42% could see the tool also for high-end products and non-artist use, respectively. 62% reported that the tool is useful for early-stage prototyping, while the remaining 38% find it somewhat useful. 85% rated the reconstruction quality as good, while 60% found that the result matches the expected intent either good or very good. They all rated the time to generate the result as average to fast. Only 1 user reported that the tool is unsatisfactory. The 2 users who had experience with alternative strategies to create density keyframes both reported that they prefer our method over alternatives.

The question of whether the sketch style is intuitive was answered by 64% as good to very good. Only 1 user reported that it took too long to adapt to the imposed style, the others all gave neutral or positive scores. Similarly, only 1 user mentioned that the sketch style negatively impacts artistic creativity. The most negative point was attributed to the sketching of different views, which 40% found difficult, and 30% somewhat difficult or not difficult. We evaluated the impact of the different loss functions and illustrate the results in Figure 20. Using only the density loss (Equation 1) like in previous work [DAI * 18] we observe large discrepancies between the input sketch and the sketch of the reconstructed density (see also SI). When using only the sketch loss (Equation 2) results are very detailed but at the cost of depth ambiguity. If both density and sketch losses are used, the sketch correspondence, as well as the depth reconstruction, are improved, but noise is well visible. Adding the total variation loss in the depth direction (Equation 3) eliminates these artifacts and generates the best results (our model).

Figure 20 also embeds a quantitative quality measure (value of density and sketch loss). Lower values stand for higher accuracy.

Recursive Passes

We evaluate our network with various numbers of passes at training and test time. We show the results on front view only in Figure 21, where each row corresponds to 1, 2, and 3 passes during training, respectively, and each column to 1, 3, and 6 successive refinements at test time that we denote by 'f', 'fff' and 'ffffff'. We observed that the difference between training pass 3 and 4 is marginal; we therefore used 3 passes in all our examples. They result in a clearly better reconstruction than the single-pass version, which justifies the use of our rollback loss.

On the other hand, with 3 and more passes during training, the number of successive optimization passes at test time has less influence. The results in 'f' already recovered most of the features of the input sketches and are slightly improved by 'fff', which is hardly distinguishable from 'ffffff'. Next, we alternate front and left view refinement steps (f, fl, flf, flfl) and show the impact on the reconstruction quality in

Sketch Augmentation

We argue that using augmented sketches in the training favors generality. We evaluate this property in Figure 22 and show comparisons of reconstructions from different sketches modified to be far from our initial parameters. On the top row, we input our synthetic sketch of the character scene, where we reduced the brightness and the line width. On the bottom row, we use one of the artist examples that is far from the prescribed style. The middle column shows our results with data augmentation, while the right one shows the results without. Several parts of the sketches were not reconstructed, which shows the need for data augmentation for more generality in the sketch style.