Juliana L D Lebeau
email: juliana.lebeau@gmail.com

Mekala Venkatachalam

Mireille Fouillaud

Laurent Dufossé
email: laurent.dufosse@univ-reunion.fr

Yanis Caro

Doru Abrudan

Samuil Micu

Carmen Socaciu

Adela Pintea

Andrea Bunea

Sanda Andrei Loredana

Florina Leopold

Cristina Coman

Dan Cristian Vodnar

Olivia Dumitrița

Rugină Zorița

Diaconeasa Oana

Lelia Pop

Francisc Dulf

Andreea Elena Pop

Lucian Cuibus

Dorottya Domokos

Andrei Lazar

Beatrice Dan

Norway Øyvind M Andersen

Marco Arlorio

George Britton

Vincenzo Fogliano

United Kingdom Paul D Fraser

Adriana Mercadante

Antonio Melendez Martinez

Jozsef Deli

Reinhold Carle

Damaso Hornero-Mendez

Monica Monica Giusti

Gerhard Sandmann

Begona Olmedilla -Alonso

Maria Roca

Antonia Melendez

Martinez -József Deli

Dámaso Hornero-Méndez

Judith Mueller-Maatsch

Florence Kreiser

Judith Hempel

Oliver Paquet-Durand

Ralf Schweiggert

Saniye Işıl İlter

Mehmet Akyıl

Zeliha Koç

Meltem Demirel

Figen Conk- Dalay

Kaymak-Ertekin

Daniele Giuffrida

Francesco Cacciola

Margita Utczas

Domenica Mangraviti

Ivana Bonaccorsi

Veronica Inferrera

Paola Donato

Paola Dugo

Luigi Mondello

A Ligia Focsan

Lowell D Kispert

Luis E Rodriguez-Saona

Cheikbeye

El Thami

Jacqueline Mejdoub

Philippe Destain

Thonart

Stefan Martens

Cathie Martin

Ingo Appelhagen

Trine Hvoslef-Eide

Anne Oertel

Doris Marko

Gudrun Pahlke

Hans-Peter Mock

Andrea Matros

B Olmedilla

Alonso -Marco Arlorio

Kewei Chen

Jose Julían Ríos

Antonio Pérez-Gálvez

María Roca

Mihaela Turturică

Nicoleta Stănciuc

Gabriela Bahrim

Gabriela Râpeanu

Co2 Miriana Durante

Pier Marcello Salvatore Lenucci

Paolo Marrese

Vito Rizzi

Monica De Caroli

Gabriella Piro

Paola Fini

Giovanni Mita

Manman Lu

Jianxiong Cai

Wei Xu

Jing Chen

Shujuan Zhao

Pu Jing

R M Arlorio

G Negri

J D Del Favero

V Coïsson

F Papillo

V Travaglia

D Gokmen

G B Marko

Giovenzana

Øyvind Andresen

María Roca

Myriam Benarroch

Justine Lallement

Larbi Rhazi

Camille Boroch

Cindy Hugot

Claude-Narcisse Niamba

Hassan Younes

Flore Depeint

Juan Garrido-Fernández

José J Ríos

Ana A O Xavier

Josefa Aguayo Maldonado

Andreina Bruno

Mirana Durante

Pier Elisabetta Pace

Eszter Gábor Palotás

Sarkadi Nagy

Hussein Daood

Éva Martos

Lajos Helyes

Andrea Lugasi

Philippe Thonart

Manfred Eggersdorfer

Ulrich Killeit

Dirk Cremer

Michael Murkovic

Alam Zeb

Marini Damanik

Antoanela Patras

Razvan Filimon

Antonio J Meléndez-Martínez

Enrique Murillo

Attila Agócs

Erika Turcsi

Szabolcs Béni

András Darcsi

József Deli

olives. Miriana Maria Durante

Luca Tufariello

Gianluca Tommasi

Giovanni Bleve

Mita

Anna Rafaela Braga

Katia Biazotto

Veridiana Vera De Rosso

Cerrado Kátia

Regina Biazotto

Anna Rafaela

Cavalcante Braga

F Blando

H Berland

M Durante

C Gerardi

A Mazzucato

M Andersen

Dominika Szot

Sławomir Wybraniec

Monica Locatelli

Valentina Azzurra Papillo

Fabiano Travaglia

Jean Daniel Coïsson

Stefania Monteduro

Debora Giordano

Amedeo Reyneri

Massimo Blandino

Jean Daniel Coisson

Emerging greener extraction systems for fungal pigments isolation Context & Problem

Keywords: fungal pigments, separation technology, green chemistry, ionic liquids. th

7 Round Table 11.15-12.15 Presentation of

The demand for natural pigments has been growing over the last two decades and is now becoming a hot topic of research. However, to fully satisfy such demand, there is an urge for the development and design of green and sustainable extraction and isolation techniques that would allow higher recovery and greater selectivity in a quicker and more benign manner than conventional methods. Indeed, the current main extraction methods are still relying on liquid-liquid extraction using organic solvents. These conventional techniques usually involve extended contact with large volumes of solvents or mixture of solvents, high temperatures; They often result in relatively low extraction yield, poor selectivity and in the necessities for further purification steps. Thus, both scale-up and cost-effectiveness of the production of natural pigments at industrial scale are yet limited and require optimisation [1].

Herein, we aim to describe emerging extraction and isolation technologies and methods that would support the development of sustainable and cost-effective production of natural products at industrial scale with a particular focus on fungal pigments for food, cosmetic, pharmaceuticals, and fabrics applications. New trends of new green extraction technologies have been developed and techniques have been reinvented by performing association with conventional methods in a more sustainable manner. However, the main limitations are the difficulty to scale up at industrial level due to instability of the methods, the price of the extraction materials (solvents, chemicals and apparatus) as well as the cost for maintenance of the process. Once these would be mastered, promising outcomes are expected.

Filamentous fungi produce a mixture of various metabolites such as pigments, fatty acids, proteins and other cellular metabolites. Thus, extraction and isolation of the pigmented molecules of interest are necessary steps before proceeding to any further utilization of these metabolites for commercial applications. Pigments can be stored within the biomass, excreted in the fermentation broth or both, suggesting that extraction methods need to be developed accordingly to the cellular localisation of the pigments and to their chemical natures. Despite the wide range of separation techniques currently available, most of industrial extraction processes rely on conventional liquid-liquid systems using organic solvent (hexane, dichloromethane, toluene, THF, etc.). Since environmental pollution generated by chemical industries has become a central concern in societal and political decisions, there is a growing demand for the development of efficient and greener separation technologies. Over the last two decades, and with the emergence of the concept of green chemistry, tremendous progresses have been performed in extraction and separation technologies. Methods such as ultra-sound-assisted extraction, microwave-assisted extraction, pressurised fluid extraction, supercritical fluid extraction, enzyme-assisted extraction, and more recently, ionic liquid assisted extraction were developed. The trend is to create extraction techniques involving cheap and environmental-friendly solvents that can be used under milder conditions with good final efficiency in quicker processes (ideally automatized). From the newest systems, ionic liquids have been recognized as very promising new extraction solvents due to the possibility to tailor their chemical properties and affinities to the target biomolecules. The focus of this work is to present an overview of the potentialities of the main emerging extraction methods on fungal matrixes for pigments isolation and recovery, with a particular interest for ionic liquids.

BOOK of ABSTRACTS

Fig, 1 :Fig. 2 :Fig. 4 :Fig. 5 :

 1245 Fig, 1: Schematic description of a Pressurised Fluid Extraction apparatus (A) and a CO 2 -Supercritical Fluid (B) apparatus[2]

Fig. 3 :

 3 Fig. 3: Schematic description of an IL-based aqueous micellar two-phase system



  Tuneable to the molecule to be extracted with high selectivity  Mild conditions (room temperature)  High biocompatibility  Can be associated with other techniques (MAE, US)  Possibility of recycling the IL • Limits for industrial applications:  Very limited recovery yield of biomolecule from the IL-ATPS  Cost of IL

PIF2016 a great opportunity

  to visit the old academic city Cluj-Napoca, to enjoy the traditional food and life in the beautiful Transylvanian landscape and to discover its cultural heritage!

en I n t e r n a t i o n a l C o n g r e ss

	Cluj-Napoca ROMANIA a n l o C i t a n F r e t n I d o o g o n g r e s s o n P i n i m s i n F o o s t o i m ent g P n d
	8th International Congress
	"Pigments in Food"
	Colored Food for Health Benefits
	BOOK of ABSTRACTS
	28th of June -1st of July 2016
	Cluj-Napoca, Romania

Editors: Prof. dr. Carmen Socaciu Prof. dr. Adela Pintea

The publication include abstracts of oral and poster presentations.The editors does not assume any responsibility regarding the content of the abstracts, the authors are responsible for the content originality. Author(s) have the copyright ownership of the data presented in the abstracts book.

IV Pigments

in Food-VIII (2016) in Cluj-Napoca, Romania are dedicated to health benefits! PREFACE On

  behalf of the Organizers of the Eight International Congress PIGMENTS IN FOOD, it is our pleasure to welcome you, at the Department of Chemistry and Biochemistry at the University of Agricultural Sciences and Veterinary Medicine Cluj-Napoca, Romania.

	After seven successful Congresses "Pigments in Food", starting in Sevilla, Spain (1999)
	and passing through Lisbon, Portugal (2002), Quimper, France (2004), Stuttgart-
	Hohenheim, Germany (2006), Helsinki, Finland (2008), Budapest, Hungary (2010), and
	Novara, Italy (2013), we are honored to host this edition, PIF2016 in the capital or
	Transylvania region, Romania.
	The program includes 5 plenary lectures and 7 keynote lectures by best international experts in the field, as well as 20 oral presentations and more than 65 poster contributions. An important aim of PIF2016 is also represented by the opportunity of meeting and discussion for scientists dealing with different aspects of pigments, such as chemists, food chemists, food technologists, biotechnologists, nutritionists, but also industry representa-tives. A comprehensive scientific program, adapted to the new approaches and trends in this scientific area was designed and realized -such approach had a particularly strategic aim, to demonstrate that and how pigments and colored food bringhealth benefits. We are delighted to offer you, meanwhile, the opportunity to visit the historical city of SCIENTIFIC PROGRAM -
	Cluj-Napoca, to enjoy the traditional food and life in the beautiful Transylvanian landscape
	and to discover our cultural and academic heritage!
	On behalf of the Organizing Committee,
	Carmen Socaciu -chair
	Adela Pintea -co-chair

It was the best opportunity for scientists from all over the world (17countries: UK, USA, Germany, France, Reunion Island, Spain, Italy, Denmark, Hungary, Poland,Turkey, Brazil, Uruguay, Panama, Mexico, Chinaand Romania) to show their advances in scientific and technological topics,related to plant pigments and food colorants. The 6 sessions were well adapted to cover the most important and outstanding directions of research and technology development in this research e.g. chemical characterization and bioprospecting, to advanced bioanalytical methods, (bio)technological production and formulation, pigment processing and stability, bioactivity, functionality and health effects, as well quality and safety aspects.

PIF 2016 at USAMV, Cluj-Napoca Tuesday June 28 th Wednesday, June 29 th Thursday, June 30 th Friday, July 1 st

	Session 6	Quality and safety of food pigments	9.00-9.30 Key note lecture 7: Manfred	Eggersdorfer	9.30-10.00 Oral presentations (15min)	Michael Murkovic, Antoanela	Patras	10.00-10.35 Selected poster presentations (6	x 6 min)
	Session 4	Food processing, pigments stability and color	development	8.30-9.15 Plenary lecture 5: Monica Giusti	9.15-9.45 Key note lecture 5: Laurent Dufossé	9.45-11.00 Oral presentations (15 min)	Kewei Chen, Mihaela Turturică, Miriana	Durante,	Pu Jing, Marco Arlorio
	8.00-9.00 Registration		9.00-9.45 Plenary lecture 2: Neal Craft	9.45-10.00 Welcome EUCHEMS FCD -Marco Arlorio	Session 1A	Chemical characterization and Bioprospecting	10.00-10.30 Key note lecture 1: Antonio Meléndez Martinez	10.30-11.00 Oral presentations (15min):	Jozsef Deli, Mekala Venkatachalam	11.00-
										14.00-18.00 Registration	16.30-17.00	Opening Ceremony	17.00-17.45	Plenary lecture 1:	George Britton	17.45-19.00	Poster session	19.00	Welcome party

11.30: Coffee Break & Poster session 11.00-11.30 Coffee Break & Poster session 10.35-11.15 Coffee Break & Poster session

	Session 7	Round Table	11.15-12.15 Presentation of EU projects:	COST EUROCAROTEN	FP-7 DISCOvery	PROCAR	12.15-13.00 Poster
	Session 5	Bioactivity, functionality and health effects	11.30-12.00 Key note lecture 6: Begoña Olmedilla-	Alonso	12.00-13.15 Oral presentations (15 min)	María Roca, Flore Depeint,	Antonio Pérez-Gálvez, Marcello S.	Lenucci, Gábor Palotás
	Session 1B	Chemical characterization and Bioprospecting	11.30-12.00 Key note lecture 2: Dámaso H.Méndez	12.00-12.30 Oral presentations (15min)	Judith Müller-Maatsch, Işıl İlter

Awards and Closing ceremony 12.30-13.30 Lunch break 13.15-14.15 Lunch break and Poster session 13.00-14.00 Lunch 13.30-14.00 Poster session 14.30 Transylvanian Landscape Trip: from Salt Mine Turda to Coltesti, a typical Transylvania village (including Dinner) Optional social events:

	City Tour	and concerts at International Festival "Jazz in	the Park"
	14.00-14.45 Plenary lecture 3: Øyvind Andersen	Session 2	Advanced bioanalytical methods	14.45-15.15 Key note lecture 3: Daniele Giuffrida	15.15-15.45 Oral presentations (15min)	A. Ligia Focsan, Luis E. Rodriguez-Saona	15.45-

16.15 Coffee Break & Poster session 16:45-17:30 Plenary lecture 4: Gerhard Sandmann Session 3 Emerging (bio)technologies of pigment production and

  Quality Lab., Department of Nutrition & Food Science, UniversidaddeSevilla Seville, Spain 10.30-10.45 O1. Carotenoid composition and biosynthesis of β-carotenoids in red mamey (Pouteria Sapota) József Deli, Erika Turcsi, Attila Agócs, Veronika Nagy, Gergely Gulyás-Fekete, Enrique Murillo. Department of Biochemistry and Medical Chemistry, University of Pécs, Hungary 10.45-11.00 O2. Isolation and characterization of pigments from Talaromyces Albobiverticillius: a marine derived fungus producing polyketide red pigments Mekala Venkatachalam, Kristian Fog Nielsen, Daniele Giuffrida, Yanis Caro, Emmanuelle GirardValenciennes, Laurent Dufossé, Mireille Fouillaud Laboratoire de Chimie des Substances Naturelles et des Sciences des Aliments -LCSNSA EA 2212, Université de la Réunion, 15 Avenue René Cassin, CS 92003, F-97744 Saint-Denis Cedex 9, Ile de la Réunion, France 11.00-11.30 Coffee Break & Poster session

		SCIENTIFIC PROGRAMME
		INSTITUTE OF LIFE SCIENCES
		UNIVERSITY OF AGRICULTURAL SCIENCES AND
		VETERINARY MEDICINE CLUJ-NAPOCA
		BLUE HALL
		28.06.2016. Tuesday
	14.00-18.00 Registration
	16.30-17.00 Opening Ceremony
		Chair: Carmen Socaciu
	17.00-17.45 PL1. Reflections from a colour fan.
		George Britton
		University of Liverpool, School of Biological Sciences, Liverpool L69
		7ZB, U.K.
	17.45-19.00 Poster session
	19.00-21.00 Welcome party
		29.06.2016.Wednesday
	8.00-9.00	Registration
		Chair: George Britton -Carmen Socaciu
	9.00-9.45	PL2. Chromatography tool box for the separation of pigments.
		Neal E. Craft
		Craft Technologies, Inc., Wilson, NC USA
	9.45-10.00	Welcome EUCHEMS FCD
		Marco Arlorio
		Session 1A Chemical characterization and Bioprospecting
	10.00-10.30 K1. Phytoene and phytofluene in the diet and health.
		Antonio J. Meléndez-Martínez
		Food Colour &	formulation	17.30-18.00 Key note lecture 4: Paul Fraser	18.00-18.30 Oral presentations	Philippe Thonart, Stefan Martens	20.00 Gala Dinner in Cluj-Napoca
		VI

Acknowledgements:

This work was financed by the University of Reunion Island and the Conseil Régional de La Réunion, COLORMAR project (DIRED / 20140704).


Guillaume Serrière
email: guillaume.serriere@telecomnancy.netcerisara

Christophe Cerisara

Dominique Fohr

Odile Mella
email: mella@loria.fr

Weakly-supervised text-to-speech alignment confidence measure

This work proposes a new confidence measure for evaluating text-to-speech alignment systems outputs, which is a key component for many applications, such as semi-automatic corpus anonymization, lips syncing, film dubbing, corpus preparation for speech synthesis and speech recognition acoustic models training. This confidence measure exploits deep neural networks that are trained on large corpora without direct supervision. It is evaluated on an open-source spontaneous speech corpus and outperforms a confidence score derived from a state-of-the-art text-to-speech aligner. We further show that this confidence measure can be used to fine-tune the output of this aligner and improve the quality of the resulting alignment.

Introduction

This work focuses on the text-to-speech alignment (T2SA) task, which consists in temporally aligning a given speech sound file with its known text transcription. The standard objective quality metric is the expected alignment error, measured in seconds and defined as L = E[| t -t|], where t is the gold timestamp of a word boundary, and t the corresponding timestamp estimated by the aligner [START_REF] Keshet | Phoneme alignment based on discriminative learning[END_REF].

Text-to-speech alignment is an important task for many applications, including: (i) Lip-syncing in cartoons production and film dubbing; (ii) Anonymization of audio corpus; (iii) Pre-processing of audio corpora for training new speech recognition systems; (iv) Indexing audio-visual corpora for browsing and querying; (v) Sampling sounds for speech synthesis; (vi) Second-language learning.

We propose in this work a novel confidence measure for detecting erroneous word boundaries at the output of an existing T2SA system. Accurately detecting misplaced word boundaries is crucial to reduce post-processing costs in every previous application. For instance, only the most reliable segments may be chosen for acoustic model training and speech synthesis, and manual corrections may be limited to the less reliable boundaries for lip syncing and corpus anonymization.

The proposed confidence measure is computed by a deep neural network (DNN) that is trained on a large corpus without any manually annotated word boundaries. We show on a gold corpus of French spontaneous speech that the proposed model is able to detect correct boundaries with a significantly better accuracy than the acoustic models used in the T2SA system, thanks to the acoustic features automatically captured by the deep neural model on the large unlabelled corpus. We further show that the proposed confidence measure may be used to post-process the T2SA output and improve its precision.

Related works

Every text-to-speech aligner faces three main challenges: (i) Handling imperfect transcriptions; (ii) Supporting noisy acoustic conditions; (iii) Finding the globally optimal alignment on long (up to a few hours) audio files. Many solutions have been proposed to address these issues. For instance, "anchorbased" approaches [START_REF] Moreno | A recursive algorithm for the forced alignment of very long audio segments[END_REF][START_REF] Jong | Automated speech and audio analysis for semantic access to multimedia[END_REF][START_REF] Hazen | Automatic alignment and error correction of human generated transcripts for long speech recordings[END_REF] automatically infer highconfidence words timestamps at regular intervals in a long audio file in order to enable regular batch alignment between two successive anchors. The issue of aligning highly imperfect text to speech may be addressed with standard acoustic adaptation [START_REF] Zhao | Refining phoneme segmentations using speakeradaptive context dependent boundary models[END_REF], or by performing recognition at the phoneme level only with monophthongs and fricatives, which appear to be more robust to noise than other phonemes [START_REF] Haubold | Alignment of speech to highly imperfect text transcriptions[END_REF]. Complementary, better phonetization models of unknown words may also be used [START_REF] Bigi | A phonetization approach for the forced-alignment task[END_REF]. More generally, various models have been proposed for phoneme alignment, such as discriminative shallow large-margin alignment models in [START_REF] Keshet | Phoneme alignment based on discriminative learning[END_REF], nonneural unsupervised acoustic models in [START_REF] Milde | Unsupervised acquisition of acoustic models for speech-to-text alignment[END_REF] and in [START_REF] Lanchantin | The development of the cambridge university alignment systems for the multi-genre broadcast challenge[END_REF], where two DNNs are used respectively for acoustic modelling for speech transcription and for segmenting the speech file into speech and non-speech segments. A remarkable HMM-based architecture is also proposed in [START_REF] Brognaux | Hmm-based speech segmentation: Improvements of fully automatic approaches[END_REF], where the acoustic models are trained solely on the target corpus to align. The authors of [START_REF] Yuan | Automatic phonetic segmentation using boundary models[END_REF] demonstrate the importance of producing high-precision temporal limits with dedicated models, and propose in [START_REF] Stolcke | Highly accurate phonetic segmentation using boundary correction models and system fusion[END_REF] a neural network to fine-tune the alignment. We follow this line of work, but rather focus on estimating the actual quality of the proposed boundaries with a confidence measure. A confidence measure is proposed in the aligner ALISA [START_REF] Stan | Alisa: An automatic lightly supervised speech segmentation and alignment tool[END_REF], but its role is to filter-out wrongly recognized sentences. Conversely, few publications address the problem of detecting reliable temporal boundaries after T2SA. [START_REF] Paulo | Automatic phonetic alignment and its confidence measures[END_REF] proposes a confidence measure that is based on a synthetic speech signal, while [START_REF] Keshet | Phoneme alignment based on discriminative learning[END_REF] discriminatively trains base functions that define an alignment confidence measure, but which is not evaluated per se: thanks to the decomposability property of the base functions, this measure is rather used with a dynamic programming algorithm to output a final alignment. Our work is, to the best of our knowledge, the first proposal to use the modelling potential of deep networks to compute successful confidence measures of text-to-speech alignment outputs.

3 Proposed model

Model description

The proposed model is shown in Figure 1. Two models, respectively called the Boundary inspector and the Boundary selector, are built to compute a confidence measure that any candidate word boundary is correct or not. Both the Boundary inspector and selector take as input an acoustic window of ±0.05s around the candidate word boundary, plus two categorical inputs representing the left and right phonemes. They can thus be viewed as acoustic models that are specialized in identifying boundaries between two segments, as opposed to classical acoustic models that are designed to discriminate between phonemes that may generate a given segment.

The Boundary inspector is a standard feed-forward deep network with two output neurons, which encodes the probability that the central input frame1 t is a true word boundary. It thus focuses on a single frame, the middle one, and makes a decision about it. It is completed with another model, the Boundary selector, which rather considers simultaneously all possible candidate frames in the interval t ± 5, and decides which one is the most likely to be the target word boundary. Because we know that consecutive frames are more correlated than distant frames, we use a recurrent LSTM model to capture correlation between frames. Because no privileged direction is assumed, we use a bi-directional LSTM. The output of this LSTM is then merged with the contextual phonetic information in a feedforward network with 11 outputs: one for each input frame. Both models are finally combined with a deep feedforward network called Aggregator, which is trained separately on another corpus.

Training

Our choice to use a Deep Neural Network (DNN) is motivated by the potential of deep networks to infer complex hierarchical features from data that would have been difficult to design by hand. But this is only possible on large training corpora, while only our small gold corpus is manually annotated with temporal boundaries. We thus have to rely on one of the common deep learning "tricks" for building a large enough training corpus, such as transfer learning, the use of auxiliary tasks or data augmentation. In this Figure 1: Proposed model. The input vector is composed of 11 frames plus 2 phonemes, shown in the center. Below, the Boundary inspector detects whether the middle frame t is a true word boundary or not. Above, the Boundary selector is composed of a bi-directional LSTM with 11 timesteps (only 5 are shown) plus 3 feed-forward layers that select the most likely word boundary frame in the input segment. Both models outputs are fed into the Aggregator model, on the right, which outputs a confidence probability that frame t is a word boundary.

work, we have decided to combine two state-of-the-art French text-to-speech aligners, ASTALI [START_REF] Fohr | De l'importance de l'homogénéisation des conventions de transcription pour l'alignement automatique de corpus oraux de parole spontanée[END_REF] and JTrans [START_REF] Cerisara | Jtrans, an open-source software for semi-automatic text-to-speech alignment[END_REF] 2 in order to align part of the open-source ORFEO corpus, composed of 3 million words of French spontaneous speech manually transcribed and available at http: //www.projet-orfeo.fr.

We then compare both ASTALI and JTrans alignments on this corpus and consider that any word boundary that has the same timestamp in both alignments, within a tolerance of ±0.02s, is correct 3 . This procedure allows us to automatically build a large training corpus of positive examples which is then completed with 3 times more negative instances obtained by randomly sampling frames that are distant from any ASTALI and JTrans word boundary by at least 0.04s, leading to a training corpus of 377662 examples, which is used to train both the Boundary inspector and selector.

The same process is used on another set of files from the ORFEO corpus to create a second training corpus of 105406 examples, on which both model output probabilities are computed and used to train the Aggregator. During the training of each model, 20% of the training corpus is further reserved to compute a validation loss.

The evolution of the training and validation loss for the three models is shown in Figure 2. These curves suggest that overfitting is not a major issue at this stage. The hyper-parameters of the DNN, including the number and size of the layers, have been set-up empirically with a few trials and errors 2 JTrans is available on github https://github.com/synalp/jtrans and ASTALI is released by its authors 3 The exact timestamp chosen for this positive temporal limit is the average of the timestamps proposed by JTrans and ASTALI. The tolerance of 0.02s is standard in the phoneme alignment literature [START_REF] Hosom | Speaker-independent phoneme alignment using transition-dependent states[END_REF]. on the training and validation corpus. In particular, we have not used any automatic hyper-parameter tuning strategy. This search of an appropriate model topology has mainly been driven by our motivation to design an architecture that is deep enough to model rich transformations and at the same time that limits the number of parameters to prevent overfitting. Note however that we have only tried a few alternative hyperparameters and thus that the proposed topology is certainly not optimal. The DNN has been implemented with Keras [START_REF] Chollet | Keras[END_REF] and trained on these positive and negative instances with the ADAM stochastic gradient descent for 150 epochs.

Test

The proposed system is evaluated on a gold corpus that is composed of 10988 words extracted from the original ORFEO corpus, and for which 16264 word boundaries, obtained with ASTALI, have been manually corrected. There is no overlapping between this gold corpus and the previous corpora. At test time, JTrans is run on the test corpus to compute candidate temporal limits of words.

An example of inputs/outputs is shown in Figure 3. Let t be a temporal word boundary4 given by JTrans, with h l and h r respectively the left and right phonemes that are separated by t. For instance, in Figure 3, t = 186, h l =õ and h r =s. The Boundary inspector and selector are run on the temporal limits proposed by JTrans and their output probabilities are then passed to the Aggregator, which finally returns, for each JTrans temporal limit t, the probability that t is correct or not.

Evaluation

Confidence measure evaluation

Similarly to most other confidence measures in the literature [START_REF] Yu | Calibration of confidence measures in speech recognition[END_REF], we evaluate the proposed confidence measure as a detector of correct vs. erroneous examples. We evaluate next its performances with a Detection Error Tradeoff (DET) curve, which is easier to interpret than the ROC curve [START_REF] Martin | The det curve in assessment of detection task performance[END_REF].

We compare our proposed model first with a baseline confidence measure derived from the acoustic Hidden Markov Models (HMM) used in the text-to-speech alignment process. Let (w i ) 1≤i≤N be the sequence of words in the transcription. For ease of notation, we assume here without loss of generality that every word is modelled with a single-state Hidden Markov Model (HMM); in fact, every word is actually composed of a sequence of phonemes, and every phoneme is modelled by an HMM with 3 emitting states. However, this hierarchy of models would lead to excessively long equations, and we prefer to simplify the presentation of this baseline.

For a given possible alignment, let the random variable Q t = i with 1 ≤ i ≤ N represents the index of the word aligned with frame t. By definition, in the context of text-to-speech alignment, a confidence measure for the transition (Q t = i, Q t+1 = i + 1) is given by the posterior probability:

P (Q t = i, Q t+1 = i + 1|X, λ)
where λ represents the parameters of the acoustic models used in JTrans, and X = (X t ) 1≤t≤T represents all observed acoustic frames.

For our baseline confidence measure, we rely on the acoustic models used in the JTrans system. These acoustic models are Hidden Markov Models, and it is thus well known that the previous posterior can be computed with the forward-backward algorithm:

P (Q t = i, Q t+1 = i + 1|X, λ) = α i (t)a i,i+1 β i+1 (t + 1)b i+1 (X t+1 ) N k=1 N l=1 α k (t)a k,l β l (t + 1)b l (X t+1 )
where b i (X t ) is the observation likelihood of frame X t in state i. b i (X t ) is modeled in JTrans by a Gaussian Mixture Model. a i,j = P (Q t+1 = j|Q t = i) is the prior transition probability between words i and j, which is irrelevant in the context of text-to-speech alignment, where we just set a i,i = 1 2 and a i,i+1 = 1 2 . α and β are respectively the matrices of forward and backward probabilities, which can be computed recursively:

α j (t) = N i=1 α i (t -1)a ij b j (X t ) β i (t) = N j=1 a ij b j (X t+1 )β j (t + 1)
We first evaluate the quality of the DNN as a detector of correct limits, assuming that any JTrans output boundary is a correct limit when its distance to the corresponding gold limit is smaller than 0.02s, as done during training. The corresponding DET curve is shown in Figure 4.

In the DET plot, the closer the curve is to the bottom-left origin, the better it is. We can observe that the proposed confidence measure is a better detector of true boundaries than the acoustic baseline for all possible detection thresholds. The first row in Table 1 also shows the Equal Error Rate (EER), which is the intersection between the y = x diagonal and the DET. With 36% of equal errors, the proposed confidence measure is significantly better than random and it is the first efficient confidence measure for word boundaries based on acoustic information that we are aware of.

While the EER is a good summary of the DET curve, it can only be computed assuming knowledge of the true labels. The next rows in Table 1 thus show standard detection performances at another operating point, the median, which corresponds to the threshold that tags half of the corpus as positive, and half as negative. The proposed model is then compared with a second baseline, called JTrans/ASTALI agreement, which tags every JTrans boundary as positive when it lies within the ±0.02s interval around the corresponding ASTALI limit. This baseline has already been used to automatically annotate the training corpora of the DNN models (see Section 3.2), except that for training, all boundaries tagged as negative are removed, while they are used here to compute the detection metrics in Table 1.

The acoustic baseline confidence measure is not significantly better than random, which confirms for text-to-speech alignment what has already been reported in the literature for speech recognition, i.e., that 1: Detection performances at fixed operating points of the proposed DNN and two baselines: an acoustic baseline, which computes the posterior of each boundary given JTrans' acoustic models, and a deterministic baseline that tags a boundary as correct when JTrans and ASTALI give close timestamps.

confidence measures based solely on acoustic observation likelihoods usually fail to reliably detect correct words [START_REF] Willett | Confidence measures for hmm-based speech recognition[END_REF]. This is why state-of-the-art confidence measures for speech recognition mainly exploit other types of features, in particular language-model features [START_REF] Seigel | Confidence Estimation for Automatic Speech Recognition Hypotheses[END_REF]. However, language-model information is irrelevant in text-to-speech alignment applications, which makes the task of detecting reliable word boundaries especially challenging. With an F1 of 60%, the performances obtained with our proposed DNN-based confidence measure are thus encouraging, because:

• Our DNN only exploits the same information as speech acoustic models, i.e., acoustic observations and phoneme identities;

• It is trained without manual supervision, only exploiting agreement between two automatic T2SA systems.

• Despite the relatively weak precision of 60% for annotating positive labels in the DNNs' training corpus, the DNN is able to learn relevant acoustic information and provide the first working confidence measure for detecting true word boundaries.

The JTrans/ASTALI agreement baseline has a low recall of 45%. Although this low recall penalizes its performances as a confidence measure, we can note that the recall is actually not crucial when this JTrans/ASTALI agreement approach is used to automatically annotate training examples for the proposed deep models, because all negative limits are discarded, as explained in Section 3.2. In fact, it may even be preferable to tune this automatic annotation method so that its precision is further increased, at the expense of an even lower recall, so that the positive examples that are kept have a higher likelihood of being correct. However, the JTrans/ASTALI agreement baseline may not easily be tuned in order to increase its precision above 60%. An interesting future work would then be to replace this agreement process with another detector, like the proposed DNN itself, for which the operating point can be tuned, and eventually iteratively retrain the DNN in a self-training fashion on larger corpora, without the need to rely on two different aligners.

Enhanced aligner evaluation

We propose next a simple post-processing module that enhances the precision of the original T2SA system. This fine-tuning algorithm basically detects suspicious JTrans temporal limits and replaces them by temporal limits with a higher confidence measure in their neighbourhood. It proceeds as follows:

Algorithm 1: Simple fine-tuning of JTrans' output alignment Figure 5: Fine-tuning example. The Jtrans initial limit is 1.86s. The DNN output probability is computed for frames (185,187) first, and then for frames (184,188). The best output is obtained for 1.84s.

Figure 6 plots the original (top horizontal dashed line) and resulting alignment error for various D and δ. Although the global impact of our fine-tuning algorithm is small, it is positive for all D and δ. Because our fine-tuning algorithm just looks for the most confident limits in a neighbourhood of the original JTrans boundary, the iterative application of the confidence measure onto more and more distant frames increases the probability of misclassification. Furthermore, whenever it moves a word boundary, the resulting impact on the previous or following words should be handled, for instance with a Viterbi algorithm. So this experiment is merely a proof of concept that confirms the possibility to post-process a text-to-speech aligner output with the proposed confidence measure; the main focus of this work is rather confidence measure evaluation, which may benefit to many other applications, as discussed in the introduction. These results are thus encouraging to further pursue efforts into investigating weakly supervised deep neural networks for fine-tuning existing text-to-speech aligners.

CONCLUSIONS AND FUTURE WORKS

We develop a weakly supervised approach that exploits two existing text-to-speech aligners to automatically annotate a corpus for training a deep neural network-based confidence measure without direct supervision. We propose two different types of neural networks for this task and combine them within a single model. For now, all three components of the proposed model are trained independently, but we plan to train them jointly in a future work to further improve the resulting model. Experimental results show that the proposed confidence measure outperforms a baseline acoustic confidence measure derived from the original text-to-speech aligner. We further show that it outperforms another baseline, which results from a voting ensemble of both original text-to-speech aligners. This is, to the best of our knowledge, the first good performances ever reported for confidence measure detection of true word boundaries. The performances reached are also interesting because the deep models only exploit acoustic information, which has been shown to be otherwise unsuccessful for confidence estimation in the context of speech recognition, and because these models are trained without manual supervision. These results open the way to further improvements in automatic annotation of unlabelled corpora for text-tospeech alignment, for instance by iteratively re-labelling the training corpus with the proposed model setup in high-precision mode and retraining new confidence models. We further apply the trained confidence measure with a simple corrective algorithm that fine-tunes the output timestamps given by the original text-to-speech aligner. This experiment shows encouraging results for improving text-to-speech alignments thanks to the proposed confidence measure. Possible ways to improve these results include designing a better exploration strategy for fine-tuning the initial alignment, as well as investigating other DNN topologies.

The complete source code as well as links to all datasets is available at https://github.com/ cerisara/speechAlignConfidence.

Figure 2 :

 2 Figure 2: Loss curves during training of the models.

Figure 3 :

 3 Figure 3: Example of a segmented sentence with its context for the fragment: "no, that's a bit [...]"

Figure 4 :

 4 Figure 4: Detection Error Tradeoff curves for detecting word boundaries. The X-axis is the False Accept Rate, while the Y-axis is the False Reject Rate. Three curves are shown, from the worst (top-right corner) to the best (bottom-left corner): random baseline detector (straight grey line), baseline (dash) and our proposed DNN (plain line).

Figure 6 :

 6 Figure 6: Average alignment error (Y-axis, in seconds) for various maximum distance D (from ±1 to ±8 frames), as a function of the minimum confidence threshold (X-axis). The top dashed horizontal line is the original JTrans alignment error.

•

  For every JTrans output word boundary t:-For every distance d ∈ 1, 2, 3, 4, 5, • • • , D up to a maximum distance D: * Compute both DNN output probabilities at distance d from t: P (t -d) and P (t + d) * Pick the best of both frames t = arg max t ∈t-d,t+d P (t ) * If the new frame is better than the original JTrans frame P ( t) > P (t) and better than a minimum confidence threshold P ( t) > δ, then move the word boundary to t and continue with the next boundary t.

		Gold		Jtrans
	t-4	t-3	t-2	t-1	t	t+1	t+2	t+3	t+4
	1.83	1.84	1.85	1.86	1.87	1.88	1.89	1.90
		DNN P(t-2)=0.95	DNN	DNN P(t+2)=0.03
				P(t)=0.15

A frame is a time-segment of 10ms length encoded into an acoustic vector of dimension 39 composed of 12 MFCC (Mel-Feature Cepstral Coefficients) plus their derivative and acceleration.

Time variables such as t represent an integer number of frames since the start of the audio file. Hence, t + 2 is the second frame after the JTrans limit t.

Acknowledgments

This work has been partly funded by the ANR ORFEO project. Some experiments presented in this paper have been made possible thanks to the donation of a GPU Titan X card by Nvidia, and were carried out using the Grid'5000 testbed, which is supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https: //www.grid5000.fr).


Gen Nishida

Ignacio Garcia-Dorado

Daniel G Aliaga

Bedrich Benes

Adrien Bousseau

Interactive Sketching of Urban Procedural Models

Keywords: Inverse Procedural Modeling, Sketching, Machine Learning Concepts:, Computing methodologies → Computer graphics, Shape modeling, •Theory of computation → Computational geometry

a) Sketch b) Suggested snippets c) Generated building ⋯ ⋯ ⋯ ⋯ ⋯ d) Rendering of city corner Figure 1: Procedural model sketching.

Our interactive sketching tool allows the user to quickly and easily author procedural 3D building models. The user only performs interactive sketching and our system automatically generates the procedural model and its parameters yielding the sketched shape. Our system partitions the design process of a 3D model into sketching various object parts that together form the overall model. a) The user sketches a few strokes of the current object type to specify the desired shape. For instance, (top row) the user sketches the shape of the building mass. b) Using a deep-learning based method, the system finds which pre-defined grammar snippets yield visually similar shapes and chooses the best one by default (red square). c) The selected snippets and their parameter values are merged into a single grammar that represents the entire building. d) An offline rendering of variations of the procedural building model generated automatically by changing the snippets and/or parameters that encode them.

Introduction

Object design and modeling is a crucial skill in various areas of entertainment, science, and engineering. Designers wish to create custom objects, seek variations of existing models, and explore new uncharted designs. However, designing geometric models is notoriously hard and unintuitive. It often requires extensive learning that makes it difficult especially for novices. Although humans can quickly draw and sketch in 2D and we are well-trained to do those tasks since childhood, the transition to 3D is difficult for automated processes.

Prior work has addressed the modeling problem from various directions. Sketch-based modeling [START_REF] Olsen | Sketch-based modeling: A survey[END_REF]] exploits human intuition and experience in drawing objects. Nevertheless, the quality of the 3D model depends on the sketching skills of the user, the details added to the drawing, and the ability to resolve inherent ambiguities of the sketching process. Another important and popular direction is procedural modeling [START_REF] Smelik | Interactive creation of virtual worlds using procedural sketching[END_REF][START_REF] Vanegas | Modelling the appearance and behaviour of urban spaces[END_REF]] that has been successfully applied to the creation of detailed and complex cities [Parish and Müller 2001], realistic and growing vegetation [START_REF] Prusinkiewicz | The algorithmic beauty of plants[END_REF], and other objects [START_REF] Ritchie | Controlling procedural modeling programs with stochastically-ordered sequential monte carlo[END_REF]]. However, procedural modeling is difficult to control and thus hard to use as an exploratory design tool making it accessible only to experts.

Our approach is to use machine learning to seamlessly merge procedural modeling and interactive sketching, thus enabling an interactive design process leveraging both the intuitiveness, freedom, and flexibility of sketching and the precision, exactness, and detail amplification [START_REF] Smith | Plants, fractals, and formal languages[END_REF]] of procedural modeling. The user does not need to specify tedious procedural rules or rule parameters; instead, they are recognized from the sketch automatically thus enabling untrained users to quickly create complex procedural models.

Consider a user who begins sketching a 3D model in 2D by using a mouse or a digital pen on a tablet. Inspired by the way artists draw the general structure before adding details [START_REF] Loomis | I'd Love to Draw[END_REF], our approach allows the user to progressively add details by sketching various object types (e.g., building mass, roof, windows) from coarse to fine. Each object type is supported by multiple predefined grammar snippets. The partitioning into object types is not explicitly predetermined by our approach; rather it results from the types associated with each of the provided snippets. The user can select the next object type to sketch using gestures or via GUI buttons. For each object type, the user begins sketching a few strokes until a match with a snippet is found.

In order to achieve interactivity, we use convolutional neural networks (CNNs) to identify the grammar snippet and snippet parameter values that best explain the current user sketch. The CNNs are trained by generating many instances of each snippet and by sampling its parameters and rendering them in a sketch style. We employ a cascade of networks, where a first network recognizes the snippet, while a second snippet-specific network estimates its parameters. At run-time, we feed the CNNs with the user sketch to almost instantaneously recover the intended snippet and parameter values.

The final output is a single generated grammar composed of an assembly of grammar snippets collectively representing the sketched 3D model. The user can render the procedurally-generated model at any moment in one of a variety of rendering styles (e.g., sketchstyle rendering, ambient occlusion, or shaded and textured rendering). Further, the result is not limited to a single instantiation of a model. Rather, we exploit the procedural nature of our system to create variations that are inspired by the originally-sketched object.

We demonstrate our approach in the context of urban modeling. Nonetheless, our system is applicable to other modeling domains as well. Our approach does not make any particular assumptions about the 3D model except assuming it can be decomposed into a set of object types and a set of pre-defined snippets per type.

Our main contributions include

• combining sketching and procedural modeling for designing complex 3D models without having to explicitly sketch all details and without having to write the underlying procedural grammars, and • using a cascade of CNNs that enables interactive classification of partial sketches to one of multiple possible grammar snippets and estimating their parameters.

Previous work

Our work relates to research in sketch-based modeling, examplebased modeling, and forward and inverse procedural modeling. Sketching attempts to make 3D modeling as direct and intuitive as drawing [START_REF] Olsen | Sketch-based modeling: A survey[END_REF]]. However, recovering a 3D model from a 2D drawing is fundamentally ill-posed because strokes do not provide depth information. Early approaches made this problem wellposed by assuming that the lines in the drawing obey specific geometric constraints in 3D, or by using a well-defined set of gestures to specify one of a set of primitive shapes (e.g., [START_REF] Zeleznik | Sketch: An interface for sketching 3d scenes[END_REF]). For smooth shapes, the lines can be assumed to denote contours and silhouettes [START_REF] Igarashi | Teddy: a sketching interface for 3d freeform design[END_REF], while for polyhedrons geometric relationships such as parallelism, orthogonality and planarity can be detected and imposed [START_REF] Lipson | Optimization-based reconstruction of a 3d object from a single freehand line drawing[END_REF], or even learned from line-renderings of 3D models [START_REF] Lipson | Conceptual design and analysis by sketching[END_REF]. Unfortunately, such assumptions only hold for a limited family of shapes. Recent methods allow the creation of complex freeform shapes by exploiting geometric constraints present in pro- fessional design drawings, such as polyhedral scaffolds [START_REF] Schmidt | Analytic drawing of 3d scaffolds[END_REF]] and cross-section lines [START_REF] Xu | True2form: 3d curve networks from 2d sketches via selective regularization[END_REF]. Interactive systems also rely on axis-aligned planes and other transient surfaces such as 3D canvases to support 3D strokes [START_REF] Bae | Ilovesketch: as-natural-as-possible sketching system for creating 3d curve models[END_REF][START_REF] Zheng | Smartcanvas; context-inferred interpretation of sketches for preparatory design studies[END_REF]. Since all the above methods derive constraints from the drawn lines, they require relatively accurate drawings as input. In addition, these methods only reconstruct what is drawn, which means that users have to draw very detailed sketches to obtain detailed 3D models. In contrast, our system relies on procedural grammars as a strong prior to regularize inaccurate and ambiguous sketches as well as to suggest intricate details from a handful of lines.

Our approach is closer to example-based methods, which reconstruct drawings by fitting parts from a database of 3D models. The database can be composed of generic parameterized shapes such as cubes, cones and cylinders [START_REF] Xue | Example-based 3d object reconstruction from line drawings[END_REF][START_REF] Shtof | Geosemantic snapping for sketch-based modeling[END_REF], or detailed 3D models from a particular domain, such as furniture and vehicles [START_REF] Xie | Sketch-to-design: Context-based part assembly[END_REF] or building parts [Chen et al. 2008b]. Similarly to our offline pre-training, [START_REF] Xie | Sketch-to-design: Context-based part assembly[END_REF] and [START_REF] Eitz | Sketch-based shape retrieval[END_REF] pre-compute non-photorealistic renderings of the 3D models to perform shape retrieval during sketching. The expressiveness of these systems greatly depends on the size of the 3D model database. On the one hand, while generic shapes can be represented compactly with a few parameters, many basic shapes need to be combined to create complex models. On the other hand, large databases of detailed 3D models are expensive to collect and store, and the resulting model is not editable. We offer a middle-ground between these two extremes by matching drawings to procedural grammars, which are compact and editable. Moreover, we do not match a complete model but progressively match parts which are then combined into the final model. This has the additional benefit of needing a relatively small database yet our approach is capable of generating a large variety of detailed 3D models.

Procedural modeling offers an effective way of generating complex, parameterized 3D models [START_REF] Wonka | Instant architecture[END_REF]Müller et al. 2006;[START_REF] Smelik | A survey on procedural modelling for virtual worlds[END_REF]]. Among procedural models, grammarbased models are commonly used in urban modeling and vegetation. Procedural systems can quickly generate many 3D models with wide variation by either changing the grammar or by varying its attributes. However, creating a grammar requires programming expertise and domain knowledge to be able to write compact rules, and setting the parameters of a grammar is non-trivial be-cause of the intricate relationship between the procedural parameters and the output. To address this issue, [START_REF] Lipp | Interactive visual editing of grammars for procedural architecture[END_REF] introduce a visual editor akin to standard 3D modeling software, allowing direct editing of architectural models by selecting and dragging procedural components. Several sketch-based systems have also been proposed for specific domains, such as trees [START_REF] Ijiri | The sketch lsystem: Global control of tree modeling using free-form strokes[END_REF], terrains [START_REF] Smelik | Interactive creation of virtual worlds using procedural sketching[END_REF], and roads [START_REF] Applegate | A sketch-based system for highway design with user-specified regions of influence[END_REF]Chen et al. 2008a]. However, these methods rely on applicationdependent heuristics rather than on a generic sketch-recognition algorithm like ours.

Inverse procedural modeling estimates the parameters of procedural models by minimizing an objective function defined by the user input and the parameter values. Monte Carlo Markov Chain (MCMC) [START_REF] Talton | Metropolis procedural modeling[END_REF][START_REF] Vanegas | Inverse design of urban procedural models[END_REF][START_REF] Stava | Inverse procedural modelling of trees[END_REF] and Sequential Monte Carlo (SMC) [START_REF] Ritchie | Controlling procedural modeling programs with stochastically-ordered sequential monte carlo[END_REF]] are so far the most promising solutions to explore the large parameter space and find near-optimal parameter values. However, these iterative sampling algorithms require many steps to converge, preventing their use in an interactive context. Recently, [START_REF] Emilien | Worldbrush: Interactive example-based synthesis of procedural virtual worlds[END_REF] learned localized procedural models from examples and reused them for sketching virtual worlds. However, their approach is suited for stochastic models and fails to represent structure and its repetition. We exploit recent advances in machine learning to perform inverse procedural modeling at runtime without the cost of iterative optimization. This approach is inspired by recent work on informed samplers that rely on discriminative inference to accelerate MCMC for computer vision tasks [START_REF] Jampani | The informed sampler: A discriminative approach to bayesian inference in generative computer vision models[END_REF].

Our machine learning approach achieves high accuracy in a fraction of the time required by standard MCMC, opening the door to interactive application of inverse procedural modeling. Our approach is enabled by the strength of CNNs. CNN is a type of artificial neural network and has been applied to many areas, such as recognizing hand-written characters [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]], image classification [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], single-image depth estimation [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF], and cross-domain image matching [START_REF] Bell | Learning visual similarity for product design with convolutional neural networks[END_REF]. CNN has also been applied to 3D shape retrieval [START_REF] Wang | Sketch-based 3d shape retrieval using convolutional neural networks[END_REF], which significantly improves the accuracy compared to a feature-based approach [START_REF] Eitz | Sketch-based shape retrieval[END_REF]].

Overview

We provide an overview of our offline training and online sketching phases (Figure 2). The input to our system is a collection of pre-defined grammar snippets (Figure 3). During an interactive design session, the user draws partial or complete sketches which are automatically matched and completed with procedurally-generated content. The final output is a complete 3D model and the corresponding generated grammar.

Snippets: To provide design flexibility, our system supports multiple grammar snippets for each object type of the sketching process. One usual partitioning of buildings into object types is to provide snippets for different building shapes (i.e., building masses), roofs, window styles, ledges, and other facade ornaments (Figure 3). These snippets are arbitrarily combined during sketching thus enabling the construction of a wide variety of building models.

Training: As a preprocess, our system trains a collection of CNNs. The snippets are fed to a data generator which then generates a large number of variations by randomly sampling the snippet's parameter values. The rendered images for each variation are then used to train two types of CNNs: one type for snippet recognition and the other type for snippet-specific parameter estimation.

Sketching: Online sketching is done via a mouse or 3D pen. The designer may interactively draw a new sketch on an empty canvas or use a lasso to select a region (e.g., a face) where to anchor the next sketched part. For a new sketch, the anchored region corresponds to the ground plane and the object type defaults to an agreed upon first object type. In all other cases, the object type depends on the label specified in the snippet for the selected anchor region. Nonetheless, the object type to sketch can be explicitly expressed by the designer. As the user sketches, the system uses the snippet recognition CNN to match the sketch to an instance of a particular snippet type and then uses that snippet's parameter estimation CNN to recover its parameter values. The procedurally-generated model is visualized in one of a variety of rendering styles. At any time the generated grammar of the entire 3D model can be output.

Representation

Without loss of generality, our system implements the grammar snippets and the generated grammar as XML-based split grammars. While our current implementation makes use of split grammars (i.e., as used by [START_REF] Wonka | Instant architecture[END_REF]] and [Müller et al. 2006]), the approach does not depend on the exact grammar type.

Split Grammar

Our system uses split grammars to represent each snippet as well as the entire model. The i'th snippet is defined as Gi = {α, τ, η, ρ} where α is the axiom, τ is the set of terminal symbols, η is a set of non-terminals, and ρ is a set of rewriting rules using the terminals and non-terminals. Moreover, each snippet is labeled with an object type and the axiom α is named Start. The snippet may also contain one or more object-type labeled non-terminals (Figure 4).

Snippet Combination

Every time the user draws new strokes, a best fitting snippet is recognized and added to the grammar representing the 3D model. At this moment, the non-terminal of the current anchor is replaced by the recognized snippet and its automatically determined parameters. For example, when the name of the selected anchor non-ion 4.

<!--grammar for a building mass --> <param building_height="20"/> <rule name="Start"> <extrude height="building_height"/> <comp> <top name="TopFace"/> <!--roof --> <side name="Façade"/> <!--façade --> <bottom name="Base"/> </comp> </rule>

Figure:

Each grammar has one axiom named "Start" and contains one or more nonterminals. Some of the nonterminals will be used by the other stages as their starting shape. In the above example, "TopFace" will be used by the roof stage, and "Façade" will be used by the façade stage.

Figure 4: Snippet Grammar. Each snippet has an axiom named Start and contains one or more non-terminals that can be rewritten by procedural rules (i.e., replaced by other snippets). In this example, TopFace will be used by a roof-type snippet, and Facade will be used by a facade-type snippet.

<!--grammar for a building mass --> <param building_height="20"/> <rule name="Start"> <extrude height="building_height"/> <comp> <top name="TopFace"/> <side name="Façade"/> <bottom name="Base"/> </comp> </rule> <!--grammar for a roof --> <param roof_slope="50"/> <rule name="TopFace"> <!--originally was "Start" --> <roofGable slope="roof_slope"/> </rule> Figure 5: Grammar Combination. When a snippet is recognized, the name of its starting axiom is modified to the name of the nonterminal to which the snippet is anchored. In this way, multiple snippets can be linked into one generated grammar for the entire 3D model.

terminal is TopFace and the user sketches a grammar for a roof, the axiom of a copy of the snippet grammar is modified to TopFace (Figure 5). Also, the object type of this axiom is set accordingly so that the system knows that the added snippet is for a roof in the generated grammar. Afterwards, the derivation tree [Müller et al. 2006] is regenerated producing a new instance of the 3D model.

Training

Our system trains one CNN per object type to recognize snippets and one CNN per pre-defined snippet to estimate parameters. The training process is performed once as a preprocess and during runtime the trained CNNs are loaded and used.

Convolutional Neural Networks

CNNs can be used for both recognition/classification problems [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Bousseau | Convolutional neural networks for speech recognition[END_REF], in which the output is the probability distribution of discrete values for each category, and regression problems [START_REF] Pfister | Flowing convnets for human pose estimation in videos[END_REF], in which the output are continuous valued parameters. Even though it may require a one-time several hours preprocess, once the network is trained, the aforementioned problems can be solved almost instantaneously. Thus, it is suitable for interactive usage (Figure 6).

Training Process

For the recognition CNNs, we use the BVLC AlexNet architecture [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]] starting from the weights obtained from the Caffe Model Zoo [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF]]. The initial weights are obtained by training with hundreds of thousands of images so as to learn to discriminate the features [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF]]. We fine-tune the networks by using our training images so as to optimize the classifiers to achieve high accuracy on our recognition application (Table 1).

For each parameter estimation CNN, we use a modified architecture with three convolutional layers followed by two fully connected layers. Finding the best network architecture is still an open problem especially for regression problems. In practice, deeper convolutional networks can extract more features but are more difficult to train. We use three convolutional layers that achieved satisfactory results. Since there are no pre-trained weights for this network, we train CNNs from scratch, which requires many more iterations to converge (Figure 7). We use normalized parameter values as input for the CNNs. The j'th parameter has a value vj ∈ [vmin, vmax] and the normalized value is computed based on this range by

v j -v min vmax-v min .
Note that the output of the first CNN type is a probability distribution of the snippet types, whereas the outputs of the second CNN types are normalized parameter values.

Data Generation

Our preprocessing tool generates a large number of training images (over 40,000 images for the first object type to sketch, building mass, and around 7,000 images for other object types) for each snippet by randomly changing its parameter values. In an early version of our system, we trained the network with multiple viewpoints. However, we found in practice that most people use a predefined vantage point, such as a three quarter view. Thus, in order to reduce the complexity of snippet recognition and parameter estimation, our system fixes the viewing orientation and position of the sketched objects. We consider the XY plane to be the ground plane in the world coordinate system, θ be the rotation angle around X axis, and φ be the rotation angle around Z axis. Our system enables specifying the viewing orientation for each object type (e.g., for building mass and roof, θ = 30 • and φ = 45 • ; for ledge, θ = 0 • , φ = ρ -72 • ; for other object types θ = 0 • , φ = ρ, where ρ is the Z rotation such that the anchor plane is perpendicular to the camera view direction). The viewing position is fixed such that the projection of the center of the bounding box of the anchored region is located at the center of the drawing canvas. Note that for the agreedupon first object type to sketch (e.g., building masses), the system considers the ground plane as the anchor. Since the anchor can be any shape at any location on the ground plane, the preprocessing tool for training changes the originating position of the snippet on the ground plane to a sampling of potential positions.

Sketching

At run-time, our system uses the pre-trained CNNs to recognize which snippet is being partially sketched. Moreover, additional tools are provided to support object repetition, to generate a variety of similar 3D models, to render using one of several styles, and to provide standard move/resize/copy/paste/delete/undo editing operations.

Recognition and Parameter Estimation

During sketching, the snippet with the highest probability is selected, and then the system performs snippet parameter estimation.

When the user selects an anchor-region for the next sketch, the system automatically alters the viewpoint (Section 5.3) so that the user sketch is best aligned to the training images. Our system resizes the Once the parameter values are estimated, the system inserts or replaces the new snippet into the generated grammar (Section 4.2) and executes the derivation to update the 3D model.

Repetition Management

In some cases, the user may want to apply an object type repetitively (e.g., a row of windows, stack of floors). Instead of sketching the object many times or making extensive use of copy and paste, we provide a quick mechanism to subdivide the anchor region, sketch an object type, and generate repetitions. Our tool infers the intended horizontal or vertical 1D repetitive application from just a few userdrawn line segments. Two examples of currently supported patterns are 1) (A) * that evenly splits the anchor into multiple anchors (or faces) of the same shape and 2) (AB) * A that splits the anchor into shapes A with shape B in between (e.g., shape B might be a thin separator between shape A's). Both A and B correspond to non-terminals of different object type labels than the anchor region. Note that our system does not require any additional action from the user to select the repetition mode; it is automatically selected when drawing facades and floors.

Design Variability

Since our end result is a grammar, we can exploit procedural modeling not just to create one model but to create a wide variation of buildings that are inspired by the original sketch. This is akin to the notion of concept drawing used in sketching to explore a design space: an artist draws subtle variations to an initial sketch and then potentially a more attractive option than the initial sketch is discovered [START_REF] Eissen | Sketching: Drawing Techniques for Product Designers[END_REF]. We support two approaches to provide variability: manual swap of snippets (for complete control; Figure 8) and stochastic snippet/parameter replacement. For the latter option, we use a similarity value between snippets (provided as input) to define the probability of the swaps. Moreover, we randomly explore parameter value changes to create more variability. We show results in Section 7 and Figure 16.

Rendering and Editing Tools

Our system provides several rendering and editing tools. The 3D model can be rendered using sketch style, screen space ambient occlusion (SSAO), pen-and-ink style, or shaded texture-mapping (Figure 9). In addition, our framework supports selecting, moving, resizing, deleting, copying and pasting subsets of the entire model as well as an undo function. For example, when the user selects a Figure 12: Objects Generated from Snippets. Our approach uses CNN to find the most appropriate snippet and its parameter values. We show a few sketches for four different object types.

building mass, control points appear on its top and side faces, and the user can resize it by moving the control points.

Results and discussion

We implemented our system on an i7-based PC Workstation with 24GB of memory and NVidia GTX980 graphics card. Our tool was implemented in C++ using OpenGL/GLSL and supports mouse and tablet input. We used the Caffe library [START_REF] Jia | Caffe: Convolutional architecture for fast feature embedding[END_REF] as the framework to train the CNNs using the GPU and to recognize snippets and estimate snippet parameters during run-time.

The pre-trained CNNs and all the snippets are loaded at system startup. Our implementation includes 4 object types and 26 snippets, which results in 30 CNNs in total. We used images of resolution 256x256 for training the recognition CNNs and we used lower resolution 128x128 images to decrease the size of the input data and increase the batch size for the parameter estimation CNNs. We trained the recognition CNNs for 20,000 iterations using the pre-trained network as initial weights [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]] and trained the parameter estimation CNNs for 80,000 iterations each. Every 10,000 iterations of training took around 30 minutes on our computer. Note that once trained, we can use these trained CNNs to recognize and estimate the parameters within 400ms.

a) h) f) i) e) c) g) j) k) d) b)
Figure 13: Example Buildings. Some of the buildings generated by our tool: Paris-style buildings (a and b), complex shape of buildings (c, d, e, f, g, j, and k), and skyscrapers (h and i). The creation time was around 5 minutes for each building.

Sketching Examples

Example sketches are shown in Figure 1, Figures 10-14 and the accompanying video. In Figure 10a, the user draws a sketch of a building mass. After selecting the top face of the generated building mass using the lasso tool (Figure 10b), the user draws a sketch of a roof (Figure 10c). The user selects a facade (Figure 10d) to subdivide it into a repeating arrangement of floors (Figure 10e). Similarly, the user selects a floor to subdivide it into walls and windows (Figure 10f). Then, the user selects one of the windows and sketches more details of the window (Figure 10g). Lastly the user sketches a ledge (Figure 10h). Further, the user can select color and texture that will be a part of the output grammar (Figure 10i). Also, the user can choose to preview a final expected model at any moment (Figure 11). In this building, the system has filled-in the details using the default snippets for the object types the user has not yet selected.

Figure 12 shows some examples of the object geometry generated by our tool for several exemplary object types. Notice how well our system can estimate parameter values. Figure 13 shows examples of buildings generated by our tool, including Paris-style buildings, skyscrapers, and complex shaped buildings. As a mean of evaluation, we used our system to reproduce a random set of buildings by using the 40 first "office" images from ImageNet and Flickr. We provide the full set of results as supplemental materials, and show a subset of representative ones in Figure 14. It took 5-15 minutes to create each of the buildings. Note that we managed to capture the overall shape and look of most buildings, although some details differ when they cannot be expressed by the snippets we've included in our current implementation (highlighted by red arrows in Figure 14).

Table 2: Training Performance and Computation Time. The table shows the training performance and computation time for recognition and parameter estimation of the various models and snippets shown in this paper. The high accuracy and the relatively small root mean squared error (RMSE) show that our approach can find an appropriate grammar and parameter values precisely within a very short time.

Recognition

Parameter

Performance and Comparison

In this section, we show training performance, compute time, and system comparisons. Table 2 shows the training performance and computation time for recognition and parameter estimation of the various models and snippets shown in this paper. The total training accuracy averages to 97.5% and the recognition plus parameter estimation time is about 340ms.

We compared our approach to MCMC, the de-facto approach for inverse procedural modeling (Figure 15). In this experiment, our approach can precisely find the parameter values within 320ms, whereas MCMC cannot find a good estimate in such a short time. Also, note that we used only one grammar for MCMC in order to skip finding the best grammar among others. The "CNN only" approach already produces very accurate results, but we add a few MCMC steps in order to further improve parameter estima-

5) 6) 7) 1) 3) 4) 2) 8) 9) 10)
Figure 14: Expressivity. We have created 40 buildings using the images from ImageNet and Flickr as inspiration. Only a subset of buildings is shown here. Please refer to the supplemental materials for all the created buildings and the source images. The red arrows highlight some building features and shapes not well supported by our currently implemented set of snippets, such as non-axis aligned shapes (4 and 10), certain window shapes (5), and irregular facade patterns (6 and 7).

Photo Credits: 8) c Jennifer C., 9) c rjp, and 10) c Alper C ¸ugun.

tion while remaining fast enough for interactive use. While the improvement is subtle, it can be noticed on the second column in Figure 15, for example. Another naive approach for parameter estimation would be to generate a very large number of training images by changing parameter values and find the best match image given a user sketch. However, this approach can only choose the best one from a discrete set of options. In contrast, our approach can estimate continuous values.

To improve accuracy, we experimented with several rendering strategies for use in training: 1) sketch style rendering, 2) blurry rendering, and 3) incomplete renderings (e.g., some edges removed). For the first option, we implemented our custom rendering algorithm. In a first pass, it renders an image by using a standard line rendering algorithm. In a second pass, it detects edges by Edlines [START_REF] Akinlar | Edlines: Real-time line segment detection by edge drawing[END_REF], and replaces them by predefined polylines that imitate hand-drawn lines. For the second option, we use Gaussian filtering with σ =image size x 0.02. For the third option, we use a similar algorithm to the first one but remove half of the detected edges. Our experiments showed that sketch style rendering achieved the best accuracy (Table 3).

Model Variation

Since the output of our tool is always a consistent grammar, the user can easily produce a wide variety of geometry by changing snippets and snippet parameter values. Figure 16 shows some of the automatically generated buildings after the user completes the design sessions. A user-provided probability density function based on the visual similarity between grammar snippets is used to change grammars, while the parameter values are randomly selected. In this example, we manually defined a similarity value between different snippets of the same object type. Then, when so desired the designer can use this function to randomly select an alternative rule; upon selection the system then performs parameter estimation.

Robustness

Our system tolerates significant sketch inaccuracy and incompleteness and always generates a valid procedural model. Figure 17a shows examples of how our system is able to find the correct grammar snippet despite of inaccurate sketches and different sketching styles. Note that if the user draws a novel shape, i.e., a shape that has not been introduced in our grammar snippet set, the CNN will select the closest shape expressible by the available snippets (Figure 17b).

User Experience

We had a group of eight users aged from 25 to 34 evaluate our system. All of them are graduate students who received technical education, but they all reported limited knowledge of 3D modeling tools and basic drawing skills. Each participant followed a simple tutorial about our system, which took them around 10 minutes to complete. They then had to reproduce two target buildings using our system, without any time limit. The two targets were both in a Parisian style, but contained subtle differences of mass, roof, windows and ledge to assess user's ability to specify different details. Figure 18 shows the quantitative evaluation of our tool. For each of two target buildings, it shows the number of clicks, the creation time, and the number of rules in the generated grammar which is an indication of generated building complexity. All participants successfully created buildings that look similar to the targets in a very short time (Figure 19). We also asked some participants to create some original buildings after showing them inspirational examples (Figure 20). On a Likert scale from 1 (strongly disagree) to 5 (strongly agree), five participants strongly agreed and three agreed that our system allowed them to achieve the buildings they wanted. They also all agreed that the system interpreted well the shapes they drew. Finally, several participants suggested areas of improvement. In particular, some participants had limited skills in perspective drawing and suggested the use of more traditional front/top/side views to sketch the building mass. It is important to note that even for those who are not good at sketching, our system interpreted their intended shapes and enabled the participants to achieve the desired output.

Limitations

Our proposed method cannot recover the procedural grammar from an existing sketch. Although our coarse-to-find sketching workflow reflects the way people draw complex objects, the sketching procedure requires the user to mentally decompose a target building. Also, the output variability is limited to that provided by the grammar snippets currently implemented in the system. One option to ameliorate the highlighted differences of Figure 14 is to increase the number of supported snippets. While this may increase the range of buildings features and shapes that can be supported, it will eventually also decrease the CNN's accuracy in recognizing which snippet is currently being sketched. Also, for some irregular shapes using procedural modeling might not be suitable -in those cases, it might be preferable to integrate non-procedural modeling methods such as example-based and/or pure sketch-based methodologies. Figure 19: User trial output. We show the buildings created by the users for each of the two target buildings in Figure 18.

Conclusions and Future Work

We have presented a procedural modeling sketching system that enables a novice user to create a grammar and a corresponding 3D geometry quickly and easily, without the need of writing procedural rules. We achieve interactive sketching of complex grammars by both decomposing the creation process of hierarchical models into coarse-to-fine stages, and by leveraging modern machine-learning algorithms to recognize grammar snippets and estimate their parameters in less than a second. The results indicate that our approach easily and efficiently creates a complex grammar of a desired building shape even for people who do not have any knowledge of procedural modeling grammars.

As future work, we will pursue several items. Currently, our system requires grammar snippets as input. In the future, we would like to automatically partition large grammar definitions into a suitable set of grammar snippets. Also, our system currently uses a fixed-view for sketching. Supporting arbitrary viewpoints significantly increases the complexity of data generation for training, while reducing the accuracy of snippet recognition and parameter estimation. Nevertheless, recent work on CNN-based viewpoint estimation suggests that such an ability is within reach [START_REF] Su | Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views[END_REF]. Finally, our current machine-learning approach requires a significant amount of memory (e.g., up to few hundred MB per network). We implemented our system using standard deep CNNs using Caffe and have not experimented with the performance of shallower CNNs to find the best tradeoff between memory and accuracy. Another possible solution would be to use a single CNN for all object types, which saves memory but might sacrifice accuracy. We asked some of the participants to create more complex buildings after showing them examples. These imaginary buildings were sketched in less than 15 minutes.

Figure 2 :

 2 Figure 2: System Pipeline. Our system consists of offline training and interactive online sketching. During training, a set of predefined grammar snippets are used to generate a large number of training images. Training and validation datasets are used to train the CNNs. During runtime, the pre-trained CNNs are used to find the best snippets and parameter values that match the current sketch. Then, the grammar parser generates an instance of the entire 3D model.

Figure 3 :

 3 Figure 3: Example Snippets and Object Types. As opposed to a standard 3D model, each snippet contains parameters defining a range of potentially generated geometry. Further, each snippet is associated with an object type. The set of shapes in each row are example outputs generated by randomly selecting snippets and parameter values.

Figure 6 :

 6 Figure 6: CNN Training. An example set of objects generated by our building mass snippet for CNN training. The objects were rendered as sketches to optimize the CNN for human-drawn strokes. For other object types, we generated the training images in a similar manner.

Figure 7 :Figure 8 :

 78 Figure 7: Parameter Estimation CNN. Training for parameter estimation requires a large number of iterations to converge because no initial pre-trained weights are used.

Figure 9 :Figure 10 :

 910 Figure 9: Rendering Styles. Our tool provides several rendering options, a) screen space ambient occlusion (SSAO), b) sketch lines, c) sketch lines + hatching texture.

  user sketch to the same resolution as the training images and the recognition CNN produces a normalized probability distribution of the snippet types. Afterwards, the system resizes the user sketch to the training image resolution for the parameter estimation CNN of the recognized snippet type, and the parameter estimation CNN outputs the normalized parameter values. As an extra step of refinement, the system also uses a few iterations (i.e., 10 in our current implementation) of a MCMC engine to further improve the parameter values. We use a single thread and Metropolis-Hastings algorithm to accept or reject randomly-generated state change proposals. Sketch-style rendering is used to generate a 2D image Iprop, and state change proposal score sprop is computed based on the user sketch I sketch by sprop = exp -D(Iprop) -D(I sketch ) F (1) where D( * ) denotes distance transform * F denotes Frobenius norm.

Figure 11 :

 11 Figure11: Final Model Previewing. Our tool allows the user to see the final expected model at any moment even if the user has not selected a grammar for some stages. The system uses a default grammar for those stages and generates a complete building model. The user can optionally stop modeling if the result is satisfactory.

Figure 15 :

 15 Figure 15: MCMC Comparison. Our approach uses both CNN and 10 iterations of MCMC-based refinement to yield much better parameter estimation than MCMC alone. Even if the MCMC iterations are removed from our approach, CNN still yields much better accuracy as compared to MCMC and in a fraction of the time. The time, RMSE, and standard deviation for each grammar are the averaged values of ten executions.

Figure :

 : Figure:Our stage-based modeling process also allows the interchange of grammars for each stage. After the user completes designing a grammar (leftmost images), our system generates a wide variety of geometry by changing grammars of each stage based on the probability density function that were manually defined and randomly changing parameter values within a predefined range.

Figure 16 :Figure 17 :

 1617 Figure16: Variability. Our system can automatically change snippets and snippet parameters to yield model variation. After the user completes a design (leftmost images), our system generates a wide variety of geometry by changing snippets using a probability density function and by randomly changing parameter values within a predefined range.

Figure 18 :

 18 Figure 18: Quantitative evaluation. All the subjects could successfully create a building that looks very close to the target in less than 7 minutes. The number of rules in the output grammar indicates its complexity

Figure .

 . Figure.The output geometry in the second user test.

Figure 20 :

 20 Figure20: Original buildings. We asked some of the participants to create more complex buildings after showing them examples. These imaginary buildings were sketched in less than 15 minutes.

Table 1 :

 1 Recognition CNNs. We compared the accuracy after 20,000 iterations of recognition CNN training using pretrained weights as initial weights and training from scratch. The former training approach improves the accuracy on our dataset.

		Training using the Training from
		pretrained weights	scratch
	Building mass	0.99	0.92
	Roof	0.98	0.90
	Window	0.98	0.87
	Ledge	0.95	0.86

Table 3 :

 3 Comparison of accuracy with different rendering options. We experimented with several rendering options to generate training images. Our sketchy rendering achieved the best accuracy in recognizing parts of human-drawn sketches.

		Building mass Roof Window Ledge
	Sketchy rendering	0.94	0.83	0.89	0.75
	Blurred image	0.59	0.46	0.37	0.38
	Missing edges	0.75	0.50	0.41	0.50

Acknowledgements

We would like to thank Jennifer Neville and Hogun Park for their comments and suggestions about the CNN architectures, our user study participants for their time and feedbacks, and the anonymous reviewers for their constructive suggestions. This research was partially funded by NSF CBET 1250232, NSF IIS 1302172, a Google Research Award, and the French ANR project SEMAPOLIS (ANR-13-CORD-0003).