Dmitry Kuzovkin
email: dmitry.kuzovkin@technicolor.com

Tania Pouli
email: tania.pouli@technicolor.com

Rémi Cozot
email: remi.cozot@irisa.fr

Olivier Le Meur
email: olivier.le_meur@irisa.fr

Jonathan Kervec
email: jonathan.kervec@technicolor.com

Kadi Bouatouch
email: kadi.bouatouch@irisa.fr

Context-aware Clustering and Assessment of Photo Collections

Keywords: Photo selection, Clustering, Image quality

Figure 1: Our context-aware framework performs the assessment and labeling of images in photo collections by considering image quality in the context of the collection as well as photos captured in the same scene. (a) Extract from a photo collection and visualization of clustering. (b) Photos captured in the same scene and close-ups of image details. (c) Top: image labeling obtained from independent assessment. Bottom: labels assigned after our context-aware adaptation of the independent score.

INTRODUCTION

With digital photography and low storage costs, users have the ability to capture many photos of the same scene, object or event with little overhead in terms of cost or e ort -an estimated 1.1 trillion photos were captured in 2016 alone1 . However, this increase in the number of photos captured, brings a signi cant overhead in the time and e ort necessary for organizing one's photo collections, and eventually selecting which images to keep or reject.

This task is commonly addressed with individual image aesthetics assessment approaches, where an input image is evaluated irrespective of its context [START_REF] Chang | Automatic triage for a photo series[END_REF][START_REF] Datta | Studying aesthetics in photographic images using a computational approach[END_REF][START_REF] Lu | RAPID: Rating Pictorial Aesthetics using Deep Learning[END_REF][START_REF] Luo | Photo and Video Quality Evaluation: Focusing on the Subject[END_REF][START_REF] Mavridaki | A comprehensive aesthetic quality assessment method for natural images using basic rules of photography[END_REF][START_REF] Redi | The beauty of capturing faces: Rating the quality of digital portraits[END_REF][START_REF] Tang | Content-Based Photo Quality Assessment[END_REF]]. However, these approaches often tend to be modeled over average user preferences, extracted from a vast, generic range of content, thus diminishing the subjective aspect of selection and neglecting the context of other photos present in the same collection.

Generally, as a pre-selection step, users tend to remove clearly awed photos, such as for example blurred or wrongly exposed ones, to facilitate the subsequent step of subjective selection, where aspects related to the depicted scene or people present in the photo are assessed. Detection of such low-quality photos is generally assumed to be an objective process that can be performed on each image independently.

Nevertheless, users implicitly consider the characteristics of the entire collection, as well as surrounding photos, to make a decision for a particular image. For example, photos taken by a nonexperienced user with a smartphone camera might present more blur and other artifacts, compared to photos acquired by a professional photographer in a studio. In a non-professional scenario, the user is likely to keep the best photos of their collection as a representation of an event, even if they are not objectively perfect photos. As an opposite example, pre-selection of sharp photos taken in a professional photo session presents stricter quality requirements, especially when numerous similar photos from the same scene are available. Regardless of the user skills and collection characteristics, if many photos of a particular object, landscape or moment are available, the user might apply stricter criteria to select whether to keep each instance or not. At the same time, they might be more lenient in their decision if only a single version is available. Therefore, the notion of quality context is important, and computation of context characteristics should be considered in the process of photo pre-selection.

To that end, we propose a novel framework that facilitates the process of photo pre-selection by evaluating image quality in the context of the collection in which it belongs and the relevant photos captured in the same scene. Given a collection of photos, originating from the same user and equipment, and representing a single event (e.g. trip, holiday, party, wedding), we employ a hierarchical clustering approach to organize collection into similarity-based clusters and de ne the context of each image.

This clustering, in addition to o ering a convenient structure for browsing photos, also provides the context in which quality may be evaluated, using any desired criteria. The context is modeled on three levels, with the following intuition behind each level:

(1) Collection level statistics can be used to re ect the features of the capturing devices and skills of a photographer.

(2) Scene level statistics focus on images of the same general scene but not necessarily completely overlapping, and aim to re ect environment properties, such as in uence of illumination.

(3) Near-duplicate level information focuses on images that depict the same scene or object with minor variations due to occasional user mistakes and other arbitrary changes between similar images.

Using the obtained context information, an independent quality score is adapted according to surrounding photos and transformed into a user friendly selection label. For demonstrating our framework, we focus on sharpness as the quality criterion, however our system can be extended to include further assessment criteria. Fig. 2 shows an overview of our clustering and assessment framework.

RELATED WORK

Image quality assessment methods aim to automatically evaluate the visual quality of an image based on particular criteria. The majority of existing approaches is represented by machine learning based solutions, where subjective factors (such as image composition, color characteristics and scene complexity) are aggregated together with objective metrics (such as sharpness and scene exposure metrics) [START_REF] Chang | Automatic triage for a photo series[END_REF][START_REF] Datta | Studying aesthetics in photographic images using a computational approach[END_REF][START_REF] Lu | RAPID: Rating Pictorial Aesthetics using Deep Learning[END_REF][START_REF] Luo | Photo and Video Quality Evaluation: Focusing on the Subject[END_REF][START_REF] Mavridaki | A comprehensive aesthetic quality assessment method for natural images using basic rules of photography[END_REF][START_REF] Tang | Content-Based Photo Quality Assessment[END_REF]. At the same time, the evaluation of particular aspects of lowquality pictures is re ected in more specialized research. Detection of blurred photos is often performed using wavelet transform based approaches [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF][START_REF] Ke | The Design of High-Level Features for Photo Quality Assessment[END_REF]], or by computing blur likelihood over local image windows in Fourier domain [START_REF] Chakrabarti | Analyzing spatiallyvarying blur[END_REF][START_REF] Mavridaki | No-Reference Blur Assessment In Natural Images Using Fourier Transform And Spatial Pyramids[END_REF]. Exposure artifacts, such as over-exposure, are usually detected using straightforward approaches based on average intensity information [START_REF] Datta | Studying aesthetics in photographic images using a computational approach[END_REF]] or image histograms statistics [START_REF] Redi | The beauty of capturing faces: Rating the quality of digital portraits[END_REF].

Whether relying on speci c criteria or a combination of multiple factors, most methods assess general image quality or particular defects by analyzing each image independently, and not considering the individual preferences, the photographer's skills, the capturing environment, or the camera properties.

To encode personal preferences in the context of aesthetic image assessment, [START_REF] Yeh | Personalized photograph ranking and selection system[END_REF] proposed a photo ranking system combining general aesthetics features with personal preferences, which are manually de ned or learned from a chosen photo example. In the work of [START_REF] Bychkovsky | Learning photographic global tonal adjustment with a database of input/output image pairs[END_REF], an extensive dataset of photo pairs before and after processing by di erent professional retouchers was created. Based on this dataset, a tone style adjustment model was learned, which could be adapted to a particular user using a small set of training photos.

The described preferences-based approaches perform the analysis of photos independently, where the information about other photos, captured in the same conditions, is not taken into consideration. The knowledge of the photo context -characteristics for the entire album and speci c scenes -could potentially assist the process of automatic photo assessment and selection, even without explicit user modeling.

To extract this necessary context information, we need to nd natural boundaries between di erent captured scenes within a collection, which can be achieved using clustering methods. Photo album clustering is typically obtained using temporal [START_REF] Cooper | Temporal event clustering for digital photo collections[END_REF][START_REF] Platt | PhotoTOC: Automatic clustering for browsing personal photographs[END_REF] or similarity [START_REF] Ceroni | To keep or not to keep: An expectationoriented photo selection method for personal photo collections[END_REF][START_REF] Chu | Automatic selection of representative photo and smart thumbnailing using near-duplicate detection[END_REF][START_REF] Alexander | Automated event clustering and quality screening of consumer pictures for digital albuming[END_REF] information. Among clustering approaches, hierarchical clustering takes a prominent place. As it does not require a pre-de ned number of clusters, it is versatile in its application and suitable for di erent types of data, and, in particular, images. Hierarchical clustering is often applied for creating tree-like representations for image browsing, for example based on color histogram similarities [Krishnamachari and Abdel-Mottaleb 1999] or on geo-location information [START_REF] Epshtein | Hierarchical photo organization using geo-relevance[END_REF]]. In our proposed method we also rely on hierarchical clustering, but, in contrast to previous methods, our clustering approach is based on similarity computed from matches de ned over SIFT descriptors. More importantly, instead of a typical tree-like browsing structure, we obtain a at representation of the entire collection, which can serve as a convenient way of image browsing (which is demonstrated in Section 4.4).

Collection clustering and quality assessment may also be performed in conjunction, either in an independent manner [START_REF] Alexander | Automated event clustering and quality screening of consumer pictures for digital albuming[END_REF] or by considering both tasks simultaneously [START_REF] Ceroni | To keep or not to keep: An expectationoriented photo selection method for personal photo collections[END_REF]]. In the latter case, in the method of Ceroni et al., a set of features is accumulated on intra-and inter-cluster levels, and a nal decision to keep or not to keep a particular photo is obtained using a machine learning system. Although some of their features depend on the clustering, their method does not perform adaptive per-album and per-cluster evaluation, but rather learns a binary classi er that re ects a model learned over numerous non-related albums and users. Another approach for context-aware selection was recently introduced by [START_REF] Chang | Automatic triage for a photo series[END_REF], where pairwise comparisons are performed within photo series of similar (nearduplicate) photos using deep convolutional neural networks, to obtain a relative ranking of images. This method demonstrates impressive results, however it is limited to ranking and selecting among small sets of nearly identical photographs, although their approach also presents a potential for further extension to handle complete collections.

In contrast, in our framework we consider the complete collection as well as di erent levels of image context simultaneously. By performing hierarchical similarity clustering, the necessary multi-level context is obtained, which gives an intuitive collection representation and provides a base for adaptive image scoring and labeling. While several previous methods are based on a prede ned criteria for image quality assessment, in our work we opt for a more exible approach that can adapt any image quality criteria to consider the context of each photo. This way, our method can provide the evaluations that are closer to user's expectations.

CONTEXT-BASED ASSESSMENT 3.1 Method Overview

The goal of our framework is to simultaneously organize and assess the quality of photographs belonging to a photo collection by considering not only each image independently but also its surrounding context. To achieve this, our approach consists of the following stages, which are illustrated in Fig. 2.

First, for each image, an independent image quality score is computed based on the selected quality criteria (Fig. 2a). Second, an input collection is grouped into temporal windows, and then each temporal window is hierarchically clustered into scene-level clusters and near-duplicate clusters (Fig. 2b). These clusters are used to de ne the similarity-based context of each image and create the intuitive collection visualization through the context-aware representation (Fig. 2c).

The per-image independent quality score computed initially is then re-estimated for the three context levels, de ned by collection, scene-level and near-duplicate level statistics (Fig. 2d). Finally, the weighted sum of context-based scores for di erent levels is used to obtain a global scoring and subsequent labeling of each image, agging images as Accept, Reject or Maybe, similar to common photo editing work ows.

We have opted to demonstrate our framework using sharpness as the underlying image quality criterion. Accordingly, a photo is suggested to be removed when its sharpness is not su cient (or, in other words, a photo is too blurred) in the context of the collection. Among the criteria leading to image rejection, blur in its di erent forms is one of the most important factors [START_REF] Marchesotti | Learning beautiful ( and ugly ) attributes[END_REF][START_REF] Maria K Wolters | The art of deleting snapshots[END_REF], as it can a ect both professional and amateur photographs and it is hard to remove in post-processing. At the same time, the sharpness requirements largely vary depending on content type and user intentions, hence the need for context-aware adaptation.

Our framework can be adapted to other independent metrics or extended to include multiple factors, such as exposure issues or noise. In e ect, any existing method can be used to obtain initial independent image quality scores, as long as the scores provided are de ned within a limited range of values (e.g. 0 to 1, not 0 to in nity).

Independent Image Quality Score

Existing techniques are capable of assessing sharpness or blur in individual images, but they cannot be easily adapted to the nature of particular collections. For example, a collection captured in di cult illumination conditions by an amateur photographer might exhibit a large number of blurred photos. However, typical blur assessment methods are pre-trained on a wide range of photos, thus they can underestimate and reject many photos in a low-quality collection. In our method, we use a traditional blur assessment method as an initial quality estimator for each image, and adapt its result to the particular context at a later stage.

We adopt the wavelet-based blur detection by [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF]. The Haar wavelet transform is applied to an image, and edge maps E i are generated on three scales, using the pyramid images LH (vertical details), HL (horizontal details), and HH (diagonal details):

E i = LH 2 i + HL 2 i + HH 2 i (i = 1, 2, 3). (1)
Non-maximum suppression is applied to process all maps into the equivalent scale, using scale-dependent local windows. Depending on the edge properties across scales, each pixel is assigned an edge or non-edge value, and each edge pixel is further assigned one of four edge structure types: Dirac-structure, Roof-structure, Astep-structure and Gstep-structure. In particular, Dirac-structure and Astep-structure pixels are considered sharp (as they tend to disappear in presence of blur), and their total number N da , along with the total number of edge pixels N ed e is used to compute a nal sharpness score:

s I = N da N ed e . ( 2
)
The obtained sharpness score is used as an individual image score in our approach. Sharpness histograms for three di erent photo collections are shown in Fig. 4. Context-based adaptation of the obtained score is described in Section 3.4.

Context Extraction by Hierarchical Clustering

After individual image quality scores are obtained, they can be adapted to the context of each image. This allows our framework to consider the quality of the image in relation to images surrounding it (same scene, taken at a similar time and so on). In this section, we describe how the context of each image is determined. As a rst step, the entire collection is organized into time-based groups. Then, inside each time group, a hierarchical tree is constructed, using the proposed image similarity metric. Finally, each time group is clustered into two levels: scene level and near duplicate level, which group the photos according to the visual similarity between them.

Time-based Grouping. As an input collection can span a large time range, varying from several minutes to days, we rst split the entire collection into sequential temporal windows, to facilitate further similarity context extraction. Based on the photo time stamps, temporal windows are computed using the solution by [START_REF] Platt | PhotoTOC: Automatic clustering for browsing personal photographs[END_REF]. In their method, temporal window boundaries are created when a time gap N between two images (at the position N ) exceeds the time gap average across neighboring images, satisfying the following condition:

lo ( N ) ≥ K + 1 2d + 1 d i=-d lo ( N +i ), (3)
where the parameters are set by the authors as K = lo (17) and d = 10. Time stamps for this analysis are extracted from EXIF data, typically available for most photographs. If no reliable timestamps are available, the complete collection is considered as a single temporal window in further analysis.

Similarity-based Distance. Each obtained temporal window is further clustered in a hierarchical manner, providing scene-level clusters and near-duplicate clusters of images. Before clustering, a similarity-based distance metric is computed between all images. Our distance metric is based on the SIFT features [Lowe 1999], due to their advantage in identifying the image matches even in presence of distortions (which often appear in the series of similar photos). For each pair of images within a temporal window, two sets of SIFT descriptors are compared using the Euclidean distance.

Given a pair of images I and , the number of matching descriptors [Lowe 1999] is computed in both directions: m I → and m →I . Then, our distance between a pair of images is given by:

d(I, ) = 10 M(I, ) • P(I, ) , (4)
where M(I, ) is the average of number of matches (m I → +m →I )/2, and P(I, ) is a measure of pair matches consistency, de ned as:

P(I , ) = min(m I → , m →I ) max(m I → , m →I ) . ( 5
)
The nominator in Eq.( 4) is set to 10, to scale the output distance to approximately a 0-1 range for most images.

Based on the computed pair-wise image similarities d(I, ), we obtain a similarity distance matrix (an example is shown in Fig. 3a). The distance matrix is computed within each temporal window and then used to perform hierarchical agglomerative clustering [START_REF] Everitt | Cluster Analysis[END_REF].

Hierarchical Tree Construction. We chose the hierarchical clustering approach, as it does not require a pre-de ned number of clusters, and it allows to create a speci c cluster structure, where clusters of di erent similarity are enclosed into each other, as it explained below. Such structure allows to re ect natural organization of photos, where photos are usually considered in connection with similar photos from the same scene.

Using the similarity distances computed in the previous step, the clustering is performed in a bottom-up approach, where each image initially belongs to a cluster containing only the image itself. At each step, the two clusters with the lowest similarity distance are merged together into a new cluster. The distance between singleimage clusters can be directly retrieved from the similarity distance matrix. The distance between two clusters with multiple images C 1 and C 2 is de ned as the shortest distance over all possible pairs of images, where one image from a pair belongs to the cluster C 1 and another image belongs to the cluster C 2 (this way of cluster merging is also known as the single-linkage criteria).

The merging process continues recursively until only a single cluster is left, where all clusters are merged together into one hierarchical binary tree. It is worth noting that in our case there will be pairs of images with no matches present between them, thus leading to in nite similarity distance (denoted as X in Fig. 3a). In this case, the merging stops when one cluster is achieved or when the distance between all remaining clusters is in nite. Thus, it is possible to obtain multiple, unconnected hierarchical trees. An example of such case is demonstrated in Fig. 3b, where two trees are obtained, due to the absence of similarity between the two scenes. Note that the depth of branches in the hierarchical tree corresponds to the similarity distances between the two connected clusters.

Tree Cutting. After the hierarchical tree is created, it can serve to partition images into nal clusters corresponding to di erent levels of similarity. To achieve this, the hierarchical tree can be cut at di erent levels, where the tree is traversed from top to the bottom, and all connected nodes below a particular cut level are assigned a single cluster. To obtain scene-level and near-duplicate clusters, the tree is cut at two levels of similarity distance, using cut thresholds c N D and c SC , where c N D < c SC , as the near-duplicate clusters with higher similarity should be enclosed into the scenelevel clusters. As a result of the conducted experiments, the cut thresholds were set as c SC = 1 and c N D = 0.15, since we found that these distances lead to a clear separation into scene-level and near-duplicate groupings for all collections tested. A visualization of the tree cuts and resulting clusters is given in Fig. 3b-c.

Context-based Photo Scoring and Labeling

The nal step of our framework is the adaptation of individual image scores computed in Section 3.2, according to the context extracted at di erent levels (Section 3.3). The context of each image is de ned on three levels: collection level, scene cluster level and near-duplicate cluster level. In the following we refer to the three levels using the C, SC, N D identi ers. To obtain context-dependent image scores, we utilize the z-score [START_REF] Kreyszig | Image browsing using hierarchical clustering[END_REF]] (based on the assumption that the independent scores follow Gaussian distribution), which can be computed from the independent image score s I as follows:

z I L = s I -µ L σ L , (6)
where µ L is a mean of quality scores on the level L ∈ C, SC, N D, and σ L denoting a standard deviation on the level L. It is worth pointing out, that each image belongs to only one cluster from each level (it can be observed in Fig. 4).

The intuition behind the z-score is that it normalizes score values around the mean value of the corresponding level, therefore adapting to its quality statistics. Examples of collection-level distributions are shown in Fig. 4, showing the necessity of such adaptation: overall sharpness can vary signi cantly across di erent collections.

In this manner, a z-score is computed on three levels, where each level provides an estimation how good the photo in the context of entire collection, its containing scene and among very similar nearduplicate photos. If the image is unique at its level (for example, no other similar photos have been taken in the same scene), then no other images are available for the computation of the statistics information, and we consider its z-score as unde ned.

The global score Z I of an image is computed by considering the z-scores computed previously at each context level:

Z I = z I C + z I SC + z I N D n z , (7)
where z I C is an image score on the collection level, z I SC is a scene level score, z I N D is a near-duplicates level score, and n z is the number of de ned z-scores for this image (number of non-singleton clusters). Unde ned z-scores for a given image are set to 0 for this calculation.

The obtained score can either be used directly for sorting and displaying the photo collections, or it can be transformed into user-friendly selection label. To facilitate the process of further photo selection, we provide users with three types of labels for each image: Accept, Reject or Maybe. The threshold values for this output labeling can be determined based on user input, for example, from the percentage of photos that user wants to keep, or set to prede ned values. In our experiments, we de ne thresholds Z I ≥ 0 for the label Accept, -0.5 ≥ Z I < 0 for the label Maybe, and Z I < -0.5 for the Reject label. This way the labels re ect the nature of zscores, where Z I = 0 corresponds to the mean of the scores across three levels of the context, and images with scores below zero are worse quality-wise than other neighboring images from their corresponding context. To demonstrate the e ectiveness of our approach, we have conducted a user study, where people's pre-selection decisions have been collected for various types of collections. Then, the obtained data has been used to verify the bene t of our context-aware method in comparison with independent methods for detection of low quality images, which do not adapt to the collection properties.

EXPERIMENTS

Dataset

Although a number of image datasets for aesthetic assessment (with corresponding evaluations by users) is publicly available, such as the Photo.Net dataset [START_REF] Datta | Studying aesthetics in photographic images using a computational approach[END_REF]] or the AVA dataset [START_REF] Murray | AVA: A large-scale database for aesthetic visual analysis[END_REF]], these datasets are created to address the task of independent image assessment. Thus each image is given without any context information about the original collection. Moreover, these datasets are acquired from photographers' peer-review social networks, hence the majority of presented photos are high-quality post-processed photos, which represent the nal outcome of a photographer's work.

In our work however, we aim to assess the photos at a very early stage of pre-selection, when a large collection of photos is just captured and needs to be organized. The recently proposed Photo Triage dataset [START_REF] Chang | Automatic triage for a photo series[END_REF] partially addresses these problems, as this dataset contains photos taken by a wide range of users and is organized into series of similar shots from the same scene. Nevertheless, these photo series still lack the context of the corresponding containing collections, which better re ect the overall intentions and skills of a photographer.

Since our approach performs photo evaluation based on multilevel context rarely available in publicly available datasets, we decided to evaluate its e ciency with a user study, performed on 5 photo collections of di erent content type. The rst collection consists of travel photos, taken mostly outdoors, with a small number of low-quality images. The second collection consists of wedding photos, captured indoors in di cult lighting conditions, and containing a noticeable number of blurred pictures. The third collection is a photo session conducted by a professional photographer (the photos are extracted before any processing is applied to them), with a consistent level of photo quality. The fourth collection covers a sport event (a volleyball match), where multiple pictures present motion blur due to the players' movements. Finally, the fth collection is taken during a Halloween party and presents cases of out-of-focus and motion blurred photos. All collections, except the professional collection, were acquired from the photo albums in YFCC100m dataset [START_REF] Kalkowski | Real-time analysis and visualization of the YFCC100M dataset[END_REF][START_REF] Murray | AVA: A large-scale database for aesthetic visual analysis[END_REF]]. The professional photo session is acquired directly from a photographer.

As the initial photo albums are of di erent size, for our analyzed collections we extract 100 photos from each album, in their original consecutive order, hence keeping the scene context unaltered. Each image is resized, so that the longest side of the photograph is 1920 pixels. In this manner, ve collections, each of 100 photos, are created. Example clusters of three collections along with the statistics on their sharpness level (Section 3.2) can be seen in Fig. 4.

User Study

The user study has been performed with 15 participants (9 male and 6 female), where each participant regularly takes personal photos in every-day life and is familiar with the task of photo selection and organization. Two of the participants occasionally use professional cameras, and can be considered as experts. Every participant was presented with each of the collections (selected in a random order), and was given the task of labeling sharp and non-sharp photos. The user could assign one of three labels: Accept if he considered a photo sharp enough to keep, Reject if a photo was too blurry and not worth keeping, and Maybe if a photo was not absolutely sharp but still worth keeping or if they could not otherwise make a decision. A user could freely browse through the entire collection during the labeling process, as in a typical real-life selection scenario, and zoom to view the images at full resolution.

The agreement between observers for each collection is analyzed using Fleiss' kappa measure [START_REF] Joseph | The equivalence of weighted kappa and the intraclass correlation coe cient as measures of reliability[END_REF]. Detailed results of the assigned labels by participants as well as the corresponding Fleiss' kappa value for each collection are given in Table 1. According to the interpretation of Fleiss' kappa measure, the level of user agreement ranges from fair to moderate on most of the collections, with the exception of the professional session, where a κ = 0.1827 indicates only slight agreement between observers. This is not a surprising result, as this collection contains very little blur as well as many repetitive images. This observation was supported by participants' feedback as well, many of which mentioned that they had trouble assessing this particular collection. Given the low degree of blur as well as the low agreement in this set, it is not included in further analysis.

Results and Discussion

To evaluate the performance of the proposed approach, the output labels of our method were compared with the labels given by each participant in the experiment described above. The metrics of accuracy, precision, recall and F-measure [START_REF] Fawcett | An introduction to ROC analysis[END_REF]] were calculated for each label and user separately, and average values of these metrics were computed across all participants. The performance comparison is given in Table 2. For each metric, a higher score indicates better performance. Both precision and recall values are important to estimate the correctness of label prediction in our task, therefore the F-measure is a good indicator of the overall performance, since it is computed as the harmonic mean between precision and recall.

To assess the bene t of our adaptation of quality scores according to context, we compare the performance of our solution against the performance of the original wavelet-based sharpness estimation method by [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF], which performs image assessment in an independent manner, i.e. without considering the image context. To obtain labels for each photo according to the method of Tong et al., the sharpness output of their method was thresholded. The thresholds were estimated as follows: sharpness scores were computed and averaged over 1600 di erent images, leading to an average score of 0.34. Half a standard deviation (σ /2 = 0.1) was considered above and below this average, obtaining the following thresholds and corresponding labels: an image is labeled as Reject if sharpness s I < 0.24, Maybe if 0.24 ≤ s I ≤ 0.44, and Accept if s I > 0.44. Then, for the obtained labels, the same performance metrics are computed.

As shown in Table 2, the adaptation of image quality scores using our framework o ers a signi cant improvement on all collections tested, relative to the results obtained with the independent method of [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF]. In collections containing more challenging photographs and conditions (Wedding ceremony, Sport event and Halloween party), our method showed the most improvement. As these collections no longer t the implicit criteria of the sharpness estimation method, falling well below the expected quality, most of the photos are rejected according to the original non-adapted score. On the contrary, the user study participants adapt to the album quality level and the context, and tend to accept even not perfectly Table 2: Comparison of labeling performance based on independent blur assessment scores using [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF]] and contextdependent scores obtained using our approach, for several image collections. Higher value of each measure indicates better performance, where F-measure can be used as an indicator of the overall performance.

sharp photos that might be important for the coverage of such an event. Our method manages to adapt to the quality level in a similar way and provides labeling results, which are more consistent with the users' decisions. At the same time, noticeable improvements were measured even for the travel collection, where overall image quality was relatively high and few blurry photos were present, showing the ability of our approach to adapt to the context of each collection and image. Additional performance analysis was conducted using receiver operating characteristic (ROC) curves, which were computed for each of three labels over the all evaluated collections. The shape of the curves, shown in Fig. 5, as well as the Area Under the Curve measures, given in Table 3, demonstrate better performance of our method for each label type. It can be noted that the independent method tends to correctly identify photos pre-labeled by users for rejection, while it does not show high performance on two other labels. This agrees with our intuition that the non-adapted method is likely to reject the lower quality photos, while users try to nd acceptable photos even among imperfect ones.

Another interesting observation that can be made from the ROC curves concerns the Maybe labels. Both the independent method and our context-dependent method suggest performance that is close to labeling by chance for that label. Indeed, the introduction of the Maybe labels makes the task of automatic pre-labeling more challenging. Despite the ambiguity that a Maybe label may introduce, we have opted for including it in our pre-labelling of images to facilitate the task of the end user. Our approach is aimed as a pre-process prior to the user's nal selection of images, enabling them to browse, and decide which images they want to keep or not more e ciently. Given that our approach relies on objective image assessment criteria and therefore cannot consider more subjective preference aspects, using a binary pre-labelling would oblige users to review all photos for making their nal selection. In contrast, using a Maybe label, allows us to use the more de nite Accept and Reject labels with higher con dence. As such, the end user in this case would only need to verify images with a Maybe label assigned.

In addition, to demonstrate the exibility of our framework, we perform a similar analysis using a second blur assessment method, namely the approach of [START_REF] Mavridaki | No-Reference Blur Assessment In Natural Images Using Fourier Transform And Spatial Pyramids[END_REF]. As their method relies on an SVM-based classi cation, we take the probability estimates as an output [START_REF] Chang | LIBSVM: A library for support vector machines[END_REF]. For the computed probability estimates, P i = 1 indicates that the image is sharp, while P i = 0 indicates that the image is blurred. The probability is used as an independent image quality score, and it is adapted to the context as described in Section 3. The output photo labeling for this method was performed in the following way: an image is labeled as Reject if P i < 0.4, Maybe if 0.4 ≤ P i ≤ 0.6, and Accept if P i > 0.6. Table 4 provides a comparison of the independent and context-dependent results for the di erent image collections. Similar to our analysis using the method of Tong et al., we nd that in this case our approach also improves the initial results and leads to labels that better correspond to user assessments. [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF] and our context-dependent method (computed over all collections given in Table 2). Higher value indicates better performance.

Although our approach shows promising results for many collections, there are some challenging cases that merit future study. First, even aided by the context, our method largely depends on the independent score used as an initial quality estimator. In the demonstrated sharpness-based example framework, it is not uncommon that the sharpness estimation method itself fails and provides a wrong score (for instance, a blurred image can get a high sharpness score due to the speci c nature of blur). In case of such failure, the context-based adaptation cannot correct the original score. Furthermore, an incorrect score assigned to one image from a multiple-images cluster can negatively a ect score adaptation for the surrounding images. As a second challenge, the context extraction by clustering can be unreliable in some conditions. Photos of low quality, especially captured in low-light conditions, often lack reliable information for the SIFT matching, thus leading to unreliable clustering and inaccurate context-adaptation as a result.

Finally, we would like to emphasize that our context-adaptive assessment does not aim to achieve the best results in the task of blurred image detection itself. Since our adapted scores rely on the original scores provided by the independent quality assessment [START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF] and our context-dependent method (computed over all collections given in Table 2). The closer the curve extends to the top left corner, the better performance for particular label is achieved. The diagonal line represents performance of random labeling. method used in each case, they are implicitly dependent on the e ectiveness of the image quality criterion chosen. The exibility of our framework however, allows for di erent criteria to be used interchangeably.

F-measure

Independent method by [START_REF] Mavridaki | No-Reference Blur Assessment In Natural Images Using Fourier Transform And Spatial Pyramids[END_REF] Context adaptation by our method

Implementation & Visualization

In addition, we would like to discuss a few aspects of our results visualization, which provides a user-friendly way of viewing the photo collection. Previous approaches that rely on hierarchical clustering for organizing photographs present images in a tree browsing structure, where one representative image replaces the contents of the cluster [START_REF] Epshtein | Hierarchical photo organization using geo-relevance[END_REF]Krishnamachari and Abdel-Mottaleb 1999]. However, in the typical photo viewing and managing software (e.g. Picasa, Lightroom) at representations are preferred, with ags and other identi ers used to label photographs.

With our approach, we construct a at representation and rely on enclosing borders to identify clusters at di erent context levels (temporal, scene-level, near-duplicate), as shown in Fig. 4. The selection labels can be assigned to images, similar to decision ags in popular photo management software, allowing the user to quickly understand the structure of their collection and perform their nal decision, without having to navigate several levels within a tree structure.

The cluster-based context visualization of collections is implemented in a form of auto-generated HTML pages, which show image thumbnails with suggested labels, in a manner similar to image browsers (examples are shown in Fig. 4). The HTML page can be freely browsed and zoomed without a ecting the frames delimiting photo clusters.

Our complete solution was implemented in Matlab, and timings for processing photo collections were measured on an Intel Core i7 PC 2.80 GHz with 16 GB RAM, running Windows 7 64-bit. Using our current framework, a collection of 100 images of HD resolution can be fully processed in about 90 seconds, with a linear growth to 460 seconds for 500 images. The clustering step takes around 40% of the execution time, and the sharpness estimation step takes around 55% of the execution time.

CONCLUSION

In this work, we propose a novel context-aware approach of photo assessment, where an independent image quality metric can be adapted to the content of an analyzed photo collection. Our approach extracts the context of each photo by means of hierarchical clustering, where each level re ects di erent degrees of similarity between photos. Along with statistical information, our clustering method provides a at collection representation with intuitive indications of the scene boundaries, which can be used to assist the user in the task of photo pre-selection. The extracted context information is used to adapt the initial image quality score for each level of context and compute a nal weighted score for every image, from which an output photo label is obtained. The conducted experiments demonstrate that we are able to model user pre-selection behavior, and that the context-adapted score performs signi cantly better than the original computed score in the scenarios where pre-selection decision cannot be made independently.

We believe that the proposed approach is the rst step towards user-aware photo assessment methods. Modeling the entire spectra of user preferences is indeed a big challenge, and in this research we attempt to address one of the multi-dimensional quality aspects. However, adapting to the collection context already allows us to model the photographer's intentions, skills and capturing environment to some extent. In this work, the three context levels are considered as equally important. Nevertheless, further study of the in uence of each context level on the nal selection of users could provide useful insights on the relative importance of z-scores from di erent levels in the weighting step. Finally, evaluating the performance of our approach with other image quality assessment metrics is also an interesting direction for future research, as well as searching for a solution that would allow us to combine different types of quality scores into a more complete image quality assessment.

Figure 2 :

 2 Figure 2: An overview of our context-aware framework. (a) An independent image assessment is performed based on the selected quality criteria. (b) Using computed SIFT matches and time stamps, photos are clustered in a hierarchical manner. (c) Obtained clustering is used to de ne the context for each photo and create the context-aware visualization of the collection. (d) An independent image score is re-estimated for the three context levels, leading to a nal scoring and labelling of the photo collection.

Figure 3 :

 3 Figure 3: Demonstration of our similarity-based hierarchical clustering on an example set of six images. (a) Similarity distance matrix between each pair of images. Crosses indicate that no matches between images are found. (b) Dendrogram of hierarchical clustering based on similarity distances. Cuts at speci c height (corresponding to the similarity distance) produce clusters grouped by di erent levels of similarity. (c) Output of clustering. The images are clustered into two scenes (yellow borders), where two images from the second scene (5 and 6) are grouped into near-duplicates cluster (red borders). Image credits: INRIA Holidays dataset [Jegou et al. 2008].

Figure 4 :

 4 Figure 4: Visualization of clustering and labeling on example clusters extracted from the test collections. Blue borders indicate temporal clusters, yellow borders indicate scene level clusters, and red borders indicate near-duplicates clusters. (a) Professional photo collection with a consistent level of quality. (b) Sport collection containing multiple pictures with motion blur. (c) Halloween party collection presenting large number of blurred photos, represented by out-of-focus and motion blurred shots. On the right: histograms of sharpness values corresponding to all photos in the collections containing presented clusters. Image credits: professional photo session provided by Nikolay Kuzovkin, sport event and Halloween party collections provided under Creative Commons by Flickr user Parker Knight. The travel collection and the wedding ceremony collection are not demonstrated in the gure, due to copyright restrictions.

Figure 5 :

 5 Figure5: Receiver operating characteristic (ROC) curves for label prediction by independent method of[START_REF] Tong | Blur detection for digital images using wavelet transform[END_REF] and our context-dependent method (computed over all collections given in Table2). The closer the curve extends to the top left corner, the better performance for particular label is achieved. The diagonal line represents performance of random labeling.

Table 3 :

 3 Area Under the Curve measured for independent method of

http://blog.infotrends.com/?p=21573


Benoît Desmarchelier

Faridah Djellal

Faïz Gallouj

Services in innovation networks and innovation networks in services: from traditional innovation networks (TINs) to public service innovation networks (PSINs) 1

Keywords: market services, public services, innovation, networks 2 ServPPIN:

This article is dedicated to a consideration of the tertiarisation of innovation networks. While the concept of traditional innovation network (TIN) has been the object of an extensive literature, new expressions of the innovation network appear in a service and sustainable development economy: in particular Public Private Innovation Networks in Services (PPINSs), Public Service Innovation Networks (PSINs) and Public Service Innovation Networks for Social Innovation (PSINSIs). They reflect the rise to prominence of market and non-market services and of the public-private relationship in collaborative innovation. This article investigates and compares these different expressions of innovation networks. In particular, it sheds light on the different roles played by public services in each of them.

Introduction

In contemporary economies, innovation is a universal and ubiquitous phenomenon present in every economic sector and every sphere of social life. However, whatever the discipline considered (economics, management, sociology, political science and so on), our analytical and conceptual tools have often been unable to grasp this innovation dynamic in its full magnitude. Thus, entire sectors of our economies (in particular, the service sectors, including non-market services) and essential forms of innovation (non-technological innovations, including social innovations) have long remained marginal in the field of "Innovation Studies". This innovation gap (which covers particular sectors and particular forms of innovation) may largely be explained by the inertia of our conceptual tools designed in and for manufacturing economies. It reflects, after all, invisible or hidden innovations, which do not fall within the traditional industrial and market indicators such as R&D, patents, and material technologies.

Considerable efforts have been made in recent years to bridge this innovation gap, taking into account both hidden forms of innovation and forgotten sectors. Thus, a field of "Service Innovation Studies" has enriched the traditional field of "Innovation Studies" that focuses on technological and industrial innovation [START_REF] Gallouj | Services and Innovation, The International Library of Critical Writings in Economics series[END_REF]Djellal and Gallouj, 2018a). An additional step forward in reducing the innovation gap has been achieved by taking into account the innovation dynamics in public services (Windrum and Koch, 2008;[START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF][START_REF] Vries | Innovation in the public sector: a systematic review and future research agenda[END_REF]Osborne and Brown, 2013;[START_REF] Miles | Public Service Innovation: What messages from the collision of Innovation Studies and Services Research[END_REF][START_REF] Potts | Public sector innovation research: what's next?[END_REF][START_REF] Fuglsang | Innovation in public service systems[END_REF]Fuglsang et al., 2014).

Ben [START_REF] Martin | Twenty challenges for innovation studies, SPRU Working Paper Series[END_REF] considers this gradual opening (to services and service innovation) of the field of innovation as one of the twenty main challenges in "Innovation Studies", since their advent, nearly a half-century ago. Djellal and Gallouj (2018a) for their part consider this opening as one of the fifteen main advances in "Service Innovation Studies", since their advent, nearly a quarter century ago. It is also described as "the shift from visible innovation to invisible innovation". It is parallel to another fundamental evolution in "Innovation Studies" which is the shift from a linear and closed model of innovation to an interactive and open or network model [START_REF] Martin | Twenty challenges for innovation studies, SPRU Working Paper Series[END_REF]. This rise of services, of service innovation and of the networked organization of innovation also lies at the heart of the shifts in public administration paradigms [START_REF] Osborne | The New Public Governance?[END_REF](Osborne, , 2010)). Indeed, in the traditional public administration paradigm, innovation is, for the most part, associated with the industrial rationalization of production processes and the adoption of technical systems, the aim being to provide passive citizens with homogeneous quasiproducts. This innovation activity, which excludes the user, is organized in a linear and topdown way. In the new public management paradigm, the industrialist perspective remains dominant, and innovation continues to be organized in a linear (non-interactive) way. The main novelty compared to the previous paradigm is the introduction of market management techniques in public services. The new public governance paradigm, currently at work in all developed countries, fundamentally changes the perspective of innovation. Indeed, this new paradigm considers public services as services and not as goods, and thus allows a broad and open concept of innovation integrating both technological and non-technological dimensions (new services, new processes, new organizations…). From the point of view of the organization of innovation, this paradigm emphasizes the collaborative dimension, and in particular the participation of citizens in innovation networks [START_REF] Osborne | The New Public Governance?[END_REF](Osborne, , 2010)). The importance given to networks, whether they be production or innovation networks, also leads to designating this new paradigm as a paradigm of "Networked Governance" [START_REF] Kelly | Creating public value: an analytical framework for public service reform[END_REF]. This article discusses the concept of innovation networks and the place that is given to services and especially public services in them. The aim is to show how, in parallel with the shift from visible innovation to invisible innovation, services in general and public services in particular are gradually moving from a peripheral to a central position in the innovation networks. Based on a review of the literature and on empirical work carried out under two European funded projects (ServPPIN and COVAL 2 ), we discuss how traditional innovation networks can be enriched by other types of networks more focused on services and public services, namely Public-Private Innovation Networks in Services (PPINSs), Public Service Innovation Networks (PSINs) and Public Service Innovation Networks for Social Innovation (PSINSIs).

This article is organized into four sections. In section 1, we provide a general description of these different expressions of innovation networks, to compare them from a morphological and functional point of view and to identify the relationships among them. The following sections are devoted to a more in-depth discussion of each of these network forms. Special attention is given in the last section to the most recent and least known expression of innovation networks, namely Public Service Innovation Networks (PSINs).

A general description of the different expressions of innovation networks: TINs, PPINSs, PSINs and PSINSIs

The notion of innovation network is often defined according to two complementary functional and morphological perspectives. In the functional perspective, the innovation network is defined as a mode of coordination between economic agents, considered to be more effective than market and hierarchy, in that it prevents the risk of bureaucratization of innovation that can occur in the hierarchy and the risk of disclosure of strategic secrets that characterizes the market. In the morphological perspective, the innovation network is defined as a structural arrangement for bringing together multiple actors around a common objective, namely innovation. While hierarchal governance is based on a central authority and market governance is based on contracts, innovation network governance is based on trust, reputation and mutual dependence between selected partners.

The notion of innovation network has been a great success in the literature, a success that is manifested on the theoretical, methodological, empirical and political levels (see [START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF]. This success of what are called here traditional innovation networks (TINs), is confirmed, in a way, by its spread to new socio-economic contexts (services, public services) and the emergence of new forms of innovation networks, namely the public private innovation networks in services (PPINSs) highlighted in the European ServPPIN project [START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF], the public service innovation networks (PSINs) and the public service innovation networks for social innovation (PSINSIs) discussed in the COVAL European project (see footnote 1). In this first section, we provide an overview, from a morphological (or structural) and functional point of view, of these different forms of innovation networks and the possible relationships among them.

The different forms of networks from a morphological and functional point of view

The different types of innovation networks considered, namely TINs and PPINSs as well as PSINs and PSINSIs can be described, in a general way, by the following variables: i) the types of agents involved in the network, ii) the role played by the public agent (the public administration), iii) the nature of the targeted innovation and iv) the main sector concerned by the innovation in question (see Figure 1).

The actors involved in the network may belong to the following sectors: the manufacturing sector (M), the public services sector (PS), the market services sector (MS), the third sector (TS) consisting of associations, non-governmental organizations (NGOs), voluntary groups, social enterprises, cooperatives and mutual societies. The network can also involve individuals (C) considered from different facets: individual citizens, users and especially lead users and consumers. In theory, actors belonging to each of these categories (M, PS, MS, TS, C) can play a role, in one way or another, in each of the types of networks. But, in reality, depending on the type of network considered, some of these sectors or agents are predominant in the network. They are represented in bold large letters in Figure 1. These networks may be set up to achieve different forms of innovation (technological and/or non-technological innovations), different scales of innovation (incremental or radical innovation, simple innovation or complex/architectural innovation) and innovations originating from different sources (adopted innovation or produced innovation). These innovations can be aimed at different sectors (manufacturing industry, market services, public services).

The public agent (actually the public administration) can play two different roles, exclusively or jointly: on the one hand, a role of co-production of the innovation strictly speaking (innovator in its own right) and, on the other hand, a role of support/facilitator of the innovation or the constitution of the network.

Figure 1 shows the general definitions of each of the innovation networks that we are investigating, and which we will discuss further in the following paragraphs. TINs are networks that focus on the manufacturing industry and technological innovation (visible innovation) and in which the public administration is not a co-producer of innovation, but a facilitator. PPINSs are systems of service-oriented collaborations, public-private collaborations, open to non-technological innovation. PSINs focus on innovation in public services. The main actors in this type of network are citizens, public sector and third sector organizations. Finally, PSINSIs are a special subcategory of PSINs dedicated to social innovation. The different types of networks envisaged, which are not independent of each other, can be characterized by their degree of visibility, that is to say, their level of recognition by economic analysis. Traditional innovation networks (TINs) thus constitute the visible tip of the iceberg of innovation networks (see Figure 2), while the other less known types of innovation networks are the submerged parts.

Over the last 30 years, the concept of (traditional) innovation network has been quite successful and has given rise to a great deal of literature. As we shall see in section 2, this success can be explained by the simplicity of the concept, its powerful heuristic value, its congruence with both broader concepts (the concept of innovation system in its various forms), and narrower concepts relating in particular to the different learning processes and the dynamics of collaborative innovation.

The extension of innovation networks to services and service innovation has attracted attention from researchers more recently (see [START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF]. For the most part, PPINSs remained invisible to economic analysis (submerged part of the iceberg in Figure 2). There are, however, a number of exceptions that correspond to innovation networks centred on market and/or non-market services, but focused primarily on technological innovation. The most obvious exceptions are health innovation networks [START_REF] Djellal | Innovation in hospitals: a survey of the literature[END_REF]. But there are others in the field of transport, tourism, defence and broadcasting services (the EU funded ServPPIN project provides a number of case studies of such exceptions). As can be seen in Figure 2, this "visible" part of the PPINSs is illustrated by the exposed part of the PPINSs block, which overlaps the TINs block.

The literature on PSINs and PSINSIs is the least extensive. It is still in its "infancy" [START_REF] Sørensen | Collaborative Innovation in the Public Sector: An Analytical Framework[END_REF]. This gap in the literature can be explained by the existence of a certain mistrust vis-à-vis notions (collaboration, partnership, network) which, in the case of immaterial, non-spectacular and frugal innovations, at work in PSINs and PSINSIs, may appear to be mere rhetorical tools [START_REF] Atkinson | Discourses of partnership and empowerment in contemporary British urban regeneration[END_REF][START_REF] Hastings | Unravelling the process of "Partnership" in urban regeneration policy[END_REF][START_REF] Lyon | Social innovation, co-operation and competition: inter-organisational relations for social enterprises in the delivery of public services Third Sector Research Centre[END_REF] rather than desirable and effective innovation arrangements. This is not the case for traditional innovation networks, which are taken seriously because they are designed to develop and implement sophisticated R&D-based industrial and technological innovations.

The different types of innovation networks are not independent of each other (see Figure 2). As we have already pointed out, there is an intersection between TINs and PPINSs. This intersection equates to certain PPINSs which are focused on technological innovation. PSINSIs are a sub-category of PSINs whose target is social innovation in public services. These two forms of innovation networks (PSINs and PSINSIs) are themselves sub-categories of PPINSs. PSINs PSINSIs can be explained in different ways [START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF]. It is explained, first of all, theoretically, by its great simplicity and its great heuristic value. After all, an innovation network seems to be nothing more than a set of nodes and links. The strong theoretical scope of this concept is, moreover, reinforced by its ability to be part of concepts that are themselves quite successful, in particular the concepts of innovation systems in their various expressions (local, regional, national systems, sectoral systems, innovative milieus, technology districts, technopoles or clusters). Indeed, (innovation) networks constitute the core elements of these concepts [START_REF] Grabher | Trading routes, bypasses and risky intersections: mapping the travels of "networks" between economic sociology and economic geography[END_REF][START_REF] Glückler | Economic geography and the evolution of networks[END_REF][START_REF] Freeman | Technology policy and economic performance: lessons from Japan[END_REF][START_REF] Carlsson | On the nature, function and composition of technological system[END_REF]. The strong theoretical scope of this concept also owes much to its ability to assimilate itself to other concepts (learning, absorption capacity, scale, scope and agglomeration economies, transaction costs, network externalities, etc.) and other theories (collective innovation, userdriven innovation, open innovation, innovation communities...). The success of the traditional innovation network concept is also due to its operational and political use. The notion of innovation network and the associated notion of innovation system, in its various forms, give rise to interesting operational frameworks for mapping innovation dynamics for auditing, performance comparison and benchmarking. The concept of innovation network is also a key component of many public policies supporting innovation at different levels (supra-national, national, regional, local). Thus, the notions of National Systems of Innovation (NSI) and the networks that constitute them remain key components of national and European innovation policies. The notions of regional innovation systems (RIS) and clusters (like innovative milieus or industrial districts in past decades) are today central to local and national policies in many countries, though they may have different names: "skill clusters" in Germany, "competitiveness clusters" in France, "knowledge clusters" and "industrial clusters" in Japan.

However, as theorized and experienced, innovation networks (and also systems), have a number of weaknesses, particularly when viewed from the perspective of a service and sustainable development economy. These weaknesses concern the nature of the stakeholders involved in the network and the nature of the innovation addressed by the collaboration. They reflect three biases (industrialist, market and technologist), which are not independent of each other and which contribute to a fourth bias in terms of public policy (see Figure 4). Traditional innovation networks (TINs) are characterized by an industrial, technological and market bias. After all, the dominant agents within these networks generally belong to the industrial sector (M) and the market sector, and the main targets of the network are technological innovations with a strong scientific and technical R&D dimension (see Figure 4). The pre-eminence of agents from the industrial and market sectors does not mean that agents belonging to other sectors are totally absent from these networks, just that their "role" is less important.

Thus, certain market services (MS) can take part in TINs. But the market services concerned are usually limited to Knowledge Intensive Business Services (KIBS): consultants of all types and financial services (investment banks or business angels). Other types of services are most often absent from traditional innovation networks. Moreover, regarding their role, these KIBS are not core elements of the network: after all, most generally, they only play a support role in favour of manufacturing firms that are the central agents.

Similarly, even if TINs are dominated by a market logic, public service actors also frequently take part in them. However, here again, as for market services, the public services concerned and, for some of them, their scope for action in terms of innovation are limited. Only two groups of public services are involved: universities and public research laboratories on the one hand, and local, regional or national public administrations on the other. The functions assigned to each of these groups in TINs are well known and documented: for public research bodies in science and technology (research centres, universities), the purpose is to participate upstream in the production of technological innovation (basic and applied research), and for public administrations, the purpose is to ensure meta-governance, in other words to promote an environment conducive to innovation and to the formation of partnerships (establish a favourable legal environment, provide financial support, encourage industrial firms to work more closely with universities and research centres). It is important to emphasize that the technological and market bias that characterizes traditional innovation networks makes it impossible to consider innovation activity specific to public administrations that would be the fruit of collaboration between different agents. The purpose of the PSINs and PSINSIs we discuss in Section 4 is to account for innovation in public services or for public service innovation strictly speaking and how it can also emerge from networks.

Overall, the triad composed of an industrial firm (producer of innovation), public research (co-producer of innovation) and public administration (promoter of innovation), is the standard form of TIN. It has been the subject of many theoretical models. These include, for example, the so-called "triple helix" model (Etzkovitch and Leydesdorff, 2000), which describes the processes of knowledge production in hybrid networks involving companies, universities and government agencies. These also include the so-called "mode 2" of knowledge production developed by [START_REF] Gibbons | The new production of knowledge: the dynamics of science and research in contemporary societies[END_REF], see also [START_REF] Gibbons | Mode 2 society and the emergence of context-sensitive science[END_REF] which describes a network of multidisciplinary actors, interacting to find solutions to the technological problems raised by industry. It is the industrial firm that is the centre of these collaborative modes of knowledge production, or which is intended to be their centre, as the life cycle of the network evolves. The analyses of network life cycles illustrate a decline in the participation of public actors over time. The maturity phase of innovation networks is clearly dominated by private industrial firms.

It should be noted that, especially starting from the precursor work of Von [START_REF] Hippel | Lead Users: A Source of Novel Product Concepts[END_REF], these (traditional) innovation networks also begin to take into consideration the user (C) and in particular the lead user as a significant actor in innovation dynamics.

The industrial, technological and market biases that characterize TINs, and which interact with each other, lead to a bias in the public policy designed to promote innovation (see Figure 4). Indeed, TINs, whether as a public policy instrument or as a public policy target, mainly promote technological innovation based on R&D and science and technology. The PPINSs addressed in the next section help to reduce all four of these biases.

Public-Private Innovation Networks in Services (PPINSs)

PPINSs are networks that have begun to interest research more recently (ServPPIN project funded by the European Commission, see [START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF]. They describe collaborations in the field of innovation between public and private service organizations. They should not be confused with public-private partnerships (PPPs), which develop within the framework of the new public management paradigm. PPPs are generally focused on service production and not on innovation, and their rationale is based on the idea that introducing a market logic is good for performance, whereas in PPINSs, what is good for performance is the hybridization of knowledge and skills. Finally, PPPs are formalized in contracts while PPINSs are more flexible structural arrangements.

In this new type of innovation network, the dominant agents belong to market services (MS) and non-market services (PS and TS) (see Figure 1). In addition, a new target appears alongside technological innovation, namely non-technological innovation which is given great importance. Thus, PPINSs bypass the technological, industrial and market biases of TINs that we have outlined previously.

• PPINSs correct the industrial bias of traditional INs by giving a central place to market services (MS). The status of services is raised both in terms of the nature of the services concerned and their function/place in the innovation process. First of all, in PPINSs, not just KIBS and financial services, but any service activity can be part of the innovation network.

The PPINSs database3 of the ServPPIN project provides the following examples: consultants, a TV channel, travel agencies and tour operators, private elder care services, transport companies, etc. Second, in PPINSs, these services no longer occupy a peripheral position in the innovation network, but rather a central one. They are now the key actors, the nodes of the networks and the main actors of innovation, which itself is broader in nature, since it includes the different forms of so-called invisible innovation (see Figure 4).

• PPINSs also correct the market bias of TINs by giving a central place to public and nonmarket services and to public-private collaboration in the network. Thus, a wide range of organizations belonging to the public sector (PS), but also to the semi-public and the so-called third sector (TS) (associations, non-governmental organizations, etc.) take part and occupy an important place in the network. The PPINs database of the ServPPIN project provides the following examples: the Red Cross, a municipality, a development agency, a chamber of commerce and industry, a tourism union, a transport union, the institutions of the labour market (collaboration between employers and unions), a foundation and so on. The new public actors involved also include research networks in human and social sciences.

• One of the key characteristics of PPINSs that distinguishes them from TINs is that any public service activity/organization, and not just public research organizations (universities, research laboratories), can perform a co-innovation activity strictly speaking. As in the case of services (see previous point), PPINSs thus make it possible to include non-technological forms of innovation in networks. They also make it possible to account for an area of innovation that is still largely under-exploited, namely innovation in public services (Windrum and Koch, 2008;[START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF]Fuglsang et al., 2014).

Thus, while TINs are focused, for the most part, on technological innovation, PPINSs are based on a broader and open concept of innovation that includes both visible (technological) innovations and invisible (non-technological) innovations, predictable (planned) innovations and unpredictable (unplanned or emerging) innovations (see Figure 4).

Visible innovations are those that are perceived by traditional analytical tools, such as R&D and patents. They reflect a technologist and assimilationist conception of innovation in services, which renders much of the innovation dynamics in services invisible [START_REF] Gallouj | Innovation in the service economy: the new wealth of nations[END_REF]. Invisible innovations are a heterogeneous category, often grouped under the term nontechnological innovations. They can take different forms: organizational, social, marketing, and so on. They reflect a service-oriented or demarcative conception of innovation in services [START_REF] Gallouj | Innovation in the service economy: the new wealth of nations[END_REF] (Iceberg A in Figure 4).

Predictable or planned innovations are incorporated into well-identified and formalized structures (for example, R&D or innovation departments, permanent or transitional innovation project groups and so on) and into well-established, more or less complex, stagegate processes (linear NPD-NSD models) or interactive, chain-linked models as described by Kline and Rosenberg [START_REF] Kline | An overview of innovation[END_REF]. These models are the application to services of traditional (manufacturing) innovation models. They therefore reflect an assimilation view of innovation organization modes, and they are included in the visible tip of Iceberg B in Figure 4. Unpredictable, unprogrammed or non-planned innovations are embedded into informal and loosely coupled structures and in "emerging" spontaneous processes. Within this general category, the literature distinguishes several types of innovation models that were for many years underestimated (submerged part of Iceberg B): bricolage model [START_REF] Fuglsang | Bricolage and invisible innovation in public service innovation[END_REF], ad hoc or a posteriori recognition model [START_REF] Gallouj | Innovation in the service economy: the new wealth of nations[END_REF], rapid application model [START_REF] Toivonen | Different types of innovation processes in services and their organisational implications[END_REF] and so on. In the bricolage or tinkering model, innovation is the result of unplanned activities carried out in response to random events and characterized by trial and error and 'learning on the job' [START_REF] Sanger | Using Old Stuff in New Ways: Innovation as a Case of Evolutionary Tinkering[END_REF][START_REF] Styhre | Tinkering with material resources: Operating under ambiguous conditions in rock construction work[END_REF][START_REF] Fuglsang | Bricolage and invisible innovation in public service innovation[END_REF]. Ad hoc innovation [START_REF] Gallouj | [END_REF] is described as the process of co-construction with the customer of a (novel) solution to a problem. This interactive, unplanned and "emerging" process cannot be separated from the process of service provision (or only in retrospect after the event). In the rapid application model, finally, once the idea has emerged, it is immediately developed as the service in question is being provided. Planning does not precede production. The service provision process and the innovation process are one and the same [START_REF] Toivonen | Innovation process interlinked with the process of service delivery: a management challenge in KIBS, Economies et Sociétés[END_REF]. The latest application of the concept of innovation network is to public services themselves and collaborative innovation in public services. We call these new kinds of innovation networks Public Service Innovation Networks (PSINs). We start by providing a general definition of PSINs, and then we examine their general characteristics using various typologies.

The definition of PSINs

PSINs, which are very successful within the "new public governance paradigm", are collaborative arrangements implemented in public services in order to create value through a process of co-innovation. They bring into play various public and private agents, especially citizens, in order to co-produce innovations in the field of public services (sector) or of public service (function), whatever the nature of the innovation in question: new service, new organization, new process or new delivery method, mix of these innovations, new reform.

As Figure 1 illustrates, although any type of public and private actor can be part of PSINs, the main actors generally belong to the following three groups: public services (PS), third sector (TS) and individual citizens (C). A key element in PSINs is that the target of collaborative innovation is the public service itself. It is the public service that is the subject of innovation. Although this is not visible in Figure 1, it should nevertheless be noted that it is not uncommon for the public actor to be absent from the PSINs throughout their life cycle or at certain periods of the life cycle. After all, as already stated, PSINs are concerned with both innovation in public services as an activity or sector and with public service innovation with public service viewed as a function of general interest even beyond public sectors. Indeed, an innovation of general interest can be provided by a network of private (market or non-market) actors, specifically because the public actor has been failing on a given "market", either because it has withdrawn from or does not have the resources or the desire to serve that market. This failure or lack of interest of the public actor is not uncommon in the particular case of PSINs centred on the resolution of wicked social problems and promoters of social innovation, networks that we call PSINSIs.

Different typologies of PSINs

PSINs can be described by using a number of typologies, which can be based on the following criteria: 1) the (sectoral or functional) fields where networks are set up; 2) the type of actors involved; 3) the nature of the innovation provided by the network; 4) the mode of formation and functioning of the network.

PSINs according to the fields where they are set up

The fields where PSINs are set up can be addressed in different ways: for example, through accountancy-based typologies of public service activities or through typologies that reflect the major problems or social needs of the moment.

In accountancy-based typologies, a distinction can be made, for example, between the following sub-sectors:

-sovereign public services (order and security), -public services regulating private activities, -public health and social protection services, -educational and cultural public services, -industrial and commercial public services. This typology can be simplified by distinguishing between general services, social services and utilities. PSINs can be created in any one of these categories. However, it should be noted that social services constitute a particularly favourable ground for PSINs set up for the implementation of social innovation (i.e. PSINSIs).

In typologies that reflect major social problems or needs, a distinction can be made, for example, between: health, education, mobility, employment, transport, security and so on. All these major social problems or needs can be the subject of PSINs or PSINSIs. For example, the Danish CLIPS project presents 14 case studies of collaborative public service innovation related to crime prevention in a local environment [START_REF] Sørensen | Enhancing Social Innovation by Rethinking Collaboration, Leadership and Public Governance, Paper presented at NESTA Social Frontiers[END_REF]. Social problems at the origin of PSINs include what the literature calls "wicked problems". Wicked problems are complex, multiform, systemic and often conflicting problems, which cannot be solved by a single actor, but which require multi-stakeholder collaboration. They include problems related to caring for an aging population (in terms of health, housing, mobility and so on), the decay of certain suburbs, environmental degradation, caring for refugees and so on. Regardless of the field of activity, PSINs are concerned by wicked problems, but PSINSIs centred on social innovation are even more focused on these problems. It is this focus on solving major social problems through social innovation that defines PSINSIs and distinguishes them from PSINs in general.

However, whether wicked or not, "problems" shouldn't necessarily be given a negative and reactive meaning (in this case social difficulties). As Milan Kubr (1988) suggests (in the context of consulting, it is true), though there are "corrective" problems, there are also "progressive" and "creative" problems. In the former case, innovation is a therapy undertaken to correct a difficult situation. In the second case, it is a matter of improving a given situation that is not yet bad, but which is expected to deteriorate over time. In the third case, it involves designing a totally new and better solution, without there being any real problem to be solved a priori.

PSINs according to the type of actors involved

A typology of PSINs based on the nature of the actors involved in the network can be envisaged. Such a typology would include the following categories: 1) Networks made up of both public and private agents. This first group can itself be broken down into different sub-types, in particular by dividing the category of private actors into market private actors (companies, consultants) and non-market private actors (associations, citizens, and so on).

2) Networks consisting only of public agents belonging to different public organizations. It is necessary to distinguish, on the one hand, the relationships between different levels of the same administration, which do not constitute a network strictly speaking (since these relationships remain embedded in a given hierarchy: a given administration being the equivalent of a company, which can be broken down at different geographical levels), and, on the other hand, the relationships between different public organizations, which do involve a networked structure. Such networks are more often formed in the context of non-social public service innovations (PSIs) rather than social PSIs. They may seek economies of scale when they involve public actors who deliver the same services in different geographical areas (for example, waste processing) or when they involve public actors which deliver different but complementary services, e.g. health and social care or police, fire and housing [START_REF] Entwistle | Collaboration and Public Services Improvement: Evidence Review Prepared for the Commission on Public Service Governance and Delivery, Public Policy Institute for Wales[END_REF].

3) Networks consisting only of private agents, working collectively to co-produce an innovation that falls within the scope of public service, not in its sectoral sense but in its functional sense (i.e. services of general interest). As already mentioned above, this configuration is a public service innovation network but not an innovation network in public services. These networks are more often formed to develop social innovations strictly speaking. They are therefore PSINSIs. The distribution of these different types of networks follows a Gaussian law, in which the dominant form is the first one (networks made up of both public and private agents). Although, this is not statistically significant, it can be noted that, in the Danish CLIPS project mentioned above [START_REF] Sørensen | Enhancing Social Innovation by Rethinking Collaboration, Leadership and Public Governance, Paper presented at NESTA Social Frontiers[END_REF], from the 14 cases of PSINs (or more precisely of PSINSISs) envisaged, 6 belong to the first category, 4 to the second and 4 to the third. Taking the public organization as a point of reference, these three types of networks might be called, respectively, hybrid PSINs, endogenous PSINs and exogenous PSINs.

PSINs according to the nature of the innovation

As we pointed out in section 2, traditional innovation networks are essentially devoted to technological innovation. , 2013). Given the particular nature of public services and their purposes, some authors have no hesitation in considering all public innovations as social innovations, or even in considering these two categories as synonyms. In our opinion, this is neither correct nor helpful. These two sets intersect, but they are not identical. After all, the scope of social innovation goes far beyond public innovation and the scope of public innovation far beyond social innovation alone. Not all public service innovations are social innovations and not all social innovations are public service innovations. PSINs are dedicated to all forms of public service innovation, and social innovation is just one form among others, which can go beyond the scope of public service. For example, a network that is formed to facilitate the implementation of an electronic service in the administration (for example an online tax system) has no (or little) reason to be considered as involving a social innovation.

The same applies to a network of municipalities, chambers of commerce and private stakeholders set up to improve the efficiency and usability of business support services (OECD, 2014). Many other examples of these types of PSINs (not focused on social innovation) can be found in the field of general public services and support services for economic activities.

The nature of innovation can provide the basis for a fairly simple typology of PSINs that distinguishes: 1) Networks created for social innovation in public services. This is what we call PSINSIs (see Figures 1 and2).

2) Networks created for other forms of public service innovations (i.e. non-social public service innovations). In the latter group, we can distinguish between networks built for service innovations and networks built for policy innovations.

PSINS according to their mode of formation and functioning

The question of network formation distinguishes planned networks from spontaneous networks [START_REF] Doz | Formation process of R-D consortia: which path to take? where does it lead?[END_REF], Schön and Pyka, 2012[START_REF] Green | A life cycle-based taxonomy of innovation networkswith a focus on public-private collaboration[END_REF].

Planned or engineered PSINs are established under the impetus of an initiating agent, a triggering entity that will invite other potential members to join the network. In theory, the initiator of the network may be any agent. In reality, however, it seems that in PSINs, the initiating agent is very often the public administration itself. The situation is different for planned PSINSIs which are most often initiated by private agents (citizens, associations and so on).

Spontaneous or emerging PSINs emerge in a self-organized way because of the convergence of the activities of agents facing a given problem, in a given context (a district, a city, a region, etc.). Here again, although, in theory, the spontaneous emergence may involve any agent, the spontaneous (self-organized) networks more often involve citizens (and not government). The principle of "self-organization", also called "self-governance", reflects the emergence of collective action within non-public agents without the intervention of the public decision-maker (government) [START_REF] Bekkers | The Social Innovation Perspective in the Public Sector: Co-creation, Self-organization and Meta-Governance[END_REF]. The spontaneous emergence of this type of network can be explained by the lack of public solutions to a given social problem or the ineffectiveness of the existing solutions.

The modes of formation of PSINs lead to a (simplified) distinction between two opposite modes of functioning [START_REF] Pyka | Taxonomy of innovation, cooperation and networks in service industries[END_REF][START_REF] Sundbo | Public-private networks and service innovation in knowledge intensive services: a report of European case studies, ServPPIN project[END_REF]:

-a vertical or institutional or top-down mode of functioning, in which, after the network is established, the initiating agent continues to enjoy a privileged "hierarchical" position: it is the conductor, the hub actor or the system integrator.

-a horizontal or bottom-up mode of functioning, which favours local interactions and in which responsibilities and leadership are more shared. The terms "distributed networks" or "distributed leadership" (as opposed to traditional entrepreneurial (heroic) leadership) are used to describe this second mode of functioning.

Conclusion

Recognition of the importance of collaborative innovation occupies a key place in Innovation Studies. Thus, in the list of the 20 main advances in this field, over the last fifty years, established by Ben [START_REF] Martin | Twenty challenges for innovation studies, SPRU Working Paper Series[END_REF], four explicitly concern the collaborative and network nature of research and innovation. Martin states these advances in the following terms: 1) From the linear model to an interactive "chain-link" model; 2) From individual actors to systems of innovation; 3) From closed to open innovation; 4) From "Mode 1" to "Mode 2".

The advances discussed by Martin mainly concern collaborations and networks whose key actors are manufacturing firms and whose main purpose is technological innovation, based on scientific and technical research. For the most part, market services are absent from this type of collaboration, and public services are only present through research laboratories and universities and certain regulatory (metagovernance) activities targeting innovation and networks. In this traditional collaborative arrangement, non-technological innovation (new services, new organizations, new methods, etc.) is not considered as being the possible target of a network activity.

However, collaboration and networks are also at work in the field of services in general, and they may focus on non-technological innovations, as was extensively analysed, from a theoretical and empirical view point, in the ServPPIN European project [START_REF] Djellal | Two decades of research on innovation in services: Which place for public services?[END_REF].

Collaborative innovation and innovation networks are also increasingly at work in the field of public services themselves (or of public service as a function of general interest beyond public sectors strictly speaking), as the paradigm of "new public management" gives way to the paradigm of "new public governance", and as the perspective of assimilation (to industrial goods, then to market services), gives way to a perspective of integration (through the Public Service-Dominant Logic: PSDL) and demarcation (through the Public Service Logic -PSL). The rise of this type of network (in the field of public services or public service) can be explained by economic and social reasons: the limited resources of public administrations to carry out (or carry out on their own) certain existing public service activities (or new/potential and necessary ones), and the complex and multifaceted nature of "wicked" social problems which, by their nature, cannot be solved (or not satisfactorily) by the activity of a single actor.

In this article, we have discussed and compared all these old and new expressions of the notion of innovation network. The emergence of new expressions of the innovation network reflects the tertiarisation of this concept, a tertiarisation that itself reflects a broadening of the forms of innovation taken into account (not just technological innovation, but any form of innovation) and the modes of organization of innovation taken into account (not just the formal and linear modes, but also the informal and interactive modes).

The new "tertiarised" forms of innovation networks that we have discussed, whether they be PPINSs, PSINs or PSINSIs, constitute an important socio-economic issue now acknowledged by the public authorities at the national and European level. They therefore require further study by academic researchers. In the future, research should strive in particular to i) consolidate our theoretical and empirical knowledge of the modes of formation and functioning of these tertiarised networks, ii) define and build the systems to accurately measure the results and the performance of these networks, iii) suggest public policies (in particular vertical or specific ones) that would help support the formation, functioning and performance of these networks.

Figure 1 :

 1 Figure 1: Different types of innovation networks: TINs, PPINSs, PSINs, PSINSIs

  . PS = Public Service. MS = Market Service. TS = Third Sector. C = Citizens, Users (Lead Users), Consumers. Bold large letters reflect the relative importance of the agent in the network

Figure 2 :

 2 Figure 2: The innovation network iceberg

Figure

  Figure 4: The biases characterizing TINs

Figure

  Figure 5: The service innovation icebergs

  PPINSs break away from this technological bias, as they take into account both technological and non-technological innovation. The same is true, in theory, for PSINs. But, in reality, PSINs are formed, above all, in order to design and implement non-technological innovations (demarcation perspective): for example a new service, a new process, a new delivery mode, a new organization or, more generally, a mix of all of the above. Non-technological innovation can also take the form of a new public reform or a new public policy. The term "public innovation" is often used to encompass innovations in both service and policy.Among the innovations developed within PSINs, social innovation occupies an important place. It is incidentally the only object of the sub-category of PSINs that we called PSINSIs. Social innovation can cut across all the categories mentioned above, insofar as it may concern a new service, a new process, a new organization, a new reform, a new social model (as opposed to a business model) or a mix of them. Whatever its form, social innovation is social "in its ends and means", according to a now standard definition attributed to the European Commission (European Commission

This work was undertaken within the EU-funded COVAL project: "Understanding value co-creation in public services for transforming European public administrations", H2020 project

2017-2020.

This database comprises

in-depth case studies of PPINSs conducted (by means of interview-based qualitative surveys) by project participants in the following countries: France, the UK, Spain, Austria, Denmark, Norway, Slovenia and Hungary. The case studies cover health, transport, knowledge-intensive services and tourist services.