Marcus Barkowsky

Romain Cousseau

Patrick Le Callet

Is visual fatigue changing the perceived depth accuracy on an autostereoscopic display?

Keywords: 3DTV, Visual Fatigue, Depth perception, Autostereoscopic display, Subjective experiment

In this paper, a subjective study is presented which aims to measure the minimum perceivable depth difference on an autostereoscopic display in order to provide an indication for visual fatigue. The developed experimental setup was used to compare the subject's performance before and after 3D excitation on an autostereoscopic display. By comparing the results to a verification session with 2D excitation, the effect of 3D visual fatigue can be isolated. It was seen that it is possible to reach the threshold of acuity for stereo disparity on that autostereoscopic display. It was also found that the measured depth acuity is slightly higher after 3D viewing than after 2D viewing.

INTRODUCTION

Recently, 3DTV services have been introduced to the public together with a variety of stereoscopic and autostereoscopic displays. Shutter glasses based stereoscopic systems are currently most frequently used at the home but in the near future, autostereoscopic displays might become available that offers a sufficiently high resolution and that benefits from the fact that wearing eye-glasses is not mandatory to see the 3D effects.

At the moment, it is often considered that one of the major risks in the wide acceptance of 3D television is visual discomfort and visual fatigue. Some sources of immediate visual discomfort have been identified, e.g. the vergence/accommodation conflict [START_REF] Yano | A study of visual fatigue and visual comfort for 3D HDTV/HDTV images[END_REF], rapid changes in horizontal disparity [START_REF] Yano | Two factors in visual fatigue caused by stereoscopic HDTV images[END_REF] [START_REF] Emoto | Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television[END_REF], and objects moving regularly in depth, e.g. continuous changes of horizontal disparity [START_REF] Speranza | Effect of disparity and motion on visual comfort of stereoscopic images[END_REF]. Recent surveys on the most likely causes of visual fatigue and visual discomfort can be found in [START_REF] Li | Measurement of 3D Visual Fatigue using Event Related Potential (ERP): 3D Oddball Paradigm[END_REF] [START_REF] Lambooij | Visual Discomfort in Stereoscopic Displays: A Review[END_REF].

Several methods have been tested in order to measure visual fatigue with questionnaires [START_REF] Kuze | Subjective evaluation of visual fatigue caused by motion images[END_REF] and to identify observers that are particularly sensitive to it [START_REF] Lambooij | Measuring Visual Discomfort associated with 3D Displays[END_REF]. Other alternative have been tested , ranging from a task performance measurement [START_REF] Barkowsky | The influence of autostereoscopic 3D displays on subsequent task performance[END_REF] to analyzing an Electroencephalogram (EEG) [START_REF] Ukai | Visual fatigue caused by viewing stereoscopic motion images: Background, theories, and observations[END_REF]. However, so far, no reliable method has been found that is simple to apply to a large panel of naîve observers. The human visual system (HVS) is capable of automatically controlling the accommodation and vergence of the eyes. In fact, each change in accommodation immediately triggers a change in vergence and vice versa. The rate at which one system influences the other is known as the accommodation/convergence ratio (CA/C) and convergence/accommodation ratio (AC/A). A recent detailed study on the relationship between the two ratios and the individual difference that might occur can be found in [START_REF] Fukushima | The relationship between CA/C ratio and individual differences in dynamic accommodative responses while viewing stereoscopic images[END_REF]. The experimental setup is similar to the situation found on stereoscopic flat panel displays. It was seen that a significant overshoot of the accommodation/vergence system is seen for some subjects and that the stereo presentation might lead to a conflict between the two coupled adaptation mechanisms AC/A and CA/C. This seems in line with an experiment that we conducted earlier [START_REF] Barkowsky | The influence of autostereoscopic 3D displays on subsequent task performance[END_REF]. We conducted an optometric fusional vergence facility test at near [START_REF] Carlson | Clinical procedures for ocular examination[END_REF] before and after viewing the same content in 2D and 3D presentation mode. In this optometric prism facility test, the ability of a person is measured to change its convergence voluntarily because he needs to compensate for the offset induced by a prism in his visual pathway. This decoupling ability may be seen as inhibiting to a certain extent the automatic adaptation ability measured in the AC/A and CA/C factors. The speed of correct convergence was faster after 3D viewing than after 2D viewing. One possible reason is that this might be due to the unnatural viewing condition: The observer learns how to decouple the convergence system from the accommodation.

It might be anticipated that the ability of changing the vergence independently from the accommodation reduces the acuity of the HVS to depth differences. This will be evaluated in this paper by displaying stimuli that only exhibit disparity information and by asking the observer to indicate whether a difference between two stimuli is visible and thus above their depth acuity threshold.

The paper is organized as follows: In Section 2, the subjective experiment method will be explained and in Section 3 the results of the study will be presented including a detailed discussion of the results before concluding the paper in the last section.

SUBJECTIVE EXPERIMENT SETUP

The subjective experiment was split into two sessions. Each observer participated in both sessions which were separated by at least three days in order to avoid memorization effects.

The overall setup of the sessions is depicted in Figure 1. A total of ten naïve observers participated in the experiment. All observers started with the 3D session (session A). The 2D session (session B) only differs in the excitation part. In order to excite the observers with 3D content and to assure that they stayed focused on the display, the observers were playing a simple car racing videogame (flatout). As the own car was displayed coming out in front of the screen, this setup exhibited visual fatigue in the 3D viewing case. In the second session which was separated by at least three days from the first session, the same setup was used except that they played in 2D instead of 3D.

At the beginning of their respective first session, they were screened for visual acuity using a Snellen Chart, for depth perception using a Randot Stereo Test, and for correct color vision using Ishihara plates. Their dominating eye was also determined in two ways, firstly using the room corner as a fix-point for covering with the thumb (far vision) and taking the view-finder of a camera close to the dominating eye (near vision). For all our observers, the two tests agreed about the position of the dominating eye. In total, there were 5 right-eye dominant and 5 left-eye dominant viewers.

Figure 1: Session setup

The display that was used for displaying the stimulus was the Philips 42" autostereoscopic display. This display was used for the depth experiments and for the 2D and 3D excitation. By using a lenticular array foil in front of an HDTV panel, this display is able to produce nine distinct views in horizontal direction. In order to facilitate the rendering of the nine views, it contains a rendering unit that takes a texture and a depth image as input. The advantage is that the user does not need to generate the nine views himself but the disadvantage is that the influence on the final rendering of the screen can not be fully controlled. In particular, we have already shown that, in a direct comparison, observers prefer to view 2D presentation to 3D presentation on this screen for most contents [START_REF] Barkowsky | Influence of Depth Rendering on the Quality of Experience for an Autostereoscopic Display[END_REF]. As it was seen that the display processing adds blur to planes that are far away from the screen plane (either in the foreground or in the background), this might influence the results of the depth test as it plays the role of an additional monocular depth cue. Thus, it was decided to use only a small disparity offset in the depth experiment.

Optometric test: Fusional vergence facility test at near

The excitation session was preceded and followed by an examination of the fusional vergence facility using a prism test: Each observer viewed a vertical line of letters at a comfortable reading distance of about 40cm. A prism of 12 diopters outwards was then introduced into the view of one eye which led to double vision. The participant was asked to fuse the two lines again into one. As soon as he succeeded, he changed the prism to 3 diopters inwards and again had to fuse the two appearing lines of characters. After a training phase, the number of full cycles was counted for one minute.

Stimulus for the depth experiment

In the experiment, the observer was asked to compare the position in depth of two plates textured with a Maltese Cross. The stimulus itself was chosen to contain only disparity information and no additional depth cues. In order to support the depth impression of the observers, the rest of the scene contained some depth information. The stimulus is shown in Figure 2. On the left side, the texture is displayed and on the right side, the 8-bit depth map coded into luminance values can be seen. In total, for symmetry reasons, four plates with a Maltese Cross were displayed but only the upper two were used. As can be seen from the Figure 2, no positional cue was available for the subjects as the stimuli are in front of a blue sky. In pretests, it was discovered that a more realistic condition stabilizes the perception of binocular depth. The torus form around the stimuli was added in order to produce a consistent depth perception for the subjects as this object extends largely in depth. The background was displayed at the maximum possible distance to the observers (depth value equals zero, therefore black).

Texture

Depth, black equals background

Depth experiment

In the depth experiment, the two upper stimuli showing the Maltese cross were located at different depth. One of the two stimuli had a depth of 80 while the other one varied with depth values larger than 80. In each trial, a random choice was performed, whether the fixed stimulus was displayed to the left or to the right. The observer was asked to indicate which of the stimuli was closer to the background, e.g. further away from them..

A standard 3up-1down staircase method was used in the experiment with each presentation having a maximum displayed duration of five seconds. The forced choice method was used. Visual feedback about the correctness of the answer was provided to the observers as we were interested in the minimum perceivable depth offset.

The initial stepsize for the depth value decrement was set to two and was successively halved after forty trials. This was done six times, so that in each depth measurement, each observer viewed 240 conditions with the smallest increment being 1/16. The first 6 out of the 240 conditions were discarded for training.

In order to analyze the experimental data from this experiment later on, several preconditions regarding the display need to be known. For this experiment, the settings of the screen were left at the default setting for "signature content". While the rendering system of the Philips screen automatically determines the disparity of the objects based on the values of the depth map, for our analysis it is important to know exactly the disparity of the stimulus on the screen. Several test stimuli were displayed on the screen at different depth and using a macro-capable camera, we were able to determine for each setting the corresponding disparity in pixels and in meters in between two adjacent views. The disparity on the screen d measured in millimeters in function of the transmitted depth value v was thus empirically determined to correspond to: ݀ = -0.036 • ݒ( -128). This is the distance between two corresponding points on the screen, e.g. in x-direction. The resulting functions in terms of visual angle α and absolute distance to the screen f in function of the 8-bit depth value v used in the transmitted depth map are displayed in Figure 3. The distance to the screen specifies the virtual position of the object in front (negative value) or behind (positive value) the screen, thus in z-direction. Please note that the screen has a fixed viewing distance of g=3.5m due to its lenticular array geometry. An average inter-ocular distance of e=65mm was used. As can be seen, both the visual angle and the distance of the displayed objects to the screen behave mostly linear to the transmitted depth value which simplifies the analysis of the results later on. The linear gradients correspond to approximately 2.1 arcsec/depth-value and -2 mm/depth_value.

RESULTS AND DISCUSSION

Fusional vergence facility test at near

The number of cycles per minute that the observers were able to perform with the two prism settings was recorded for the dominating eye and the non-dominating eye. The results are depicted in Figure 4. On the left hand side, the dominant eye for each observer is examined before and after each of the two sessions. It can be seen that most observers were able to do about 20 cycles per minute but two observers were significantly slower with one not being able to perform the fusion at all. On the right hand side the additional number of cycles after each session is displayed. As can be seen, both after 2D excitation and after 3D excitation, the observers performed the optometric test faster than at the start of the session.

For the dominating eye, in 6 cases the increase after 3D viewing was larger than after 2D viewing which is particularly visible for the observers 2, 6, 9 and 10 with respectively 4, 8, 12 and 7 additionnal cycles for 3D compared to 2D. The observers 5 and 8 showed a slightly larger increase after 2D viewing than after 2D viewing with 3 and 2 cycles more. It should be noted that the onetime counting of the number of cycles for one minute in the optometric test involves a certain quantization error as the minute usually finishes within performing a cycle.

For the non-dominating eye, the distinction between 2D and 3D viewing is not clearly visible. Four observers showed a higher increase after 2D viewing while another four observers showed a higher increase after 3D viewing. However, in both cases, the number of additional cycles is very small, the maximum being 5 for observer 6 in preference of the 3D session.

Depth plate test

The result of the subjective experiment using the depth plates consists of 240 Forced-Choice Paired Comparison decisions per observer. As the first six trials were disregarded for training, the analysis was done with 234 data points.

The number of correct decisions is counted for each disparity value displayed on the screen. When using a quantization step size for the disparity data of one, this leads to the bar chart shown in Figure 5. Although the data is noisy, it is possible to fit a cumulative Weibull function as shown in this example using Matlab and the Psychophysics Toolbox extension [START_REF] Brainard | The Psychophysics Toolbox[END_REF]. A second fit using the Quest algorithm was performed in addition. Both results are shown in the figure.

The diagram also shows the 75% and 92% points of correct detection for both function fits. In the remaining paper, the discussion will focus on the Weibull function fit.

Figure 5: Example for function fit on the experimental data using the Weibull function

From those functions, the 75% point of correct detection can be determined for each observer before and after 2D and 3D viewing, leading to four values per observer. In Figure 6 it can be seen that the 75% threshold is reached on average with a difference in depth of 3.7 levels while the maximum is at 6 levels. For the 92% detection threshold, the mean value is at approximately 6.0 depth values with the maximum close to 10. Using the results presented in Section 2.3, showing that on this display, the visual angle is approximately linear to the depth value with a gradient of 2.1 arcsec/depth-value, the average value for 75% detection corresponds to 7.8 arcsec and a correct detection of 92% is achieved at 12.6 arcsec.

These values are at the lower end of the range of the stereo acuity reported in the literature, e.g. 20 arcsec have been reported in [START_REF] Schor | Disparity range for local stereopsis as a function of luminance spatial frequency[END_REF]. This result is surprising as it was anticipated that the large amount of crosstalk on the display would render the detection task more difficult. In [START_REF] De Silva | 3D video assessment with Just Noticeable Difference in Depth evaluation[END_REF] a similar experiment was performed on the same display but with a different display setting: The depth range was compressed by a factor of about 1.7. The authors reported that about 20 depth levels are required to see a difference which would correspond to about 12 depth levels in our case. As they used an automatic staircase method that is likely to provide a higher detection threshold, the results can be considered coherent.

In order to compare the subject's behavior for the 2D and 3D viewing sessions, Figure 7 displays the difference for the 75% and the 92% detection threshold. In contrast to Figure 6, the ordinate directly provides the accuracy gain measured in arcsec. It is obvious that for all except one observer, the accuracy increase was higher after the 3D session than after the 2D session. For the 92% threshold, the average value for the 2D case is -0.3 arcsec indicating a slight decrease of the threshold while for the 3D case, an average of 2.3 arcsec is obtained. Under the assumption that the observeracuity follows a Gaussian distribution, a Student T-Test can be conducted to compare the mean values of the two distributions. It is found that there is a significant difference between the 2D case and the 3D case on a 95% confidence level for the 92% threshold but not in the 75% threshold. However, the number of subjects should be considered as small. These results indicate that there is an increase in accuracy after having watched 3D content on the same screen. One drawback of the experiment conducted so far is that the Philips autostereoscopic screen induces a lot of crosstalk and that the rendering process might introduce artifacts that facilitate or harden the task. It should also be noted that for all observers the first session was the 3D session and thus, they might have performed differently at the very first presentation. However, with 234 depth trials this can be rejected and no indication has been found in Figure 6.

The results also indicate a very high stereo acuity compared to the literature, in particular when the experimental setup is taken into consideration. It should be noted that the measured stereo acuity of 12.6 arcsecs is higher than the typical value for the spatial acuity of the human observer which is in the range of 60 arcsecs for a 20/20 observer. This might indicate that the horizontal resolution of a 3D display needs to be higher than the vertical resolution in order to provide the correct disparity information on the location of objects in depth while the texture information has a lower cut-off frequency.

CONCLUSIONS

In this study, a psychometric experiment has been conducted on a Philips autostereoscopic display in order to find a way to measure differences between a 2D excitation session and a 3D excitation session. It was found that observers were able to detect disparity differences close to and below the acuity threshold previously reported in literature. It was also seen that the 3D excitation session did not reduce the acuity due to visual fatigue but actually increased the acuity significantly. This effect was not seen in the 2D case.

It was also possible to verify that the fusional vergence test at near for the dominant eye shows a higher improvement for the 3D session than for the 2D session, probably because of the ability of the observers to decouple their accomodation/vergence system during the 3D session. Thus, two measurements were identified which show objectively that 2D and 3D viewing on the same screen affects differently the fusional vergence facility and a measurement of depth acuity performed on the same screen. This difference might be in relation with visual fatigue as observers reported symptoms of visual fatigue after the 3D session. A larger panel of observers is however necessary to quantify the relationship between the measurements and the visual fatigue.

In particular, there are two not necessarily conflicting hypothesis that need to be investigated. In the following, it is assumed that some adaptation process took place for the observers in the 3D case and that this adaptation process is inevitable for long-term viewing on this type of screen. The first hypothesis would state that those observers which adapted only partially were experiencing the negative effects of 3D viewing, e.g. the accommodation-vergence conflict, for a long time. This would lead to higher accumulated visual fatigue with content duration. In this case, the observers with a smaller difference in terms of prism cycles or depth acuity between 2D and 3D viewing would have experienced more visual fatigue. The second hypothesis would state that the adaptation process itself is tiring for the observers. In this case, the observers that showed a larger difference between the 2D and the 3D case would have felt more visual fatigue. It is often reported that strong visual fatigue is reported only for the first few times when people start viewing 3D, so this would provide a possible explanation.

As the autostereoscopic display used in this experiment has many drawbacks, the next step will be to reproduce the experiment using a stereoscopic screen for the 2D and 3D sessions and a stereoscope for the depth acuity experiment.

Figure 2 :

 2 Figure 2: Content displayed on the screen for the depth accuracy experiment

)Figure 3 :

 3 Figure 3: Visual angle and the absolute distance to the screen in function of the transmitted depth value

Figure 4 :

 4 Figure 4: Number of cycles performed at the fusional facility test per observer

Figure 6 :

 6 Figure 6: Results of the function fit with the cumulative Weibull function at 75% and 92% correct detection threshold

Figure 7 :

 7 Figure 7: Comparison of the depth acuity before and after viewing 2D and 3D respectively


Kun Wang
email: kun.wang@acreo.se

Marcus Barkowsky
email: marcus.barkowsky@univ-nantes.fr

Romain Cousseau

Kjell Brunnström

Roger Olsson

Patrick Le Callet

M Sjöström

Subjective evaluation of HDTV stereoscopic videos in IPTV scenarios using absolute category rating

Keywords: ACR, QoE, 3DTV, Simulcast coding, Multiview coding, spatial resampling, temporal resampling

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

3D videos are riding their success from cinema to home entertainment markets such as TV, DVD, Blu-ray, video games, etc. As those popular 3D films go through the box office, a huge profit could be foreseen from bringing the 3-D experience to the home. A number of techniques have been invented and are in most cases still under fast development for representing and presenting 3D videos, e.g. multi-view, 2D video plus depth, volumetric, however, the side-by-side S3D is the most common and mature technology that has been widely used in the movie industry and today's 3DTV broadcasting for its simplicity and compatibility. S3D videos present viewers two similar images with slight visual disparity, i.e. two perspectives of the same object. With the help of the glasses each eye of the viewers will only see one of the pictures. The brain will then put corresponding points and objects together in the two images and create a sense of 3D depth through disparity. In the home environment, the channels currently used for IPTV seem most appropriate for S3D services in HDTV quality. However the transmission bitrates has a strong impact on the delivered video quality, the bandwidth requirements still a challenge for both broadcast and streaming.

Video compress encoding is one of the most common techniques for saving the transmission bandwidth. The encoding technology for 2D videos has progressed with the development of H.264/AVC coding scheme. Today, most of common 3D coding methods are based on H.264 [START_REF]Advanced video coding for generic audiovisual services[END_REF] , e.g. H.264 simulcast coding (independent bitstream coding) and multi-view coding [START_REF] Vetro | Status of MVC text, software, and conformance[END_REF] (MVC, dependent bitstream coding). For stereoscopic videos, simulcast coding encodes the left and right views separately with H.264/AVC in the very same way as it was handled in normal 2D videos. The advantage of H.264/AVC is that it can be achieved by current standards and existing hardware, hence it allows the broadcaster to use most of its 2D infrastructure even for 3D. However as the images of the different views are highly correlated a lot of information between the two views is redundant. MVC is one of the codec that uses this redundancy to improve coding efficiency by introducing inter-view prediction, where images are not only predicted from temporally neighboring images but also from corresponding images in adjacent views. Besides encoding techniques, specific data processing operations such as temporal and spatial resampling are tools that are also frequently used for more efficient recording and transmission of video signals. However when it comes to 3D videos, the question is still open which ones of the processing steps are efficient for saving bitrate as well as their impacts on perceived 3D visual quality.

To save the transmission bitrate of 3D videos, some researchers have focused on asymmetric video processing. Saygili, G.(2009) [START_REF] Saygili | 3D display dependent quality evaluation and rate allocation using scalable video coding[END_REF] employed asymmetric coding where one view was encoded by H.264/AVC and the other using scalable video coding. Scalable multiview coding (SMVC) with temporal and spatial scalability had been studied in Ozbek,N.'s (2007) paper [START_REF] Ozbek | Rate Allocation Between Views in Scalable Stereo Video Coding using an Objective Stereo Video Quality Measure[END_REF] . In Stelmach, L. (2000) [START_REF] Stelmach | Stereo image quality: effects of mixed spatiotemporal resolution[END_REF] the mixed resolution or frame rate reduction in one of the stereo views were explored.

The evaluation of the perceived video quality is of highest importance from the end user's point of view. In the 2D video case, it has been well studied in many publications, e.g. Pinson, M (2010) [START_REF] Pinson | HDTV subjective quality of H.264 vs. MPEG-2, With and Without Packet Loss[END_REF] , Barkowsky, M. (2010) [START_REF] Barkowsky | Analysis of Freely Available Dataset for HDTV including Coding and Transmission Distortions[END_REF] , and several different methods for measuring the subjective quality have been developed and tested.In the 3D case, the most common approach is to use the Double-Stimulus Continuous Quality-Scale (DSCQS) method as specified in ITU-R BT.500 8 on stereoscopic displays [START_REF] Saygili | 3D display dependent quality evaluation and rate allocation using scalable video coding[END_REF][START_REF] Ozbek | Rate Allocation Between Views in Scalable Stereo Video Coding using an Objective Stereo Video Quality Measure[END_REF][START_REF] Stelmach | Stereo image quality: effects of mixed spatiotemporal resolution[END_REF] for 3D presentation. However, the DSCQS method is time consuming since each video sequence is played at least twice, which limits the number of test sequences that can be evaluated during the subjective test. As an alternative, the Absolute Category Rating (ACR) as specified in ITU-T P.910 [START_REF]ITU-T P.910 Subjective video quality assessment methods for multimedia applications[END_REF] may be used which only plays each video once, therefore it is capable of collecting more votes from subjects. This allows to test a broader range of qualities for example on varies scenarios of high definition (HD) S3D videos. There are some other methods that have been used for the 3D subjective quality evaluation as well, such as Single Stimulus Multimedia (SSMM) stated in Merkle, P. (2007) [START_REF] Merkle | Efficient Prediction Structures for Multiview Video Coding[END_REF] which is a modified version of the Single Stimulus Impairment Scale where the same video was shown twice to viewers before rating, but there is no unimpaired reference videos in the test sequence set. Strohmeier (2010) [START_REF] Strohmeier | Open Profiling of Quality as a mixed method approach to study multimodal experienced quality[END_REF] introduced a mixed method called Open Profiling of Quality (OPQ) which combines the evaluation of quality preferences and elicitation of idiosyncratic experienced quality factors.

In this paper, we have adopted the ACR-HR (absolute category rating with hidden unimpaired reference video) subjective method which has allowed us to investigate the users' experience of stereoscopic video quality and to compare different coding and transmission scenarios and performances of state of the art video compression standards. Subjective tests in two different laboratories with the ACR method gave quality judgments based on two panels of observers. In addition to answering on a general five point ACR scale, the subjects were asked to indicate visual comfort. This paper is organized as follows. In Sec. 2, the video preparation processing is described. The subjective experiment is presented in Sec. 3. The results are reported and discussed in in Sec. 4, before concluding the work in Sec. 5.

VIDEO PROCESS AND CODING SCENARIOS

Source videos

In total, 11 source stereoscopic video sequences (SRC) were adopted for the subjective experiment. Each SRC was about 10 seconds long and had higher resolution than SDTV, covering the full range from low motion and low detail through high motion and high detail content. The scenes are summarized in Table 1.

Video processing

Several different scenarios, called Hypothetical Reference Circuits (HRC) according to the terminology of the Video Quality Experts Group (VQEG) [START_REF] Cermak | Test Plan for Evaluation of Video Quality Models for Use with High Definition TV Content[END_REF] were used in creating the Processed Video Sequences (PVS). The SRCs were transformed into PVS according to Figure 1.

Figure 1: Setup of encoding and processing steps in order to generate the PVS A SRC was firstly processed with spatial or temporal downsampling (an optional step used for certain HRCs). The spatial downsampling was performed symmetrically using a "lanczos-3 filter" with 1/4 and 1/16 of the original video size; in the temporal downsampling the video frame rate was reduced to 1/2 and 1/3 of the original frame rate, which resulting in approximately 12 and 8 frames per second. Secondly, video encoding process was added to the sequence, the H.264/AVC video encoder in its reference implementation JM 17.0 was used to create the simulcast encoded sequences, and JMVC 7.1 was used to generate multiview encoded videos. After encoding the encoded bitstream was in many cases supposed to be transmitted over a potentially packet loss network, however, it was not the purpose to study such effects in this work, so in our simulation we skipped transmission and directly decoded the bitstreams. The decoded sequences were then upsampled to their original temporal frame rate and the full HD resolution which is the same as the native display resolution of the 3D screens we used in the subjective experiment. This also approximately halves the bitrate. Further information can be found in Barkowsky, M. [START_REF] Barkowsky | Analysis of Freely Available Dataset for HDTV including Coding and Transmission Distortions[END_REF] . Please note that the bitrate at the same QP also depends on the properties of the SRC. The fixed QP approach was preferred to fixed bitrate as it helps to cover the full range of quality for each SRC. Temporal and spatial resampling were based on the simulcast coding with QP at 26. HRC 1 is an uncompressed and undistorted video that acted as a reference 3D video to compare to the other conditions. For each source video, a corresponding 2D presentation was also introduced by duplicating the left view video and displaying the same view for the left and the right eye. This provides a pure 2D impression on the 3D screen. All processes were symmetric, which means in for a certain sequences the video processing were equally imposed on both the left and the right view of the side-by-side S3D videos.

SUBJECTIVE EXPERIMENT

The subjective experiments were performed at two labs independently: at the University of Nantes IRCCyN, France (Lab 1) and Acreo AB, Sweden (Lab 2). For this cross-lab comparison, the ambient and all hardware and software at both locations were adjusted as similarly as possible. The lab environments adhere to the lab setup defined in the recommendation ITU-R BT.500-11 [START_REF] Itu | Methodology for the subjective assessment of the quality of television pictures[END_REF] . A 23" Dell LCD display (120Hz, resolution 1920x1080p) was used for displaying 3D videos in the experiment together with a pair of active shutter glasses from the Nvidia 3D vision system. The display was positioned far enough from the wall to avoid any conflicts of the displayed 3D content with the real world. The viewing distance was set to 3 times of the display height which is the same value used in the VQEG HDTV testplan [START_REF] Cermak | Test Plan for Evaluation of Video Quality Models for Use with High Definition TV Content[END_REF] .

The maximum crossed and uncrossed disparity of each SRC sequence was manually determined in order to assure that the videos were displayed in the comfortable viewing zone described in Chen, W.( 2010) [START_REF] Chen | New Requirements of Subjective Video Quality Assessment Methodologies for 3DTV[END_REF] . The voting interface for the viewers to rate the video quality was shown on a separate display. At IRCCyN, backlighting was used for adjusting the lab luminance, the luminance level of the reflection from the gray wall behind the screen was set to 50cd/m 2 which corresponded to 15% of the peak luminance of the display after passing the shutter glasses when they were activated. At Acreo the room illuminance when displays were on was set to 20 (lux) which was close to darkness for the sake of avoiding reflections from objects other than the display. No flickering was perceived in any of the laboratories. The video sequences were displayed in uncompressed format in order to make sure that all observers were given the same presentation of the same video sequence. In order to assure that no temporal distortion was introduced by the player, the videos were preloaded into the computer's Random Access Memory (RAM) and special care was taken that the playout of twice the Full-HD resolution was performed without temporal jitter. A training session containing 10 sequences was pre-conducted before the formal rating session so that observers would become accustomed to the PVSs' characteristics and the rating interfaces. Both training session and rating session were using the "absolute category rating with hidden reference" (ACR-HR) method in our experiment; hence the observers had a smooth transition from training session to rating session without feeling any boundaries. The PVSs were presented in random order and they were rated independently on the ACR category scale which is five-point quality scale defined by ITU [START_REF]ITU-T P.910 Subjective video quality assessment methods for multimedia applications[END_REF] as it shown in Figure 2 (Excellent, Good, Fair, Poor and Bad, which are later mapped to the scores 5, 4, 3, 2, and 1 respectively). The subjective test instructions, questionnaires as well as rating interface were presented in the observers' native language (French in the University of Nantes, and Swedish in Acreo, see table 3), for other international observers English were used. For each sequences, besides the evaluation for the overall video quality of 3D experience, we included a visual comfort comparison scale to evaluate the visual comfort associated with the visualization of the sequences compared to viewing on a conventional 2D television. The next presentation followed immediately † after an observer validated his / her vote on both quality of experience and visual comfort. The average time for rating a single PVS and preloading the next sequence was about 16 seconds, thus about one third of the time needed for the same experiment if it were proceed with DSCQS.

The subjective experiment contained a total of 175 videos: 10 training sequences, 11 references and 154 impaired sequences which were presented in pseudo-random order. Prior to the subjective experiment, the observers were screened for visual acuity using a Snellen Chart, stereoscopic acuity using a Randot Stereo test and color blindness. The whole experiment was divided into two sessions of approximately 50 minutes each with pauses after about 15 minutes of viewing time. In total 48 naïve observers (24 at each lab) participated the subjective experiment. After the experiment all observers' votes were screened according to ITU-R BT.500 and the VQEG HDTV testplan, 7 (2 from IRCCyN and 5 from Acreo) were rejected. The remained 41 observers consisted of 21 male and 20 female with an average age at 27.9 years old (minimum 15, median 23, maximum 64).

RESULTS

Cross lab Comparison

Figure 3 is a scatter plot of the subjective experiment data collected from two labs. All PVSs were mapped in horizontal axis with Lab 1's MOS scale and in vertical axis with Lab 2's voting scale. It shows the MOS results from the two laboratories have a similar trend, though the experiments were done in different locations and different observer groups. The green diagonal line is a reference. It indicates the ideal case in which the data from the two laboratories would match perfectly to each other. However, the real data had a small deviation downside of the diagonal line, which mean there was a difference between the data of two labs, and the observers in Lab1 (IRCCyN) were giving higher score than in Lab 2 (Acreo) for the same PVS in most of the cases. In fact the MOS from Lab 1 had slightly larger span from lowest 1.7 to highest 4.4 comparing to Lab 2 which spans from 1.9 to 4. The blue regression line is Lab 1's data mapped to Lab 2's, and the black regression line shows lab 2's data mapped to lab 1 which was actually used in the final combined data analysis.

An ANalysis Of VAriance between groups (ANOVA) [START_REF] Spiegel | Schaum's outline of theory and problems of statistics[END_REF] was performed with the laboratories as one between factor and 11 SRCs times 15 HRCs as within factors. This shows that there was a significant difference in the main effect of the laboratories F(1, 41) = 5.83 (Fisher-Snedecor distribution), the significance level or critical p-value = 0.02 < 0.05, corresponding to a 2% chance of rejecting the null hypothesis when it is true. The null hypothesis here means the distribution from the two labs stems from the same statistical process.

Figure 3; Scatter plot of data from two labs with linear regression However if we applied the linear transformation of Lab 1's to Lab 2's data by formula 1, most of the difference between the laboratories can be taken out. In other words, we kept lab 1's data untouched and rescaled lab 2's data to match lab 1.

The choice was completely arbitrary and could equally well have been done the other way around.

݁ݎܿܣ ݈݀݁݊݃݅ܽ݁ݎ ܽݐܽ݀ = 1.22 × ݁ݎܿܣ ݈ܽ݊݅݃݅ݎ ܽݐܽ݀ -0.43 (1)
We then analyzed the data again with the above mentioned ANOVA. This time the main effect of laboratories was not significant F(1, 39) = 0, p= 0.99 > 0.05. The main effects of SRC and HRC were significant with F(10, 390) = 68.80, p = 0.00 < 0.05 and F(14, 546) = 197.16, p = 0.00 < 0.05 respectively. The interaction between SRC and HRC was also significant F(140, 5460) = 6.70, p = 0.00 < 0.05. The interactions of SRC with laboratories, see Figure 4, was also significant F(10, 390) = 2.52, p = 0.01 < 0.05. The interaction between the HRC and laboratories, see Figure 5, was however not significant. Therefore the two tests were combined and analyzed as a single evaluation after aligning the data from one lab to the other. Such an alignment is often necessary between two laboratories performing the same experiment, in particular if the language is different. The notion of the absolute categories has slight offsets in different languages.

Analysis of observers' experience of 3D video quality

The average MOS as shown in Figure 4 were calculated across all 11 SRCs. The error bars indicate 95% confidence intervals. It can be seen that the MOS varies for different video source content. In general, 3D videos with the original Full HD resolution (SRC1~6 and 11) were rated better than the others (SRC7~10). The videos with lower original resolution were upscaled to Full HD resolution for displaying on the screen, hence it is reasonable to expect a lower MOS. Apart from the original resolution effect, SRC 10 was rated lowest, which may be because this 3D content had the highest crossed and uncrossed disparity among all SRCs. The content was highly detailed and the video was shot in bright sunlight, thus the background of the scene appears very bright as well. On the other end of the scale, SRC11 was rated the topmost of the visual quality. This content had a good depth presentation, slow motion and the background is dark.

Figure 5 presents the MOS partitioned by the 15 HRCs. Obviously the degree of compression during the video encoding expressed as the Quantization Parameter (QP) has great impact on the MOS. Both H.264/AVC simulcast and MVC show a similar trend, the assessed quality level decreases when the encoding QP was increased. HRC12 (spatial resolution divided by 4) got a better MOS (4, "good") than the other 3 temporal and spatial processes. It shows the statistically the same MOS as the 3D encoding only scenario with QP32. This is particularly true for those SRC that have Full-HD resolution. A detailed analysis per SRC showed that people did not notice the resolution reduction by a factor of 4. For the Full-HD SRCs, the MOS value of the reference videos was equal to 4.4, while the MOS value for the resolution reduction case was equal to 4.3. This was tested to be statistically insignificant. MOS of the temporal processes dropped moderately (3.5 for FP/2, and 3.1 for FP/3), which is about the similar quality level as coding only case at QP38. Resolution/16 reduction got the worst quality evaluation with MOS= 2.8.

As we have seen from our previous work [START_REF] Barkowsky | Subjective Quality Assessment of error concealment strategies for 3DTV in the presence of asymmetric transmission error[END_REF] , in an ACR subjective experiment, the 2D unimpaired reference presentation (HRC14 in Figure 5) may get slightly better MOS than the 3D unimpaired reference sequence (HRC1) even if this absolute difference is small. By looking further at the visual discomfort data collected during the subjective test, the discomfort values of HRC1 and HRC14 rated in a MOS scale were statistically equal, with a vote of approximately 3.5, indicating a value that may be translated to "slightly more comfortable than 2DTV". Hence the visual discomfort may not have been the factor which affected the 2D preference. However this characteristic was largely dependent on the 3D video source, e.g. for SRC11, the 3D reference videos were clearly rated superior to the 2D case. We also noticed that observers' preference of 2D video disappeared when the 2D video was heavily compressed (HRC15), in contrast with HRC4-AVC encodedat QP38, they both have a similar MOS=3.

Bitrate / MOS analysis

Figure 6 plots the average MOS versus bitrate in a semi-logarithmic scale. All SRCs are compared individually for the coding performance of H.264 simulcast and MVC. The blue and green dashed lines connect the 4 data points from the experimental data of MVC and H.264 simulcast coded videos respectively. These points present QP 44, QP 38, QP 32, QP26 that were related to their corresponding bitrates along the X axis. The figure shows that the MVC and H.264/AVC curves are quite close, however the MVC performed slightly better than H.264 simulcast. It can also be noticed that the gain decreases with higher bitrates. For most of the SRCs the QP32 and QP26 are statistically indistinguishable as the curves show a flat out trend at the top. In order to have a simple way of comparing between the cases of temporal and spatial resolution reduction to the coding only cases we introduce two factors: "bitrate gain" and "quality gain". They are demonstrated by an example in Figure 7 and equations 2 and 3. Figure 7: An example of comparison of different video processing in terms of bitrate efficiency Figure 7 plots the average rating of SRC3 for H.264 simulcast coding, MVC coding, temporal and spatial artifacts as a function of bitrate. The MVC and H.264 simulcast lines were further interpolated and extrapolated by the red curve with curve fitting tools. The HRCs of frame rate and resolution reduction are indicated by 4 individual data spots. Figure 7 shows an example calculation of the two indicators for the "resolution /4" reduction to H.264/AVC coding only cases. First, we obtained the "resolution /4" data point (x1,y1), and we drew a vertical line and a horizontal line. These lines intersect with the rate distortion curves of the coding only cases. The vertical line generates an intersection point with the H.264/AVC interpolated line at (x1, y2), and the horizontal line crosses the H.264/AVC line at (x2, y1). In some cases, "resolution/4" has a higher MOS than the MVC or the H.264/AVC curves and the horizontal line passing through this point didn't intersect with the MVC or the H.264/AVC curves. This was particularly the case when they showed a flat out trend at higher bitrates. In these cases we used the bitrate of QP26 encoded points on the MVC or simulcast curves instead in order to obtain the coordinate x2. The "bitrate gain" factor is given by formula 2, which indicates the amount of bitrate that can be saved when the MOS remains constant, i.e. the service provider offers a guaranteed quality of 3DTV services.

݁ݐܽݎݐ݅ܤ ݊݅ܽܩ = ଵ ଶ (2)
The "quality gain" factor is defined in equation 3. It indicates for a given bitrate limit the quality gain that can be achieved by a resolution reduction of 4. This is a scenario in which the 3DTV service provider offers a fixed access bandwidth to the subscriber.

ݕݐ݈݅ܽݑܳ ݊݅ܽܩ = ܻ1 -ܻ2 (3)
By applying these two factors to all temporal and spatial processed points for each SRCs we can get Table 4 for the bitrate gain comparison and Table 5 for quality gain comparison. Comparing the bitrate gain of spatial and temporal resolution reduction to the H.264/AVC encoding, the "resolution / 4" is clearly superior to others with a mean value of 67%, i.e. on average, "resolution / 4" process only uses 67% of the bandwidth which was needed for transmitting H.264 simulcast coded 3D videos at the same visual quality level. There is an exception for SRC10 where all spatial and temporal processes cost more bandwidth than the coding only cases. The "resolution/16" reduction only works efficient among the videos with originally full HD resolution (using only 60% of bandwidth in average). For the videos with a lower resolution, the transmission at the original resolution with H.264 is more efficient. However, when the comparison is extended to MVC, we see that "resolution/4" outperforms MVC among most of the full HD content videos. Previously we have already seen that MVC performs slightly better than H.264 in general. Table 4 also reveals the temporal reduction didn't save any bitrates at all, on the contrary it required 2.6 (for 12fps) and 4 times (8fps) more bandwidth compared to H.264 simulcast and MVC.

Table 5 shows a comparison of the quality gain based on the same service bandwidth. The results are similar results to those of the bitrate gain. The "resolution/4" wins in the quality gain with an average improvement of 0.24 MOS compared to the H.264/AVC encoding only case. For the videos with higher resolution the "resolution/16" can get a quality gain of 0.53 MOS. In the comparison to the MVC case, only the "resolution/4" shows a few small enhancements (0.10 MOS) for the higher resolution contents. As a conclusion, it may be stated that for an HD 3DTV transmission system, actually a reduction of the resolution by a factor of four before the video encoding will result in a better quality. It will not only help the service provider to save the bandwidth but also to save some amount of hardware processing which would be needed for encoding and decoding of two times a Full HD video.

Gender analysis

An Anova was performed with gender as one between factor and 11 SRCs times 15 HRCs as within factors. The proportion of women to men is about one to one in the subjective test (21 male and 20 female). This analysis shows that there was no main effect difference between the genders. However, the interaction between gender and HRC was significant, see Figure 8, F(14, 546) = 2.38, p = 0.00 < 0.05. The interaction with the SRC was not significant, see Figure 9. We can find from Figure 8 that female observers were voting slightly more positive, and the MOS spans a bit larger range compared to the male observers.

CONCLUSIONS AND FUTURE WORK

In our study we evaluated several video processing methods as well as the state of the art 3D coding standards. The results were based on a subjective experiment using ACR of S3D videos performed in two laboratories, one in France and one in Sweden. There was a difference between the laboratories that could be modeled well by a linear transformation. Even after transformation of one of the laboratories data with the linear transformation, a significant interaction between the laboratories and the SRCs remained. It was also discovered that there was a significant interaction between the genders and the HRC i.e. females voted more positively on high quality videos than males, but also decreased their quality ratings faster that males.

We presented a specific focus on bandwidth saving using the subjective experiment results. We discovered that a resolution reduction of four may result in higher bitrate efficiency when H.264 video coding is used. The result that this scenario is also preferable to MVC coding suggests that the reduction of the spatial resolution may be advantageous for MVC as well. It was seen that MVC performs better than H.264 simulcast, however this advantage decreases when the bitrate is higher. The reduction of the frame rate did not save bandwidth but it actually reduced the video quality to a large amount. It was noted that the 3D video content is an important and largely independent factor; it has strong effects on the video quality, the visual comfort and the bitrate. The results can be applied to an HD 3DTV transmission system, as the resolution reduction not only helps service provider to save bandwidth but also to save some amount of hardware processing which would be needed for encoding and decoding.

Figure 2 :

 2 Figure 2: Subjective experiment rating interface

Figure 4 :Figure 5 :

 45 Figure 4: The MOS across SRC of the different laboratories after scaling Lab 2's (Acreo) data to Lab 1 (IRCCyN). The error bars shows 95% confidence intervals

Figure 6 :

 6 Figure 6: Bitrates versus MOS of MVC and H.264 simulcast comparison

Figure 9 :Figure 8 :

 98 Figure 9: The mean MOS across SRC for the different genders. The error bars shows 95% confidence intervals

Table 1 :

 1 Source video sequences

	Sequences num.	resolution	frame rate	characterization
	src01	1920x1080p	25 Hz	Macro-Recording, time-lapse, surprising motion
	src02	1920x1080p	25 Hz	Car racing preparation, high detail, colorful
	src03	1920x1080p	25 Hz	Car race, high motion, large depth range
	src04	1920x1080p	25 Hz	Animation, human characters, rare colors
	src05	1920x1080p	24 Hz	Mesh grid rendering, high detail, small depth range
	src06	1920x1080p	24 Hz	Rendered transparent glass ball, circular motion
	src07	1280x720p	25 Hz	Group of parachutists, unsteady camera, flapping clothes
	src08	1440xs1080p	25 Hz	Market place with groups of people, skin colors
	src09	1024x576p	25 Hz	Night scene, fireworks, large depth effects, sudden motion

Table 2 :

 2 List of processing conditions (HRC)

	HRC Nr.	Codec	QP	Other Degradation
	1	none	-	ref. 3D
	2	Simulcast	26	/
	3	Simulcast	32	/
	4	Simulcast	38	/
	5	Simulcast	44	/
	6	MVC	26	/
	7	MVC	32	/
	8	MVC	38	/
	9	MVC	44	/
	10	Simulcast	26	fps/2
	11	Simulcast	26	fps/3
	12	Simulcast	26	Res/4 (width/2 * height/2)
	13	Simulcast	26	Res/16 (width/4 * height/4)
	14	Simulcast	-	ref. 2D
	15	Simulcast	38	2D

Table 2

 2 lists all HRC conditions. In order to cover the range of typical coding qualities, in HRC1-4 and 5-8 the quantization parameter (QP) was varied from 26 to 44 with a stepsize of six. Incrementing the QP by six, doubles the quantization stepsize of the linear quantizer for the Discrete Cosine Transform (DCT) coefficients in the H.264 encoder.

Table 3

 3 displays a French and Swedish version of the subjective experiment user interface which corresponds to the English version shown in Figure2.

Table 3 :

 3 French and Swedish version of subjective experiment user interface

		French	Swedish
	Title	Comment jugez-vous la qualité d'expérience dégagée par la séquence 3D?	Vad tycker du om kvaliteten i denna 3D presentation?
	part1	Qualité d'expérience [Excellente, Bonne, Assez Bonne, Médiocre, Mauvaise]	Upplevelsekvalitet [Utmärkt, Bra,Varken eller, Dålig,Usel]
		Confort visuel	Visuella bekvämligheten
	part2	[Beaucoup plus..., plus…., Aussi…., Moins…, Beaucoup moins…. ]	[Mycket mera…, Mera…, Lika…, Mindre…, Mycket mindre... ]
		….confortable à regarder que la télé 2D	…..visuellt bekvämt än att se på 2D TV
	Validate Valider	Validera

Table 4 :

 4 Comparison of spatial and temporal reduction performance to 3D coding concerning bitrate gain at same quality level Table4gives an overview of bitrate gain of all frame rate and resolution reductions. The cells marked with green shade indicate when the corresponding process is efficient and thus saves bandwidth compared to MVC or H.264 simulcast coding. For the other cells marked with red where MVC or H.264/AVC performs better without the preprocessing of spatial or temporal downsampling. Please note, that the resolution reduction cases were only performed with H.264 simulcast coding so the comparison to MVC coding may be misleading and higher gains may be achieved when the downsampled videos are later encoded with MVC instead of H.264/AVC.

Table 5 :

 5 Comparison of spatial and temporal reduction performance to 3D coding concerning quality gain with constant bitrate limit

† There were sometimes a small delay due to the video loading time

ACKNOWLEGEMENT

This work has been partly conducted within the scope of the JEDI (Just Explore Dimension) ITEA2 project which is supported by the French industry ministry through DGCIS. In Sweden, the work was financed by VINNOVA (The Swedish Governmental Agency for Innovation Systems). The participation of the observers is gratefully acknowledged. We would also like to thank Börje Andrén for his assistance with the subjective experiment at Acreo and Sylvain Tourancheau at Mid Sweden University for his advice.


Olivier Glassey
email: olivier.glassey@idheap.unil.ch

Jean-Henry Morin

Patrick Genoud

Giorgio Pauletto
email: giorgio.pauletto@etat.ge.ch

Design Thinking and Participation: Lessons Learned from Three Case Studies

Keywords: participation, design thinking, serious games, case study, perceived usefulness

This paper examines how design thinking and serious games approaches can be used to support participation through the analysis of three case studies. Indeed we will analyze these approaches in three different contexts: (i) a state-owned multi-utilities company; (ii) a political party; (iii) an information system strategic committee. Our analysis framework relies on the concepts of "perceived usefulness" and "perceived ease of use" and we will use it to discuss the lessons learned. Our main finding is that these approaches really contributing in making complex and abstract matters more "tangible" and thus understandable.

Introduction

In this paper we will examine how design thinking and serious games approaches can be used to support participation through the analysis of three case studies. Section 1.1 addresses the issues of wicked problems, with which most public policies deal, and how innovative techniques can help solving them. Section 1.2 then presents the Think Services approach, a combination of design thinking and serious games. In section 2 we will briefly present the contexts of our three cases studies: (i) a state-owned multi-utilities company; (ii) a political party; (iii) an information system strategic committee. In order to analyze these cases we will use a framework defined in section 3 and we will discuss the lessons learned in section 4.

Tackling Wicked Problems

According to [START_REF] Sanford | Characterizing eParticipation[END_REF] governments seek to encourage participation in order to improve the efficiency, acceptance, and legitimacy of political processes. They identify the main stakeholders of participation as citizens, non-governmental organizations, lobbyists and pressure groups, who want to influence the political system, as well as the opinion forming processes. Many political processes are concerned with solving wicked problems, defined by [START_REF] Newman | Gain a Foundation in Design Thinking to Apply Gartner's Hybrid Thinking Research[END_REF] as "those that defy conventional approaches to understanding, planning, design, implementation and execution because: (i) The stakeholder interests are so diverse and divisive; (ii) Interdependencies are so complex and so little understood; (iii) Behaviors are so dynamic and chaotic (unpredictable)". One approach to address wicked problem is proposed by [START_REF] Newman | Gain a Foundation in Design Thinking to Apply Gartner's Hybrid Thinking Research[END_REF]: hybrid thinking is centered amongst others on design thinking and co-creation.

Similarly the VoiceS research project [START_REF] Holzner | Beyond Web 2.0 -What Serious Games, Semantics and Social Networks can Contribute to eParticipation[END_REF] uses serious games to support eParticipation and make "complex EU co-decision procedure accessible to a large audience (especially among younger citizens), thus providing necessary understanding and enabling them to contribute actively to the platform". A game is defined by [START_REF] Holzner | Beyond Web 2.0 -What Serious Games, Semantics and Social Networks can Contribute to eParticipation[END_REF] "as a structured or semi-structured activity, usually undertaken for enjoyment and sometimes also used as an educational tool. Key components of games are goals, rules, challenge, and interactivity." [START_REF] Susi | Serious Games -An Overview[END_REF] explain that the term serious game "came into wide use with the emergence of the Serious Games Initiative in 2002 (seriousgames.org)". The Serious Games Initiative website states that they are "focused on uses for games in exploring management and leadership challenges facing the public sector". [START_REF] Susi | Serious Games -An Overview[END_REF] also believe that serious games "can be applied to a broad spectrum of application areas, e.g. military, government, educational, corporate, and healthcare."

The Think Services Approach

The approach used by Think Services is adapted from techniques originating from design thinking (for an introduction see [START_REF] Brown | Design Thinking for Social Innovation[END_REF][START_REF] Brown | Design thinking[END_REF]) and for more on design science [START_REF] Vaishnavi | Design Science Research Methods and Patterns: Innovating Information and Communication Technology[END_REF]); from the Innovation Games of Luke Hohmann [START_REF] Hohmann | Innovation Games: Creating Breakthrough Products Through Collaborative Play[END_REF]; and from participatory focus groups. One of the objectives of Think Services is rely on a positive lens. According to [START_REF] Avital | Introduction to designing information and organizations with a positive lens[END_REF], this concepts "refers to a perspective in the social sciences that emphasizes the capacity of people, teams, and organizations to construct enriching work environments, create more fulfilling customer experiences and design better socio-technical systems (…)". [START_REF] Avital | Introduction to designing information and organizations with a positive lens[END_REF] also explain that design encompasses three organizational practices: (i) actually producing artifacts; (ii) using them; and (iii) communicating about them.

In a world where management and governance processes are focused on decision-making, [START_REF] Avital | Introduction to designing information and organizations with a positive lens[END_REF] strongly advises that decision-makers engage in design thinking practices. Indeed it has been assumed since the theory of innovation of Schumpeter in the 1930's that "the most important designs for innovations would originate from producers", even if individuals or other firms also contribute to innovation [START_REF] Baldwin | Modeling a Paradigm Shift : From Producer Innovation to User and Open Collaborative Innovation Modeling a Paradigm Shift : From Producer Innovation to User and Open Collaborative Innovation 1 . Introduction and Overview[END_REF]. Here we allow ourselves to draw a parallel with policy-making and political decision-making: we have a feeling that it is also assumed that innovations come from those who are in charge. However information and communication technology and new models such as open source software and co-creation have been changing this paradigm and leading to what has been coined open innovation [START_REF] Baldwin | Modeling a Paradigm Shift : From Producer Innovation to User and Open Collaborative Innovation Modeling a Paradigm Shift : From Producer Innovation to User and Open Collaborative Innovation 1 . Introduction and Overview[END_REF]. This is why the Think Services approach aims at co-designing policies and services. It furthermore has the goal of going beyond technological applications and making the results of multidisciplinary approaches tangible. Indeed prototypes of service innovation should be tangible [START_REF] Brown | Design thinking[END_REF], however tangible does not (necessarily) mean physical. The Think Services approach relies heavily on tangible outputs such as pictures, videos, shared documents and working spaces, and contextual mapping and presentation tools such as Prezi [11]. However it also goes further in using the leverage of tangibilization: innovation games such as Design the Service Box actually do produce a cardboard box describing the designed service. We indeed believe that in order to tackle wicked problems, tangible prototypes should be used.

Several countries and governments are currently exploring a design thinking approach similar to the one fostered through Think Services, see e.g. [START_REF] Parker | The Journey to the Interface. How Public Service Design Can Connect Users to Reform[END_REF] or [START_REF] Thomas | Innovation by Design in Public Services[END_REF]. More generally Service Design is itself becoming a more and more developed and structured discipline that addresses a growing demand from the private and public sector see [START_REF] Stickdorn | This is Service Design Thinking. Basics -Tools -Cases[END_REF], [START_REF] Kimbell | Designing for Services -Multidisciplinary Perspectives: Proceedings from the Exploratory Project on Designing for Services in Science and Technology-based Enterprises[END_REF] and [START_REF] Moritz | Service Design. Practical Access to an Evolving Field[END_REF].

Workshops

To support our tangible service design approach we assembled a toolbox of different workshops. The goal is twofold: first, it allows the elicitation of needs, i.e. exploring the feasibility and the viability of the ideas, and second, it builds energy and momentum with the stakeholders taking part in the workshops.

The toolbox is an evolving set. It uses several practical and academic sources to populate and help finding the right approach for the problem at hand. We do reference the work of the original authors, if they are identified, and encourage the reuse of our own resource. As of this day, several workshops are identified, not all of them are completely developed or tested. Each of them is described in a synthetic document, explaining why it is useful, how to carry it out and what the expected outputs are. The main elements described formally are the objective, the brief, the content, the output and the follow-up of the workshop. To give a brief gist of the current content of the toolbox, this is the list of the titles of selected workshops: "Hunt the stereotypes", "Shape the future trends", "Remember the future", "Create the service box", "Build the business model", "Brainstorming", "Play the service".

Let us explain in briefly two of them that we will describe in more details later in our applied case studies. In "Remember the future", we ask the participants to imagine they live in a distant future. Several scenarios are developed and assigned to different groups (these scenarios are usually taken from the outputs of "Shape the future trends"). The groups are then asked to describe the service and its environment as they would have seen it from that point of view. This forces to observe the future from a more distant future, therefore allowing to literally "Remember the future". The service is then described as if it had already been implemented.

Why is it important? It often is difficult to imagine concretely what the future service should look like. Several studies in cognitive psychology show that by examining the future we lack a frame of reference and get easily lost in the possible paths opening up. By reversing the point of view, the description of tangible elements is clearer, richer and more concrete. It is also easier to describe what steps were taken to reach the desired service. By selecting several scenarios, we allow to test the robustness of the service. This usually uncovers the similarities of the service in radically different settings. On the other hand, by looking at the contrasts, it also underlines peculiarities that might be essential success factors in given situations.

In "Create the service box", the participants are invited to physically design a box that virtually contains the desired imagined service in order to communicate its characteristics. This not only allows the expression of the tangible benefits and perception of the imagined service, but also lets the groups share more clearly the ideas they have about the service. These ideas will otherwise most of the time remain vague, intangible and difficult to get across. In this workshop, a service is already roughly imagined (this might be the output of a "Brainstorming" workshop combined with a "Remember the future" workshop). The process of "Create the service box" is simple. Give the participants a cardboard box, drawing material, magazines where to cut up pictures, and ask them to literally design the box to sell the service. After building the physical box, we invite them to present the result to the group and strongly encourage a narrative storytelling form. This structures the output, engages the audience and makes it easier to depict a concrete use. The short presentation of the groups is followed by a general discussion and a synthesis.

Why is the workshop useful? It mainly allows describing very vividly the service. By building a physical artifact, people usually come to a point where they stop talking and start doing. In several cases, choosing images, drawing on the box, or developing a story allow to more clearly exchange the ideas than only relying on words, without limiting them in their imagination.

Let us now explain more precisely the experiments we conducted and extract some lessons learned.

In this section we describe the design workshops and the three organisations in which they were organised: one state-owned multi-utilities company; one political party; the information system strategic committee of the Canton of Geneva. We will also detail the objectives of these workshops.

Services Industriels de Genève

The chief information officer (CIO) of the Services Industriels de Genève (SIG, the stateowned, multi-utilities company of Geneva) is in charge of a team of 10 managers responsible for the different information technology (IT) services.

The creative workshop was organized with this team about the future of the IT in the organization [START_REF] Genoud | Atelier "Remember the future[END_REF]. It was based on the Remember the future approach [START_REF] Morin | Remember The Future[END_REF]. The main goal of the CIO was to make his colleagues stand back about the evolution of their job and of their professional environment over the next 10 to 15 years.

During the first part of the workshop the participants were asked to imagine their workplace in the year 2040 and draw it onto a board. Rather than claiming to correctly describe the reality of 2040, this initial step allowed them to immerse in this far future in a very natural way, making their thoughts tangible through the drawings. It constituted a very good starting point for the rest of the workshop with people envisioning colleagues collaborating from the other side of the planet, convergence of private and professional environments, teleworking using rich interfaces, holographic avatars, ubiquitous sensors, or pervasive virtualization.

Once "projected" in 2040, everybody had to imagine himself as the CIO of SIG celebrating his/her retirement. At this occasion the CIO is asked to tell the story of the building of his new team 20 years ago (in 2020) and outline why and how this team contributed to the success of the organization in the following years. Two groups were formed and asked to elaborate their scenario in different contexts. One group was supposed to imagine the evolution of the IT team in a context of commoditization: the IT and the information systems were in this case a support to the evolution of SIG. The other group worked on a contrasted scenario where the information systems were supposed to be part of the core business of SIG.

To support their thoughts each group was suggested to elaborate on IT missions of the SIG in the proposed context, on the evolution of the competencies needed to achieve these missions and on the services proposed by their teams.

Even if constructed in two very different contexts, the story proposed by each groups conveyed some common preoccupations. First, data and information was envisioned in each context as a strategic resource for the organization. The workshop helped the team to explicit the value of the information as a primary source of knowledge. Many services were imagined based upon data, information and knowledge, with new activities and competencies (such as "service trader" or "semantician") needed to deal with that.

The CIO of SIG had no peculiar expectations with respect to the outcome of the workshop. But the ideas and reflections generated during the workshop evoked different useful perspectives regarding the evolution of his organization. He mentioned as an intangible result the fact that his team stepped back with respect to their daily activities and a reinforcement of the team spirit.

Parti Démocrate Chrétien

The Christian Democratic People's Party (Parti Démocrate-Chrétien -PDC) is a center-right wing Swiss political party. Its Geneva section invited ThinkServices to run a creative workshop for the Economic Commission on the theme of Jobs and Skills of the Future. Such a prospective issue naturally triggered their interest for co-creativity, design thinking, serious gaming thus leading to their request to experience such methods at first hand.

The roughly 2 hours Skill Box workshop was a mix of two techniques: Persona creation and Service Box. First, the participants were asked to describe the profile (persona) of the future worker. In order to be as open as possible, they were given the following categories of work to address in groups: nursing & health care provider, teacher, bank employee, librarian, blue-collar worker or one of their personal choosing. They were also given a set of thinking cues in terms of: education and training (initial and lifelong), work load, places, schedules, wages, retirement, tools, recruiting, personal vs. professional life balancing, etc.

Following the persona definition the participants were asked to design the "skill box" (i.e. service box corresponding to that profile) as an actual artifact they could display in a job fair.

Finally, the groups were asked to present two things: first, the persona they imagined followed by the box they co-designed as if they had been in a "skill store". We deliberately looked at the result from the standpoint of observers. This allowed us to witness the following thought process. The participants clearly had in mind the societal drivers and requirements in terms of preparedness level of a political party. Therefore one could easily imagine that their next step could be to conduct a SWOT analysis of some of the salient aspects revealed. Among these aspects three clearly emerged from the stories. First and foremost was the importance of technology as enabler in the areas of learning, organization and planning of work. The second aspect touched upon the growing changes in social structures and the corresponding work environment thereof. Of particular attention were the increase of working women and the redistribution of traditional family structure. Finally, the fundamental role of the networked life (i.e., social networking, virtual teams, lifelong learning, etc.) in all aspects of both private and professional lives especially considering their blurring boundaries.

Collège Spécialisé des Systèmes d'Information

The Collège Spécialisé des Systèmes d'Information (CSSI) of the State of Geneva is constituted of 12 managers in charge of information systems of each of the State of Geneva departments. During the year 2009, the members of the CSSI wrote their strategic plan for the years 2009 to 2013 [START_REF][END_REF]. This document describes the main axes considered to give coherence to the evolution of the different information systems of the State of Geneva. It also presents the key success factors and the strategy adopted to achieve this.

The writing of this strategic plan by a working group of the CSSI allowed to quickly finalise the output. A regular sharing of the reflections with all the members of the committee ensured that everybody related to the content of the document. But the short deadline was not necessarily favourable to an effective ownership of the final document, at least when considering the values it was meant to convey. Yet this phase of ownership is essential if one expects a coherent and effective communication and implementation of the strategy. So although the document has been drafted in a rather open and co-creative way, the question remained whether every member of the CSSI had really taken the ownership and appropriated the real sense of the proposed strategy. To test this and/or to make the appropriation process easier, we organized a workshop [START_REF] Genoud | Atelier d'appropriation du plan stratégique des systèmes d'information[END_REF] bringing them together around the Create the Service Box [START_REF] Pauletto | Create the Service Box[END_REF] creative method.

The workshop was conducted during two hours. The participants were divided into 3 groups. In a co-creation process, each group had 45 minutes to materialize on each of the 6 faces of the box the expected benefits of the service virtually dropped off in the box: namely the strategic plan of the information systems of the State of Geneva. With managers used to mainly talk about "costs" and "quality", we emphasized the importance to also consider other values that their strategic plan can convey, namely its "value of use" and "perception".

The box allows the participants to easily pass from concepts to something more concrete, which helps them to start up again, enriching it with new ideas made tangible through pictures, drawings, carvings, mobiles or any other media supporting their imagination. The six faces of the box also make up a constraint that forces participants to get to the main points and prioritize their messages.

During the creative phase, each group tries to make the key aspects of the strategic plan tangible on the cardboard box by enriching their creative contributions. Participants exchange their standpoint and confront their visions, by using drawings, cut-out or glue.

Once the box is complete, it serves to support the narrative, the storytelling. The group tells its story and "sells" the strategic plan to different possible stakeholders (deputies, business managers, executives) with the help of the box and the symbolism it conveys. This co-construction improves the understanding and the ownership of a service that otherwise remains purely intangible. It also helps to focus on the elements of value rather than only on the expected features.

The exercise was originally intended to facilitate the ownership of the strategic plan of the CSSI by its members. But he eventually demonstrated that this appropriation was already effective: even if the narratives offered by the three groups were different, all agreed on the consistency of the messages heard. The exercise had the merit of revealing that the strategic plan has been understood and internalized by members of the committee. This finding was in itself a remarkable result of the workshop.

All participants recognized the richness of this creative method which helped passing from abstract concepts and common values to various concrete messages. Participants appreciated the fact that this method revealed their strategic plan in the form of various metaphors which, despite their lack of precision, certainly facilitate its communication to different audiences. The Create the Service Box method is then an excellent tool to master the complexity and helps to convey simpler and more accessible messages.

Finally everyone recognized the team building dimension of such a co-creative workshop led in a playful spirit. Participants are asked to materialize the expected benefits of the strategic plan of the information systems of the State of Geneva on each of the 6 faces of a box Outputs Different but consistent narratives of the strategy; effectiveness of the symbolism of the box to convey abstract and complex messages; team-building.

Analysis Framework

Our analysis framework was built ex post in order to analyze the Think Services approach. As the workshops had been documented through blog posts, pictures, video recordings, etc. it was possible to apply it in a rather coherent way. However we have to mention that no research design was developed before the series of workshops began, and that our analysis has its limitations. Still we believe that interesting lessons can be learned from these selected case studies.

In order to discuss how workshop participants perceived the Think Services approach we focus on the creation of value they support, as well as participants' satisfaction. [START_REF] Ho | Exploring Service Science in our current socio-economic environment[END_REF] states that the value creation of a service is most of the time only looked at in terms of cost and quality, possibly of utility, but rarely in terms of perception. This is precisely what we want to look into, rather than investigating the quality or the utility of a tangible workshop's output such as a service box. We will not do a literature review here but we will use [START_REF] Wixon | A Theoretical Integration of User Satisfaction and Technology Acceptance[END_REF] as a starting point. They propose "an integrated research model that distinguishes beliefs and attitudes about the system from beliefs and attitudes about using the system". To survey the behaviour of using technology [START_REF] Wixon | A Theoretical Integration of User Satisfaction and Technology Acceptance[END_REF] rely on the well-known technology acceptance model (TAM) developed by [START_REF] Davis | Perceived usefulness, perceived ease of use, and user acceptance of information technology[END_REF]. This theory suggests that users confronted to a new technology are influenced in their use by the:

 Perceived usefulness: Davis defines it as "the degree to which a person believes that using a particular system would enhance his or her job performance";  Perceived ease-of-use: Davis describes this as "the degree to which a person believes that using a particular system would be free from effort".

Although this theory was developed for technological systems we believe it can be applied to serious games and design thinking workshop as well. Other dimensions such as perceived risks, costs, immediacy, or fun, are not formally integrated in our assessment but will be discussed in an ad-hoc manner in next section.

Lessons Learned

We used the above framework as a guideline for assessing the result of this work in terms of the impact of design thinking and serious gaming as significant factors augmenting participation. These findings, although not formally surveyed are however the result of informal debriefing sessions that took place systematically after each workshop. These sessions involved gathering feedback form the participants through discussions; later discussions also took place among the workshop organizers. It is for this reason this paper is presented as "lessons learned" rather than actual formal findings. Future work to validate our hypothesis will require the use of formal quantitative and / or qualitative survey techniques.

Perceived usefulness. Based on informal discussions and debriefing sessions following all workshops, it is clear that the vast majority of workshop participants explicitly stated that both the actual outcome of the workshop and the methods used would significantly contribute to enhancing their performance in their work. Some workshops have actually led to follow up workshops or concrete actions based on the outcome.

Perceived ease-of-use. Again, all feedbacks converged acknowledging the high degree of reusability of the methods and techniques. This is further strengthened by the fact that all our workshops are documented in open workshop briefs available for anyone to take, use and enhance. The briefs describe in detail everything that is needed to run the workshops. As an example, the Create the Service Box brief can be found here [START_REF] Pauletto | Create the Service Box[END_REF]. One participant mentioned after a workshop that he would use these techniques in work related problem-solving and mediation meetings. This illustrates the high degree of ease of use of the approach.

Cognitive shielding and safe house phenomenon. Interestingly we have found from our participants that as an external not for profit Think Tank we carry a tremendous potential as a "safe place" for experiencing, testing, trying new ideas that would have never surfaced otherwise or would have been difficult to argue in their respective professional environments. We refer to this phenomenon as "cognitive shielding" as it basically offers a protected environment for expressing such ideas and positions with much less risk.

Power of tangibilizing the intangible. All our workshops have a strong focus on tangibilization. Whether a Service Box, Stories, etc., they all rely on the production of an artifact. The key point here is that the artifact serves as the focus point allowing a much needed intermediation between people. The attention is therefore transferred on this artifact rather than between people. Moreover, it significantly contributes to reducing the complexity of the issues being discussed. Such tangible artifacts become easier to deal with than concepts. This has also proven to be a major value of the approach. The serious gaming approach has a very strong impact on engagement. Participants feel compelled to play in teams thus contributing to team-building addressing together a common challenge. The story telling aspect also contributes to tangibilization through its metaphorical side and the fact that people become part of their stories. They therefore collectively endorse the issues they work on, are able to take some distance and find there is value in the process. This was particularly true for public administrations. The key lesson is definitely the power of tangibilization.

In very rare occasions we have witnessed negative reactions to the approach. The only one we are able to report on was a case where a participant following the workshop commented with a statement: "ok, that was interesting but now we need to get back to work". Among some of the other aspects let us briefly mention the following. The perceived risks appears to be relatively low as participants are immediately put in a "safety" position through a brief introduction stressing the value of creativity, trial and error, collaboration and having fun. In terms of costs, the highest cost incurred is most likely the actual time spent for the workshop, and therefore not working as usual. From the point of view of immediacy, there is no doubt participants have instant takeaways to inspire their work. Finally having fun along the way is a key building block and success factor of the approach leveraging the creative capabilities we have.

Finally, we also learned a few useful things we share as tips and noteworthy moments to lookout for when running the workshops. Success of such techniques is greatly improved when participants are put in the proper mindset. This requires setting the context through well-known techniques like using a short video, playing a game, etc. For example, we have used the "Did you know?" video [25], a simple "yes -no" exercise among two people. Person A states ideas and person B systematically responds No arguing why it's a bad idea. Then they switch roles and person A systematically answers positively enhancing the idea of person B. Another technique is the sound ball where people exchange a virtual ball making a sound. Each person then repeats the set of sounds adding his own at the end.

Another interesting and noteworthy event is when during the workshop you see all the participants in a group get up. We have systematically witnessed this event as being the tipping point moment of success for a group. Often this is unconsciously contagious as all groups end up standing working together.

A last key element is trying to meet the deadline: indeed a good timing of the workshop is a key success factor, as well as having a dedicated timekeeper has proven to be valuable. Using observers from the organizing team can also be useful to capture noteworthy events and / or help answering questions when needed.

Conclusion and Future Work

Our experience running co-creation workshops based on design thinking and gaming in intellectually protected settings such as an independent, not for profit Think Tank has led us to witness the value of the approach for participants. This has in turn led us to discuss and consider the role played by design thinking and gaming in stimulating participation. Based on three cases we discussed the lessons learned in terms of perception of usefulness and ease of use. Stimulating and augmenting participation is a source of empowerment for people when addressing and trying to solve wicked problems. Future work involves assessing our hypothesis through formal methodologies. In doing so, we plan to study and define the criteria allowing to not only assess the workshops but also serve as best practice supporting engagement and participation based on design thinking, serious gaming, story telling and

Table 1 .

 1 Workshop "Services Industriels de Genève".

	Toolbox	Remember The Future
	Objectives	Stand back about the evolution of the work
		of the IT team over the next 10 to 15 years
	Brief	Participants are asked to imagine their
		workplace in the year 2040 and the profile
		of workers
	Outputs	New strategic data and information
		services; new activities and competencies,
		such as "service trader" or "semantician"

Table 2 .

 2 Workshop "Parti Démocrate Chrétien".

	Toolbox	Skill Box
	Objectives	Imagine jobs and skills of the future
	Brief	Participants are asked sketch the persona of
		a future worker and to design the skill box
		corresponding to that profile as a job
		announcement to be shelved in a JobStore.
	Outputs	Technology as enabler for learning,
		organizing and planning of work; growing
		changes in social structures; fundamental
		role of networked life.

Table 3 .

 3 Workshop " Collège Spécialisé des Systèmes d'Information".

	Toolbox	Create the Service Box
	Objectives	Appropriation of a strategic plan by the
		members of the IT strategy board
	Brief

tangibilization. We nevertheless think this contribution already serves as background work for further studies in this area as well as report on field experience that may be helpful to practitioners in services innovation and design.

Finally another aspect we have not addressed in this paper is the organization and structure of the different workshops as a multi-entry point process. These workshops were very briefly mentioned in section 1.3. We plan to further study this issue as we continue to develop and run the workshops.