Axel Carlier
email: axel.carlier@enseeiht.fr

Amaia Salvador
email: amaia.salvador@upc.edu

Ferran Cabezas
email: ferran.cabezas@upc.edu

Xavier Giro

I Nieto

Vincent Charvillat
email: vincent.charvillat@enseeiht.fr

Oge Marques
email: omarques@fau.edu

Xavier Giro-I-Nieto
email: giro-i-nietoxavier.giro@upc.edu

Assessment of crowdsourcing and gamification loss in user-assisted object segmentation

Keywords: GWAP, Crowdsourcing, Serious games, Object detection, Object segmentation 1

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Fig. 1 Introducing the human "in the loop" in object segmentation are even guided by these algorithms to produce clicks in the most meaningful regions. The main difference between crowdsourcing and GWAP, of course, is that while the crowdsourcing approach is potentially tedious and depends on the attention to detail, patience, and understanding of the task by the workers, the game-based approach is potentially more relaxed and fun and does not feel like a chore.

In this paper we are particularly interested in studying how much segmentation quality is lost when an interactive segmentation task is performed by either modestly paid workers with no expert knowledge (in the crowdsourcing case) or using data points from Ask'nSeek game logs. We use the data collected from a group of "expert users" (using Click'n'Cut) as a baseline for comparison. Throughout the paper we shall refer to the former type of loss as the crowdsourcing loss -the type of loss incurred when an interactive segmentation task is assigned to paid workers who are not segmentation experts, using the same tool -whereas the latter type will be called the gamification loss, which refers to how much quality of segmentation results may be lost when we switch to a game-based approach to the same task.

The rest of the paper is organized as follows: Section 2 discusses related work in the fields of interactive object segmentation, crowd-based object annotation, and games with a purpose (GWAP). Section 3 describes Click'n'Cut and Ask'nSeek, the two online tools designed to collect human clicks for the object segmentation task. Section 4 describes the experiments performed using Ask'nSeek and Click'n'Cut and discusses the most relevant results. Finally, Section 5 suggests directions for future work and Section 6 presents the final conclusions.

Related work

Interactive object segmentation The segmentation of objects by combining human interaction and image processing algorithms has been extensively explored in the literature. In such interactive setup, the graphical user interface responds to some sort of weak annotation (bounding box, scribbles, clicks...) from the user by generating and displaying a complete segmentation of the object. The typical workflow expects the user to interact with the proposed solution either by accepting it or by providing more traces that may allow the segmentation algorithm to converge to a satisfactory result. Most interactive segmentation techniques normally propagate the user-generated indications of which pixels belong to the foreground or background through a graph-based representation of the image. Among these graph-based solutions, two main families of algorithms can be identified depending on whether the nodes of the graph correspond to a pixel or to a region (superpixel).

The foundational proposal for interactive foreground segmentation was based on graph cuts [START_REF] Boykov | Interactive graph cuts for optimal boundary map; region segmentation of objects in n-d images[END_REF]. The algorithm considers every pixel as a node in a graph, connected to their spatial neighbors by an edge whose weight depends on the visual similarity between pixels. In addition, every pixel node is also connected to two special terminal nodes, each of them representing an foreground or background label, with a weight associated to the similarity of each pixel to a model of the foreground or background. Segmenting an object is equivalent to finding the min cut of the graph, that is, those edges that once disconnected minimize a certain energy function defined on the two resulting sub-graphs. The graph cuts approach has also been expanded by other authors [START_REF] Rother | grabcut": interactive foreground extraction using iterated graph cuts[END_REF], and its annotation mode mode (bounding boxes, sloppy contours or tight polygon) predicted for each image on [START_REF] Jain | Predicting sufficient annotation strength for interactive foreground segmentation[END_REF].

Superpixel-based solutions [START_REF] Lee | Interactive segmentation as supervised classification with superpixels[END_REF]22,[START_REF] Noma | Interactive image segmentation by matching attributed relational graphs[END_REF][START_REF] Wang | Interactive video cutout[END_REF] avoid the computational load of a pixel-bypixel segmentation by working with unsupervised image segmentations performed offline. These solutions rely strongly on the region boundaries defined by the segmentation algorithm, which are assumed to correspond to the semantically meaningful regions in the image. These image partitions are usually configured to generate over-segmentations from the semantic point of view, which include all object contours as well as many additional ones, which are to be removed through the interaction process. The process of mapping user interaction to regions in the partitions, so that the labeled pixels are assigned to their corresponding region, is straightforward.

The adjacency information between regions coded in a partition can be further enriched by iteratively merging pairs of neighboring regions. As a result, hierarchical partitions contain additional information capable of capturing the multiscale semantic nature of an image. Interactive segmentation solutions based on hierarchies [START_REF] Adamek | Using contour information and segmentation for object registration, modeling and retrieval[END_REF][START_REF] Arbelaez | Constrained image segmentation from hierarchical boundaries[END_REF][START_REF] Giró-I Nieto | From global image annotation to interactive object segmentation[END_REF][START_REF] Salembier | Binary partition tree as an efficient representation for image processing, segmentation, and information retrieval[END_REF] use these data structures to propagate labels not through a flat graph partition but, instead, considering regions at multiple scales. The comparative study in [23] indicated similar accuracy labels for GrabCut [START_REF] Rother | grabcut": interactive foreground extraction using iterated graph cuts[END_REF] and hierarchical solutions [START_REF] Adamek | Using contour information and segmentation for object registration, modeling and retrieval[END_REF][START_REF] Salembier | Binary partition tree as an efficient representation for image processing, segmentation, and information retrieval[END_REF], but a faster response for the latter ones.

More recent solutions have used convolutional neural networks to extract object segments from minimal user interaction. A single user click is used in [START_REF] Pinheiro | From image-level to pixel-level labeling with convolutional networks[END_REF][START_REF] Russakovsky | What's the point: Semantic segmentation with point supervision[END_REF] to condition the loss function of a network which had been previously trained for the semantic segmentation of an image [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF]. These work though does not explore though an interactive framework, as only one click is collected by user and no visual feedback is provided to augment the informativeness of the click.

The solution adopted in our work generates segmentations by combining a precomputed set of object candidates (also referred to as "saliency detectors"). Instead of considering precomputed regions which are normally generated considering only perceptual criteria, our basic processing unit are regions that have been defined by an algorithm that estimates the "objectness" of an automatically generated segment, i.e. how likely a segment is to correspond to a semantically meaningful object. Object candidate techniques [START_REF] Arbeláez | icoseg: Interactive co-segmentation with intelligent scribble guidance[END_REF][START_REF] Carreira | Constrained parametric min-cuts for automatic object segmentation[END_REF][START_REF] Liu | Learning to detect a salient object[END_REF] generate a ranked list of object candidates for the image, based on its visual features together with additional parameters learned from a collection of semantically meaningful regions. The presented approach is an extension of [START_REF] Salvador | Crowdsourced object segmentation with a game[END_REF], where crowdsourced clicks labelled as foreground or background were mapped into a collection of object candidates to select the region which better matched the captured traces. However, our system is more flexible than [START_REF] Salvador | Crowdsourced object segmentation with a game[END_REF], because it obtains solutions that can combine multiple candidates. Our Click'n'Cut system has been introduced in [START_REF] Carlier | Click'n'cut: Crowdsourced interactive segmentation with object candidates[END_REF] and uses this combination of object candidates.

Crowd-based object annotation

There have been several approaches to elicit object annotation data from the crowds in the literature. Some authors [START_REF] Salvador | Crowdsourced object segmentation with a game[END_REF][START_REF] Steggink | Adding semantics to image-region annotations with the name-it-game[END_REF] have designed games so that users would unawarely generate segmentation traces. This way, the task would not become so tedious and the gaming incentive might eliminate or reduce the financial one. Most initiatives for object segmentation have adopted a collaborative approach where users are instructed on how to generate high quality segmentations. Solutions in this family normally vary depending on the incentive, which can go from an abstract call to help science, to a very accurate pricing policy. Click'n'Cut relies on workers that were explicitly paid to generate an accurate segmentation of the objects.

One of the most popular initiatives in this direction is LabelMe [START_REF] Russell | Labelme: A database and web-based tool for image annotation[END_REF], on online platform that has collected a large amount of local annotations by asking volunteers to draw a polygon around the object.

A very ambitious initiative is related to the Microsoft COCO (Common Objects in COntext) dataset [17]. This project uses workers from Amazon's Mechanical Turk to segment the objects in images at an estimated rate of 79 seconds per object instance. Only one worker segments each object, but this worker must first pass a training stage before being qualified to segment. This segmentation effort uses the OpenSurfaces interface [START_REF] Bell | Opensurfaces: A richly annotated catalog of surface appearance[END_REF], an open source tool based on polygonal segmentation.

The authors of [11] compare the segmentation results achieved from crowdsourced workers who draw polygons around cars with the results obtained by applying a computer vision algorithm in the bounding boxes provided as ground truth. Ground truth segmentations were generated by 9 in-lab annotators using a similar interface as workers, taking on average 60 seconds to label each car. The crowdsourced task was organized in two batches: a first one paid 1 cent per annotated car and a minimum 75 % approval rate, and a second one paid 5 cents per car and 95 % approval rate required. Results showed a small (1 %) increase in the final quality of the segmentations for the higher priced case.

The crowd was also used in [START_REF] Jain | Predicting sufficient annotation strength for interactive foreground segmentation[END_REF] to assess an interface aimed at choosing the best input modality among a bounding box, a sloppy contour or a tight polygon. The authors highlight that in crowdsourced campaigns the annotation time is the basic budget constraint, and that by automatically adapting the annotation mode to the image it is possible to optimize the quantity and quality of the segmentations. The selection is based on an estimation of the average time necessary for each modality: 7 sec (seconds) for bounding boxes, 20 sec for sloppy contours and 54 sec for tight polygons. Their study was performed on 101 workers and a dataset of 420 images, collecting a minimum of 5 responses for each modality per image.

In our work we have tried to adjust as much as possible to the experiment described in [23] to be able to compare the quality of a crowdsourced solution with respect to an campaign with expert annotators.

Finally on the related topic of object co-segmentation which consists in segmenting the same object in multiple images that feature this object, it is worth describing iCoseg [4]. In this paper, the authors allow users to draw scribbles on images to annotate background and foreground. The scribbles on one image are used to co-segment all images that show the same object. In addition, the authors use an active learning formulation that allows the system to automatically detect the areas that would lead to the most informative scribbles, and propose it to the users. The same active learning paradigm is followed in [START_REF] Rupprecht | Image segmentation in twenty questions[END_REF]. In this paper, the authors introduce a system relying on oversegmentations that requires clicks from users in order to obtain an image segmentation. Active learning has also been used in [START_REF] Fathi | Combining self training and active learning for video segmentation[END_REF] for video segmentation. Our Click'n'Cut interface also displays feedback from the system to the users, to guide their interaction into the most relevant part of the image.

Games with a purpose (GWAP) Boredom will limit the duration of the annotation sessions that users will be willing to accept. Collaborative campaigns tend to produce highquality segmentations, but may result in tedious and boring tasks for the user. This limitation has been addressed in other works by designing Games With a Purpose (GWAP) capable of capturing valuable traces for object segmentations. In these cases, users (players) may be unaware that their feedback can be used for such purpose and still provide high quality traces.

There exist two fundamental differences between explicit and implicit collection of human traces. Firstly, in the work we just described where users directly interact with the segmentation resulting from their traces, their aim is explicitly the generation of high quality masks and the instant feedback guides them to generate the most informative traces. On the other hand, in a game-based scenario the goal of the user is to win the game which, in the case of Ask'nSeek (the GWAP that we propose), is completely unrelated to the quality of the segmentation coming out of the user traces. Secondly, interactive segmentation interfaces that collect foreground and background traces follow a coherent temporal sequence that try to correct the result of the last mask estimation. In our game-based approach, user interactions from different games are combined independently from the moment of their acquisition.

A popular strategy for obtaining crowd-sourced annotations is through on-line GWAPs, which exploit the high motivational levels achieved by games in such a way that the user interaction produces some type of valuable outcome. The Extra Sensory Perception (ESP) game [START_REF] Von Ahn | Labeling images with a computer game[END_REF] collects textual labels at the global scale by showing the same image to a pair of players. Players are prompted to enter words related to the shown image and, when an agreement is obtained between different players, they are rewarded with points. The label is accepted as correct when different pairs of users agree about it.

The first game used for object detection at a local scale was Peekaboom [START_REF] Von Ahn | Peekaboom: a game for locating objects in images[END_REF]. This platform is the natural evolution of the popular ESP Game from the same authors [START_REF] Von Ahn | Labeling images with a computer game[END_REF], which generated pairs of images and labels at a global scale. Peekaboom is played in pairs, where one player reveals parts of an image so that the other can guess the textual label representing the object that is being discovered. The areas to uncover are indicated with clicks, which are supposed to be placed on the objects.

The Name-It-Game [START_REF] Steggink | Adding semantics to image-region annotations with the name-it-game[END_REF] is played in pairs and collects both images and textual labels for the segmented objects. In that game, objects are outlined by a revealer player and their label must be predicted by a second guesser player upon a gradual appearance of the selected object. This interface combines freehand and polygonal segmentations, similar to LabelMe. The authors claim that by combining multiple traces obtained from games played using the same image, results are similar to the ones obtained by the LabelMe annotation campaign [START_REF] Russell | Labelme: A database and web-based tool for image annotation[END_REF]. Our experiments have also fused traces from different users on the same image to clean out noise, but the task of our workers is not freehand, but assisted by a computer vision algorithm instead.

The two-role approach is simplified in RecognizePicture [START_REF] Lux | Finding image regions with human computation and games with a purpose[END_REF], where the gradual revealing of the image is automatically chosen following different patterns. Players must choose between four possible labels describing one of the semantics contained in the image. Such approach requires a previous stage where an annotation at a global scale must be previously available to make sure that at least one of the four possible labels is indeed present in the image. Ask'nSeek [START_REF] Salvador | Crowdsourced object segmentation with a game[END_REF] also involved the participation of two players to collect the textual labels of the objects contained in an image, as well as selecting the best object candidate based on on clicks labeled as above, below, on the left, on the right or on the objects.

User interfaces

Human computation requires the capture of user interaction; taken altogether, these human contributions act as a computer that assists in solving a problem. This section describes the two online tools used in this study to solve the object segmentation problem. Intentional interaction has been collected with Click'n'Cut [START_REF] Carlier | Click'n'cut: Crowdsourced interactive segmentation with object candidates[END_REF], an interactive object segmentation tool described in Section 3.1. On the other hand, unintentional interaction has been captured by the Ask'n'Seek game [START_REF] Carlier | Ask'nseek: A new game for object detection and labeling[END_REF], described in Section 3.2.

Click'n'Cut interactive segmentation tool

In Click'n'Cut [START_REF] Carlier | Click'n'cut: Crowdsourced interactive segmentation with object candidates[END_REF], users are asked to produce foreground and background clicks to perform a segmentation of the object that is indicated in a provided description. The fundamental interactions available to the user are the left and right clicks, which generate foreground and background points, respectively. Figure 2 shows the interface, which displays the image that we wish to segment, along with a set of basic interactions (on the bottom-right of the screen) and a reminder of how the interface works (on the top-right part of the screen). There is also a description of the object to segment on the top of the screen, right above the image. Every time a user generates a click, the segmentation result is updated and displayed over the image with an alpha value of 0.5 (which can be changed by the user using a Transparency slider). This segmentation is computed using an algorithm based on object candidates, introduced in [START_REF] Arbeláez | icoseg: Interactive co-segmentation with intelligent scribble guidance[END_REF], and aims at guiding the user to provide information (i.e., meaningful clicks) that will actually help improving the quality of the final segmentation result.

Users can also correct a wrong click by just clicking on it again to make it disappear. The Clear points button removes the entire set of clicks that have been made by the user. Finally, once satisfied with the result, the user can move on to the next task by clicking the Done button.

The Ask'nSeek game

Ask'nSeek [START_REF] Carlier | Ask'nseek: A new game for object detection and labeling[END_REF] is a two-player, web-based, game that can be played on a contemporary browser without any need for plug-ins. One player, the master (Fig. 3, bottom) hides a target region somewhere within a randomly chosen image. The second player (seeker) (Fig. 3, top) tries to guess the location of the hidden region through a series of successive guesses, expressed by clicking at some point in the image. What makes the game more interesting is that, rather than just blindly clicking around, the seeker must ask the master for clues relative to some meaningful object within the image before each and every click. Once the master Fig. 4 Example of the game from the seeker's point of view receives a request for a clue from the seeker containing a label, it is required to provide a spatial relation, which is selected from a list: {above, below, to the right of, to the left of, on, partially on, none of the above}. These indications -in the form of (spatial relation, label), e.g., "on the church" -accumulate throughout the game and are expected to be jointly taken into account by the seeker during game play. Based on the previously selected points and the indications provided by the master, the seeker can refine their next guesses and -hopefully -guess the hidden region after relatively few attempts. Figure 4 illustrates a typical gameplay with Ask'nSeek. In the game featured on this example, the seeker first asked the master an indication relative to the red car, and the master answered that the region is "on the left of the red car". The seeker clicked on the image but not on the region, so he asked for a second clue, relative to the white car. The master answered that the region is "above the white car", and the seeker once again did not find the region. He finally got the indication that the region is "above the road", and clicked on the right location. Once he clicks inside the region, the actual location of the region chosen by the master is prompted to the seeker (before finding it, he could not see it). In other words, the target only appears on the image after the seeker has managed to click inside it.

The game is played cooperatively, which provides a way to encourage both players to locate the hidden region as quickly, and with as few clicks, as possible. To control for time, a timer (set to 2 minutes) is provided, which requires that the hidden object be found within this interval in order to win. Moreover, the score of both players decreases after each new click, which encourages the players to be precise in their request for clues (and associated responses) to minimize the number of clicks (and, indirectly, shorten the duration of the game).

Traces from Ask'nSeek can be processed in order to categorize a set of clicks labelled as foreground or background, as in the case of Click'n'Cut. For example, a click "on the building" is foreground relatively to the object building but background relatively to the object sky. In addition, a click "on the right of the building" is background relatively to the object building.

However, the motivation for the user is different in the Ask'nSeek case. Playing the game is the main goal, instead of obtaining an accurate segmentation. In fact, players are completely unaware that their interactions can be exploited to solve an object segmentation problem.

Experiments and results

In this section we discuss the results of a series of experiments using the interfaces described in Section 3. From the data collected during these experiments, we examine and quantify the crowdsourcing loss incurred when an interactive segmentation task is assigned to paid workers recruited on a crowdsourcing platform, comparing their results to the ones obtained when computer vision experts are asked to perform the same tasks. We also study the gamification loss that can occur when, instead of incentivizing users or workers with money, we use a GWAP to have gamers (only motivated by fun) produce meaningful information through their game logs.

Our object segmentation experiments have been conducted on a dataset proposed by [23] in their work on interactive segmentation which makes our results comparable with the results from that study. The collection contains 96 images selected from the larger Berkeley Segmentation Dataset [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF], and also includes the ground truth binary masks for 100 objects (two images have three associated objects each), and a textual description for the users of the object to segment. In order to study and control the quality of our user traces, we have added five images from the PASCAL VOC [START_REF] Everingham | The Pascal visual object classes (VOC) challenge[END_REF] segmentation dataset and have written a textual description of one object per image (see Appendix A) . Therefore our image set is composed of 101 images, and there are 105 tasks (objects to segment) to perform.

Our experiments were conducted on three different user profiles:

-Click'n'Cut -Experts: 15 computer vision researchers from academia, both students and professors. -Click'n'Cut -Workers: 20 paid workers from the platform microworkers.com, 4

Females and 16 Males with ages ranging from 20 to 40 (average: 25.6). Workers were all from Southeast Asia, with a large majority (17 out of 20) of users originating from India and Sri Lanka. Each worker was paid 4 USD for annotating 105 images. -Ask'nSeek -Players: 162 players (mostly students) played the Ask'nSeek game on any number of images they wanted to.

Preliminary figures

Table 1 presents a preliminary comparison of relevant figures for the three groups of users. It is worth mentioning that paid workers produced a substantial number of clicks: in average, workers clicked more that twice as many times as the experts on the same images (they were 20 against 15), and ten times more than the players. Another interesting difference among the groups of users is the ratio between foreground and background clicks. Expert users mostly produce foreground clicks (72 % of the times). Workers also use more foreground clicks, but the ratio is 63 %/37 %. Finally, players tend to produce 57 % of foreground clicks.

The most remarkable value in Table 1 is the percentage of erroneous clicks, defined as the number of clicks that are badly categorized, i.e. foreground clicks that are in fact on the background and vice versa. We did not consider the Partially On clicks from Ask'nSeek Moreover, Click'n'Cut users were given the possibility to produce as many clicks as they wanted to, whereas Ask'nSeek players are limited by a 2-minute timer (which includes the time to type labels, and exchange indications). The game stops when the seeker finds the target, which occurs after an average of 1.6 indications per game. Finally, Click'n'Cut users are focused on one object for each image: the object they have to segment. In Ask'nSeek, the players use all the objects they can see in the image. In other words it takes two players to play an average of 30 games that usually produce 1.6 clicks each, which, due to the game's nature, do not necessarily have to be related to a single object. This preliminary set of results also highlights the difference in number of errors performed by workers on Click'n'Cut when compared to Ask'nSeek players. As we will see in Section 4.2, the high error rate of the workers using Click'n'Cut is partly caused by a specific subset of workers who performed particularly poorly. Ask'nSeek players error rate is much more evenly spread across users (game players). The fact that Ask'nSeek is a game naturally limits the impact of some of the usual sources of errors in crowdsourcing, namely: the spammers, the incompetent workers, and workers with insufficient attention [START_REF] Oleson | Programmatic gold: Targeted and scalable quality assurance in crowdsourcing[END_REF]. Being a game, Ask'nSeek is relatively safe from spammers (the game has nothing to offer except for enjoyment; if players do not like the game, they are free to leave), and workers' attention is kept at a certain level by the non-repetitiveness of the task. Unlike Click'n'Cut where the task to perform is always the same, Ask'nSeek players regularly switch roles (from master to seeker) and since the players' pairing is random, players interact with different people over time. The major source of errors in Ask'nSeek is the misunderstanding between the master and the seeker. Misunderstanding can arise from an imprecise requested object from the seeker (e.g. "dog" in an image where there are three dogs), or from the master not understanding a word used by the seeker.

User profiling based on the types and correctness of the clicks

In this section we take a closer look at user traces on Click'n'Cut in order to explore the details behind the crowdsourcing loss suffered between the results of the experts and the paid workers. The right graph of Fig. 5 presents the percentage of errors on foreground clicks (in blue) and on background clicks (in red). Note that these percentages do not take into account the number of foreground and background clicks, which means that the mean of the two percentages is not equivalent to the total error rate. It is worth noting that the expert's highest source of errors seems to be the foreground clicks. To further understand this phenomenon, let us consider the following numbers. Expert users have produced 24,611 clicks on 15 * 105 = 1, 575 tasks, and among those clicks there were 1,042 wrong ones. The 10 tasks (out of 1,575) for which the most errors were made account for a total of 372 errors, i.e. more than one third of the errors. This error rate distribution is shown on Fig. 6, on which we can see that a very large majority of the tasks had a very low error rate.

Figure 7 presents two tasks that have created a large number of errors from the experts. On the left, only the man's hat should have been segmented. Two experts segmented the man, which created 100 errors (one tenth of the total number). On the right, the description of the fish to segment ("topmost fish on the center-right") was also misunderstood by two experts.

It is interesting to note that these errors are due to insufficient attention from the experts, which suggests that we should always have more than one expert performing a task (typically in our traces, there were never more than two experts who misunderstood a task at the same time).

Figure 8 shows the same type of plot as in Fig. 5 but this time for paid workers. The first very obvious fact is that, unlike expert users, workers have a very heterogeneous way of interacting with Click'n'Cut. Five workers out of 20 produced a majority of background clicks, whereas we previously observed that all expert users clicked a higher number of foreground clicks. The distribution of the number of clicks is also clearly biased by one user (user # 1), who produced an average of 160 clicks per image (only foreground clicks). The right plot of Fig. 8 also shows that this particular user (user # 1) made very few errors. We should be careful with the data from this user since it can affect our results a lot, without being statistically significant.

The biggest difference between experts (Fig. 5) and workers (Fig. 8) is the diversity of user profiles in the case of the workers. This heterogeneity is visible when comparing the graphs of error percentages, depicted on the graphs on the right-hand-side of each figure. This observation has led us to propose a categorization of the workers, inspired by [START_REF] Oleson | Programmatic gold: Targeted and scalable quality assurance in crowdsourcing[END_REF], and exemplified in the cases contained in Fig. 9: -Worker # 1, a.k.a. the "painter", only used foreground clicks, with an exceptional amount of clicks and an error rate almost equal to 0 %. In fact we suspect that this user misunderstood the task and believed he/she had to entirely paint the object with green clicks. -Worker # 3, a.k.a. the "mirror", has such a high error rate that by inverting his/her contributions (i.e. considering his/her background clicks as foreground, and vice versa), he/she would actually display a very low error rate. We can only assume that he/she misunderstood the instructions, confusing foreground and background clicks. -Worker # 5 and # 8, a.k.a. the "border guards", produced almost exclusively background clicks located on the border of the objects. -Worker # 18, a.k.a. the "surrounder", produced only foreground clicks, and has almost 100 % errors. He tried to surround the object with foreground clicks, in a similar way as requested in LabelMe [START_REF] Russell | Labelme: A database and web-based tool for image annotation[END_REF]. -Worker # 19, a.k.a. the "spammer", randomly placed foreground clicks over the image so that he would get paid. This worker completed the entire set of tasks in less than 5 minutes, whereas it took 30 to 60 minutes to a honest user. -Workers # 7, # 9 and # 17, a.k.a. the "tired", had used a very limited number of clicks but, unlike the "spammer", most of the clicks are correct. -Workers # 2, # 4, # 6, # 12, # 13, # 14, # 15, # 16, # 20, a.k.a. "the expert workers", understood the task and completed it carefully. These workers exhibit statistics (in number of clicks and error rate) that are comparable to expert users. -Workers # 10 and # 11 can not be categorized as only one type of users. They changed their pattern of interaction during their session. For example, worker # 11 started as "a border guard", and probably understood the interface after a while and ended up interacting as an expert.

Building up on the observations we have made to establish our categorization, we manually defined some criteria to automatically categorize users based on the following statistics:

-Average number of clicks per task #cli -Percentage of foreground clicks (i.e. left clicks) %FG -Median percentage of errors per task %err (We chose to use the median because, as we have shown in Fig. 6, a small amount of tasks can have a drastic effect on the average error rate. The median is more robust to this small amount of outlying tasks). -Average Jaccard index per task J, described below.

-Average time spent per task t -Percentage of clicks that lie near the contour of the object %Cont Fig. 9 Six types of workers: (top) the "painter", the "mirror", and the "border guard"; (bottom) the "surrounder", the "spammer" and the "tired"

We use the Jaccard index as a measure of the segmentation precision. The Jaccard Index is defined as J = P ∩GT P ∪GT between the Predicted (P) and Ground Truth (GT) masks. We then define the set of following rules that will help us categorize users:

-Rule 1: A "painter" is a worker who clicked more than 100 times per task with more than 90 % foreground clicks (#cli >= 100 ∩ %FG >= 90). -Rule 2: A "mirror" is a worker who has a median error rate greater than 90 % with less than 50 % clicks on the contour (%err >= 90 ∩ %Cont <= 50). -Rule 3: A "border guard" is a worker who clicked more than 50 % of the time on the contour with less than 50 % foreground clicks (#Cont >= 50 ∩ %FG < 50). -Rule 4: A "surrounder" is a worker who clicked more than 50 % of the time on the contour with more than 50 % foreground clicks (#Cont >= 50 ∩ %FG >= 50). -Rule 5: A "spammer" is a worker who spent less than 10 seconds per task with more than 20 % errors per task (t <= 10 ∩ %err >= 20). -Rule 6: A "tired" is a worker who spent less than 10 seconds per task with less than 20 % errors per task (t <= 10 ∩ %err < 20). -Rule 7: An "expert worker" is a worker who recorded an average Jaccard Index greater than or equal to 0.80 (J >= 0.8).

Note that, in order to be mutually exclusive, the workers defined in Rules 1 to 6 should record an average Jaccard Index lower than 0.80. If a worker does not fit any of these categories, he/she is labelled as "others".

We used the rules above to categorize users based on their statistics on the Gold Standard images. Table 2 summarizes our results:

Looking at Table 2, it can be seen that we mis-categorized three workers (out of 20), namely workers #10, #11, and #15. The main reason for these three errors was the "others" category, which currently designates users who drastically change their interaction pattern during the study. The results on Table 2 suggest that five Gold Standard images may not be sufficient to evaluate users and predict to which categories they belong with 100 % accuracy.

In any case, he main lesson of this categorization of user profiles is that, except for worker #19, who was just a spammer, the highest number of errors came from users who did not understand the job properly. This could have been avoided, or at least limited, with a

Filtering low quality workers with gold standard tasks

Section 4.2 has shown that while experts present in general a uniform and acceptable error rate on generated clicks, workers tend to offer a much more heterogeneous performance, resulting in some cases in completely misleading interactions. This section presents principled strategies to detect and discard these low quality workers.

The only data we can use to filter workers are the traces on the gold standard images, i.e., the five PASCAL images introduced to serve as a control dataset. Figure 10 displays the error rate per user on the gold standard dataset (in blue) and on the test dataset (in red).

There is a strong correlation between error rates on the gold standard and on the test set. A notable exception is worker #7, who made no mistakes on the gold standard set, but still showed an error rate greater than 30 % on the test set. Fig. 10 Error rate on the gold standard image (in blue) and on the tasks (in red) for each worker (from [START_REF] Carlier | Click'n'cut: Crowdsourced interactive segmentation with object candidates[END_REF]) Fig. 11 Evolution of the overall error rate on gold standard images (in blue) and on the tasks (in red) when filtering users based on a threshold on the gold standard error rate Nevertheless, we can filter a considerable amount of errors by just removing the workers that are above a threshold of error rate on the gold standard. In Fig. 11, we vary the threshold that we use to filter users based on the gold standard images. The blue curve (resp. red curve) represents the error rate of the remaining users on the gold standard (resp. on the test set).

In the next set of experiments we consider two thresholds based on the graph on Fig. 11. First we will keep only the users who made less than 50 % errors on the gold standard, which makes the total error rate around 20 % on the gold standard. We also try the smaller threshold of 20 %, which makes the total error rate less than 10 % on the gold standard (and on the test data as well).

Table 3 presents the results on the three experiments: Click'n'Cut with experts, Click'n'Cut with workers, and finally Ask'nSeek. We use the Jaccard index (defined earlier) as a measure of the segmentation precision.

The experts' results are computed for each expert separately, and then averaged over all experts. The results thus mean that one expert will obtain an average of 0.89 Jaccard on each task. Note that filtering experts based on their gold standard performances does not make a lot of sense, since all experts have an error rate below 10 % on the gold standard.

There are many things to be said about these numbers. First and not surprisingly, the experts obtain the best segmentation score. This is understandable because they are fully aware that they are performing segmentation (unlike Ask'nSeek players), and they already know what is a good segmentation and what are the main difficulties to obtain it. In other words, their experience helps them focus on more meaningful regions to click on than workers for example. Workers' results are very dependent on the filtering based on the gold standard images. The results range from 0.14 without filtering (which is very bad) to 0.82 when considering only users with a low error rate on gold standard images. On the other hand, Ask'nSeek results are very low compared to Click'n'Cut experiments.

Figure 12 shows the Jaccard index that can be obtained with different amounts of clicks from the three experiments. It can be observed that Click'n'Cut users are able to achieve a significantly better object segmentation result compared to Ask'nSeek players. What is interesting from this graph is the evolution of the Jaccard index for the Ask'nSeek traces, which does not seem to improve with the amount of clicks. This is due to the big difference in the distribution of clicks in Ask'nSeek and Click'n'Cut, which will be discussed later in this paper.

To conclude this section, we should add that we have explored more advanced methods for filtering data in another work [START_REF] Cabezas | Quality control in crowdsourced object segmentation[END_REF] and showed that workers can reach a Jaccard Index of 0.86 when an appropriate filtering is applied.

The crowdsourcing loss

The results introduced earlier reveal several facts that could be characterized as crowdsourcing loss, i.e. a loss induced by having a task performed by workers instead of experts. First, the data is noisier with workers than with experts. This can be visualized in Fig. 13: green and red points are densely concentrated in the central row (workers) and much more sparse on the top row (experts). This is caused by many factors: spammers, workers who do not understand the task, a lower attention level, etc. This already underlines a key message: crowdsourcing tasks must be carefully designed and include quality controls to detect errors.

Second, most of the best workers from the crowd still perform worse than average experts. The bottom row of Fig. 13 depicts the collected clicks collected from the filtered crowdworkers and we can see that, though the clicks are free of noise, they are also sparser than the expert clicks.

In fact when we compute the Jaccard index of each crowd worker, we observe that the expert workers perform worse than the average expert. Eight "expert workers" reach a Jaccard index between 0.80 and 0.86 which is below the average (0.89, see Table 3) of the real experts. Only two workers can really compare to the real experts: they reached a 0.89 and 0.90 jaccard index. Figure 13 also shows that there are areas in which users (expert and crowd) tend to click more. In our experience, these high density areas are located on regions where the underlying superpixels produced by the segmentation algorithm fail to follow the object boundaries. As such, high density areas convey a lot of information: they can help improving the superpixel segmentation for example. Filtering users, as we can see on Fig. 13, makes these high density areas disappear.

In order to understand why expert workers do not perform as well as true experts, we looked at the interaction patterns of these two groups. The results are presented in Tables 4 and5.

We regroup the interaction by considering the alternating pattern between foreground clicks and background clicks. For example if a user clicked twice on the foreground, and then once on the background, we categorize the interaction as following the FB pattern in the table (F stands for foreground, B for background).

For each pattern, we display the frequency of appearance of this pattern and the average Jaccard Index such pattern led to. We also compute the average number of clicks in the sequence. For example on the second line of Table 4, when experts follow the FB pattern, they start in average with 5.7 foreground clicks and end with 3.9 background clicks in average.

From these tables, we can first say that both experts and expert workers tend to always start with foreground clicks. But an important difference between these two groups of users is that the most frequent pattern among expert workers is to only click on the foreground, which leads them to a 0.8541 Jaccard Index in average. If we compare these numbers to the ones from true experts, we can say that expert workers do not have the same sense of a complete segmentation than experts do. For those cases when experts only click on the foreground, they tend to click more (8.9 clicks against 8.4 in average) and obtain a significantly better Jaccard Index (0.8949 against 0.8541).

In addition, experts also tend to have longer interaction patterns than expert workers. We only displayed in both tables the patterns that appear more than 5 % of the time, and the sequences FBFBFB and FBFBFBF respect this constraint among experts, but not among expert workers. This means that expert workers are less inclined to have long interaction sequences, and sometimes stop interacting too soon to achieve a very good Jaccard Index.

To sum up, expert workers do not perform as well as experts mostly because they stop interacting too soon. This could be due to two principal reasons: first they are less conscious of what a good segmentation is, and second they are motivated to complete the tasks as quickly as possible and as a consequence tend to quickly move on to the next task once the result is acceptable.

The gamification loss

This section aims at discussing why the segmentation results obtained with the Ask'nSeek game are poorer than the ones obtained with the Click'n'Cut interactive segmentation tool.

A first very simple reason for Ask'nSeek's performance is the number of clicks gathered through the game, as discussed in Section 4.1. It is important to state here that one of the limitations of our current approach is the difficulty of processing the free text entered by the seeker when asking for a clue (see Section 3.2). Even by doing it manually there are many labels that remain hard to categorize, either because they are not precise enough or because they are not understandable enough. It would be interesting to study simple ways (e.g., autocompletion, limited vocabulary, and so forth) to make the natural language processing more straightforward.

The second reason of Ask'nSeek's poor performances is the spatial distribution of the seekers' clicks. Figure 14 shows the foreground and background clicks gathered through Ask'nSeek on two images from our dataset. On the left image, the entire soldier that stands on the right should be segmented. We can see that all the foreground clicks are focused on the soldier's head. We can make a similar observation on the right image: the foreground clicks are concentrated into the duck's head whereas the entire duck should be segmented in our case.

What is even more interesting is the spatial distribution of the background clicks. We can see that on Ask'nSeek traces, the background clicks are mostly located on other objects, which is understandable: the seekers know that the target region is often located on a salient object, so their clicks are focused on objects (other soldiers, or other ducks in Fig. 14). When we look at Fig. 13, we can see that background points obtained through Click'n'Cut are almost always located near the object's boundaries. In other words, the gamification loss is a direct consequence of the nature of the game itself: the players know that the most efficient strategy to win in Ask'nSeek is to place the target region on an object, and preferably on a salient part of the object (e.g., human's face).

Future work

The obtained results provide several opportunities for research in order to reduce the crowdsourcing and gamification losses for object segmentation introduced by our tools Click'n'Cut and Ask'nSeek.

A common aspect to improve is a more accurate user categorization in order to improve any collected feedback. As discussed in Section 4.2, erroneous feedback can have different sources and discarding users completely may be a too drastic solution. For this reason, we foresee an automatic categorization of users to feed different object segmentation algorithms depending on the type of collected feedback.

Regarding Click'n'Cut, we have seen that most errors from paid workers are due to a misunderstanding of the job. Even though we see the potential of using the variety of traces produced by these workers, we believe that adding a tutorial at the beginning of the experiment would allow users to understand the task better and it would simplify the challenge of filtering errors.

Regarding Ask'nSeek, our analysis on Section 4.5 encourages the definition of an active learning setup for the game. Given that Ask'nSeek traces are biased towards the most prominent objects (and object parts) on the images, a saliency estimator could determine where to place the target region on an image in order to gain as much information as possible from the game logs. This may allow biasing the master's choice of the target region (e.g., by granting more points if the master follows the system's advice) in order to gather more informative traces through the Ask'nSeek game.

In addition, one of the major limitations of Ask'nSeek is related to the text processing of the collected tags. In this work these tags have been manually selected and clustered, while a more realistic approach should be able to perform this task automatically by resolving disambiguation problems and identifying synonyms or related terms.

Conclusion

In this paper, we have presented and studied the crowdsourcing and gamification losses incurred when attempting to solve a computer vision problem, such as object segmentation, with human users in the loop. This work has been carried on by using the Click'n'Cut interactive segmentation tool and the Ask'nSeek game, both accessed online by users. The study has considered three different groups of users: experts and workers on Click'n'Cut, and players on Ask'nSeek. Not surprisingly, the experts who use Click'n'Cut produced the best results. The crowd of workers originally produced very noisy inputs, but we have shown how a simple filtering method based on gold standard images can bring acceptable results. Finally, results obtained through Ask'nSeek are poor and significantly worse than the results obtained through Click'n'Cut, as a consequence of the nature of the task (playing a game rather than segmenting an object through a series of clicks).

We have given special attention to the analysis of the loss induced by having a crowd of paid workers perform an object segmentation task, when the quality of the segmentation results produced by such crowd is compared to the one obtained when we invited computer vision experts to perform the same tasks. We have seen that the workers are less efficient than expert users in positioning their clicks in meaningful areas. However, this loss could be compensated by a better of use of the diversity of workers' profiles that actually produce a high amount of clicks that are wrong, but informative nevertheless.

Conversely, Ask'nSeek clicks are not very informative because they are very redundant. Due to the nature of the game, players are biased towards positioning their clicks on objects, which is not necessarily the best strategy for our segmentation algorithm. In the future we plan to modify the gameplay in order to encourage a higher diversity of clicks' positions.

To conclude, we would like to emphasize once more the workers' categorization depicted in Fig. 9. Among the different categories of workers we have identified and labeled, currently the only traces that are useful to our algorithm are the ones provided by the "expert" workers. We do acknowledge, though, that the traces from the "painter", the "surrounder" and the "border guard" carry a lot of potentially useful information for the segmentation task. Devising a smart way of using this information could produce an approach that overcomes the current best performances in interactive segmentation. This suggests an interesting research avenue that, to our knowledge, may have not been explored enough: introducing several different types of user interaction to bring complementary information that would work together to achieve improved object segmentation results.

Fig. 2

 2 Fig. 2 Screenshot of the Click'n'Cut interface

Fig. 3

 3 Fig. 3 Screenshots of the Ask'nSeek game: seeker's screen (top); master's screen (bottom)

Fig. 5

 5 Fig. 5 On the left, number of foreground (blue) /background (red) clicks on the left and on the right, percentage of foreground/background errors per expert user on Click'n'Cut

Figure 5

 5 Figure5shows an analysis of the experts' clicks and errors. The numbers are averaged on all tasks. Expert users produce in average from 7 to 26 clicks per image; interestingly enough, the proportion of foreground (blue)/background (red) clicks for each expert user is fairly similar, regardless of the image.The right graph of Fig.5presents the percentage of errors on foreground clicks (in blue) and on background clicks (in red). Note that these percentages do not take into account the number of foreground and background clicks, which means that the mean of the two percentages is not equivalent to the total error rate. It is worth noting that the expert's highest source of errors seems to be the foreground clicks.

Fig. 6

 6 Fig. 6 Histogram of the error rates per task

Fig. 7

 7 Fig. 7 Two of the tasks that produced many errors. Descriptions associated to the tasks are: 'Extract just the man's hat. Do not include the rest of the man or any or any other objects.' (left) and 'Extract the topmost fish on the center-right of the image.' (right)

Fig. 8

 8 Fig. 8 On the left, number of foreground (left) /background (right) clicks; on the right, percentage of foreground/background errors (right) per paid worker on Click'n'Cut

Fig. 12

 12 Fig. 12 Jaccard Index vs. the number of clicks used for segmentation

Fig. 14

 14 Fig. 14 Spatial distribution of the clicks (foreground in green, background in red) from the Ask'nSeek players

Table 1

 1 Comparison of the number of clicks and error rates in the different setups , as 1) they represent a minority of clicks, and 2) they are a specificity of Ask'nSeek, therefore not comparable to Click'n'Cut. A very important finding derived from these early results is the difference in the amount of clicks per user that are collected with the two interfaces. There are several reasons that explain this difference. First, it takes two players to produce a click in Ask'nSeek (the master and the seeker) whereas only one user is necessary in Click'n'Cut. Secondly, the users in Ask'nSeek played an average of 30 games (i.e. images) each, whereas Click'n'Cut users performed the entire set of 105 tasks.

		Click'n'Cut	Click'n'Cut	Ask'nSeek
		Experts	Workers	Players
	# Users	15	20	162
	# Clicks	234.4	544.6	51.4
	(per image, all	168 FG	345.8 FG	29 FG
	users included)	66.4 BG	198.8 BG	21 BG
				1.4 Part. On
	# Errors	4 %	35 %	7 %
	in this percentage

Table 2

 2 User categorization obtained on the gold standard images versus ground truth user categorization gold standard images that would have taught workers what is a good click and what is a bad click. Nevertheless the collected data was processed to address a realistic scenario in which crowdsourced workers do not understand or even try to understand the provided instructions.

	Category	Ground Truth	Categorization based on Gold Standard images
	Painter	#1	#1
	Mirror	#3	#3
	Border Guard	#5, #8	#5, #8, #10, #11
	Surrounder	#18	#18
	Spammer	#19	#19
	Tired	#7, #9, #17	#7, #9, #17
	Expert Worker #2, #4, #6, #12, #13, #14, #15, #16, #20 #2, #4, #6, #12, #13, #14, #16, #20
	Others	#10, #11	#15
	better tutorial on

Table 3

 3 Average Jaccard Index on the test dataset in the three experiments

		Click'n'Cut	Click'n'Cut	Ask'nSeek
		Experts	Workers	Players
	All users	0.89	0.14	0.44
	Users with less than 50 % errors on GS	0.89	0.63	0.43
	Users with less than 20 % errors on GS	0.89	0.82	0.40

Table 4

 4 Analysis of the experts' interaction patterns

	Pattern	Frequency	Average number of clicks in the sequence	Average Jaccard index
	FB	18.4 %	5.7 ; 3.9	0.8867
	F	18.2 %	8.9	0.8949
	FBF	17.8 %	6.7 ; 2.7 ; 8.0	0.8980
	FBFB	12.1 %	4.4 ; 3.0 ; 3.6 ; 2.7	0.8907
	FBFBF	10.4 %	4.5 ; 2.6 ; 4.2 ; 2.6 ; 5.4	0.8873
	FBFBFB	6.6 %	3.9 ; 2.8 ; 3.0 ; 2.4 ; 3.2 ; 3.6	0.8728
	FBFBFBF	6.3 %	4.2 ; 2.8 ; 3.2 ; 2.5 ; 5.1 ; 2.2 ; 5.7	0.8693
	Others	10.2 %	N/A	N/A

Table 5

 5 Analysis of the expert workers' interaction patterns

	Pattern	Frequency	Average number of clicks in the sequence	Average Jaccard index
	F	28.8 %	8.4	0.8541
	FB	22.3 %	5.7 ; 3.3	0.8654
	FBF	12.5 %	6.9 ; 2.2 ; 6.9	0.8605
	FBFB	8.4 %	6.5 ; 2.6 ; 3.9 ; 2.7	0.8822
	FBFBF	5.4 %	6.8 ; 2.8 ; 4.4 ; 2.0 ; 6.1	0.8648
	Others	22.6 %	N/A	N/A

Acknowledgments This work has been developed in the framework of the project TEC2013-43935-R, financed by the Spanish Ministerio de Economa y Competitividad and the European Regional Development Fund (ERDF).

Appendix A: Images used in our experiments

We used the five images taken from the PASCAL VOC dataset (see Fig. 15) as Gold Standard in our experiments. The textual descriptions that were provided to the users during Fig. [START_REF] Jain | Predicting sufficient annotation strength for interactive foreground segmentation[END_REF] Original images (top) and ground truth segmentation masks (bottom) of the 5 images we used in our exeperiments as gold standard our experiments were :

-Extract the woman from the image. Include her hair, her clothes, and the part of her arm that holds the bottle. -Extract the cat from the image. Try to discard the dog's paw laying on the cat.

-Extract the woman from the image. Include her hair.

-Extract the bus from the image. Do not include the mirrors on the front of the bus.

-Extract the cat from the image.


Laura N Vandenberg

Wade V Welshons

Frederick S Vom Saal

Pierre-Louis Toutain

John Peterson Myers

Should oral gavage be abandoned in toxicity testing of endocrine disruptors?

Keywords: BPA, Complications, Corticosterone, Glucuronidation, Hormone, Pharmacokinetic, Sublingual

For decades, hazard assessments for environmental chemicals have used intra-gastric gavage to assess the effects of 'oral' exposures. It is now widely usedand in some cases requiredby US federal agencies to assess potential toxicity of endocrine disrupting chemicals (EDCs). In this review we enumerate several reasons why gavage is not appropriate for the assessment of EDCs using bisphenol A (BPA) as a main example. First, whereas human dietary exposures interact with the oral mucosa, gavage exposures avoid these interactions, leading to dramatic differences in absorption, bioavailability and metabolism with implications for toxicokinetic assumptions and models. Additionally, there are well acknowledged complications associated with gavage, such as perforation of the esophagus that diminish its value in toxicological experiments. Finally, the gavage protocol itself can induce stress responses by the endocrine system and confound the assessment of EDCs. These serious flaws have not been taken into account in interpreting results of EDC research. We propose the exploration of alternatives to mimic human exposures when there are multiple exposure routes/sources and when exposures are chronic. We conclude that gavage may be preferred over other routes for some environmental chemicals in some circumstances, but it does not appropriately model human dietary exposures for many chemicals. Because it avoids exposure pathways, is stressful, and thus interferes with endocrine responses, gavage should be abandoned as the default route of administration for hazard assessments of EDCs.

Background

The realization that a large number of chemicals used in products disrupt endocrine function, referred to as endocrine disrupting chemicals (EDCs), has created significant challenges for the approaches used in hazard assessments of environmental chemicals. Knowledge generated in the last several decades about EDCs questions many assumptions used in chemical risk assessments. This has led to a number of published studies disputing current hazard and risk assessment approaches and assumptions for studies of EDCs [START_REF] Diamanti-Kandarakis | Endocrine-disrupting chemical: an endocrine society scientific statement[END_REF][START_REF] Vandenberg | Hormones and endocrine-disrupting chemicals: low-dose effects and nonmonotonic dose responses[END_REF][START_REF] Zoeller | Endocrine-disrupting chemicals and public health protection: a statement of principles from the endocrine society[END_REF][START_REF] Gore | Policy decisions on endocrine disruptors should be based on science across disciplines: a response to Dietrich et al[END_REF][START_REF] Gore | Editorial: an international riposte to naysayers of endocrinedisrupting chemicals[END_REF].

In this review we examine a method commonly used in toxicology studies, intra-gastric gavage, to deliver chemicals to subjects in a controlled manner. Intra-gastric gavage involves the insertion of a tube into the mouth, the sliding of this tube through the esophagus, and the deposition of the compound directly into the stomach [START_REF]Oral gavage in mice and rats[END_REF]. This method has been widely used to test potential hazards of EDCs because it is thought to allow precise control of both the dose and timing of treatment. Two comparable delivery methods used in humans involve enclosing the chemical in pills that only release their contents in the stomach and the delivery of nutrients directly to the stomach of patients (enteral dosing) via a feeding tube.

We specifically address recent studies that challenge the use of gavage for the study of EDCs using bisphenol A (BPA) as a model EDC. We chose BPA because human exposures are widespread [START_REF] Vandenberg | Urine, serum and tissue biomonitoring studies indicate widespread exposure to bisphenol A[END_REF], low doses have been linked to adverse effects in laboratory animals [START_REF] Vom Saal | Chapel Hill bisphenol A expert panel consensus statement: integration of mechanisms, effects in animals and potential to impact human health at current levels of exposure[END_REF][START_REF] Vandenberg | Low dose effects of Bisphenol A: An integrated review of in vitro, laboratory animal and epidemiology studies[END_REF], exposures are associated with a wide range of human diseases [START_REF] Rochester | Bisphenol A and human health: A review of the literature[END_REF], and there remain unanswered questions about how best to model routes and sources of exposure [START_REF] Vandenberg | Human exposures to bisphenol A: mismatches between data and assumptions[END_REF]. Although we chose to focus on BPA, the issue of a lack of detailed understanding of all potential routes of exposure applies to many chemicals used in a wide range of products [START_REF]Toxic ignorance[END_REF].

We also review a number of studies suggesting that gavage does not appropriately model dietary chemical exposures and what the consequences might be for the interpretation of toxicokinetic studies. We discuss how gavage can induce stress responses that may interfere with hazard assessments of chemicals that have endocrine modes of action. We conclude by identifying other methods of chemical administration and address how to best mimic exposures experienced by humans.

Gavage does not model human dietary exposures

When exposures occur via the diet, the chemical can interact with numerous surfaces in the oral cavity including the buccal, sublingual, gingival, palatal and labial mucosa [START_REF] Madhav | Recent trends in oral transmucosal drug delivery systems: an emphasis on the soft palatal route[END_REF]. Each of these surfaces is fed by a rich blood supply, and the epithelia within the oral mucosa are 4 to 4000-times more penetrable than skin [START_REF] Madhav | Recent trends in oral transmucosal drug delivery systems: an emphasis on the soft palatal route[END_REF]. Chemical absorption occurs efficiently via these surfaces, leading to rapid transport to the arteries delivering the chemical to tissues. Importantly, chemicals that are absorbed via this route evade a sequential first-pass metabolism first by the gut wall and then by the liver. For example, the gut wall first-pass effect is relevant in rats for BPA, but not in humans [START_REF] Mazur | Differences between human and rat intestinal and hepatic bisphenol A glucuronidation and the influence of alamethicin on in vitro kinetic measurements[END_REF], whereas hepatic firstpass metabolism is most important in humans [START_REF] Gayrard | High bioavailability of bisphenol A from sublingual exposure[END_REF][START_REF] Vom Saal | Bisphenol A (BPA) pharmacokinetics with daily oral bolus or continuous exposure via silastic capsules in pregnant Rhesus monkeys: relevance for human exposures[END_REF]. When chemicals that are absorbed in the mouth evade first-pass metabolism, they are then bioavailable at higher levels relative to chemicals absorbed from the gut that are transported directly to the liver via the mesenteric vessels.

The toxicokinetic profiles of several chemicals have been shown to differ substantially in animals treated via gavage versus administration via other oral routes. In one example, rodents were administered donepezil, a pharmaceutical used in the treatment of Alzheimer's disease, either via gavage or in a solution consumed from a syringe [START_REF] Atcha | Alternative method of oral dosing for rats[END_REF]. Both blood and brain concentrations were lower when the drug was administered via gavage compared to when it was swallowed, differences the authors attributed to absorption via the buccal and/or sublingual surfaces. Steroid hormone absorption is more rapid and leads to higher blood concentrations when administered sublingually than when ingested via a capsule or pill [START_REF] Kuhl | Pharmacology of estrogens and progestogens: influence of different routes of administration[END_REF]. Gavage versus dosed feed administration of benzyl acetate affected the carcinogenic response by this chemical [START_REF] Yuan | Effects of gavage versus dosed feed administration on the toxicokinetics of benzyl acetate in rats and mice[END_REF]. Finally, for Sulindac, a non-steroidal antiinflammatory, oral gavage resulted in higher peak and lower trough concentrations in plasma and mammary tissue, and had a greater effect on prostaglandin E2 levels than the corresponding dietary dosing [START_REF] Kapetanovic | Effects of oral dosing paradigms (gavage versus diet) on pharmacokinetics and pharmacodynamics[END_REF].

In a recent study, circulating BPA concentrations were examined in dogs administered BPA via the sublingual mucosa or via traditional gavage methods [START_REF] Gayrard | High bioavailability of bisphenol A from sublingual exposure[END_REF]. Following sublingual exposure, BPA entered into circulation largely in an unconjugated form, consistent with a pathway that evades hepatic first-pass metabolism. This contrasted with the results of gavage administration in the same study, in which over 99% of BPA reaching the systemic circulation was in a conjugated (glucuronidated) form, as would be expected when a compound undergoes extensive firstpass metabolism [START_REF] Gayrard | High bioavailability of bisphenol A from sublingual exposure[END_REF][START_REF] Doerge | Pharmacokinetics of bisphenol A in neonatal and adult Sprague-Dawley rats[END_REF][START_REF] Doerge | Pharmacokinetics of bisphenol A in neonatal and adult rhesus monkeys[END_REF]]. An interesting additional comparison is provided by studies in which BPA was administered to rhesus monkeys via gavage [START_REF] Patterson | Concurrent determination of bisphenol A pharmacokinetics in maternal and fetal rhesus monkeys[END_REF] and less than 1% of administered BPA was bioavailable (unconjugated) in blood; in contrast, BPA fed in a piece of fruit resulted in over 7% of administered BPA being bioavailable in blood [START_REF] Vom Saal | Bisphenol A (BPA) pharmacokinetics with daily oral bolus or continuous exposure via silastic capsules in pregnant Rhesus monkeys: relevance for human exposures[END_REF][START_REF] Taylor | Similarity of Bisphenol A Pharmacokinetics in Rhesus Monkeys and Mice: Relevance for Human Exposure[END_REF].

A human de facto variant of gavage involves delivery of the chemical in hard pills or gelatinous capsules. Like gavage, this form of administration prevents the chemical from interacting with the oral mucosa, leading to toxicokinetic parameters that are unlikely to reflect all of the human experience for a chemical that people are exposed to via an oral route. Unfortunately, the only experimental human BPA toxicokinetic study conducted to date delivered the chemical using a hard gelatin capsule [START_REF] Volkel | Metabolism and kinetics of bisphenol a in humans at low doses following oral administration[END_REF], rendering their results irrelevant to real-life human exposures via the diet.

For chemicals like BPA found as contaminants in food and beverages, human exposures via dietary sources likely occur throughout the day in meals and snacks. Buccal absorption associated with discrete meals should generate a series of BPA peaks in the blood and then troughs during the inter-digestive period, rather than a low steady exposure expected following intestinal absorption. In addition, the amplitude of these intermittent bursts of BPA concentrations could be higher in arterial blood than those that can be measured peripherally [START_REF] Chiou | The phenomenon and rationale of marked dependence of drug concentration on blood sampling site. Implications in pharmacokinetics, pharmacodynamics, toxicology and therapeutics (Part I)[END_REF]. Although there are protocols to employ the gavage method of administering chemicals to animals repeatedly during the day, handling animals and inserting a tube into their stomach multiple times daily is clearly problematic. Thus, studies aimed to assess the hazards of chemicals have used gavage protocols that typically rely on a single daily administration [see for example [START_REF] Eichenlaub-Ritter | Exposure of mouse oocytes to bisphenol A causes meiotic arrest but not aneuploidy[END_REF][START_REF] Howdeshell | Gestational and lactational exposure to ethinyl estradiol, but not bisphenol A, decreases androgen-dependent reproductive organ weights and epididymal sperm abundance in the male long evans hooded rat[END_REF][START_REF] Larocca | Effects of in utero exposure to Bisphenol A or diethylstilbestrol on the adult male reproductive system[END_REF]]. These differences between real-world experience and experimental protocol in the timing and duration of exposures reduce the utility of gavage protocols for studies concerned with mimicking human dietary exposures. A study that directly compared the toxicokinetic profiles of BPA in mice exposed via gavage and those exposed in their feed revealed higher concentrations of unconjugated BPA in the blood of mice that consumed BPA via food versus those that were treated via gavage [START_REF] Sieli | Comparison of serum http://www.ehjournal.net/content/13/1/46 bisphenol A concentrations in mice exposed to bisphenol A through the diet versus oral bolus exposure[END_REF]. In fact, after adjusting for dose, the maximum serum concentration reached was 3-fold greater when BPA was consumed in feed vs. when administered via gavage. Note that the rodent mouth is more heavily keratinized compared to dog or primate [START_REF] Gayrard | High bioavailability of bisphenol A from sublingual exposure[END_REF], reducing but likely not eliminating oral absorption.

In sum, exposure by gavage or by swallowing a capsule may result in significant differences in toxicokinetics compared to an oral exposure that mimics the routes and timing of exposure to chemicals consumed in food or beverages. There are two principle reasons: gavage avoids absorption by surfaces in the oral cavity, and it typically involves only a single daily pulse (bolus). An analysis of data that showed that time since the last meal did not account for BPA levels in urine in the US population [START_REF] Stahlhut | Bisphenol A data in NHANES suggest longer than expected half-life, substantial nonfood exposure, or both[END_REF] suggested that gavage models [START_REF] Dekant | Human exposure to bisphenol A by biomonitoring: methods, results and assessment of environmental exposures[END_REF][START_REF] Lakind | Daily intake of bisphenol A and potential sources of exposure: 2005-2006 National Health and Nutrition Examination Survey[END_REF] are likely not accounting for major sources and routes of BPA exposure.

Why are these differences in toxicokinetic parameters important?

Calculating toxicokinetic parameters from data obtained in gavage and/or hard gelatin capsule experiments could lead to inaccurate conclusions about expected bioactive levels of EDCs in serum in the general population. For example, because gavage/capsule experiments show that over 99% of BPA absorbed in the gastrointestinal tract undergoes first pass metabolism, a recent study estimated that the amount of unconjugated BPA in human serum would be below the limit of detection of even the most sensitive analytical methods available [START_REF] Teeguarden | Are typical human serum BPA concentrations measureable and sufficient to be estrogenic in the general population?[END_REF]. This analysis assumes that no absorption occurs via the oral mucosa; yet as discussed above, oral absorption can be substantial and, in fact, not significantly different from exposure via IV injection [START_REF] Gayrard | High bioavailability of bisphenol A from sublingual exposure[END_REF]. The assumption that almost all BPA from oral exposures undergoes first-pass metabolism has been used to argue that any observed unconjugated BPA in human serum must be the result of contaminated lab procedures, and therefore has been used to dismiss dozens of biomonitoring studies that report unconjugated BPA in human serum samples [START_REF] Vandenberg | Human exposures to bisphenol A: mismatches between data and assumptions[END_REF].

A recent NIH-sponsored study assessed whether BPA could be measured accurately in human serum by multiple laboratories and whether there was contamination during blood collection [START_REF] Birnbaum | Consortium-based science: The NIEHS's multipronged, collaborative approach to assessing the health effects of Bisphenol A[END_REF]. The conclusion was that sources of contamination could be identified and eliminated, allowing contamination-free assays to be conducted [START_REF] Vandenberg | A round robin approach to the analysis of bisphenol A (BPA) in human blood samples[END_REF]. Other studies have similarly reported that BPA contaminations can be eliminated during blood collection. For example, one study referred to BPA contamination as "an elusive laboratory challenge" [START_REF] Ye | Potential external contamination with bisphenol A and other ubiquitous organic environmental chemicals during biomonitoring analysis: An elusive laboratory challenge[END_REF] and another noted "a propensity to introduce artifactual aglycone BPA" into assays [START_REF] Patterson | Concurrent determination of bisphenol A pharmacokinetics in maternal and fetal rhesus monkeys[END_REF], yet both of these studies report that sources of BPA contamination could be identified and eliminated prior to collection and assay of serum samples. Thus, BPA contamination can be controlled when assaying human serum and urine samples. Suggestions that unconjugated BPA cannot be detected in human serum also imply that BPA should have no effects on human health at current levels of exposure [START_REF] Teeguarden | Are typical human serum BPA concentrations measureable and sufficient to be estrogenic in the general population?[END_REF]. Yet over sixty epidemiology studies show associations between BPA exposures and disease outcomes [START_REF] Rochester | Bisphenol A and human health: A review of the literature[END_REF].

Stress associated with gavage is problematic for endocrine endpoints

A large literature has examined the effects of gavage on stress. A meta-analysis of experiments designed to assess the effects of gavage (without assessing the effects of specific test chemicals) indicates that the process of gavaging animals induces rapid, pronounced and statistically significant effects on stress-related responses [START_REF] Balcombe | Laboratory routines cause animal stress[END_REF]. Individual studies have shown that gavage can increase the secretion of the stress-response hormone corticosterone in mouse feces, evidence that the hypothalamicpituitary-adrenal axis is activated [START_REF] Walker | A less stressful alternative to oral gavage for pharmacological and toxicological studies in mice[END_REF]. The effects on corticosterone levels can vary depending on the vehicle and volume of liquid used [START_REF] Brown | Stress produced by gavage administration in the rat[END_REF][START_REF] Bonnichsen | The welfare impact of gavaging laboratory rats[END_REF]. Other studies indicate that gavage affects cardiovascular endpoints including diastolic and systolic blood pressure and heart rate [START_REF] Balcombe | Laboratory routines cause animal stress[END_REF][START_REF] Okva | Refinements for intragastric gavage in rats[END_REF]. Interestingly, Okva and colleagues demonstrated that small volumes of gavage fluid have greater effects on heart parameters than larger volumes [START_REF] Okva | Refinements for intragastric gavage in rats[END_REF], an unexpected finding that challenges the assumption that stress responses occur only when animals are gavaged with inappropriately high volumes. In these studies, physiological differences were apparent at least one hour after the gavage protocol. This is likely to be of particular importance when studying pregnant females; acute maternal stress causes elevated serum corticosterone levels to persist for 12 hours [START_REF] Montano | Serum corticosterone in fetal mice: sex differences, circadian changes, and effect of maternal stress[END_REF], indicating that a significant portion of the day could be spent with elevated circulating stress hormones due to stress during pregnancy.

Stress has a significant impact on some of the major drug-metabolizing enzyme systems; different forms of stress were reported to either increase metabolic clearance [START_REF] Bousquet | Endocrine modification of drug responses in the rat[END_REF] or decrease metabolic rate [START_REF] Pollack | Chronic stress impairs oxidative metabolism and hepatic excretion of model xenobiotic substrates in the rat[END_REF]. For example, stressed rats display significantly decreased blood levels of hexobarbital, pentobarbital and meprobamate, but not phenobarbital, after administration of these drugs [START_REF] Driever | Stress stimulation of drug metabolism in the rat[END_REF]. The ability of stress to decrease drug levels in blood was shown to be dependent upon an intact pituitary-adrenal axis.

In an ongoing FDA-sponsored study that is examining the effects of BPA on physiological and behavioral endpoints, a wide range of BPA doses were administered daily by gavage first to pregnant rats and then to the offspring from birth for up to two years [START_REF] Birnbaum | Consortium-based science: The NIEHS's multipronged, collaborative approach to assessing the health effects of Bisphenol A[END_REF]. In the first published study from this experiment, BPA produced significant effects at the two lowest doses in the experiment (2.5 and 25 μg/kg/day) on expression of the genes for estrogen receptors alpha and beta, Esr1 and Esr2, in the brains of neonates [START_REF] Cao | Prenatal bisphenol a exposure alters sex-specific estrogen receptor expression in the neonatal rat hypothalamus and amygdala[END_REF]. The authors also reported that the gavage protocol alone altered these endpoints compared to pups produced from mothers that had not been gavaged but had been subjected to all other handling and restraint procedures; restraint is commonly used as a stress-inducing procedure in rats [START_REF] Kumar | Ascorbic acid protects against restraint stress-induced memory deficits in Wistar rats[END_REF]. Cao et al. noted that gavage may confound the study of BPA and other EDCs [START_REF] Cao | Prenatal bisphenol a exposure alters sex-specific estrogen receptor expression in the neonatal rat hypothalamus and amygdala[END_REF], writing, "Interpreting the BPA and [ethinyl estradiol] results in regions where there were statistically significant differences in [estrogen receptor] expression between the naïve and vehicle controls requires caution because the BPA-or [ethinyl estradiol]related effects on expression may have been influenced by prenatal stress (from gavage)". Thus, there is evidence that restraining and gavaging animals induces stress responses that can increase the probability of false negative or false positive findings if appropriate controls are not employed in endocrine disruption research.

Unfortunately, in the next phase of the FDA study, the only control animals not treated with BPA are being subjected to both restraint and daily gavage, so the control animals will be subjected to significant multiple sources of stress on a daily basis beginning during fetal life and then continuing from birth, for as long as 2 years. As noted by Cao and colleagues [START_REF] Cao | Prenatal bisphenol a exposure alters sex-specific estrogen receptor expression in the neonatal rat hypothalamus and amygdala[END_REF], this experimental design will impair the ability to distinguish the effects of the compound from the effects of stress.

Roberts et al. reported that gavage can also induce significant levels of apoptosis in the liver, a response with obvious physiological consequences [START_REF] Roberts | Dosing-induced stress causes hepatocyte apoptosis in rats primed by the rodent nongenotoxic hepatocarcinogen cyproterone acetate[END_REF]. Because this study was not initially designed to test the effects of gavage itself, the authors concluded that without the use of appropriate control groups (the inclusion of both untreated controls and vehicle gavage controls), completely different conclusions could have been made about the effects of their test chemical on the liver. Roberts et al. also demonstrated that the effects of gavage on liver apoptosis can be masked by co-treatment with compounds that act as liver hyperplastic agents, increasing concerns that gavage can confound or alter the results of studies designed to test the health effects of pharmaceuticals or xenobiotics. These concerns were again raised by Okva et al. [START_REF] Okva | Refinements for intragastric gavage in rats[END_REF], who noted that "[p]hysiological, metabolic, endocrine and behavioural changes can be attributed to stress, and [intragastric]-gavaging procedures as such may interfere with the evaluation of novel drugs administered by this route."

Collectively, this evidence has wide-reaching implications. Thousands of studies performed by regulatory agencies, academic researchers and industry labs have used gavage to assess thousands of compounds, including EDCs. Data showing that gavage activates the stress response system indicate that these time-consuming and costly studies cannot be interpreted as showing effects (or no effect) of any chemical due to the confounding factor of stress; at best, positive findings can only be interpreted as due to an interaction between stress and chemical exposure. More problematic from a public health perspective is the interpretation of results that do not show a significant effect (i.e. negative results), since assumptions regarding the absence of hazard may be based on masking of an effect by the stress associated with gavage. Multiple controls are required to separate these interacting factors, and studies that do not include all of the appropriate controls cannot be easily interpreted.

Acknowledged complications with gavage

The issues we have raised here are largely but not completely unacknowledged in the literature and are compounded by other issues also widely recognized. For example, a common complication with gavage involves perforation of the esophagus or stomach [START_REF] Bonnichsen | The welfare impact of gavaging laboratory rats[END_REF]. Additional complications involve the accidental introduction of fluids into the trachea or lungs, asphyxia, inflammation, weight loss, hemorrhage, and reflux [START_REF] Bonnichsen | The welfare impact of gavaging laboratory rats[END_REF][START_REF] Damsch | Gavage-related reflux in rats: identification, pathogenesis, and toxicological implications (review)[END_REF]. Other studies report high frequencies of morbidity and mortality due to the use of gavage; several report rates greater than 50% [START_REF] Balcombe | Laboratory routines cause animal stress[END_REF]. Numerous factors can influence the success of gavage, including the level of experience of the technician, size and type of probe used, volume administered, repetitive dosing and vehicle used [START_REF] Okva | Refinements for intragastric gavage in rats[END_REF]. Because of the relatively high frequency of gavage-related complications and the need for highly trained animal care staff, some animal welfare groups propose that alternative dosing methods be used whenever possible [START_REF]Oral gavage in mice and rats[END_REF].

Gavage has been used in many hazard assessments because this method is thought to allow precise control of both the dose and timing of treatment. Yet, recent studies have revealed that this assumption may not be accurate. In a recent study using 'Good Laboratory Practices' (GLP), scientists from the FDA reported substantial BPA contamination (including serum concentrations of BPAmetabolites consistent with uncontrolled exposures of up to 80 μg BPA/kg/day) in two sets of negative controls: those that were administered an oil vehicle, and those that were not gavaged with any compound [START_REF] Churchwell | Comparison of life-stage-dependent internal dosimetry for bisphenol a, ethinyl estradiol, a reference estrogen, and endogenous estradiol to test an estrogenic mode of action in sprague dawley rats[END_REF][START_REF] Delclos | Toxicity evaluation of bisphenol A administered by gavage to Sprague Dawley rats from gestation day 6 through postnatal day 90[END_REF]. Unfortunately, the source of the contamination could not be identified, so it cannot be determined what role the gavage method itself played in this inadvertent exposure. Importantly, these results suggest that the positive aspects of using gavage (i.e. precise control of exposures) may overestimate the value of this exposure route, and thus other large GLP-compliant studies that have used gavage but have not measured BPA and BPA metabolites in serum to evaluate actual contamination should be examined with more caution (for example, [START_REF] Ferguson | Developmental treatment with bisphenol A or ethinyl estradiol causes few alterations on early preweaning measures[END_REF][START_REF] Ferguson | Developmental treatment with bisphenol A causes few alterations on measures of postweaning activity and learning[END_REF]).

Should endocrine disruptor screening assays use gavage?

The US EPA's Endocrine Disruptor Screening Program states that gavage delivery is preferred in its Tier 1 testing protocols [START_REF]Endocrine Disruptor Screening Program (EDSP): Assay status table[END_REF]. All of the studies included in their validation of two protocols, the Hershberger assay and the TG-407 28-day oral toxicity study, solely used the gavage method [START_REF]Preliminary draft under development: EDSP and other validation chemicals at a glance[END_REF]. Similarly, the NTP's developmental and reproductive screening assays are expected to use the most relevant route of exposure for the chemical of interest, but note that when conducting toxicokinetic studies, "[i]f feed or drinking water will be the exposure route in the toxicology study, gavage exposure should also be included to estimate the absorption parameter(s)" [START_REF] Buchanan | Purpose and guidelines for toxicokinetic studies within the National Toxicology Program[END_REF].

Data from toxicokinetic studies and data from studies investigating the effects of gavage itself do not support the use of gavage in the testing of putative EDCs. Clearly, toxicokinetic profiles can differ when exposures occur via gavage versus other routes of exposure [START_REF] Gayrard | High bioavailability of bisphenol A from sublingual exposure[END_REF]. Additionally, the stressful nature of the gavage method can alter function of the hypothalamic-pituitary-adrenal axis [START_REF] Walker | A less stressful alternative to oral gavage for pharmacological and toxicological studies in mice[END_REF] and could confound the assessment of chemicals that interfere with hormones that act on this axis. Because the endocrine system has complex positive-and negative-feedback loops, the effects of a stressful event may not be limited to endpoints associated with the hypothalamic-pituitary-adrenal axis [START_REF] Bourke | Stress-induced sex differences: adaptations mediated by the glucocorticoid receptor[END_REF] challenging the use of gavage for the assessment of any endocrine-responsive endpoint. As noted elsewhere, the gastrointestinal tract "is the largest endocrine organ system in the body, and may secrete more hormones than all other organ systems combined" [START_REF] Engelking | Metabolic and Endocrine Physiology[END_REF]. Therefore, it is essential to understand the implications of the administration of EDCs to the gastrointestinal tract, for example by requiring the inclusion of both non-gavaged controls as well as vehicle-only gavage controls.

Other options for chemical administration

For hypothesis testing, route of exposure may not be of central importance, but for hazard assessment, risk assessors typically require that studies use a route of exposure that is deemed "relevant" to humans [START_REF] Buchanan | Purpose and guidelines for toxicokinetic studies within the National Toxicology Program[END_REF]. There are exceptions to this requirement: the FDA requires that all animal studies used in hazard assessments employ oral exposure routes, even if non-oral routes occur or are expected; an example is diethyl-phthalate (DEP), a compound used in cosmetics with dermal and inhalatory human exposure routes that has been tested in animals via oral administration [START_REF] Parlett | Women's exposure to phthalates in relation to use of personal care products[END_REF].

All dosing methods have pros and cons that must be considered in the design of a study, bearing in mind how the data will be used. Most of this review has focused on limitations to the gavage method, particularly because we have not seen these issues acknowledged in a regulatory context, and because recent studies suggest that the use of gavage may interfere with the study of EDCs [START_REF] Cao | Prenatal bisphenol a exposure alters sex-specific estrogen receptor expression in the neonatal rat hypothalamus and amygdala[END_REF]. Yet, EDCs and pharmaceuticals have also been administered by milling the compound of interest into feed, dissolving it in drinking water, feeding animals from a pipette, or adding the compound to a wafer or other food and allowing the animal to consume it. Although the FDA and other regulatory or advisory panels have clearly given priority to studies that use oral and gastric exposures [61,[START_REF] Chapin | NTP-CERHR expert panel report on the reproductive and developmental toxicity of bisphenol A[END_REF], it is important to note that there are alternatives, including Silastic implants and osmotic pumps. These implanted devices are of particular interest in BPA studies because they can provide constant exposures to low doses that produce serum concentrations that approximate those found in humans [START_REF] Vom Saal | Bisphenol A (BPA) pharmacokinetics with daily oral bolus or continuous exposure via silastic capsules in pregnant Rhesus monkeys: relevance for human exposures[END_REF][START_REF] Acevedo | Perinatally administered bisphenol a as a potential mammary gland carcinogen in rats[END_REF][START_REF] Hunt | Bisphenol A alters early oogenesis and follicle formation in the fetal ovary of the rhesus monkey[END_REF]. These routes of exposure may be relevant also because there are important and significant non-oral sources of BPA exposure [START_REF] Vandenberg | Human exposures to bisphenol A: mismatches between data and assumptions[END_REF][START_REF] Geens | A review of dietary and non-dietary exposure to bisphenol-A[END_REF]. The issue of stress during the placement of implanted devices, and also the possibility of chronic inflammation that can alter disposition of many substances and hormones, however, requires further attention.

Conclusions: how well do exposure protocols in experimental studies mimic human exposures?

For a chemical like BPA, the majority of exposures are currently thought to come from food, but by no means does this mean that gavage exposures adequately model human dietary exposures, nor should this prediction eliminate from consideration non-dietary sources [START_REF] Geens | A review of dietary and non-dietary exposure to bisphenol-A[END_REF]; the European Food Safety Authority recently acknowledged that there was "uncertainty" regarding the degree to which thermal paper contributed to BPA exposure in the general population [START_REF]Draft scientific opinion on the risks to public health related to the presence of bisphenol A (BPA) in foodstuffsexposure assessment[END_REF]. In the study of chemicals with multiple routes and chronic exposures, which method of administration appropriately mimics this situation? At one end of the spectrum, it has been proposed that as long as a method produces circulating blood concentrations that are within the range of what has been measured in humans, the study should be considered relevant, regardless of how the chemical was administered [START_REF] Vandenberg | Hormones and endocrine-disrupting chemicals: low-dose effects and nonmonotonic dose responses[END_REF]. At the other end of the spectrum is the FDA's stance that only oral exposures, preferably by gavage, are relevant, even if actual human exposures occur via other routes.

We conclude that the use of gavage administration as the recommended approach may be inappropriate in the study of EDCs because it can confound studies by inducing substantial stress in animals, thus altering all endocrine and non-endocrine responses associated with a regular meal. Gavage also avoids oral absorptions that are part of human dietary exposure. Although there are limitations to all methods of dosing animals, the flaws associated with gavage are so severe for some substances, especially those subjected to a large first-pass effect, that this route of exposure should be abandoned for the study of EDCs. For some chemicals, there may be reasons why gavage is preferred over other routes of administration, but this method should not be employed using the rationale that it 'appropriately models human dietary exposures'; the data do not support this assumption.

http://www.ehjournal.net/content/13/1/46

Acknowledgements

The authors thank colleagues in the fields of endocrinology and environmental health for fruitful discussions on this topic. Funding for JPM was by the John Merck Fund, for FSvS by NIEHS grants ES021394 and ES020952 and for WVW by UMC AES MO-VMFC0018.

Abbreviations BPA: Bisphenol A; DEP: Diethyl-phthalate; EDC: Endocrine disrupting chemical; EPA: Environmental Protection Agency; FDA: Food and Drug Administration; GLP: Good Laboratory Practices; NTP: National Toxicology Program; TCDD: 2,3,7,8-tetrachlorobenzo-p-dioxin.

Competing interests WVW, P-LT and JPM declare no conflicts of interest. LNV provided expert testimony in a civil case involving a product that might contain EDCs. FVS wrote a report for attorneys involved in product labeling litigation.

Authors' contributions LNV, WVW, FSvS, P-LT and JPM wrote the paper. All authors read and approved the final manuscript.