Jean Kempf. Proposition pour le débat. Journée M. Kammen, EHESS, 10 mai 2002.

Historian of Paradox(es). Michael Kammen and the History of Memory in Mystic Chords of Memory.

Michael Kammen likes paradoxes. Nay, he thrives on them ; they have guided his historical oeuvre since the publication of his first book, and I would contend that despite, or maybe because of, his apparently gentle and balanced approach, they form the most central and stimulating aspect of his long line of interpretations of American history.

He chose George Washington's famous oxymoron as the title of his first book ( A Rope of Sand, 1968), published his (Pulitzer prize winner) « Inquiry concerning the Origins of American Civilization » under the title of People of Paradox, and then described the Constitution as A Machine that Would Go of Itself1 . When defining -and defending -his historical method he accumulated, with evident rhetorical pleasure more oxymorons, more paradoxes, more conundrums, speaking of « disciplined subjectivity », of the « thin line between acuity and vacuity » and eventually even asking the reader-in much more than a rhetorical phrase-to display « a critical but compassionate acuity » toward his research. 2 At a more structural level, he adressed one of his central preoccupations as cultural historian-the constitution of American society-from the « problem »-read the formal incompatibility-of tradition vs democracy, and declared his fascination-his core problématique -for « a society becoming its own historian ». 3 Finally, in an historiographical context of heavy deconstruction, of generalized fragmentation, and of extreme conceptualization and theorizing, Kammen unashamedly wrote in 1972 about American « Civilization » (a term that seemed to have disappeared from mainstream history since Max Lerner's famous book in 19574 ) and in 1991 published a massive, avowedly comprehensive (if not all-encompassing), history of memory in American culture.

It takes some guts to write extensively (Kammen must be one of the most prolific contemporary historians) against the grain. But here again, in a paradoxical way. His books rely heavily on secondary sources more than on direct archival research. But in handling sources he displays a truly amazing deftness and erudition, giving them an entirely new dimension. His own view of American history is thoroughly informed by the work of all American historians of the moment and not only that of his own persuasion or field. In fact, he must be, with John Higham (to whom Mystic Chords is dedicated) one of the best informed student of American historiography, a fact also attested by his numerous articles on the topic (the best of which could very well be « An Americanist's reprise » on the « histoire problème »5 ), by his many reviews, as well as his compiling for the American Historical Association of a collection of essays on historical writing in the United States and his authoring of a thick authoritative introduction;6 and eventually, no less significantly, his presiding the OAH.

Kammen goes on, teaching but not preaching, questioning but never dismissing, and simultaneouly gently agitating « prolegomenons », « codas » and « syzygies » in the face of « intextuating », « emplotting » and « essentializing ».7 I believe that nowhere is this more visible and significant than in his work on memory, and particularly in Mystic Chords, a book which encompasses all his favorite questions and allows us best to see what he can teach us as regards American history.

I -Michael Kammen and the Memory moment in American historiography. A central outsider

The modern concern for memory, in American and generally in Western European historiography, literaly burst in the early 1980s. Before that date there had been a few landmark works but they remained mostly outside the field of history. The first author to displace the concept of « memory » from the field of psychology (and/or philosophy) and its attendant anatomic-physiologic topicality, was Maurice Halbwachs (1925, 1950) with his sociological concept of « mémoire collective », identifying in fact a « mémoire sociale ». Then came the anthropologists, and particularly the structural anthropologists (Levi-Strauss) and the cultural ones (Geertz). As the scholars of antiquity (Moses Finley) they interrogated memory as part of the myth of the origin, and placed it in direct relation with the formation of identity. The first historian after Finley to really study memory in medieval and classical Europe was Francis Yates with his Art of Memory (1960). The anthropological model was thus applied to a diachronic reading in which memory formed a functionnal/structuring element in the social fabric. It appeared as a « forming » practice, but remained limited in its definition to memorization, oral transmission and in fine to the perpetuation of the social order -a sociological perspective -more than to the creative possibilities of evolution.

The new wave of historical concern for memory was much more ambitious and more ambiguous as well. It was both a reaction to the explosion of « patrimonial » concern, commemorating frenzy and « heritage fever » in Europe and the United States, and an internal reaction, in the field of professional history, to the advent of cultural history as the new overarching paradigm of historical studies. The emergence of a conception of history as « history of representations » is too well known to be repeated here. 8 Suffice it to say that as the concept of « invention » became prevalent (if not invasive), the technical shifts in the field of historical practice (oral history, minority studies, social history, cultural geography) combined with social demands (and crises : political, generational, racial, etc.) broadened the meaning of memory to a whole array of practices, ranging from education to commemoration and to collection. This mouvement gave birth to countless studies of « memorial » activity whose volume soared after 1983, just as new concerns about the place of minorities in American society appeared and as a revision (or at least a revisit) of German and Japanese war past took place, new debates took place about the Holocaust, when the land-whichinvented-its-own-history (the USSR) was beginning to thaw. 9 The largest body of works on American memory-and the bestis concerned with conflicts of memory, or more broadly put with the « politics of memory ». 10 They approch it from the perspective of commemoration (the public display of memory), « narrative » (under the influence of anthropology) and the construction of identity in specific groups. Needless to say the Civil War and ethnic groups are their favorite topoi, and their concern is-in a post-foucaldian and often more efficient way-the uncovering of the underlying discursive structures of power.

Michael Kammen's book, although conceived in the late 70s if we are to believe its author (Mystic Chords 705) takes stock of this deep movement in American historiography, participates in it and at the same time cannot be assimilated to it.

For one, Kammen, although starting like most of his colleagues from a contemporary social question, locates the debate in a discrepancy between a « true past » constructed by historians in a post-rankian model (his is only « truer ») and a false (fallacious) popular perception, essentially a nostalgic one. 11 In that respect he could be assimilated with the other historiographies of memory, considering, as David Thelen wrote, that « the debate over 'American memory' is finally about audiences for American history ». 12 Mystic Chords could thus be read as the narrative, the report of the growing realization that truth is in fine less important than belief. But in his development of what is, after all, a traditional opposition between history and memory, Kammen complexifies the picture so much that all versions of facile dichotomies blurr and that even his own attempt at categorizing does not resist the infinite layers of contradictory meanings and evidence ; in the end both are sent back to back, as beauty is in the eyes of the beholder. 13 He is, despite his strong opinions that he does not hesitate to put forward, a methodological empiricist, a point he makes in his « preface to the 1980 edition » of People of Paradox and so plunges even deeper into complexities. 14 As an empiricist, he needs large bodies of facts, huge series along something approaching la longue durée. In Mystic Chords-as in many of his other books only much more so-, Kammen embraces at least a century and a half of American history-a rather broad scope with many complex phenomena at play-and declares his object to be « American culture » at a time when most historians-except maybe Foner-wrote about sub-cultures and sub-groups, narrowly focussing on a topoi or a topic. Kammen does not ignore them, but yet sees them as part of a larger plot. This is what I read in his quoting from Robert Penn Warren (« ...to be an American is not . . . a matter of blood ; it is a matter of an idea ») and eventually from Hamlet (« I have some rights of memory in this kingdom »). 15 Concerned with the making of an American, with the acquisition of a « birth »right (one more paradox of Americanness), he reoccupies the fields of Tocqueville, Turner, Hartz, Boorstin, and so 11 « An Americanist's Reprise », 20. Mystic Chords, 38-39. 12 Mystic Chords, 703-04. 13 Mystic Chords, 703-04. 14 He speaks of his « intuitive and eclectic methodology » (People of Paradox, xv). 15 Mystic Chords, 11.

doing relentlessly explores the E Pluribus Unum, except that now his Unum is not monolithic but pluralistic, not fused but cooperating in a extraordinary, almost baroque but functional bricolage called the Nation. Hence his title: Lincoln's vibrant words, at the close of a rather uncompromising oration on the theme of the strayed sheep and the stern shepherd, evoke this unity in diversity (the chord) but definitely place it on the level of a mystery, some sort of alchemical formula that no scientist can ever hope to explain. We are ushered in the realm of revelation, a world that merely requires belief, or at least recording.

II -Michael Kammen's revision of exceptionnalism

Stating that Michael Kammen is interested in continuities is hardly the greatest academic scoop of the decade. Memory, as he reminds us is essentially about the « past-present relation » (Mystic Chords, 5), and what he choses to define as tradition is very similar to the meaning given to the word by Eric Hobsbawm. 16 His definitely anti-theoretical stance, however, makes his point less clear than it actually is, and his definitions of his « word-concepts » hardly help. The main problem comes from his use of « myth » which takes several meanings despite the attempt at clarification through « the big beige book ». Barthes (quoted in a footnote) would be more in order here, or -but is it really necessary ?-Levi-Strauss. By choosing to do away with the meaning of myth as fallacy, one can escape the (original) debate of a learned truth counterbalancing a popular fallacy, or to be even more comprehensive, the truth of today redressing the fallacies of the past. Calling them « a different perspective » does not solve the ethical debate. On the contrary, limiting (after Barthes ) myth to « the discourse of origins » greatly simplifies our inquiry. Traditions are thus seen as social practices (whether physical or discursive) validated by being handed down from the past and thus demanding repetition (Hobsbawm, 4), and -I believe this point is paramount -legitimized by their naturality : the generational transmission being probably the most central paradigm in the cultural uses of the physical world. It requires a process of formalization and/or ritualization which takes places as needed, and which is -paradoxically -permanently undergoing transformation. And what Kammen shows meticulously is how central to American society this inter-generational portage was (to use pioneer vocabulary), and how contentious, in a fundamentally dispersed more than pluralistic world. After all, the fathers of the Constitution anticipated the « past-isprologue » motto, by stating -in a well-calculated rhetorical twistthat the aim of the new system of governement they were devising was « to secure the blessings of prosperity to ourselves and our posterity».

The intellectual thread running throughout the book is once again a paradoxical one : how, in the face of so many different differences and variables, is it still possible to maintain the idea of an American difference and define an American dynamics of memory (Mystic Chords, 700) ? For Kammen is fascinated by the idea of exceptionalism 17 while at the same time knowing that the question is not operative any more.

In Mystic Chords, a large amount of the comparative chapters, of the introductory pieces and of the partial syntheses that punctate the work are precisely devoted to showing the deep parallels with other nations. In his introduction, he displays his broad knowledge of European historiography and history to reach a conclusion opposite to the neoexceptionalists of the consensus school. They looked abroad -when they did -only to see American experience as irreductible, and protective. Theirs was of course a different context and when Louis Hartz wrote The Liberal Tradition in America (1955) or Arthur Schlesinger The Vital Center (1949), or even Daniel Boorstin The Genius of American Politics (1953), liberalism appeared as a dynamic and vital alternative to the tragic failures of European ideologies. Kammen, on the contrary, first quietly but surely gets rid of a huge staple argument of historical comparatism in one page when he states that one must take into account a time lag phenomenon of about two generations between Europe and the United States. 18 By displacing in his memory studies the focus from polity to society, he then reaches the conclusion that America is not exceptional because of the type of ideology it did (or did not) develop, nor because of what is remembered or how it is remembered -those processes are fairly similar in many societies -but by the fact that it combined in a peculiar way the sphere of the private and that of the public. His conclusions are very close to that of Michael Walzer in What it means to be an American, especially when he places the « immigrant experience » as vitally constitutive of any « Americanness », which in the end is -should be for Walzer -a pluralist nation. 19 Both Kammen and Walzer have read Horace Kallen, but they may have retained different parts. In the end, however, I believe that Culture and Democracy (that Kammen does not 17 Michael Kammen, « The Problem of American Exceptionalism: A Reconsideration », American Quarterly 45 (March 1993): 1-43. 18 See his « comparisons » at the end of each major part, especially pp. 283-296. 19 Michael Walzer, What It Means to Be anAmerican. Essays on the American Experience (New York : Marsilio, 1992) quote), and Kallen's concept of « anonymity » perfectly fit his horizon : the reconciliation between tradition and democracy that he wishes to see in the history of American memory. 20 His long list of practices, from genealogy to family reunions, his study of group memories in the 1920s and 1930s, his reference to scholarship on slavery, all point to possibly the central meaning of American identity : the privatization of heritage and the collectivization of destiny, both united in the conceptual synthesis of memory, looking as much behind as forward, being in fact retrospective because prospective.

This sharing -in fact, ontological separation -of roles has been mirrored in the First Amendment and the status of « the individual » since well before the appearance of the « liberal tradition ». It is also most interestingly and tellingly at work in the construction of personal legacies and the generalized practice of genealogy (Kammen remarks that the only really American religion, Mormonism, is one which defines its idea of salvation on the construction of lineage). 21 To risk a facile comparison, being an American is partaking of this other « dual citizenship » which might constitute the true originality of American experience.

Such conceptualization might even work as a possible subsumation of the 4 modes in which he distributes his investigation about « the dynamics of tradition and the role of collective memory in American culture ». 22 Indeed the debate between tradition and progress, the ambivalence of Americans (they are not the only nation to know such interrogations) towards change and their repeated bouts of nostalgia, as well as « the extent to which historical issues are publicly contested », all fit within this dichotomy, as they all remain part of the private sphere, or to put it more bluntly of a formless market. What Kammen's inventory shows is that American government (broadly construed), even in its most extreme phases of interventionism, even in the perennial reashing of a few stock myths in public speeches and the compulsory appeal to the « promises of baptism » that form the core of political rhetoric, seem to be ontologically concerned with the management of the future. Which leaves it to communities and families, collectors and maecenas, poets and historians, and today to the media, the role of shaping memories. The United States, in that respect, would appear as different from other democracies which have steadily displaced inheritance -the major source of unequality -from the domain of the private to that of the public, through systems of redistribution. 23 That 20 Walzer, 25ff. 21 David Lowenthal concurs with his stimulating remarks on « personal legacies » in The Heritage Crusade and the Spoils of History (New York : Viking, 1996), 34, 56. 22 Mystic Chords, 700. 23 This may also be seen in the American conception of the land / the landscape might very well make America special, as if it had not undergone the sacralization of the nation-state that took place in 19th-century Europe. Or rather, it would make it the perfect child of the Reformation and the Enlightenement, the Reformation secularizing the whole world as John Gillis interestingly suggests, 24 the Enlightenement establishing the figure of the « mobile individual » (a notion that caused Halbwachs in Les Cadres sociaux de la mémoire, to worry about the future cohesion of society).

Michael Kammen, however, is not as concerned as Halbwachs was about a lack of memory, or even the adoption of substitute, fake, sham, superficial memories on the part of certain segments of society (Halbwachs was concerned about the working class). It is not even clear whether the opposition he dwells upon in his opening chapter, by redrawing the Tocquevillian question of the paradox of democracy, is really his main preoccupation. For clearly, his empirical sense and pragmatic philosophy tell him that democracy and tradition do work together and that the willful forgetfulness or re-writing of history are more totalitarian habits than democratic ones ; or rather that what makes democracy is precisely the plurality of cycles of forgetting and re-writing simultaneoulsy going on, making the citizen a partaker of many different strata of memory. What does worry him, however, is the existence of anything close to one (united) « collective » memory in the United States. He never states the question in so many words but each beam and rafter of his huge complex frame point at that possible aporia. In other words, while writing a general history of American memory, Michael Kammen wonders if such a project is possible.

III -Kammen and the paradox of the historian

A good paradoxical start would be Kammen's own comments about his prose style. 25 Answering criticisms made at his People of Paradox, he claimed a highly personal approach to the writing of history. Those who have read him know the rich -albeit sometimes convoluted -forms of a style which is often more expositional than expository. His references are obvious -Carl Becker, but also perhaps Bancroft and Jefferson. The « Prologomenons » and « codas » that he loves so much, could make him pass for a dispenser of truths and systems. Even his strong authorial pose, permanently integrating the reader in his text by outlining the parts in a rather didactic way, as well as the multiplication of proleptic marks to guide him/her might well make him appear as a which has remained a major source of memorial transmission in Europe. 24 Gillis, 18-19. 25 People of Paradox, xv. revealer, almost the equivalent of a developer for photographic images.

Despite his rich (to say the least) documentation and his use of anecdote, he is not a story-teller either. « [G]eneralizations are hazardous and ambiguities abound » (493) seems to be more his leitmotif. His « comparisons » are balanced, his hesitations numerous. He is fascinated by the detail, the idiosyncracy, and most of all the change of focal pespective, to the point that a (French) colleague once said to me: « But what does Kammen mean ? ». I first found the question rather disturbing but it made me realize that one might miss the point of Mystic Chords (to take our example of the day) by not reading it as a « work in the making » or rather as the novel of a work in the making. Such reading would perfectly tie in with its author's persistent interest in iconography and in art, as well as his aesthetic stance. If the comparison was not so overbearing I would not hesitate to place him alongside the Henry James of The Ambassadors. The numerous reprises, qualifications, even the rhytmic patterns of some sentences, unveil the fight between « the obsession with the other thing »26 and the will to « take things as they come », which is the way by which James thematizes the tensions between Europe and America, or rather between the Old and the New worlds. Kammen invites us on a journey of self-discovery, in the deep meanders of uncertainty, misgivings, doubts : history, despite the fact that it is a operation of the mind, is never what you make it / want it to be, and it escapes all possible teleologies, even those of the beholder.

Does it make, however, Michael Kammen more « readable » ? I would argue that it places him in a different paradigm, one in which the historian is less in charge of producing the all-embracing equation of American development -as Frederick Jackson Turner -than a writer entrusted with the duty to carry on a morality. He notes it himself when analyzing the historical production of the mid-nineteenth century, with Bancroft or Parkman : history for them, he wrote, had become « an appropriate surrogate for religion because they believed in the centrality of moral progress over time » (70). With Kammen, it is certainly not a question of moralizing the past, nor even of finding a moral in it, but of (merely ?) being aware of its existence, as a redemptive force.

Redeemer, or simply custodian of the past : the historian comes in when something has been lost : the sense of change, a sense that makes him close to Pierre Nora who opens his Lieux de mémoire with two literary pages of eulogy on the defunct memory. 27 Both indeed write in a sometimes lyrical mode under the influence of the great 19th century historians, both operate from a « certain idea » of their nation to paraphrase the Gaullian epithet ; which makes their theories less exegetic than hermeneutic. Kammen may hesitate as to his « word-concepts » to narrow down the meaning of (narrative) uses of « the past » in the present, and Nora may get caught in the Moebius ring of his definition of his lieux de mémoire when trying to hold in check the possibly infinite inventory of the lieux. Both assert that memory is not about remembering (or forgetting 28 ) but about the struggle between progress and decline. As Nora himself showed, the theme of decline is at the core of all American history. 29 The very definition of American time holds the perpective of decay 30 , which makes destinarian rhetoric and even the various, and sometimes subtle forms of racial/racist discourse in the United States, more a real manifestation of terror than one of superiority.

Progress as a concept, however, is recurrently present in Kammen's study of American « memory », but more often than not sideways, although it clearly is the real object of his investigation. John Bodnar in a long review essay of the American translation of Les Lieux de mémoire, compares the two works at length and reaches the same conclusion, in different terms :

Imbued with the faith of the Enlightenment in human reason and potential, these new states in places such as France and America took up the highly romantic project to make the future better than the past (951) . . . To extrapolate from the work of Nora, Kammen, and Giddens [A. Giddens, Beyond Left and Right : The Future of Radical Politics (Stanford, 1994)],

we can say that under the nation-state there as a sense of the transcendent in cultural politics that helped conservatism an democracy get along and that fostered a belief system that seemed both eternal and loftier than either conservative visions of authority or leftist dreams of mass democracy. In a sense the impulses to democratize and to reassert tradition needed each other. Democracy pushed outward for greater personal freedom ; tradition worked to constrain the individual and to maintain the collective (959) 31 But Bodnar makes democracy once again the anti-thesis of histoires, 1984), xvii-xix 28 Which makes the study of the « trous de mémoire » interesting but, in fine, selfdefeating. Is it a real historical question, ie one which does not hold its own answer ? I would contend that it is not. 29 Pierre Nora, « Le 'Fardeau de l'histoire' aux États-Unis », in Mélanges Pierre Renouvin : études d'histoire des relations internationales (Paris : P.U.F., 1966), 51-74. 30 I find Kammen's quoting T.S. Eliot (Mystic Chords, 14) quite significant of that metaphysics of the historical venture. 31 John Bodnar, « Pierre Nora, National Memory, and Democracy : A Review », Journal of American History 87 n° 3 (december 2000) : 951-63.

tradition and refocuses the debate on politics, as he reads Mystic Chords as pointing at a (contemporary) time where « a mythical nation [is] drained of politics and inequality where people [are] free to pursue a myriad of personal pleasures and leisure-time fantasies. » (957) In other words the commodification of all social relations as a result of the loss of debate between contending forces over the usability of the past, and its replacement with precisely a vague plastic commodity.

The key words here are « debate » and « contending ». Nostalgia and progress are probably the fundamental concepts -that are necessarily present in a dialectic way in a given society, American or other, -to make the present possible. Or rather to make the present more than a « tale . . . full of sound and fury, signifying nothing ».

Nostalgia and Progress are feelings, sentiments, in other words modalizations which define uses of the past (and do not oppose them to burdens). They generate objects and narratives but are not part of them. They are the form of content as semiologists say, already interpretations of the world, compasses in the ocean of time. They integrate all forms of social productions into a significant whole ; more they serve one another -progress feeds on nostalgia which is generated by progress. They are necessary objects of our historical operations, which Michael Kammen, because he is a paradoxical historian, has allowed us to see.

Works Cited

Works by Michael Kammen People of Paradox An Inquiry Concerning the Origins of American

Civilization. Cornell University Press, [1972]

  1980. The Past Before Us. Contemporary Historical Writing in the United States. Ithaca, NY : Cornell University Press, 1980. (editor) Mystic Chords. The Transformation of Tradition in American Culture. New York : Vintage, 1991. « The Problem of American Exceptionalism: A Reconsideration », American Quarterly 45 (March 1993): 1-43. « An Americanist's Reprise : The Pervasive Role of Histoire Problème in Historical Scholarship concerning the United States since the 1960's », in The Challenge of American History, edited by Louis P. Masur (Baltimore : The Johns Hopkins University Press, 1999), 1-25, originaly published as the March 1998 issue of Reviews in American History.

	Other

works Bodnar, John. « Pierre Nora, National Memory, and Democracy : A

One could even add, in a different field, the title of a volume of historiographical essays he compiled in 1980 : The Past Before Us.

« Introduction to the 1980 Edition », People of Paradox, xvi.

The phrase is borrowed from Harry Levin, see Mystic Chords, n1, 787.

The few books that have used « American Civilization » in their titles since 1957 seem to be for the most part either anthologies and/or textbooks, and were mostly published in the 1960s and early 70s.

Michael Kammen, « An Americanist's Reprise : The Pervasive Role of Histoire Problème in Historical Scholarship concerning the United States since the 1960's », in The Challenge of American History, edited by Louis P. Masur (Baltimore : The Johns Hopkins University Press, 1999), 1-25, originaly published as the March 1998 issue of Reviews in American History.

See note 1. « The Historian's Vocation and the State of the Discipline in the United States », 19-46.

See « An Americanist's Reprise », 1 and People of Paradox,

As Jean-Pierre Rioux states quite cogently in « La mémoire collective », Pour une histoire culturelle(Paris : Seuil, 1997), 325-53.

See Natalie Zemon Davis and Randolph Starn, « Introduction », Representations 26 (Spring 1989): 6, and Mystic Chords, 710-13.

For instance, John Bodnar, Remaking America. Public Memory, Commemoration, and Patriotism in the Twentieth Century (Princeton : Princeton University Press, 1992).

Eric Hobsbawm, « Inventing Traditions » in Eric Hobsbawm and Terence Ranger (eds.), The Invention of Tradition (Cambridge : Cambridge University Press, 1983), 1-4

Strether in chapter 1 of The Ambassadors says : « I'm always considering something else ; something else, I mean, than the thing of the moment. The obsession of the other thing is the terror. »

Pierre Nora, « Entre mémoire et histoire. La problématique des lieux » in Pierre Nora (ed.), Les Lieux de mémoire (Paris : Gallimard, Bibliothèque illustrée des


Paolo Crosetto
email: paolo.crosetto@inrae.fr

Thomas De Haan
email: thomas.dehaan@uib.no-

Comparing input interfaces to elicit belief distributions

Keywords: Belief elicitation, Forecasting, Scoring rules, interfaces 1

We develop an intuitive, Click-and-Drag interface to elicit continuous belief distributions of any shape. We test this interface against the state of the art in the experimental literature -a text-based interface and multiple sliders -and in the online forecasting industry -a distribution-manipulation interface similar to the one used at Metaculus, a crowd-forecasting website. By means of a pre-registered experiment on Amazon Mechanical Turk we collect quantitative data on the convergence speed and accuracy of reported beliefs in a series of induced-value scenarios varying by granularity, shape, and time constraints. We also collect subjective data on ease of use, frustration and understanding. Results show that the click-and-drag interface outperforms all others by accuracy and speed, and is self-reported as being more intuitive and less frustrating than other interfaces, confirming our pre-registered hypothesis. Besides pre-registration, we report that the click-and-drag interface generates the least drop-out rate from the task, and scores best in a sentiment analysis of an open-ended general question. Further, we use the interfaces to collect homegrown beliefs on temperature in New York City in 2022 and 2042. On average, all subjects overshoot the real temperature for 2022 by about 2°F, and all anticipate further global warming in the order of 2.3°F; these forecasts are by and large not impacted by the interface used to elicit them. We provide a free and open source, ready to use oTree and Qualtrics plugin of our click-and-drag and all other tested interfaces available at https://beliefelicitation.github.io/.

Introduction

Eliciting beliefs is hard. Subjects might not entertain exact but only fuzzy beliefs on a specific event. They might have a vague idea, but be unable to provide a point estimate. If asked to provide a distribution, they might run into cognitive problems because they do not know what a belief distribution is, or how to report it. On top of all these cognitive and perception problems, lies the added problem of how the elicitation interface might facilitate or hinder the subject in expressing her intuitive belief distribution.

Eliciting beliefs and predictions has become a popular element of study across several fields and disciplines. Beliefs are key in experimental economics and psychology [START_REF] Schotter | Belief elicitation in the laboratory[END_REF][START_REF] Trautmann | Belief elicitation: A horse race among truth serums[END_REF], in experimental research about asset markets [START_REF] Haruvy | Traders' expectations in asset markets: Experimental evidence[END_REF], in macroeconomics, where expectations on key indicators play a crucial role and are elicited in recent experiments [START_REF] Kryvtsov | Central bank communication that works: Lessons from lab experiments[END_REF][START_REF] Armantier | The price is right: Updating inflation expectations in a randomized price information experiment[END_REF][START_REF] Rholes | Should central banks communicate uncertainty in their projections[END_REF], in behavioral decision making, where the importance of graphical interfaces to elicit beliefs has been highlighted [START_REF] Goldstein | Lay understanding of probability distributions[END_REF], in welfare economics to assess preferences for redistribution [START_REF] Page | Subjective beliefs about the income distribution and preferences for redistribution[END_REF], but also in marketing to assess priors of bayesian models [START_REF] Sandor | Designing conjoint choice experiments using managers' prior beliefs[END_REF][START_REF] Delavande | Eliciting Subjective Probabilities in Internet Surveys[END_REF], or in political science [START_REF] Leemann | Eliciting beliefs as distributions in online surveys[END_REF]. Belief elicitation and aggregation has also gained traction outside of academia, for instance in terms of asking professional forecasters, or 'crowds' for predictions, which is happening more and more online (e.g. https://www.predictit.org/ or https://www.metaculus.com/). The type of information elicited has also gotten more detailed. Rather than just asking for an average or a mean, either likely intervals [START_REF] Schlag | A method to elicit beliefs as most likely intervals[END_REF][START_REF] Jain | Unpacking the future: A nudge toward wider subjective confidence intervals[END_REF], modes, medians, or entire distributions [START_REF] Harrison | Scoring rules for subjective probability distributions[END_REF][START_REF] Harrison | Subjective beliefs and statistical forecasts of financial risks: The chief risk officer project[END_REF] are starting to get elicited in experiments.

There can be good reasons to choose to ask for a complete distribution when eliciting a forecast. It is possible that a policymaker is interested in more moments than just the mean of a distribution, or that there are expectations that the distribution for an indicator of interest might have multiple peaks. Another argument is that asking an entire distribution might actually be more intuitive to visualize for a forecaster than a derived measure such as the average or mode. For instance [START_REF] Kröger | What point of a distribution summarises point predictions?[END_REF] show that asking for a point prediction can create confusion when comparing with the complete underlying distribution.

The literature on the elicitation of beliefs is still in an early stage and features fundamental discussions on how to best ask participants about their predictions. One central discussion focuses on whether and how to incentivize participants to give their best belief estimates [START_REF] Trautmann | Belief elicitation: A horse race among truth serums[END_REF][START_REF] Danz | Belief elicitation and behavioral incentive compatibility[END_REF]. A discussion missing so far in the economics literature, but present in the judgment and decision making literature [START_REF] Goldstein | Lay understanding of probability distributions[END_REF] is the explicit testing of the interface used to elicit the belief distribution. If we are going to ask participants to fill in a distribution, which is not a simple task, then entering the distribution and being able to match this to the distribution they have in mind should be made as easy as possible. Frictions caused by the input interface, or frustration with interacting with the interface could create biases or inaccuracies.

We make two contributions. First, we present a first systematic comparison of several elicitation interfaces to test their performance. Second, we introduce a newly developed "Click-and-Drag" interface and test how it compares with other methods used so far for eliciting entire distributions.

We ask participants, recruited via Amazon Mechanical Turk to perform a mimic-the-distribution task. Subjects are shown a target distribution on one part of their screen and can enter a distribution on another part of the screen. Participants are asked to mimic several distributions, where the shape of the target distribution varies, as well as the time they have to work on the task (either 15 or 45 seconds). The participants are paid according to the euclidean distance between the target distribution and theirs, and are hence incentivized to mimic the target distribution as close as possible.

We run 4 treatments, corresponding to 4 different interfaces. The Click-and-Drag interface, where the distribution is determined by support points which the participants can create and place with the mouse. The Slider interface, where each bin has a slider which the participants can individually drag up or down via the mouse. The Text interface, where participants can numerically fill in the height of each bin in the distribution. And the Distribution interface, inspired by the one used on the forecasting website metaculus.com, that starts out with an approximate normal distribution and provides 3 horizontal handles to adjust the mean, variance and skew.

The experiment was pre-registered at OSF. We find that, as pre-registered, the Click-and-Drag interface clearly outperforms all other interfaces in terms of performance and speed of convergence to a final answer. Furthermore we find that participants report the Click-and-Drag interface to be more intuitive and less frustrating than the other interfaces, including for example the Slider interface, which has been frequently used in experimental economics [START_REF] Harrison | Scoring rules for subjective probability distributions[END_REF], [START_REF] Harrison | Subjective beliefs and statistical forecasts of financial risks: The chief risk officer project[END_REF]).

The overall performance difference between the Click-and-Drag interface and the competition is clear. Only in one limited case does this interface get outperformed by the Distribution interface. This is when the distribution to mimic is random and quite erratic and the time given is short, 15 seconds. The Distribution interface appears here to benefit from starting from an initial normal distribution. The Distribution interface does poorly in all other instances. But it looks as though having a ready made reasonable/normal starting distribution helped for short time distribution assignments. This is a feature that could easily be added with Click-and-Drag -but feels like giving too much of a hint to subjects; starting from an empty distribution is instead free from default bias.

We also hope that this paper promotes the idea for experimental economists to more often explicitly test key features of their experimental interface. Experimental economists (and psychologists) often come up with original design solutions to test their hypotheses, however this might leave important features of the designs, which could be sources of bias and interference, untested.

The Click-and-drag belief interface

In creating Click-and-Drag, we had two main aims. First, we wanted an interface which could be understood with a few lines of instructions at most, or be picked up from practicing with a short tutorial or even no instructions at all. Second, we wanted to make an interface that scaled well, i.e. that was equally simple when creating a distribution over 5 or 50 bins, up to a (near) continuous distribution. One could imagine an oil price forecast where small differences matter for the world economy, yet the typical price variance and hence uncertainty of the future price is large.

Our solution to the aforementioned design challenges was to create an interface where one could draw a curve, and this curve would then dictate the shape of the underlying distribution. Drawing a curve is an intuitive exercise, and it effortlessly scales to any different number of bins by simply discretizing the continuous curve. To make this curve-drawing intuitive, we built on top of standard JavaScript libraries. In Click-and-Drag, subjects click to create a point, to which the curve is linearly attached. Points can be added, moved around, and deleted by simply clicking and dragging them. Figure 1 shows a screenshot of the interface in use; an interactive demo of the interface is available at https://beliefelicitation.github.io/.

Under the hood, the application uses the Otree platform (Python/Django framework, Chen et al. ( 2016)). The user interface is developed using the JavaScript libraries Highcharts.js and jQuery. The interface is inspired by the Highcharts demo tools "Click to add a point" and "Basic column". The sliders use the noUiSlider JavaScript library from refreshless.com. And the graphic elements are designed with the Bootstrap css library. The interface was developed by Mu Numérique SAS, Grenoble.

Experimental test

Task

To test the different interfaces we set up a "mimic-the-distribution" task. In this task subjects were asked to recreate a target discrete probability distribution. The target distribution was depicted as a series of bins of varying height, shown in the top right corner of the screen. The rest of the screen was taken up by the belief elicitation interface, where subjects could enter the target distribution. The nearer subjects got to the target distribution, the higher their payoffs. Subjects were paid according to a score (between 0 and 100) which depended on the sum of all the differences between the bin heights of the targeted and entered distributions. Figure 2 shows a screenshot of the task, for the Click-and-Drag interface. Click-and-Drag As described in detail above.

Text-based This rudimentary elicitation interface has long been the norm in experimental economics, mainly for technical reasons. Subjects have to input a number for each of the discrete bins of the chosen support. The sum of the inputted numbers must be equal to 1 (or, equivalently, 100). This interface has been the one most largely used in earlier experiments for its technical simplicity. It is usually limited at 2 to 5 bins, and only seldom used beyond this threshold for practical concerns. In our implementation, the normalization is carried out by the software and no "sum to 100" constraint is imposed on the subjects. One example of the use of the text-based method can be found in the Survey of Consumer Expectations of New York (FRBNY) in the United States where, inspired by Manski [START_REF] Manski | Measuring expectations[END_REF][START_REF] Dominitz | How should we measure consumer confidence[END_REF][START_REF] De Bruin | Measuring consumer uncertainty about future inflation[END_REF] the New York Federal Reserve decided to ask experts the probability that future inflation would lie between two certain percentage levels. There have been critiques to this questionnaire method, for example by [START_REF] David | Response bias in survey measures of expectations: Evidence from the survey of consumer expectations' inflation module[END_REF] who argues this type of density elicitation performs worse than a simpler inflation directional question.

Slider-based This interface is an incremental improvement over the text-based interface. For each bin subjects can move a slider to increase or decrease the probability mass of the bin. Normalization to 1 (equivalently, 100) can be automatic or delegated to the subject. This interface has been used when the tools available to experimental economists have evolved, and among others by [START_REF] Harrison | Scoring rules for subjective probability distributions[END_REF]; [START_REF] Andreozzi | The stability of conditional cooperation: beliefs alone cannot explain the decline of cooperation in social dilemmas[END_REF]; [START_REF] Harrison | Subjective beliefs and economic preferences during the covid-19 pandemic[END_REF]; [START_REF] Fairley | Risky health choices and the balloon economic risk protocol[END_REF]. The Slider interface is also reminiscent of -though not identical to -the Distribution Builder interface [START_REF] Goldstein | Lay understanding of probability distributions[END_REF], where subjects hit on a "+" button to visually add probability mass to any of a series of bins. In our implementation the normalization is carried out by the software.

Distribution The rise of on-line predictions markets and crowd-prediction websites, like, for instance, Metaculus or PredictIt, has created the need for intuitive tools to enter predicted values. For instance, Metaculus uses an interface based on a bounded bell-shaped curve, that is controlled with Left and right handles add mass on the left (or right) of the distribution, increasing in passing its dispersion. We reached out to Metaculus to ask for their code, but did receive a limited reply that made us unable to exactly replicate their interface. We hence created a Distribution elicitation interface based on a skew normal distribution, defined by three parameters: ξ for the location, ω for the scale, and α for the shape. We translated these three parameters into the three-handles interface proposed by Metaculus assigning ξ to the middle (location) handle, ω as the (normalized) distance between the left and right handles, and α as the difference between the distance of the left handle from the center and the distance of the right handle from the center. While we cannot be sure that this is exactly the function used by Metaculus, extensive testing shows us that our interface's behavior is qualitatively very similar to (a discretized version of) Metaculus' interface.

Treatments

We tested the four interfaces between subjects. To assess the robustness of the interfaces to scale, difficulty, and time constraints we added three within-subjects variations.

Target distribution shape. We chose four different continuous shapes increasing in complexity. A truncated normal distribution, symmetric and single-peaked, is the baseline shape. We then added skew; made it bimodal; and finally added random noise to the asymmetric, bimodal distribution. Some shapes are hence harder than others -i.e. they require more tinkering with the interfaceto reproduce.

Level of discretization. The four shapes are then discretized over 7, 15 or 30 bins. This is to assess the ability of the elicitation tool to easily scale to a finer granularity.

Time constraint. Subjects are given 45 or 15 seconds to complete the task. They have to spend the full time constraint on the screen -i.e., they are not allowed to submit the distribution before the time is up. This is done to measure how the different interface's performance scales when subjects are forced to reply in limited time.

Subjects faced 4 shapes × 3 discretizations × 2 times constraints, for a total of 24 screens. The screens were presented in a fixed order, in increasing level of difficulty. Screens started at 45 seconds, symmetric, 7 bins. We cycled through the different shapes, keeping the bins constant, and then through the number of bins, varying shape. The same sequence was run a second time, with all distributions mirrored, but this time with 15 seconds of allotted time. The target distributions for the first cycle of 12 screens are reported in Appendix C; the 12 screens for the second cycle featured the exact same distributions, but mirrored.

Measures

We collect data on each click the subjects make. This allows us to derive two main quantitative measures of the performance of each interface.

Accuracy. We measure accuracy as the complement to 100% of the normalized distance of the final, submitted distribution from the target. Better interfaces have higher accuracy.

Adjustment path. We measure accuracy at each moment subjects interacted with the interface. 4 This allows us to see the path followed by the subjects to get nearer to the target, and hence to assess the speed of convergence to the final, submitted distribution. Some interfaces could allow subjects to quickly get the main strokes done, to then fine-tune the resulting distribution, while others might require to advance by smaller steps. Better interfaces allow for quicker convergence to the target distribution.

We also collect qualitative measures for each interface via three questions focusing on ease of use, frustration, time needed to understand the interface (on 1-to-7 Likert scales) and an open question asking for general comments. We further asked which device was used for input (keyboard, mouse, touchpad, touchscreen) as the interface performance can vary with input methods.

Finally, as a first test of the interfaces to elicit homegrown beliefs, we asked subjects to report their beliefs about the maximum daily temperature in New York City for the 4 th of July 2022 -a date that was in the near future for the experimental subjects -and for the 4 th of July 2042, 20 years in the future. We did this to have a first test of the Click-and-Drag interfaces with homegrown beliefs rather than induced values. The aim is to assess whether there are measurable biases introduced by the interfacesi.e. whether the aggregated beliefs elicited with one interface systematically vary with respect to the same beliefs elicited with another. We do this using temperatures, that is something on which any subject can have at least a fuzzy distribution in mind, and over a 20 years period to assess whether subjects predict an increase in temperature related to global warming.

Sample and session details

We implement the experiment using oTree, and run it on Amazon Mechanical Turk. The oTree application was developed by Mu Numérique SAS, Grenoble. We pre-registered and aimed for 360 Mturkers, 90 per each elicitation interface. Each subject was required to be using a PC (as opposed to a phone or tablet), and had to go through 24 screens, for a grand total of 8640 mimic-the-distribution tasks.

Following common practice, we limited recruitment to subjects geolocated in the United States, having completed at least 500 HITs on Mechanical Turk, and having an approval rate of at least 95%. After instructions (available in Appendix E), subjects faced a playground where they could practice the interface and the task for up to 1 and a half minutes. In order to weed out bots, we then implemented a set of four control questions (see Appendix F). Subjects had up to three attempts to clear the control question screen. After the first and second attempts, they were given feedback on their answers and provided with the correct replies. Subjects that failed three times the control questions were excluded from the experiment, with no bonus. If the case bots were able to pass the control question screen, they were usually unable to interact with the elicitation interfaces, since those are coded in Javascript, with quite custom code, and are much harder for bots to interact with than HTML fields. We labeled as bots and dropped from the sample all subjects not interacting at all on any screen, and thus earning zero.

We pay all screens. For each screen, recreating exactly the target distribution is worth 20 US$ cents; this means that each increase in 5% of the accuracy of the inputted distribution is worth 1 US$ cent. The maximum theoretical payoff is of 4.8 dollars, plus a 50cent fixed bonus. This makes our experiment reasonably well paid for Amazon Mechanical Turkers.

We ran the experiment over three sessions, a first session with 40 subjects to test for eventual bugs and problems (there were none), a session with what we thought would be the right number of subjects (we had to set the number of invited MTurkers taking into account an estimation of the incidence of bots) and a last session once we realized that we had more bots than anticipated, and had to fill in the missing real subjects. All sessions ran from the 20 th to the 22 nd of June 2022.

Confirmatory results

We pre-registered our main hypotheses on OSF, at https://osf.io/ft3s6. All data and analysis script to reproduce all the results in this paper are available at the OSF page of this project or in the dedicated github repository.

We hypothesized that the Click-and-Drag interface would outperform the other three tested interface in terms of accuracy, robustness to increased number of bins, different distribution shapes, and stricter time constraint, convergence in time, and that it would be self-reported as more intuitive and less frustrating. 56

Sample

Table 1 reports the demographics of the sample. As required, all subjects used a PC. We have slightly more usable data than pre-registered (372 vs. 360), slightly above the pre-set 90 per treatment for all treatments. The demographic distribution is not different across treatments by gender (Kruskal-Wallis test, p-value = 0.462), but it is different by age (Kruskal-Wallis test, p-value = 0.035), with the Slider treatment being significantly older than all others (Mann-Withney test, p-values: 0.015 vs. Click-anddrag, 0.018 vs. Text, 0.016 vs. Distribution), that are not different from each other (Mann-Withney test, all p-values > 0.88).

Subjects had to clear a screen of control questions to be able to perform the main task. They had three trials. After the first trial, they were given the correct answers. Between 40 and 50% of subjects cleared the control questions screen at their first trial. This share, a proxy for the average understanding in a treatment, is not significantly different across treatments (Kruskal-Wallis test, p-value = 0.353).

Subjects earned 2 to 3 euros on average, depending on the treatment; the payoffs are statistically different across treatments (Kruskal-Wallis test, p-value < 0.001). The difference is driven by Click-and-Drag, that generates significantly higher payoffs than all other interfaces (Mann-Withney, all p-values < 0.001). Slider and Distribution do not generate statistically different payoffs among each other (Mann-Witheny, p-value = 0.64). Slider does not generate higher payoffs than Text (Mann-Withney, p-value = 0.136), and Distribution yields higher payoff than Text (Mann-Withney, p-value = 0.026). The higher payoffs are an early indication that subjects reached higher accuracy with the Click-and-Drag interface with respect to all other interfaces.

Accuracy

Table 2 gives an overview of the results of our experiment with respect to the accuracy of submitted final distributions across different interfaces, overall and by allotted time, number of bins and shape of the distributions to be mimicked. As pre-registered, the Click-and-Drag interface shows a better overall accuracy than all other interfaces (Mann-Withney, all p-values < 0.001). It is also notably robust to most variations. When moving from the 45 to the 15 seconds condition, it loses less accuracy when compared to Slider and Text (Mann-Withney, all p-values < 0.001), but more than Distribution, that is the only interface starting above zero, and yielding "average" results with minimal (or no) effort. 7 It is also remarkably robust to increasing the number of bins (lowest loss of performance when moving from 7 to 15 bins, and from 15 to 30 bins, against Slider and Text: Mann-Withney, all p-values < 0.001) -again with the exclusion of Distribution. When looking at the shapes, Click-and-Drag outperforms all others for Symmetric (Mann-Withney, all p-values < 0.001), Skewed (Mann-Withney, all p-values < 0.001) and Bimodal (Mann-Withney, all pvalues < 0.03) target distribution shapes; it is not statistically different from Slider for messy, Random shapes (Mann-Withney, p-values = 0.44), but still better than Text and Distribution (Mann-Withney, all p-values < 0.03).

These results are driven mainly by the number of bins dimension, where the performance drop of Slider and Text is most notable. Table A.7, in Appendix Appendix A gives all the detailed results of each of the 24 screens for the interested reader.

Adjustment path

Given the increasing importance of response times, and, more generally, choice processes in experimental economics (see for instance [START_REF] Spiliopoulos | The BCD of response time analysis in experimental economics[END_REF], for a review) the elicitation interfaces were designed to record the state of the distribution after each interaction with the interface -be it a click or entering a number. This allows us to track the speed with which subjects arrive at the final submitted distribution. We pre-registered that Click-and-Drag would be the faster tool, in the sense of being the tool with which subject take less time to cover most of the ground to the final, submitted distribution, while other tools would need more time. This is important both theoretically and practically. In theory, a tool that allows you to get near to the final answer with the first strokes is less likely to induce misreporting, or to be impacted by fatigue or carelessness, with respect to tools that need labor and effort to get where the subject wants. Practically, in applied settings belief elicitation might be done as a side task among many others, and the fact that subjects can quick sketch their beliefs is an important feature of an elicitation interface.

Figure 3 shows the accuracy of subjects, for all screens, separately for the 15 and 45 seconds conditions, in time. Click-and-Drag clearly shows a faster (i.e., in the plot, steeper) curve, especially in the first seconds. Note that Distribution here starts from a mechanical advantage, since the starting distance is not zero; still, its slope is the shallower of all the interfaces, indicating a slow advance towards the final submission. The result is even more clear in the 15 seconds condition, where by the 5 th second Click-and-drag subjects reach an accuracy of 20%, while Slider and Text subjects linger around 3%. The main result -Click-and-drag allows subject to converge faster to the target distribution, is replicated also when looking separately at screens with differing number of bins and differing shapes -see Figures A.5 and A.6 in Appendix Appendix A.

Self-reported assessment

Subjects faced three questions where they could self-report their qualitative assessment of the interface they had used in the task. All three questions were graded on a Likert scale from 1 to 7, and focused on ease of use, frustration, and ability to quickly understand. Table 3 reports the mean and 95% confidence interval of the subjects self-reported assessment on the three dimensions.

Click-and-drag Slider Text Distribution

Hard to use 3. 77 [3.42,4.12] 3. 95 [3.57,4.33] 4.16 [3.8,4.52] 4.51 [4.18,4.84] Frustrating 3. 59 [3.22,3.96] 3.74 [3.35,4.13] 4.24 [3.88,4.6] 4.28 [3.93,4.63] Difficult to understand 3. 48 [3.14,3.82] 3.41 [3.06,3.76] 3.43 [3.07,3.79] 4.05 [3.75,4.35] Table 3: Likert scale (1 to 7) self-reported interface assessment, mean and 95% conf.int.

The Click-and-drag interface ranks first of all interfaces in ease of use and in generating least frustration, and third in the speed of understanding (indeed, it is easier to just input numbers, as there is likely nothing to learn there). Most results are not significant, though. Click-and-drag is perceived as less frustrating than Text (Mann-Withney, p-value = 0.02) and beats Distribution in all dimensions (Mann-Withney, all p-values < 0.02), the rest not being significant.

Exploratory results

This section includes analyses that were not pre-registered. This was due mainly to the fact that even in this age of replication crisis and pre-registration, discovery is still a thing, and some ex-post obviously important analyses did not occur to us ex-ante.

Slackers

Elicitation interfaces can be (very) frustrating. This can lead subjects to drop out -i.e. not to finish the task, get distracted, and just let the time pass without collecting payoffs. Especially in the case of Mechanical Turk subjects in an online, unsupervised setting, subjects could just leave their browser tab open and do other things if the burden of fulfilling the task became heavier than the expected earnings. This could be particularly true for screens with 30 bins, and for the (in effect tedious) Slider and Text interfaces.

The number of persons dropping out on one or more screens -we call them slackers -can then be used as a proxy for the engaging (or frustrating) nature of the interface. We define subjects as having slacked on a screen if they had no or minimal interaction with the screen, and having improved the score from the starting point by less than 5 percentage points over the whole allotted time. 4 reports the mean number of screns slacked, and the distribution of the number of screens (out of 24) on which a subject has slacked, by treatment. The number of slackers and the severity of their disengagement from the task is severely affected by the treatments. The Click-and-drag interface proves to be the most engaging, with nearly half the subjects never slacking on any screen, and most of the other half slacking on only one to three screens. The Slider interface shows significantly more slacking, and the Text interface even more so. For both interfaces, it is indeed extremely frustrating to complete the task for the 15 and 30 bins targets, especially so for the Text interface. The Distribution interface shows a different pattern. In this interface, subjects do not start from zero, but from a roughly normal distribution. They are hence "spoiled" insofar as they start from a more advantageous point. Still, moving the sliders in order to approximate the given picture is sometimes hard, and subjects might have decided for a satisficing strategy keeping things "as is" as they got effortlessly "enough" money from the start. This might explain why no subject exerted effort on all screens; and the majority of subjects is classified as a serious slacker.

Click-and-Drag features the lowest mean amount of screens with no interactions (Mann-Withney, all p-values < 0.001), and thus proves to be, on an online sample of a possibly scarcely motivated population like MTukers, the most engaging interface, generating the least fatigue and drop-out rates.

Robustness of final results to slackers

Given the above results on slackers, it is unclear whether the advantage of the Click-and-drag interface is limited to avoiding slackers. Slackers perform rather badly, and bias the mean performance of the affected interfaces downwards. It is possible that non-slackers reach a similar performance across all interfaces. To check whether the results are robust to focusing only on subjects having devoted full effort on most screens, Table 5 replicates the analysis of 52.73 [50.85,54.61] 26.72 [25.18,28.26] 27.5 [24.9,30.1] 68.88 [50.88,86.88] by shape Symmetric 71. 26 [69.65,72.87] 47.11 [44.28,49.94] 52.5 [48.09,56.91] When limiting the analysis to subject devoting near to full effort, Click-and-Drag loses a bit of its edge. It still outperforms Slider and Text in all the dimensions it did so before (Mann-Withney, all p-values < 0.001), but is not statistically different from Distribution (Mann-Withney, p-value = 0.484). Nonetheless, only two subjects are included for Distribution in this table, showing how hard it was for this interface to be used consistently -98% of subjects slacked more than 3 screens when using this interface, and voiding of all meaning any statistical test. Nonetheless, the selection at play is clear: the 2 heavily self-selected subjects working hard on all screens had a performance non distinguishable from the one of the 90 (out 95) subjects devoting effort with Click-and-Drag. It takes motivation and dedication to reach good results with Distribution, and the frustration it generates discourage 98% of subjects from doing so.

Sentiment analysis of the open-ended comments

Subjects had the possibility of leaving an open-ended comment on their experience within the final questionnaire. This reply was not compulsory, but 149 subjects out of 372 did fill it in. The modal reply was a variation of "good" or "no problems", but some subjects made longer comments, voicing their frustration or showing appreciation for the task. We ran a sentiment analysis on the corpus of replies. Sentiment analysis is a text-mining technique using dictionaries that associate a valence (positive or negative) to any work in a dictionary, and then applying this valence to sentences. The sentiment analysis was run using the R package syuzhet [START_REF] Jockers | Syuzhet: Extract Sentiment and Plot Arcs from Text[END_REF], that works by assigning an emotional value to each word in a text, in the form of a numerical value, that can be positive or negative and whose magnitude indicates the intensity of the emotion. A positive overall mean sentiment means that the message were more positive than negative.

Overall, sentiment over our experiment was positive. Nonetheless, the mean sentiment of the comments varied by interface. The Click-and-drag interface scored the highest mean sentiment, at 0.58 (st.dev. 0.59); followed by Slider (0.43,st.dev. 0.5),Text (0.4,st.dev. 0.41) and Distribution (0.36,st.dev. 0.47). Differences are not significant (Kruskall-Wallis test, p-value = 0.330) and in general all sentiments are positive; but the ranking of the sentiment analysis confirms the ranking of most of the other considered criteria.

The belief elicitation interfaces in action: predicting temperatures in NYC and climate change

On top of the pre-registered horse race of belief elicitation on the mimic-the-distribution task, we asked two non-incentivized direct belief elicitation questions. Subjects used the interface they had been exposed to to fill in this open belief question.

This was a first, unstructured, attempt at observing whether the interface does impact the reported beliefs. We chose to ask two questions related to maximum temperature in New York City on the 4 th of July of two given years -2022, the year the experiment took place, and 2042, twenty years in the future. We chose this topic as one with which everyone is familiar, on which reasonable information is available to anyone, and to gauge average belief in an uncertain, but highly discussed, topic, namely climate change. Given the widely known topic, we could reasonably expect everyone of our subjects to hold a well-formed belief, and, with ∼90 subjects per treatment, we had a reasonable ex-ante expectation that the distribution of truly held beliefs would be roughly similar across treatments.

For each of the two questions, subjects could enter their belief using the interface they had been assigned to, over 31 bins, ranging from 60°F to 120°F, each bin representing a step of 2°F.

The target day was 14 days in the future at the moment of the sessions, and hence the temperature forecast for that day was unknown both to us and our subjects. But reliable, long-term weather data exist that place the target, expected maximum temperature for July 4 th in New York City to be of 83.6°F, with a 95% confidence interval of [82.3,84.9]°F. Actual maximum temperature on the 4 th of July 2022 in New York City (Central Park) turned out to be higher, at 85°F.8

The questions were not incentivized. This resulted in some subjects slacking -i.e., as detailed above, not interacting with the screen in any way or in a very limited way. We identify slackers as having moved the interface by less than 0.05 percentage points overall. Confirming the frustration generated by the Text interface, only 1 to 3% of subjects slacked in the Click-and-Drag and Slider treatments, but 7.6% (for the 2022 question) and 13% (for 2024) of subjects slacked in the Text interface. In the following, we report results excluding slackers.

The mean and 95% confidence interval of the elicited distributions, by interface, are detailed in Table 6. The mean shape of these distributions -i.e., the mean probability allotted to each bin in a treatment -and its 95% confidence interval are plotted in Figure 4.

Slider

Text Distribution 4th July 2022 87. 69 [76.15,99.23] 84.96 [57.61,112.3] 85.83 [53.84,117.82] 89.66 [72.81,106.52] 4th July 2042 89.92 [78.73,101.11] 87.07 [58.55,115.6] 88.91 [52.43,125.4] 91.69 [73.15,110.23] All beliefs overshoot the correct historic mean. Nonetheless, the temperature has been going up in recent years. While the overall secular average maximum temperature over 122 years is of 83.6°F, it is of 84.2°F since 2000. Overall hence, perceived temperatures are above real temperatures, even accounting for recent warming. Expected warming over 20 years is on average 2.3°F.

There is very little difference across elicitation interfaces in means. While point estimates do vary, aggregate beliefs do not significantly vary by interface, as confidence intervals are quite large. The aggregate beliefs elicited using the Click-and-Drag interface nonetheless show smaller variance, and have a smoother shape with respect to those entered using any other interface. Overall, the interface, as one would hope, seem not to radically affect the given beliefs in any particular way, if not for generating a smoother distribution.

Conclusion

We introduce Cick-and-Drag, a new belief elicitation interface, and test its performance against three other interfaces used in belief elicitation in the experimental literature or by a crowd-prediction website. We find considerable variance in performance of different interfaces across different task characteristics, such as time given, number of bins and shape of a target distribution. The elicitation interface appears to clearly matter.

We find that our Click-and-Drag interface overall outperforms the other three tested elicitation interfaces, and is the least impacted by changes in shape, number of bins, and time allotted to the task. Participants were able to closer mimic a target distribution within the allotted time of either 15 or 45 seconds. Especially in the case when the target distribution was one consisting of more than a few bins, Click-and-Drag very clearly outperformed both the Slider and Text input interfaces. The third competing interface, a Distribution 'shifting and stretching' interface, inspired by the interface of metaculus.com, tended to perform a little better (however overall still worse), but suffered from being found unintuitive and frustrating according to a participant questionnaire. Moreover, the Distribution interface generated the highest drop-out rate of all interfaces, being arguably harder to manipulate.

Our experiment, including the detailed data on participant performance over time, gives us a good insight into how people react to the different elicitation interfaces. One element to be possibly explored for future research is how changing the starting distribution affects the performance of the Click-and-Drag interface, as we saw that the interface where we provided participants with a ready made normal distribution, performed quite well under certain settings.

In the end our results suggest that researchers who plan to elicit belief distributions as part of their experiment, will likely benefit from adopting the Click-and-Drag interface. Moreover with an improved interface to elicit entire distributions, needing less time and feeling more intuitive, the choice to elicit a distribution, rather than just a mean or a mode, might become more attractive for both researchers and principals requiring a forecast.

We believe that the very existence of this test of the Click-and-drag interface can contribute to its adoption and robustness on top of the fact that the interface did indeed come ahead of the others. We know from hard data what we can reasonably expect of this interface, and this should make it easier for fellow researchers -or, indeed for the crowd-prediction industry -to safely adopt it for their belief elicitation studies.

Figure 1 :

 1 Figure 1: Screenshot of the Click-and-drag interface in action

Figure 2 :

 2 Figure 2: A screenshot of the main mimic-the-distribution task, for the Click-and-Drag interface

Figure 3 :

 3 Figure 3: Performance dynamics by interface

Figure 4 :

 4 Figure 4: Distirbution of elicited beliefs on maximum temperature on the 4 th of July 2022 and 2042 in New York City, by treatment. Mean beliefs in color; true mean in black; 95% confidence interval shaded.

Figure

  Figure A.5: Performance dynamics by interface -for different target shapes

Figure

  Figure A.6: Performance dynamics by interface -for different number of bins

Table 1 :

 1 N % female mean age (sd) mean payoff (sd) % no error in CQ Mechanical Turk Sample: demographics and final payoffs, by treatment

	Click-and-drag 95	41%	36.73 (9.69)	2.92 (0.75)	43%
	Slider	91	42%	40.56 (10.93)	2.35 (0.7)	49%
	Text	91	48%	37.07 (10.94)	2.17 (0.89)	46%
	Distribution	95	37%	37.11 (11.17)	2.23 (0.43)	37%

Table 2 :

 2 Final accuracy in percentage points, mean and 95% confidence interval, by condition for all interfaces

Table 4 :

 4 Mean number of screens with no interactions, and distribution of slackers types by treatment

Table

  Table 2 limited to the subjects slacking on up to 3 screens out of the 24.

		Click-and-drag	Slider	Text	Distribution
		(N = 90)	(N = 83)	(N = 47)	(N = 2)
	Overall
		61.81 [60.72,62.9]	50.7 [49.35,52.05]	52.71 [50.7,54.72]	67.12 [59.41,74.83]
	by time constraint
	45 seconds 67.19 [65.72,68.66] 61.76 [59.83,63.69] 63.95 [61.08,66.82] 69.62 [57.96,81.28]
	15 seconds	56.3 [54.76,57.84]	39.69 [38.08,41.3] 41.19 [38.75,43.63]	64.62 [53.1,76.14]
	by number of bins
	7 bins	70.09 [68.21,71.97] 72.19 [70.24,74.14]	74.3 [71.4,77.2]	70.62 [55.57,85.67]
	15 bins	62.64 [60.97,64.31] 54.06 [52.07,56.05] 54.7 [51.71,57.69]	62.9 [49.94,75.86]
	30 bins

Table 5 :

 5 Final performance, mean and 95% confidence interval, for subjects with limited slacking

Table 6 :

 6 Mean and 95% conf.int. of the elicited beliefs -maximum temperature on the 4 th of july

This is a click for all interfaces but Text, for which it is the moment a subject leaves one text field for the next one.

We pre-registered an additional hypothesis about learning, positing that Click-and-drag would yield the best increase in performance from the first to the last time subjects saw a similar screen, an hypothesis to be tested via diff-in-diff. Unfortunately, this hypothesis was based on a previous iteration of the experimental design, based on 36 screens, where subjects would face the same screen twice, at the beginning and at the end of the experiment. For reasons of time and budget, we scrapped these screens, but we failed to update the pre-registration; we simply do not have the data to test this hypothesis, since each screen is different in one or more dimensions, and all comparisons of performance in the first and last screens would be confounded by the change in screen type (shape, time constraint, number of bins) and would not identify a learning effect.

We further pre-registered a robustness check by input interface (mouse vs. touchpad vs. touchscreen) but the share of subjects using other input devices than the standard mouse + keyboard is so low that such an analysis is not warranted. See Appendix D for details.

All interfaces start with an "empty" distribution (i.e. all bins set at zero), but the Distribution interface, that, being a discretized bell-shaped curve, cannot start at zero but starts from a symmetric, normal-like shape. The Distribution interface starts with a huge advantage, by design, and the time-constraint and dynamic adjustment path results are biased in its favor.

Average temperature was computed using the R rnoaa package, that allows to download accurate temperature data from the NOAA database; we downloaded the daily maximum temperature in Central Park, NYC, for all July 4 th starting from July 1900 up to July 2021.

Acknowledgments

Funding for this article was provided within the Priority Research Program FAST "Facilitate public Action to exit from peSTicides" financed by the French Agency for Research (ANR).

The experiemnt could technically be run by any browser, on any device. Nonetheless, for practical reasons linked to screen size, we required all participants to run the experiment on a PC -as opposed to tablet or phone. Still, subjects could do so using a variety of input devices -mouse, touchpad, keybard, touchscreen -and in principle the performance of the different interfaces is not orthogonal to the interface used to perform the task. Without a keyboard, the Text interface is unusable. With a touchpad (as opposed to a mouse), the Click-and-drag and Slider interfaces are harder to use. Not surprisingly since 100% of the sample sat in front of a PC, most subjects used a mouse and a keyboard (when needed). The share of subjects that used also a touchpad or a touchscreen is low; but the share that used just those is so small (9 subjects using only a touchpad, 1 subject using only a touchscreen) as not to warrant a robustness analysis by type of input interface.

Appendix E. Experimental instructions

Common instructions

Welcome to this research project! In this study, you will have the opportunity to earn money by working on a number of tasks.

Procedures and Participation. This study takes approximately 20 minutes and participation is voluntary. Please complete the task until the end. If you drop out of the task before finishing it you will get no reward. You are only allowed to participate in this study once.

Confidentiality. Information collected in this study is for academic purposes only and will be kept strictly anonymous.

Payment. If you complete this study, you will receive $0.50 for your participation (HIT reward). According to your performance, you may earn additional money (bonus) during the study. The number of points for your bonus depends on how carefully you solve the tasks you will be asked to work on. You will be credited your Total reward (HIT reward + bonus reward) shortly after the completion of this study.

Match the graph Task. Read these instructions carefully. Your payment will depend on it. Moreover, You will later face control questions, and you will get no reward if you do not answer them correctly.

You will see in the top-right corner of your screen a small figure with a bar-graph in it. Your task is to recreate the same bar-graph, but then in a larger frame in the middle of the screen. You will receive money depending on how close the shape of your created bar-graph is to the target shown in the top-right corner of the screen.

How close your graphs is to the target picture is measured in percentage points, where you can attain a maximum score of 100%. Each screen is worth $0.20. This means that if you reach 100%, you earn 20 USD cents; if 50%, 10 cents; if 10%, 2 cents. Each 5% increase in your score is worth 1 cent.

You will have to complete 24 graph-matching tasks. Your bonus is given by the sum of the amounts earned in each of the 24 tasks. A perfect score results in a bonus of 4.8 USD.

In the next screen, you can try the interface to familiarize yourself with the task. You can test the interface as you wish without affecting your score. Try to get the best score during the 90 seconds.

Playground: familiarize yourself with the task (no bonus) Click-and-Drag You can adjust the bar-graph by adding, moving or removing anchor-points: You can add anchor-points by clicking anywhere on the graph creates. You can move anchor-points around by dragging them. You can remove anchor-points by clicking on them.

Slider

You can adjust the bar-graph by dragging each bar up or down. Click on the top of the bar to drag it.

Text

You can adjust the bar-graph by entering a numerical bar height for each bar in the respective text field below the horizontal axis.

Distribution

You can adjust the bar-graph by adjusting the position of the horizontal slider buttons below the graph. You can add additional sliders to fine-tune the bar-graph.

Appendix F. Control Questions

The four control questions were implemented as multiple choice questions. The available replies are in square brackets, in bold the correct answers. For control question 4, each answer was correct for one and only one interface, in this order: Click-and-drag, Slider, Text, Distribution.

1. How much is the fixed participation fee (HIT reward)? [0.10$; 0.20$; 0.50$; 1.00$] 2. How much (bonus) can you earn on each of the 24 screens? [0.10$; 0.20$; 0.50$; 1.00$] 3. The more your bar-graph matches the target picture, the lower your score.