Lucas Desport

Carlos Andrade

Sandrine Selosse

Net-zero emission opportunities for the Iron and Steel industry at a global scale

Keywords: Negative emission technologies, bioenergy, CCS, energy modelling, steel industry

We use a bottom-up prospective model to explore how the iron and steel industry can decarbonize its activity by combining bioenergy with carbon capture and storage technologies.

INTRODUCTION

According to the latest IPCC Assessment Report, achieving the objective of limiting temperature increase to 1.5°C will require the massive roll-out of solutions to reduce CO2 emissions and remove CO2 from the atmosphere. For the industry sector, which represented around 25% of global CO2 emissions in 2018 (IPCC 2021), the decarbonization pathway is particularly complex. The improvement in energy efficiency might be overcompensated by the increase in production and emissions would further increase. This is especially the case for the iron and steel industry (ISI), which is responsible for 7% of global emissions in 2019 (IEA 2020a). Notably, steel is a very important product for the energy transition because most low-carbon technologies depend on it. The decarbonization pathway for this sector becomes more challenging as part of its CO2 emissions coming from production processes are inevitable (Suopajärvi et al. 2017).

The current options to produce steel are very polluting. Around 70% of the world's steel production is based on the blast furnace-basic oxygen furnace (BF-BOF) technology that relies heavily on the use of coke for iron reduction (World Steel Association 2020). Coke production is a high CO2 emitting process but is also vital for iron production through the BF-BOF route because it shows the most suitable characteristics to produce highquality iron. Hence, it is difficult to replace it with other materials (Yang, Meerman, and Faaij 2021). Many efforts have been made to reduce energy consumption and emissions in the BF-BOF route, however further reductions within the current technologies are hard to achieve as they are really mature (Remus et al. 2013). The remaining steel production comes from the electric arc furnace route (EAF) based on steel scrap (23%) and from the direct reduction of iron coupled with an EAF (DRI-EAF) (7%). The production based on the EAF using steel scrap replaces most of the use of coal by electricity which significantly reduces emissions. However, steel production cannot fully rely on steel scrap as its availability cannot cover the increasing steel demand, and because some of the steel scrap does not present the required characteristics to produce high-quality steel end-products. On the other hand, DRI-EAF uses natural gas as the main iron reducing agent, producing up to 60% fewer emissions compared to the BF-BOF route. This might be a good alternative to produce less polluting steel in regions having access to natural gas. However, it is not feasible that DRI-EAF will completely replace BF-BOF steelmaking, as there are some locations where BF-BOF is clearly the less costly route (MIDREX 2018). New steel producing technologies shifting to the direct use of coal (HISARNA or COREX) or natural gas (ULCORED) could allow the reduction of emissions however they might not be commercially available before 2030. There is also the possibility to shift the use of natural gas to hydrogen decreasing even further the emissions. This option requires a very low price for electricity to keep the competitiveness of the industry. Complete neutral carbon steel producing technologies (ULCOWIN, ULCOLYSIS) relying on the electrolysis of iron ore to produce steel might be available by the middle of the century. The transition to new steel producing technologies would be affected by economic aspects as they are more expensive. In this sense, the use of the BF-BOF route might still play an important role in the production of steel in the future, which requires additional efforts to reduce emissions. Subsequently, to further reduce CO2 emissions in this sector, it is possible to integrate carbon capture and storage (CCS) and/or utilization (CCU) technologies into the different steel production routes. Another option consists of replacing part of the fossil fuels with biomass products. Charcoal can replace some of the coke used in the BF, nevertheless, complete replacement of coke is not possible because charcoal does not feature the same physical properties as coke. On the other hand, most of the use of coal can be replaced by charcoal, and biomethane can completely replace natural gas [START_REF] Mousa | Biomass Applications in Iron and Steel Industry: An Overview of Challenges and Opportunities[END_REF]. Finally, the CO2 captured can be utilized by mineralizing steel slags, a by-product of BF-BOF. Thus, options appear very promising for decarbonizing the ISI, although the study of these options combined together, and on a global scale, has received little attention, although the use of bioenergy with CCS or CCU (resp. BECCS and BECCU) may offer negative emissions (NE). Indeed, as biomass is considered carbon neutral, by capturing and storing CO2, the latter can be subtracted from the atmosphere (they are thus commonly referred to as Negative Emission Technologies (NETs)).

In this sense, the objective of this work is to analyze the role of NETs in decarbonizing the ISI. To what extent could NETs contribute to this target? What would be the most cost-efficient technologies? How do NETs interact with other decarbonization options available for this sector? Depending on biomass potentials, which regions of the world are the most likely to rely on NETs? These are the questions we will investigate in this paper.

METHODOLOGY

The TIAM-FR model

This analysis is carried out with TIAM-FR, the French version of the TIMES Integrated Assessment Model (TIAM). TIAM is the global version of the TIMES family models developed under the Energy technology System Analysis Program (ETSAP). TIMES is a generator of partial equilibrium techno-economic models representing the energy system of geographical area -or regions, on a long-term horizon. Thus, TIAM-FR is a bottom-up model describing the world energy system disaggregated into 15 regions. For each of them, the model depicts year-byyear the energy system with a detailed description of different energy forms, technologies, and end-uses, constituting the Reference Energy System (RES) (Figure 1). TIAM-FR allows evaluating and discussing the different perspectives of energy systems evolution with respect to the envisioned objectives and pathways.

The structure of the model enables us to consider variations across the 15 regions regarding their socioeconomic properties (cost of capital, labor, and energy), energy demand projections and their commercial routes. Several thousand existing and alternative technologies described by their techno-economic parameters are connected into this RES for all sectors of the energy system (industry, commercial, residential, agriculture, transport). Technologies are also characterized by the energy carriers and materials they consume, the energy services they provide, and the GHG they emit. Driven by end-use demands, the model aims to supply energy services at a minimum discounted cost by choosing the most strategic investments to operate the energy system, dealing with several environmental and technical constraints. Besides, the model is equipped with a climate module allowing accounting for every GHG emitted by the energy system and calculates the impact on temperature elevation in the atmosphere. This type of modelling offers the opportunity for each region to explore the possible energy pathways in the long-term through different scenarios, i.e., consistent assumptions on the trajectories of the energy system. The modelling of the ISI includes the different decarbonization options presented previously and developed in the following paragraphs.

Modelling of iron and steel technologies and potential biomass use

The Table 2 presents a summary of the different potentials (found in the literature) to substitute fossil fuels with bioproducts for the different iron and steel producing routes. In general, charcoal can substitute only a small share of the use of coke as it does not present the same strength and porosity. On the other hand, charcoal and biomethane are perfect substitutes to coal and natural gas respectively. Raw biomass cannot be used directly in any of these processes as it presents high moisture content. It is also important to notice that biogas or syngas produced directly from anaerobic digestion and gasification cannot be used directly in the ISI as they do not present the same chemical composition as natural gas, so purification and upgrading are required beforehand. The model can freely choose the amount of bioproducts (any combination between 0% and the maximum substitution potential) that can replace fossil fuels for each technology and in any period from 2030 to 2100. Before 2030, charcoal can be consumed in Brazil as around 20% of its steel production is based on this commodity (SINDIFER 2020), and in Norway that uses some charcoal in the steel industry. The use of bioproducts in the rest of the regions is made possible starting from 2030. The harvesting potentials of the different bioproducts (wood, agriculture residues, organic waste, etc.) are taken from [START_REF] Kang | La Place de La Bioénergie Dans Un Monde Sobre En Carbone : Analyse Prospective et Développement de La Filière Biomasse Dans Le Modèle TIAM-FR[END_REF].

Scenarios

The analysis of the role of NETs in decarbonizing the ISI will be carried out through four different scenarios.

The first run consists of a reference scenario (REF), without any specific decarbonization plans targeted. This allows having an initial vision on the role of the different steel assets to satisfy steel demand, and to capture the efforts needed to reduce emissions in the future. Also, this scenario enables us to identify whether biomass products can be developed in the absence of specific policies favoring its use.

The next scenarios consist of limiting the atmospheric temperature increase to 2°C and 1.5°C by 2100 (respectively entitled 2C and PA, in reference to the Paris Agreement). Solving these scenarios, the model might require massively deploying alternative technologies in all sectors including the ISI, so it is possible to analyze the contribution and roles of different decarbonization options (CCS, CCU, NETs). As these scenarios constrain all sectors of the economy, the model is free to maintain a certain level of emissions in the ISI which might be eventually offset by negative emissions generated in other sectors (e.g. power sector, DAC), as long as this paradigm is the most cost-effective. The underlying assumption behind this paradigm is that economic sectors could buy negative emissions.

The final scenario (IS0) forces the iron and steel industry to achieve carbon neutrality by 2050 in a world where temperature increase is limited to 1.5°C by 2100. Through this ambitious target, it will be possible to analyze more deeply the potential contribution of NETs in ISI, as the model has to compensate the residual CO2 emissions released by fossil-based processes or by the residual emissions of carbon capture assets.

Through the analysis of these different scenarios, it will be assessed how the different decarbonization options would interact with each other and with the rest of the energy system in order to reach the proposed decarbonization objectives, and how NETs could further contribute to face the current climate challenge.

All scenarios are consistent with an SSP1, based on a recent post-COP26 study (Climate Resource 2021), projecting the demand for commercial steel to be multiplied by 2.5 by 2100.

Results

To capture the challenges underlying the decarbonization of the ISI at the global scale, we first analyze the emissions of CO2 in each scenario until 2080 (Figure 2). In the REF scenario, we first notice that the emissions of CO2 are steadily increasing and are multiplied by more than 3, which denotes the huge efforts to be accomplished by this industry to become carbon neutral in 2050. The emissions of CO2 from ISI represent roughly 5% of the cumulative CO2 emitted over the century, which guides the world towards 2,8°C temperature elevation by 2100. To lower global warming either to 2°C or 1°5C, the efforts engaged by the ISI can be appreciated with the figure below. Comparing the PA and ISI0 scenarios reveals that it is more cost-effective to delay the carbon neutrality of ISI to 2060 and further compensate the CO2 emitted before, rather than investing massively and rapidly between 2040 and 2050 to become carbon negative.

In terms of technology, these ambitions are achieved mainly thanks to CCS and hydrogen processes, as Error! Reference source not found. shows. Notably, even in a REF scenario, the DRI process becomes cost-competitive and penetrates the world steel production mix significantly, with a cost of hydrogen of roughly 1.2$/tH2 starting from 2040. In the other scenarios, the industry heavily deploys carbon capture units, that avoid CO2 emissions from BF and Hisarna processes either to store it or enhance it into gaseous and liquid fuels. From 2050 onwards, roughly 2Gt (resp. 3,5 Gt) of CO2 are captured from the ISI in a 2C scenario (resp. PA and IS0).

In the pivotal period of 2040, one can appreciate the huge technological efforts required for the ISI to become carbon neutral, which replaces and equips almost all BF processes with BF-CCS processes in the IS0 scenario. The transition is more progressive in the PA scenario, in which the industry prefers to delay the roll-out of some CCS assets but will generate more negative emissions (NE) than the IS0 scenario in the last decades. These NE are achieved by combining bioenergy from charcoal and CCS. In a 2C scenario, the ISI starts using charcoal as a substitute to coal and coke from 2030 with shares below 10%, but in the more constrained scenarios (PA and IS0), between 40 and 50% of coal is replaced by charcoal. We notice a big difference between the PA and the IS0 in the shares of charcoal and coke used, as Figure 4 below shows, necessary to offset the CO2 emitted during the 2040-2060 period (Figure 2). Thus, there is a trade-off between achieving carbon neutrality of the ISI in 2050 (IS0) or delaying it to 2060 (PA); the first requires a massive deployment of CCS assets but a moderate charcoal use, while the other prefers a more progressive penetration of CCS but a massive use of bioenergy in the future.

Although there is no specific policy in the REF scenario biogas is used in high proportions only in India and Africa which have low or expensive access to natural gas resources but affordable biomass potentials. For those regions biogas is used as a perfect substitute to natural gas in existing assets. In the more constrained scenarios, biogas is used as a reducing agent combined with CCS to generate minor NE in MIDREX processes by less than 2% of the total amount of NE at the global level. According to the IS0 scenario, negative emissions from the ISI are generated unequally around the globe. The USA, Western Europe and Africa are the regions relying the most on charcoal by up to 85% to compensate the emissions of other regions such as Japan and Western Europe using only 40% of charcoal roughly in 2050, due to the higher cost of biomass in these regions.

Discussion

The latter statement underlines a major assumption made in this modelling that is the global ISI is united to become carbon neutral by 2050 and agrees that industries of some regions would generate more NE than they require to offset the emissions of others. This involves that a global carbon market is set up. Besides, the emissions of CH4 and N2O were not considered in achieving carbon neutrality but those would constrain even more the ISI. If all GHG were considered, we would expect an even greater role for negative emissions in the ISI.

CONCLUSION

The ISI can achieve its decarbonization by midcentury under the condition to be able to massively use charcoal and invest in CCS units, which would roughly double the production costs of steel. To reach global decarbonization objectives the cooperation of different regions is required.

ACKNOWLEDGEMENT

Figure 1 :

 1 Figure 1: Simplified representation of the energy system (RES) for each of the regions in the TIAM-FR model

  energy consumption of the base year (i.e. 2018) relies on the energy balances of the steel industry from the IEA database (IEA 2020b). The different iron and steel technologies that are developed through the modelling horizon have been represented in the TIAM-FR model with their respective energy and materials consumption based on (ETSAP n.d.; Griffin, Hammond, and Norman 2013; Keys, Hout, and Daniëls 2021; Sikström 2013). Economic parameters have been based on (ETSAP n.d.; Keys, Hout, and Daniëls 2021; Kuramochi et al. 2012; Vogl, Åhman, and Nilsson 2018; West 2020; Wörtler et al. 2013). With the different techno-economic parameters represented in the model it is possible to calculate the emissions of each steel producing route as well as the levelized cost of materials (see

Figure 2 :

 2 Figure 2: CO2 emissions of the ISI according to the scenarios

Figure 4 :

 4 Figure 4: Average shares of charcoal and coke used globally

Figure 3 :

 3 Figure 3: Finished steel production from different processes over time through the 4 scenarios

Table

Table 1 :

 1 ). The emissions and the levelized cost of CO2 avoidance are in coherence with the data presented by (Yang, Meerman, and Faaij 2021). Characteristics of the candidate technologies

	Iron producing routes	CO2 emissions (kt)	Levelized cost of steel ($/t)	CO2 avoidance cost ($/CO2 avoided)
	BF-BOF	1653	590
	BF-BOF CCS	401	694	83
	BF-BOF TGR	1861	777
	BF-BOF CCS TGR	774	852	69
	COREX	2907	665
	COREX CCS	1231	727	37
	HISARNA	1355	628
	HISARNA CCS	256	724	87
	MIDREX	785	584
	MIDREX CCS	412	615	83
	ULCORED	586	557
	ULCORED CCS	224	588	84
	ULCOWIN	289	706
	ULCOLYSIS	28	696
	DRI-H2	101	791
	DRI-H2 INT	280	742
	SCR-EAF	149	630

Process Availa bility date Fossil fuel use Bioproduct substitution Maximum substitutio n potential Reference

	Coke oven	2018	Coal	Charcoal	0%-5%	(Mousa et al. 2016)
	Pelletizat ion	2018	Coal	Charcoal	0%-100%	(Nwachukw u, Wang,
						and
	Sintering	2018	Coke	Charcoal	0%-40%	Wetterlund
						2021)
	Blast		Coke	Charcoal	0%-6%
	Furnace /		Coal	Charcoal	0%-100%
	with CCS
	(includin g the Top Gas	2018	Natur al gas	Biomethane	0%-100%	(Suopajärvi et al. 2017)
	recycling
	option)
	Direct
	Reductio					(Tanzer,
	n of Iron (MIDREX)	2018 / 2025	Natur al gas	Biomethane	0%-100%	Blok, and Ramírez
	/ with					2020)
	CCS
	COREX / with CCS HISARNA / with	2020 2030	Coal Coke Coal	Charcoal Charcoal Charcoal	0%-100% 0%-45% 0%-100%	(Norgate et al. 2012)
	CCS
	ULCORE		Coal	Charcoal	0%-100%
	D / with CCS	2030	Natur al gas	Biomethane	0%-100%	(Tanzer,
	ULCOWI N	2050	Natur al gas Coal	Biomethane Charcoal	0%-100% 0%-100%	Blok, and Ramírez 2020)
	Cupola	2018	Natur al gas	Biomethane	0%-100%
			Coal	Charcoal	0%-100%	(Yang,
	EAF	2018	Natur al gas	Biomethane	0%-100%	Meerman, and Faaij 2021)
	DRI-H2		Coal	Charcoal	0%-100%	(Tanzer,
	integrate d steel plant	2030	Natur al gas	Biomethane	0%-100%	Blok, and Ramírez 2020)
	Final					(Tanzer,
	producti on of	2018	Natur al gas	Biomethane	0%-100%	Blok, and Ramírez
	steel					2020)

Table 2 :

 2

Possible uses of biomass in the ISI in TIAM-FR

Copyright © CEN2022

https://doi.org/10.1016/j.apenergy.2021.116558. Remus, Rainer, Aguado-Monsonet Miguel A, Serge Roudier, Luis Delgado Sanc, Miguel A. Aguado-Monsonet, Serge Roudier, and Luis Delgado Sancho. 2013. Best Available Techniques (BAT) Reference Document for Iron and Steel Production (Integrated Pollution Prevention and Control). https://doi.org/10.2791/97469. Sikström, Peter. 2013. "ULCORED Direct Reduction COncept for ULCOS. A Brief Introduction," no. November. SINDIFER. 2020. "Produção de Ferro-Gusa Em Minas Gerais e No Brasil." Suopajärvi, Hannu, Antti Kemppainen, Juho Haapakangas, and Timo Fabritius. 2017. "Extensive Review of the Opportunities to Use Biomass-Based Fuels in Iron and Steelmaking Processes." Journal of Cleaner Production 148: 709-34. https://doi.org/10.1016/j.jclepro.2017.02.029. Tanzer, Samantha Eleanor, Kornelis Blok, and Andrea Ramírez. 2020. "Can Bioenergy with Carbon Capture and Storage Result in Carbon Negative Steel?" International Journal of Greenhouse Gas Control 100 (July): 103104. https://doi.org/10.1016/j.ijggc.2020.103104. Vogl, Valentin, Max Åhman, and Lars J Nilsson. 2018. "Assessment of Hydrogen Direct Reduction for Fossil-Free Steelmaking." Journal of Cleaner Production 203: 736-45. https://doi.org/10.1016/j.jclepro.2018.08.279. West, Kira. 2020. "LOW-TEMPERATURE ELECTROWINNING FOR STEELMAKING (ULCOWIN)." World Steel Association. 2020. "Steel Statistical Yearbook 2020 Extended Version," 46. Wörtler, Martin, Felix Schuler, Nicole Voigt, Torben Schimidt, Peter Dahlmann, Hans Bodo Lüngen, and Jean-Theo Ghenda. 2013. "Steel ' s Contribution to a Low-Carbon Europe 2050." Yang, F., J. C. Meerman, and A. P.C. Faaij. 2021. "Carbon Capture and Biomass in Industry: A Techno-Economic Analysis and Comparison of Negative Emission Options." Renewable and Sustainable Energy Reviews 144 (April): 111028. https://doi.org/10.1016/j.rser.2021.111028.


Julien P Chanal

Dr Philippe G Sarrazin
email: philippe.sarrazin@ujf-grenoble.fr

Running head: UPWARD COMPARISONS IN THE CLASSROOM Big-Fish-Little-Pond Effect versus Positive Effect of Upward Comparisons in the Classroom: How does one reconciliate contradictory results?

Keywords: social comparison, Big-Fish-Little-Pond effect, academic self-concept educational context, upward comparison, group effect

Studies investigating social comparison in the classroom have lead to contradictory results concerning upward comparison effects. Research demonstrates that they can lead either to enhancing student's academic performance without influencing his/her self-concept or to decreasing student's academic self-concept through Big-Fish-Little-Pond Effect (i.e., BFLPE)

 postulates. Effects of implicit or explicit social comparison on student's self-concept and performance in Physical Education classes were thus considered simultaneously in the same study. Multilevel modelling analyses results demonstrate simultaneously positive and negative effects depending on the outcome and on the explicitness or implicitness of frame of reference considered. Counterbalancing the negative effects of the class-average level demonstrated in BFLPE studies, our results clearly support for assimilation effects of the small group comparison explicitly selected by the student on both self-concept and performance.

Introduction

Social comparison processes have been pointed out to provide standards of comparison used by individuals in self-evaluations (e.g., [START_REF] Festinger | A theory of social comparison processes[END_REF] and in particular among key antecedents of academic self-concept (e.g., [START_REF] Skaalvik | Issues in research on self-concept[END_REF]. In the educational context, different frames of reference can be used by students: the whole class, a small group of peers, or a particular classmate [START_REF] Skaalvik | Internal and external frames of reference for academic self-concept[END_REF]. In the literature, effects of these different frames of reference are quite divergent. As [START_REF] Wheeler | Social comparison and self-evaluations of competence[END_REF] recently emphasized, an apparent discrepancy exists between two set of results concerning social comparison effects in the classroom. A large number of studies (e.g., [START_REF] Marsh | Big-Fish-Little-Pond Effect on Academic Self-Concept. A Cross-Cultural (26-Country) Test of the Negative Effects of Academically Selective Schools[END_REF] investigated and supported the negative effects of class-average achievement level on individual student's self-concept in the Big-Fish-Little-Pond Effect model (BFLPE, e.g., [START_REF] Marsh | The-big-fish-little-pond effect on academic self-concept[END_REF]. By contrast, other researches [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF][START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] recently showed positive effects of social comparison with one (or two) classmate(s) on student's performance. This paper aims to reconcile these two seemingly divergent results and to further understand social comparison effects of various frames of reference on self-evaluations and performance in the classroom.

The Big-Fish-Little-Pond Effect: The group as frame of reference BFLPE occurs when equally able students have lower self-concepts when they compare themselves to more able students and higher self-concepts when they compare themselves with less able students. For example, if an average-ability student is in a class of highly able students, this student's academic abilities would be lower than the average of the other students in this class, and this discrepancy would lead to academic self-concepts that are below average. Conversely, if the student is in a class of less able students, then this student's academic ability would be above the average of the other students in the class, and that difference would lead to academic self-concepts that are above average. According to the BFLPE model, academic self-concept is positively correlated with individual achievement, but negatively related to class-average achievement. These hypotheses are illustrated in bold lines in Figure 1. A considerable number of research now exists in support of these predictions (e.g., Marsh & Craven, 2002, for review). These effects have been estimated in various cultures and domains (e.g., [START_REF] Marsh | Big-Fish-Little-Pond Effect on Academic Self-Concept. A Cross-Cultural (26-Country) Test of the Negative Effects of Academically Selective Schools[END_REF] including the physical activity domain [START_REF] Chanal | Big-Fish-Little-Pond effects on gymnastics self-concept: Social comparison processes in a physical setting[END_REF].

Choices of comparison target: The individual as frame of reference

Other researchers [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF][START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] have check that choosing to compare upward with a better performing target results in improved performance in naturalistic educational contexts. For example, [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] ran a study investigating the effects of choices of comparison targets. Students were asked to name a classmate in seven disciplines with whom they compared their grades. It was revealed that the comparison target had slightly better grades than the student who had chosen him/her, indicating an upward comparison tendency. Grades obtained by the comparison target were also put in bond with student's subsequent own grades. Results demonstrated that they predicted obtaining higher grades for the subject, controlling for his/her previous grades. These results appeared for the first choice of comparison [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and were extended to the second ones [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF]. These hypotheses are illustrated by the dotted lines in Figure 1.

--------------------Insert Figure 1 -----------------------How can one explain of divergent effects of upward comparison on self-evaluation? BFLPE studies have clearly demonstrated that upward social comparison induced negative self-evaluations (e.g., [START_REF] Marsh | Big-Fish-Little-Pond Effect on Academic Self-Concept. A Cross-Cultural (26-Country) Test of the Negative Effects of Academically Selective Schools[END_REF]. The central interest of [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] was concerning the student's performance (i.e., grades), however they did not completely ignore self-perceptions. They measured a similar variable called comparative evaluation which corresponds to the evaluation of one's ability compared to others. They found that this variable was not predicted by the grades of the comparison target but only by participants' own grades. In other words, comparing upward did not lower comparative evaluation. As pointed out by [START_REF] Wheeler | Social comparison and self-evaluations of competence[END_REF], "there is a potential problem here because upward comparison should lead logically to lower comparative evaluation. If individuals are comparing with people better than themselves, they should be less likely to claim that they are better than their peers" (p. 572). Upward comparison seems to lead to lower self-concepts when the frame of reference is the class (i.e., BFLPE) and to higher grades without influencing student's self-concept when the frame of reference is a target choice. How can we explain why upward comparison did not lead to the same negative effects on student's self-concept and produce positive effects on student's grades?

One of the causes of conflicting results could be found in the different methodologies used across studies. [START_REF] Stapel | Method matters: Effects of explicit versus implicit social comparisons on activation, behavior and self-views[END_REF] demonstrated that "method matters" when social comparison processes are investigated. Specifically, these authors carried out five studies to demonstrate effects of social comparison on activation, behaviour and self-views differ in function with the implicit or explicit nature of the social comparison. The protocol of this five-study design included presenting implicit and explicit social comparisons to the subjects before they assessed a similarity focus test (study 1), working on a lexical decision task (Study 2), and answering a general knowledge test (Study 3). An explicit comparison is said to occur "when a person is provided with a comparison target and a comparison is explicitly asked for" whereas an implicit comparison appears "when relevant comparison others are identified immediately in the absence of an explicit directive or force to compare" (Stapel & Suls, 2004, p. 861). In the explicit comparison conditions, participants were then asked to compare themselves (e.g., "Are you more or less intelligent than XX") with specific others that have been presented to them (Britney Spears, Marie Curie, or Pamela Anderson Lee) whereas in the implicit comparison conditions this direct comparison induction was absent.

Results demonstrated that implicit comparisons result in contrast (i.e., a negative effect) and explicit comparisons produce assimilation (i.e., a positive effect) whether it was for behaviour or self-views. This distinction between social comparison processes might explain why previous results are contradictory. According to [START_REF] Stapel | Method matters: Effects of explicit versus implicit social comparisons on activation, behavior and self-views[END_REF], asking students to choose a comparison target might have activated trait priming and similarity focus with the selected target and so on exerted assimilation effect towards him/her. Thus, the students in both lowand high-ability classes can actively compare themselves with those who have slightly higher grades. By an assimilation process, this upward comparison can increase motivation or serve as a model of effective learning practices. As a result, they can improve achievement as shown by [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF]. At the same time, students in highability classes are also involuntarily exposed to higher performing classmates. By a contrast process, they may suffer a decline in academic self-concept [START_REF] Wheeler | Social comparison and self-evaluations of competence[END_REF].

The present investigation

The goal of the present study was to consider both of these paradigms simultaneously to further understand social comparison processes in the classroom. As underlined by Skaalvik and Skaavik (2002): "To provide a better understanding of how social comparisons affect students' self-concept, researchers need to consider both group and individual comparisons" (p. 237) as both are indispensable to adequately capture the complexity of social comparisons.

To our knowledge, only one study has specifically treated these two sources of social comparison [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF][START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF]. Specifically, this group of researchers collaborated in the reanalysis of the studies of [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF]. Using multilevel modelling, preliminary analyses replicated some original results: T1 grades, comparative evaluation and target comparison level were significant predictors of T2 grades. More interesting, when comparative evaluation was used as a dependent variable, results showed a negative and large class-average ability effect, controlling for T1grades. Moreover, a small positive effect of the target comparison level was found, but only in the reanalysis of [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] data. Whereas the effects of class-average ability on selfevaluation were consistently negative in both studies, the effects of choosing a more able comparison target on self-evaluation were never negative and even were sometimes positive.

The first objective of the present study was thus to replicate findings from [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] in the Physical Education (PE) domain and to respond to some limitations of this previous reanalysing study. [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] underlined, impacts of social comparison effects on student's self-concept were not the original intention of [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] studies. Therefore, instead of using a self-concept scale with strong psychometric properties, [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] used a single item variable named comparative evaluation. Potential problems and inconsistent results described above concerning effects of target comparison level on self-evaluation might be due to the use of this scale. We therefore wanted to replicate these results using an appropriate self-concept measure.

The second objective of the present study was to test various implicit and explicit social comparisons simultaneously on student's self-concept and performance. Specifically, we tested the effect of three frames of reference -an explicit target of comparison, an explicit small group of comparison and an implicit group of comparison (i.e., the class) -to further explain the discrepancy. In accordance with [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] and [START_REF] Wheeler | Social comparison and self-evaluations of competence[END_REF], we hypothesised that upward comparisons can simultaneously contribute to improved performance as shown in [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] studies, and lower self-evaluations as predicted by BFLPE studies. More precisely, following [START_REF] Stapel | Method matters: Effects of explicit versus implicit social comparisons on activation, behavior and self-views[END_REF] we presumed that (1) explicit social comparison with a selected target but also with a selected small group would lead to assimilation effect on both student's self-concept and performance. At the same time, we expected (2) that implicit social comparison with all the classmates would lead to contrast (i.e., negative BFLPE on self-concept) on student's selfconcept as previous studies on BFLPE demonstrated (e.g., [START_REF] Marsh | Big-Fish-Little-Pond Effect on Academic Self-Concept. A Cross-Cultural (26-Country) Test of the Negative Effects of Academically Selective Schools[END_REF]. Models tested in this study are summarized in Figure 2.

--------------------Insert Figure 2 ---------------------- Participants completed a questionnaire assessing their self-concept in PE, sex, age and their targets of comparison. This questionnaire was administered in the classroom to all students present on that day. The anonymity of the answers was assured. It was indicated to the participants that there were no good or bad answers to the questionnaire and that each participant must answer individually and honestly. At the same time, a teacher and two trainees in PE per class assessed the level of each student. The teacher and trainees were different for each class. After data collection, grades of the second trimester were recovered in order to be used as a dependent variable.

Measures

The PE self-concept was measured with 6 items (e.g., "I am at ease in the majority of the physical and sporting activities") assessing the Sports Competence factor resulting from the French version of the Physical Self-Description Questionnaire (Guerin, [START_REF] Guérin | Generalizability of the PSDQ and its relationships to physical fitness: The European French connection[END_REF]. The answers were related to a Likert-scale in 6 points with markers at the ends going from 1 (False) to 6 (True). In the present investigation, there is support for construct validity of responses to this scale in that responses were internally consistent (α = .91). The mean of the items was calculated and used as an indication of students' PE self-concepts.

Targets of comparison. As in previous studies [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF][START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF], the participants were asked to nominate with whom they typically compared in the class. Instead of nominating one or two students, a table enabled them to indicate the names of up to five classmates with no number being imposed. A box left the possibility of answering "nobody". In order to allot to each student his/her trimester grade, we found the names and PE achievement of each pupil based on their birthdates, their sex and their class. However, this information could not be collected when the birthdates were erroneous or when the sex did not correspond to the data collected. Thus, grades and PE achievement could only be found for 357 participants. The small group comparison level was then obtained by averaging the individual PE achievement of students nominated when they were available.

Class-average and modified PE achievement

As [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] underlined : "Without special instruction, intervention, or moderation, teachers tend to "grade on a curve" such that there is not much variation between classes in terms of the level assigned even when there are substantial differences between classes in terms of the ability levels of students within classes" (p. 11). This observation raises a problem for purposes of evaluating the BFLPE, because it is critical to have a class-average level that reflects the differing ability levels of the classes. Unfortunately, we couldn't select schools involved in our study so achievement levels across the schools were not comparable.

To overcome this difficulty, we followed the procedure used by [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] 1 .

Information relative to schools from which our classes came from allowed us to categorize them (relative to scores they obtained in a national standardized exam at the end of the junior high school) in 3 different levels: low (below the mean level of our sample) medium (between the mean regional level and the mean level of our sample) and high (above the regional level).

We used these categorizations to rescale the average ability levels on a 0-to-3 scale, with low = 0, medium = 1.5 and high = 3. This constant value was then added to the (within-class) standardized PE achievement level of each student, depending on the student's class. These values were then standardized so that the grand mean across all students was zero. Following these recommendations we standardized (z-scoring) all variables to have M = 0, SD = 1 across the entire sample (see also [START_REF] Marsh | The negative effects of school average ability on academic self-concept-An application of multilevel modelling[END_REF]. Class-average level was then determined by averaging individual student achievement in each class (but not restandardizing these scores so that individual student and class-average achievement scores were in the same metric).

Statistical analysis

Descriptive analyses concerning the number of comparison targets and the direction of social comparison were initially led similarly to [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF]. Models of multilevel multiple regressions were then built using MLWin software (Rasbah, Browne, Healy, Cameron, & Charlton, 2001). Two main dependent variables were used, student's (1) PE self-concept and (2) grades. Four models were built with student's selfconcept as the dependent variable. In model 1, gender, age, a product between those terms and the student's individual achievement were introduced. In model 2, we added class-average achievement level. Then, we added the individual comparison target achievement (Model 3) or the small group comparison target achievement (Model 4). Concerning student's grades as the outcome, four models have been built. In the first one (Model 5), gender, age, a product between those terms, student's self-concept and target comparison level were introduced. In model 6, the student's achievement level was added. Models 7 and 8 were identical to models 5 and 6 but target comparison level was replaced by small group comparison level.

Results

Descriptive analysis

Among students for whom data were available, 81 participants out of 357 did not choose anybody as a target of comparison (23 %). Approximately 70 % of the participants who selected at least one individual with whom to compare (i.e., 192 out of 276) chose more than two comparison targets. The number of comparison targets is higher than 2 (M = 2.59) if we consider the entire sample and 3 (M = 3.36) if we consider only those who declare to compare. The choices of comparison partners are mainly students of the same sex (87%), with boys choosing more partners of the same sex (90%) that girls (85%) do.

Choice of comparison

Table 1 presents students and comparison targets means grade as well as paired t-tests.

Results plead in favour of an upward comparison tendency for the first choice (mean = 13.60 vs. students mean = 12.75, t = 3.85, p <.001), the second choice (mean = 13.35 vs. students mean = 12.83, t = 2.16, p <.05), or for the small group (mean = 13.36 vs. students mean = 12.86, t = 2.90, p <.01).

- ------------------Insert Table 1---------------Links between various frames of reference and student's PE self-concept Table 2 presents the results of analyses intended to test the links between frames of reference and student's PE self-concept. Model 1 showed that student's gender and student's achievement (β = .56, and β = .66, p <.05, respectively) were positively related to student's PE self-concept. Boys have a higher self-concept than girls and better achievement leads to better self-perceptions in PE. Model 2 demonstrated BFLPE, class-average level was negatively related to student's PE self-concept (β = -.72, p <.05) whereas gender and individual achievement remain significant predictors (β = .52, and β = .77, p <.05, respectively) 2 . In model 3, the target comparison level appeared not statistically related to student' PE self-concept (β = .04, p >.05) whereas in model 4, the small group of comparison did (β = .19, p <.05). In these two models, class-average level still remain statistically significant and negative (β = -.79, et β = -.73, p <.05, respectively).

- ------------------Insert Table 2---------------Links between various frames of reference and student's grades Table 3 presents results of analyses intended to test the links between frames of reference and student's grades. Model 5 demonstrated that student's self-concept and target comparison level were both positively related to student's grade (β = .44, and β = .15, p <.05, respectively). However, in model 6, when the student's achievement was added, student's self-concept and target comparison level did not reach significance (β = .08, p =.18 and β = .09, p =.07, respectively). Model 7 demonstrated that student's self-concept and small group comparison level were both positively related to student's grade (β = .46, and β = .29, p <.05, respectively). Moreover, these effects remain statistically significant and positive (β = .12, and β = .15, p <.05, respectively) when student's achievement was added in model 8.

-------------------Insert Table 3---------------

Discussion

The purpose of this study was to reconcile two sets of contradictory results concerning social comparison effects of upward comparisons. Previous studies in naturalistic educational contexts have provided contradictory evidence of potential benefits on student's performance and negative effects on self-perceptions but have not been able to say if these processes were exclusive or not.

Social comparison choices

Results are in conformity with previous studies [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF][START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF]) concerning student's upward comparison tendency. Indeed, grades of the first two targets of social comparison are slightly higher than grades of students who chose them.

Analysis of the number of comparison targets seems to indicate that the choice of one or two targets of comparison is not the most frequent. Indeed, the general mean of the number of selected partners is higher than 2 if the entire sample is considered, and 3 if only those who declare having at least one target of comparison are taken. This result confirms the existence of small group comparison as posited by [START_REF] Skaalvik | Internal and external frames of reference for academic self-concept[END_REF]. The small group comparison targets seems to have slightly better grades than the student who choose them, also reflecting an upward comparison tendency.

Links between frames of reference and student's PE self-concept and grades Multilevel modelling enabled us to test simultaneously the impact of variables from individual level (i.e., the target comparison and the small group of comparison level) and from class level (i.e., the average-class level).

Links with student's PE self-concept

Results reveal important information when comparing the frames of reference under consideration concerning student's self-concept. First of all, one of the purposes of this study was to replicate BFLPE in the PE domain. Our results clearly show negative BFLPE across models 2 to 4. They replicate previous results in PE classes during a gymnastics training program [START_REF] Chanal | Big-Fish-Little-Pond effects on gymnastics self-concept: Social comparison processes in a physical setting[END_REF]. In addition, we wanted to test the influence of selected targets on an appropriate self-concept measure. Results corroborate previous findings of the reanalysis of [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF][START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] concerning the target comparison level. As model 3 showed (see Table 2), target comparison level was not related to student's PE self-concept. However, model 4 brings additional indications concerning group comparison. Results showed that PE self-concept was positively related with the small group of comparison level (i.e., the selected group) and negatively with the class-average level (i.e., the non-selected group). This result demonstrates that various frames of reference could simultaneously impact student's self-concept. It is also in accordance with [START_REF] Stapel | Method matters: Effects of explicit versus implicit social comparisons on activation, behavior and self-views[END_REF] predictions concerning effects of implicit or explicit social comparison. When the comparison with a group was explicit (i.e., the selected group of comparison), assimilation occurred whereas contrast appeared when the comparison was implicit (i.e., the class). This result needs to be replicated before drawing final conclusions but it is interesting to consider that a selected group can have consequences opposed to a non-selected group.

Nevertheless, we could have assumed that explicit comparison with an individual target might also have led to assimilation whereas in fact, it did not. Previous results showed that this result is inconsistent depending on the data set and the subject [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF][START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF]. Our point of view is that an individual frame of reference might be less common in student's self-views than the favourite group is. This could be particularly true in PE courses where small groups are frequently formed (probably more than in others academic domains).

Analyses of the number of comparison targets demonstrated that the majority of students chose more than two classmates with whom they compare themselves. Few students (less than 30) chose only one classmate to compare with. We thus were not able to build multilevel models specific to these students. Our assumption was that these students' self-concepts may have been more influenced by this single target and that an assimilation effect might have been found.

Links with student's grades

First of all, our results replicate former studies of [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] concerning independent and positive effects of upward comparison with a target comparison and student's self-concept on student's grades. The better the target comparison level and the higher the self-concept were, the higher the grades were (model 5). However, these effects disappear when individual achievement was added as in model 6. This result suggests that this effect could be an artefact of students choosing comparison targets similar to themselves. Nevertheless, if the student's self-concept effect seems to be totally confounded with the effect of student's achievement, the target comparison level still remain fairly significant (p = .07) indicating a potential additive effect of this variable. We also tested the impact of a selected group on student's grades. Results were in accordance with our hypothesis. Specifically and as predicted, model 7 showed that the small group of comparison level had a positive impact on student's grades independently of student's self-concept.

Moreover, these effects remain significant when student's achievement level was added to the model. In other words, for equally able students, those who chose to compare themselves with a better comparison group obtained better grades (i.e., assimilation effect).

Limitations Some limitations of this study have to be pointed out. BFLPE studies necessitate standardized performance measures to provide a common metric for comparing students' level from different classes or schools. Even if this was not the case in our study, we were able to scale classes in relation to schools they attended. To take the differences between schools into account, we used a macroscopic indicator (i.e., scores obtained in a national standardized exam) because PE contributes to this indicator. Nevertheless, this indicator is not the most reliable considering PE. It would have been interesting to use the regional rank of the school in this discipline.

Moreover, our study used correlational data and a cross-sectional design. Contrary to [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF] and [START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] studies, this design did not allow us to control longitudinally for previous grades when student's performance was considered. However, although we did not control these effects longitudinally, we added to our model a measure of the student's achievement level likely to replace grades obtained in the first trimester.

Conclusion

The ambition of the study was to reconcile two paradigms related to social comparison processes providing paradoxical results in natural educational context. It appeared that these processes are not exclusive but complementary. Results of this study (1) demonstrated that various frames of reference may have simultaneous effects on student's self-concept. They (2) confirmed previous studies on social comparison choices [START_REF] Blanton | When better-than-others compare upward: Choice of comparison and comparative evaluation as independent predictors of academic performance[END_REF][START_REF] Huguet | Social comparison choices in the classroom: Further evidence for students' upward comparison tendency and its beneficial impact on performance[END_REF] and BFLPE (e.g., Marsh & Haw, 2003) concerning upward social comparison tendency, positive effects of target comparison level on student's performance (i.e., grades), negative effect of class-average level and a null effect of target comparison level on student's self-concept, extending previous results in PE classes. Lastly, the results (3) bring new information related to group comparison effects whether the group is explicitly selected or not. When social comparison is explicit, a group as a frame of reference might have potential benefits for students' self-concept or performance. When social comparison is implicit, negative effects of the group (i.e., negative BFLPE) on self-concept occur. Footnotes 1. More precision concerning the procedure can be found in [START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF][START_REF] Seaton | In Search of the Big Fish: A re-analysis investigating the coexistence of the Big-Fish-Little-Pond Effect with the positive effects of upward comparisons[END_REF] 2. As suggested by an anonymous reviewer, we tested (1) if students who did not choose any comparison target differ from others students in their self-concept or grades and (2) if they were less prone to the BFLPE. Results of these additive analyses showed that these students did not differ from others and that the size of the BFLPE did not vary significantly for these students. Note. All outcome and predictor variables at the individual student level were standardized (M = 0, SD = 1). All parameter estimates are statistically significant when they differ from zero by more than two standard errors (SEs). a A positive score indicates that boys have better grades than girls.

  -Method Participants and procedure Participants included 385 students (185 boys, 185 girls and 15 not identified) from 19 classes of all levels of French high schools during the 2004-2005 school year. The average age was 13.35 years old (SD = 1.1). The study was conducted during scheduled physical education lessons, compulsory part of the French educational system. Data were collected after the staff meetings and the reception of the grade reports of the first trimester.

  PE achievement. The PE teacher and two trainees in PE carrying out a teacher-training course in the class during the year, were requested to assess each student's PE level, answering individually to the question: "according to you, what is the PE achievement level of this student?" on a scale from 1 (Very bad) to 7 (Very good). Answers were strongly consistent (α = .89) and the mean of the three indicators was calculated. Data analysis Student's grade, PE achievement and group comparison level

Figure 1 .

 1 Figure CaptionFigure 1. Postulates of a BFLPE model in the classroom are in bold, postulates of the upward comparison studies with dotted line.Figure 2. Models tested in our study. Dotted lines are those tested separately.

Figure 2 .

 2 Figure CaptionFigure 1. Postulates of a BFLPE model in the classroom are in bold, postulates of the upward comparison studies with dotted line.Figure 2. Models tested in our study. Dotted lines are those tested separately.

Table 3

 3 Multilevel models with student's PE grades as a dependent variable

		Model 5	Model 6	Model 7	Model 8
	Variables	Effect SE Effect SE Effect SE Effect SE
	Fixed effects
	Intercept	.09	.13	.15	.10	.27*	.09	.23*	.09
	Gender	-.10	.15 -.23* .11 -.41* .14 -.34* .11
	Age	.02	.10	.10	.08	.13	.08	.12	.08
	Gender × Age	-.12	.13	-.15	.10	-.17	.12	-.12	.10
	PE self-concept	.44*	.07	.08	.06	.46*	.07	.12*	.06
	Target Comparison Level	.15*	.07	.09	.05
	Small group comparison Level					.29*	.07	.15*	.06
	Student's achievement			.72*	.06			.64*	.07
	Residual variance components
	Level 2 class	.10	.06	.09	.05	.00	.00	.06	.03
	Level 1 students	.68	.08	.38	.04	.63	.07	.39	.04
	N	178		178		179		179

Note. All outcome and predictor variables at the individual student level were standardized (M = 0, SD = 1). All parameter estimates are statistically significant when they differ from zero by more than two standard errors (SEs). a A positive score indicates that boys have better self-concept than girls.


Xingdi Yuan

Tong Wang
email: tong.wang@microsoft.com

Yen-Hsiang Wang

Rania Abdelghani

Pauline Lucas

Hélène Sauzéon

Pierre-Yves Oudeyer

Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches to selecting highquality questions from a set of LLM-generated candidates. Our method works under the constraints of 1) a black-box (non-modifiable) question generation model and 2) lack of access to human-annotated references -both of which are realistic limitations for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation.

Introduction & Related Work

Large Language Models (LLMs) have recently gained tremendous popularity in the NLP community [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF][START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF]. The ever-increasing size in both models and training data renders many traditional learning methods impractical/intractable. As a result, prompt-based learning has emerged as a new paradigm tailored specifically towards leveraging the power of LLMs [START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Petroni | Language models as knowledge bases?[END_REF][START_REF] Raffel | Exploring the limits of transfer learning with a unified text-totext transformer[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF]Schick and Schütze, 2021b;[START_REF] Gao | Making pre-trained language models better few-shot learners[END_REF][START_REF] Liu | Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing[END_REF]. In the zero-shot setting (such as in this study), a data sample is first "verbalized" into an input prompt and a ground-truth response -both often in natural language forms. The prompt is then issued to a pre-trained LLM to obtain a predicted response, which can then be compared to the ground-truth for evaluation. This new technique has been successfully applied to many applications including text classification [START_REF] Yin | Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach[END_REF]Schick and Schütze, 2021a), QA [START_REF] Jiang | How can we know when language models know? on the calibration of language models for question answering[END_REF], natural language generation (Li and Liang, 2021) and NLG evaluation [START_REF] Yuan | Bartscore: Evaluating generated text as text generation[END_REF].

Despite the impressive results on popular NLP benchmarks, however, the back-end LLMs are usually pre-trained with general-domain data, leading to sub-optimal performance in new domains for prompt-based learning. There are two major challenges in successful domain adaptation. Firstly, aside from the many known issues of LLMs [START_REF] Webson | Do promptbased models really understand the meaning of their prompts[END_REF][START_REF] Min | Rethinking the role of demonstrations: What makes in-context learning work?[END_REF][START_REF] Zhao | Calibrate before use: Improving few-shot performance of language models[END_REF][START_REF] Andrew K Lampinen | Can language models learn from explanations in context?[END_REF], their sheer size and/or accessibility (e.g., served via API over the internet) makes it prohibitively expensive and impractical for domain adaptation. These limitations have inspired a recent line of work known as prompt editing/tuning [START_REF] Gao | Making pre-trained language models better few-shot learners[END_REF][START_REF] Xiang | Prefix-tuning: Optimizing continuous prompts for generation[END_REF][START_REF] Madaan | Memory-assisted prompt editing to improve gpt-3 after deployment[END_REF]. The general idea is to systematically study the correlation between prompt construction and the performance on a specific task. Prompt construction comes in a wide variety of flavours ranging from adapting real-valued prompt embeddings to the order/wording/etc. of few-shot in-context learning examples. Meanwhile, it also introduces a second challenge: prompt-tuning often relies on the availability of ground-truth labels of the data, which imposes much uncertainty in applications where labeled data are scarce.

Given the ubiquity of the aforementioned challenges, we focus our study on alleviating the constraints on both annotation availability and access to model parameters, and consequently making LLMs more accessible to be deployed and used in real-world applications. We take a mainstream NLG task, namely question generation, as a case study [START_REF] Du | Learning to ask: Neural question generation for reading comprehension[END_REF][START_REF] Yuan | Machine comprehension by text-to-text neural question generation[END_REF][START_REF] Du | Harvesting paragraph-level question-answer pairs from Wikipedia[END_REF][START_REF] Pan | Recent advances in neural question generation[END_REF][START_REF] Liu | Asking questions the human way: Scalable question-answer generation from text corpus[END_REF][START_REF] Pyatkin | Asking it all: Generating contextualized questions for any semantic role[END_REF]. In this task, a model is trained to generate a natural language question conditioned on a context and an answer, such that the generated question can be answered by the provided answer using the context as supporting evidence. Question generation is the corner stone for many NLP applications including education [START_REF] Kurdi | A systematic review of automatic question generation for educational purposes[END_REF][START_REF] Abdelghani | Conversational agents for fostering curiositydriven learning in children[END_REF], automatically FAQ generation [START_REF] Mass | Unsupervised faq retrieval with question generation and bert[END_REF], information seeking [START_REF] Qi | Stay hungry, stay focused: Generating informative and specific questions in information-seeking conversations[END_REF], etc. In an educational setting, for example, a question generation system can generate demonstrations that inspire students' curiosity and thinking (teaching), or to help assess students' proficiency on certain knowledge or skills (examining). These use cases would benefit greatly from reduced dependency on computing resources, data availability, and the required expertise for fine-tuning an LM.

To align with these real-world scenarios, our goal is to obtain better outputs from an inferenceonly LLM (i.e., as a "black-box", which is relatively more accessible, e.g., through online APIs). In particular, given the common practice of sampling multiple outputs for improved generation diversity, we propose a method that aims at selecting the best candidate based on multiple aspects of question quality in a zero-shot manner -notably without model adaptation or human annotations. Our method can be seen as a post-hoc selection process within a larger NLG pipeline, and thus is orthogonal and applicable to zero-shot and incontext learning methods [START_REF] Rubin | Learning to retrieve prompts for in-context learning[END_REF][START_REF] Lu | Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity[END_REF][START_REF] Liu | What makes good in-context examples for GPT-3?[END_REF].

Problem Setting

Notations Formally, we consider a dataset of context-answer pairs (c, a) both as strings. The task of question generation is to generate a question q that can be answered by a using c as supporting evidence. We use an off-the-shelf pretrained LLM-based question generator in a zeroshot setting (prompt construction detailed in Appendix A). To simulate the black-box generator scenario, we refrain from any form of model tuning. We do, however, assume access to a set of output sequences stochastically sampled from the question generator. We thus ground our study to this application scenario by sampling k questions Q = {q i ∶ i = 1, . . . , k}. For comparison as a baseline, we also denote q g as the question generated with a greedy algorithm (i.e., generating the most probable token at each time step).

Our goal is to devise an algorithm S which selects the best candidate q i * that maximizes some evaluation metric M ∶ Q ↦ R, i.e., S(Q) = i * = arg max i M (q i ). We use M s , M s , and M s to denote the mean, min, and max of {M (q) ∶ q ∈ Q}, resp., and M g for the greedy output M (q g ). Semantically, M s ≤M s ≤M s is tautologically true, and a positive result on the design of S would translate to M (q S(Q) ) outperforming both M s and M g .

Datasets In this work, we adopt two question generation datasets with distinctive characteristics, namely SQuAD [START_REF] Rajpurkar | SQuAD: 100,000+ questions for machine comprehension of text[END_REF] and Fairytale QA [START_REF] Xu | Fantastic questions and where to find them: Fairytaleqaan authentic dataset for narrative comprehension[END_REF]. SQuAD was originally proposed as an extractive question answering (QA) dataset. In the question generation literature [START_REF] Du | Harvesting paragraph-level question-answer pairs from Wikipedia[END_REF][START_REF] Yuan | Machine comprehension by text-to-text neural question generation[END_REF][START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF], it has been used as a sentence-level question generation task, i.e., a context c is a single sentence that contains the corresponding answer a as a substring. Fairytale QA has also been used for both question answering and question generation. It features paragraph-level question generation (with c being one or more paragraphs), and the answer a is not necessarily a sub-string of c. Since we do not perform any form of model/prompt tuning, we use the testing split for datasets, which consist of 11,877 data points for SQuAD and 1,007 for Fairytale QA.

Model We leverage a pre-trained GPT-3 model [START_REF] Brown | Language models are few-shot learners[END_REF] for both question generation and selection (detailed in §3). In all our experiments, we prompt the GPT-3 model in a 0-shot manner. Details on all our prompts are provided in Appendix A.

Evaluation Metrics

We use two quantitative methods to evaluate the selected question q ′ = M (q S(Q) ):

• Reference-based evaluation: Following prior works, we use BLEU-4 for SQuAD [START_REF] Du | Harvesting paragraph-level question-answer pairs from Wikipedia[END_REF][START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF] and ROUGE-L for Fairytale QA [START_REF] Xu | Fantastic questions and where to find them: Fairytaleqaan authentic dataset for narrative comprehension[END_REF]. These metrics compare q ′ against the reference question q (a.k.a. the "groundtruth" question in the existing literature).

• Human evaluation: we solicit human annotations on a subset of the data. We postulate that an over-all score given holistically to rate a question would be highly subjective and thus less likely to induce agreement among annotators. Accordingly,

[context] Old Dragonbeard must have been a master swordsman standing midway between those of the first and of the second order. Molo, however, of whom this story tells, was a sword hero. At that time there lived a young man named Tsui, whose father was a high official and the friend of the prince. And the father once sent his son to visit his princely friend, who was ill. The son was young, handsome and gifted. He went to carry out his father's instructions. When he entered the prince's palace, there stood three beautiful slave girls, who piled rosy peaches into a golden bowl, poured sugar over them and presented them to him. After he had eaten he took his leave, and his princely host ordered one of the slave girls, Rose-Red by name, to escort him to the gate. As they went along the young man kept looking back at her. And she smiled at him and made signs with her fingers. First she would stretch out three fingers, then she would turn her hand around three times, and finally she would point to a little mirror which she wore on her breast. When they parted she whispered to him: "Do not forget me!" [question] Who was Rose-Red? [input1] Is the [question] related to the [context]? Why? [output1] Yes, the question is related to the context. Rose-Red is one of the three beautiful slave girls who served the prince. The young man, Tsui, was attracted to her and she seemed to be attracted to him as well.

[input2] Based on the above response [output1], which one of the following [options] best described [question] wrt [input1]?

[options] 1: They are not at all related; 2: They are remotely related; 3: They are somewhat related; 4: They are closely related.

[output2] 3: They are somewhat related. we decompose the quality of a question into seven dimensions 2 , and ask human annotators to rate a question on each dimension followed by an overall rating of the question. We collect three annotations from different annotators for each data points. We provide details of the human study in Appendix B.

Method

In this section we propose three question selection methods. As described in §2, each method is used to score k sampled questions in Q and the candidate with the highest score is proposed as the final output.

n-gram similarity We use n-gram similarity between a question and its corresponding context to measure their relevance. This method reflects the intuitive assumption that favorable question be closely related to the information provided by the context. Specifically, we extract all unique ngrams 3 s n (c) from a given context c, s n (q) from a question q. The n-gram similarity score is then defined as:

sim n = |s n (c) ∩ s n (q)| |s n (q)| , (1)
where |s| indicates the size of set s.

Round-trip

Intuitively, the answer to a generated question should be semantically equivalent to the answer that has been used to generated the question. Formally, a question generation model QG and a QA model (both with reasonable performance) should satisfy the following:

q ′ = QG(c, a); a ′ = QA(c, q ′ ); a ′ = a.
(2)

2 Namely, grammatical correctness, offensiveness, clarity, relevance, importance, specificity, and answerability.

3 In all our experiments n ranges from 1 to 5.

SQuAD

Fairytale QA (BLEU-4) (ROUGE-L)

prior works (models trained/fine-tuned on these datasets) [START_REF] Du | Harvesting paragraph-level question-answer pairs from Wikipedia[END_REF] 0.152 - [START_REF] Zhang | Addressing semantic drift in question generation for semisupervised question answering[END_REF] 0.184 -UniLM Large [START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF] 0.228 -UniLM v2 Base [START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF] 0.244 -ERNIE-GEN Large [START_REF] Xiao | Ernie-gen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation[END_REF] 0.254 -BART (Xu et This idea is closely related to cycle consistency in the existing literature on image generation [START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], machine translation [START_REF] Artetxe | Unsupervised neural machine translation[END_REF], and QA [START_REF] Alberti | Synthetic QA corpora generation with roundtrip consistency[END_REF][START_REF] Shah | Cycle-consistency for robust visual question answering[END_REF]). Here, we use GPT-3 as an off-the-shelf QA model to obtain a ′ for each pair of c and q ′ , resulting in k answers A = {a ′ 1 , . . . , a ′ k } for the k sampled questions in Q. We then measure the similarity between each a ′ i and the ground-truth answer a using F 1 score for SQuAD and ROUGE-L for Fairytale QA (in accordance with the evaluation setup from the original papers for the two datasets). Finally, we select the question corresponding to the generated answer a ′ i * that overlaps the most with a (i.e., that can be best answered by GPT-3) Prompts used in these experiments are detailed in Appendix A.

Prompt-based Score

We propose a two-step procedure (Figure 1) for prompting GPT-3 to answer the same set of meta-questions (i.e., questions about the quality of a given question) used for human evaluation ( §2).

In step 1, given a context-question pair, GPT-3 is prompted to answer a meta-question as an open question (as opposed to choosing among a list of options) as well as to verbalize a reason for its answer. In step 2, GPT-3 is prompted to choose from a list of options representing the rating scale of the meta-question.

We empirically observe that without the first step, GPT-3 output tends to have a low-entropy distribution, i.e., often choosing the same option for a given meta-question disregarding the different context-question pairs. In contrast, the model appears to be better primed wrt output diversity with the additional first-step, which is inline with observations made in some existing studies [START_REF] Nye | Show your work: Scratchpads for intermediate computation with language models[END_REF][START_REF] Wei | Chain of thought prompting elicits reasoning in large language models[END_REF].

Similar to human evaluation, we also prompt GPT-3 to generate an overall score to a question. We use overall prompt-based score (OPS) to denote this GPT-3-labeled score, and averaged promptbased score (APS) to denote the average score over all individual meta-questions.

Results and Discussion

To measure the performance of a selection method ( §3), we use it to select one out of k questions stochastically sampled from GPT-3, and score the selection with the evaluation metrics outlined in §2. We set k = 5 for all our experiments. Additionally, we test the ensemble performance with multiple methods. To ensure comparability, we normalize the scores obtained from each selection method into the range between 0 and 1, and use their average score to perform question selection.

Reference-based evaluation

Reference-based evaluation are automatic metrics that are applied to the entire test sets of SQuAD and Fairytale QA. We observe in Table 1 that on both datasets, all question selection methods outperform M s , the average score over all five sam-pled questions, validating the effectiveness of the proposed methods. While all individual methods outperform the greedy generation baseline M g on SQuAD, round-trip is the best performing one, outperforming M g on both datasets. It can be further improved via ensemble with n-gram and/or promptbased scores (using uniform weights).

Note that prior studies require a large amount of labeled data for model training/fine-tuning, while GPT-3 performs zero-shot inference. Despite this major difference in learning paradigm, most GPT-3-based models proposed here outperform previous results by significant margins on the SQuAD dataset -even the least performant samples M s (lowerbound) achieve competitive results. For Fairytale QA, however, only the best samples M s (upperbound) outperform previous results [START_REF] Xu | Fantastic questions and where to find them: Fairytaleqaan authentic dataset for narrative comprehension[END_REF], indicating margins for improvement on question selection strategies for future work.

Human Evaluation

Human evaluation consists of 16, 800 annotations (collected from 87 annotators) evenly split across the two datasets (details in Appendix B). For question generation (among many language generation tasks), model outputs may exhibit linguistic diversity while maintaining semantic equivalence. It is thus highly problematic to evaluate such outputs against a single reference (i.e, "ground-truth" questions). Figure 2 empirically shows that the groundtruth (GT) questions provided in the datasets often fail to receive the highest human ratings, on many occasions scoring lower than stochastic samples from GPT-3 (M s ). Consequently, we strongly advocate for human evaluation, which we believe is higly effective in improving generalizability of our results to real-world applications.

Another prominent observation is that n-gram and APS perform quite differently on the two datasets. On SQuAD, n-gram similarity outperforms other individual methods, with further noticeable improvements via ensemble with roundtrip. APS, on the other hand, does not work nearly as well, performing the worst for almost all metaquestions. In contrast, n-gram (particularly trigram) similarity shows the worst performance on Fairytale QA, while APS outperforms all other methods by a noticeable margin.

We posit that the reversed trend in comparing n-gram and APS can be explained by the distinct natures of the datasets. For SQuAD, the sentencelevel contexts are relatively short and simple with strictly extractive answers (i.e., the answers being sub-strings of the corresponding contexts). As a result, paraphrasing the context can be a rather effective question generation strategy, hence the stronger correlation between question quality and the c-q n-gram similarity. On the other hand, with multiparagraph contexts and abstractive, open-ended answers, questions are more likely posed about abstract ideas rather than simple context paraphrasing. Consequently, n-gram similarity, which favors local context paraphrasing, can no longer serve as a good question selection strategy.

Limitations and Future Work

We acknowledge that our system has some limitations that warrants further investigation. For example, one needs to be mindful of the specific downstream applications of the proposed methods, both in terms of potentially large variance in outof-distribution performance (e.g. divergent question generation, [START_REF] Abdelghani | Conversational agents for fostering curiositydriven learning in children[END_REF]) and of mitigating harmful/toxic contents in educational applications [START_REF] Bender | On the dangers of stochastic parrots: Can language models be too big?[END_REF].

We also acknowledge the prohibitively restrictive access to the GPT-3 model at the time of writing. We do believe that this constraint will relax over time, and meanwhile, hoping that our proposal can shed light on research and applications with more accessible LLMs such as GPT-J [START_REF] Wang | [END_REF] and BLOOM (BigScience, 2022) for future work.

Conclusion

In this study, we investigate the practical problem of selecting the best output from multiple samples generated by an LLM. Using question generation as a case study, we propose two prompt-based approaches that select high-quality questions according to question quality from multiple perspectives. To alleviate real-world constraints on using large LMs such as computation resources and data availability, the proposed methods do not rely on model fine-tuning nor human annotation. Extensive experiments with both automatic and human evaluations evince the effectiveness of our approach on selecting high-quality questions from stochastic samples.

Contents in Appendices:

• In Appendix A, we report all prompt templates we used in this work.

• In Appendix B, we provide details on the human study.

• In Appendix C, we provide the full set of our experiment results.

• In Appendix D, we report implementation details.

A Prompt Designs

We report an example of our prompt for question generation in Figure 3.

We report an example of our prompt for QA (used in round-trip) in Figure 4.

We report an example of our prompt in obtaining prompt scores in Figure 1.

B Human Study

We randomly sample 50 documents from each of the two datasets SQuAD and Fairytale QA. Each document correspond to one ground-truth question and six questions generated by GPT-3 (five by stochastic sampling and one by greedy search). Each question is then rated by three human annotators wrt seven meta-questions and one over-all rating, altogether constituting 50 × 2 × (1 + 5 + 1) × 3 × (7 + 1) = 16, 800 annotations. There are in total 87 annotators involved in the annotation process, all annotators are English speakers, they are recruited from regions including Europe, the United States and United Kingdom. Each annotator on average performed 193 annotations and was paid on average $14.1 USD per hour.

We perform a basic spam filtering process on the raw annotations. We observe a 15.4% spam rate. All human scores reported in this paper are computed after spam removal.

We report the eight meta-questions we used for human annotation in Figure 5. The eight metaquestions correspond to columns in Figure 2. We collect three annotations from different annotators for every meta-question, we report the averaged human agreement rate in Table 2.

C Additional Results

In Table 3, we report the full experiment results for reference-based evaluation. 2: Averaged human agreements among three annotators. An agreement indicates that all three annotators selected the same option for a meta-question.

SQuAD

Fairytale QA (BLEU-4) (ROUGE-L)

prior works (models trained/fine-tuned on these datasets) [START_REF] Du | Harvesting paragraph-level question-answer pairs from Wikipedia[END_REF] 0.152 - [START_REF] Zhang | Addressing semantic drift in question generation for semisupervised question answering[END_REF] 0.184 -UniLM Large [START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF] 0.228 -UniLM v2 Base [START_REF] Bao | UniLMv2: Pseudo-masked language models for unified language model pre-training[END_REF] 0.244 -ERNIE-GEN Large [START_REF] Xiao | Ernie-gen: An enhanced multi-flow pre-training and fine-tuning framework for natural language generation[END_REF] 0.254 -BART (Xu et al., In Table 4, we report the full results for human evaluation on SQuAD.

In Table 5, we report the full results for human evaluation on Fairytale QA.

Story:

As soon as the lady had departed the fisher's son awoke, and the dark lad told him of her visit, and how he would never see her as long as he lived. At this the fisher's son felt the cold creeping up to his heart, yet he knew the fault had not been his that sleep had overtaken him. 'I will search the whole world through till I find her,' cried he, and the dark lad laughed as he heard him. But the fisher's son took no heed, and off he went, following the sun day after day, till his shoes were in holes and his feet were sore from the journey. Nought did he see but the birds that made their nests in the trees, not so much as a goat or a rabbit. On and on and on he went, till suddenly he came upon a little house, with a woman standing outside it. Instruction: Read the above story, ask a question and answer it. Question: GPT-3 FILLS IN THIS BLANK Answer: search the whole world through till he found her Figure 3: An example of prompting GPT-3 for question generation. We use the text before green as prompt, and text after green as suffix. We refer readers to the GPT-3 documentation for more details about GPT-3's inserting completion mode.

[Document]: is cheeks were red with passion, and his eyes were bright, for he could not but notice that, now that she was safe at Orphir under her true love's protection, the Lady Morna's manner had grown cold and distant again, and he was beginning to lose faith in Snorro's charm.

Angry and disappointed, he had sought his mother's room to pour out his story of vexation to her.

He stopped short, however, when he saw the wonderful waistcoat lying on the table, all gold and silver and shining colours. It was like a fairy garment, and its beauty took his breath away.

[Question]: Why did Harold lose faith in Snorro's charm?

[Answer]: Harold lost faith in Snorro's charm because the Lady Morna's manner had grown cold and distant again.

D Implementation Details

In all experiments, we use the text-davinci-002 (175B parameters) variant of GPT-3. It is currently the most capable GPT-3 model variant. Compared to other variants, text-davinci-002's support to inserting completions can better facilitate our question generation tasks (as shown in Figure 3).

We use a temperature of 0.7 during the sampling process of question generation. In all other use cases (e.g., QA round-trip, prompt score), we use greedy generation (temperature is set to 0). 1. Is the question gramatically correct? 1) It is grammatically incorrect 2) It has some grammatical issues 3) It is grammatically correct 2. Is the question offensive to people? 1) It is very offensive 2) It may be offensive 3) It is not at all offensive 3. Is the question clear? 1) It is not at all clear 2) It is mostly clear 3) Is is very clear 4. Is the question related to the context of the attached document? 1) It is not at all related 2) It is somewhat related 3) It is closely related 5. Is the question asking about an important aspect of the context of the attached document? 1) Not at all important 2) It may be important 3) It is very important 6. Is the question asking about a specific piece of information in the attached document? 1) The question is very generic 2) The question is somewhat generic 3) The question is very specific 7. Can the question be answered using information in the attached document? 1) No, answering the question requires completely different information 2) The question can be partially answered using information from the document 3) The question can be perfectly answered using information from the document 8. What is your overall rating of the question generated based on the attached document? 1) The question is very bad 2) The question is okay 3) The question is very good 5: Human eval results (Fairytale QA). Abbreviations in the first row denote Grammatical correctness, Offensiveness, Clarity, Relevance, Importance, Specificity, Answerability, Averaged Human Rating (over all dimensions to the left), Overall Human Rating (an overall score given by annotators). Best and second best numbers (excluding baselines) are highlighted with boldface and underline, respectively.

Figure 1 :

 1 Figure1: Prompting GPT-3 to rate a question's relevance. GPT-3 output is highlighted in green.

Figure 2 :

 2 Figure 2: Human evaluation results, averaged over three annotators' scores, normalized per column. Left: SQuAD; right: Fairytale QA. Abbreviations in x-axis denote Grammatical correctness, Offensiveness, Clarity, Relevance, Importance, Specificity, Answerability, Averaged Human Rating (over all dimensions to the left), Overall Human Rating (an overall score given by annotators). Exact scores are provided in Appendix C.

Figure 4 :

 4 Figure 4: An example of prompting GPT-3 for QA. GPT-3 output is highlighted in green.

Figure 5 :

 5 Figure 5: Meta-questions we designed for human evaluation.

Table 3

 3

	: Reference-based evaluation scores on various
	question selection methods. Best and second best num-
	bers (excluding baselines) are highlighted with bold-
	face and underline, respectively.

Table 4 :

 4 .980 0.952 0.950 0.920 0.937 0.917 0.948 0.893 tri-gram + RT 0.983 0.987 0.952 0.953 0.927 0.933 0.925 0.951 0.900 bi-gram + APS 0.972 0.983 0.943 0.932 0.903 0.922 0.867 0.932 0.857 tri-gram + APS 0.972 0.983 0.945 0.928 0.910 0.912 0.878 0.933 0.862 bi-gram + APS + RT 0.977 0.980 0.947 0.932 0.905 0.918 0.905 0.938 0.877 tri-gram + APS + RT 0.972 0.980 0.948 0.928 0.910 0.912 0.902 0.936 0.878 Human eval results(SQuAD). Abbreviations in the first row denote Grammatical correctness, Offensiveness, Clarity, Relevance, Importance, Specificity, Answerability, Averaged Human Rating (over all dimensions to the left), Overall Human Rating (an overall score given by annotators). Best and second best numbers (excluding baselines) are highlighted with boldface and underline, respectively.

		G	O	C	R	I	S	A	AHR	OHR
	GT	0.937 0.987 0.943 0.930 0.925 0.922 0.887 0.933 0.870
	M g	0.950 0.983 0.927 0.953 0.925 0.905 0.870 0.930 0.833
	M s	0.957 0.981 0.940 0.937 0.909 0.921 0.879 0.932 0.857
	bi-gram	0.968 0.990 0.952 0.953 0.918 0.937 0.885 0.943 0.880
	tri-gram	0.968 0.990 0.943 0.957 0.928 0.917 0.890 0.942 0.885
	round-trip (RT)	0.962 0.992 0.927 0.933 0.900 0.922 0.903 0.934 0.872
	APS	0.945 0.980 0.912 0.912 0.880 0.902 0.863 0.913 0.837
	APS + RT	0.947 0.977 0.928 0.915 0.890 0.898 0.903 0.923 0.875
	bi-gram + RT	0.983 0