Ola Helenius
email: ola.helenius@ncm.gu.se

Uffe Thomas Jankvist

Linda Marie Ahl

Mario Sánchez Aguilar

On the implicitness of the 'theory of change' in implementations in mathematics education research

Keywords: Implementation research, theory of change, professional development

Articles in mathematics education that report on the implementation of innovations do not always report models for how the innovation program is supposed to attain its effects. We analyze an article with no such explicit theory of change with the aim of providing a background case for discussing the role of theories of change in implementation research. Our method consists of finding hypothetical determinants, which are the retrospectively deduced and explicated mechanisms for how the program reaches its effects. For the problem-solving promoting professional development program we analyze, we find a set of hypothetical determinants that are nested in a complex way. The use of predefined problems for both the lesson model and the professional development model and the collegial development of guiding questions for teachers to use with the predefined problems seems to form the core of a hypothetical theory of change for the program.

Introduction

Within general implementation research (IR), it is often stressed that IR studies should make the 'Theory of Change' (ToC) of the innovation explicit and that this should play an active part in the final evaluation of a given implementation [START_REF] Chen | Theory-driven evaluation: Conceptual framework, application and advancement[END_REF][START_REF] Weiss | Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families[END_REF]. In IR within mathematics education, however, it has sometimes been found that the ToC is not made explicit-and furthermore that there is not always alignment between the ToC that underpinned the design of the innovation and that which was applied in the final evaluation, as exemplified by [START_REF] Jankvist | Illustrating the need for a 'Theory of Change' in implementation processes[END_REF] when examining the large-scale Swedish professional development project, the Boost of Mathematics [START_REF] Helenius | A stakeholder analysis of the development of a large scale professional development program[END_REF]. Our observation of this 'implicitness' of a ToC has been further strengthened during an ongoing literature review on IR in mathematics education research concerned with studies that self-identify as dealing with implementations by using that term in the title or abstract. Preliminary results of this review have been presented elsewhere (Ahl, Aguilar et al., 2022a[START_REF] Ahl | Professional development as a for implementing mathematics education innovations: Results from a systematic review[END_REF], 2023). In our work of summarizing the whole review [START_REF] Ahl | Implementation research in mathematics education: A systematic mapping review[END_REF], we find that in order to identify a ToC, we often have to 'work backward', meaning that we have to analyze the evaluations with respect to finding the underlying objects of concern, i.e., the implicit ToC. This led us to question: what potential consequences it might have had for the evaluation that a ToC was not explicated in the first place? In particular, successful studies, or at least studies reported by the authors as being successful, were often much less explicit in identifying factors of influence, making it harder for us to produce hypotheses on what could be central elements in the ToC of the implemented innovation reported in the study.

To contribute to the discussion on what it entails to construct a ToC, in this paper, we analyze what we find to be a critical case [START_REF] Flyvbjerg | Five misunderstandings about case-study research[END_REF] with the purpose of reconstructing a possible ToC. We see this case as critical since the selected article reports a successful implementation and does not explicitly discuss a ToC. The chosen article for analysis is both well-written and presents an implementation case that is complex enough to make it possible to illustrate phenomena that are represented in a substantial selection of articles presenting implementations in mathematics education. The case concerns the reporting of a professional development program for promoting problem-solving teaching that turned out to have positive effects on students' problem-solving skills. Before we get to the case, we first provide a bit of background on the notion of theory in mathematics education, ToC in general, and theory-driven evaluations.

Briefly on the notion of theory

The literature, including the mathematics education literature, is rich in definitions and descriptions of the notion of 'theory'. For example, [START_REF] Radford | Connecting theories in mathematics education: challenges and possibilities[END_REF] considers a theory a triplet (P, M, Q), where P is a set of principles, M is a methodology, and Q is a set of paradigmatic research questions. Also, in the context of mathematics education research, [START_REF] Niss | On the mathematical competencies framework and its potentials for combining and networking with other theories and theoretical constructs[END_REF] suggest the following "minimalist definition":

A theory is a theory of something, i.e., it deals with certain sorts of objects and phenomena and includes terms for these. Its purpose is to produce corroborated claims about these objects and phenomena, typically in response to questions posed about them. These claims are generated by some means, on some grounds, involving some fundamental methodology/ies. (p. 17) From a more general perspective, they do, however, state that in writings on the notion of 'theory', there are mainly three meanings that are prevalent, whether explicit or implicit:

• A theory can be a hypothesis that cannot be, or has not yet been, substantiated;

• A theory can be a collection of beliefs, rules or principles that are meant to guide action or behaviour;

• A theory can be a more or less connected edifice of claims intended to explain or predict phenomena occurring within some domain. (p. 17)

As can be seen, the first of these is in line with [START_REF] Mason | The role of theory in mathematics education research[END_REF], who describe a theory as a "hypothesis, or possibility such as a concept that is not yet verified but that if true would explain certain facts or phenomena" (p. 1055).

Theory of change and theory-driven evaluation

A ToC may be defined as "a collection of theoretical constructs that may be used to guide the design and implementation of a program and to evaluate its success or failure" [START_REF] Jankvist | Illustrating the need for a 'Theory of Change' in implementation processes[END_REF](Jankvist et al., , p. 1048)). These theoretical constructs may originate from both inside and outside of mathematics education research. This is to say that "a ToC may be something developed locally as part of an implementation, for example, one that relies on subject-specific theoretical constructs" (p. 1048). Hence, a ToC may often be a 'local' theory. Furthermore, a ToC may also involve descriptions of a set of assumptions that explain mini-steps leading to long-term goals and connections between program activities and outcomes occurring at each step [START_REF] Chen | Theory-driven evaluation: Conceptual framework, application and advancement[END_REF]. Hence, a ToC may often be closer to the definition of 'theory', which considers theory a kind of hypothesis rather than the minimalist definition (cf. section above).

When talking about theory-driven evaluations, the 'theory' in theory-driven may refer to the ToC, but also sometimes to a larger model encompassing the ToC, i.e., an "explicit theory or model of how the program causes the intended or observed outcomes and an evaluation that is at least partly guided by this model" (Rogers et al., 2000, p. 5)

A critical case

To illustrate the difference between reporting some given result of an implemented program and approaching the evaluation and reporting from an implementation perspective, we look at the professional development (PD) program focusing on problem-solving (PS) reported by [START_REF] Saadati | Assessing impact of a teacher professional development program on student problem-solving performance[END_REF]. The article is published in a ZDM special issue on empirical research on problemposing and problem-solving. So, while the authors use the term 'implementation' when discussing the PD-program ARPA, the authors do not use concepts or constructs from IR to frame their implementation. Note that this is a well-written article, and our decomposition of it is a result of our intention to illustrate the potential of taking an implementation approach and the efforts involved when trying to reconstruct a ToC. We are not intending to criticize the choices made by the authors when writing the article.

The PD program under scrutiny was designed as a series of nine collegial workshops and seven monthly intermediate problem-solving lessons that teachers would carry out with their classes. From the program attendants, two teachers who taught two grade five classes each were chosen. The teachers implemented the monthly problem-solving lessons in one of their classes, while the other class acted as a control. By administering a problem-solving test in a pre/post fashion and analyzing the variance, the authors showed that the experimental group increased their problem-solving skills more than the control group (statistically significant at the p<0.05 level). The authors point out that it is important to analyze the student results in this fashion (and we agree). If improved student results in the experimental group relative to the control group can be established, then the project can be considered to be successful. But, from an implementation perspective, questions remain: What is it that was successful and why?

Since Saadati and Felmer describe the organizational details of the PD project quite well, it is possible for us to extract elements of a ToC from their description. We do that as a series of if-then statements, which are the hypothetical determinants that are supposed to explicate "causal processes [that] are expected to happen to attain program goals" (Chen, 2012, p. 18).

On the surface, the program follows [START_REF] Desimone | Improving impact studies of teachers' professional development: Toward better conceptualizations and measures[END_REF] suggested change model: a PD with the elements of content focus, active learning, coherence, duration, and collective participation that leads to changes in teachers' knowledge, skills, attitudes, and beliefs, which again leads to changes in the instruction, which in turn leads to improved student learning. However, the components of the change model proposed by Desimone are not detailed enough to capture which ideas and teacher behaviors actually lead to change. Saadati and Felmer describe the design of the program in much more detail, and several related and quite intricate determinants can be explicated. First, the program involves seven PS lessons that are supposed to follow a detailed model involving work in randomized groups with problems specific to the program. Also, a core idea is that the teacher should not give hints to struggling students but instead respond to student questions with guiding questions. A hypothetical determinant is, hence, that if students attend lessons following this model, then students' PS skills will improve (D1). Since students are evaluated individually but practice problem-solving in groups, a further determinant is that if students improve their PS skills in groups, then their individual PS skills will improve (D2). We can also explicate the sub-aspect of the previous hypothetical determinants.

For example, if teachers support students with high-quality guiding questions, then students' PS skills will improve (D3), or if students practice problem-solving with well-chosen tasks, then PS skills will improve (D4). Note that D3 and D4 are (hypothetically) independent of each other and also independent of the PS model as a whole and are thus instead assumed to have independent effects on students' skill development. However, the guiding questions and specific problems are elements of the teaching model, so another hypothetical determinant involving these elements is: if teachers are prepared with guiding questions for a particular problem, then they will be able to conduct the PS lesson with fidelity (D5).

We also need to look at the role of the PD sessions. Here, one possible hypothetical determinant is if teachers participate in PD sessions according to the model, then teachers will be able to carry out the PS lessons with fidelity (D6). The PD sessions also follow a model, so the role of the expert PD session monitor might be important: if an expert monitor oversees the PS session, then it is carried out according to the model (D7). We can also look at more specific elements tying the lesson design and PD design; for example, if teachers work collegially with a problem, then they can develop good guiding questions (D8). Moreover, the role of the specifically chosen problems might not only have a role in being good for practicing problem-solving, but these may also play a role in the PD: if problems are pre-chosen for the PS lessons, then the same problems can form the basis for the PD sessions (D9). It should be noted that there are many more possible determinants to explicate for the PD program, but our intention here is not to explicate all possible determinants, only a few that are enough to make our point. We also want to note that we believe some of the determinants above really are vital as to why ARPA worked, while others are examples that we do not think matter much but which cannot be ruled out either because no evaluation of them is reported in this analyzed paper.

Discussion

In this paper, we analyzed elements of an implicit ToC, in terms of hypothetical determinants, in a critical case [START_REF] Flyvbjerg | Five misunderstandings about case-study research[END_REF] reporting the implementation and student-level results of a PD program focusing on problem-solving. This exercise aims to illustrate the difference between reporting PD results with a focus on the problem-solving component and reporting on the connections between the implementation determinants in relation to the program's impact, i.e., having implementation research in focus.

The case we analyzed is interesting since PD programs promoting problem-solving are relatively common in mathematics education research. The program reported by [START_REF] Saadati | Assessing impact of a teacher professional development program on student problem-solving performance[END_REF], ARPA, was reported as successful. But, to answer our research question, from an implementation research perspective, we need to know more about the program determinants to be able to adapt the innovations to other contexts. Unless internal mechanisms in a program are explicated and preferably evaluated, it is difficult to identify why the program works. The more one knows about the determinants and, more generally, the theory of change, the easier it will be to plan for replication.

Transparency with determinants is also important for comparisons with other problem-solving programs, showing more mixed results. This is the general idea behind the theory-driven evaluation movement [START_REF] Chen | Theory-driven evaluation: Conceptual framework, application and advancement[END_REF]. The article by [START_REF] Saadati | Assessing impact of a teacher professional development program on student problem-solving performance[END_REF] represents a good example of a relatively typical article within the problem-solving subfield of mathematics education research. It is not written as an article in implementation research. The work we presented in this paper can be seen as an illustration of what researching the ARPA project from an implementation perspective could entail.

Of all the particularities of ARPA, we identified and focused on a set of hypothetical determinants, of which two are of particular interest: the role of the guiding questions and the role of the specific problems. In our view, it is likely that these hypothetical determinants have a core role in the functioning of the program. First, it is well-known from mathematics education research that a challenge when implementing problem-solving is that teachers often feel tempted to give too many hints or even give away the solution, in particular when students struggle (Heibert & Grouws, 2007). Since it is often argued that it is the struggle that induces the learning effect in problem-solving-based teaching, extensive support would ruin the learning effect (Heibert & Grouws, 2007). Unless one has great experience with a problem, providing students with the right guiding questions without giving away the solution is hard. In ARPA, teachers prepare productive guiding questions in collegial PD sessions. This brings us to the second aspect: having the problems for the problem-solving lessons fixed beforehand. It is this design choice that allows the teachers to prepare guiding questions. The role of the guiding questions and the role of the chosen problems are thus nested together and also nested with both the PD-session design and the lesson design in ARPA. This nesting is, in our view, probably an important aspect of the ARPA design.

Can ARPA be considered an implementation of research results from mathematics education? Yes, because the role of a productive struggle for learning, as well as the tendency of teachers to see such struggle as unproductive, together with a preference to instead explain solutions or give hints that lead to the avoidance of struggle, are well-documented empirical findings (e.g., [START_REF] Hiebert | Problem Solving as a Basis for Reform in Curriculum and Instruction: The Case of Mathematics[END_REF], Heibert & Grouws, 2007). Therefore, we can say that the theory of change for ARPA is, in fact, an explanatory type of theory that connects the important role of struggle with teachers' tendency to avoid it and suggests that well-prepared guiding questions are a way of providing both teachers and students with more acceptance for the struggle phase. If we posit the role of teachers being prepared with guiding questions as an enabling factor for teachers being likely to conduct problem-solving lessons where students are kept in enough productive struggle, then the set of determinants we have focused on can be seen as part of a more conclusive theory of change. The interdependent roles of the pre-specified problems and associated guiding questions in both the PD sessions and PS lessons then fit as a concrete realization of such a theory of change.

It should be noted again that our analysis of ARPA is theoretical and based only on the reporting of the program by [START_REF] Saadati | Assessing impact of a teacher professional development program on student problem-solving performance[END_REF]. Because the article we analyzed did not report any evaluation other than on student problem-solving skill development, we cannot know if the determinants we explicated are, in fact, the important ones. We also, on purpose, chose to discuss only a few of the possible determinates of the program and, for example, did not say anything about the role of group work in the PS lessons or the role of teachers spending time in the PD-sessions to work on and discuss the solution to the problems. In replications of ARPA, it would be worthwhile to include measures of the determinants we have highlighted, as well as other possible determinants.

  Figure 1 gives an indication of how the hypothetical determinants D1 to D9 are nested in a hypothetical ToC within the general change model, where the PD program supports a particular lesson model, which in turn leads to the development of students' PS skills and how the different determinants relate to each other.

Figure 1 :

 1 Figure 1: Elements of a hypothetical ToC in relation to the PD program's implementation described by Saadati and Felmer (2021)

  . Still, according to[START_REF] Jankvist | Illustrating the need for a 'Theory of Change' in implementation processes[END_REF], a theory-driven evaluation presupposes an explicit ToC. Theory-driven evaluations have two key components: a conceptual one, that is, that the evaluation should explicate a program theory or model, and an empirical one, that is, that the evaluation should investigate how the programs (or innovations) cause intended or observed changes and outcomes. (p. 1048)

In

[START_REF] Chen | Theory-driven evaluation: Conceptual framework, application and advancement[END_REF]

account of theory-driven evaluation, the corresponding concept to a ToC is called a change model. Chen described that "each program must identify a leverage or mechanism upon which it can develop a treatment or intervention to meet a need. That leverage or mechanism is variously called the determinant or the intervening variable" (p. 18). When we analyze our critical case, we will do so by looking for such mechanisms and referring to them as hypothetical determinants. The term hypothetical is relevant for two reasons. First, we do not know exactly what the designers of the program under scrutiny consider to be the most important mechanisms of the program. There are several aspects of the program that we do not choose to discuss that might, in fact, be important. Second, before an evaluation of the role of possible determinants of the program has been made, any presentations of important mechanisms will remain hypothetical.

Acknowledgments

This paper is part of grant 2022-04811 under the Swedish Research Council.


Boris Koichu
email: boris.koichu@weizmann.ac.il

Aamer Badarneh

Menucha Farber

What can researchers and teachers co-learn from data on students' problem-solving experiences?

Keywords: Problem solving, implementation chain, research-practice partnership, questionnaire on student struggle

Problem solving is an endorsed but still rare activity in school mathematics instruction. The current study explores the assumption that research-practice partnerships conducted in a co-learning agreement mode can support implementation of problem solving in middle school. Within a colearning agreement, three researchers developed a questionnaire aimed at characterizing students' struggle with mathematical problems, and four teachers administered the questionnaire to their students after conducting problem-solving lessons. Both parties attended to the student responses in the hope to learn something useful. A protocol of the data-analysis session by the researchers and semi-structured interviews with three teachers were analysed to reveal what exactly each party could infer from the same data. Essential differences were found. However, the researcher and the teacher learning seemed to be mutually enriching, which is important from the implementation perspective.

Introduction

This paper is part of a series of reports based on a long-term study on systematic implementation of problem solving (PS) as a desirable innovation in regular mathematics classrooms. In the previous reports, we proposed that the complexity of PS implementation can usefully be tackled by means of a theoretical-organizational framework entitled Problem-Solving Implementation Chain (Cooper & Koichu, 2022;[START_REF] Koichu | Implementation of problem solving in school: from intended to experienced[END_REF]. The chain consists of the following links: (i) PS as intended by those specialists who design and propose PS resources (e.g., mathematics education researchers, task designers), (ii) PS as planned by teachers who interact with the proponents or with the resources, (iii) PS as enacted by the teachers in their classrooms, and (iv) PS as experienced by students.

The framework relies on two assumptions. First, PS resources can attain different characteristics at each link of the chain or, metaphorically speaking, evolve. Second, since the evolution occurs in interaction of different stakeholders (e.g., researchers, task designers, professional development facilitators, teachers, and students), study of these interactions can (partially) explain the evolution and shed light on several long-standing queries, such as: Why PS is still a rare activity in a regular mathematics classroom [START_REF] Schoenfeld | Why are learning and teaching mathematics so difficult?[END_REF]; why instructional materials intended to provide students with opportunities for PS are frequently experienced as drill and practice (Felmer et al., 2019); how PS can be taught [START_REF] Lester | Can mathematical problem solving be taught? Preliminary answers from 30 years of research[END_REF]; how evidence of enactment of PS resources with students can be used by the resource-enactors and the resource-proponents [START_REF] Koichu | Implementation of problem solving in school: from intended to experienced[END_REF].

Our current study is related to the last query, and, accordingly, focuses on the fourth link of the implementation chain and its relation to the rest of the links. In addition, our study is deeply rooted in a perspective on educational research as a co-learning agreement between researchers and practitioners [START_REF] Wagner | The unavoidable intervention of educational research: A framework for reconsidering researcher-practitioner cooperation[END_REF]. To recall, this perspective aspires to create opportunities for co-learning by reducing asymmetry between educational researchers and practitioners, recognizing their different types of expertise and agendas as equally and mutually important. As Wagner states, By working together, each [researchers and practitioners] might learn something more about the world of the other. Of equal importance, however, each may learn something more about his or her own world and its connection to institutions of schooling. (Wagner, 1997, p. 16) Co-learning agreements that involve joint collecting and analysing data on teaching and learning mathematics have been shown (e.g., by [START_REF] Pinto | Implementation of mathematics education research as crossing the boundary between disciplined inquiry and teacher inquiry[END_REF] to be promising for implementation research on innovations [START_REF] Century | Implementation research: Finding common ground on what, how, why, where, and who[END_REF], in our case, on implementation of PS in regular classrooms. In these terms, our study addresses the following question: What can mathematics education researchers and mathematics teachers, who cooperate within a co-learning agreement, infer from shared research data on students' PS experiences? For practical considerations and with an eye on scaling up, we focus on a single data source, namely, a questionnaire on students' experienced PS which is designed to be administered immediately after a PS lesson.

Theoretical framework and the questionnaire in question

In this section we operationalize PS as experienced, present the rationale and structure of the questionnaire and discuss its learning potential from two perspectives, of researchers and of teachers.

A researcher perspective: Focus on characteristics of student struggle

As mentioned, instructional resources intended to facilitate PS sometimes reach students in ways that facilitate drill and practice. To this end, the main conceptual distinction is between a problem and an exercise, the former referring to a task for which the solution method is not known in advance to the solver, and the latterto a task whose solution method requires recall rather than invention (e.g., [START_REF] Kilpatrick | A retrospective account of the past 25 years of research on teaching mathematical problem solving[END_REF]. Note that a mathematical task cannot be classified as problem or exercise based on its formulation only. As Kilpatrick asserted, "Researchers in mathematics education have long accepted the truth that a problem for you today may not be one for me today or for you tomorrow" (Kilpatrick, 1985, p. 3). Therefore, the solvers' subjective experiences should be considered. [START_REF] Hiebert | The effects of classroom mathematics teaching on students' learning[END_REF] use the word struggle to mean that students must expend effort to figure out something that is not immediately apparent. In our study, we treat evidence of struggle as a signal that a solver experienced a task as a problem. We operationalize struggle as a self-reported difficulty that has required from the solver effort or help to overcome, as well as a difficulty that the solver experienced but did not overcome. Such difficulties can be of different nature and be handled by different means. Capitalizing on Pólya's PS model and additional resources, [START_REF] Warshauer | Productive struggle in middle school mathematics classrooms[END_REF] developed a categorization of kinds of struggle, as follows: (1) struggle to get started (e.g., confusion about what the task is asking), (2) struggle to carry out a process (e.g., encountering an impasse), (3) struggle related to uncertainty in explaining and sense-making (e.g., difficulty in communicating the solution), (4) expressing misconceptions and errors (e.g., difficulty in finding mistakes in the solution). Based on this categorization, we developed 10 multiple-choice questions, four of which are presented in Table 1 as examples. For us, low levels of struggle (1-2) in relation to all 10 items is a signal that the task was too easy for the solver to qualify as a problem, and the intermediate or high level of struggle (3-5) in relation to at least one item means that the task was experienced as a problem, and also indicates which aspects of the task were problematic. We also wished to obtain evidence of what the students did when encountering difficulties. In line with [START_REF] Warshauer | Productive struggle in middle school mathematics classrooms[END_REF], we assumed that PS lessons are likely to include independent or small-group PS and whole-class discussions, so the questions are formatted accordingly (see the right column below). The open part of the questionnaire (see Table 2) was designed to elicit student's elaborated reflections on the lesson. It was designed with an eye on characterizing the productiveness of the student struggle. Productive struggle is associated with a positive attitude towards challenges and setbacks and with students' satisfaction with learning new mathematics [START_REF] Warshauer | Productive struggle in middle school mathematics classrooms[END_REF]. This is why the questions request students to recall significant, novel or pleasant for them moments and aspects of the PS lesson. In other words, we chose to use evidence of students' satisfaction with aspects of the lesson, if any, as indicators of whether their struggle was productive for them in some sense.

Table 2: Open questions from the questionnaire on PS as experienced 1. While working on the problem, were there any significant moments for you? If so, what was their significance? 2. Complete the following sentence: While working on the problem, I was pleased that… 3. Complete the following sentence: I would be more pleased while working on the problem if… 4. Did the whole-class discussion raise something that was new for you? If so, what was it? 5. Complete the following sentence: During the whole-class discussion, I was pleased that… 6. Complete the following sentence: I would be more pleased by the whole-class discussion if…

A teacher perspective: Focus on what was successful or could be improved

Even though struggle is a widespread term in the mathematics education professional literature (e.g., [START_REF] Hiebert | The effects of classroom mathematics teaching on students' learning[END_REF][START_REF] Warshauer | Productive struggle in middle school mathematics classrooms[END_REF], this term is hardly used in mathematics teachers' discourse on their lessons, at least not in Israel where the study was conducted. Therefore, we deemed it unlikely that mathematics teachers would find a research agenda of characterizing students' struggle particularly attuned to their interests. However, we hoped that students' responses to the questionnaire would be of interest to the teachers as artefacts beyond their usual reach, and that these artefacts would stimulate the teachers' in-depth reflection and create new learning opportunities.

To initially make sense of the learning opportunities for teachers in relation to the developed questionnaire, we rely on a teaching triad model [START_REF] Jaworski | Investigating Mathematics Teaching: A Constructivist Enquiry[END_REF][START_REF] Potari | Tackling the complexity of mathematics teaching: Using the teaching triad as a tool for reflection and analysis[END_REF]. To recall, the teaching triad consists of three domains of activity in which teachers were seen to engage when using open-ended problems in teaching: management of learning (ML); sensitivity to students (SS) and mathematical challenge (MC). In the words of Potari and Jaworski:

Management of learning describes the teacher's role in the constitution of the classroom learning environment by the teacher and students. It includes classroom groupings; planning of tasks and activity; setting of norms and so on. Sensitivity to students describes the teacher's knowledge of students and attention to their needs; the ways in which the teacher interacts with individuals and guides group interactions. Mathematical challenge describes the challenges offered to students to engender mathematical thinking and activity. (Potari & Jaworski, 2002, pp. 352-353) In our study, ML, SS and MC are used as categories in the analysis of teachers' assertions on what they learned on their PS lessons when exposed to the student responses to the questionnaire.

Method

This study was conducted as part of the Raising the Bar in Mathematics Classrooms (RBMC) R&D project. The project team consists of 10 specialists of different professional profiles. Since 2020, the team developed a bank of challenging mathematical problems. These problems have been exposed to more than 200 secondary-school mathematics teachers (teachers of grades 8-10 with 14-16-year-old students) in the projects' workshops and professional learning communities. In these workshops and communities, the problems, and their schemes of enactment (see [START_REF] Koichu | Implementation of problem solving in school: from intended to experienced[END_REF], for details and examples) are discussed, and then the teachers implement the problems in their classes. As a rule, the teachers describe the enactment orally during the workshop and in writing, by means of a semistructured reflective protocol (Cooper & Gruenhut, 2023).

The data on the teacher-perspective for the current study were collected in semi-structured interviews with three teachers, Yana, Maya, and Ella (the names are pseudonyms), who took active part in the RBMC activities and were familiar with the authors of this paper for at least a year. All three teachers hold a M.Ed. degree in mathematics education and have more than 15 years of experience in teaching mathematics. In January 2023, these teachers agreed to administer the questionnaire to their students after their PS lessons, then to read the students' responses and be individually interviewed. The interviews were conducted via Zoom and lasted for 45 minutes each. The interviews included five thematic clusters: (1) background questions (e.g., "tell us about your class"); (2) questions about the lesson (e.g., "how did you organize the class in small groups?"); (3) questions about particular students (e.g., "tell us about two students who were the first to solve the problem"; (4) questions on the lessons learned from the questionnaire (e.g., "is there something new that you learned from the closed/open part of the questionnaire?"); (5) questions on the questionnaire itself (e.g., "would you advise us to remove or add some questions?"). The interviews were recorded and transcribed. Parts of the transcripts related to the fourth cluster were categorized using thematic analysis [START_REF] Saldaña | The coding manual for qualitative researchers[END_REF]. The categorization was in terms of the teaching triad presented above.

The data on the researcher-perspective were collected from а three-hour recorded Zoom meeting of the three authors. A protocol of this meeting was used as the main data source. During the meeting, we attended to the Excel table consisting of responses to the questionnaire of 71 students, but mainly focused on the data from one class taught by Sorina. We raised suggestions about what was happening in the lesson and then tried to support or refute these conjectures by means of applying descriptive statistics to the closed part of the questionnaire or freely interpreting the open responses.

Findings

To recall, the research question was about what mathematics education researchers and mathematics teachers can infer from consideration of the questionnaire data. In this section, we provide a two-part answer to this question. Due to the space constraints, we address the first part by providing characteristic examples and the second part by presenting thematic summaries of findings.

A researcher-perspective on learning from the questionnaire

Given the goals of the questionnaire, it is not surprising that we first learned about the extent and characteristics of the students' struggle with a particular problem (see Figure 1) taught by Sorina, Yana and Maya1 . Overall, we felt that we could see quite a nuanced picture of the lessons without being there, though some of our conjectures called for additional data to be tested. Based on student-responses to Q1 (see Table 1) we learned that 11 out of 26 students who took part in Sorina's lesson had not struggled to understand the problem formulation and 15 students had. We also learned that 25 students eventually understood the problem, 16 by themselves and ninewith the help of the teacher. From Q2 we learned that most of the students (21) immediately had an initial idea of how to approach the problem once the formulation had been understood and that five students had not. However, these five students overcame the difficulty, onewith the help of a classmate, and fourof the teacher. Interestingly, there was no significant Pearson correlation between the answers to Q1 and Q2 (r=0.003, p>0.05), which means that the students could begin to solve the problem regardless of whether it was difficult or not to understand the problem formulation. This observation led us to the following conjecture: if teachers wish to preserve students' PS autonomy, assistance with understanding the problem formulation might not be a hindrance to the independent search for solution ideas. However, one would need more information to test this conjecture.

Based on Q3, we learned that 20 students felt that their initial idea was unsuccessful at least to some extent, but all 26 students figured out how to proceed, including six students who consulted peers and five who consulted the teacher. Twelve students reported, in response to Q4, that they had encountered a dead-end at some stage but 10 of them found a way out. The means of getting out of the dead-end varied, but, interestingly, the group work was less common when trying to overcome impasses, and more common in consideration of solution ideas produced by individual effort. We also learned that the Three-Square Problem was indeed a problem for all students in the class, and, as intended, its solution was surprising but within reach.

The open part of the questionnaire was informative to us mainly regarding the whole-class discussion. For example, we learned that some students wanted more room for student-presentations, somefor the teacher's presentation, which might have put things in good order, and some wanted more time for the discussion to attain both goals. Regarding the rest of the open items, we concluded that most of the students positively valued their experience, but we assessed many answers as too general.

A teacher-perspective on learning from the questionnaire

In contrast to the researchers, the teachers found the open part of the questionnaire more instructive than the closed part. Yana explained that drawing conclusions from the closed part was difficult. She said: "I think that I, and many other teachers, would consider the first part only if you [the researchers] tell us what we can learn from it". Maya indicated that the open part was more interesting because it gave her ideas for what she could have done differently and how to improve in the future. Ella noted that reading through the closed questions had been difficult, and also said that the closed part gave her a picture per student whereas the open parta picture of the entire class, which she valued. When the interviewers presented the teachers with examples of what could be concluded from the closed questions (see above), Ella and Yana reconsidered. They re-engaged in reading the quantitative data and made several observations. For example, Ella proudly said: "See? I really did not help them!" In contrast, Yana was surprised to find that several of her students credited her for help when she thought she was not helping them. Upon elaboration, it became clear that Yana made encouraging remarks (e.g., "keep trying") that the students interpreted as help and Yana did not.

We continue with brief summaries of teachers' assertions about their learning from the questionnaire in terms of three elements of the teaching triad, ML, SS and MC (see above).

In relation to ML: Yana and Maya indicated that the questionnaire made them think about time allocation in the different parts of the lessons. Yana said: "I should have known better how to divide time between the problem presentation, work in small groups and whole-class discussion." Maya discovered that time she allocated for the whole-class discussion was insufficient.

In relation to SS:

The teachers discovered some surprising aspects of their students' experiences and sometimes transformed these surprises into ideas for what they might do in the future. For example, Maya said: "I did not realize that they are so eager to discuss. I thought that [small-group and whole-class] discussions were like a competition for them, but they really wanted to learn from each other." Yana was also surprised that her students were willing to discuss but in her view, "the students competed with each other for whose explanation or solution is better." Ella learned that some of those students that she perceived as "passive" based on her in-the-field impression, were engaged in the whole-class discussion, and enjoyed it. All three teachers were surprised to see the diversity of their students' reflections on different stages of the lesson and reasoned how to personalize their teaching.

In relation to MC: Maya and Yana made comments about the difficulty and appropriateness of the Three-Square Problem for their students. Yana was encouraged by the fact that the students apparently liked the problem as "not standard and inviting discussions", whereas Maya pointed out that she identified several students for whom the problem was very challenging, more so than she had expected. Ella noticed that her prior perception of some of her students as "stronger" and "weaker" did not correspond with the students' responses on how difficult or easy the lesson had been for them.

Concluding remarks

Accumulating and analysing data on enacting an innovation with students is a cornerstone of implementation research [START_REF] Century | Implementation research: Finding common ground on what, how, why, where, and who[END_REF]. However, the question of how different stakeholders of implementation can be involved in this process along the links of the implementation chain remains open [START_REF] Koichu | Implementation of problem solving in school: from intended to experienced[END_REF]. In addition, the potential of learning from research data for teachers has been recognized decades ago (e.g., Bishop, 1977, cited in Kilpatrick, 1981) but little is known about how this learning can actually take place. These observations motivated our small-scope study, which showcases that co-learning of researchers and teachers from the shared datasets is possible and even desirable.

In our study, the researchers and the teachers chose to attend to different clusters of the data on students' experiences in problem-solving lessons, looked at the data from different angles and reached different conclusions. The researchers mainly produced conjectures about what was happening in the lessons and about phenomena related to students' struggle in individual and group work, and in turn, about characteristics of the instructional materials as intended vs. as enacted and experienced. The teachers mainly sought practical insights in relation to what could be improved in the lessons. Both parties were surprised by some of the findings as well as by some of the observations made by the other party. To this end, the researcher learning and the teacher learning seem to be complementary and mutually enriching. Needless to say, our study is only a small step in revealing the potential of co-learning agreements in implementation research, and that many measures for making such colearnings relevant, useful and productive can be thought of.

Figure 1 :

 1 Figure 1: Three-Square Problem (adapted from https://adasha.weizmann.ac.il)

Table 1 : Sample multiple-choice questions from the questionnaire on PS as experienced

 1

		Extent of agreement,		How did I overcome the difficulty? (denote by X all appropriate options)
	Statement of difficulty	from 1 (fully disagree) to 5 (fully agree)	I did not	By myself	With the classmates help of the	With the help teacher's	Not relevant
	Q1. At first, it was difficult for me to
	understand the problem formulation.
	Q2. When I had understood the
	problem formulation, I felt that I
	didn't know how to begin solving it.
	Q3. My first initial idea appeared to
	be not really productive.

Q4. I began solving the task and felt that I reached a dead-end.

Ella was teaching another problem, which is not presented here due to the space constraints.Three identical squares are given, and three identical right-angle triangles are placed on them, such that the vertex of the right angle is exactly on the center of the square. Arrange the figures according to the area that is common to the square and the triangle, from least to greatest. Justify your claims!

Acknowledgment

The research was funded by the Trump Foundation (371) and the Israel Science Foundation (2930/21). Opinions expressed in this paper are not necessarily shared by the foundations.


Morten Misfeldt
email: misfeldt@ind.ku.dk

Thomas Brahe

Uffe Thomas Jankvist

Raimundo Elicer

Eirini Geraniou

Kajsa Bråting

Andreas Lindenskov Tamborg

Implementation through adaptation: Relations between mathematics, programming, and computational thinking

Keywords: Co-creation, computational thinking, implementation, mathematical competencies

This paper examines the experience of three teachers who participated in developing teaching resources for programming and computational thinking (PCT) as part of a project that integrates mathematics and PCT. Using interviews, we explore the teachers' understanding of this integration and their approach to transmitting it through teaching sequences. The findings reveal that the teachers face challenges in exerting agency over the innovation and adapting it for implementation, affecting their ability to integrate PCT and mathematics effectively.

Introduction

There is growing interest in integrating programming and computational thinking (PCT) into primary and secondary education. In Denmark, a pilot project was launched in 2018 to teach "technology comprehension" to K-9 students in 46 schools, using a curriculum focused on four areas: digital empowerment, digital design, computational thinking, and technological agency. The project tested two approaches: technology comprehension as an independent subject and as part of traditional subjects, including mathematics. The evaluation showed that teachers largely followed the resources developed, thus highlighting the importance of teaching resources for successful implementations (Rambøll, 2021). Analysis of existing resources show varying levels of integration between programming and mathematics [START_REF] Elicer | Nature of the relations between programming and computational thinking and mathematics in Danish teaching resources[END_REF][START_REF] Elicer | Comparing programming in elementary mathematics in Sweden and Denmark[END_REF]. The authors of this paper have conducted comparative explorations of PCT integration in Denmark, England, and Sweden [START_REF] Elicer | Comparing programming in elementary mathematics in Sweden and Denmark[END_REF], accompanied by several design experiments testing preliminary findings in Danish classrooms. The findings from these experiments were used to initiate a co-creation process with five teachers, who were invited to develop resources intended to take advantage of the explored synergies between mathematics and PCT. This paper describes our collaboration with three of those teachers, here referred to as Michael, Lucas, and John. There was no prior relationship between these teachers and the team of researchers. Michael was recruited through a local municipality driven makerspace, he had only limited experience with resource development. Lucas and John had more experience, and they were introduced to us by the publishing house that will conduct the dissemination of the resources. The development of these resources is the outset of this paper. We focus on the experience of three of the five engaged teachers. We aim at answering the following question: How do the three teachers adapt the idea of integrating PCT and mathematics in their design of teaching resources, and what considerations did they have in this respect?

In order to study conceptual integration, we have based ourselves in the emerging literature of implementation research in mathematics education [START_REF] Koichu | Implementation-related research in mathematics education: the search for identity[END_REF]. The idea is that challenges, difficulties and concerns that teachers have with integrating PCT and mathematics can support implementation and change, if these difficulties lead to adaption and personalization of the resource.

In the following, we first describe in detail the interplay between PCT and mathematics as well as the adopted framework of co-creation implementation. Next, we describe the empirical bases and results. Finally, we discuss potential challenges with PCT in the Danish mathematics classrooms.

Programming, computational thinking and mathematics

A number of characterizations of computational thinking already exists. Yet, they generally refer to a collection of knowledge and skills to solve problems modeled in such a way that problems can be unambiguously communicated to and solved by a computer [START_REF] Wing | Computational thinking[END_REF]. [START_REF] Wing | Computational thinking[END_REF] describes computational thinking as decomposition, data representation, pattern recognition, abstractions, and algorithms. Here, we refer to programming and computational thinking (PCT) in order to stress the inclusion of actual programming in our focus. This is related to mathematics, of course, e.g., through problem solving, data handling, and modeling. [START_REF] Elicer | Nature of the relations between programming and computational thinking and mathematics in Danish teaching resources[END_REF] developed six categories or "levels" describing the integration between mathematics and PCT. These are: (1) no mathematics involved; (2) no PCT involved; (3) mathematics as a context; (4) PCT as a context; (5) conceptual integration involving concepts from both mathematics and PCT in the scene-setting and wrap-up, but not necessarily in the specific solution methods involved; and (6) operational integration where mathematics and PCT competencies are interdependent. Building on a competencies approach to mathematics [START_REF] Niss | Mathematical competencies revisited[END_REF], these categories are defined by distinguishing actions and concepts in both mathematics and PCT as depicted in available teaching resources. In turn, we aim at using them for the production of new ones.

Implementation as collaborative creation

In education, there is always a continuity between implementation and creation. As stated by [START_REF] Bruckheimer | Creative implementation[END_REF], "Generally, curricula are not implemented" (p. 43). He continues to argue that implementation should be regarded as a dynamic process, and that the process of adoption is also a process of manipulation, adaptation, and additions. [START_REF] Jankvist | Launching Implementation and Replication Studies in Mathematics Education (IRME) [Editorial[END_REF] argue that "the actual implementations should themselves become creative" (p. 4). In order to understand implementation in this way, we build on a definition of implementation developed by Koichu and colleagues (2021).

They conceptualize implementation as an ecological disruption to state of affairs that happens through a combination of: "(a) endorsements of innovations; (b) an action plan; and (c) a perceived problem or challenge that can be addressed through (a) and (b)" (p. 986). [START_REF] Koichu | Implementation-related research in mathematics education: the search for identity[END_REF] describe the different stakeholders as proponents of an innovation and innovation plan and adapters of the innovation, respectively. Innovation adapters will go through a process in four parts:

(1) constructing agency over innovation, (2) gradually changing within-community communication or across-community communication, (3) gradually changing practice so that it accommodates the innovation, (4) adapting the innovation to their needs and aspirations. (p. 986) This does, however, also mean that there is a constant interplay between proponents and adapters.

Hence, neither plan nor innovation is left unaffected by the implementation process. Building on the four sub-processes from [START_REF] Koichu | Implementation-related research in mathematics education: the search for identity[END_REF] and the six categories from [START_REF] Elicer | Nature of the relations between programming and computational thinking and mathematics in Danish teaching resources[END_REF], we aim to understand how our three teachers combine elements of PCT and mathematics in the various processes of developing teaching resources.

Method

The empirical foundation for this paper consists of interviews conducted with the three teachers to develop resources and was led by the second author of this paper. The teachers were given: 1) relative freedom on deciding and developing the themes of the resources; 2) support for developing their ideas and discuss the integration of PCT and mathematics with the researchers from the team; and 3) a clear objective to create tasks and teaching scenarios that require combination of PCT and mathematics. In the period dedicated for the teachers to develop their resources, three workshops were held where members from the research team jointly discussed what integrating PCT and mathematics means.

Despite the large degree of "freedom" and open-endedness in how to scope the resources, we asked the teachers specifically to address the interplay between PCT and mathematics. We were interested in how this setup led practitioners to reflect, develop and find solutions and attach meaning to this integration. Looking at the teachers initial suggestion for the resources, it became clear that the teachers used rather different approaches and levels of integration between PCT and mathematics.

The interview study

To understand the teachers' difficulties, the second author conducted interviews with Michael, John and Lucas. Michael was a mathematics teacher in his 40s with around 20 years of experience. In this project, Michael worked with resources directed towards lower secondary school. He had no previous experience as an author of teaching resources, but broad experience in teaching technology, albeit not programming. John, a younger mathematics teacher in his 30s, had experience with teaching programming. As author of teaching resources about block programming in this project, John worked with resources directed towards middle school. Lucas also worked with resources directed towards middle school. He was in his 50s and had a mixed background. He graduated as a teacher in the mid 1990s and always had an interest in integrating programming and computational thinking into school subjects. After ten years as a teacher, he began as a developer of educational technologies for children.

The interviews were audio recorded and later transcribed verbatim for analysis. A semi-structured approach was developed in order to explore the participants' experiences and conception. The interview guide was focused on the interplay between mathematics and PCT. We analyzed the interviews in a thematic manner [START_REF] Braun | Using thematic analysis in psychology[END_REF], in which we began by identifying their perception of the synergy between PCT and mathematics. Subsequently, we identified the teachers' conception of the implementation, and what they believed was needed to support the implementation of the teaching resources they had produced.

The synergy between PCT and mathematics

The synergy between PCT and mathematics was difficult for the teachers to use and adapt. Michael pointed to a specific presentation slide describing the computational thinker and said: "It was only really clear to me when you showed me the slide" because "it was very practical. The name 'construct' for example. Exactly that word helped me." This indicates both difficulties in the operationalization of the synergy, i.e., levels 5-6 in [START_REF] Elicer | Nature of the relations between programming and computational thinking and mathematics in Danish teaching resources[END_REF] integration modelbut it also says something about the value of tangible manifestations. According to Michael, what PCT can offer is actually not unique to mathematics: Michael: I would rather say that PCT provides something special in ALL subjects. I understand why we are doing it in mathematics due to the official curricula, but the exploring approach to create something physical, I think is a fit for all subjects.

Michael believed that the creation of tangible solutions to problems should be a guiding principle in all education. He traced this back to his childhood, building things with his father, where they learned from mistakes and adjusted their work until they succeeded. This hands-on approach is evident in his teaching resource, where he leads students on a fictional journey through a dystopian universe that requires them to solve tasks and survive through hands-on work.

For John, the connection between PCT and mathematics was more artificial. In his first iterations, John presented a specific model for project-oriented work without any relation to PCT and mathematics. This may be due to his general educational philosophy with a high degree of open-endedness for the students to work within. Still, it could also signal that he found it difficult to integrate PCT into mathematics. He touches upon this, later in the interview, when talking about the transferability to other teachers, who may have different backgrounds and different prerequisites: John: (...) We talked about the teacher guide and whether it makes sense to other teachers, or only to me. Does it make sense? And the first time that we talked, you were talking about mathematics and analog to digital coding… And now, I've ended up in a completely different place without the concrete coding part and without the concrete mathematics part. It is a more diffuse way of working.

For John, the interplay between PCT and mathematics was not easy to plan. It takes reflection, work and re-thinking in order to arrive at a higher level of integration concerning Elicer and Tamborg's (2022) integration model and to get past level 3, where for example mathematics is used as a context to a programming task without adding anything from a mathematical learning perspective.

Lucas, on the other hand, immediately seemed to adopt the idea of integrating PCT into mathematics.

He had a background as a mathematics teacher in primary school, and he had worked for more than 15 years developing educational technology in the industry. At first, the interview touched on how to teach programming and technology in general:

Lucas: In my opinion, a lot of it [PCT] comes by itself if you provide the students with a concrete problem or task that they have to solve. I think that the best way to learn programming and computational thinking is to actually do it and try it out.

His approach was centered round small, simple scenarios in which there is a problem for students to solve. For example, in one scenario, a girl wants to build a treat-dispensing robot for her dog. The student use a Lego Spike set and a Scratch-like programming environment to program the robot to dispense a treat when a sensor detected a certain distance. This task required both PCT and mathematical skills, and therefore demonstrated level 6 integration [START_REF] Elicer | Nature of the relations between programming and computational thinking and mathematics in Danish teaching resources[END_REF], where PCT and mathematics are tightly intertwined and a solution requires both.

The resources as implementation

An important aspect of the integration between PCT and mathematics is how to make its use as clear as possible to practitioners. As seen above, each teacher had his own personal background, with different knowledge, experience, values and worldviews. To varying degrees, this may have helped shape the resources and the underlying and implicit demands for uses to possess. Hence, the question is how the teachers reflected on this aspect with regard to making the resources accessible. For Michael, there is a certain degree of self-contradiction in that a resource calls for innovation and at the same time has a detailed set of instructions or guidelines for the users-but he also acknowledges the need for such guidelines: Michael: (...) Of course, the teacher guide has huge implications. If it [the resource] seems too unmanageable, some teachers may give it a pass.

The view on teacher guiding was one of the central themes in the interview guide as part of our assessment of the path from initial idea to eventual implementation. This in particular with regard to the PCT area that is highly demanding from a technological point of view. As for John, he sees a potential implementation problem that may stem from the basic structures that mathematics teachers are working within; a mathematics teacher is required to complete or touch upon certain competency areas during a school year. Hence, adding PCT, which the teacher may not be familiar with, can prove to be an obstacle that some want to avoid: John: Certainly, there are teachers who, from the outset of the school year, have made specific plans as to what to accomplish and when: "We are going to slavishly follow the textbook."

In math, that is pretty easy, since all schools have a textbook system for the individual age groups and classes. It is easy to go: "Well then, we'll go with the mathematics textbook and don't look back." This is important to remember when developing resources that are trying to "shake things up" by implementing new ways of working and combining subject areas, i.e., a lot hinges on the individual teacher's personal preferences. Hence, having a very stringent focus on teacher guiding is crucial. It needs to be easy and accessible. Along those lines, Lucas provided a similar perspective and mentioned the lack of programming in the Danish teacher education (as he recalled it):

Lucas: One could argue that it is a problem that is amplified by the fact that Danish teachers are not trained in programming. During the teacher training, you are not provided with a coherent and elaborate course in how to program-and that creates uncertainty. They don't believe that they can. Hence, they don't believe that their students can. Now, Lucas' utterances are of course based on his personal beliefs and experiences. Yet, he does touch upon a theme that may be a real-life barrier in some cases where individual teachers choose not to use PCT resources because of lack of experience with (basic) programming.

Discussion: Implementing the interplay between PCT and mathematics

Even if teachers agree on the potential value of integration of PCT into mathematics education, other factors (including prior experiences, expertise, beliefs and motivations) come into play, when we aim for implementing the interplay between PCT and mathematics in teaching practices and resources.

Applying [START_REF] Koichu | Implementation-related research in mathematics education: the search for identity[END_REF] implementation framework, we recognized the challenges that the three innovation adapters, John, Michael and Lucas, were faced with, as well as their views and strengths. In terms of the first stage in the implementation process, i.e., (1) constructing agency over the innovation, both John and Michael had difficulties with constructing agency over the strong interrelation between PCT and mathematics. For both of them, the interplay between mathematics and PCT was considered rather abstract. On the contrary, Lucas found this first stage easier, since he had much more experience in developing resources as compared to John and Michael. In addition, he seemed to have already internalized the interplay between PCT and mathematics. Moving on to the second stage, (2) gradually changing within-community communication or across-community communication, the co-designing and co-editing work, and the ongoing debate with the second author, did gradually change John's and Michael's resources in that the sequences and tasks developed closer and more interdependent relationship between PCT and mathematics. This, quite crucially, also changed their conception of the interplay between PCT and mathematics in their own practice and the teaching practice in general. Over the duration of the project, the communication and interactions between the researchers and John and Michael progressed from "telling" John and Michael about how PCT and mathematics integration may be achieved to offering feedback on their ideas and prototypes of teaching resources. This process was not required for Lucas. Lucas was able to accommodate the PCT and mathematics integration to his existing ways of thinking about teaching resources. Regarding the third stage in the implementation process, (3) gradually changing practice so that it accommodates the innovation, we find that the main problem for John and Michael was the lack of time for achieving a change in their teaching practice in order to accommodate the integration of PCT into mathematics. This of course was not an issue for Lucas, as the innovation in question was already embedded (to a certain level) in his practice. Finally, the fourth stage in the implementation process involves (4) adapting the innovation to their needs and aspirations. This happened in all three cases. John relied on students finding the meeting points between PCT and mathematics on their own, while he wanted to have a relatively "hands-off" approach. He adapted the integration of PCT to mathematics, as his aspirations were to prepare the ground for projectoriented pedagogy. Michael adapted the integration between PCT and mathematics in the direction of narrative teaching. Finally, for Lucas the whole idea of PCT-mathematics integration was already part of his way of thinking. He adapted the integration in question in a direction of problem solving scenarios, closely aligned with how he had worked with the development of resources before.

The question that navigated the work presented in this paper was: "How do the three teachers adapt the idea of integrating PCT and mathematics in their design of teaching resources, and what considerations did they have in this respect?" Co-designing resources with researchers was a valuable, yet complex exercise for the participant teachers, which also revealed the different needs each teacher had. An awareness of these challenges and difficulties, as well as the contributions and expertise offered by researchers, places us in as position towards integrating PCT into mathematics education on a basis informed and enlightened by implementation research.

  John: In what I ended up working with, I actually found it difficult to integrate mathematics and programming. It has been more like a project, where the students are working on an innovative project, and I was careful not to tell them "You MUST use a specific mathematical concept, or you MUST use a specific way to program." So, I left it open for the students to think along those lines themselves.

Acknowledgment

This work is funded by the Novo Nordisk Foundation, grant number NNF19OC0058651.


John Prytz
email: johan.prytz@edu.uu.se

Linda Marie Ahl

Uffe Thomas Jankvist

Innovation and the problem of measuring effects in large scale professional development projects

Keywords: Innovation, effects, measurement, development project, replication

This paper concerns the relation between the research fields of implementation research (IR) and professional development (PD). Our main interest is in the concept of innovation and measurement of effects in large scale development projects. The case for our discussion is the Swedish large-scale development project Boost for Mathematics (BM). A project designed according to theories from PD research. Our tentative conclusion, in an IR perspective, is that it is difficult to replicate the BM project. This despite the fact there are six studies that measure the effects of the project. We argue that replication requires better knowledge about the innovations as possible causes of effects. We suggest further research in this direction.

Introduction

The study presented in this paper is part of the project Implementation research as an emerging field of mathematics education. The project's aim is to create an empirically based theoretical framework for implementation research (IR) in the field of mathematics education research, in particular for research on large-scale development projects. Our project examines existing theories in implementation research and which parts of these are applicable to implementation projects related to mathematics education. To verify this, we test the theories and related concepts that we find relevant through comparisons of five selected cases. Historical comparisons are made of three Swedish projects: Boost for Mathematics (BM) (Matematiklyftet) (2013)(2014)(2015)(2016); PUMP (Processanalyser av Undervisning i Matematik/Psykolingvistik, Process analysis of Teaching in Mathematics/Psycholinguistics) (1974)(1975)(1976)(1977); and New Mathematics (1961Mathematics ( -1968)). Also, contemporary comparisons are made between Boost for Mathematics and MIST (Middle-school Mathematics and the Institutional Setting of Teaching) (2007-) and TRIUMPHS (TRansforming Instruction in Undergraduate Mathematics via Primary Historical Sources) (2015-), the two latter are projects from the USA.

In this paper we present an initial and tentative study of the relation between theories from IR and professional development (PD) research. These two fields are entangled since large scale professional development projects, concerning mathematics education, are enterprises where innovations often are implemented; we know this from our recent research overview [START_REF] Ahl | Professional development as a means for implementing mathematics education innovations: results from a systematic review[END_REF]. By studying, through an IR lens, a large-scale development project based on PD theory we seek to understand in what way central elements of IR theory, in comparison to PD theory, can provide new insights; this is our purpose. As a case for our discussion, we take the Swedish BM project.

The BM was an in-service training program for mathematics teachers in Sweden, covering school years 1 to 12. The project ran between 2013 and 2016. 76% of all mathematics teachers joined the project. It is well known the BM was designed according to PD theories [START_REF] Boesen | National-scale professional development in Sweden : theory, policy, practice[END_REF][START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF]. A core design idea was peer learning among teachers. [START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF] concluded that teachers rather than researchers were supposed to complete the innovations. Teachers were given much freedom in relation to the training material. Another feature of the BM is that its impact on teachers' behaviour and students' learning have been measured in several studies [START_REF] Grönqvist | Lyfter Matematiklyftet matematikkunskaperna?[END_REF][START_REF] Lindvall | Two large-scale professional development programs for mathematics teachers and their impact on student achievement[END_REF][START_REF] Lindvall | Impact and design of a national-scale professional development program for mathematics teachers[END_REF][START_REF] Lindvall | Critical features of professional development programs: Comparing content focus and impact of two large-scale programs[END_REF][START_REF] Ramböl | Slututvärdering av matematiklyftet 2013-2016[END_REF][START_REF] Österholm | Utvärdering av Matematiklyftets resultat: slutrapport[END_REF].

However, from an IR perspective the idea to measure impact or effect of a project where teachers are given much freedom is problematic. In particular when causes of the effects are considered. In this paper we develop a theoretical discussion on why this is problematic, with the BM as our case. Our guiding questions for this discussion are:

1. What type of project was the BM in an IR-perspective (adaption or fidelity centred)? 2. What were the innovations of the BM and in what way were they innovations? 3. What effects have been measured and how have they been explained in studies of the BM?

As to the paper's disposition, this introduction is followed by a section on theory. In that section, we present a theoretical difference between the research fields of IR and PD. The difference concerns the concept of innovation. We also present two concepts used to answer question 1: fidelity to an innovation and adaption of an innovation. Thereafter, in a section about analytical approach, there is a brief description of how the guiding questions are answered. Then an initial analysis is presented, where the main point is that there are much to do in order map all innovations of the BM. In a final section we discuss, with the BM as example, how theories from the fields of IR and PD are related and how that relation can be problematic. We also suggest further empirical studies of the BM.

IR theoretical constructs

In the field of IR, the concept of innovation is central. It is the thing to be implemented. For instance, [START_REF] Century | Implementation research: Finding common ground on what, how, why, where, and who[END_REF], in their overview on IR, defines the field as follows:

… we offer a working definition of implementation research as systematic inquiry regarding innovations enacted in controlled settings or in ordinary practice, the factors that influence innovation enactment, and the relationships between innovations, influential factors, and outcomes. (p. 170)

In contrast, the role of innovations is not as salient in the PD field. Take for instance the following definition in an overview on PD research by [START_REF] Darling-Hammond | Effective teacher professional development[END_REF]:

In this review, we define effective professional development as structured professional learning that results in changes to teacher knowledge and practices, and improvements in student learning outcomes. We conceptualise professional learning as a product of both externally provided and job-embedded activities that increase teachers' knowledge and help them change their instructional practice in ways that support student learning. Thus, formal PD represents a subset of the range of experiences that may result in professional learning. (p. 2) Here, PD is understood as structured professional learning. And if PD changes the knowledge and practice of teachers or student learning, it is effective. Teachers are also supposed to engage in specific activities and be exposed to experiences. No explicit formulations about innovations are made, though. Still, in the seven properties which, according to [START_REF] Darling-Hammond | Effective teacher professional development[END_REF], define effective PD programs, the innovation concept appears in an implicit way. More precisely, teachers are supposed to do or focus on certain things, which can mean that teachers meet something new. For instance, a PD program should focus on discipline-specific pedagogy, teachers should share ideas, experts can give support about evidence for best practice, and there should be time to reflect on best practice. From this we conclude, when the concept of innovation is part of a PD research, it is an implicit part and far from its main focus. Nevertheless, this means there is a potential overlap between IR and PD when it comes to innovations. However, as soon as the impact of PD-projects are measured, the role and nature of the innovations become relevant. In particular, when one enters discussions about causal relations. Our basic thought is this: if there is an effect, there should be a cause. In an educational context, it is thus relevant to understand this cause as something new (an innovation) going into the teachers' mind or into their teaching.

If we now turn to the IR approach to measurement of effects, it clearly involves the innovation concept, but also the fidelity concept. Fidelity in this context means that teachers understand and use the innovation as planned by its proponents. In order to measure the effects of an innovation, we need to know if teachers have used the innovation with fidelity. Following [START_REF] Century | Implementation research: Finding common ground on what, how, why, where, and who[END_REF] overview on IR, fidelity is a matter of degrees. Perfect fidelity is not possible to achieve in practice and is neither the only ideal to pursue. The teaching situation is a complex one, and we must assume that teachers adapt an innovation to local circumstances. Coburn (2003) even points out that to achieve sustainable implementation of an innovation over time (a quality and desirable effect), adaptation can be necessary. However, this does not make the concepts of innovation and fidelity irrelevant when considering effects. It is still important to identify the core components of an innovation, to identify influential factors in the implementation process, and to know in what respect the enactment of the innovation differs from the theoretical ideal [START_REF] Century | Implementation research: Finding common ground on what, how, why, where, and who[END_REF]. Even in cases where the sought effect or outcome is sustainability, which can require adaptation of the given innovation, it remains relevant to compare how the innovation is used in practice with the original plan. Because if the core components of the innovation have been lost, it is not a matter of sustainability. For a longer discussion of the fidelity concept, see [START_REF] Jankvist | How about Fidelity? Implementa tion and Replication[END_REF].

Analytical approach

In order to answer the first guiding question, we draw on results from our previous studies which in part concern the BM. The original contribution of this paper is to apply certain IR concepts (fidelity and adaptation) not used in the previous studies. For the second and third guiding questions, our material consists in reports and research articles about the BM. We then seek to distinguish what the planned innovation of the BM were, but also which effects of the BM have been measured, and how these have been explained. For the second question, we have also used two online search engines to distinguish in what respect the content of the BM material was innovations for teachers. Or if the teachers can have encountered the essential BM content through other channels.

One search engine is Retriever (https://www.retrievergroup.com/sv/product-mediearkivet). Retriever has the biggest media archive in the Nordic countries covering printed and online editorial media as well as radio and television since the 1980s. We have restricted our search to media in the Swedish language. The other engine is the article search engine of the teacher journal Nämnaren (https://ncm.gu.se/namnarenopenaccess-sok). The journal is dedicated to mathematics education and is targeted towards teachers in mathematics. Its search engine only handles Nämnaren material. From Retriever, we wanted to see when important elements of the BM got general media coverage in Sweden. From the search engine for Nämnaren, we wanted to see when important elements of BM were introduced more specifically to teachers. In this way, we can start to discern what were actually the new or innovative important elements of the BM and in what way.

Initial analysis of the Boost for Mathematics project

The BM was in certain respects a top-down project. In the two decades before the BM, Swedish results in international large-scale assessments had decreased very much, which received much media attention. Politicians were forced to act. One of the attempts to counter this development was the BM. But it was not the only attempt, for instance, a new mathematics curriculum was issued in 2011, that is two years before the launch of the BM. During the preparations of the BM, a plausible cause of the poor results was identified: teachers were too seldom leading the teaching and students were too often working alone without supervision and feedback from the teachers. This was reflected in the general goals of BM: to achieve a changed teaching culture and a changed in-service training culture. A new teaching culture meant that teachers should have more teaching methods at their disposal and reflect better about teaching decisions [START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF]. So, initiated by the national government and organised by the National School Agency (Skolverket), the BM was a top-down project.

However, the top-down character did not entail that fidelity was central in the design of the BM. Indeed, there were certain and official goals, but they were quite open; teachers should get more teaching methods (not one) and develop their ability to reflect and make own decisions [START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF]. This openness was further emphasised by the project's basic design principle: peer learning among teachers. This entailed in-service training in the schools, managed by selected senior teachers at the schools. Before the training began, the selected teachers received an eight-day course in mathematics education. These courses were given by university departments specialised in mathematics education [START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF]. Given these circumstances, Prytz (2021) concluded that teachers were given the role of completing the innovations, while the material for the training constituted a context of innovation.

As mentioned in the introduction, it is well-known that the BM was designed according to PD theories [START_REF] Boesen | National-scale professional development in Sweden : theory, policy, practice[END_REF][START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF], which included the peer learning design. For example, in official reports and in decision documents concerning BM, there were explicit references to an international overview on PD research. However, Prytz (2021) pointed out that this free role of the teachers was also a product of a school governance policy that favoured decentralisation and teacher autonomy.

In the light of basic IR theory, we therefore argue that adaption rather than fidelity was central in the BM. Teachers were given freedom to adapt innovations to local circumstances; they were not given specific guidelines they had to abide by.

So, what about innovations in the BM? To understand what the planned innovations were, the training material is an important source. But, it is a vast material and, in this paper, we only scratch the surface. For the training, there were so-called modules for the topics of arithmetic, geometry, algebra, and functions. There were also modules for the topics problem solving, digitalization, and language in mathematics. For all topics, there were sets of modules dedicated to the year-spans 1-3, 4-6, 7-9, and 10-12. All sets of modules also covered four themes: abilities (competencies), formative assessment, interaction, and socio-mathematical norms. The concrete content of the modules was films, audio clips, and texts with instructions and questions for the peer learning sessions. The modules also contained scientific articles: international overviews of each topic and the four themes; and specific articles concerning the same topics and themes in a Swedish context. The modules were distributed via the National School Agency website [START_REF] Prytz | When research met policy: A history of innovation and a complicated relationship in three Swedish development projects in mathematics education, 1960-2018[END_REF].

Participation in the program meant that each group of teachers in a school had to take two sets of modules. The modules in arithmetic were mandatory, while the second was optional, but one set had to be chosen. The modules had been created by researchers working in university departments specialised in mathematics education. In certain respects, the modules per se, in combination with the peer learning idea, can be seen as an innovation of BM. As far as we can see, this type or form of PD was something new in Sweden, at least at this scale. And the innovation was supposed to meet the goals of changing the in-service training culture and the teaching culture.

As to the content of the modules, there were also innovations. At the time of the beginning of the BM, two of the modules' four themes (formative assessment and socio-mathematical norms) were quite new to teachers. We used the two media search engines to see how often the words "formative assessment" and "socio-mathematical norms" were used in Swedish media in general and in the teacher journal Nämnaren in the period of 1980 to 2022. The Swedish words are "formativ bedömning" and "sociomatematiska normer". As to media in general, "formative assessments" began to appear much more frequently after 2009, 11 publications that year, increasing fast to a peak with 266 publications in 2015. In total we got 1617 publications. In Nämnaren we got 37 publications, almost all in 2009 or after, except for two publications. This picture is confirmed by an overview by Hirsch and Lindberg (2015) about Swedish research on formative assessment. According to this overview the number of Swedish PhD theses about formative assessment increased after 2010. In contrast, the expression "sociomathematical norm" was very rare, it hardly appeared in media in general (7 publications in total) or in Nämnaren (4 publications in total). All 11 publications appeared after 2010, with a concentration to the period 2011-2015.

Regarding the concepts of "abilities" and "interaction", it is not meaningful to search for those, since they have a wider meaning and can appear in many contexts. However, we know they appeared already in the 1994 mathematics curriculum; both had a prominent role [START_REF] Prytz | Swedish mathematics curricula, 1850-2014. An overview[END_REF]. We also know that the curriculum messages about abilities and interaction were complex and teachers had difficulties understanding them (Boesen et al, 2014). In the 2011 curriculum reform, a similar type of message was pushed once again [START_REF] Prytz | Swedish mathematics curricula, 1850-2014. An overview[END_REF]. Thus, the BM can be considered a new reform tool in that implementation process. The innovation was then not the ideational content, but the form in which the content was presented to teachers. The fact that a message or a theme was implemented via a curriculum reform can mean that teachers could get further information about it through for instance textbooks, which also were affected by the curriculum reform. This can be compared to how socio-mathematical norms was barely mentioned outside the BM-modules. Our point is that the four themes as innovations appear to have been different in nature (ideational or form); and teachers' possibilities to learn about them through other media may have differed. But, further studies of for instance textbooks and curriculum documents are required to corroborate this point. Further studies of the BM material are also needed to better understand how the four themes were concretized. Now to the measuring and explanations of effects of the BM. Two major studies measuring a changed teacher culture have been made: [START_REF] Ramböl | Slututvärdering av matematiklyftet 2013-2016[END_REF] and Österholm et al. (2016). Both studies were planned parts of the BM. In brief, both studies only involved teacher which participated in the BM; the [START_REF] Ramböl | Slututvärdering av matematiklyftet 2013-2016[END_REF] analysis was focused on organisational aspects, while Österholm et al. (2016) was also focused on ideational aspects and included the modules' four themes (abilities, formative assessment, interaction, and socio-mathematical norms) in the analysis. Both studies indicated that teachers perceived a new culture and a change in teaching. The studies also indicate that the actual teaching had changed. Concerning their explanations, [START_REF] Ramböl | Slututvärdering av matematiklyftet 2013-2016[END_REF] identified the BM in-service training model as a cause of the positive effects. Österholm et al. (2016) could not identify any particular organisational factor causing positive effects; but they concluded that participation in the BM seems to have been the important factor. They also identify that the teaching culture had changed, in varying degrees, along the four themes. However, the possibility that the teachers might have received relevant information about some themes from elsewhere was not considered in the analysis.

Another set of studies consider the BM's effect on student learning: [START_REF] Grönqvist | Lyfter Matematiklyftet matematikkunskaperna?[END_REF], [START_REF] Lindvall | Two large-scale professional development programs for mathematics teachers and their impact on student achievement[END_REF], [START_REF] Lindvall | Impact and design of a national-scale professional development program for mathematics teachers[END_REF][START_REF] Lindvall | Critical features of professional development programs: Comparing content focus and impact of two large-scale programs[END_REF]. [START_REF] Grönqvist | Lyfter Matematiklyftet matematikkunskaperna?[END_REF] was a planned part of the BM. A very brief and general summary of these four studies is that they seek to find out if and in what way the BM and its PD design had an impact on student learning. All four had a controlexperiment-group-design. Students whose teachers had participated in the BM were in the experiment group, while students whose teachers had not participated in the BM were the control group. Different types of test results were used. All four studies discerned positive effects on student learning in some school years, but far from all. To varying degrees the four studies considered the content of the BM and how it affected the teachers and student learning. [START_REF] Lindvall | Critical features of professional development programs: Comparing content focus and impact of two large-scale programs[END_REF] was the most specific and focused only on the arithmetic modules, providing also a detailed content analysis of these modules. A common feature of the other three studies is that the content of the BM is treated as one unity. As to the four BM themes, they are mentioned in all four studies, but they constitute one unit rather than several different innovations. Consequently, the measured effects were caused by this unit. How different innovations played out was not studied.

Conclusions

To measure and assess effects of a large-scale PD program such as the BM is relevant, in certain perspectives. The fact that the BM reached 76% of all Swedish teachers in mathematics and cost nearly 100 million Euros of tax-money are good reasons; the relevant question for tax-payers, politicians, and teachers is then: was it worth it? But also, from a PD research perspective it is relevant, since the basic design of the BM was based on PD theory, more precisely a principle about peer learning among teachers. And the six studies, referred to in this paper, measuring different effects of the BM, do give relevant answers to how the peer learning design functioned and in what way. Still, as we have indicated, from an IR perspective, the measurements of the BM are problematic. In IR, innovation is essential when effects are measured. You need to know what it is, its core components, and you need to know to what degree teachers had fidelity to the innovation. Our initial analysis suggests a number of potential problems in the BM with respect to this:

1. Adaptation was favoured over fidelity. 2. Different types of innovations (ideational or form) were at play at the same time. 3. Innovations could to varying degrees have been mediated through other channels. This makes it difficult to identify which innovations or types of innovations should be considered a cause of the measured effects. This also makes it difficult to discern the degree of fidelity. Add to that our analysis, so far, does not concern innovations connected to mathematical content. Thus, there might be more innovations that can complicate the situation further.

But what makes up a problem from an IR point of view does not need to be a problem within PD research. In this paper, it is also noted that IR and PD theories put different emphasis on the innovation concept; in IR theory it is explicitly positioned as essential, while in PD theory it is more implicitly in the background. Hence, it is not surprising that we in IR see problems in a project based on PD theory. And, as pointed out above, it makes sense in a PD perspective to measure the BM in the way it has been measured. So why should PD people care? The key concept in our answer is replication. If you want to carry out a Boost for Mathematics 2-or at least draw on experiences from the BM in later large-scale projects-it is relevant to retain those innovations that functioned and omit those that did not function well. It then becomes relevant to analyse further what the BM innovations actually were and determine their nature. See for example [START_REF] Koichu | Implementation-related research in mathematics education: the search for identity[END_REF] for a discussion on different natures of innovations (material-centered vs. interaction-centered).

We therefore intend to make further studies into the BM materials (modules) and make comparisons with for instance textbooks and curriculum documents, which we presented at CERME. More precisely, it is possible and relevant to compare BM modules with curriculum reform innovations. We know for instance that the new national curriculum in 2011 had stronger focus on communication, but also problem solving. To such comparisons we can add comparisons of BM modules with popular textbooks published both before and after the 2011 curriculum reform.

Acknowledgements

This paper is part of grant 2020-04090 under the Swedish Research Council.