Ioanna Lykourentzou
email: i.lykourentzou@uu.nl

Antonios Liapis
email: antonios.liapis@um.edu.mt

Costas Papastathis
email: costas@uop.gr

Konstantinos Papangelis
email: k.papangelis@xjtlu.edu.cn

Costas Vassilakis

Exploring Self-organisation in Crowd Teams

Keywords: crowd teams, self-organization, computer-supported collaboration, creative writing

Online crowds have the potential to do more complex work in teams, rather than as individuals. Team formation algorithms typically maximize some notion of global utility of team output by allocating people to teams or tasks. However, decisions made by these algorithms do not consider the decisions or preferences of the people themselves. This paper explores a complementary strategy, which relies on the crowd itself to self-organize into effective teams. Our preliminary results show that users perceive the ability to choose their teammate extremely useful in a crowdsourcing setting. We also find that self-organisation makes users feel more productive, creative and responsible for their work product.

Introduction

As the nature of work is becoming more and more distributed and flexible, creating effective remote teams is becoming an increasingly pertinent problem. Recently, team collaboration has been the subject of increasing research in the crowd work domain. The reason is simple: as the problems that task providers want to delegate to the crowd increase in complexity, individual contributions are not enough, and it becomes evident that people from the crowd need to collaborate. Complex problems where crowd collaboration has proven valuable range from mass scale scientific research and article authoring [START_REF] Vaish | Crowd research: Open and scalable university laboratories[END_REF], to designing software prototypes [START_REF] Retelny | Expert crowdsourcing with flash teams[END_REF], and from writing stories [START_REF] Kim | Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration[END_REF] to collaborative idea generation [START_REF] Siangliulue | Ideahound: Improving largescale collaborative ideation with crowd-powered real-time semantic modeling[END_REF].

Creating an effective team is well-known to be a problem involving multiple challenges. In a typical workplace setting these include balancing the need for skill diversity, personality compatibility and schedules, among other parameters [START_REF] Gilley | Integrated Theoretical Model for Building Effective Teams[END_REF]. Crowd team formation poses further challenges since crowd team mem-bers (i) have usually never worked together before the crowdsourcing task begins, (ii) must perform effectively in relatively little time, and (iii) cannot be assumed to share common values or loyalty to a specific firm (which may be the case when bringing together a remote team from within the same organisation). These challenges mean that the typical methods of team formation, which usually involve pre-profiling team members (in regards to skill, personality etc.), and then placing them to work together, may not suffice in a crowd setting. Instead, new methods are needed to form crowd teams fast and efficiently. In our previous work we proposed "team dating": a method for crowd team formation in cold-start conditions, i.e. assuming nothing about the profile of the individual workers or the way they will collaborate [START_REF] Lykourentzou | Team dating: A self-organized team formation strategy for collaborative crowdsourcing[END_REF]. Team dating allows people to "try out" different candidate teammates for a number of rounds, before placing them into teams with an appropriate teammate of their choice, with the help of an algorithm that takes into account their in-between evaluations. This ad-hoc manner proved to help create efficient ad-hoc teams on a creative task.

Similarly to most existing algorithm-based methods for crowd management (see for example [START_REF] Rahman | Optimized group formation for solving collaborative tasks[END_REF][START_REF] Salehi | Hive: Collective design through network rotation[END_REF]), our previous method suffered from one important disadvantage: not actively involving the workers in the process, but rather assigning them directly to a task or to a team. However, as latest research in management sciences [START_REF] Lawler | Designing Organizations That Are Built to Change[END_REF] and also crowdsourcing [START_REF] Xia | No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work[END_REF] indicates, too close a monitoring can stifle worker creativity and initiative-taking; two features that are absolutely necessary in creative, complex teamwork.

In this work we explore a new concept: self-organisation, which aims to empower crowd workers with the opportunity to choose their teammates, and "guide" the algorithmic process of team formation. Self-organisation is a wellknown concept in domains such as Open Source Software Development, or online content co-creation communities like Wikipedia, but -to the best of our knowledge -it has never been explored in a crowd setting. In this first study, we aim to explore how people behave in such a setting, what choices they make, and how.

The rest of this paper is organised as follows. First, we present related literature, focusing on team formation algorithms for crowdsourcing, and findings from management literature on the nature of self organisation. Then we present our methodology, including the description of the interface and algorithm we used to explore crowd team self-organisation, as well as our data collection method. Next we present our preliminary experimental results, focusing on the way people behave when given the choice to select their teammate. We conclude this paper with limitations and future work.

2 Related Work

Team Formation Algorithms: A trend to micro-manage crowd workers

Team formation in crowdsourcing is often managed by algorithms. Whereas in a traditional work setting a human manager is often enough to decide which person should work with which in the context of a task, the scale of crowdsourcing often necessitates automation. A number of algorithms managing the crowd team formation process have emerged recently, which can be categorized to two broad categories: i) crowd team building algorithms, which select which worker should collaborate with whom before the task begins, and ii) crowd team coordination algorithms, which control the team processes after the task has begun. Crowd team building algorithms, such as the ones proposed in [START_REF] Rahman | Optimized group formation for solving collaborative tasks[END_REF], view team formation as a mathematical optimisation problem. They tap on the scale of crowdsourcing, which makes it impossible for traditional methods (e.g. a human manager) to put together an effective team. Assuming a large pool of workers with known profiles (e.g. skill level) and a very pool of tasks, the objective of crowd team building algorithms is to match each task with a group of workers so as to accomplish the task optimally within given constraints (deadline, upper budget threshold etc.).

Crowd team coordination algorithms come to play after the task has begun. For example, the algorithm proposed by Salehi et al. [START_REF] Salehi | Hive: Collective design through network rotation[END_REF] rotates workers across teams based on their viewpoint diversity, in an effort to achieve idea cross-fertilisation and thus increase the innovation capacity of the participating collective as a whole. Workers however are not asked whether they would like to switch teams or not, and the rotation decision is only taken by the algorithm. Other works in the area are those by Valentine et al. [START_REF] Valentine | Flash organizations: Crowdsourcing complex work by structuring crowds as organizations[END_REF] and Kim et al. [START_REF] Kim | Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration[END_REF], who use computational methods to assign crowd workers to specific parts of the work, either in teams or individually, according to a top-down decision manner and pre-defined roles.

The problem with team management algorithms like the above, is that they largely micro-manage the workers, by assigning them directly to a team. This approach is indeed appropriate for microtask crowdsourcing, where the crowdsourcing task can be clearly delimited to discrete parts and given to specific workers with specific roles. However, when it comes to more complex work, which is usually the type of work that crowd teams are called to address, such algorithms can stifle creativity and initiative-taking, as indicated by recent research in management sciences [START_REF] Lawler | Designing Organizations That Are Built to Change[END_REF] and crowdsourcing [START_REF] Jahanbakhsh | You want me to work with who?: Stakeholder perceptions of automated team formation in project-based courses[END_REF][START_REF] Xia | No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work[END_REF].

Self organization for team building

Self-organisation is a management term often used to describe the functionality of software development teams, either within a company or in Open Source Software Development communities. As part of the Agile Manifesto [START_REF] Beck | The Agile Manifesto[END_REF], it is defined as a process followed by teams that manage their own workload, shift tasks based on needs and best fit, and participate in the group decision making [START_REF] Highsmith | Agile Project Management: Creating Innovative Products[END_REF]. It has been found to improve the performance of participating teams as it "brings the decision making authority to the operational level, thus increasing speed and accuracy of problem solving" [START_REF] Moe | Scrum and team effectiveness: Theory and practice[END_REF].

Self-organizing teams have certain characteristics [START_REF] Takeuchi | The new product development game[END_REF]. First, they are driven by "zero information", where prior knowledge does not apply. This enables the team members to challenge existing knowledge status quo and have the potential to create something truly novel. Second, they exhibit autonomy, as they do not have a top-down appointed leader, but rather leadership is a property that emerges as the team members divide their roles [START_REF] Hoda | Organizing Self-Organizing Teams[END_REF]. Third they exhibit self-transcendence, i.e. the team pursues ambitious goals, and fourth crossfertilization, i.e. team members have a variety of backgrounds, viewpoints and knowledge.

In crowd teams, not a lot of works currently exist giving workers the option to self-organise. An early attempt is the work by Lykourentzou et al. [START_REF] Lykourentzou | Team dating: A self-organized team formation strategy for collaborative crowdsourcing[END_REF] who partially delegate the team building process to the crowd workers themselves, by enabling them to try out different candidate teammates, evaluate them, and then make crowd teams based on these indications. Although this work does take worker preference into account, it does so indirectly.

In this paper we start exploring the notion of self-organisation for crowd teams. Given the effectiveness of the approach on other types of teams, such as corporate ones, we experiment with allowing individuals from the crowd to explore the "space" of candidate teammates available to them, discover those with whom they might work best, through a trial-and-error approach, and finally directly indicate their preferences.

Methodology

Interface Design

Current crowdsourcing platforms do not encourage collaboration, let alone selforganisation and choice of who to work with. Therefore, to explore the properties of self-organisation, and how people behave in this context, we designed a tailormade framework and interface, outlined in Fig. 1.

The task given to the crowd workers is a creative writing challenge inspired by the exquisite corpse method [START_REF] Brotchie | A Book of Surrealist Games[END_REF], where participants co-create a story by gradually building on each others' contributions. To give a sense of competition and inspire motivation, the framework has been designed as a game, played in three rounds. Before the start of the game, a pre-authored story is presented to all users. Then during each round, users work in teams of two and collaborative write a possible continuation to this story. At the end of the round, individuals vote for the best story (they cannot vote for their own team's story), and decide whether they would like to continue with the same teammate or not. In case they want to change, they can indicate their preference for another person, choosing from the list of all possible candidate teammate profiles. The most voted story is then appended to the main story, the team that wrote it receives an award, in the form of a score bonus, and a new round begins.

In the next round, teams are formed with the help of an algorithm, hereby referred to as "self-organisation algorithm", which attempts to best satisfy each user's desired (or undesired) teammate. Each team will have to continue the main story, as it was formed in the previous. This cycle repeats until, at the Start of the Game Once the system is synchronized to all players, the instructions of the game are presented. Afterwards, players are asked about their demographic information, measuring: age, gender, ethnicity, education level, employment status, whether they have won or participated in a similar writing task before and their self-perceived creativity level. Then, each player is presented with the start of the pre-authored story and is asked to individually write a continuation for it. We use this input as a "writing sample", and add it to the profile of the individual, for other users to see

Teammate selection Once players have submitted their individual story writing sample, it is time to choose their first teammate from the list of profiles from all players. A player's profile at this stage consists of the player's username, demographic data, as well as their writing samples (Fig. 1a). In the following stages, where players have already collaborated with and have rated one another, profiles will be enriched to also include each player's ratings, both from the person looking at the profile, and on average. Players can choose up to two possible teammates, with whom they would like to work with. The self-organisation algorithm will use these choices to form the pairs that will be invited to work together in the next round.

Team collaboration As soon as the algorithm has placed users in pairs, based on their profile choices, the teams must collaborate to continue the story so far ("main story"). In the first round, the main story consists of the initial preauthored story. Players' individual writing samples are not taken into account, since all teams must start from the same basis, however each user is free to propose their sample to their teammate, or write something new together. The interface at this stage, shown in Fig. 1b, consist of four main parts: 1) a clock that indicates how long they have for their collaboration, 2) a chat area, serving as their communication channel, 3) the story so far ("main story"), and 4) a collaborative text writing area, where users can see in real time what their teammate writes, and work together to continue the main story.

Peer evaluation Once they complete their collaborative writing (under the "Team Collaboration task) players evaluate their teammate on a Likert scale of 1 to 5 on: Skillfulness ("How skillful was [teammate's username] in continuing the story?"), Collaboration ability ("How good is [teammate's username] as a collaborator?"), Helpfulness ("[teammate's username] comments were helpful").

Players can also rate their own helpfulness level from 1 to 5. Finally, players can tick one or more options regarding how they are similar with their teammate (the options are "commitment to working hard on this task", "how we think the work should be done", "general abilities to do a task like this", "personal values"). These ratings are used to enrich each player's profile, as explained above.

Voting for Best Story After providing feedback on their teammate, each individual player votes for their preferred story continuation. Given a number of P participants in the experiment, there are P/2 candidate story continuations to choose from. Users can vote for any of them, except the one that they coauthored (they cannot see it as an option). Once all players vote, the story with the most votes is presented to them along with the winning team .

Teammate Selection (enriched) In case there are more rounds remaining, users start a new round by first selecting a teammate, in the same way as in Section 3.1. The only difference is that now players are also asked whether they would like to continue with the same teammate or not. This choice is critical for self-organisation. It renders players responsible for calculating the relative gain they will have from continuing with the same teammate (e.g. lower communication overhead, since the team has learnt to work together) versus the risk of losing the chance to work with another teammate that could potentially get them access to a better story in the next round.

Self-organization algorithm for allocating crowd workers to teams The aim of the algorithm is to assist the team self-coordination process, by matching users with those teammates that they mostly prefer working with. The algorithm does so by ranking the possible candidate teams based on their pairwise average profile ratings (affinity), and then gradually selecting those teams with the highest affinity values, until all users have been placed into teams.

End of Game After three rounds, all users are prompted to the "Final Survey" screen, which shows the final story with all winning stories appended, the ranking of the users (based on a point system rewarding each round's winning teams), and a questionnaire that users must fill in regarding their overall experience.

Data Collection

As a first exploratory study, we conducted five experiments with teams of crowdworkers, using only the self-organisation team formation algorithm described above. In total, 34 crowdworkers participated. While more crowdworkers joined each experiment, they swiftly dropped out and were not part of any phase of the team-based tasks; their data is omitted without affecting the validity of the findings. Three of the sessions had 6 participants working in 3 teams, while two session had 8 participants working in 4 teams. The majority of participants selfidentified as aged 18-24 (94%), Asian (91%), male (62%) and with a Bachelors (53%) or High school degree or equivalent (29%). While only a few participants (18%) reported some prior experience with creative writing, most of them reported being highly creative, with an average score of 3.9 out of 5.

Results

Research Questions

A large number of metrics and observations were collected from the five experiments detailed in Section 3.2. Since this paper explores an algorithm that accounts for the crowdworkers' choice for teammate , the analysis in this section will focus on this aspect. Specifically, we wish to analyse the participants' opinions on self-organization as supported in this experiment and implications in their selection of team-mate, relating to the following questions:

1. Do people like being able to choose their teammate? 2. How does having the choice to select who one works with whom, affects participants? 3. What matters the most when having the option to select a teammate? 4. How did players decide which team story to vote?

The questions are analysed at the level of the individual, using the questionnaire data that players filled in at the end of the experiment. Since we are dealing with a single population, we employ the one sample t-test and Chi-Square Goodness-of-Fit Test methods.

Player behavior in a self-organisation context

RQ1: Users find it useful to choose their teammates To answer whether users find it useful to choose their teammates, we conduct a one-sample ttest on their answers to the question "How useful was being able to choose your teammate?" of the final questionnaire, against the hypothetical mean of 3.0, denoting a neutral opinion. Results show that users show a significantly higher than neutral preference to being able to choose their teammate, with t(29) = 2.134, p < 0.005. A mediation analysis revealed that this preference is not affected by being in a winning team or not.

RQ2: Having the choice to select one's teammates makes users feel productive, responsible and creative. Next we examine how having the choice to select one's teammates affects participants. A Chi-Square Goodnessof-Fit Test revealed that there are significant differences among the different ways that this choice could affect users, with x 2 (5) = 12.00, p < 0.005. Specifically, having the choice to select one's teammates made users feel more productive, creative, responsible for what they write and more in control of their work (Table 1). Post-hoc binomial pairwise tests a statistically significant differences across revealed statistical significant differences across all user-reported answers with at least p < 0.05, except between answers "more tired" and "other" where no statistically significant difference was found. No statistically significant differences were found depending on whether the user was in a winning team or not, and whether the user was in the placebo condition or not.

What mattered the most when choosing teammate? Choosing the person that would make me win 27% I chose randomly 13% I chose the people whose initial story I liked the most 27% I chose the people whose profile information I liked the most 27% Other 6%

Table 2: Reasons for choosing a teammate.

RQ3: Users selected teammates based on a variety of reasons -no statistical significance Next we examine what drove the selection of teammates by the users. A Chi-Square Goodness-of-Fit Test revealed that there are no statistically significant differences among the different ways that users select teammates, with x 2 (4) = 5.33, p = 0.26. Player choices from our sample seem to be distributed primarily among selecting a teammate based on winning potential, writing sample and profile, and to a lesser extent randomly or for other reasons (Table 2).

RQ4: Users voted for the story they liked the most, not strategically Next we examine what drove the selection of teammates by the users. A Chi-Square Goodness-of-Fit Test revealed that there are very significant differences among the different ways that users select the winning story, with x 2 (3) = 133.625, p < 0.001. The dominant reason for voting for a story is how much users liked it (Table 3). Users did not vote strategically (i.e. for a worse story in hope that their would win) or randomly. Post-hoc binomial pairwise tests revealed statistical significant differences only across the dominant reason mentioned above ("I voted for the story I liked the most") and the rest, but not among the other reasons.

Sample Session

To shed more light in the process followed by different crowd workers when collaboratively writing in teams, we analyse an indicative session with 8 crowd workers. The session is one of the five sessions analyzed in the quantitative results section above. The winning story of the first round is the one by users shuyuanliu16 and yihao15. Interestingly this story is very unlike to the initial writing sample that any of the two users contributed. The harmonious collaboration of this team was rewarded by other players, and the winning team chose to stay together, with both users rating each other highly (pairwise rating of 4.5 out of 5). From the other competing teams, at least one member declared that they wanted to change teammate. It is also interesting to note that both the two members of the winning team declared to have 3 out of 4 possible collaboration style elements in common ("task commitment", "way of thinking", "general abilities", differing only in the "personal values" element). In contrast, the members of the losing teams either declared no collaboration style element in common ( 2How did you decide which story to vote? I always vote for the story I like the most 83% I sometimes voted for a not good story, to have more chances to win 3% I voted randomly 14% Other 0%

Table 3: Story voting criteria.

out of the 3 losing teams), or they had a large imbalance in the collaboration style elements they declared (in the last team one member declared to share no elements, while the other member declared sharing all four). This observation indicates that the dissimilarity in regards to the collaboration styles may play an important role to the perceptions of the team members and their eventual teams performance. Further research is however needed to statistically examine this observation.

Moving forward to the next rounds we observe that the winning story of the next round is that of a newly formed team, that of users HanbinQin and grizzHuang. Similarly to the previous winning team, this team declared to share many collaboration style elements (4outof 4) and its members rated each other highly (4.25 out of 5). From the losing teams, two out of three also shared at least one collaboration style element, and these teams declared that they wanted to continue working together, despite having lost. The members of these teams also rated each other highly (4.75 out of 5, and 5 out of 5 respectively). The losing team that expressed a wish to change teammates was the team that shared no common collaboration style element, and the members of which rated each other very low (2 out of 5). This qualitative observation also indicates that there seems to be a connection between the number of collaboration style elements a team shares, their inter-team evaluations and their wish to stay together, as well as their performance. Further research is needed to confirm this observation. Given that rest of the teams wanted to stay together, all four teams remained unchanged by the algorithm in the third and final round. The winning story was again by a different team, this time by the team of users nakashimaritsu and Hecate, who had rated each other highly in the previous round. The inter-team ratings and collaboration styles that the teams had declared to share in the previous round remained unchanged.

A final interesting observation with regards to this sample is the fact that the number of total users votes on the winning stories decreases from round to round. After the first round, all 8 users voted for their preferred story. However in rounds 2 and 3 only 6 users voted. This pattern seems to repeat in the rest of our samples. Although users declared that they did not intentionally vote strategically, this finding indicates that perhaps some of them avoided to vote, to give more chances to their team, by not taking any action "against" it. Further research is nevertheless required as to the reasons why some users choose not to vote, and whether this finding is statistically valid or not.

Conclusion, Limitations and Future Work

The results of our exploratory study show that indeed users consider being able to choose their teammates, and thus being able to affect the decision of the team formation process, extremely useful in a crowdsourcing setting. Being involved in the process, and not a mere component of it, made users feel significantly more productive, responsible for their choices and creative. These findings are fully in line with what organisational sciences research has shown, i.e. that involving users (in that case workers) in the decision-making process, helps motivate feelings of empowerment, responsibility and ownership of one's work, with eventual positive effects on team performance.

As an exploratory work, this study has a number of limitations, which can eventually form part of future work. The study only explored the behavior of users in a self-organisation context, without a control condition. This is appropriate to gain insights regarding the process, but it also means that further work is needed to examine, for instance, the differences between self-organised team formation and more traditional team building methods. Moreover, the results of this study relied only on the final questionnaire that users answered. Much richer information can be extracted by quantitatively analysing their pairwise evaluations or their profile choices after each round, and this can also be the object of future work. Such an analysis can help reveal what makes a "winning team" and if these teams share some common characteristics in comparison to non-winning teams. An deeper analysis can also be made on the quality of the team stories, both from those voted as the winning and those not voted, to reveal the extent to which users may vote strategically, even without realising it. Finally, a sentiment and text analysis can be performed in the future on the text produced during the team collaborations, on the shared collaborative document or on the chat. Such an analysis can reveal more fine-grained elements of team collaboration, such as turn-taking styles, signs of social or cultural affinity, trust and common ground building, equality of communication etc. Exploring these elements, especially across the rounds can help reveal the reasons behind some teams persisting while others dissolving, and therefore help understand to a deeper level the properties of human self-organised collaboration.

Fig. 1 :

 1 Fig. 1: Screens from the User Interface

Table 1 :

 1 Having the choice to select who I work with, made me. . . Effects of choice.

	More productive