Danielle Bragg
email: danielle.bragg@microsoft.com

Oscar Koller
email: oscar.koller@microsoft.com

Mary Bellard

Larwan Berke
email: larwan.berke@rit.edu

Patrick Boudreault
email: patrick.boudreault@gallaudet.edu

Annelies Braffort
email: annelies.braffort@limsi.fr

Naomi Caselli

Matt Huenerfauth
email: matt.huenerfauth@rit.edu

Hernisa Kacorri
email: hernisa@umd.edu

Tessa Verhoef
email: t.verhoef@liacs.leidenuniv.nl

Christian Vogler
email: christian.vogler@gallaudet.edu

Meredith Ringel Morris
email: merrie@microsoft.com

Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective

Keywords: sign language; recognition, translation, and generation;

Developing successful sign language recognition, generation, and translation systems requires expertise in a wide range of felds, including computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture. Despite the need for deep interdisciplinary knowledge, existing research occurs in separate disciplinary silos, and tackles separate portions of the sign language processing pipeline. This leads to three key questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the feld? and 3) What are the calls to action for people working in the feld? To help answer these questions, we brought together a diverse group of experts for a two-day workshop. This paper presents the results of that interdisciplinary workshop, providing key background that is often overlooked by computer scientists, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community.

INTRODUCTION

Sign language recognition, generation, and translation is a research area with high potential impact. (For brevity, we refer to these three related topics as "sign language processing" throughout this paper.) According to the World Federation of the Deaf, there are over 300 sign languages used around the world, and 70 million deaf people using them [START_REF]Our Work[END_REF]. Sign languages, like all languages, are naturally evolved, highly structured systems governed by a set of linguistic rules. They are distinct from spoken languages -i.e., American Sign Language (ASL) is not a manual form of English -and do not have standard written forms. However, the vast majority of communications technologies are designed to support spoken or written language (which excludes sign languages), and most hearing people do not know a sign language. As a result, many communication barriers exist for deaf sign language users. Sign language processing would help break down these barriers for sign language users. These technologies would make voice-activated services newly accessible to deaf sign language users -for example, enabling the use of personal assistants (e.g., Siri and Alexa) by training them to respond to people signing. They would also enable the use of text-based systems -for example by translating signed content into written queries for a search engine, or automatically replacing displayed text with sign language videos. Other possibilities include automatic transcription of signed content, which would enable indexing and search of sign language videos, real-time interpreting when human interpreters are not available, and many educational tools and applications.

Current research in sign language processing occurs in disciplinary silos, and as a result does not address the problem comprehensively. For example, there are many computer science publications presenting algorithms for recognizing (and less frequently translating) signed content. The teams creating these algorithms often lack Deaf members with lived experience of the problems the technology could or should solve, and lack knowledge of the linguistic complexities of the language for which their algorithms must account. The algorithms are also often trained on datasets that do not refect real-world use cases. As a result, such single-disciplinary approaches to sign language processing have limited real-world value [START_REF] Erard | Why Sign-Language Gloves Don't Help Deaf People[END_REF].

To overcome these problems, we argue for an interdisciplinary approach to sign language processing. Deaf studies must be included in order to understand the community that the technology is built to serve. Linguistics is essential for identifying the structures of sign languages that algorithms must handle. Natural Language Processing (NLP) and machine translation (MT) provide powerful methods for modeling, analyzing, and translating. Computer vision is required for detecting signed content, and computer graphics are required for generating signed content. Finally, Human-Computer Interaction (HCI) and design are essential for creating end-to-end systems that meet the community's needs and integrate into people's lives.

This work addresses the following questions:

Q1: What is the current state of sign language processing, from an interdisciplinary perspective?

Q2: What are the biggest challenges facing the feld, from an interdisciplinary perspective?

Q3: What calls to action are there for the feld, that resonate across disciplines?

To address these questions, we conducted an interdisciplinary workshop with 39 participants. The workshop brought together academics from diverse backgrounds to synthesize the state-of-the-art in disparate domains, discuss the biggest challenges facing sign language processing efforts, and formulate a call-to-action for the research community. This paper synthesizes the workshop fndings, providing a comprehensive interdisciplinary foundation for future research in sign language processing. The audience for this paper includes both newcomers to sign language processing and experts on a portion of the technology seeking to expand their perspective.

The main contributions of this work are:

• orientation and insights for researchers in any domain, in particular those entering the feld

• highlighting of needs and opportunities for interdisciplinary collaboration

• prioritization of important problems in the feld for researchers to tackle next

BACKGROUND AND RELATED WORK

Building successful sign language processing systems requires an understanding of Deaf culture in order to create systems that align with user needs and desires, and of sign languages to build systems that account for their complex linguistic aspects.

Here, we summarize this background, and we also discuss existing reviews of sign language processing, which do not take a comprehensive view of the problem.

Deaf Culture

Sign language users make up cultural minorities, united by common languages and life experience. Many people view deafness not as a disability, but as a cultural identity [START_REF] Stephen | Deaf Identity Development: Construction and Validation of a Theoretical Model[END_REF] with many advantages [START_REF] Bauman | Deaf Gain: Raising the Stakes for Human Diversity[END_REF]. When capitalized, "Deaf" refers to this cultural identity, while lowercase "deaf" refers to audiological status. Like other cultures, Deaf cultures are characterized by unique sets of norms for interacting and living. Sign languages are a central component of Deaf cultures, their role in Deaf communities even characterized as sacred [START_REF]George Veditz Quote -1913[END_REF]. Consequently, development of sign language processing systems is highly sensitive, and must do the language justice to gain adoption.

Suppression of sign language communication has been a major form of oppression against the Deaf community. Such discrimination is an example of "audism" [START_REF] Bauman | Audism: Exploring the Metaphysics of Oppression[END_REF][START_REF] Eckert | Audism: A Theory and Practice of Audiocentric Privilege[END_REF][START_REF] Humphries | Audism: The Making of a Word[END_REF]. In 1880, an international congress of largely hearing educators of deaf students declared that spoken language should be used for educating deaf children, not sign language [START_REF] Lane | A Chronology of the Oppression of Sign Language in France and the United States[END_REF]. Subsequently, oralism was widely enforced, resulting in training students to lip-read and speak, with varying success. Since then, Deaf communities have fought to use sign languages in schools, work, and public life (e.g., [START_REF] Geers | Early sign language exposure and cochlear implantation benefts[END_REF]). Linguistic work has helped gain respect for sign languages, by establishing them as natural languages [107]. Legislation has also helped establish legal support for sign language education and use (e.g., [START_REF] Un General | Convention on the Rights of Persons with Disabilities[END_REF]). This historical struggle can make development of sign language software particularly sensitive in the Deaf community.

Sign Language Linguistics

Just like spoken languages, sign languages are composed of building blocks, or phonological features, put together under certain rules. The seminal linguistic analysis of a sign language (ASL) revealed that each sign has three main phonological features: handshape, location on the body, and movement [106]. More recent analyses of sign languages offer more sophisticated and detailed phonological analyses [START_REF] Brentari | Trilled Movement: Phonetic Realization and Formal Representation[END_REF]114,[START_REF] Sandler | Sign Language and Linguistic Universals[END_REF][START_REF] Brentari | Sign Language Phonology[END_REF]. While phonological features are not always meaningful (e.g., the bent index fnger in the sign APPLE does not mean anything on its own), they can be [START_REF] Brentari | Handshape in Sign Language Phonology[END_REF]. For example, in some cases the movement of the sign has a grammatical function.

In particular, the direction of movement in verbs can indicate the subject and object of the sentence.

Classifers represent classes of nouns and verbs -e.g., one handshape in ASL is used for vehicles, another for fat objects, and others for grabbing objects of particular shapes. The vehicle handshape could be combined with a swerving upward movement to mean a vehicle swerving uphill, or a jittery straight movement for driving over gravel. Replacing the handshape could indicate a person walking instead. These handshapes, movements, and locations are not reserved exclusively for classifers, and can appear in other signs. Recognition software must differentiate between such usages.

Fingerspelling, where a spoken/written word is spelled out using handshapes representing letters, is prevalent in many sign languages. For example, fngerspelling is often used for the names of people or organizations taken from spoken language. Its execution is subject to a high degree of coarticulation, where handshapes change depending on the neighboring letters [START_REF] Keane | Coarticulation in ASL Fingerspelling[END_REF]. Recognition software must be able to identify when a handshape is used for fngerspelling vs. other functionalities. Sign languages also make extensive use of depiction: using the body to depict an action (e.g., showing how one would fllet a fsh), dialogue, or psychological events [START_REF] Dudis | Depiction of events in ASL: Conceptual integration of temporal components[END_REF]. Subtle shifts in body positioning and eye gaze can be used to indicate a referent. Sign language recognition software must accurately detect these non-manual components.

There is great diversity in sign language execution, based on ethnicity, geographic region, age, gender, education, language profciency, hearing status, etc. As in spoken language, different social and geographic communities use different varieties of sign languages (e.g., Black ASL is a distinct dialect of ASL [START_REF] Mccaskill | The hidden treasure of Black ASL: Its history and structure[END_REF]). Unlike spoken languages, sign languages contain enormous variance in fuency. Most deaf children are born to hearing parents, who may not know sign language when the child is born. Consequently, most deaf sign language users learn the language in late childhood or adulthood, typically resulting in lower fuency [START_REF] Rachel | Rethinking the critical period for language: New insights into an old question from American Sign Language[END_REF]. Sign language processing software must accurately model and detect this variety, which increases the amount and variety of training data required.

It is diffcult to estimate sign language vocabulary size. Existing ASL-to-English dictionaries contain 5-10k signs (e.g., [103]). However, they are not representative of true size, as they lack classifers, depictions, and other ways signs are modulated to add adjectives, adverbs, and nuanced meaning.

Reviews

Existing reviews of sign language processing are largely technical and out-of-date, written before the advent of deep learning. Most reviews focus on a specifc subarea, such as the software and hardware used to recognize signs [START_REF] Ebrahim | Review in sign language recognition systems[END_REF]117,[START_REF] Joudaki | Vision-based sign language classifcation: a directional review[END_REF][START_REF] Cooper | Sign Language Recognition[END_REF]. Few reviews discuss multiple subfelds of sign language technologies (e.g., recognition, translation, and generation).

In this work, we provide a broader perspective that highlights common needs (e.g., datasets), and applications that blend multiple technologies. Unlike past reviews, we also articulate a call to action for the community, helping to prioritize problems facing the feld.

Existing reviews also incorporate limited perspectives outside of computer science. In particular, few reviews incorporate the linguistic, social, and design perspectives needed to build sign language systems with real-world use. Some reviews consider a related discipline (e.g., linguistics in [START_REF] Cooper | Sign Language Recognition[END_REF]), but do not consider the full spectrum of disciplines involved. This work integrates diverse interdisciplinary perspectives throughout, providing deeper insight into how technologies align with human experiences, the challenges facing the feld, and opportunities for collaboration.

There is already interest among researchers in various felds in applying their methods to sign language applications. In particular, some technical reviews of gesture recognition touch on sign language recognition as an application domain (e.g., [120,[START_REF] Grs Murthy | A Review of Vision Vased Hand Gestures Recognition[END_REF]108,[START_REF] Zaman Khan | Hand Gesture Recognition: a Literature Review[END_REF]). These reviews focus on algorithms for detecting fngers, hands, and human gestures. However, by framing sign language recognition as an application area, they risk misrepresenting sign language recognition as a gesture recognition problem, ignoring the complexity of sign languages as well as the broader social context within which such systems must function. In this work, we provide linguistic and cultural context in conjunction with algorithms, to establish a more accurate representation of the space.

METHOD

To help answer our guiding questions, we convened a two-day workshop with leading experts in sign language processing and related felds. Many of these participants continued on to synthesize the workshop fndings in this paper.

Participants

A total of 39 workshop attendees were recruited from universities and schools [START_REF] Brentari | Handshape in Sign Language Phonology[END_REF] and a tech company [START_REF] Camgoz | Neural Sign Language Translation[END_REF]. Academic participants were based in departments spanning computer science, linguistics, education, psychology, and Deaf studies. Within computer science, specialists were present from computer vision, speech recognition, machine translation, machine learning, signal processing, natural language processing, computer graphics, human-computer interaction, and accessibility. Attendees from the tech company had roles in research, engineering, and program/product management. The number of participants present varied slightly over the two days.

Participants were demographically diverse:

• Nationality: Attendees were an international group, currently based in the U.S., Europe, and Asia.

• Experience: Career level ranged from recent college graduates through senior professors and executives.

• Gender: 25 male, 14 female

• Audiological status: 29 hearing (6 with Deaf immediate family members), 10 Deaf and hard of hearing (DHH)

Procedure

The workshop activities were structured to facilitate progress toward our three guiding questions (current landscape, biggest challenges, and calls to action). Day 1 provided the necessary domain-specifc background knowledge, and Day 2 addressed our three questions as an interdisciplinary group. Interpreters and captioners were available throughout.

Day 1: Sharing interdisciplinary domain knowledge.

• Domain Lectures: A series of 45-minute talks, covering relevant domains and given by domain experts: Deaf culture, sign language linguistics, natural language processing, computer vision, computer graphics, and dataset curation.

• Panel: A 45-minute panel on Deaf users' experiences with, needs for, and concerns about technology, with a Deaf moderator and four Deaf panelists.

Day 2: Discussing problems and mapping the path forward.

• Breakout Sessions: Participants divided into smaller groups (8-9/group) to discuss specifc topics, for 3.5 hours.

The topic areas, outlined by the organizers and voted on by participants, were: • Breakout Presentations: Each breakout group reported back on their topic, through a slide presentation mixed with discussion with the larger group (about 20 minutes each).

In the following sections, we summarize the content generated through this workshop exercise, organized by our three guiding questions.

Q1: WHAT IS THE CURRENT LANDSCAPE?

In this section, we synthesize each group's formulation of the current state-of-the-art. We note that some topics overlap. In particular, data is central to progress on all fronts, so we start with a summary of the landscape in sign language datasets.

Datasets

Existing sign language datasets typically consist of videos of people signing. Video format can vary, and is often dependent on the recording device. For example, video cameras often produce MP4, OGG, or AVI format (among others). Motioncapture datasets have been curated, often by attaching sensors to a signer (e.g., [START_REF] Lu | Collecting a motion-capture corpus of American Sign Language for data-driven generation research[END_REF][START_REF] Heloir | Captured Motion Data Processing for Real Time Synthesis of Sign Language[END_REF][START_REF] Berret | Collecting and Analysing a Motion-Capture Corpus of French Sign Language[END_REF]). These datasets can be pulled from to generate signing avatars, and are often curated for this purpose. Depth cameras can also be used to capture 3D positioning. For example, the Kinect includes a depth sensor and has been used to capture sign language data (e.g., [START_REF]Chinese Sign Language Recognition Dataset[END_REF][START_REF] Cooper | Sign Language Recognition Using Sub-Units[END_REF][START_REF] Oszust | Polish sign language words recognition with kinect[END_REF]). Table 1 summarizes public sign language video corpora commonly used for sign language recognition. (See [START_REF] Reiner | Sign language corpora survey[END_REF]'s survey for a more complete list of datasets, many of which are intended for linguistic research and education.)

The data collection method impacts both content and signer identity. For example, some corpora are formed of professional interpreters paid to interpret spoken content, such as news channels that provide interpreting [START_REF] Forster | Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather[END_REF][START_REF] Koller | Re-Sign: Re-Aligned End-To-End Sequence Modelling With Deep Recurrent CNN-HMMs[END_REF][START_REF] Camgoz | Neural Sign Language Translation[END_REF]. Others are formed of expert signers paid to sign desired corpus content (e.g., [START_REF] Kak | Purdue RVL-SLLL ASL Database for Automatic Recognition of American Sign Language[END_REF]124,118]). Yet other corpora consist sign language videos posted on sites such as YouTube (e.g., [START_REF] Reza | MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language[END_REF]) -these posters may be fuent signers, interpreters, or sign language students; such videos are typically of "real-life" signs (i.e., self-generated rather than prompted). The geographic region where the data is collected also dictates which sign language is captured. Many datasets have been curated by researchers and startups in the U.S., where ASL is the primary language of the Deaf community, and consequently contain ASL. Fewer datasets have been curated of other sign languages, though some exist (e.g., [START_REF] Matthes | Dicta-Sign-building a multilingual sign language corpus[END_REF]). The vocabulary size of these datasets varies from about 100-2,000 distinct signs (see Table 1).

Annotations may accompany video corpora. These annotations can demarcate components of signs (e.g., handshapes and movements), the identity and ordering of the signs, or a translation into another language like English. These annotations can take various formats, including linguistic notation systems (for sign components), English gloss (for sign identity and order), and English text (for translations). The annotations can be aligned at various levels of granularity. For example, the start and end of a handshape could be labeled, or the start and end of a full sentence. Generating annotations can be very time-intensive and expensive. Annotation software has been developed to facilitate annotating videos, and is often used by linguists studying the language (e.g., ELAN [112] and Anvil [START_REF] Michael | Anvil[END_REF]). Because sign languages do not have a standard written form, large text corpora do not exist independent of videos.

Recognition & Computer Vision

Glove-based approaches to sign language recognition have been used to circumvent computer vision problems involved in recognizing signs from video. The frst known work dates back to 1983, with a patent describing an electronic glove that recognized ASL fngerspellings based on a hardwired circuit [START_REF] Gary | Digital Data Entry Glove Interface Device[END_REF]. Since then, many other systems have been built for "intrusive sign recognition," where signers are required to wear gloves, clothing, or other sensors to facilitate recognition (e.g., [START_REF] Charayaphan | Image Processing System for Interpreting Motion in American Sign Language[END_REF][START_REF] Fels | Glove-Talk: A Neural Network Interface between a Data-Glove and a Speech Synthesizer[END_REF][START_REF] Liang | A Real-Time Continuous Gesture Recognition System for Sign Language[END_REF][START_REF] Oz | American Sign Language Word Recognition with a Sensory Glove Using Artifcial Neural Networks[END_REF][START_REF] Cooper | Sign Language Recognition[END_REF]).

Non-intrusive vision-based sign language recognition is the current dominant approach. Such systems minimize inconvenience to the signer (and, unlike gloves, have the potential to incorporate non-manual aspects of signing), but introduce complex computer vision problems. The frst such work dates back to 1988, when Tamura et al.

[110] built a system to recognize 14 isolated signs of Japanese Sign Language using skin color thresholding. As in that seminal work, many other systems focus on identifying individual signs (e.g., [START_REF] Grobel | Isolated sign language recognition using hidden Markov models[END_REF][START_REF] Lang | Sign language recognition using kinect[END_REF]).

Real-world translation typically requires continuous sign language recognition [105,[START_REF] Forster | Modality Combination Techniques for Continuous Sign Language Recognition[END_REF], where a continuous stream of signing is deciphered. Continuous recognition is a signifcantly more challenging and realistic problem than recognizing individual signs, confounded by epenthesis effects (insertion of extra features into signs), co-articulation (the ending of one sign affecting the start of the next), and spontaneous sign production (which may include slang, non-uniform speed, etc.).

To address the three-dimensionality of signs, some visionbased approaches use depth cameras [113,123], multiple cameras [START_REF] Brashear | Using Multiple Sensors for Mobile Sign Language Recognition[END_REF] or triangulation for 3D reconstruction [START_REF] Schmidt | Using Viseme Recognition to Improve a Sign Language Translation System[END_REF][START_REF] Schmidt | Enhancing Gloss-Based Corpora with Facial Features Using Active Appearance Models[END_REF]. Some use colored gloves to ease hand and fnger tracking [START_REF] Cooper | Sign Language Recognition Using Linguistically Derived Sub-Units[END_REF].

Recent advances in machine learning -i.e., deep learning and convolutional neural networks (CNNs) -have improved state-of-the-art computer vision approaches [START_REF] Koller | Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs[END_REF] A real-life continuous sign language video dataset has also been released [START_REF] Forster | Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather[END_REF], and is used as a community benchmark. Given utterance-or sentence-level segmentation, recognition systems can reliably identify sign boundaries [START_REF] Koller | Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos[END_REF].

For such challenging datasets (still only covering a vocabulary of around 1,000 different signs), recognition algorithms can achieve a Word Error Rate (WER) of 22.9% [START_REF] Cui | A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training[END_REF] when trained and tested on the same signers, and a WER of 39.6% when trained and tested on different sets of signers.

Modeling & Natural Language Processing

Because sign languages are minority languages lacking data, the vast majority of work in MT and NLP focuses on spoken and written languages, not sign languages. While recognition handles the problem of identifying words or signs from complex signals (audio or video), MT and NLP typically address problems of processing language that has already been identifed. These methods expect annotated data as input, which for spoken languages is commonly text (e.g., books, newspapers, or scraped text from the internet). Translation between spoken and signed languages (and vice versa) also typically requires intermediary representations of the languages that are computationally compatible.

Various notation systems are used for computational modeling. Human-generated annotations are often in gloss, a form of transliteration where written words in another language (e.g., English) are used to represents signs (e.g., ASL). Other writing systems have also been developed for people to use, including si5s [START_REF] Clark | How to Write American Sign Language[END_REF] and SignWriting [109]. Separate notation systems have been developed for computers to represent sign languages during modeling and computation; HamNoSys [START_REF] Hanke | HamNoSys-representing sign language data in language resources and language processing contexts[END_REF] is one of the most popular, designed to capture detailed human movements and body positioning for computer modeling. To facilitate structured storage, XML-based markup languages have also been developed, e.g. Signing Gesture Markup Language (SiGML) [START_REF] Elliott | The development of language processing support for the ViSiCAST project[END_REF], which is compatible with HamNoSys.

Sign language translation systems can either use predefned intermediary representations of the languages involved, or learn their own representations (which may not be humanunderstandable). Methods that use predefned representations are highly compatible with grammatical translation rules (e.g., [START_REF] Davydov | Information system for translation into Ukrainian sign language on mobile devices[END_REF]125,[START_REF] Elliott | The development of language processing support for the ViSiCAST project[END_REF]116]). Methods that do not use such representations typically use some form of deep learning or neural networks, which learn model features (i.e. internal representations) that suit the problem. These methods have been used for recognition combined with translation, processing complete written sentences in parallel with signed sentences [START_REF] Koller | Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs[END_REF][START_REF] Koller | Deep Sign: Hybrid CNN-HMM for Continuous Sign Language Recognition[END_REF][START_REF] Camgoz | Neural Sign Language Translation[END_REF]. Such techniques are often used in computer vision systems, and overlap with works presented in the previous section.

Avatars & Computer Graphics

Sign language avatars (computer animations of humans) can provide content in sign language, making information accessible to DHH individuals who prefer sign language or have lower literacy in written language [START_REF] Huenerfauth | Sign language in the interface: access for deaf signers[END_REF]. Because sign languages are not typically written, these videos can be preferable to text. Videos of human signers or artist-produced animations provide similar viewing experiences, but avatars are more appropriate when automatic generation is desirable (e.g., for a website with unstable content). Current pipelines typically generate avatars based on a symbolic representation of the signed content prepared by a human author (e.g. [START_REF] Al-Khazraji | Modeling the Speed and Timing of American Sign Language to Generate Realistic Animations[END_REF][START_REF] Adamo | ASL-Pro: American Sign Language Animation with Prosodic Elements[END_REF][START_REF] Braffort | KAZOO: A Sign Language Generation Platform Based on Production Rules[END_REF]115,[START_REF] Bangham | Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project[END_REF]). When the avatar is generated as part of a translation system (e.g., [START_REF] Karpouzis | Educational resources and implementation of a Greek sign language synthesis architecture[END_REF][START_REF] Elliott | Linguistic modelling and language-processing technologies for Avatar-based sign language presentation[END_REF]), an initial translation step converts spoken/written language into a symbolic representation of the sign language (as described in the previous section). Whether human-authored or automatically translated, a symbolic plan is needed for the sign-language message. While multiple representations have been proposed (e.g. [START_REF] Elliott | The development of language processing support for the ViSiCAST project[END_REF][START_REF] Braffort | KAZOO: A Sign Language Generation Platform Based on Production Rules[END_REF][START_REF] Adamo | ASL-Pro: American Sign Language Animation with Prosodic Elements[END_REF]), there is no universal standard.

Beginning with this symbolic plan, pipelines generating avatars typically involve a series of complex steps (e.g., as outlined in [START_REF] Huenerfauth | Sign language in the interface: access for deaf signers[END_REF][START_REF] Gibet | The SignCom system for data-driven animation of interactive virtual signers: Methodology and Evaluation[END_REF]). Animations for individual signs are often pulled from lexicons of individual signs. These motion-plans for the individual signs are produced in one of several ways: key-frame animations (e.g. [START_REF] Huenerfauth | Release of Experimental Stimuli and Questions for Evaluating Facial Expressions in Animations of American Sign Language[END_REF]), symbolic encoding of subsign elements (e.g. [START_REF] Ebling | Building a Swiss German Sign Language avatar with JASigning and evaluating it among the Deaf community[END_REF]), or motion-capture recordings (e.g., [100,[START_REF] Gibet | The SignCom system for data-driven animation of interactive virtual signers: Methodology and Evaluation[END_REF]). Similarly, non-manual signals are pulled from complementary datasets (e.g., [START_REF] Huenerfauth | Generating American Sign Language classifer predicates for English-to-ASL machine translation[END_REF]) or synthesized from models (e.g., [START_REF] Kacorri | Continuous Profle Models in ASL Syntactic Facial Expression Synthesis[END_REF]). These elements are combined to create an initial motion script of the content. Next, various parameters (e.g., speed, timing) are set by a human, set by a rule-based approach (e.g. [START_REF] Ebling | Building a Swiss German Sign Language avatar with JASigning and evaluating it among the Deaf community[END_REF]), or predicted via a trained machine-learning model (e.g., [START_REF] Al-Khazraji | Modeling the Speed and Timing of American Sign Language to Generate Realistic Animations[END_REF][START_REF] Lu | Collecting a motion-capture corpus of American Sign Language for data-driven generation research[END_REF]). Finally, computer animation software renders the animation based on this detailed movement plan.

The state-of-the-art in avatar generation is not fully automated; all parts of current pipelines currently require human intervention to generate smooth, coherent signing avatars. Prior research has measured the quality of avatar animations via perceptual and comprehension studies with DHH participants [START_REF] Huenerfauth | Evaluating American Sign Language Generation Through the Participation of Native ASL Signers[END_REF], including methodological research [START_REF] Kacorri | Regression Analysis of Demographic and Technology-Experience Factors Infuencing Acceptance of Sign Language Animation[END_REF] and shared resources for conducting evaluation studies [START_REF] Huenerfauth | Release of Experimental Stimuli and Questions for Evaluating Facial Expressions in Animations of American Sign Language[END_REF].

UI/UX Design

The state-of-the-art of sign language output in user interfaces primarily centers around systems that use sign language video or animation content (e.g., computer-generated human avatars) to display information content. (Surveys of older work appear in [START_REF] Huenerfauth | Sign language in the interface: access for deaf signers[END_REF] and [START_REF] Al-Khazraji | Modeling the Speed and Timing of American Sign Language to Generate Realistic Animations[END_REF].) These projects include systems that provide sign language animation content on webpages to supplement text content for users. There has also been some work on providing on-demand defnitions of terminology in ASL (e.g., by linking to ASL dictionary resources [START_REF] Hariharan | Evaluation of an English Word Look-Up Tool for Web-Browsing with Sign Language Video for Deaf Readers[END_REF]). As discussed in [START_REF] Hariharan | Evaluation of an English Word Look-Up Tool for Web-Browsing with Sign Language Video for Deaf Readers[END_REF], prior work has found that displaying static images of signs provides limited beneft, and generally users have preferred interfaces that combine both text and sign content.

Research on designing interactive systems with sign language recognition technologies has primarily investigated how to create useful applications despite the limited accuracy and coverage of current technology for this task. This has often included research on tools for students learning ASL, either young children (e.g., [122]) or older students who are provided with feedback as to whether their signing is accurate [START_REF] Huenerfauth | Evaluation of language feedback methods for student videos of american sign language[END_REF].

While there have been various short-lived projects and specifc industry efforts to create tools that can recognize full phrases of ASL to provide communication assistance, few systems are robust enough for real-world deployment or use.

Q2: WHAT ARE THE FIELD'S BIGGEST CHALLENGES?

In this section, we summarize the major challenges facing the feld, identifed by the interdisciplinary breakout groups.

Datasets

Public sign language datasets have shortcomings that limit the power and generalizability of systems trained on them.

Size: Modern, data-driven machine learning techniques work best in data-rich scenarios. Success in speech recognition, which in many ways is analogous to sign recognition, has been made possible by training on corpora containing millions of words. In contrast, sign language corpora, which are needed to fuel the development of sign language recognition, are several orders of magnitude smaller, typically containing fewer than 100,000 articulated signs. (See Table 2 for a comparison between speech and sign language datasets.)

Continuous Signing: Many existing sign language datasets contain individual signs. Isolated sign training data may be important for certain scenarios (i.e., creating a sign language dictionary), but most real-world use cases of sign language processing involve natural conversational with complete sentences and longer utterances.

Native Signers: Many datasets allow novices (i.e., students) to contribute, or contain data scraped from online sources (e.g., YouTube [START_REF] Reza | MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language[END_REF]) where signer provenance and skill is unknown. Professional interpreters, who are highly skilled but are often not native signers, are also used in many datasets (e.g., [START_REF] Forster | RWTH-PHOENIX-Weather: A Large Vocabulary Sign Language Recognition and Translation Corpus[END_REF]).

The act of interpreting also changes the execution (e.g., by simplifying the style and vocabulary, or signing slower for understandability). Datasets of native signers are needed to build models that refect this core user group.

Signer Variety: The small size of current signing datasets and over-reliance on content from interpreters mean that current datasets typically lack signer variety. To accurately refect the signing population and realistic recognition scenarios, datasets should include signers that vary by: gender, age, clothing, geography, culture, skin tone, body proportions, disability, fuency, background scenery, lighting conditions, camera quality, and camera angles. It is also crucial to have signer-independent datasets, which allow people to assess generalizability by training and testing on different signers. Datasets must also be generated for different sign languages (i.e., in addition to ASL).

Recognition & Computer Vision

Despite the large improvements in recent years, there are still many important and unsolved recognition problems, which hinder real-world applicability. Annotations: Producing sign language annotations, the machine-readable inputs needed for supervised training of AI models, is time consuming and error prone. There is no standardized annotation system or level of annotation granularity. As a result, researchers are prevented from combining annotated datasets to increase power, and must handle low inter-annotator agreement. Annotators must also be trained extensively to reach suffcient profciency in the desired annotation system. Training is expensive, and constrains the set of people who can provide annotations beyond the already restricted set of fuent signers. The lack of a standard written form also prevents learning from naturally generated texte.g., NLP methods that expect text input, using parallel text corpora to learn corresponding grammar and vocabulary, and more generally leveraging ubiquitous text resources.

Generalization: Generalization to unseen situations and individuals is a major diffculty of machine learning, and sign language recognition is no exception. Larger, more diverse datasets are essential for training generalizable models. We outlined key characteristics of such datasets in the prior section on dataset challenges. However, generating such datasets can be extremely time-consuming and expensive.

Modeling & Natural Language Processing

The main challenge facing modeling and NLP is the inability to apply powerful methods used for spoken/written languages due to language structure differences and lack of annotations. 2. Comparison of sign language vs. speech datasets. Existing sign language corpora are orders of magnitude smaller than speech corpora. Because sign languages are not typically written, parallel written corpora do not exist for sign languages, as they do for spoken (and written) languages.

Structural Complexity: Many MT and NLP methods were developed for spoken/written languages. However, sign languages have a number of structural differences from these languages. These differences mean that straightforward application of MT and NLP methods will fail to capture some aspects of sign languages or simply not work. In particular, many methods assume that one word or concept is executed at a time. However, many sign languages are multi-channel, for instance conveying an object and its description simultaneously. Many methods also assume that context does not change the word being uttered; however, in sign languages, content can be spatially organized and interpretation directly dependent on that spatial context.

Annotations: Lack of reliable, large-scale annotations are a barrier to applying powerful MT and NLP methods to sign languages. These methods typically take annotations as input, commonly text. Because sign languages do not have a standard written form or a standard annotation form, we do not have large-scale annotations to feed these methods. Lack of large-scale annotated data is similarly a problem for training recognition systems, as described in the previous section.

Avatars & Computer Graphics

Avatar generation faces a number of technical challenges in creating avatars that are acceptable to Deaf users (i.e., pleasing to view, easy to understand, representative of the Deaf community, etc.). Some of these problems may be addressed by including Deaf people in the generation process [START_REF] Kipp | Assessing the Deaf User Perspective on Sign Language Avatars[END_REF].

Uncanny Valley: Sign language avatars are subject to an uncanny valley [START_REF] Mori | The uncanny valley [from the feld[END_REF]. Avatars that are either very cartoonish or very human-like are fairly pleasing, but in-between can be disconcerting. For example, in addition to providing semantically meaningful non-manual cues (e.g., raised eyebrows indicating a question), avatars must also have varied, natural facial expressions (i.e., not a robotic, stoic expression throughout). It can be diffcult to design avatars that fall outside of this valley.

Realistic Transitions: To illustrate why transitions between signs are diffcult, consider a generation system that pulls from motion-capture libraries. The system can pull complete sign executions, but must then piece together these executions. One sign might end with the hands in one position, while the subsequent sign starts with the hands in another position, and the software must create a smooth, natural transition between.

Modeling Modulations:

In some sign languages, adjectives and adverbs are executed by modulating a noun or verb. For example, in ASL, PLANE RIDE is executed by moving a certain handshape through the air. BUMPY PLANE RIDE is identical, but with the movement made bumpy. Infnite such descriptors can be executed, and capturing them all in a motion-capture database is infeasible. Acceptable abstractions have not been standardized (e.g., in a writing system), so it is unclear how much real-life variation avatars must portray.

Finding Model Holes: It is diffcult to fnd holes in generation models, because the language space is large and rich, and the number of ways that signs can be combined in sequence grows exponentially. Testing all grammatical structures empirically is not scalable. This "unknown unknown" problem is common to other machine learning areas (e.g., speech recognition [START_REF] Hermansky | Multistream recognition of speech: Dealing with unknown unknowns[END_REF]).

Public Motion-Capture Datasets Many motion-capture datasets used for avatar generation are owned by particular companies or research groups. Because they are not publicly available, research in this area can be impeded.

UI/UX Design

Sign language UI/UX design is currently confounded by technical limitations that require carefully scoped projects, many potential use cases requiring different solutions, and design choices that may have powerful social ramifcations.

Technical Limitations A long-term goal in this space is full universal design of conversational agents. For example, if a system supports speech-based or text chat interaction, then it should also support input and output in sign language. However, given the current limitations of the component technologies, it may be useful for researchers to focus on more nearterm research aims: for instance, if we have a sign language recognition system capable of recognizing some fnite number of signs or phrases, then what types of applications can be supported within this limit (for different vocabulary sizes)?

Varied Use Cases: There are a huge number of use cases for sign language processing, requiring different interface designs. For example, sign language recognition could be useful for placing a meal order in a drive-through restaurant, or for commanding a personal assistant. Similarly, sign language generation may be used in various situations. For people who want to create websites that present signed content, avatars may be the most reasonable solution, as they allow for ease in editability, creation from anywhere, and scalability (cost). However, people also want websites to be searchable and indexable, and videos and animations are diffcult for current text-based search engines to index and search. Combining text and video introduces layout problems, especially when text is automatically replaced with video. These situations, and many others, have drastically different design criteria.

Language and Dialect Choice: Many different sign languages exist, with many dialects for each. Choosing which one(s) a system will recognize or portray is a diffcult problem with societal implications. Minorities within Deaf communities may be further marginalized if their dialects are not represented. Similarly, failure to represent other types of diversity -e.g., gender, race, education level, etc. -could also be detrimental.

Q3: WHAT ARE THE CALLS TO ACTION?

In this section, we outline an interdisciplinary call to action for the research community working on any piece of the endto-end sign language processing pipeline. Once stated, these calls to action may seem intuitive, but have not previously been articulated, and have until now been largely disregarded.

Deaf Involvement

In developing sign language processing, Deaf community involvement is essential at all levels, in order to design systems that actually match user needs, are usable, and to facilitate adoption of the technology. An all-hearing team lacks the lived experience of Deafness, and is removed from the use cases and contexts within which sign language software must function. Even hearing people with strong ties to the Deaf community are not in a position to speak for Deaf needs. Additionally, because of their perceived expertise in Deaf matters, they are especially susceptible to being involved in Deaf-hearing power imbalances. People who do not know a sign language also typically make incorrect assumptions about sign languagese.g., assuming that a particular gesture always translates to a particular spoken/written word. As a result, all-hearing teams are ill-equipped to design software that will be truly useful.

It is also important to recognize individual and community freedoms in adopting technology. Pushing a technology can lead to community resentment, as in the case of cochlear implants for many members of sign language communities [104]. Disrespecting the Deaf community's ownership over sign languages also furthers a history of audism and exclusion, which can result in the Deaf community rejecting the technology. For these reasons, a number of systems built by hearing teams to serve the Deaf community have failed or receive mixed reception (e.g., sign language gloves [START_REF] Erard | Why Sign-Language Gloves Don't Help Deaf People[END_REF]).

Deaf contributors are essential at every step of research and development. For example, involvement in the creation, evaluation, and ownership of sign language datasets is paramount to creating high-quality data that accurately represents the community, can address meaningful problems, and avoids cultural appropriation. Future datasets might take cultural competency into account by 1) being open-source and publicly available, 2) providing cultural context for challenges to ensure that computer vision experts competing on algorithmic performance understand the nature, complexity, and history of sign languages, and/or 3) providing more appropriate metrics developed by the Deaf community, beyond the current standard of WER. Similarly, Deaf community involvement is fundamental to the creation of appropriate computational models, interface design, and overall systems.

The importance of Deaf involvement is heightened by technology's impact on language. Written English is evolving right now with new spellings based on technological constraints like character limits on Twitter, and diffculty typing long phrases on phone keyboards. It is possible that signers would similarly adapt sign languages to better suit the constraints of computing technologies. For example, people might simplify vocabulary to aid recognition software, constrict range of motion to ft the technical limits of video communications [START_REF] Keating | Cybersign and new Proximities: Impacts of New Communication Technologies on Space and Language[END_REF], or abstract away richness to support standardized writing or annotation.

Call 1: Involve Deaf team members throughout.

Deaf involvement and leadership are crucial for designing systems that are useful to users, respecting Deaf ownership of sign languages, and securing adoption.

Application Domain

There are many different application domains for sign language processing. Situations where an interpreter would be benefcial but is not available are one class of applications. This includes any point of sale, restaurant service, and daily spontaneous interactions (for instance with a landlord, colleagues, or strangers). Developing personal assistant technologies that can respond to sign language is another compelling application area. Each of these scenarios requires different solutions. Furthermore, these different use cases impose unique constraints on every part of the pipeline, including the content, format, and size of training data, the properties of algorithms, as well as the interface design. Successful systems require buy-in from the Deaf community, so ensuring that solutions handle application domains appropriately is essential.

Technical limitations impact which domains are appropriate to tackle in the near-term, and inform intermediary goals that which will ultimately inform end-to-end systems. Many of these intermediary goals are worth pursuing in and of themselves, and offer bootstrapping benefts toward longer-term goals. For example, a comprehensive, accurate sign language dictionary that lets users look up individual signs would be an important resource for sign language users and learners alike, and would also inform model design for continuous sign language recognition. In addition, support for everyday use of sign language writing would make text-based resources accessible to sign language users in their language of choice, and would also organically generate an annotated corpus of sign language that could be used to learn language structure. Call 2: Focus on real-world applications.

Sign language processing is appropriate for specifc domains, and the technology has limitations. Datasets, algorithms, interfaces, and overall systems should be built to serve real-world use cases, and account for real-world constraints.

Interface Design

The feld currently lacks fundamental research on how users interact with sign language technology. A number of systems have been developed explicitly serving sign language users (e.g., messaging services [111,121], games [START_REF] Lee | A gesture-based American Sign Language Game for Deaf Children[END_REF][START_REF] Brashear | American sign language recognition in game development for deaf children[END_REF], educational tools [START_REF] Reis | ASL CLeaR: STEM education tools for deaf students[END_REF][START_REF] Alshammari | Building an Interactive E-Learning Tool for Deaf Children: Interaction Design Process Framework[END_REF], webpages [START_REF] Petrie | Augmenting icons for deaf computer users[END_REF], dictionaries [102,[START_REF] Bragg | A User-Powered American Sign Language Dictionary[END_REF], and writing support [START_REF] Savina Bianchini | SWift: a SignWriting improved fast transcriber[END_REF][START_REF] Bragg | Designing an Animated Character System for American Sign Language[END_REF]). However, accompanying user studies typically focus on evaluating a single system, and do not outline principles of interaction that apply across systems. As a result, each team developing a new system must design their interface largely from scratch, uninformed by general design guidelines based on research.

Since many technologies required for end-to-end sign language translation are under development, it may be necessary for researchers to use Wizard-of-Oz style testing procedures (e.g., [START_REF] Dahlbäck | Wizard of Oz studiesâA ˘ Ťwhy and how[END_REF]) to better understand how Deaf users would react to various types of user-interface designs. Recent work has used such approaches. For instance, researchers have used Wizardof-Oz methodologies to study how Deaf users would like to issue commands to personal assistants [START_REF] Rodolitz | Accessibility of voice-activated agents for people who are deaf or hard of hearing[END_REF] or how Deaf users may beneft from a tool that enables ASL dictionary lookup on-demand when reading English text webpages [START_REF] Hariharan | Evaluation of an English Word Look-Up Tool for Web-Browsing with Sign Language Video for Deaf Readers[END_REF].

Returning to the personal assistant application mentioned above, a Wizard-of-Oz methodology could be used to investigate interaction questions, such as how the user might "wake up" the system so it expects a command, and how the system might visually acknowledge a signed command (e.g., by presenting written-language text onscreen) and provide a response to the user (e.g., as written-language text or as sign-language animation). Additionally, such simulations may also be used to determine how good these technologies must be before they are acceptable to users, i.e., what threshold of recognition accuracy is acceptable to users in specifc use cases. Such work can set an agenda for researchers investigating the development of core sign-language recognition or synthesis technologies.

Call 3: Develop user-interface guidelines for sign language systems.

Because sign language processing is still developing, we lack a systematic understanding of how people interact with it. Guidelines and error metrics for effective system design would support the creation of consistently effective interfaces.

Datasets

As highlighted throughout this work, few large-scale, publicly available sign language corpora exist. Moreover, the largest public datasets are orders of magnitude smaller than those of comparable felds like speech recognition. The lack of large-scale public datasets shifts the focus from algorithmic and system development to data curation. Establishing large, appropriate corpora would expedite technical innovation.

In particular, the feld would beneft from a larger body of research involving reproducible tasks. Publicly available data and competitive evaluations are needed to create interest, direct research towards the challenges that matter (tackling depiction, generalizing to unseen signers, real-life data), and increase momentum. Furthermore, having open-source implementations of full pipelines would also foster faster adoption.

There are four main approaches for collecting signing data, each of which has strengths and weaknesses. Developing multiple public data resources that span these four approaches may be necessary in order to balance these tradeoffs.

1. Scraping video sites (e.g., YouTube) has many potential benefts: low cost, rapid collection of many videos, the naturalistic nature of the data, and potential diversity of participants. Its pitfalls include: privacy and consent of people in the videos, variability in signing quality, and lack of accompanying annotations.

2. Crowdsourcing data through existing platforms (e.g., Amazon Mechanical Turk) or customized sites (e.g., [START_REF] Bragg | A User-Powered American Sign Language Dictionary[END_REF]) offers potential cost savings (particularly if participants contribute data for free), and the ability to reach diverse contributors (i.e., by removing geographic constraints). However, crowdsourcing is subject to quality control issues. In paid systems people may rush or "cheat" to earn more money, and in unpaid learning activities, well-intentioned learners may submit low-quality or incorrect data.

3. Bootstrapping, where products are released with limitations and gather data during use, is common to other AI domains (e.g., voice recognition [START_REF] Schalkwyk | Google search by voice: A case study[END_REF]). This approach is cheap, collects highly naturalistic data, and may scale well. However, privacy and informed consent are potential pitfalls, and there is a cold-start problem -can a useful application be created from current datasets to support this bootstrapping process, and can it achieve a critical mass of users?

4. In-lab collection allows for customized, high-end equipment such as high-resolution, high-frame-rate cameras, multiple cameras, depth-cameras, and motion-capture suits. However, this type of controlled collection may result in less naturalistic content, higher costs that limit scalability, and lower participant diversity due to geographic constraints. Models trained on such high-quality data also may not generalize to users with low-quality phone or laptop cameras.

Some metadata impacting data utility can only be gathered at the time of capture. In particular, demographics may be important for understanding biases and generalizability of systems trained on the data [START_REF] Gebru | Datasheets for datasets[END_REF]. Key demographics include signing fuency, language acquisition age, education (level, Deaf vs. mainstream), audiological status, socioeconomic status, gender, race/ethnicity, and geography. Such metadata can also beneft linguistics, Deaf studies, and other disciplines.

Metadata regarding the data collection process itself (i.e., details enabling replication) are also vital to include so that others can add to the dataset. For example, if a dataset is gathered in the U.S., researchers in other countries could replicate the collection method to increase geographic diversity.

Call 4: Create larger, more representative, public video datasets.

Large datasets with diverse signers are essential for training software to perform well for diverse users. Public availability is important for spurring developments, and for ensuring that the Deaf community has equal ownership.

Annotations

A standard annotation system would expedite development of sign language processing. Datasets annotated with the standard system could easily be combined and shared. Software systems built to be compatible with that annotation system would then have much more training data at their disposal. A standard system would also reduce annotation cost and errors. As described earlier, the lack of standardization results in expensive training (and re-training) of annotators, and ambiguous, error-prone annotations.

Designing the annotation system to be appropriate for everyday reading and writing, or developing a separate standard writing system, would provide addition benefts. With such a system, email clients, text editors, and search engines would become newly usable in sign languages without translating into a spoken/written language. As they write, users would also produce a large annotated sign language corpus of naturally generated content, which could be used to better train models. However, establishing a standard writing system requires the Deaf community to reach consensus on how much of the live language may be abstracted away. Any writing system loses some of the live language (i.e., a transcript of a live speech in English loses pitch, speed, intonation, and emotional expression). Sign languages will be no different.

Computer-aided annotation software has been proposed (e.g., [START_REF] Onno | Transcription and Notation Methods[END_REF][START_REF] Dreuw | Towards automatic sign language annotation for the elan tool[END_REF][START_REF] Chételat-Pelé | Sign Language Corpus Annotation: toward a new Methodology[END_REF]), but could provide increased support due to recent advances in deep learning applied to sign language recognition. Current sign language modeling techniques could be used to aid the annotation process in terms of both segmenting and transcribing the input video. Aided annotation should leverage advances in modeling whole signs and also sign subunits [START_REF] Koller | Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data Is Continuous and Weakly Labelled[END_REF][START_REF] Koller | Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos[END_REF]. Annotation support tools could also alleviate problems with annotating depictions, as they could propose annotations conditioned on the translation and hence circumvent the problem of detailing the iconic nature of these concepts.

Call 5: Standardize the annotation system and develop software for annotation support.

Annotations are essential to training recognition systems, providing inputs to NLP and MT software, and generating signing avatars. Standardization would support data sharing, expand software compatibility, and help control quality. Annotation support would help improve accuracy, reliability, and cost.

CONTRIBUTIONS

This paper provides an interdisciplinary perspective on the feld of sign language processing. For computer scientists and technologists, it provides key background on Deaf culture and sign language linguistics that is often lacking, and contextualizes relevant subdomains they may be working within (i.e., HCI, computer vision, computer graphics, MT, NLP). For readers outside of computer science, it provides an overview of how sign language processing works, and helps to explain the challenges that current technologies face.

In synthesizing the state-of-the-art from an interdisciplinary perspective, this paper provides orientation for researchers in any domain, in particular those entering the feld. Unlike disciplinary reviews that focus on relevant work in a particular domain, we relate these domains to one another, and show how sign language processing is dependent on all of them.

In summarizing the current challenges, this work highlights opportunities for interdisciplinary collaboration. Many of the problems facing the feld cross disciplines. In particular, questions of how to create datasets, algorithms, user interfaces, and a standard annotation system that meet technical requirements, refect linguistics of the language, and are accepted by the Deaf community are large, open problems that will require strong, interdisciplinary teams.

Finally, in articulating a call to action, this work helps researchers prioritize efforts to focus on the most pressing and important problems. Lack of data (in particular large, annotated, representative, public datasets) is arguably the biggest obstacle currently facing the feld. This problem is confounded by the relatively small pool of potential contributors, recording requirements, and lack of standardized annotations. Because data collection is diffcult and costly, companies and research groups are also incentivised to keep data proprietary. Without suffcient data, system performance will be limited and unlikely to meet the Deaf community's standards.

Our workshop methodology used to provide this interdisciplinary perspective on sign language processing can be used as a model for other similarly siloed felds. While the general structure of the workshop is directly duplicable, some work would need to be done to tailor it to other felds (e.g., identifying relevant domains and domain experts).

CONCLUSION

In this paper, we provide an interdisciplinary overview of sign language recognition, generation, and translation. Past work on sign language processing has largely been conducted by experts in different domains separately, limiting real-world utility. In this work, we assess the feld from an interdisciplinary perspective, tackling three questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the feld? and 3) What are the calls to action for people working in the feld?

To address these questions, we ran an interdisciplinary workshop with 39 domain experts with diverse backgrounds. This paper presents the interdisciplinary workshop's fndings, providing key background for computer scientists on Deaf culture and sign language linguistics that is often overlooked, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community. In doing so, this paper serves to orient readers both within and outside of computer science to the feld, highlights opportunities for interdisciplinary collaborations, and helps the research community prioritize which problems to tackle next (data, data, data!).

Acronyms AI Artifcial Intelligence. ASL American Sign Language.

CNN convolutional neural network.

DHH Deaf and hard of hearing.

HCI Human-Computer Interaction.

MT machine translation.

NLP Natural Language Processing.

UI/UX User Interface/User Experience.

WER Word Error Rate.

Glossary

annotation Descriptions of sign language content, typically in a writing system, note-taking system, or other discrete system that a computer can read. ASL American Sign Language, the primary language of the Deaf community in the United States (and several other parts of the world). audism Oppression or discrimination of the Deaf minority by the hearing majority. This marginalization has a long history, and may be overt covert.

classifer A handshape used to represent a particular class of objects, actions, or ideas. CNN A Convolutional Neural Network is a specifc type of neural network that makes use of effcient parameter sharing and has revolutionized computer vision. co-articulation The infuence of surrounding signs/context on the production of the current sign, for example the ending of one sign affecting the beginning of the next. continuous Refers to long phrases or full sentences as opposed to single, isolated signs (or words).

Deaf culture The culture of Deaf communities, of which sign languages are a central component. Capitalized "Deaf" refers to these cultures, while lowercase "deaf" refers to audiological status.

depiction Depiction refers to the ability to represent semantic information visually. Enacting an event or using classifers are examples of depiction. Depiction is structured and highly fexible and many aspects of depiction have no spoken language corollary.

epenthesis Insertion of phonological features into or between signs, often in the form of movement for sign languages.

fngerspelling The use of a series of manual letters (handshapes representing letters), to spell out a written/spoken word in a sign language.

gloss Transliteration of one language (e.g., ASL) with words of another language (e.g., English). This retains the grammatical structure of the target language (e.g., ASL).

HamNoSys The Hamburg Notation System [START_REF] Prillwitz | HamNoSys. Version 2.0[END_REF] is a phonetic transcription scheme that allows explicit annotation of individual subcomponents of sign language such as handshapes, movements, etc.

phonological Having to do with phonology, the smallest units of a language, and how they are put together. For spoken languages the components are sounds, and for sign languages they are handshapes, locations, and movements etc.

real-life Describes sign language that has not been scripted or tailored to suit machine learning algorithms, but is rather produced naturally.

signer-independent Characterizes a realistic sign language recognition experimental setup where the signers that are tested on are explicitly excluded from the training set.

User Interface/User Experience UI refers to user interface, while UX refers to user experience. Technology designers (and HCI researchers) are often trained to focus on these crucial aspects of technology.

vocabulary The set of unique signs (or words) that occur in a language or dataset.

Word Error Rate A standard evaluation criterion in speech recognition and sign language recognition, and is based on the edit distance between reference and hypothesis.

Table 1 .

 1 Popular public corpora of sign language video. These datasets are commonly used for sign language recognition.

	Dataset	Vocabulary	Signers	Signer-independent	Videos Continuous	Real-life
	Purdue RVL-SLLL ASL [65]	104	14	no	2,576	yes	no
	RWTH Boston 104 [124]	104	3	no	201	yes	no
	Video-Based CSL [54]	178	50	no	25,000	yes	no
	Signum [118]	465	(24 train, 1 test) -25	yes	15,075	yes	no
	MS-ASL [62]	1,000 (165 train, 37 dev, 20 test) -222	yes	25,513	no	yes
	RWTH Phoenix [43]	1,081	9	no	6,841	yes	yes
	RWTH Phoenix SI5 [74]	1,081	(8 train, 1 test) -9	yes	4,667	yes	yes
	Devisign [22]	2,000	8	no	24,000	no	no
	alistic scenarios are still very challenging for state-of-the-art
	algorithms. As such, recognition systems achieve only up to
	42.8% letter accuracy [101] on a recently released real-life fn-
	gerspelling dataset.

, though lack of suffcient training data currently limits the use of modern Artifcial Intelligence (AI) techniques in this problem space.

Automatic recognition systems are transitioning from small, artifcial vocabularies and tasks to larger real-world ones. Re-

Table

		Sign Language	Speech
	Modality	visual-gestural	aural-oral
	Articulators	manual, non-manual	vocal tract
	Seriality	low	high
	Simultaneity	high	low
	Iconicity	high	low
	Task	recognition, generation, translation	recognition, generation, translation
	Typical articulated corpus size	<100,000 signs	5 million words
	Typical annotated corpus size	<100,000 signs	1 billion words
	Typical corpus vocabulary size	1,500 signs	300,000 words
	What is being modelled	1,500 whole signs	1,500 tri-phonemes
	Typical corpus number of speakers	10	1,000

ACKNOWLEDGMENTS

The authors would like to thank all workshop participants. We also thank Bill Thies for helpful discussions and prudent advice. This material is based on work supported by Microsoft, NIH R21-DC016104, and NSF awards 1462280, 1746056, 1749376, 1763569, 1822747, 1625793, and 1749384.