Martina Fischer

Paul Rikeit

Petra Knaus

Catherine Coirault
email: catherinecoiraultc.coirault@institut-myologie.org

Martina Krüger

John Joseph Mccarthy

Roberta Sartori

YAP-Mediated Mechanotransduction in Skeletal Muscle

Keywords: YAP, mechanotransduction, skeletal muscle, hippo pathway, muscle homeostasis

INTRODUCTION

Mechanotransduction refers to the conversion of mechanical inputs to intracellular biochemical and biophysical signals [START_REF] Wang | Mechanotransduction at a distance: mechanically coupling the extracellular matrix with the nucleus[END_REF]. Based on the observation that muscle grows in response to exercise and degrades when underutilized, studies on mechanotransduction have been performed already decades ago in skeletal muscle [START_REF] Goldberg | Protein synthiesis during work-induced growth of skeletal muscle[END_REF][START_REF] Vandenburgh | In vitro model for stretch-induced hypertrophy of skeletal muscle[END_REF].

Current studies on mechanotransduction consider different models. The first relies on the initiation of mechanosignaling via stimulation of "mechanosensors." These are thought to be adhesive, structural or transmembrane proteins, which could react with conformational changes to applied forces, transmitted by the extracellular matrix (ECM) or neighboring cells. These mechanical stimulations are then integrated into signaling pathways induced by soluble factors and consequently regulate transcriptional changes. In an alternative model, the cell itself is considered a compartmentalized mechanical body with given physical properties such as its viscosity, elasticity or stiffness. Here, the cellular mechanics are mainly defined through the actin, tubulin or septin cytoskeleton, intermediate filaments and the nuclear envelope and nuclear skeleton. These intracellular networks are connected to the ECM through adhesion complexes so that the cellular mechanics are in a permanent coordination with the extracellular constraints. According to this model, mechanical changes are not translated into one specialized mechanosensing pathway but into a simultaneous change in various cell processes, which are regulated by cytoskeletal dynamics, including the activation of signaling pathways.

The IGF-1-Akt-mTOR (Insulin-like growth factor I -protein kinase B/Akt -mammalian target of Rapamycin) pathway has emerged to be the main positive regulator of muscle mass [START_REF] Sandri | Signaling in muscle atrophy and hypertrophy[END_REF][START_REF] Miyazaki | Early activation of mTORC1 signalling in response to mechanical overload is independent of phosphoinositide 3-kinase/Akt signalling[END_REF][START_REF] Schiaffino | Mechanisms regulating skeletal muscle growth and atrophy[END_REF] and Myostatin-Smad3 has been identified as the main negative regulator of muscle mass [START_REF] Wackerhage | Signal transduction pathways that regulate muscle growth[END_REF][START_REF] Rodriguez | Myostatin and the skeletal muscle atrophy and hypertrophy signaling pathways[END_REF]. Furthermore, Yes-associated Protein (YAP), a transcriptional coactivator downstream of the Hippo pathway, has been shown to be involved in myogenesis, muscle homeostasis and muscle disorders. In parallel, YAP emerged as a key player in mechanotransduction [START_REF] Dupont | Role of YAP/TAZ in mechanotransduction[END_REF][START_REF] Wackerhage | The Hippo signal transduction network in skeletal and cardiac muscle[END_REF] and several crosstalks between Akt/mTOR or TGFß/SMAD and the Hippo/Yap pathways have been identified which point to a role of YAP in regulating muscle mass through mechanical cues [START_REF] Jang | Akt phosphorylates MstI and prevents its proteolytic activation, blocking FOXO3 phosphorylation and nuclear translocation[END_REF][START_REF] Alarcón | Nuclear CDKs Drive Smad Transcriptional Activation and Turnover in BMP and TGF-β Pathways[END_REF][START_REF] Tumaneng | YAP mediates crosstalk between the Hippo and PI(3)K-TOR pathways by suppressing PTEN via miR-29[END_REF][START_REF] Grannas | Crosstalk between Hippo and TGFβ: subcellular localization of YAP/TAZ/Smad complexes[END_REF].

YAP

The transcriptional co-activator YAP was first described in 1994 as a 65 kDa binding partner of the Yes protein-tyrosine kinase [START_REF] Sudol | Characterization of the Mammalian YAP (Yes-associated Protein) gene and its role in defining a novel protein module, the WW Domain[END_REF]. YAP contains a transcription activation domain (TAD) located at the carboxy-terminal half [START_REF] Yagi | A WW domain-containing yes-associated protein (YAP) is a novel transcriptional co-activator[END_REF], while the amino-terminal half contains one or two WW domains [START_REF] Sudol | Structure and function of the WW domain[END_REF]. These WW domains mediate interactions with proteins containing PPxY motifs. A plethora of different proteins bind to the WW domains of YAP including LATS1/2 [START_REF] Oka | Mst2 and Lats kinases regulate apoptotic function of Yes kinase-associated protein (YAP)[END_REF], angiomotin (AMOT) [START_REF] Zhao | Angiomotin is a novel Hippo pathway component that inhibits YAP oncoprotein[END_REF] and Smad7 [START_REF] Ferrigno | Yes-associated protein (YAP65) interacts with Smad7 and potentiates its inhibitory activity against TGF-β/Smad signaling[END_REF]. In its active state, YAP localizes to the nucleus and regulates the activity of several transcription factors including RUNX, SMAD, p73 and ErbB4 and most importantly TEAD family transcription factors, since YAP and TEAD occupy about 80% of the same genomic loci [START_REF] Zhao | TEAD mediates YAPdependent gene induction and growth control[END_REF]. Prominent target genes of YAP include CTGF, Cyclin D1, AREG, Birc5, and myogenic transcription factor Myf5 [START_REF] Dong | Elucidation of a universal size-control mechanism in Drosophila and mammals[END_REF][START_REF] Zhao | TEAD mediates YAPdependent gene induction and growth control[END_REF][START_REF] Zhang | YAP-dependent induction of amphiregulin identifies a non-cellautonomous component of the Hippo pathway[END_REF][START_REF] Watt | Yap is a novel regulator of C2C12 myogenesis[END_REF].

Together with its paralog TAZ (Transcriptional co-activator with PDZ-binding motif), YAP controls a wide range of cellular functions. During embryogenesis in mice YAP is expressed at all stages from blastocyst to perinatal stage. Homozygous disruption of the YAP allele in mice results in embryonic lethality and causes developmental arrest at E8.5. In contrast, TAZ shows a later onset and is not yet expressed at blastocyst stage [START_REF] Morin-Kensicki | Defects in yolk sac vasculogenesis, chorioallantoic fusion, and embryonic axis elongation in mice with targeted disruption of Yap65[END_REF]. This indicates a unique role of YAP in embryonic development, for which TAZ does not compensate.

Nuclear YAP activity controls the transcription of genes involved in cell cycle control, typically driving proliferation and survival and inhibiting apoptosis [START_REF] Dong | Elucidation of a universal size-control mechanism in Drosophila and mammals[END_REF]. Thus, by balancing cell proliferation and death, YAP mediates cell contact inhibition in vitro [START_REF] Zhao | Inactivation of YAP oncoprotein by the Hippo pathway is involved in cell contact inhibition and tissue growth control[END_REF] and regulates organ size in vivo [START_REF] Camargo | YAP1 increases organ size and expands undifferentiated progenitor cells[END_REF][START_REF] Dong | Elucidation of a universal size-control mechanism in Drosophila and mammals[END_REF][START_REF] Xin | Regulation of insulin-like growth factor signaling by Yap governs cardiomyocyte proliferation and embryonic heart size[END_REF]. As a regulator of cell-cycle control, YAP misregulation can also lead to tumorigenesis. In several tissues YAP is known to function as an oncogene, while its upstream negative regulators and their adaptors have tumor suppressor function. Furthermore, YAP expression in MCF10A cells induces epithelial to mesenchymal transition (EMT) [START_REF] Overholtzer | Transforming properties of YAP, a candidate oncogene on the chromosome 11q22 amplicon[END_REF].

In summary, dysregulated YAP activity has been implicated in a wide range of tumor types including intestinal stem cells, hepatocellular, pancreatic, renal, colorectal, breast, and skeletal muscle cancer [START_REF] Dong | Elucidation of a universal size-control mechanism in Drosophila and mammals[END_REF][START_REF] Tremblay | The Hippo transducer YAP1 transforms activated satellite cells and is a potent effector of embryonal rhabdomyosarcoma formation[END_REF][START_REF] Patel | Niche appropriation by Drosophila intestinal stem cell tumours[END_REF] which has been further reviewed by S. Plouffe [START_REF] Plouffe | Disease implications of the Hippo/YAP pathway[END_REF].

Furthermore, YAP is involved in cell fate decisions and serves as a "stemness" factor in different progenitor cell pools of the body. Examples include progenitors in the intestinal crypt, neural progenitor cells in the neural tube, epidermal stem cells and satellite cells of skeletal muscle, where YAP activity promotes proliferation and blocks differentiation [START_REF] Camargo | YAP1 increases organ size and expands undifferentiated progenitor cells[END_REF][START_REF] Cao | YAP regulates neural progenitor cell number via the TEA domain transcription factor[END_REF][START_REF] Watt | Yap is a novel regulator of C2C12 myogenesis[END_REF][START_REF] Schlegelmilch | Yap1 acts downstream of α-catenin to control epidermal proliferation[END_REF].

YAP also influences cell migration, which is also critical for muscle development and regeneration as activated satellite cells need to migrate out of their niche and along the basal lamina of the myofiber. It was shown, that YAP overexpression in MCF10A or HEK293 cells leads to increased migration and YAP knockdown abolishes migration in T47D cells and renal carcinoma cell lines [START_REF] Haskins | Neuregulin 1-activated ERBB4 interacts with YAP to induce Hippo pathway target genes and promote cell migration[END_REF][START_REF] Schütte | Hippo signaling mediates proliferation, invasiveness, and metastatic potential of clear cell renal cell carcinoma[END_REF][START_REF] Sorrentino | Metabolic control of YAP and TAZ by the mevalonate pathway[END_REF][START_REF] Moroishi | A YAP/TAZ-induced feedback mechanism regulates Hippo pathway homeostasis[END_REF].

YAP capacity to balance proliferation, apoptosis and migration also makes it a regulator of regenerative processes in different tissues including intestine and heart muscle tissue as YAP knockdown severely impairs their regenerative capacity [START_REF] Cai | The Hippo signaling pathway restricts the oncogenic potential of an intestinal regeneration program[END_REF][START_REF] Xin | Hippo pathway effector Yap promotes cardiac regeneration[END_REF]. In zebrafish, YAP activity in fin regeneration is based on cell density differences along the regenerating tissue, which leads to a graded control of tissue growth [START_REF] Mateus | Control of tissue growth by Yap relies on cell density and F-actin in zebrafish fin regeneration[END_REF].

In brief, YAP is a regulator of the cell cycle and cell fate decisions and consequently of development, organ size and tumorigenesis.

The Hippo Pathway and YAP Regulation

YAP activity is regulated very tightly by the Hippo pathway and a great number of crosstalks, whose interplays are not completely uncovered today (Figure 1). Originally identified by genetic studies in Drosophila, the Hippo signaling cascade functions as a highly conserved canonical upstream regulator of YAP activity [START_REF] Harvey | The drosophila mst ortholog, hippo, restricts growth and cell proliferation and promotes apoptosis[END_REF][START_REF] Wu | hippo Encodes a Ste-20 family protein kinase that restricts cell proliferation and promotes apoptosis in conjunction with salvador and warts[END_REF]. At the core of the mammalian pathway is a kinase cassette containing Mammalian Ste20-like 1/2 kinase (MST1/2) and large tumor suppressor 1/2 kinase (LATS1/2) with their adaptors Salvador and MOBKL1A/1B (Mps 1 binder kinase activator-like 1A and 1B) (Figure 1). YAP activity is regulated by phosphorylation, predominantly through five different phosphorylation sites, which are located in HXRXXS consensus motifs for LATS1/2 kinases. The most intensely studied LATS mediated phosphorylation is at serine 127, which leads to binding of 14-3-3 proteins, sequestration of YAP in the cytoplasm and consequently termination of its nuclear activity [START_REF] Zhao | Inactivation of YAP oncoprotein by the Hippo pathway is involved in cell contact inhibition and tissue growth control[END_REF]. Phosphorylation at Serine 381 by LATS1/2 however, primes YAP for phosphorylation by casein kinases CK1δ or CK1ε and subsequent ubiquitination via SCF βTRCP E3 ubiquitin ligase and proteasomal degradation [START_REF] Zhao | A coordinated phosphorylation by Lats and CK1 regulates YAP stability through SCF(β-TRCP)[END_REF]. LATS1/2 kinases are activated by phosphorylation of activated MST1/2 kinases. The activity of MST1/2 and LATS1/2 kinases are further FIGURE 1 | Actin associated proteins regulate YAP activity. The transcriptional coactivator YAP shuttles into the nucleus, where it activates TEAD mediated gene expression. After phosphorylation by LATS1/2 kinase, YAP binds to 14-3-3 proteins, leading to its cytoplasmic retention and degradation. YAP activity is regulated by the actin cytoskeleton. Actin stress fibers connect to the lamin meshwork in the nucleus via the LINC-complex. Rho GPTases are regulated by GPCR signaling which in turn regulates actin dynamics and YAP activity (dashed lines). Actin-binding proteins, like angiomotin (AMOT) or neurofibromin 2 (NF2/Merlin) are also known to regulate YAP activity, either through LATS or by direct interaction with YAP. Akt, a key regulator of the IGF-1-mTor pathway also binds to actin stress fibers, crosstalks to the Hippo pathway by interacting with MST1/2 and by YAP induced expression of a microRNA (miR-29) which inhibits the inhibition of Akt by targeting PTEN. regulated via different trans-and autophophorylation sites. For further reading see the reviews of [START_REF] Visser | LATS tumor suppressor: a new governor of cellular homeostasis[END_REF] and [START_REF] Rawat | Regulation of mammalian Ste20 (Mst) kinases[END_REF].

Moreover, YAP activity is balanced through a negative feedback loop. YAP-TEAD activity induces LATS2 kinase expression and activation of LATS1/2 kinases through Merlin/Neurofibromin 2 (NF2), leading to phosphorylation and inactivation of YAP [START_REF] Moroishi | A YAP/TAZ-induced feedback mechanism regulates Hippo pathway homeostasis[END_REF]. Thus, overshooting YAP activity including its tumorigenic potential can be counteracted by this intrinsic regulatory mechanism.

The regulation of YAP by canonical Hippo signaling in mammals was revealed in the context of contact inhibition of proliferation (CIP). Cells grown at low density show nuclear YAP and an inactive Hippo cascade, while at high cell density, the Hippo pathway is switched on and YAP is inactivated via LATS1/2 mediated phosphorylation [START_REF] Zhao | Inactivation of YAP oncoprotein by the Hippo pathway is involved in cell contact inhibition and tissue growth control[END_REF]. A specific Hippo receptor as the primary trigger of the Hippo signaling cascade has not been identified yet and the dependence of YAP regulation on Hippo signaling in other contexts has been questioned [START_REF] Aragona | A mechanical checkpoint controls multicellular growth through YAP/TAZ regulation by actin-processing factors[END_REF]. The main upstream elements regulating YAP activity are discussed in the following paragraphs and are summarized in Figure 1. For further informations see the following publications [START_REF] Schroeder | Regulation of the Hippo pathway by cell architecture and mechanical signals[END_REF][START_REF] Johnson | The two faces of Hippo: targeting the Hippo pathway for regenerative medicine and cancer treatment[END_REF][START_REF] Hansen | YAP and TAZ: a nexus for Hippo signaling and beyond[END_REF].

Apicobasal Cell Polarity, Tight Junctions, and Adherens Junctions

YAP/TAZ proteins interact with several components of the Crumbs polarity complex and disrupting the Crumbs complex increases nuclear YAP [START_REF] Varelas | The Crumbs complex couples cell density sensing to Hippo-dependent control of the TGF-β-SMAD pathway[END_REF]. Furthermore, AMOT proteins, which also localize to the Crumbs complex regulate YAP activity [START_REF] Paramasivam | Angiomotin family proteins are novel activators of the LATS2 kinase tumor suppressor[END_REF][START_REF] Zhao | Angiomotin is a novel Hippo pathway component that inhibits YAP oncoprotein[END_REF]. Also, cadherin-catenin complexes were shown to regulate YAP localization and activity [START_REF] Schlegelmilch | Yap1 acts downstream of α-catenin to control epidermal proliferation[END_REF][START_REF] Silvis | α-catenin is a tumor suppressor that controls cell accumulation by regulating the localization and activity of the transcriptional coactivator Yap1[END_REF]. Expression of E-cadherins as well as their association with α-and β-catenin are required for density dependent nuclear exclusion of YAP [START_REF] Kim | E-cadherin mediates contact inhibition of proliferation through Hippo signalingpathway components[END_REF]. Non-receptor tyrosine phosphatase PTPN14, which plays a role in regulating phosphorylation of β-catenin in adherens junctions [START_REF] Wadham | The protein tyrosine phosphatase Pez is a major phosphatase of adherens junctions and dephosphorylates β-catenin[END_REF] was also shown to inhibit YAP activity by promoting its cytoplasmic localization, independently of its phosphatase activity [START_REF] Michaloglou | The tyrosine phosphatase PTPN14 is a negative regulator of YAP activity[END_REF].

Also Merlin/NF2, another membrane-associated protein, which links cytoskeletal components with proteins in the cell membrane, regulates YAP. The tumor suppressor function of Merlin/NF2, inactivated in Neurofibromatosis type II, acts through the activation of the Hippo cascade most likely by binding and recruiting LATS to the plasma membrane, which in turn promotes LATS phosphorylation by MST [START_REF] Yin | Spatial organization of Hippo signaling at the plasma membrane mediated by the tumor suppressor Merlin/NF2[END_REF].

Soluble Cues and Receptor Signaling

YAP can also be regulated through G-protein coupled receptors (GPCRs) and serum starvation inhibits YAP activity via reduced GPCR signaling. G 12/13 -, G q/11 -, and G i/o -coupled receptor agonists (e.g., Lysophosphatidic acid (LPA), sphingosine-1phosphate (S1P)) activate YAP/TAZ while Epinephrine and Glucagon inhibit YAP/TAZ via G s -coupled GPCR signaling [START_REF] Yu | Regulation of the Hippo-YAP pathway by G-protein-coupled receptor signaling[END_REF]. Recently, the G 12/13 subunit was also identified to be activated by the Wnt-FZD/ROR receptor, which would represent an alternative WNT signaling pathway acting through YAP [START_REF] Park | Alternative Wnt signaling activates YAP/TAZ[END_REF]. The regulation of YAP by G-proteins has been shown to be either mediated by the Rho family of GTPases, actin dynamics and LATS [START_REF] Yu | Regulation of the Hippo-YAP pathway by G-protein-coupled receptor signaling[END_REF] or PI3-kinase (PI3K) and phosphoinositide-dependent kinase (PDK1) [START_REF] Gumbiner | The Hippo-YAP signaling pathway and contact inhibition of growth[END_REF]. In addition, epidermal growth factor receptor (EGFR) signaling activates YAP through activation of PI3-kinase (PI3K) and phosphoinositide-dependent kinase (PDK1), which binds to the Hippo core kinases complex [START_REF] Fan | Regulation of Hippo pathway by mitogenic growth factors via phosphoinositide 3-kinase and phosphoinositidedependent kinase-1[END_REF]. The neuregulin 1 (NRG1) activated receptor ERBB4, another member of the EGFR family interacts with YAP through its PPxY domain and activates YAP-mediated transcription [START_REF] Haskins | Neuregulin 1-activated ERBB4 interacts with YAP to induce Hippo pathway target genes and promote cell migration[END_REF].

YAP: A KEY REGULATOR IN MECHANOTRANSDUCTION

There is increasing evidence that YAP is a key regulator of mechanotransduction. Pioneering work of Piccolo and coworkers showed that mechanical forces can serve as inputs for the regulation of YAP. By analyzing YAP localization and transcriptional response, they showed YAP activity to be regulated by ECM stiffness, cell-spreading or substrate rigidity [START_REF] Dupont | Role of YAP/TAZ in mechanotransduction[END_REF]; Figures 2A-C). Along with these findings, the group of Sasaki proposed a model where cell morphology alone modulates YAP activity. By the use of a microdomain culture system the cell area of a single cell was defined preventing cell-cell contact [START_REF] Wada | Hippo pathway regulation by cell morphology and stress fibers[END_REF]. Guan and co-workers further excluded the requirement of focal adhesion sites for the regulation of YAP through cell morphology, by seeding epithelial cells on poly-lysine and attaching them via electrostatic forces [START_REF] Zhao | Cell detachment activates the Hippo pathway via cytoskeleton reorganization to induce anoikis[END_REF].

In addition, Piccolo and colleagues showed that YAP can be reactivated in postconfluent culture conditions by stretching the cells, while preventing that these cells lose cell-cell contact [START_REF] Aragona | A mechanical checkpoint controls multicellular growth through YAP/TAZ regulation by actin-processing factors[END_REF]. The reactivation of YAP by cyclic stretching has also been confirmed on soft surfaces together with an increase in cell spreading, stress fiber formation and proliferation [START_REF] Cui | Cyclic stretching of soft substrates induces spreading and growth[END_REF]; Figure 2D). The regulation of YAP by shear stress has only been rarely characterized so far and more research is needed to prove a shear stressdependent YAP regulation. However, YAP was shown to be activated by fluid shear stress in osteoblasts [START_REF] Kaneko | Integrin αv in the mechanical response of osteoblast lineage cells[END_REF]. Furthermore, increased YAP expression, triggered by fluid shear stress, increased osteogenesis and decreased adipogenesis of hMSCs and initiated dedifferentiation of chondrocytes [START_REF] Zhong | Mesenchymal stem cell and chondrocyte fates in a multishear microdevice are regulated by Yes-associated protein[END_REF]; Figure 2E).

In addition, a mechanical memory was claimed. Long term culture of hMSCs on supraphysiologically stiff substrates can persistently activate YAP and transfer to soft hydrogels cannot inactivate YAP anymore [START_REF] Yang | Mechanical memory and dosing influence stem cell fate[END_REF].

YAP Regulation by Intracellular Stress

Cell shape and size, ECM stiffness and forces like traction or shear stress all reflect on the cytoskeleton. Published work provides compelling evidence for a critical role of actin dynamics in the regulation of YAP through mechanical cues. Particularly by correlating the activity of YAP with actin stress fiber formation and showing YAP inactivation by the use of F-actin or Rho inhibitors, but not by inhibiting microtubules or Rac1-GEFs [START_REF] Dupont | Role of YAP/TAZ in mechanotransduction[END_REF][START_REF] Halder | Transduction of mechanical and cytoskeletal cues by YAP and TAZ[END_REF][START_REF] Zhao | Cell detachment activates the Hippo pathway via cytoskeleton reorganization to induce anoikis[END_REF]. Also in vivo experiments on Drosophila reveal increased actin stress fiber assembly to correlate with YAP nuclear localization and overgrowth of the wing disc [START_REF] Fernández | Actin-capping protein and the hippo pathway regulate Factin and tissue growth in Drosophila[END_REF][START_REF] Sansores-Garcia | Modulating F-actin organization induces organ growth by affecting the Hippo pathway[END_REF]. However, the specificity of this effect and the mechanism linking stress fiber formation to YAP activity is controversial and the focus of ongoing research.

Data providing deeper insight into how YAP is regulated by cell morphology and in coordination with cell-contact inhibition in epithelial cells was further published by the Piccolo lab. They found mechanical forces to be the overarching regulators of YAP in a multicellular context. The actin-capping andsevering proteins Cofilin, GapZ, and Gelsolin were identified as gatekeepers, limiting YAP activity in cells which experience low mechanical stress. By depleting actin-capping/severing proteins they showed that increased actin stress fiber formation can restore YAP activity in dense monolayers [START_REF] Aragona | A mechanical checkpoint controls multicellular growth through YAP/TAZ regulation by actin-processing factors[END_REF]. Assuming the cytoskeleton as the key transducer of mechanical cues into YAP signaling, the role of adaptor proteins like AMOT or other cytoskeletal structures like the tubulin or septin network remain to be further investigated. In regard to skeletal muscle, the role of the structural proteins of the sarcomere needs to be considered as well and their effects on YAP regulation remain to be examined. In the same way, the impact of the polynucleated organization of myofibers in mechanotransduction remain to be investigated.

In addition to changes in cytoskeletal structures, extracellular forces are also transmitted to the nucleus as the cytoskeleton is coupled to the nuclear envelope. The LINC-complex (Linker of Nucleoskeleton and Cytoskeleton), consisting of Nesprin and SUN proteins, thereby connects actin stress fibers to the nucleoskeleton (for further reading, see [START_REF] Lombardi | Keeping the LINC: the importance of nucleocytoskeletal coupling in intracellular force transmission and cellular function[END_REF]. Recently, also YAP nuclear translocation was found to be dependent on the force transition to the nucleus through the LINC-complex. By the use of traction force maps, the transfer of the strain to the nucleus was considered essential for YAP localization and activity. Moreover, YAP nuclear relocalization after strain can be prevented by knocking down Nesprin, a protein of the LINC-complex [START_REF] Driscoll | Cytoskeletal to nuclear strain transfer regulates YAP signaling in mesenchymal stem cells[END_REF]. A-type lamin, an intermediate filament located at the inner nuclear membrane, binds to SUN proteins [START_REF] Haque | SUN1 interacts with nuclear lamin A and cytoplasmic nesprins to provide a physical connection between the nuclear lamina and the cytoskeleton[END_REF] and accumulates at the LINC-complex after applied tension [START_REF] Guilluy | Isolated nuclei adapt to force and reveal a mechanotransduction pathway in the nucleus[END_REF]. Consistently, satellite cell-derived myoblasts carrying a mutation in A-type lamin are unable to reactivate YAP after cyclic stretch [START_REF] Bertrand | Cellular microenvironments reveal defective mechanosensing responses and elevated YAP signaling in LMNA-mutated muscle precursors[END_REF]. Mutations in nucleoskeletal proteins like A-type lamin or Emerin can cause several forms of muscular disorders whose pathophysiology is still not understood.

Transcription factors whose activity is regulated by actin dynamics are already known. For example, megakaryoblastic leukemia 1 (MKL1) binds to G-actin and is released when G-actin polymerizes to form F-actin [START_REF] Miralles | Actin dynamics control SRF activity by regulation of its coactivator MAL[END_REF]. Moreover MKL-1, together with YAP, has already been implicated in mechanosensing defects of LMNA mutant myoblasts [START_REF] Bertrand | Cellular microenvironments reveal defective mechanosensing responses and elevated YAP signaling in LMNA-mutated muscle precursors[END_REF], presumably through modulation of actin dynamics [START_REF] Ho | Lamin A/C and emerin regulate MKL1-SRF activity by modulating actin dynamics[END_REF].

Role of LATS in Mechanotransduction

The dependence of mechanically induced YAP activation on the Hippo core kinases has been challenged. The Guan and Sasaki lab claimed YAP to be regulated by LATS, and LATS to be regulated by stress fibers [START_REF] Wada | Hippo pathway regulation by cell morphology and stress fibers[END_REF][START_REF] Zhao | Cell detachment activates the Hippo pathway via cytoskeleton reorganization to induce anoikis[END_REF]. By contrast, the Piccolo group found LATS phosphorylation not to be the primary mediator of YAP activity through mechanical cues. YAP and TAZ activity could not be rescued by knockdown of LATS1 and LATS2 after inhibition of actin polymerization [START_REF] Aragona | A mechanical checkpoint controls multicellular growth through YAP/TAZ regulation by actin-processing factors[END_REF]. Also for the highly discussed adaptor protein AMOT it is not sufficiently clarified if AMOT regulates YAP by either direct binding or via interaction with LATS2 protein [START_REF] Paramasivam | Angiomotin family proteins are novel activators of the LATS2 kinase tumor suppressor[END_REF][START_REF] Zhao | Angiomotin is a novel Hippo pathway component that inhibits YAP oncoprotein[END_REF]. Nevertheless, if actin stress fibers inactivate LATS or sequester a LATS-independent inhibitor of YAP, or both, remains to be clarified.

YAP Crosstalks with Other Mechanosensitive Pathways

YAP interacts with components of other signaling pathways, which play a role in mechanotransduction as well. Canonical TGFβ/BMP signaling acts through SMADs and has been shown to be sensitive to mechanical inputs into the cell [START_REF] Maeda | Conversion of mechanical force into TGF-β-mediated biochemical signals[END_REF][START_REF] Kopf | BMP growth factor signaling in a biomechanical context[END_REF]. YAP was shown to bind to activated SMAD1 proteins and to enhance their BMP induced transcriptional activity [START_REF] Alarcón | Nuclear CDKs Drive Smad Transcriptional Activation and Turnover in BMP and TGF-β Pathways[END_REF]. YAP also interacts with SMAD2/3 in a TGFβ and cell density dependent manner [START_REF] Grannas | Crosstalk between Hippo and TGFβ: subcellular localization of YAP/TAZ/Smad complexes[END_REF]. In addition, YAP nuclear exclusion sequesters SMAD2/3 proteins to the cytoplasm and therefore suppresses TGFβ signaling [START_REF] Varelas | The Crumbs complex couples cell density sensing to Hippo-dependent control of the TGF-β-SMAD pathway[END_REF][START_REF] Narimatsu | Distinct polarity cues direct Taz/Yap and TGFβ receptor localization to differentially control TGFβ-induced Smad signaling[END_REF]. Furthermore, YAP interacts with the TGFβ signaling inhibitor SMAD7 [START_REF] Ferrigno | Yes-associated protein (YAP65) interacts with Smad7 and potentiates its inhibitory activity against TGF-β/Smad signaling[END_REF].

Wnt/β-catenin has been implicated in mechanotransduction as well [START_REF] Huang | Mechanotransduction in bone repair and regeneration[END_REF][START_REF] Kang | New insights into Wnt-Lrp5/6-β-catenin signaling in mechanotransduction[END_REF]. Cytoplasmic YAP inhibits Wnt/β-catenin activity by sequestering β-catenin in the cytoplasm [START_REF] Imajo | A molecular mechanism that links Hippo signalling to the inhibition of Wnt/β-catenin signalling[END_REF] and by interacting with its regulators disheveled and SHP2 [START_REF] Barry | Restriction of intestinal stem cell expansion and the regenerative response by YAP[END_REF][START_REF] Tsutsumi | YAP and TAZ, Hippo signaling targets, act as a rheostat for nuclear SHP2 function[END_REF]. Overexpression of active YAP in mouse cardiomyocytes leads to increased β-catenin activity via the IGF-PI3K-AkT-GSK3β axis [START_REF] Xin | Regulation of insulin-like growth factor signaling by Yap governs cardiomyocyte proliferation and embryonic heart size[END_REF]. Furthermore, Protein Kinase C zeta can phosphorylate both YAP and β-catenin to inhibit their nuclear activity [START_REF] Llado | Repression of intestinal stem cell function and tumorigenesis through direct phosphorylation of β-Catenin and Yap by PKCζ[END_REF]. However, the impact of the crosstalk between YAP and TGFβ/BMP or Wnt/βcatenin signaling on mechanotransduction and its relevance in skeletal muscle homeostasis remain to be elucidated.

MECHANOSENSING AND -SIGNALING IN SKELETAL MUSCLE

Muscle activity is known to be a major regulator of skeletal muscle mass, with an increase in mechanical loading resulting in muscle hypertrophy, and a decrease in mechanical loading resulting in muscle atrophy [START_REF] Goldberg | Mechanism of work-induced hypertrophy of skeletal muscle[END_REF]. This activityinduced muscle growth has been extensively studied (for recent review see [START_REF] Schiaffino | Mechanisms regulating skeletal muscle growth and atrophy[END_REF][START_REF] Piccirillo | Mechanisms of muscle growth and atrophy in mammals and Drosophila[END_REF]. Numerous signaling molecules have been identified to be involved and robust literature supports the role of the IGF-1-PI3K-Akt-mTOR pathway as a positive regulator and Myostatin-Smad3 as a negative regulator of muscle mass [START_REF] Sandri | Signaling in muscle atrophy and hypertrophy[END_REF][START_REF] Wackerhage | Signal transduction pathways that regulate muscle growth[END_REF][START_REF] Miyazaki | Cellular mechanisms regulating protein synthesis and skeletal muscle hypertrophy in animals[END_REF][START_REF] Rodriguez | Myostatin and the skeletal muscle atrophy and hypertrophy signaling pathways[END_REF]. Also YAP was identified to contribute to the regulation of muscle mass, as overexpression of YAP is sufficient to induce skeletal muscle hypertrophy and the amount of YAP protein is increased in skeletal muscle cells after mechanical overload [START_REF] Goodman | Yes-associated protein is up-regulated by mechanical overload and is sufficient to induce skeletal muscle hypertrophy[END_REF].

Mechanosensitive calcium channels and the kinase domain of titin, a structural protein of the sarcomere, have so far been identified as mechanosensors in skeletal muscle [START_REF] Lange | The kinase domain of titin controls muscle gene expression and protein turnover[END_REF][START_REF] Benavides Damm | Calcium's role in mechanotransduction during muscle development[END_REF][START_REF] Bogomolovas | Titin kinase is an inactive pseudokinase scaffold that supports MuRF1 recruitment to the sarcomeric M-line[END_REF]. They undergo conformational changes in response to mechanical load and thereby initiate signaling pathways, which regulate muscle mass. However, the detailed mechanisms and involved signaling pathways remain controversial.

IGF-1-PI3K-Akt-mTOR

Upon contraction, Insulin-like growth factor I (IGF-I) is released by the muscles and acts as an autocrine muscle hormone leading to muscle growth. The importance of the IGF-I-PI3K-Akt-mTOR pathway signaling has been largely confirmed and the multi-protein complex mTORC1 emerged to play a fundamental role in the regulation of skeletal muscle mass by regulating protein synthesis and cell size [START_REF] Sandri | Signaling in muscle atrophy and hypertrophy[END_REF][START_REF] Frost | Multifaceted role of insulin-like growth factors and mammalian target of rapamycin in skeletal muscle[END_REF]. The activation of the serine/threonine-specific protein kinase Akt appears to be the crucial determinant of the cellular signaling processes and the transition point between atrophy and hypertrophy [START_REF] Brooks | Skeletal muscle wasting with disuse atrophy is multi-dimensional: the response and interaction of myonuclei, satellite cells and signaling pathways[END_REF]. Nevertheless, the regulation of Akt/mTOR, especially its dependence on autocrine IGF-1 stimulation, has been an ongoing discussion point [START_REF] Hornberger | Mechanotransduction and the regulation of protein synthesis in skeletal muscle[END_REF][START_REF] Spangenburg | Changes in muscle mass with mechanical load: possible cellular mechanisms[END_REF]. Clear evidence has been provided that mechanical loading is sufficient for Akt activation [START_REF] Nader | Intracellular signaling specificity in skeletal muscle in response to different modes of exercise[END_REF][START_REF] Bolster | Immediate response of mammalian target of rapamycin (mTOR)-mediated signalling following acute resistance exercise in rat skeletal muscle[END_REF][START_REF] Sakamoto | Akt signaling in skeletal muscle: regulation by exercise and passive stretch[END_REF] albeit mTOR signaling can also be mechanically activated in the absence of Akt in the mouse model [START_REF] Miyazaki | Early activation of mTORC1 signalling in response to mechanical overload is independent of phosphoinositide 3-kinase/Akt signalling[END_REF]. Interestingly, it has been shown that Akt binding to the cytoskeleton is dependent on mechanical stretch [START_REF] Sawada | Force transduction by Triton cytoskeletons[END_REF]. However, Hornberger and colleagues report that the inhibition of actin polymerization did not prevent Akt activation after mechanical strain [START_REF] Hornberger | Intracellular signaling specificity in response to uniaxial vs. multiaxial stretch: implications for mechanotransduction[END_REF]. However, the mechanisms responsible for the mechanical activation of mTORC1 signaling are not yet fully elucidated and have recently been reviewed by [START_REF] Goodman | The role of mTORC1 in regulating protein synthesis and skeletal muscle mass in response to various mechanical stimuli[END_REF].

Crosstalk between YAP and the mTOR/Akt Signaling

Since mTOR regulates organ size through cell growth by regulating protein synthesis and the Hippo pathway regulates organ size by regulating proliferation it seems apparent that these pathways are coordinately regulated [START_REF] Csibi | Hippo-YAP and mTOR pathways collaborate to regulate organ size[END_REF].

Akt has been reported to interact with the Hippo pathway via several routes. Akt has been shown to regulate YAP phosphorylation [START_REF] Basu | Akt phosphorylates the Yes-associated protein, YAP, to induce interaction with 14-3-3 and attenuation of p73-mediated apoptosis[END_REF] but not by phosphorylating YAP directly [START_REF] Zhao | Inactivation of YAP oncoprotein by the Hippo pathway is involved in cell contact inhibition and tissue growth control[END_REF] but presumably through its interaction with MST1/2. In Drosophila, PI3K mediated Akt activity was shown to regulate the phosphorylation of yorkie (Yki), the YAP ortholog, most likely via activation of the MST1/2 ortholog hippo (hpo) or even upstream of hpo [START_REF] Straßburger | Insulin/IGF signaling drives cell proliferation in part via Yorkie/YAP[END_REF]; Figure 1). In mammals, MST1/2 was already shown to be a binding partner of Akt and to reduce Akt activity [START_REF] Cinar | The pro-apoptotic kinase Mst1 and its caspase cleavage products are direct inhibitors of Akt1[END_REF]. Vice versa, MST1/2 was also found to be inhibited by the interaction with Akt [START_REF] Jang | Akt phosphorylates MstI and prevents its proteolytic activation, blocking FOXO3 phosphorylation and nuclear translocation[END_REF]. Finally, data demonstrate that the phosphorylation of MST1/2 can be induced by the mTOR signaling pathway and restrict MST1/2 function to inhibit cell growth in prostate cancer cells [START_REF] Collak | Threonine-120 phosphorylation regulated by phosphoinositide-3-Kinase/Akt and mammalian target of rapamycin pathway signaling limits the antitumor activity of mammalian sterile 20-Like Kinase 1[END_REF].

Besides protein-protein interactions, transcriptional crosstalks have been identified. Hippo pathway activity negatively regulates Akt transcription in Drosophilia [START_REF] Straßburger | Insulin/IGF signaling drives cell proliferation in part via Yorkie/YAP[END_REF][START_REF] Ye | Akt is negatively regulated by Hippo signaling for growth inhibition in Drosophila[END_REF]. In human cell lines, the Hippo pathway was shown to regulate mTOR activity via the microRNA-29 (miR-29). YAP activity leads to the expression of miR-29 which inhibits the translation of PTEN, a phosphatase, which in its active state inhibits Akt (Figure 1). Consistently, YAP overexpression or Lats1/2 knockdown increase mTOR activity in skin sections [START_REF] Tumaneng | YAP mediates crosstalk between the Hippo and PI(3)K-TOR pathways by suppressing PTEN via miR-29[END_REF]. However, YAP overexpression induced muscle hypertrophy was recently shown to act through an mTORC1-independent mechanism [START_REF] Goodman | Yes-associated protein is up-regulated by mechanical overload and is sufficient to induce skeletal muscle hypertrophy[END_REF].

YAP IN SKELETAL MUSCLE PHYSIOLOGY AND DISEASES

The role of YAP in cardiac muscle and its regeneration has emerged as a promising field of research [START_REF] Papizan | Hippo in the path to heart repair[END_REF][START_REF] Wackerhage | The Hippo signal transduction network in skeletal and cardiac muscle[END_REF][START_REF] Zhou | The hippo pathway in heart development, regeneration, and diseases[END_REF], but knowledge about YAP function in skeletal muscle is still limited. Interestingly, even before the discovery of YAP protein itself, evidence of its importance in muscle was supported by the identification of the muscle promoter elements MCAT, which are regulated by TEAD family transcription factors and are found in promotors of genes coding for contractile proteins (e.g., β-myosin heavy chain, skeletal α-actin) and regulators of myogenic differentiation (Myf5, Mrf4, myogenin; [START_REF] Mar | A conserved CATTCCT motif is required for skeletal muscle-specific activity of the cardiac troponin T gene promoter[END_REF][START_REF] Yoshida | MCAT elements and the TEF-1 family of transcription factors in muscle development and disease[END_REF][START_REF] Ribas | Members of the TEAD family of transcription factors regulate the expression of Myf5 in ventral somitic compartments[END_REF][START_REF] Benhaddou | Transcription factor TEAD4 regulates expression of myogenin and the unfolded protein response genes during C2C12 cell differentiation[END_REF]. Furthermore, transgenic overexpression of TEAD-1 in mouse muscle leads to a change in myosin heavy chain isoform expression and therefore to a transition from fast to slow oxidative fiber phenotypes [START_REF] Tsika | Overexpression of TEAD-1 in transgenic mouse striated muscles produces a slower skeletal muscle contractile phenotype[END_REF]. This indicates, that YAP activity regulates the transcription of genes important for muscle development, homeostasis and plasticity.

YAP in Skeletal Muscle Myogenesis and Regeneration

For muscle growth and regeneration, activated satellite cells expand, differentiate and then fuse with existing myofibers [START_REF] Zhang | Use of antibodies to identify satellite cells with a light microscope[END_REF]. In their quiescent state, Pax7 expressing muscle stem cells are located between the basal lamina and plasma membrane of the myofiber [START_REF] Lepper | Inducible lineage tracing of Pax7-descendant cells reveals embryonic origin of adult satellite cells[END_REF]. Upon activation, satellite cells start expressing Myf5 and MyoD and proliferate via asymmetric division. Part of this expanded satellite cell pool then undergoes differentiation, marked by myogenin expression and complete downregulation of Pax7. Finally, activated myoblasts fuse with existing myofibers, while the remaining pool of satellite cells self-renews and returns to quiescence [START_REF] Tedesco | Repairing skeletal muscle: regenerative potential of skeletal muscle stem cells[END_REF]; Figure 3).

In this process, high YAP activity promotes proliferation of activated Pax7+ and MyoD+ muscle progenitor cells while YAP inactivation is needed for myogenic differentiation (Figure 3). Changes in YAP activity during satellite cell maturation have been shown in vitro and ex vivo on murine myoblasts. They show predominantly nuclear YAP during culture and YAP cytoplasmic translocation after myogenic differentiation along with decreased Yap mRNA and protein levels and increased YAP phosphorylation [START_REF] Watt | Yap is a novel regulator of C2C12 myogenesis[END_REF][START_REF] Judson | The Hippo pathway member Yap plays a key role in influencing fate decisions in muscle satellite cells[END_REF]. Moreover, YAP knockdown reduces proliferation of satellite cellderived myoblasts but has no impact on progression of their differentiation [START_REF] Nagata | Entry of muscle satellite cells into the cell cycle requires sphingolipid signaling[END_REF]. Further, evidence for the inhibition of skeletal muscle differentiation by YAP activity has been found in vivo on Xenopus laevis embryos, as YAP overexpression leads to inhibition of MyoD expression [START_REF] Gee | Yes-associated protein 65 (YAP) expands neural progenitors and regulates Pax3 expression in the neural plate border zone[END_REF] and on mouse skeletal myofibers, which show reduced YAP levels during postnatal maturation [START_REF] Watt | The Hippo pathway effector YAP is a critical regulator of skeletal muscle fibre size[END_REF]. In vitro overexpression of constitutively active YAP in myoblast precursors results in increased Cyclin D1 and Myf5 expression as well as decreased myogenin, Mef2c and p21 expression, which inhibits terminal myogenic differentiation [START_REF] Ishibashi | MyoD induces myogenic differentiation through cooperation of its NH2-and COOH-terminal regions[END_REF][START_REF] De Falco | Involvement of cdks and cyclins in muscle differentiation[END_REF][START_REF] Watt | Yap is a novel regulator of C2C12 myogenesis[END_REF].

Regarding YAP regulation, MST1 is activated during myoblast differentiation by caspase3 and active MST1 is needed for proper myoblast differentiation [START_REF] Fernando | Caspase 3 activity is required for skeletal muscle differentiation[END_REF]. Furthermore, YAP has also been claimed to be involved in the activation of satellite cells by sphingosine-1-phosphate (S1P) mediated YAP activation [START_REF] Nagata | Entry of muscle satellite cells into the cell cycle requires sphingolipid signaling[END_REF][START_REF] Yu | Regulation of the Hippo-YAP pathway by G-protein-coupled receptor signaling[END_REF]; Figure 1).

Also, culture conditions show evidence for a YAP dependent regulation of satellite cell differentiation. Established protocols optimized for myogenic differentiation share similarities with those for inactivation of YAP as they recommend high cell density, reduced serum conditions and substrates softer than standard cell culture plastic [START_REF] Yaffe | Serial passaging and differentiation of myogenic cells isolated from dystrophic mouse muscle[END_REF][START_REF] Kaushik | From stem cells to cardiomyocytes: the role of forces in cardiac maturation, aging, and disease[END_REF].

YAP in Skeletal Muscle Homeostasis and Disease

In adult skeletal muscle, major Hippo pathway components including YAP are expressed in fast and slow muscle [START_REF] Watt | Yap is a novel regulator of C2C12 myogenesis[END_REF]. In healthy muscle sections, YAP staining is weak and predominantly cytoplasmic, suggesting that YAP does not play a transcriptional role in the function of adult muscle [START_REF] Crose | Alveolar rhabdomyosarcoma-associated PAX3-FOXO1 promotes tumorigenesis via Hippo pathway suppression[END_REF]. However, there are conflicting data on the role of YAP in muscle homeostasis and organ size, including atrophy and hypertrophy. Judson and colleagues report that high levels of a constitutively active YAP mutant drive degeneration, atrophy and necrosis in skeletal muscle fibers by use of a skeletal muscle fiber but not satellite cell specific knock-in mouse model [START_REF] Judson | Constitutive expression of Yes-associated protein (Yap) in adult skeletal muscle fibres induces muscle atrophy and myopathy[END_REF]. Gene expression profiling of these mice show similarities to muscles from mdx mice, a model for Duchenne muscular dystrophy [START_REF] Hoffman | Dystrophin: the protein product of the Duchenne muscular dystrophy locus[END_REF]. Interestingly, this muscle wasting phenotype is largely reversible as inactivation of the transgene rescues the degenerative phenotype.

Watt and colleagues on the other hand found YAP as a positive regulator of skeletal muscle size through a TEAD-dependent but mTOR-independent regulation of protein synthesis after knockdown or overexpression of YAP. Furthermore, they report YAP to limit neurogenic atrophy following muscle denervation, since YAP knock out prior to denervation dramatically increased atrophy of muscles [START_REF] Watt | The Hippo pathway effector YAP is a critical regulator of skeletal muscle fibre size[END_REF]. Goodman and colleagues support the hypertrophic role of YAP in muscle. The chronic mechanical overload model in mouse, which leads to a progressive increase in muscle mass, shows increased YAP expression and phosphorylation. Vice versa, overexpression of YAP in the mouse tibialis anterior leads to hypertrophy [START_REF] Goodman | Yes-associated protein is up-regulated by mechanical overload and is sufficient to induce skeletal muscle hypertrophy[END_REF]. Furthermore, increasing muscle mass by blocking myostatin and activin signaling in mice in vivo leads to increased total YAP and YAP phosphorylation. Finally, physical exercises also increase YAP phosphorylation levels in mouse limb muscles [START_REF] Hulmi | Muscle protein synthesis, mTORC1/MAPK/Hippo signaling, and capillary density are altered by blocking of myostatin and activins[END_REF].

A possible explanations for the contrasting results on the role of YAP in muscle might be the different time points, which have been analyzed, or the use of different YAP mutants, as the constitutive active YAP S127A mutant, in contrast to wild type YAP, cannot be subject to negative feedback regulation.

The upstream regulation of YAP during muscle homeostasis remains poorly characterized so far. By examining neurogenic atrophy, MST1 expression was found to be upregulated in fast-but not slow-dominant muscle and knockout of MST1 attenuated fast-dominant skeletal muscle wasting. Whether YAP phosphorylation and activity are affected here, has not been determined [START_REF] Wei | MST1, a key player, in enhancing fast skeletal muscle atrophy[END_REF].

YAP signaling is also implicated in skeletal muscle diseases. Rhabdomyosarcomas are cancers of skeletal muscle tissue that are divided into different subtypes, the two main ones being embryonal rhabdomyosarcoma (eRMS) and alveolar rhabdomyosarcoma (aRMS). Levels of YAP phosphorylation show high variability between different RMS cell lines. Total YAP protein levels, however, are elevated in RMS cells and histological RMS tumor sections show increased nuclear YAP stainings [START_REF] Crose | Alveolar rhabdomyosarcoma-associated PAX3-FOXO1 promotes tumorigenesis via Hippo pathway suppression[END_REF]. Overexpression of constitutively active YAP in activated but not quiescent satellite cells leads to muscle tumors similar to eRMS. In vitro and in vivo YAP knockdown experiments revealed that lowering YAP expression in human eRMS can rescue tumorigenicity [START_REF] Tremblay | The Hippo transducer YAP1 transforms activated satellite cells and is a potent effector of embryonal rhabdomyosarcoma formation[END_REF]. aRMS is characterized by expression of the paired box 3-forkhead box protein O1 (PAX3-FOXO1) [START_REF] Galili | Fusion of a fork head domain gene to PAX3 in the solid tumour alveolar rhabdomyosarcoma[END_REF][START_REF] Shapiro | Fusion of PAX3 to a member of the forkhead family of transcription factors in human alveolar rhabdomyosarcoma[END_REF]. PAX3-FOXO1 directly upregulates RASSF4 in aRMS cells and tumors, which associates with MST1 and inhibits its tumor suppressor function, leading to tumorgenesis [START_REF] Crose | Alveolar rhabdomyosarcoma-associated PAX3-FOXO1 promotes tumorigenesis via Hippo pathway suppression[END_REF].

Furthermore, muscles of mdx mice show elevated levels of phosphorylated and total YAP protein [START_REF] Judson | Constitutive expression of Yes-associated protein (Yap) in adult skeletal muscle fibres induces muscle atrophy and myopathy[END_REF]. Finally, the first evidence of an involvement of YAPmediated mechanosensing defects in patients with LMNArelated congenital muscular dystrophy has been reported recently [START_REF] Bertrand | Cellular microenvironments reveal defective mechanosensing responses and elevated YAP signaling in LMNA-mutated muscle precursors[END_REF]. Along with several defects in the organization of the cytoskeleton, YAP was found not to respond to the changing mechanical properties of their environment. While YAP is excluded from the nucleus in soft environment in healthy control cells, patient derived cells maintain nuclear YAP localization in soft environment.

CONCLUDING REMARKS AND PERSPECTIVES

YAP has emerged as an important player in mechanotransduction, transmitting mechanical cues into a transcriptional cell response. At the same time, YAP has been shown to be involved in skeletal muscle development and regeneration, as YAP contributes to the regulation of activation, proliferation and differentiation of satellite cells. Beyond that, YAP signaling is also important in adult skeletal muscle homeostasis as misregulation can lead to atrophy or hypertrophy and aberrant YAP activities have been observed in disease states, including skeletal muscle dystrophies. Adult muscle homeostasis again is mainly regulated by muscle activity, which is a mechanical cue itself. Akt-mTOR signaling is widely accepted as the main regulatory pathway defining muscle mass, but the autocrine activation of this pathway by IGF-1 is controversial and several crosstalks to YAP have been identified. Precise mechanisms by which YAP is regulated by mechanical cues are still unknown. Cytoskeletal and presumably also nucleoskeletal tension, in particular actin dynamics and Rho signaling, have been identified as important players of mechanotransduction on YAP, but the detailed mechanism still remains to be elucidated. Uncovering this mechanism, also in regard to the specially organized cytoskeleton of postmitotic myofibers, could reveal important insights into our understanding of muscle homeostasis and subsequently into the physiopathology of muscle diseases.

FIGURE 2 |

 2 FIGURE 2 | Different mechanical inputs regulate YAP activity. YAP (Yes-associated protein) is localized to the nucleus and active under mechanical conditions that lead to high intracellular tension such as a large adhesive area (A), stiff extracellular matrix (ECM) (B), non-bendable substrates (C), cell stretching (D), or fluid shear stress (E). Conditions favoring low contractile forces in the cell, such as small adhesive areas, soft ECM, bendable substrates, relaxation of stretching forces or culture in static media, lead to YAP inactivation by nuclear exclusion.

FIGURE 3 |

 3 FIGURE 3 | Regulation of YAP level and activity during satellite cell differentiation. After activation of quiescent satellite cells (SC), SCs divide and differentiate into myotubes that fuse with existing myofibers or self-renew and return to quiescence. During SC activation, YAP expression increases until this fate decision has been made. In differentiating SCs, YAP is inactivated by increased phosphorylation at Serine 127 (pYAP S127).

Frontiers in Physiology | www.frontiersin.org

February 2016 | Volume 7 | Article 41

ACKNOWLEDGMENTS

This work has been supported by DFG funding and the Sorbonne Universités UPMC Univ Paris through Myograd (fellowship to MF); by the DFG through the Berlin-Brandenburg School for Regenerative Therapies GSC 203 (fellowship to PR) and by the DFG through the Research Unit RU2165 "Regeneration in Aged Individuals" (to PK). We also thank Prof. Dr. Hans-Georg Simon for helpful discussions.

AUTHOR CONTRIBUTIONS

MF conceived and wrote the article, PR contributed to the writing and designed the figures. PK and CC revised the article. All authors approved the work for publication.

Conflict of Interest Statement:

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Fischer, Rikeit, Knaus and Coirault. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.


Sagar Verma
email: sagar.verma@centralesupelec.fr

Richa Verma

P B Sujit
email: sujit@iiitd.ac.in

MAPEL: Multi-Agent Pursuer-Evader Learning using Situation Report

Keywords: multi-agent learning, deep reinforcement learning, recurrent neural network

In this paper, we consider a territory guarding game involving pursuers, evaders and a target in an environment that contains obstacles. The goal of the evaders is to capture the target, while that of the pursuers is to capture the evaders before they reach the target. All the agents have limited sensing range and can only detect each other when they are in their observation space. We focus on the challenge of effective cooperation between agents of a team. Finding exact solutions for such multi-agent systems is difficult because of the inherent complexity. We present Multi-Agent Pursuer-Evader Learning (MAPEL), a class of algorithms that use spatio-temporal graph representation to learn structured cooperation. The key concept is that the learning takes place in a decentralized manner and agents use situation report updates to learn about the whole environment from each others' partial observations. We use Recurrent Neural Networks (RNNs) to parameterize the spatio-temporal graph. An agent in MAPEL only updates all the other agents if an opponent or the target is inside its observation space by using situation report. We present two methods for cooperation via situation report update: a) Peer-to-Peer Situation Report (P2PSR) and b) Ring Situation Report (RSR). We present a detailed analysis of how these two cooperation methods perform when the number of agents in the game are increased. We provide empirical results to show how agents cooperate under these two methods.

I. INTRODUCTION

Multi-agent systems have received much attention in the past decade [START_REF] Kuo | Multiagent cooperative learning strategies for pursuit-evasion games[END_REF]- [START_REF] Mirowski | Learning to navigate in complex environments[END_REF]. In such systems, agents share a common environment, where they act independently or in cooperation with each other to achieve a combined goal. We focus on the problem where multiple agents achieve a single task in cooperation with each other. In such problems, the agents must have the ability to handle unknown and uncertain scenarios and take the success of the whole team into account.

A multi-agent pursuit-evasion game complexity depends on many variables like the type of environment, the observation of the agents, their actions, cooperation strategy, and the reward structure. Being complex and dynamic, pursuit-evasion problems are challenging to solve [START_REF] Antoniades | Pursuit-evasion strategies for teams of multiple agents with incomplete information[END_REF]. Such complexities have been addressed by stochastic modeling of agents motion in [START_REF] Hespanha | Greedy control for hybrid pursuit-evasion games[END_REF], [START_REF] Guibas | A visibility-based pursuit-evasion problem[END_REF]. There has been growing interest in modeling the game, in which the evader is intelligent and has certain sensing capabilities [START_REF] Vidal | Probabilistic pursuit-evasion games: theory, implementation, and experimental evaluation[END_REF]. This paper focuses on the problem of partial observation of agents where structured message passing is used for cooperation. We present a zero-sum game based on pursuit and evasion between two teams of equal number agents. Since either the pursuer or the evader wins the game, therefore, the game can be represented as a zero-sum game. The multi-agent pursuit-evasion problem is shown in Figure 1. We assume partial observability of the environment to ensure that the solution is usable in many real-world applications that closely correspond to the task in hand. However, learning becomes more difficult under partial observation along with complex interactions between agents and the environment. Each agent perceives the environment locally and even though the effect of other agents' actions on the environment is visible but the agents, themselves, are not. Reinforcement learning (RL) has been used to solve multi-agent pursuit-evasion in [START_REF] Kuo | Multiagent cooperative learning strategies for pursuit-evasion games[END_REF], [START_REF] Parker | Distributed algorithms for multi-robot observation of multiple moving targets[END_REF]- [START_REF] Bilgin | An approach to multi-agent pursuit evasion games using reinforcement learning[END_REF]. Recent works like [START_REF] Zhang | Model-free reinforcement learning for fully cooperative multi-agent graphical games[END_REF]- [START_REF] Leibo | Multi-agent reinforcement learning in sequential social dilemmas[END_REF] use deep reinforcement learning for different multi-agent problems where centralized policy learning is employed. All these works deal with full observation and are not suitable for our problem. We propose MAPEL, a class of deep reinforcement learning based methods that uses spatio-temporal graphs for structured cooperation between agents under partial observation.

To solve our multi-agent pursuit-evasion game, we present MAPEL which uses spatio-temporal graphs to structure the cooperation between agents in a team. We propose using abstract messages called situation reports which are shared among agents for cooperation. We present two different methods for situation report update which are based on dense and sparse communication. MAPEL can handle pursuers and evaders which move at the same speed throughout the game, which means that neither pursuers nor evaders have an advantage over each other. We show that MAPEL cooperation methods lead to a high degree of cooperation between agents. We also show how the two cooperation methods perform when the number of agents in the team is increased.

The remainder of this paper is organized as follows. Section 2 mentions the existing works related to this paper. Section 3 describes the problem and its formulation as a multi-agent reinforcement learning (MARL) problem. Section 4 presents MAPEL and other proposed baselines. Section 5 explains the experimental setup followed by the results and their explanation in section 6. Section 7 concludes this paper.

II. RELATED WORK

Reinforcement learning has been successfully used to play games like Atari [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] and Go [START_REF] Silver | Mastering the game of go without human knowledge[END_REF]. In [START_REF] Liu | A pursuit-evasion algorithm based on hierarchical reinforcement learning[END_REF], the authors suggest an approach based on hierarchical RL for the same, while enabling the players to learn through tasks with less complexity. Multi-agent reinforcement learning (MARL) consists of a set of learning agents that share a common environment [START_REF] Busoniu | A comprehensive survey of multiagent reinforcement learning[END_REF]. Learning in such a framework is fundamentally difficult because of the interaction arising between the agents and the environment and amongst themselves. Conventional decentralized learning techniques like Q learning for each agent [START_REF] Tan | Multi-agent reinforcement learning: Independent vs. cooperative agents[END_REF] assume the other agents to be a part of the environment. Such methods don't work in multi-agent settings because the theoretical convergence guarantee no longer holds and makes the learning unstable due to the fact that changes in the policy of any agent will affect the policies of the other agents, as well [START_REF] Matignon | Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems[END_REF].

Joint action learning or centralized policy learning is one way to do multi-agent reinforcement learning. [START_REF] Hong | A deep policy inference q-network for multi-agent systems[END_REF] present a deep policy inference Q-network that targets multi-agent systems composed of controllable agents. A centralized policy for the controllable agent is learned from its raw observations. [START_REF] Omidshafiei | Deep decentralized multi-task multi-agent reinforcement learning under partial observability[END_REF] presents joint and independent policy learning methods. In an independent policy learning method, the joint learned policy is transferred to individual agents in an iterative manner. [START_REF] Gupta | Cooperative multi-agent control using deep reinforcement learning[END_REF] discusses why centralized policy learning fails in case of multi-agent setting and presents methods to learn policy for heterogeneous agents as well as homogeneous agents. Sunehag et al. [START_REF] Sunehag | Value-decomposition networks for cooperative multi-agent learning based on team reward[END_REF] discuss the problem of "lazy agents" which is when some agents remain inactive when a centralized policy learning is used. They present a value-decomposition network which enables better reward sharing between agents to solve the problem of inactive agents.

Decentralized learning requires effective cooperation between different agents. [START_REF] Foerster | Learning with opponent-learning awareness[END_REF] suggest learning with opponentlearning awareness method in which each agent anticipates other agent's policy. This method only works for complete observation. [START_REF] Palmer | Lenient multi-agent deep reinforcement learning[END_REF] discusses the problem of experience replay in multi-agent deep reinforcement learning (MA-DRL). They state that transitions stored in experience replay memory (ERM) can become outdated because agents update their policies in parallel. They apply leniency to MA-DRL by mapping agents state-action pairs to decaying temperature values that control the amount of leniency applied towards negative policy updates that are sampled from the ERM. They also state that this help in better cooperation among agents. [START_REF] Lowe | Multi-agent actor-critic for mixed cooperative-competitive environments[END_REF] use actor-critic to learn policies for complex cooperation. [START_REF] Foerster | Counterfactual multi-agent policy gradients[END_REF] uses centralized critic to estimate the Q-value, decentralized actors are used to optimize agents' policies, and counterfactual baselines are used to solve multi-agent credit assignment problem. [START_REF] Yong | Cooperative coevolution of multiagent systems[END_REF] presents co-evolution methods to learn better coordination between agents. [START_REF] Pinheiro | Local wealth redistribution promotes cooperation in multiagent systems[END_REF] present a method to solve cooperation between agents that can act selfishly.

Some multi-agent problems can be explicitly described as graphs. [START_REF] Shao | Cooperative reinforcement learning for multiple units combat in starcraft[END_REF] presents cooperative reinforcement learning for multiple agents in StarCraft game. [START_REF] Hu | Reinforcement learning for multi-agent patrol policy[END_REF] uses graph representation with reinforcement learning for coordination and cooperation in multi-agent patrol task. Efficient state representation based on the distance between agents and different game entities is used to reduce the observation state complexity. [START_REF] Marzagão | Multi-agent flag coordination games[END_REF] presents a flag coordination game where graph structure is explicitly present and is utilized to model multi-agent coordination. Some problems where there are no explicit graph structures present, game states can be decomposed into some weak time-varying structures. Such structures can be learned using factor graph representation and graphical learning methods. [START_REF] Bryant | A Neuroevolutionary Approach to Adaptive Multi-agent Teams[END_REF] discusses use cases where cooperation is explicitly required. A genetic algorithm variation is used to solve the adaptive team of agents (ATA) problem. Their method can adapt an agent to a new role based on the overall structure of the environment. [START_REF] Guestrin | Coordinated reinforcement learning[END_REF] presents joint policy learning method for coordinated reinforcement learning through structured communication between agents. [START_REF] Zhang | Clique-based cooperative multiagent reinforcement learning using factor graphs[END_REF] presents a way to decompose a global Q-function into local Q-function based on the task decomposition between agents expressed using factor graphs. [START_REF] Zhang | Cooperative multiagent reinforcement learning using factor graphs[END_REF] divide agents into cliques based on specific tasks. [START_REF] Amato | Scalable planning and learning for multiagent pomdps[END_REF] uses factor graphs to learn implicit structures present in multi-agent settings. Factor graphs reduce the action and observation space and learning agents' policy becomes easier.

In the literature, RL has been used earlier for the classic pursuer-evader game [START_REF] Isaacs | Differential games: a mathematical theory with applications to warfare and pursuit, control and optimization[END_REF]. In [START_REF] Liu | A novel approach based on evolutionary game theoretic model for multi-player pursuit evasion[END_REF], a learning technique for multi-player pursuit-evasion games is presented for discrete state and action spaces. The proposed algorithm is only applicable for multi-player pursuit-evasion games with superior pursuers (in terms of speed). The article [START_REF] Wang | Research on pursuit-evasion games with multiple heterogeneous pursuers and a high speed evader[END_REF] is another work suggesting a technique using learning in differential multiplayer pursuit-evasion games that have superior evaders. In [START_REF] Alexopoulos | Cooperative pursue in pursuit-evasion games with unmanned aerial vehicles[END_REF] hierarchical decomposition is used to solve games having two pursuers and one evader.

III. PROBLEM DEFINITION

We model the multi-agent pursuer-evader problem as a grid world of dimension M × N in which obstacles are placed randomly (uniform distribution N (0, σ)). In this grid, there are P pursuers, E evaders, and a single target T . At any time t, a pursuer p ∈ P has the global knowledge about all pursuer locations and the current target location in the environment. An evader e ∈ E is assumed to know the locations of all other evaders and the target. We assume each agent can sense a rectangular region of length l and width w. However, the agents cannot sense on the other side of the obstacle. That is, a pursuer can detect an evader if they are in line-of-sight and within the sensed region. The speed of all the pursuers and the evaders is given by v and remains constant throughout the game. The target, T remains stationary throughout the game.

A game starts with randomly sized obstacles placed on the grid at random locations as shown in Figure 1 (for a 2-pursuers vs 2-evaders game). The target is spawned at a random location near the middle of the grid (M/2, N/2) and it is of length t s . The pursuers and the evaders are randomly spawned on the opposite sides of the grid. The pursuers and the evaders can move to any of the adjacent cells of the grid only if the cell is either empty or occupied by any of the agents. An agent reaches the target when its location is same as the target's location. Also, a pursuer captures an evader only if their locations are the same. Once an evader is captured by a pursuer, it cannot move anywhere else but the pursuer can move to an adjacent cell after catching the evader.

There are three conditions for a game to complete. 1) An evader reaches the target, in which case the evaders win the game. 2) A pursuer reaches the target before an evader, in which case the pursuers win the game. 3) All the evaders are captured by the pursuers, in which case the pursuers win the game. Based on the three different winning criteria we have the following reward structure:

1) When the evaders win by capturing the target, a reward of w e = 0.5 is awarded to them and a penalty of w p = -0.5 is given to the pursuers. 2) When the pursuers win by reaching the target before the evaders, a reward of w p = 0.5 is awarded to, and a penalty of w e = -0.5 is given to the evaders. 3) When the pursuers win by capturing all the evaders, a reward of w p = 1 is awarded to the pursuers and a penalty of w e = -1 is given to the pursuers. Rewards are equally divided among all the agents of a team. This makes it sure that agents in a team do not compete with each other.

IV. METHODS

In this section we first present a naive method in which an agent greedily moves towards the target, followed by the second method which is a multi-agent formulation of deep Qlearning and then we introduce the proposed method MAPEL with two different cooperation strategies.

A. Naive Method

A naive agent tries to move towards the target, T . Each agent has a partial view of the environment and knows the location of the other agents of its team. It also knows the location of the target. A naive agent moves towards the target in a straight line. If the next location on the line of sight towards the target is obstructed, it randomly chooses an adjacent location that is closest to the line. If a pursuer observes an evader in its observation space, it computes the shortest path to the evader and chooses its next location along that path. Similarly, if a pursuer/evader observes the target in its observation space, it computes the shortest path to the target and chooses its next location along that path. Also, if a pursuer observes the target and an evader or multiple evaders in its field of view, it computes the shortest paths to all of them and chooses its next location along the path that has the smallest length.

B. Multi-agent Q-learning

An N -agent stochastic game E is formalized by the tuple E = (S, A 1 , . . . , A N , R 1 , . . . , R N , T , γ), where S denotes the state space, and A i is the action space of agent j {1, . . . , N }. The reward function for agent j is defined as

R i : S × A 1 × • • • × A N → R,
determining the immediate reward. The transition probability is given by T : S × A 1 × . . . A N → P r(S). P r(S) is the collection of probability distributions over the state space S. The goal of agents is to find a policy π which maximizes the expected return G t , which is the discounted sum of rewards given by

G t = N i=1 T τ =t γ τ -t R i
τ , where T is the time-step when an episode ends, t denotes the current time-step, γ [0, 1) represents the reward discount factor, and R i τ is the reward received at time-step τ by agent A i .

The agents choose actions according to their policies. For agent i, the corresponding policy is defined as π i : S → P r(A i ), where P r(A i ) is the collection of probability distributions over agent i's action space A i . The joint policy of all the agents is given by π : π 1 ×, • • • × π N . The joint actions of all the agents is given by a : A 1 ×, . . . , ×A N . The value function of agent i given state s under the joint policy π is written as the expected cumulative discounted future reward:

v i π (s) = v i (s; π) = ∞ t=0 γ t E π,p r i t |s 0 = s, π (1)
The Q-function can then be defined within the framework of N -agent game based on the Bellman equation given the value function in equation ( 1) such that the Q-function Q i π : S × A 1 ×, . . . , A N → R of agent i under the joint policy π can be formulated as

Q i π = R i (s, a) + γE s -p v i π (s ) , (2)
where s is the state at the next time step. The value function v i π can be expressed in terms of the Q-function in equation (2) as

v i π = E a π Q j π (s, a)]. (3)
The Q-function for N -agent game in equation ( 2) extends the formulation for a single-agent game by considering the joint action taken by all agents a, and by taking the expectation over the joint action in equation (3).

C. Multi-Agent Pursuer-Evader Learning (MAPEL)

In the Q-learning method presented in the previous section, the joint policy π is dependent only on the current observation of all the agents combined. It is impossible for an agent to know anything about the observation of the other agents in that setting. Also, the size of the combined observation and action spaces increases exponentially with the number of agents. For a large number of agents, this could be problematic.

In this section, we present a spatio-temporal (st) architecture called MAPEL which allows agents to learn their individual policies by sharing their observations with each other by cooperating via situation reports. We represent a team of agents as an st-graph G = (N , E M , E T ), where N denotes the total number of agents, E M is the total number of edges between the agents i.e. the edges used to pass situation reports, and E T is the number of edges connecting agents at time T . Figure 3 shows an example st-graph capturing agent-agent interactions during a game. In the unrolled st-graph, two agents at a given time step t are connected with an undirected spatiotemporal edge e = (a i , a j ) E M , and two nodes at adjacent time steps are connected with an undirected temporal edge iff (a i , a j ) E T .

We parameterize the nodes N and edges E T using RNNs in our st-graph. The edges E M are used by nodes to pass situation reports to each other. The situation reports are used by agents to compute their actions at time t. The network architecture of MAPEL is illustrated in figure 2, each agent is represented by an RNN, the agents compute their observational features using a CNN and pass their own observations to other agents via situation reports. Each agent uses the situation reports received by other agents along with its current observation to compute its next action. RNN maintains history information about an agent. The situation report coupled with RNN is used to handle partial observability. Situation report provides an abstract and clear representation of the observation. This helps in reducing the hidden state representation noise which arises due to other agents changing their strategies. For example, if a pursuer observes the target or an evader, it can inform other pursuers about its observation via situation report. This could help the other pursuers in changing their decision to not go in the direction of this particular pursuer and search in other areas for the target or other evaders.

In real-world applications, we can have hundreds of agents and interaction among all of them may not be possible due to some physical constraints or simply because of high computational complexity. In most of the cases, it is not necessary to have dense communication between all the agents. Sparse communication structures can be used to learn effective cooperation. We present two situation report update methods that use the structures present in the game. Peer-to-Peer Situation Report (P2PSR): In Peer-to-Peer Situation Report method, all the agents can share situation report with each other. This is the case of dense communication. This means that in our st-graph representation for N nodes, we have E M = N (N -1)/2 edges. Figure 3 shows the st-graph representation of P2PSR. This type of cooperation is required when an agent wants to know what other agents are observing so that it does not explore their regions. The objective of the agents then becomes to minimize the search time and exploration area to complete the task.

Ring Situation Report (RSR): In Ring Situation Report method, agents are randomly chosen to form a ring. Each agent can only pass messages to its adjacent agents. The st-graph representation for this type of cooperation is given by Figure 3. For N nodes, we have E M = N edges for N > 2. This type of cooperation can be used to cordon off an area and search inside it. This does not require all the agents to know about the other agents' observations. An agent only needs to know what its adjacent agents are observing. This decreases the number of messages required to cooperate.

V. EXPERIMENTAL SETUP

We perform our experiments on the multi-agent pursuerevader environment presented in section III. We begin with explaining the environment representation, agent observation featurization, and representation of messages under different situation report methods. All the experiments have been conducted on a workstation with 1.2 GHz CPU, 256 GB RAM, NVIDIA V100 GPU and running Ubuntu 18.04. We use PyTorch [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] for network implementation.

A. Environment

The environment is a grid world composed of multiple grids of size 32 × 32. Figure 4 shows partial observation of agent in a 2 vs 2 game. The white regions are the empty regions where evaders and pursuers can move. An agent considers all grid cells outside its observation space to be empty. Evaders are blue, pursuers are green, and the target is red. Figure 4 (a) shows the observation space of evader 1, it can see all the grid cells in its observation space, it knows the locations of other evaders and the target. Similarly figure 4 (b) shows the observation space of evader 2. Figure 4 (c) shows the observation space of pursuer 1, it can see all the grid cells in its observation space, it knows the locations of other pursuers and the target. Similarly, the observation space of pursuer 2 is represented in figure 4 (d).

B. Agent Observation

For Q-learning, we need to represent an agent's observation as meaningful features. In our experiments, we found that raw RGB frames provide good observational for 2 vs 2 games but fail to generalize for more number of agents. We represent each type of entity in our environment as separate channels. We have five channels in our feature space, each of size 32 × 32. In the bottom left portion of figure 2, we show the featurization of observation of one of the pursuers in a 4 vs 4 game. The first channel shows the observation space of the agent, the second channel shows the position of the agent itself, the third channel shows the position of other agents, the fourth one shows the location of the target, and the fifth channel shows the location of the opponent(s) observed. This feature representation accurately incorporates all the information observed by an agent.

C. Message Representation

In P2PSR, an agent a i receives a situation report m t i in form of a vector of size N -1 at time t. The N -1 elements of the vector represent the messages from other agents, if the value of an element in the vector is 1, then it means the corresponding agent has seen the target or opponent(s) in its observation space. An element with value 0 means that the observation space is empty. In RSR, an agent a i receives a situation report message m t i in form of a vector with size 2 from two of its adjacent agents at time t.

D. Training

We train multi-agent DQN for learning evaders against naive pursuers and multi-agent DQN for learning pursuers against naive evaders. We train both MAPEL cooperation methods against naive agents. All our are trained for 400 epochs, 500 episodes per epoch. We use Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] optimizer to train all our models. Learning rate is varied over epochs, it starts with 0.001 and decays at every 200 epochs by one-tenth. To ensure exploration, -greedy starts at 1.0 and ends at 0.1. A discount factor of 0.99 is used. While training multi-agent DQN models, a history length of 5 observations is used. We use a batch size of 64 in all our experiments. We vary the number of agents per team from 2 to 5 for all our models. Following are the model variations that we train, 1) MA-DQN pursuers against naive evaders. 2) MA-DQN evaders against naive pursuers. 3) MAPEL-P2PSR pursuers against naive evaders. 4) MAPEL-P2PSR evaders against naive pursuers. 5) MAPEL-RSR pursuers against naive evaders. 6) MAPEL-RSR evaders against naive pursuers.

E. Evaluation

We evaluate our MA-DQN, MAPEL-P2PSR, and MAPEL-RSR evaders against naive pursuers and vice-versa. 100,000 episodes are used for all the evaluations. Average reward is reported for both evaders and pursuers. In the case of pursuers running different methods, we also report the total number of times pursuers were able to capture all the evaders. We call these results as "complete wins".

VI. RESULTS

Figure 5 compares different models' learning curve under the different number of agents for both evaders and pursuers. For the number of agents N = 1, 2, 3, both MAPEL cooperation methods, i.e., P2PSR and RSPRP have the same message length. In such cases, there is no fundamental difference between these methods. Therefore, we only train both methods when team sizes are more than 3. Figure 5 (a) shows the learning curve for evaders with MA-DQN and MAPEL-P2PSR when the number of agents is 2 for both evaders and pursuers. Similarly figure 5 (b) is for a 3 vs. 3 scenario for evaders against naive pursuers. Figure 5 (d) and (e) are for pursuers with MA-DQN and MAPEL-P2PSR against naive evaders when the numbers of agents are 2 and 3 respectively. Figure 5 (c) and (f) show all three methods for evaders and pursuers against their naive opponents when the number of agents is 5.

In all the scenarios, pursuers are able to score better rewards than evaders. We believe that the pursuers are able to learn about the strategy where capturing all the evaders maximizes their rewards. From figure 5 (c) and (f), it is evident that MAPEL-P2PSR for pursuers learns about capturing all evaders quickly as compared to MAPEL-RSR. After 350 epochs both the methods converge to same average rewards which shows that MAPEL-RSR has similar learning capabilities as MAPEL-P2PSR. We believe this is due to the fact that in the case of MAPEL-P2PSR, all pursuers know about all other pursuers' observations explicitly which helps them in knowing about "capture all evaders" strategy early. In the case of MAPEL-RSR, more epochs are required to learn about this strategy.

Table I compares the average rewards and complete wins of different methods for pursuers against naive evaders in four scenarios, i.e., 2 vs. 2, 3 vs. 3, 4 vs. 4, and 5 vs. 5. It can be seen that the naive method performs better than MA-DQN in all the scenarios. On rendering a few episodes, we find that MA-DQN pursuers are not able to find the shortest paths as compared to the naive method. For some of the successful episodes, we find that pursuers are able to beat the opponents when some of the team members are closer to the target as compared to the evaders. In 4 vs 4 and 5 vs 5 scenarios, we can see that MAPEL-P2PSR is ahead of MAPEL-RSR by 0.023 and 0.025 units of average reward respectively. This is in line with our earlier hypothesis that MAPEL-P2PSR is better at learning about "capture all evaders" strategy because of dense communication. This is evident from the "complete wins" in column 7 and 9. The difference in the average reward is still less when compared to difference in "complete wins" between the two methods.

Table II compares the average rewards and complete wins of different methods for evaders against naive pursuers in four scenarios, i.e., 2 vs. 2, 3 vs. 3, 4 vs. 4, and 5 vs. 5. Similar to the case of evaders, the naive method performs better than MA-DQN in all the scenarios. We also observe that the rewards from MAPEL-P2PSR and MAPEL-RSR for evader are smaller than the pursuers. The reason for this is that pursuers can learn about "capture all evaders" strategy to get more reward whereas pursuers don't have any such strategy to maximize their rewards further. The reason MAPEL methods perform better than naive and MA-DQN methods is that evaders can avoid the regions where pursuers have been observed by some members of the team.

VII. CONCLUSIONS AND FUTURE WORK

In this paper, we presented a variation of multi-agent pursuit-evasion game with partial observability. We also present MAPEL for multi-agent cooperative reinforcement learning to solve the game. We compare proposed MAPEL with two benchmarks; the naive method which is a greedy solution and a multi-agent DQN formulation. We perform experiments with varying number of agents to show the generalizability of the MAPEL cooperation methods. We empirically show that MAPEL cooperation methods are better at learning cooperation strategy by reporting the results of "capture all evaders" in the case of pursuers.

In the future, our goal would be to test the transfer-ability of MAPEL methods to games with more number of agents. We would also like to experiment under different game conditions like opponents with different speeds, non-equal team sizes, moving target, etc. We would also like to find effective ways of analyzing and comparing proposed cooperation methods.

Fig. 1 :

 1 Fig. 1: Top-left image shows the rendering of our multi-agent pursuerevader game. Top-right image shows the labelled description of the game environment. Bottom image shows the proposed cooperation method.

Fig. 2 :

 2 Fig. 2: Generalized architecture of MAPEL.

Fig. 3 :

 3 Fig.3: Spatio-temporal representation used for cooperation between agents. Top image shows cooperation between three agents using spatio-temporal graph unrolled in time. Bottom-left image shows peer-to-peer situation report method between four agents and bottomright image shows ring situation report method between four agents.

Fig. 4 :

 4 Fig. 4: Partial observation of individual agents in 2-pursuers vs 2evaders game. Each agent can see everything in its observation space and knows about the target and other team members' locations. An agent cannot see the observation of its team members.

Fig. 5 :

 5 Fig. 5: Learning curve comparison of different methods in different scenarios.

TABLE I :

 I Evaluation result of different methods for pursuers against naive evaders in different scenarios.

	Scenario Naive MA-DQN MAPEL-P2PSR MAPEL-RSR
	2 vs. 2	0.134	-0.279	0.419	NA
	3 vs. 3	0.153	-0.247	0.429	NA
	4 vs. 4	0.157	-0.225	0.423	0.416
	5 vs. 5	0.161	-0.217	0.417	0.419

TABLE II :

 II Evaluation result of different methods for evaders against naive pursuers in different scenarios.


Jean-Claude Bermond

Bela Bollobas

The Diameter of Graphs: a survey

come