Dr Iony Randrianirina

Keywords: Metaverse, Inventions, Utility Patent, Design Patent, Patentability, Public Order

The metaverse offers to users a lot of opportunities to develop inventions such as hardwares, softwares, tools and technologies enabling a completely realistic immersive experience. Inventions can be developed for and in the metaverse. Regarding such new technologies, any patent lawyer may raise the question of patentability: Under which conditions can an invention be patentable? The answer depends on the choice of law. Some inventions may be patentable according to major jurisdictions, providing that the patent applicant meets the legal requirements for patentability. But certain legal systems refuse to grant utility patents on the grounds of public order. This Chapter intends to analyse and discuss the patentability of inventions developed for and in the metaverse, with the understanding that utility patent issues are far more insightful than design patent ones.

INTRODUCTION

The metaverse has much to offer to users, who are able to engage in creative collaboration with other users via their own avatars (or virtual representations), thus to design, build, and own virtual real creations. There are three categories of metaverse: Online game platforms such as Minecraft, Fortnite, and Roblox, also known as 'massively multiplayer online role-playing games' or MMORPG; Social media platforms such as Horizon Workrooms, created by Meta, and Second Life known to be the first metaverse platform; And corporate or industrial metaverse platforms, designed for internal use only, like the one created by the French Corporation Dassault Group. Roblox is another online game platform which allows users to program and play games through a virtual currency called Robux. In contrast to a private or centralized version of a metaverse, a crypto metaverse is decentralized and based on blockchain technologies. No one person or entity owns or controls a decentralized metaverse, as compared with proprietary metaverses. Meta Platforms, Inc. and Microsoft are both pursuing proprietary metaverses, whereas The Sandbox and Decentraland are creating decentralized metaverse platforms that are developed and governed by the users.

A large number of metaverse platforms use extended reality technologies (XR) in the form of augmented reality (AR) technology, virtual reality (VR) system or mixed reality program (MR). In the first case, the metaverse places virtual objects in actual space: Such a technology is called augmented reality (AR). For example, Niantic, Inc., the publisher of the popular Pokémon game series, published Pokémon Go, an AR mobile game. Each user in Pokémon Go had its own avatar, which was placed in a virtual map corresponding to a realworld geographic location. As users walked around, so did their avatars. The mobile game also offered an AR mode, which displayed different Pokémon appearing in the realworld environment by superimposing them on the view taken in with the camera on a user's device. Ikea also uses AR technology allowing consumers to place virtual copies of Ikea furniture in a room to test how they look and if they fit with existing furniture. In contrast, in the second case, the user may be placed in an immersive experience of a virtual world to potentially interact with others: Such technology is known as virtual reality (VR). With this VR system, metaverse users can attend virtual shows, like the live one staged by Young Thug, Chainsmokers, and David Guetta in Horizon Worlds. The VR headset-the Oculus Rift-offered a new, immersive portal to virtual worlds and metaverses. This is what Second Life offered back in 2003, and what the metaverses are trying to offer today and promises to offer in the future. In February 2019, Fortnite, one of the most popular online video games, where players would build forts and defend them against attacks of zombie armies or travel to distant virtual worlds, began to evolve from its game playing roots into a more immersive shared experience, with its first ever live concert by the band Marshmello, attended by over 10 million players. Fortnite's virtual world (or metaverse) showcases that the opportunity for virtual shared experiences in the metaverse is constantly evolving. Some technologies allow physical and virtual objects co-exist in mixed reality environments (MR) and interact in real time. Mixed reality that incorporates haptic technology has sometimes been referred to as visuo-haptic mixed reality. Mixed reality has been used in applications across fields including design, education, entertainment, military training, healthcare, product content management, and human-in-the-loop operation of robots. The movie Ready Player One, 1 directed by Steven Spielberg in 2018, provides a perfect overview of the effects of haptic technology. The film featured people entering an interactive virtual world called Oasis, using goggles like the Oculus Rift.25 and body armor which provides them with haptic feedback, so that when the avatar is hit in Oasis, its user feels it in the real world.

Obviously, the metaverse is also an ideal advertising space for corporations, with virtual billboards being seen by millions of people. Such an immersive world brings with it both new opportunities and new challenges for intellectual property, especially patents. The legal issue concerns the patentability of creations made in the virtual worlds such as Decentraland and The Sandbox. Are inventions made in the metaverse patentable?

Patents come in two forms: utility patents covering novel and useful inventions, and design patents covering ornamental design. Creations made in the metaverse could be granted both utility or design patents. Indeed, designers may patent only creations which are qualified as inventions. However, the notion of invention is not legally defined. For example, neither the Paris Convention for the Protection of Industrial Property, 2 nor the Patent Cooperation Treaty 3 provide a definition of 'invention'. The European Patent Convention only gives a negative definition when expressly excluding discoveries, scientific theories and mathematical methods, aesthetic creations, schemes, rules and methods for performing mental acts, playing games or doing business, programs for computers and presentations of information. 4 Western legislators have not made any attempts to define the term 'invention'. The legal definition was left to scholars and the courts through a number of case law tried to aid what constitutes invention. 5 According to French Professor Michel Vivant, "there is invention each time an intellectual process, whatever it is, allows to end in an innovation being supported on knowledge stemming from hard sciences."6 Invention is also defined by U.S. case law to include "any new and useful process, machine, manufacture or composition of matter and thus is broad enough to include method patents."7 In short, the World Intellectual Property Organization (WIPO) defines the invention as "a product or a process that provides, in general, a new way of doing something, or offers a new technical solution to a problem."8 Regarding the conditions for patentability, "European patents shall be granted for any inventions, in all fields of technology, provided that they are new, involve an inventive step and are susceptible of industrial application." 9 The usefulness of the invention is a criterion specific to U.S. patent law. According to 35 U.S. Code § 101, "Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title."

In the past five years, metaverse-related patent applications covering hardware and software in a variety of technologies have exploded, indicating that the colonization of the metaverse is well under way. Many of the hardware patents are related to devices used to enter the metaverse, such as VR and AR glasses, while most of the software patents are also related to the user experience. Such observation leads us to distinguish between two types of inventions: (A) inventions developed for the metaverse and (B) inventions developed in the metaverse.

A. INVENTIONS DEVELOPED FOR THE METAVERSE

The metaverse is a digitally-enabled virtual reality environment. It uses AR and VR to enable users to interact with 3D digital objects. It can be explored by users using VR headsets and gloves as well as sensors. The most far-reaching vision of the metaverse is the development of brain-computer interfaces. These devices aim to replace screens and physical hardware. A few models include sensory inputs such as sound, smell, and touch. All of these devices and systems consist in inventions developed for the metaverse. Some of them are patentable, but some of them are not. Thus, the principle is the patentability of the inventions, but this principle is limited by the compliance with public policy.

a. Patentable inventions

In general terms, patents can be granted for either hardware components or for software processes. Generally speaking, if they provide a technical solution to a technical problem, then it is possible to patent them. As regards hardware patents, they are usually sought in relation to devices and gear that are used to enter into the metaverse, like VR or AR glasses, processors and storage mediums. Obtaining software patents for metaverse technologies is comparatively more complex. As to software-related patents, computer programs per se are not patentable, whereas computer-implemented methods are. If a software-based invention can be proven to have a technical application or implementation, then patents can be granted. Much of the software in the metaverse is used for simulation of either, the real world or virtual situations.

For example, algorithms for generating and moving virtual shapes and scenes in a VR environment based on hand gestures are patented, as well as head motion, or line of sight of the user; systems for generating haptic feedback corresponding to users' interaction with virtual objects in a virtual environment; and methods for generating 3D avatars of the users, which emulate users' appearance and behavior, to name a few.

Following Articles 2(c) and 3 of the European Patent Convention, a computer program claimed 'as such' is excluded from patentability. In IBM Case, 10 the Board of Appeals for the European Patent Office (EPO) concluded that "computer programs as such" referred only to those that were non-technical in character. It does not make any difference whether a computer program is claimed by itself or as a record on a carrier. In short, software-based inventions are patentable provided that they have technical character, if they cause a technical effect when run on a computer. 11 In 2014, the U.S. Supreme Court, in its Alice Corp. v. CLS Bank International decision, 12 provided additional considerations for software-related inventions. In particular, the U.S. Supreme Court warned that software inventions directed to mere 'abstract ideas' are not 'patenteligible' (i.e., cannot be patented). What is an 'abstract idea' exactly? Unfortunately, the Supreme Court did not provide a bright-line test, and lower U.S. courts continue to grapple with the definition. The Supreme Court, in Alice, did provide a general test: We know that it is not enough to claim a generic computer with an overbroad idea. In the words of the Supreme Court: "[S]tating an abstract idea while adding the words 'apply it [with a computer]' is not enough for patent eligibility." Since the Alice decision, however, the Federal Circuit, a specialized court that reviews all patent-related appeals in the U.S., has routinely found software-related inventions patentable. For example, post-Alice, the Federal Circuit stated: "We do not read Alice to broadly hold that all improvements in computer-related technology are inherently abstract". 13 A review of Federal Circuit decisions reveals that a best practice to demonstrate patenteligibility is to focus on how a software-related invention provides a 'specific improvement' over the 'prior art' (i.e., existing technology). Such a strategy includes the description of an improvement in the patent specification, the distinction between the improvement and the prior art, and the recital of the improvement in the patent claims. The Federal Circuit has routinely found patents as eligible that use this strategy. 14 Fortunately, because software typically executes on some type of underlying computing device (e.g., such as a smartphone, a laptop, a server, a cloud platform, etc.), an inventor can typically describe how a given software-related invention provides one or more improvements to the underlying computing device. Examples of 'improvements' include the following describing: Increasing the speed or efficiency of the underlying computing device; Reducing the processing requirement or memory usage of the LG Electronics, 880 F.3d 1356 (Fed. Cir. Jan. 25, 2019), analyzing an invention directed to a graphic user interface (GUI) software invention and identifying a pattern for "improvement" (based patents, stating that: "[l]ike the improved systems claimed in Enfish, Thales, Visual Memory, and Finjan, these claims recite a specific improvement over prior systems, resulting in an improved user interface for electronic devices.").

underlying computing device; Increased security; Improved computer network communication and/or operation; Describing a specific improvement over the prior art (existing technology).

Indeed, the difference relies on whether the industrial application requirement is fulfilled, in the sense that if the computer program brings a technical solution to a technical problem, it can be patentable. Patent claims must emphasize technical improvements of the technology, incorporate technical steps that can be argued as non-abstract, and ensure that a specication sets out a problem-solution combination that can be tied to a practical application. When preparing a patent application for a software or a computer-implemented invention, a best practice includes describing, in the patent specification, how the software or computer-implemented invention improves an underlying computing device. And of course, the invention needs to comply with the usual requirements for patent registration: Novelty, inventive step, and industrial application. 15 Many patented inventions concern cryptocurrencies and blockchain, AR and VR headsets, and other technologies prevalent in the metaverse. Here are some examples of patented inventions: Meta has filed patent applications to alter the users' experience in the metaverse, by allowing them to customise the objects and locations that can be found in the metaverse. Other patentable inventions includes an 'avatar personalisation engine' that reproduce the appearance of a real person into an avatar, only by using one picture. Moreover, another patent was filed by the company, consisting of a device that is able to track the users' facial expression and based on the response, will adapt the content displayed. Therefore, the objective pursued by those inventions seems to be designed to allow advertisers to launch more targeted advertisements of their products in the metaverse. Apple, Inc. filed in 2021 a patent application before the United States Patent and Trademark Office (USPTO) related to the metaverse, consisting in a technical solution that will allow to exclude individuals who have violated the interaction rules established. In other words, the technical solution designed by Apple will allow users to block an avatar that insulted or attacked another one, just like on social media platforms. In this way Apple seeks to obtain protection for its technical solution specifically for the metaverse.

A company called Immersion has patented six inventions covering what are called haptic effects, the process of feeling or touching virtual objects. Disney is reported to have obtained a patent in the US for a virtual world simulator.

Back in 2019, Nike obtained a patent for including blockchain technology to their physical sneakers, hence when a consumer would buy a pair of shoes, they would automatically obtain a token that stated the ownership of the product. In this way, the company sought to foster the authenticity of their shoes. Once purchased, the user will be able to virtually enjoy its brandnew Nike's CryptoKicks shoes in Nikeland, the metaverse created by the company.

The issue of patentability of interoperability or compatibility between two or more different metaverse platforms may also be raised. This is not currently the case, and it will be interesting to see who ultimately decides the standards for interoperability: Meta or the marketplace in general. In the meantime, it can already be stated that interoperability and compatibility systems are patentable as they cause a technical effect. A key aspect of interoperability is standardsetting and the use of patents necessary to implement such standards, which can lead to disputes between market players on the determination of FRAND (fair, reasonable and nondiscriminatory) terms for the licensing of standard-essential patents.

Hardware-related and software-related inventions are patentable as long as they are licit, that is, they are not formally excluded from patentability. That would be the exception to patentability.

b. Exceptions to patentability

Some legislations expressly prohibit the granting of patents over certain types of inventions. Following Article 53 of European Patent Convention, European patents shall not be granted in respect of inventions the commercial exploitation of which would be contrary to 'ordre public' or morality; Such an exploitation shall not be deemed to be so contrary merely because it is prohibited by law or regulation in some or all of the Contracting States.

For example, most patent offices would refuse to grant utility patents over inventions which are contrary to human dignity or disregard the dignity of a creature, or in any other way contrary to public order or morality, as well as over methods of cloning human beings, and methods of modifying the genetic identity of the human being.

The Court of Justice of the European Union (CJEU), in its Brüstle decision, excludes an invention from patentability where the technical teaching which is the subject-matter of the patent application requires the prior destruction of human embryos or their use as base material, whatever the stage at which that takes place and even if the description of the technical teaching claimed does not refer to the use of human embryos. 16 Mr Brüstle was the holder of a German patent, which concerns isolated and purified neural precursor cells, processes for their production from embryonic stem cells and the use of neural precursor cells for the treatment of neural defects. It was claimed in the patent specification filed by Mr Brüstle that the transplantation of brain cells into the nervous system is a promising method of treatment of numerous neurological diseases. The first clinical applications have already been developed, in particular for patients suffering from Parkinson's disease. In order to remedy such neural defects, it is necessary to transplant immature precursor cells, still capable of developing. In essence, that type of cell exists only during the brain's development phase. The use of cerebral tissue from human embryos raises significant ethical questions and means that it is not possible to meet the need for the precursor cells which are required to provide publicly available cell treatment. According to the CJEU, Mr Brüstle's invention did not fulfill the legal requirements for patentability, in its last condition: The non-compliance with public order. Public policy infringement could also be blamed to Palmer Luckey, Oculus founder who claims he designed a new VR headset that actually kills you if you die in a game. Famous VR developer Palmer Luckey recently developed a virtual reality headset rigged with bombs such that, if you lose the game, the headset explodes and kills the user. Palmer Luckey said the device was inspired by Sword Art Online, the Japanese novel series-turned-anime in which players are trapped in an online role-playing game where death in the game means death in the real world because of the killer 'NerveGear' headset they wear. "The idea of tying your real life to your virtual avatar has always fascinated meyou instantly raise the stakes to the maximum level and force people to fundamentally rethink how they interact with the virtual world and the players inside it," Luckey wrote in a now-viral blog post. "Only the threat of serious consequences can make a game feel real to you and every other person in the game." Luckey explained that the device is connected to "three explosive charge modules" that are tied to a "narrow-band photosensor that can detect when the screen flashes red at a specific frequency." "When an appropriate game-over screen is displayed, the charges fire, instantly destroying the brain of the user," he said. Way back in 2002, two German designers developed a form of Pong that painfully shocked users when they lost. The console, titled the "Painstation", was, hopefully like Luckey's user-killing VR headset, more of an art project than a real product destined for market.

Luckey's new creepy User-Killing VR Headset invention would never be patentable under jurisdictions that prohibit patentability of inventions contrary to public policy. But in some states, such legal prohibition does not exist. For example, the U.S. Patent Act (35 U.S. Code) does not provide any exception to patentability. The USPTO once rejected 'immoral' inventions. American courts once invalidated patents that seemed morally questionable, such as patents directed to making cheap tobacco look expensive and patents directed to putting imitation seams into stockings to make them look more expensive. That legal doctrine applied by the USPTO and the Courts (that is, the so-called doctrine of 'Moral Utility') is long-since gone, and now virtually all matter of trickery can be included in a patent application. Historically, the idea that 'immoral' inventions were unpatentable was a tenet of U.S. patent law. The rationale for such a rule originated from the idea that immoral inventions were not 'useful' as required by 35 U.S.C. § 101. 17 Controversial inventions are very much patent-eligible today. Juicy Whip, Inc. v. Orange Bang, Inc. 18 killed the doctrine of 'Moral Utility'. The patent at issue in Juicy Whip, was related to a beverage dispenser "with an associated simulated visual display of beverage." 19 Specifically, the invention (the 405 Patent) comprised a display bowl filled with "a permanent sterile and stable fluid simulating the color and texture of a beverage to be dispensed," but actually dispensed beverages from hidden tanks of "pressurized water and concentrate." The claim explained that this display bowl acted as a "powerful merchandising tool for stimulating impulse buying." Defendants Orange Bang, Inc. and Unique Beverage Dispensers, Inc., argued that the patent was invalid because the '405 Patent' described an immoral invention that lacked utility. The Federal Circuit disagreed, finding that "[t]he requirement of 'utility' in patent law is not a directive to the Patent and Trademark Office or the courts to serve as arbiters of deceptive trade practices." After all, the Federal Circuit held that "Congress never intended that the patent laws should displace the police powers of the States, meaning by that term those powers by which the health, good order, peace and general welfare of the community are promoted." In this ruling, the Federal Circuit explicitly declined to follow Rickard [START_REF] Du Bon | d Rickard v[END_REF] and Aristo Hosiery, 21 asserting that those cases did not "represent[] the correct view of the doctrine of utility under the Patent Act of 1952." American authors seem to agree with the overturning. 22 Nowadays, the USPTO's utility requirement is mostly used as a weapon against inventions that are outright inoperative (e.g., an invention that doesn't work at all for any purpose) and/or unbelievable (e.g., a perpetual motion machine). In other words, if Luckey wanted to, he could absolutely file a patent application for his user-killing VR device (though it might not fare so well against existing prior art, especially given how much Luckey admits his ideas came from anime).

After examining inventions developed for the metaverse, we can raise a second kind of patent issue: what would happen with the inventions that are generated in the metaverse? What would happen with inventions that are created by an avatar in the metaverse and that can be used both in the metaverse itself as well as in the real world? Could those inventions be patentable?

B. INVENTIONS DEVELOPED IN THE METAVERSE

It is possible that an avatar evolving in a metaverse platform happens to invent a new virtual technology. The question is: Would such a virtual invention be patentable? What if the invention infringes actual valid patent? It may be affirmed that since such inventions are virtual and not real, they could not be patentable, unless they produce technical effect, thus the infringement would not be punishable.

a. Non patentability

If an avatar user happens to develop an invention in the metaverse, it would be virtual and not real. Since inventions developed in the metaverse would be virtual, would they be protected with an intellectual property right on the grounds of design patent? Following Article 3(a) and (b) of Community Design Regulation (CDR), 23 'design' means the appearance of the whole or a part of a product resulting from the features of, in particular, the lines, contours, colours, shape, texture and/or materials of the product itself and/or its ornamentation; And 'product' means any industrial or handicraft item, including inter alia parts intended to be assembled into a complex product, packaging, get-up, graphic symbols and typographic typefaces, but excluding computer programs. The CDR should be interpreted as meaning that a patentable design must be a physical shape in 3D form.

In fact, metaverse design patent cases should be identified as 'computer program products' in the specification so as to be distinguishable from physical product design patent cases. 'Design' means a new design of the shape, pattern, or a combination thereof, as well as a combination of the color, shape and pattern, of the entirety or a portion of a 'product', which creates an aesthetic feeling and is fit for industrial application. Consequently, the first thing to consider is whether 21 Scott Williams, Inc. v. Aristo Hosiery Co., 7 F.2d 1003 (2d Cir. 1925). 22 Kirk A. Sigmon, "Pay to Play: Video Game Monetization Patents and the Doctrine of Moral Utility", in Georgetown Law Technology Review, 72 (2021): "After all, granting a patent covering a useless invention is unlikely to be of much harm to society, whereas denying a patent to an ultimately useful invention might significantly harm the inventors' ability to realize the value of their invention". 23 the 'products' in the virtual world of the metaverse belong to the design products defined by design patent laws. Virtual products in the metaverse do not seem to be considered a design product as defined by design patent laws. This is because neither are these virtual products a kind of industrial product after all, nor are they designs suitable for industrial applications even if they can bring users good experiences in the metaverse. Product designs in the metaverse, on the other hand, are not considered to be able to achieve mass production in the conventional sense. Thus, Nike's CryptoKicks would not be patentable, despite the meeting of the novelty criterion.

b. Patentability of inventions producing technical effect

Dwiss, a Swiss watch microbrand, offered for sale NFTs allowing their holders to design their own and unique virtual watches. What if the company could offer to conceive a physical version of the NFT version of the watch? It is likely to consider that only the physical aspect of watches would be patentable under design patent law. Of course, if the virtual watch involves a new virtual inventive step, only the physical version would be patentable under utility patent law. However, inventors like Palmer Luckey may have the brilliant idea to develop a software that can cause technical effect sensed by users in their real life. It has been demonstrated that such software-related inventions are patentable. So far, virtual reality does not entail any physical consequence in real life. But what if the invention developed in the metaverse produces virtual technical effect? Indeed, some softwares and websites enable users to create virtual artifacts, tools and weapons that can be imported in a MMORPG such as Fortnite. Like in the movie Ready Player One,24 users can invent artifacts giving their avatars new super powers making them gain a competitive edge in the game. Would a new virtual pair of shoes enhancing the avatar's strength, a super powered virtual grenade, an orb, a virtual magical key, a new weapon that has never been designed, be patentable? It has been demonstrated that a software-based invention allowing users to block an avatar in a game is patentable because the result is legally considered as a technical effect. An analogical reasoning, permits to patent virtual tools producing technical effects.

Here is a more realistic example: Kerbal Space Program is a space flight simulation video game developed by Mexican studio Squad for Linux, OS X, Windows, PlayStation 4, PlayStation 5, Xbox Series X/S and Xbox One. In the game, players direct a nascent space program, staffed and crewed by green humanoid aliens known as 'Kerbals'. The game features a realistic orbital physics engine, allowing for various real-life orbital maneuvers such as Hohmann transfer orbits and orbital rendezvous. From the space center, players can build various vehicles such as rockets, aircrafts, spaceplanes, and rovers from a provided set of components. Constructed craft can be launched from the space center's launch pad or runway to accomplish various tasks while avoiding partial or catastrophic failure, such as lack of fuel or structural failure. Players control flight with little assistance to keep their rocket oriented. Provided it maintains sufficient thrust and fuel, a virtual spacecraft can enter orbit or even travel to other virtual celestial bodies. Missions involve goals such as reaching a certain altitude, escaping the atmosphere, reaching a stable orbit, landing on a certain planetary body, rescuing stranded astronauts, capturing asteroids, and creating space stations and surface bases. Players may also set challenges for each other on the game's forums.

Since the creation of virtual rockets, aircrafts, spaceplanes, and rovers must meet all the reallife physical and atmospheric conditions, the issue of their patentability is of crucial importance. What if an aircraft manufacturer actually builds a physical engine which is the exact replica of a virtual engine designed by a game player in Kerbal Space Program? Non-patentability of the virtual version of the invention would be unfair if the manufacturer is entitled to patent the physical version, unless the latter does not meet the novelty criterion on prior art grounds. Then the manufacturer could not be granted the patent. Thus the legislator should allow the patentability of virtual inventions which produce technical effect, even if the result remains in the metaverse.

As a consequence, virtual inventions developed in the metaverse could not be covered either with a utility patent nor with a design patent, unless they produce technical effects in real life or in the metaverse. What about virtual patent infringement?

c. Virtual patent infringement

Consider a virtual headset created by an avatar in a metaverse, which would be similar to a patented headset. From the perspective of a user in the virtual world, the virtual headset is operating in exactly the same way as the apparatus claim, with the exception of being in a virtual environment. However, the virtual headset does not actually exist. Seen from the real world, the virtual headset is a series of ones and zeros representing a headset in memory. Should courts apply the U.S. doctrine of equivalents? Under the doctrine of equivalents, "a product or process that does not literally infringe upon the express terms of a patent claim may nonetheless be found to infringe if there is 'equivalence' between the elements of the accused product or process and the claimed elements of the patented invention". 25 The doctrine of equivalents has deep roots in U.S. patent law. Infringement by equivalence "requires a showing that the difference between the claimed invention and the accused product [is] insubstantial." 26 One way of proving insubstantial difference is the 'Function-Way-Result' test : "Showing on a limitation by limitation basis that the accused product performs substantially the same function in substantially the same way with substantially the same result as each claim limitation of the patented product." The difficult question is whether each operates in an equivalent way. Perkin-Elmer Corp. v. Westinghouse Elec. Corp. stated that "a claimed invention and an accused device may perform substantially the same function and may achieve the same result will not make the latter an infringement under the doctrine of equivalents where [the accused device] performs the function and achieves the result in a substantially different way."27 Thus, it can be assumed that what we call virtual patent infringement could not be punishable, as the argued copy would simply consists in an image of the patented invention.

CONCLUSION

Only inventions developed for the metaverse are patentable, on the grounds of both utility and design patents, whereas inventions developed in the metaverse would hardly be patentable, since they are only virtual, and as long as they do not cause technical effect in real life. For the same reason, it would be difficult to find a person guilty of patent infringement for having reproduced a virtual copy of a patented invention.

Meanwhile, major international companies such as Meta, Microsoft and NVIDIA have recently formed the Metaverse Standards Forum, aiming to standardize the metaverse industry. It is believed that countries or regions will gradually open up or further relax the conditions for metaverse design patent applications in response to this trend in the foreseeable future.

  Council Regulation (EC) No 6/2002, 12 December 2001 on Community designs (OJ EC No L 3 of 5.1.2002, p. 1).

Michel Vivant, "La privatisation de l'information par la propriété intellectuelle", in Revue internationale de droit économique 2006/4 (t. XX, 4), p. 373.

Cardiac Pacemakers, Inc. v. St. Jude Med., Inc., 576 F.3d 1348, 1362 (Fed. Cir. 2009).

https://www.wipo.int/patents/en/

European Patent Convention, Art. 52 (1).

Article 52(1) of the European Patent Convention: "European patents shall be granted for any inventions, in all fields of technology, provided that they are new, involve an inventive step and are susceptible of industrial application".

Case C-34/10, Oliver Brüstle v. Greenpeace e.V., Judgment of the Court (Grand Chamber) of 18 October 2011.

Ready Player One, directed by Steven Spielberg, Warner Bros. Pictures 2018.

Warner-Jenkinson Co., Inc. v. Hilton Davis Chemical Co., 520 U.S.17, 21, 29 (1997).

Crown Packaging Technology, Inc. v. Rexam Beverage Can Co., 559 F.3d 1308, 1312 (Fed. Cir. 2009).

Perkin-Elmer Corp. v. Westinghouse Elec. Corp., 822 F.2d 1528, 1531 n.6 (Fed. Cir. 1987).


Crosstalk between PKR1 and miR-124 is crucial for EMT regulation of tc21+epicardial mesothelial cells during cardiac development and repair process

Abstract Background

The epicardial-mesothelial cells (EMCs) are essential regulators of cardiac growth and repairment process. The epicardial-to-mesenchymal transition (EMT) of epicardial fate determination are controlled by both cell autonomous and cardiomyocyte-originated mechanisms. Here, by using an in vitro and in vivo model of epicardial EMT, we investigated the role of miRNAs as regulators of these process and their potential targets.

Methods

EMT was induced in mice embryonic tcf21+EMCs through an angiogenic cytokine, prokineticin treatments. Cre-dependent tracing of TCF21-tm-iCRE-EMCs was utilized to abrogate prokineticin receptor-1 (PKR1) in mice epicardium. Human tcf21+ cardiac fibroblast (mainly originated from epicardial origin also utilized to study for their repairment signaling.

Results

Upon EMC-specific abrogation of PKR1 in early stages of cardiac development the TCF21-tm-iCREPKR1-/-mice

Oxford University Press uses cookies to enhance your experience on our website. By selecting 'acce you are agreeing to our use of cookies. You can change your cookie settings at any time. More infor can be found in our Cookie Policy.

Conclusion

This study shows the importance of a crosstalk between PKR1

Funding Acknowledgement

Type of funding sources: Foundation. Main funding source(s): Fondation de France

Oxford University Press uses cookies to enhance your experience on our website. By selecting 'accept all' you are agreeing to our use of cookies. You can change your cookie settings at any time. More information can be found in our Cookie Policy.

  exhibited 13±4% embryonic lethality due to a disconnection of epicardial cells from compact layer, failed expansion of the sub-epicardial space, and disruption of heterotypic cell interaction between epicardium and myocardium. EMT-RT profiler and enrichment analysis revealed an impaired EMT in TCF21-tm-iCREPKR1-/-hearts. Ingenuity Pathway Analysis on the EMT molecular network (e.g., Snai1, Snai2, B-catenin, N-cadherin, vimentin), specific to cardiac development and defects revealed MiR-124 as a part of this network that was inversely associated and down-stream of PKR1 in cultured Tcf21+ cells. Furthermore, protein expression of SNAIL2 was significantly downregulated in TCF21-tm-iCREPKR1-/-hearts and cultured Tcf21+ cells upon treatment with mir124 mimic. The luciferase reporter assay showed that miR124 directly targeted the 3'untranslated region of SNAIL2 in EMCs. In a counter experiment, PKR1 gene replacement or mir124 inhibitor was able to rescue the impaired EMT-genes, and an increase in apoptosis and impaired proliferation in epicardium of embryonic TCF21-tm-iCREPKR1-/-hearts. Importantly, miRNA mimic in cultured tcf21 cells maintained epithelial features in EMCs, and increased EMT-associated transcripts and traits, such as apoptosis. In contrast, MiR124 is involved in wound healing process in tcf21+human CFs and EMCs.

04314536 Crosstalk between PKR1 and miR-124 is crucial for EMT regulation of tc21+epicardial mesothelial cells during cardiac development and repair process 

	, ,
	European Heart Journal, Volume 43, Issue Supplement_2, October
	2022, ehac544.3003, https://doi.org/10.1093/eurheartj
	/ehac544.3003
	Published: 03 October 2022

H Arora, A Mammadova, Canan G. Nebigil To cite this version: H Arora, A Mammadova, Canan G. Nebigil. Crosstalk between PKR1 and miR-124 is crucial for EMT regulation of tc21+epicardial mesothelial cells during cardiac development and repair process. European Society of Cardiology annuel meeting, Aug 2022, Barcelona, Spain. 10.1093/eurheartj/ehac544.3003. hal-


CONTROLLING GENERATIVE MODELS WITH CONTINU-OUS FACTORS OF VARIATIONS

Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent works have shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like position or scale of the object in the image. Our method is weakly supervised and particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

, showing that the position of the object can be controlled within the image.

INTRODUCTION

With the success of recent generative models to produce high-resolution photo-realistic images [START_REF] Karras | A Style-Based Generator Architecture for Generative Adversarial Networks[END_REF][START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF][START_REF] Razavi | Generating Diverse High-Fidelity Images with VQ-VAE-2[END_REF], an increasing number of applications are emerging such as image in-painting, dataset-synthesis, deep-fakes... However, the use of generative models is often limited by the lack of control over the generated images. Indeed, more control could, for instance, be used to improve existing approaches which aim at generating new training examples [START_REF] Bowles | GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks[END_REF] by allowing the user to choose more specific properties of the generated images.

First attempts in this direction showed that one can modify an attribute of a generated image by adding a learned vector on its latent code [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF] or by combining the latent code of two images [START_REF] Karras | A Style-Based Generator Architecture for Generative Adversarial Networks[END_REF]. Moreover, the study of the latent space of generative models provides insights about its structure which is of particular interest as generative models are also powerful tools to learn unsupervised data representations. For example, [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF] observed on auto-encoders trained on datasets with labels for some factors of variations, that their latent spaces exhibit a vector space structure where some directions encode the said factors of variations.

We suppose that images result from underlying factors of variation such as the presence of objects, their relative positions or the lighting of the scene. We distinguish two categories of factors of variations. Modal factors of variation are discrete values that correspond to isolated clusters in the data distribution, such as the category of the generated object. On the other hand, the size of an object or its position are described by Continuous factors of variations, expressed into a range of possible values. As humans, we naturally describe images by using factors of variations suggesting that they are an efficient representation of natural images. For example, to describe a scene, one likely enumerates the objects seen, their relative positions and relations and their characteristics [START_REF] Berg | Understanding and predicting importance in images[END_REF]. This way of characterising images is also described in [START_REF] Krishna | Visual genome: Connecting language and vision using crowdsourced dense image annotations[END_REF]. Thus, explaining the latent space of generative models through the lens of factors of variation is promising. However, the control over the image generation is often limited to discrete factors and requires both labels and an encoder model. Moreover, for continuous factors of variations described by a real parameter t, previous works do not provide a way to get precise control over t.

In this paper, we propose a method to find meaningful directions in the latent space of generative models that can be used to control precisely specific continuous factors of variations while the literature has mainly tackled semantic labeled attributes like gender, emotion or object category [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF][START_REF] Odena | Conditional Image Synthesis With Auxiliary Classifier GANs[END_REF]. We test our method on image generative models for three factors of variation of an object in an image: vertical position, horizontal position and scale. Our method has the advantage of not requiring a labeled dataset nor a model with an encoder. It could be adapted to other factors of variations such as rotations, change of brightness, contrast, color or more sophisticated transformations like local deformations. However, we focused on position and scale as these are quantities that can be evaluated, allowing us to measure quantitatively the effectiveness of our method. We demonstrate both qualitatively and quantitatively that such directions can be used to control precisely the generative process and show that our method can reveal interesting insights about the structure of the latent space. Our main contributions are:

• We propose a new method to find interpretable directions in the latent space of generative models, which correspond to parametrizable continuous factors of variations of the generated image.

• We show that properties of generated images can be controlled precisely by sampling latent representations along linear directions.

• We propose a novel reconstruction loss for inverting generative models with gradient descent.

• We give insights of why inverting generative models with optimization can be difficult by reasoning about the geometry of the natural image manifold.

• We study the impacts of disentanglement on the controllability of generative models.

LATENT SPACE DIRECTIONS OF A FACTOR OF VARIATION

We argue that it is easier to modify a property of an image than to obtain a label describing that property. For example, it is easier to translate an image than to determine the position of an object within said image. Hence, if we can determine the latent code of a transformed image, we can compute its difference with the latent code of the original image to find the direction in the latent space which corresponds to this specific transformation as in [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF].

Let us consider a generative model G : z ∈ Z → I, with Z its latent space of dimension d and I the space of images, and a transformations T t : I → I characterized by a continuous parameter t.

For example if T is a rotation, then t could be the angle, and if T is a translation, then t could be a component of the vector of the translation in an arbitrary frame of reference. Let z 0 be a vector of Z and I = G(z 0 ) a generated image. Given a transformation T T , we aim at finding z T such that G(z T ) ≈ T T (I) to then use the difference between z 0 and z T in order to estimate the direction encoding the factor of variation described by T .

LATENT SPACE TRAJECTORIES OF AN IMAGE TRANSFORMATION

Given an image I ∈ I, we want to determine its latent code. When no encoder is available we can search an approximate latent code ẑ that minimises a reconstruction error L between I and Î = G( ẑ) ( Î can be seen as the projection of I on G(Z)) i.e.

L(I, G(z)) (2) 2.1.1 CHOICE OF THE RECONSTRUCTION ERROR L
One of the important choice regarding this optimization problem is that of L. In the literature, the most commonly used are the pixel-wise Mean Squared Error (MSE) and the pixel-wise cross-entropy as in [START_REF] Lipton | Precise Recovery of Latent Vectors from Generative Adversarial Networks[END_REF] and [START_REF] Creswell | Inverting The Generator Of A Generative Adversarial Network[END_REF]. However in practice, pixel-wise losses are known to produce blurry images. To address this issue, other works have proposed alternative reconstruction errors. However, they are based on an alternative neural network [START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF][START_REF] Johnson | Perceptual Losses for Real-Time Style Transfer and Super-Resolution[END_REF] making them computationally expensive.

The explanation usually given for the poor performance of pixel-wise mean square error is that it favors the solution which is the expected value of all the possibilities [START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF] 2 . We propose to go deeper into this explanation by studying the effect of the MSE on images in the frequency domain.

In particular, our hypothesis is that due to its limited capacity and the low dimension of its latent space, the generator can not produce arbitrary texture patterns as the manifold of textures is very high dimensional. This uncertainty over texture configurations explains why textures are reconstructed as uniform regions when using pixel-wise errors. In Appendix A, by expressing the MSE in the Fourier domain and assuming that the phase of high frequencies cannot be encoded in the latent space, we show that the contribution of high frequencies in such a loss is proportional to their square magnitude pushing the optimisation to solutions with less high frequencies, that is to say more blurry. In order to get sharper results we therefore propose to reduce the weight of high frequencies into the penalisation of errors with the following loss:

L(I 1 , I 2 ) = ||F{I 1 -I 2 }F{σ}|| 2 = ||(I 1 -I 2 ) * σ|| 2 (3)
where F is the Fourier transform, * is the convolution operator and σ is a Gaussian kernel. With a reduced importance given to the high frequencies to determine ẑ when one uses this loss in equation 2, it allows to benefit from a larger range of possibilities for G(z), including images with more details (i.e with more high frequencies) and appropriate texture to get more realistic generated images. A qualitative comparison of some reconstruction errors and choices of σ can be found in Appendix C.

Algorithm 1: Create a dataset of trajectories in the latent space which corresponds to a transformation T in the pixel space. The transformation is parametrized by a parameter δt which controls a degree of transformation. We typically use N = 10 with (δt n ) (0≤n≤N ) distributed regularly on the interval [0, T ]. Input: number of trajectories S, generator G, transformation function T , trajectories length N , threshold Θ.

Result: dataset of trajectories D for i ∈ 1, S do D ← {} ; z 0 ∼ N (0, I) ; I 0 ← G(z 0 ) ; z δt ← z 0 ; for n ∈ [1, N ] do z t ← arg min z L(G(z), T δtn (I 0 )) ; if L(G(z), T δtn (I 0 )) < Θ then D ← D ∪ {(z 0 , z δt , δt n )} ; end end end

RECURSIVE ESTIMATION OF THE TRAJECTORY

Using equation 2, our problem of finding z T such that G(z T ) ≈ T T (I), given transformation T T , can be solve through the following optimisation problem:

z T = arg min z∈Z,||z||≤ √ d L(G(z), T T (I)) (4)
In practice, this problem is difficult and an "unlucky" initialization can lead to a very slow convergence. [START_REF] Zhu | Generative Visual Manipulation on the Natural Image Manifold[END_REF] proposed to use an auxiliary network to estimate z T and use it as initialization.

Training a specific network to initialize this problem is nevertheless costly. One can easily observe that a linear combination of natural images is usually not a natural image itself, this fact highlights the highly curved nature of the manifold of natural images in pixel space. In practice, the trajectories corresponding to most transforms in pixel space may imply small gradients of the loss that slowdown the convergence of problem of Eq. ( 2) (see Appendix E). To address this, we guide the optimization on the manifold by decomposing the transformation T T into smaller transformations [T δt0 , . . . , T δt N ] such that T δt0=0 = Id and δt N = T and solve sequentially:

z n = arg min z∈Z L (G (z; z init = z n-1 ) , T δtn (G (z 0 ))) for n = 1, . . . , N (5)
each time initializing z with the result of the previous optimization. In comparison to [START_REF] Zhu | Generative Visual Manipulation on the Natural Image Manifold[END_REF], our approach does not require extra training and can thus be used directly without training a new model. We compare qualitatively our method to a naive optimisation in Appendix C.

A transformation on an image usually leads to undefined regions in the new image (for instance, for a translation to the right, the left hand side is undefined). This is why we designed L to ignore the value of the undefined regions of the image. Another difficulty is that often the generative model cannot produce arbitrary images. For example a generative model trained on a given dataset is not expected to be able to produce images where the object shape position is outside of the distribution of object shape positions in the dataset. This is an issue when applying our method because as we generate images from a random start point, we have no guarantee that the transformed images is still on the data manifold. To reduce the impact of such outliers, we discard latent codes that give a reconstruction error above a threshold in the generated trajectories. In practice, we remove one tenth of the latent codes which leads to the worst reconstruction errors. It finally results into Algorithm 1 to generate trajectories in the latent space.

ENCODING MODEL OF THE FACTOR OF VARIATION IN THE LATENT SPACE.

After generating trajectories with Algorithm 1, we search regularities in these trajectories to find a direction u ∈ Z which encodes a the parameter t of specific factor of variation in the latent space. Assuming a model f : Z → R that describes how factors of variations are encoded in the latent space, we pose t = f (z) = g( z, u ), with g : R → R and •, • the euclidean scalar product in R d , to formulate the hypothesis that a factor of variation is encoded in a direction u in the latent space.

For instance, if we consider the factor corresponding to the horizontal position of an object x in an image, and the dSprite dataset, we have x that follows a uniform distribution U(-0.5, 0.5) while the projection of z onto an axis u follows a normal distribution N (0, 1). Thus, it is natural to choose a model of the form:

x = f (z) = g( z, u ) = u,z 0 N (t; 0, 1)dt = 1 2 erf u, z √ 2 with ||u|| = 1 (6)
In general, the distribution of the parameter t of a factor of variation is not known. A simple approach is to suppose that t follows a centered Gaussian distribution (which results in a linear model as z is sampled from a centered Gaussian too). It is also possible to use a model which does not rely on such prior by learning g:

t = f (θ,u) (z) = g θ ( u, z ) with ||u|| = 1 (7)
with g θ : R → R and (θ, u) trainable parameters of the model. We typically used piece-wise linear functions for g θ . However, this model cannot be trained directly as we do not have access to t (in the case of horizontal translation the x-coordinate for example) but only to the difference δt = t G(z1) -t G(z0) between an image G(z 0 ) and its transformation G(z 1 ) (δx or δy in the case of translation). We solve this issue by reformulating the problem as:

δt = f (θ,u) (z 1 ) -f (θ,u) (z 0 ) with ||u|| = 1 and g θ (0) = 0 (8)
Hence, u and θ are estimated by training f (θ,u) with gradient descent on a dataset produced by Algorithm 1 for a given transformation.

An interesting application of this method is the estimation of the distribution of the images generated by G. Indeed, when z is normally distributed, then u, z ∼ N (0, 1). With our model, t = g θ ( u, z ) thus, when g θ is an monotonic differentiable function, the distribution ϕ : R → R + of t can be estimated by:

ϕ(t) = N (g -1 θ (t); 0, 1) d dt g -1 θ (t) (9)
With the knowledge of g θ we can also choose how to sample images. For instance, let say that we want to have t ∼ φ(t), with φ : R → R + an arbitrary distribution, we can simply transform z ∼ N (0, 1) as follows:

z ← z -z, u u + (h φ • ψ)( z, u )u (10)
with h φ : [0, 1] → R and ψ such that:

ψ(x) = x -∞ N (t; 0, 1)dt ; h -1 φ (x) = x -∞ φ(g θ (t)) d dt g θ (t) dt (11)
These results are interesting to bring control not only on a single output of a generative model but also on the distribution of its outputs. Moreover, since generative models reflect the datasets on which they have been trained, the knowledge of these distributions could be applied to the training dataset to reveal potential bias.

EXPERIMENTS

Datasets: We performed experiments on two datasets. The first one is dSprites [START_REF] Matthey | dsprites: Disentanglement testing sprites dataset[END_REF], composed of 737280 binary 64 × 64 images containing a white shape on a dark background. Shapes can vary in position, scale and orientations making it ideal to study disentanglement. The second dataset is ILSVRC [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF], containing 1.2M natural images from one thousand different categories.

Implementation details: All our experiments have been implemented with TensorFlow 2.0 [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF] and the corresponding code is available at https://anonymised.for.review.

We used a BigGAN model [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF] whose weights are taken from TensorFlow-Hub allowing easy reproduction of our results. The BigGAN model takes two vectors as inputs: a latent vector z ∈ R 128 and a one-hot vector to condition the model to generate images from one category.

The latent vector z is then splited into six parts which are the inputs at different scale levels in the generator. The first part is injected at the bottom layer while next parts are used to modify the style of the generated image thanks to AdaIN layers [START_REF] Huang | Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization[END_REF]. We also trained several β-VAEs [START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF] to study the importance of disentanglement in the process of controlling generation. The exact β-VAE architecture used is given in Appendix B. The models were trained on dSprites [START_REF] Matthey | dsprites: Disentanglement testing sprites dataset[END_REF] with an Adam optimizer during 1e5 steps with a batch size of 128 images and a learning rate of 5e-4.

QUANTITATIVE EVALUATION METHOD

Evaluating quantitatively the effectiveness of our method on complex datasets is intrinsically difficult as it is not always trivial to measure a factor of variation directly. We focused our analysis on two factors of variations: position and scale. On simple datasets such as dSprites, the position of the object can be estimated effectively by computing the barycenter of white pixels. However, for natural images sampled with the BigGAN model, we have to use first saliency detection on the generated image to produce a binary image from which we can extract the barycenter. For saliency detection, we used the model provided by [START_REF] Hou | Deeply supervised salient object detection with short connections[END_REF] which is implemented in the PyTorch framework [START_REF] Paszke | Automatic differentiation in pytorch[END_REF]. The scale is evaluated by the proportion of salient pixels. The evaluation procedure is:

1. Get the direction u which should describe the chosen factor of variation with our method.

2. Sample latent codes z from a standard normal distribution.

3. Generate images with latent code zz, u u + tu with t ∈ [-T, T ].

4. Estimate the real value of the factor of variation for all the generated images.

5. Measure the standard deviation of this value with respect to t.

Jahanian et al. ( 2019) proposed an alternative method for quantitative evaluation that relies on an object detector. Similarly to us, it allows an evaluation for x and y shift as well as scale but is restricted to image categories that can be recognized by a detector trained on some categories of ILSVRC. We thus gain in genericity with our approach.

RESULTS ON BIGGAN

We performed quantitative analysis on ten chosen categories for which the object can be easily segmented by using saliency detection approach. Results are presented in Figure 2 (top). We observe that for the chosen categories of ILSVRC, we can control the position and scale of the object relatively precisely by moving along directions of the latent space found by our method.

However, one can still wonder whether the directions found are independent of the category of interest.

To answer this question, we merged all the datasets of trajectories into one and learned a common direction on the resulting datasets. Results for the ten test categories are shown in Figure 2 (bottom). This figure shows that the directions which correspond to some factors of variations are indeed shared between all the categories. Qualitative results are also presented in Figure 3 for illustrative purposes. We also checked which parts of the latent code are used to encode position and scale. Indeed, BigGAN uses hierarchical latent code which means that the latent code is split into six parts which are injected at different level of the generator. We wanted to see by which part of the latent code these directions are encoded. The squared norm of each part of the latent code is reported in Figure 4 for horizontal position, vertical position and scale. This figure shows that the directions corresponding to spatial factors of variations are mainly encoded in the first part of the latent code. However, for the y position, the contribution of level 5 is higher than for the x position and the scale. We suspect that it is due to correlations between the vertical position of the object in the image and its background that we introduced by transforming the objects because the background is not invariant by vertical translation because of the horizon.

THE IMPORTANCE OF DISENTANGLED REPRESENTATIONS

To test the effect of disentanglement on the performance of our method, we trained several β-VAE [START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF] on dSprites [START_REF] Matthey | dsprites: Disentanglement testing sprites dataset[END_REF], with different β values. Indeed, β-VAE are known for having more disentangled latent spaces as the regularization parameter β increases.

Results can be seen in Figure 5. The figure shows that it is possible to control the position of the object on the image by moving in the latent space along the direction found with our method. As expected, the effectiveness of the method depends on the degree of disentanglement of the latent space since the results are better with a larger β. Indeed we can see on Figure 5 that as β increases, the standard deviation decreases (red curve), allowing a more precise control of the position of the generated images. This observation motivates further the interest of disentangled representations for control on the generative process.

RELATED WORKS

Our work aims at finding interpretable directions in the latent space of generative models to control their generative process. We distinguish two families of generative models: GAN-like models which do not provide an explicit way to get the latent representation of an image and auto-encoders which provide an encoder to get the latent representation of images. From an architectural point of view, conditional GANs [START_REF] Odena | Conditional Image Synthesis With Auxiliary Classifier GANs[END_REF] and InfoGan [START_REF] Chen | InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets[END_REF] allow the user to choose the category of a generated object or some chosen properties of the generated image but these approaches require a labeled dataset and use models which are explicitly designed to allow control. Our method has the advantage of being model agnostic and of not directly requiring labels. A more recent attempt at controlling generative models is described in [START_REF] Bau | GAN Dissection: Visualizing and Understanding Generative Adversarial Networks[END_REF] in which the authors analyze the activations of the network's neurons to know which ones are causing the presence of specific objects in the generated image. With their method, they managed to control the presence of objects in the generated image. On the other side, our work focuses on the latent space and not on the intermediate activations inside the generator.

One of our contribution and a part of our global method is a procedure to find the latent representation of an image when an encoder is not available. Several previous works have studied how to invert the generator of a GAN to find the latent code of an image. [START_REF] Creswell | Inverting The Generator Of A Generative Adversarial Network[END_REF] showed on simple datasets (MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] and Omniglot (Lake et al., 2015)) that this inversion process can be achieved by optimising the latent code to minimize the reconstruction error between the generated image and the target image. [START_REF] Lipton | Precise Recovery of Latent Vectors from Generative Adversarial Networks[END_REF] introduced tricks to improve the results on a more challenging dataset (CelebA [START_REF] Liu | Deep learning face attributes in the wild[END_REF]). However we observed that these methods fail when applied on a more complex datasets (ILSVRC [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF]). In this paper we introduce a new reconstruction loss adapted to this particular problem which improves the quality of reconstructions significantly. We also give a hint about why the problem of inverting a generative model can be difficult compared to other optimization problems.

The closest works were released on ArXiv very recently [START_REF] Goetschalckx | GANalyze: Toward Visual Definitions of Cognitive Image Properties[END_REF][START_REF] Jahanian | On the "steerability" of generative adversarial networks[END_REF] indicating that finding interpretable directions in the latent space of generative models to control their output is of high interest for the community. In these papers, the authors describe a method to find interpretable directions in the latent space of the BigGAN model [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF]. If their method exhibits similarities with ours (use of transformation, linear trajectories in the latent space), it also differs on several points. From a technical point of view our training procedure differs in the sense that we first generate a dataset of interesting trajectories to then train our model while they train their model directly. Our evaluation procedure is also more general as we use a saliency model instead of a MobileNet-SSD v1 [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] trained on specific categories of the ILSVRC dataset allowing us to measure performance on more categories. We provide additional insight on how auto-encoders can also be controlled with the method, the impact of disentangled representations on the control and on the structure of the latent space of BigGAN. Moreover we also propose an alternative reconstruction error to invert generators. However, the main difference we identify between the two works is the model of the latent space used. Our model allows a more precise control over the generative process and can be being adapted to more cases.

CONCLUSIONS

Generative models are increasingly more powerful but suffer from little control over the generative process and the lack of interpretability in their latent representations. In this context, we propose a method to extract meaningful directions in the latent space of such models and use them to control precisely some properties of the generated images. We show that a linear subspace of the latent space of BigGAN can be interpreted in term of intuitive factors of variation (namely translation and scale).

It is an important step toward the understanding of the representations learned by generative models.

A PENALTY ON THE AMPLITUDE OF FREQUENCIES DUE TO MSE

In Section 2.1, we consider a target image I ∈ I and a generated image Î = G(ẑ) to be determined according to a reconstruction loss L (Equation 1). Let us note F{•} the Fourier transform. If L is the usual MSE, from the Plancherel theorem, we have || Î -I|| 2 = ||F{ Î} -F{I}|| 2 . Let us consider a particular frequency ω in the Fourier space and compute its contribution to the loss. The Fourier transform of I (resp. Î) having a magnitude r (resp. r) and a phase θ (resp. θ) at ω, we have:

|F{ Î}(ω) -F {I}(ω)| 2 = |re i θ -re iθ | 2 = (rcos( θ) -rcos(θ)) 2 + (rsin( θ) -rsin(θ)) 2 = r2 + r 2 -2rr cos( θ)cos(θ) + sin( θ)sin(θ) = r2 + r 2 -2rr cos( θ)cos(θ) + sin( θ)sin(θ) (12)
If we model the disability of the generator to model every high frequency patterns as an uncertainty on the phase of high frequency of the generated image, i.e by posing θ ∼ U([0, 2π]), the expected value of the high frequency contributions to the loss is equal to:

E |F{ Î}(ω) -F {I}(ω)| 2 = r2 + r 2 -2rr     E cos( θ) =0 cos(θ) + E sin( θ) =0 sin(θ)     = r2 + r 2 (13)
The term r 2 is a constant w.r.t the optimization of L and can thus be ignored. The contribution to the total loss L thus directly depends on r2 . While minimizing L, the optimization process tends to favour images Î = G(ẑ) with smaller magnitudes in the high frequencies, that is to say smoother images, with less high frequencies.

B β-VAE The curvature of the natural image manifold makes the optimisation problem of Equation 2 difficult to solve. This is especially true for factors of variation which correspond to curved walks in pixel-space (for example translation or rotation by opposition to brightness or contrast changes which are linear).

To illustrate this fact, we show that the trajectory described by an image undergoing common transformations is curved in pixel space. We consider three types of transformations, namely translation, rotation and scaling, and get images from the dSprites [START_REF] Matthey | dsprites: Disentanglement testing sprites dataset[END_REF] dataset which correspond to the progressive transformation (interpolation) of an image. To visualize, we compute the PCA of the resulting trajectories and plot the trajectories on the two main axes of the PCA. The result of this experiment can be seen in Figure 9. In this figure, we can see that for large translations, the direction of the shortest path between two images in pixel-space is near orthogonal to the manifold. The same problem occurs for rotation and, at a smaller extent, for scale. However this problem does not exist for brightness for example, as its change is a linear transformation in pixel-space. This is problematic during optimization of the latent code because the gradient of the reconstruction loss with respect to the generated image is tangent to this direction. Thus, when we are in the case of near orthogonality, the gradient of the error with respect to the latent code is small. Indeed, let us consider an ideal case where G is a bijection between Z and the manifold of natural images. Let be z ∈ Z, a basis of vectors tangent to the manifold at point G(z) is given by ∂G(z) ∂z1 , ..., ∂G(z) It shows that when the direction of descent in pixel space is near orthogonal to the manifold described by the generative model, optimization gets slowed down and can stop if the gradient of the loss with respect to the generated image is orthogonal to the manifold.

For example, let assume we have an ideal GAN which generates a small white circle on a black background, with a latent space of dimension 2 that encodes the position of the circle. Let consider a generated image with the circle on the left of the image and we want to move it to the right. Obviously, we thus have ∇ z ||G(z) -T T (G(z 1 ))|| 2 = 0 if the intersection of the two circles is empty (see Figure 9) since a small translation of the object does not change the reconstruction error.

F ADDITIONAL QUALITATIVE EXAMPLES

We show qualitative examples for images generated with the BigGAN model for position, scale and brightness. The images latent codes are sampled in the following way: zz, u u + αu with α ∈ [-3, 3] and u the learned direction. We have chosen the categories to produce interesting results: for position and scale categories are objects, for brightness categories are likely to be seen in a bright or dark environment. Notice that for some of the chosen categories, we failed to control the brightness of the image. It is likely due to the absence of dark images for these categories in the training data. for position and scale, the direction is learned on the ten categories presented here while for brightness only the five top categories are used.

Figure 2 :

 2 Figure 2: Quantitative results on the ten categories of the ILSVRC dataset used for training (Top) and for ten other categories used for validation (Bottom) for three geometric transformations: horizontal and vertical translations and scaling. In blue, the distribution of the measured transformation parameter and in red the standard deviation of the distribution with respect to t. Note that for large scales the algorithm seems to fail. However, this phenomenon is very likely due to the poor performance of the saliency model when the object of interest covers almost the entire image (scale ≈ 1.0). (best seen with zoom)

Figure 3 :

 3 Figure 3: Qualitative results for some categories of ILSVRC dataset for three geometric transformations: horizontal and vertical translations and scaling.

Figure 4 :

 4 Figure 4: Squared norm of each part of the latent code for horizontal position, vertical position and scale.

Figure 5 :

 5 Figure 5: Results of our evaluation procedure with four β-VAE for β = 1, 5, 10, 20. Note the erf shape of the results which indicates that the distribution of the shape positions has been correctly learned by the VAE. See Figure 2 for additional information on how to read this figure.

CFigure 6 :Figure 7 :Figure 8 :

 678 Figure 6: Reconstruction results with different σ values. We typically used a standard deviation of 3 pixels for the kernel.

Figure 9 :

 9 Figure 9: Two trajectories are shown in the pixel space, between an image and its transformed version, for three types of transformations: translation, scale and orientation. Red: shortest path (interpolation) between the two extremes of the trajectory. Blue: trajectory of the actual transformation. At each position along the trajectories, we report the corresponding image (best seen with zoom).

  (z) L(G(z), I target ) is near orthogonal to the manifold then: ∀i ∈ 1, ..., d : ∇ G(z) L(G(z), I target ),

Figure 10 :

 10 Figure 10: Qualitative results for 10 categories of ILSVRC dataset for three geometric transformations (horizontal and vertical translations and scaling) and for brightness.

Table 1 :

 1 ARCHITECTUREThe β-VAE framework was introduced by[START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF] to discover interpretable factorised latent representations for images without supervision. For our experiments, we designed a simple convolutional VAE architecture to generate images of size 64x64, the decoder network is the opposite of the encoder with transposed convolutions. β-VAE architecture used during experiments with the dSprites dataset.

	Encoder	Decoder
	Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Dense + ReLU units=256 Dense + ReLU units=256 µ: Dense + Identity σ: Dense + Exponential units=10	Dense + ReLU units=256 Dense + ReLU units=256 Reshape shape=4x4x32 Transposed Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Transposed Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Transposed Convolution + ReLU filters=32 size=4 stride=2 pad=SAME Transposed Convolution + Sigmoid filters=1 size=4 stride=2 pad=SAME

We could have used a L

penalty on the norm of z to encode a centered Gaussian prior on the distribution of z. However the L 2 penalty requires an additional hyper-parameter β that can be difficult to choose.2 Indeed, if we model the value of pixel by a random variable x then arg min x E (xx) 2 = E [x]. In fact, this problem can easily generalized at every pixel-wise loss if we assume that nearby pixels follows approximately the same distribution as arg min x E [L(x, x)] will have the same value for nearby pixels.