The perversity of the relationship between national economies and war

Jacques Fontanel

Economistes contre la course aux armements CESICE, Université Grenoble-Alpes, 2019

Two conceptions coexist concerning the relationship between the economy and war. The first considers that the costs of national defense constitute a burden for the national economy, while in another conception, war and economy are often considered as historically linked products. However, until very recently, the possession of wealth (through predation) and power were the two main stated reasons for wars. War is analyzed either as an expression of the struggle of national entities in the face of economic scarcity, but also in their hegemonic will, or as a burden to be overcome. War is the culmination of the political struggle for hegemonic power, against scarcity or for opulence.

Deux conceptions coexistent concernant les rapports entre l'économie et la guerre. La première considère que les coûts de la défense nationale constituent un fardeau pour l'économie nationale, alors que, dans une autre conception, la guerre et l'économie sont souvent considérés comme des produits historiquement joints. Cependant, jusqu'à une période très récente, la possession des richesses (par la prédation) et la puissance constituaient les deux principales raisons avouées des guerres. La guerre est analysée soit comme l'expression de la lutte d'entités nationales face à la rareté économique, mais aussi dans leur volonté hégémonique, soit comme un fardeau dont il faudrait arriver à s'en passer. La guerre est le point culminant de la lutte politique pour la puissance hégémonique, contre la rareté ou pour l'opulence.

The link between economy and war is complex, both being, in turn, ends and means. Thus, a war is prepared for and requires financing intended to ensure the security of the country (Si vis pacem, para bellum). The economy is itself a weapon, intended to impoverish or destabilize the enemy country (sanctions, blockade, destruction). In this situation, it is not directly a question of trying to conquer markets, but rather of engaging in a negative-sum game, at the highest possible price for the enemy, hoping for a return on investment with a defensive or offensive victory. In this context, economic optimization gives way to the relative power of states. Two conceptions coexist concerning the relationship between the economy and war. The first considers that the costs of national defense constitute a burden for the national economy, while in another conception, war and economy are often considered as historically linked products. However, until very recently, the possession of wealth (through predation) and power were the two main stated reasons for wars. War is analyzed either as an expression of the struggle of national entities in the face of economic scarcity, but also in their hegemonic will, or as a burden to be overcome.

War is the culmination of the political struggle for hegemonic power, against scarcity or for opulence.

Not every war is to be condemned, especially when its aim is the defense of human rights or the rejection of imperialism. The American Civil War allowed the rise of the industrial and financial power of the strongly protectionist North-East against the agricultural producer South, which favored free trade. This resulted in rapid industrial development. On the other hand, the 1914-1918 war considerably weakened the national economies and demographics of European countries. At the end of the last century, the launch by the United States of the "Star Wars" program sounded the death knell for the Soviet experiment. This time, the economy dictated its law.

From the beginning of time, war was conceived as a normal way to acquire wealth. Predation was a practical way to get rich. The defeated people are led to slavery, which constitutes an essential instrument of production in the Greek economy. War is always a moment of overcoming a country either to preserve its freedoms and protect its wealth, or to seize the goods of others in order to improve both the national economic conditions and to increase the relative power of the state. In these extreme conditions, there is a strong incentive for technological, scientific, organizational, human and managerial innovations. As Rabelais said, the sinews of war are "pecuni". States must have regular revenues to support military needs, and in doing so, they must also ensure that the national economy is able to provide the means for national security and regalian issues.

For the German historical school, war is adorned with a thousand virtues, including the development of productive forces. For Sombart, war presupposes colossal national efforts, which leads to a strong incentive to the spirit of competition, to the learning of social discipline by modern armies, to industrialization and industrial concentration, to the constitution of large markets, with the centralization of orders and mass consumption, to technological innovation, to the standardization necessary for economies of scale, to the rational organization of work, and even to the growth of the financial sector, through the management of the military debt of the state. War is a necessary evil for the economic development of capitalism. The European industrial revolution and the capitalist system were concretely the products of war. The military needs of armed conflict created the preconditions for the development of capitalism. The state has promoted the development of capitalism.

The imperialist state set out to develop its markets in the still weakly developed countries, sometimes with the help of violent means. The entrepreneurial spirit is expressed in military expeditions, which allow the development of civilization. The victorious country can then exploit raw materials and cheap means of subsistence, sometimes at the expense of the economic progress of the colonized countries. War serves to develop the economy of the victors at the expense of the vanquished. Under these conditions, the army is a productive force that energizes national capitalism.

Moreover, historically, military technology, financed by the state, has always been a vector of development. Many scientific discoveries have been financed by modern armies, which are great consumers of new technologies developed by the industrial system. In this context, the economic interests of industrialists and the military complement each other, and industry has learned a great deal in times of war. With the Manhattan Project, the great military projects received the organized help of modern science and technology, to the point of greatly influencing the civilian sector. With the establishment of military secrecy on high technologies, science, technology and economy became hostages of political power. Influenced by the dominant economic and social forces, technology is not apolitical.

For the mercantilists, the power of the Prince is the fundamental objective of the whole national society, and thus of the economy. The State is then an organization of power devoted to reinforcing the prosperity of the national economy. It is a question of weakening the opposing forces, by all means, in particular the violence of weapons, colonization or enslavement. Power does not exist without a strong economy and a dominating army. For Veblen, nationalism and economic warfare are essential to the development of the capitalist system. The thesis of unequal exchange or the theory of underdevelopment as a product of the development of great powers perpetuates this type of economic analysis in terms of power.

In 2005, Washington wants to maintain its world leadership, through international organizations (in charge of transforming the law of the strongest into international law) and through the ill-considered application of the ideology of globalization. The dollar owes its value to the military hegemony of the United States, but also to the importance of multinational firms of American origin. Even if the US government defends, in its discourse, an impartial and apolitical free trade, its military interventions in the world will undoubtedly weaken its "soft power" and possibly its military "hard power", even if the development of the digital economy gives it considerable potential power in terms of information, It also has the capacity to develop cyber programs intended for military-civilian operations against a "rogue state" or for a policy of dissuasion against countries that voluntarily or involuntarily oppose the political and economic priorities of the United States.

In an anticipated context of capitalist globalization, Marx considered that war belonged to the field of superstructures and that it was conditioned by the antagonistic social relations of capitalism. War and capitalism go hand in hand, only the revolutionary class struggle leads to peace. For Lenin, the military effort is a powerful instrument for the development of capitalist economies, through the development of primitive accumulation, colonial domination and the search for hegemony of the economically and militarily most advanced countries within the framework of the law of evolution of economic systems. Imperialism, the supreme stage of capitalism, leads to the capitalist exploitation of the world and to recurrent regional or world wars, within the framework of the sharing of foreign markets. The export of capital becomes necessary in order to fight against the laws of the tendency of profit rates to fall and equalize, a chronic and deadly disease of capitalism. If Marxist thought is above all marked by the concept of class struggle (which refers rather to a civil war), the concept of imperialism explains that, at a certain stage of its development, capitalism is also a producer of conflicts between bourgeois states.

For Baran and Sweezy, capitalism secretes a surplus, defined as the gap between production and solvent demand. This surplus can be absorbed mainly by military spending at the expense of civilian government spending or the consumption of capitalists, whose importance cannot grow at the same rate as their profits and income. Moreover, military expenditure stimulates collective values, which ensure the continuity of capitalism. Military power allows the exploitation of other nations, either through occupation of territory, intimidation, or the establishment of international laws that facilitate capitalist exploitation. Conflicts are immanent to capitalism, only their expressions evolve.

Without taking into account the Marxist analysis, J.K. Galbraith considers that the threat of war constitutes an indispensable element of control of social disagreements and anti-social tendencies. It is an instrument of stability and control of national economies, through the subordination of citizens to the state. In American capitalism, there are no valid substitutes for the military functions of conflict. For Jacques Attali, war is the extreme manifestation of industrial competition. Conflict leads to the revival of production, it transforms consumption patterns and social habits. The phase of non-battle favors the rise of military expenditures and the development of automated networks of selfsurveillance. The extended war phase allows the redistribution of power control in favor of the dominant industrial countries. Capitalism implies the emergence of a brutal phase which, if it does not lead to military war, leads to the development of symbolic alienation. Today, the latter could be found in the modern ideology of globalization.

War is an irrational economic burden Today, armed conflicts are particularly destructive and therefore very costly. Even the preparation for war can be destructive, as the failure of the Soviet Union bears witness. The potential attacker must know that he will suffer losses (economic sanctions and destruction of men and equipment) and experience a reduction in his gains (resistance, industrial sabotage, terrorism). Because of the extreme seriousness of the use of nuclear weapons, states use economic weapons more frequently.

For the Physiocrats and the great Classics, in order to ensure the sovereignty of States, it is necessary to build up military forces, in order to dissuade the predatory desires of other States, in a world of relative scarcity. For Ricardo, the unreasonable increase in military expenditure leads inevitably to war, because it produces a process of impoverishment which becomes a fundamental source of conflict. Malthus adds that economic development often proves insufficient in relation to the natural growth of the population. In this context, war becomes a privileged means of combating overpopulation, when famine or birth control have not done their job.

For Walras (1979), national defense is rejected from the field of study of economic science. However, if all countries adopted free trade, standing armies would be abolished and international disputes would be settled by arbitration. As a general rule, liberal economists consider that war has no future in a developed society (Nef, 1949). Military expenditure, with its weak and debatable knock-on effects, is unbearably expensive. Moreover, Pareto (1887Pareto ( /1901) is opposed to state intervention in economic life, and considers that wars are illegitimate in developed societies and that they lead to the decadence of civilizations. Deterrence is to be preferred, but the increase in military budgets increases the probability of war [START_REF] Richardson | Arms and insecurity. A mathematical study of the causes and origins of war[END_REF]. Arms race models highlight the three essential factors in the emergence of wars, namely the evolution of military spending by potential adversaries, grievances, and the economic "fatigability" resulting from the enormous costs of military spending. The economy plays a rather moderating role, but if grievances are strong, war can intervene, causing human and economic disaster. Keynes (1934), a convinced pacifist, wanted to eradicate both war and communism. While rearmament may lead to a temporary revival of the economy, in the long run it does not meet social needs and it slows down the national potential for economic development. It is preferable to direct public investment towards construction or public works, which are socially useful. However, security is also a matter for the economy, because the economic crisis sometimes leads to the overthrow of democracies and to the application of power politics. The economic crisis in post-World War I Germany is a threat to the economic development of Europe and to democracy. Sustainable peace is inconceivable without international economic solidarity of the democracies. Scientific knowledge of the economy is a factor of peace, in the face of totalitarian temptation and the personal and random games of politicians. Peace is also a condition for economic development. If a national system engages in an arms race, it must be opposed by economic warfare, which in the long run limits the military danger itself.

Economic warfare, the indirect weapon with perverse effects

To save a country is better than to destroy it. Subduing the enemy without fighting is the best (Sun Zi, Art.14). Today, economic warfare is more insidious, it concerns international rules considered "liberating" in the long term, while ensuring, in the short term, deadly inequalities. Economic warfare has always existed, from sieges to embargoes, in situations of war (scorched earth policy or economic terrorism) or peace (boycotts or embargoes). The primacy of the economy in the new international relations is affirmed. There are two objectives to economic warfare: to coerce the enemy and to dominate partners. All economic strategies do not have as their objective the well-being of the national economy, nor its development, but rather the weakening or subjugation of another economy, as already presented by mercantilist thinking (Fontanel, et al. 2008). Sanctions to cause significant economic damage to a country to change its policy (apartheid, violation of minority rights, tyranny or war) are essential instruments of economic and political power (Lachaux, 1992). Economic factors become permanent weapons which, however, in order to regenerate themselves, cannot be constantly diverted from their essential functions. With the strategy of encirclement, it is possible to develop links of economic interdependence that can guarantee peace (Realpolitik). The new economic solidarities are the best instrument for deterring aggression. However, economic rationality is not always that of economic, political or ideological powers. Insecurity depends on the arms race, social inequalities or inter-state dominations. The economy has become an instrument of power.

The collapse of the Soviet economy may not have been due to a single reason, but it was strongly affected by the uninterrupted effort to prepare for war in the face of the dominant economic force of the United States. The arms race comes at a cost, which the richer countries can more easily take into account by applying the strategy of impoverishment by imposing an economic "arm wrestle" through the arms race. The least developed countries are impoverished more rapidly, as they do not have enough "discretionary income". Under these conditions, the arms race leads to the weakening of the poorest state, with a view to destabilizing it socially, politically and economically, to the point of renouncing power.

Direct military strategies are made difficult to apply with the apocalyptic power of nuclear weapons. The great powers must therefore have oblique strategies of dissuasion, economic measures of retaliation, decisions on embargoes or boycotts, whose economic and political effects depend on self-defence measures, international solidarity and the potential for substitution. This may include controlling the sale of arms that could be turned against it, or developing strategies to control the export of dual products, half-civilian, half-military, in order to prevent the opposing army from increasing its economic potential. These strategies are very costly for the targeted country, but they are often also costly for the countries that engage in such action, especially when globalization has led all actors to a common dependence on essential products. It is undoubtedly possible to "punish" an oil-producing country internationally, but by refusing to buy its production, the result is an increase in the price of oil and a relative scarcity that will be passed on to all the national economies concerned by the action.

The strategy of domination leads dominant countries to influence the dominated countries in their strategic decisions. The economic weapon allows a country, or group of countries, to dominate another country thanks to the power conferred by the monopoly of the supply of goods and services vital for its survival. The strategy of political violence consists in taking the economic power of another country when it is hostile and in weakening its dominant social strata. It is then necessary to control the political forces of the state apparatus and the trade unions, by nationalization or privatization, by strikes or riots.

Today, all international economic relations must be interpreted from a strategic perspective, as the result of the complex interplay of international power relations. Contradictory hypotheses have been put forward. For neo-liberals, the end of nations is inscribed in the peaceful and rather benevolent logic of the market economy. For heterodox economists, the inevitable disintegration of national economies risks increasing insecurity and impoverishment. Capital flows or trade negotiations or international aid are powerful instruments of political pressure (or reaction). Protectionism, economic sanctions or the control of strategic products are at the service of a nation's political objectives, but sometimes, within the framework of the World Trade Organization for example, these states are obliged to respect the rules of free trade, unless they withdraw from the organization. [START_REF] Baldwin | Economic Statecraft[END_REF] considers that, in a situation of growing economic interdependence, the national interest must take into account the overlap of national interests. In this context, the distinction between war and the search for power is blurred. For [START_REF] Thurow | Head to head : the coming economic battle among Japan, Europe and America[END_REF] the United States cannot afford to lose the economic war. It is therefore necessary to protect high-tech sectors, to support American companies that are victims of foreign subsidies, to help sectors with high added value and to act firmly in international forums to promote the development of the American economy. The struggle is obviously expressed against the "rogue states", but also against allied countries, when power is too well shared between members of a group of states with common interests. In the United States, supposed military imperatives are often put forward to hide a real industrial policy of state subsidies and research and development. Economic factors often dominate the agenda of strategic considerations. As each nation competes with others in global markets, national security depends first on economic strength. The provision of a war room to conduct economic warfare highlights the application of Sun Zi's Principle 102, according to which "no spy is too beloved and no reward is too great for them."

For Labarde and Maris (1998), globalization is universal, civil and permanent war. It is an oligopolistic and cartelized organization of the world, in the respect of financial logic, which defines a new organization of the work of the capitalist enterprise and the freedom of installation of the owners of capital. Capital has always been international, stateless, more financial than industrial, more speculator than producer of wealth. Globalization can only be affirmed by reducing social protection and solidarity. Multinational firms are the great beneficiaries of this globalization. It is a daily and generalized civil war. The economic war is a social and societal suicide.

The opening of borders is not a necessary step towards peace. Yet today's military authorities combine an insatiable demand for new weapons with an increased aversion to their use. The nationality of a company loses its meaning, as currency, technology, and factories cross borders with fewer and fewer restrictions. Transnational firms and markets, now the main actors in the international economy, favor the mobility of capital and the relocation procedures and the development of new technologies, which are important factors of income. However, alongside islands of prosperity, the world economy is in crisis and the gap is widening. Globalization, dominated by the richest countries, based on inequalities, does not lead to peace, but to a war (without or with weapons) for the distribution of wealth.

Armed war remains an instrument of conflict management between States, but it is very costly, it is no longer socially "recognized" as a mode of crisis management and, above all, with nuclear weapons, it can trigger an apocalyptic situation. The multiplication of the number of states, the absence of a supranational order and the inability to enforce the law everywhere, open up a bright future for war For Jaurès, capitalism was already war. War" has changed in nature, it is not the main concern of today's societies, but it is a recurrent threat. But with the rise of autocracies, plutocracy and imperialist ideologies, it remains a course of action that the most authoritarian regimes will be able to impose on their external opponents and on their own people.


Valentin Gabeur

Paul Hongsuck Seo

Arsha Nagrani

Chen Sun

Karteek Alahari

Cordelia Schmid

AVATAR: Unconstrained Audiovisual Speech Recognition

Keywords: speech recognition, video, audiovisual

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Automatic Speech Recognition (ASR) is often applied to edited or streamed media (for example, TV, online videos, video conferencing), where the input signal consists of both an audio and a visual stream. For these applications, the visual stream can provide strong cues for improving ASR, particularly in cases where the audio is degraded or corrupted. This has been largely exploited by AV-ASR works which focus on lip motion [START_REF] Noda | Audio-visual speech recognition using deep learning[END_REF][START_REF] Tamura | Audio-visual speech recognition using deep bottleneck features and high-performance lipreading[END_REF][START_REF] Chung | Lip reading sentences in the wild[END_REF][START_REF] Afouras | Deep audio-visual speech recognition[END_REF][START_REF] Petridis | End-to-end audiovisual speech recognition[END_REF][START_REF] Makino | Recurrent neural network transducer for audio-visual speech recognition[END_REF][START_REF] Ma | End-to-end audio-visual speech recognition with conformers[END_REF][START_REF] Serdyuk | Audio-visual speech recognition is worth 32x32x8 voxels[END_REF] (using video crops centered around the speaker's mouth). While lip motion is a strong signal in videos centered on the speaker, it may be less useful in some online videos (those with egocentric viewpoints, face coverings, poor video quality, speaker at a distance etc.). A more recent and less-explored direction is the contribution of additional visual context, for example, the hand movements of a speaker, the presence of certain objects that are being described or even the background location [START_REF] Miao | Open-domain audio-visual speech recognition: A deep learning approach[END_REF].

In this paper, we focus on the latter case. The main existing benchmark for this task is the How2 dataset, which consists of instructional videos where the ground truth is obtained from user uploaded transcripts [START_REF] Sanabria | How2: a large-scale dataset for multimodal language understanding[END_REF]. While extremely valuable, the How2 dataset was created by keeping the audio samples that were most aligned to user-uploaded transcripts. This was done using an automatic alignment tool, and biases the dataset towards 'clean' audio samples. We posit that in this case, a model trained on clean audio would never be incentivised to learn from the visual modality, as all the information for ASR is present in the audio stream. This has led to a number of AV-ASR works to doubt whether the visual modality is useful at all in a clean audio context, or if it is simply used as a regularizer [START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Srinivasan | Analyzing utility of visual context in multimodal speech recognition under noisy conditions[END_REF][START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF].

To resolve this issue, we explore masking strategies that degrade the audio samples during training, and then evaluate our model under noisy audio conditions. By dropping out key words in the audio signal, we incentivise our model to pay attention to the visual stream. Our model is based on a Seq2Seq encoder-decoder architecture. Unlike previous works that use full-frame pre-extracted visual features [START_REF] Miao | Open-domain audio-visual speech recognition: A deep learning approach[END_REF][START_REF] Sanabria | How2: a large-scale dataset for multimodal language understanding[END_REF][START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF][START_REF] Gupta | Visual features for context-aware speech recognition[END_REF][START_REF] Moriya | Lstm language model adaptation with images and titles for multimedia automatic speech recognition[END_REF][START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF][START_REF] Paraskevopoulos | Multimodal and multiresolution speech recognition with transformers[END_REF][START_REF] Srinivasan | Looking enhances listening: Recovering missing speech using images[END_REF], our encoder performs audio-visual fusion early, and is trained directly from pixels and spectrograms. We show that large performance gains can also be achieved by pretraining our model on the large HowTo100M [START_REF] Miech | HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips[END_REF] dataset (not to be confused with How2).

Given the clean audio on the How2 test set, we also simulate noise at test time [START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF]. Unlike [START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF] which explicitly drops 'visual words', we simulate noise in a more objective way, by randomly dropping audio segments, or adding external environmental sounds from the AudioSet dataset [START_REF] Gemmeke | Audio set: An ontology and human-labeled dataset for audio events[END_REF]. We show that under these conditions, our model with audio-visual inputs consistently outperforms an audio only model for the task of AV-ASR. In addition, we also create a small, real-world test set with naturally occurring noisy audio. This dataset is created by filtering out samples where automatic ASR gets perfect results, and hence is a challenging test bed. On this dataset, we show that the visual modality makes a significant contribution to performance.

In this work we make the following contributions: (i) We propose a new encoder-decoder AudioVisual ASR TrAns-formeR (AVATAR) which is trained end-to-end from spectrograms and full-frame RGB. Our encoder fuses audio and visual inputs and is trained jointly with the decoder; (ii) To prevent the audio stream from dominating training, we propose and compare a number of masking strategies during training, thereby encouraging our model to pay attention to the visual stream; (iii) Our model achieves state-of-the-art performance on the How2 AV-ASR benchmark, with visual information improving performance under various simulated noise conditions; and finally (iv) we create a new, challenging real-world test bed for AV-ASR called VisSpeech. The dataset is created using a combination of automatic techniques and manual annotation and unlike other works, allows us to demonstrate the performance of our method under realistic noise conditions. We have released this dataset publicly to the research community at https://gabeur.github.io/avatar-visspeech.

Related Work

Audio-visual speech recognition. CTC [START_REF] Graves | Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[END_REF] and Seq2Seq [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF][START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF] are the two most popular losses for performing ASR. In the context of AV-ASR focused on lip motion, they have been compared [START_REF] Afouras | Deep audio-visual speech recognition[END_REF] and combined [START_REF] Ma | End-to-end audio-visual speech recognition with conformers[END_REF]. While early approaches [START_REF] Noda | Audio-visual speech recognition using deep learning[END_REF][START_REF] Tamura | Audio-visual speech recognition using deep bottleneck features and high-performance lipreading[END_REF] use pre-extracted lip visual features, recent works [START_REF] Chung | Lip reading sentences in the wild[END_REF][START_REF] Afouras | Deep audio-visual speech recognition[END_REF][START_REF] Petridis | End-to-end audiovisual speech recognition[END_REF][START_REF] Makino | Recurrent neural network transducer for audio-visual speech recognition[END_REF][START_REF] Ma | End-to-end audio-visual speech recognition with conformers[END_REF][START_REF] Serdyuk | Audio-visual speech recognition is worth 32x32x8 voxels[END_REF] adopt an end-to-end approach by directly processing the pixels of the speaker's lips. In contrast, the contribution of full frames for AV-ASR beyond the speaker's mouth movements has only been studied through pre-extracted visual context features: either action features [START_REF] Sanabria | How2: a large-scale dataset for multimodal language understanding[END_REF][START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF][START_REF] Paraskevopoulos | Multimodal and multiresolution speech recognition with transformers[END_REF], place features [START_REF] Miao | Open-domain audio-visual speech recognition: A deep learning approach[END_REF][START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Gupta | Visual features for context-aware speech recognition[END_REF][START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF][START_REF] Srinivasan | Looking enhances listening: Recovering missing speech using images[END_REF] or object features [START_REF] Miao | Open-domain audio-visual speech recognition: A deep learning approach[END_REF][START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Gupta | Visual features for context-aware speech recognition[END_REF][START_REF] Moriya | Lstm language model adaptation with images and titles for multimedia automatic speech recognition[END_REF][START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF][START_REF] Srinivasan | Looking enhances listening: Recovering missing speech using images[END_REF]. Unlike these works, we train directly from full frame pixels for AV-ASR. Full-frame AV-ASR datasets. The How2 dataset [START_REF] Sanabria | How2: a large-scale dataset for multimodal language understanding[END_REF], built from instructional videos, is the main benchmark for the fullframe AV-ASR task. Prior to this benchmark, full-frame AV-ASR works [START_REF] Miao | Open-domain audio-visual speech recognition: A deep learning approach[END_REF][START_REF] Gupta | Visual features for context-aware speech recognition[END_REF][START_REF] Palaskar | End-to-end multimodal speech recognition[END_REF] have also evaluated on instructional videos datasets but those have not been released. Audiocaptioned image datasets [START_REF] Srinivasan | Analyzing utility of visual context in multimodal speech recognition under noisy conditions[END_REF][START_REF] Srinivasan | Looking enhances listening: Recovering missing speech using images[END_REF] have also been used for AV-ASR. User-uploaded transcripts are the main source of ground truth for large scale AV-ASR video datasets [START_REF] Makino | Recurrent neural network transducer for audio-visual speech recognition[END_REF][START_REF] Serdyuk | Audio-visual speech recognition is worth 32x32x8 voxels[END_REF][START_REF] Sanabria | How2: a large-scale dataset for multimodal language understanding[END_REF]. As these are often misaligned or inaccurate, transcripts are typically first audio-aligned and then automatically filtered [START_REF] Liao | Large scale deep neural network acoustic modeling with semi-supervised training data for youtube video transcription[END_REF]. Audio signal degradation for AV-ASR evaluation. In the case of lip motion for AV-ASR, several works have experimented with adding 'babble noise' [START_REF] Afouras | Deep audio-visual speech recognition[END_REF][START_REF] Makino | Recurrent neural network transducer for audio-visual speech recognition[END_REF] or extra speech tracks [START_REF] Makino | Recurrent neural network transducer for audio-visual speech recognition[END_REF][START_REF] Serdyuk | Audio-visual speech recognition is worth 32x32x8 voxels[END_REF]. Additive Gaussian noise has also been used in [START_REF] Serdyuk | Audio-visual speech recognition is worth 32x32x8 voxels[END_REF]. In the case of full-frame AV-ASR, it is common to completely mask some segments of the audio signal that correspond to visual words. Srinivasan et al. [START_REF] Srinivasan | Looking enhances listening: Recovering missing speech using images[END_REF] mask the audio segments corresponding to nouns and places. Ghorbani et al. [START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF] compute the similarity between words and visual frames and mask only the visual words. We instead attempt to mimic more realistic settings by simulating audio degradations on How2 and releasing our 'in the wild' benchmark VisSpeech. In order to prevent audio from dominating the task, we extend audio degradation to the training phase by randomly masking input words at training time.

Method

Model Architecture

In this section we provide an overview of our audiovisual ASR model called AVATAR (Figure 1). Our model consists of a multimodal encoder to encode both RGB frames and audio spectrograms, and a transformer decoder which produces the natural language speech recognition output. Unlike previous AV-ASR works, we do not use frozen visual features, but have a single multimodal encoder that allows early multimodal fusion [START_REF] Nagrani | Attention bottlenecks for multimodal fusion[END_REF]. Audio Inputs. Our model operates on 25 second audio inputs. We follow common practice and extract 80-dimensional filter bank features from the 16kHz raw speech signal using a Hamming window of 25ms and a stride of 10ms, giving us 80 × 2500 size spectrograms for 25 seconds of audio. We then extract 16 × 16 non overlapping patches, giving us a total of 5 × 156 = 780 input tokens for audio. RGB Inputs. We randomly extract 2 frames at 2.5 fps from each input video clip, which are then converted into tokens by extracting 16 × 16 × 2 tubelets resulting in a total of 14 × 14 = 146 input tokens (the image resolution is 224 × 224). This is based on our observations that visual signals are highly redundant for most videos and can therefore be efficiently captured from few frame samples. Audiovisual Encoder. We adopt the recently proposed MBT architecture [START_REF] Nagrani | Attention bottlenecks for multimodal fusion[END_REF], which is a transformer based multimodal encoder. Given both sets of audio and RGB tokens, MBT first adds positional encodings to each token, and append a CLS token to each set. The sets are then fed to the MBT encoder. MBT relies on bottleneck tokens to model the cross-modal interactions. Here we use the best parameters from [START_REF] Nagrani | Attention bottlenecks for multimodal fusion[END_REF] (4 bottleneck tokens Decoder. All hidden units from the encoder are then passed to an auto-regressive transformer decoder [START_REF] Vaswani | Attention is all you need[END_REF] consisting of 8 layers and 4 attention heads.

Training Strategies

In this section we describe our training loss and strategy for AVATAR. Our model is first pretrained on a large dataset with transcripts obtained using an audio-only ASR API [START_REF] Miech | HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips[END_REF]. To entice the model to pay attention to the visual modality, we introduce a word masking strategy, which is described below. End to end training. The model is trained end-to-end using a cross-entropy loss on each decoded token. Word Masking. To prevent our model from ignoring the visual modality, we introduce word masking techniques during training. We randomly sample target words and mask out the input audio signals that correspond to those words using pre-extracted alignments between words and the input signal. We obtain the alignment either from ASR results for pretraining or by using an off-the-shelf forced-alignment tool for finetuning. For selecting target words to mask, we experiment with two strategies: random and content word masking. The former strategy selects target words randomly whereas the latter chooses the targets among non-stop (henceforth known as content) words. For random word masking, we randomly mask out 10% of the words. For content word masking, there are fewer candidate words to be masked so content words are masked at a higher rate to match the 10% overall masking on the entire dataset.

VisSpeech Dataset

In this section we describe our new AV-ASR benchmark called VisSpeech. Our dataset is a subset of the publicly released HowTo100M dataset [START_REF] Miech | HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips[END_REF], and is curated using a combination of automatic filtering stages and manual verification.

Dataset creation pipeline:

Our dataset creation pipeline is driven by two objectives: (1) We want to find challenging audio conditions in which regular audio-based ASR fails. For this we seek videos where there is a large word error rate between automatic ASR and usergenerated transcripts. [START_REF] Tamura | Audio-visual speech recognition using deep bottleneck features and high-performance lipreading[END_REF] We are also interested in video seg-ments where there is high audio-visual correspondence, in order to create a suitable multimodal test set for AV-ASR. Our pipeline consists of the following steps:

Step 1: Obtain videos with user uploaded transcripts. We first search for HowTo100M videos that have both manually uploaded transcripts and automatic ASR. We then align the transcripts using the Levenshtein algorithm in order to compute the global word error rate (WER) between the two. Videos with a global WER greater than 100% are removed, as we find this helps filter out completely wrong user uploaded transcripts or ASR. This gives us 85K videos.

Step 2: ASR vs user transcripts. Videos are split into segments, at silences detected using an open-source VAD model [START_REF] Team | Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier[END_REF]. Using the user transcripts and the ASR from each segment, we filter out segments with WER greater than 50% between the two transcripts to remove samples with significantly low quality user transcripts or ASR. This is similar to the rationale for the filtering in Step 1, which is performed at a video level, but now we perform it at a segment level. Next and most importantly we remove "easy" samples, namely those with low WER of less than 20% for non-stop words (too clean) and samples with less than 9 words (too short). This leaves us with 773K sentences from 7.5K videos.

Step 3: Visual-Text similarity. To measure visual-text similarity, we run a video-text similarity model [START_REF] Nagrani | Learning audio-video modalities from image captions[END_REF] trained on the Howto100M dataset to get similarity scores between each video and sentence pair. This helps highlight challenging samples where the visual modality can compensate for corrupted audio.

Step 4: Manual annotation. Finally, we manually check the highest similarity segments and correct the user-uploaded ASR if necessary.

VisSpeech consists of 508 segments from 495 unique videos. The average transcript length is 12.2 words and average segment duration is 4.3 seconds. By filtering for segments where ASR fails, we find that our dataset is truly a challenging test bed 'in the wild', with the audio containing background chatter, laughter, music and other environmental sounds. During the manual verification phase, we also noticed that many examples contain speech spoken with challenging English accents from various regions all over the world.

Experiments

In this section we first describe the datasets, metrics and implementation details for training (Section 5.1). We then describe the simulated noise we use for evaluating our models on the How2 dataset (Section 5.2). Finally we discuss the results of our model under different training strategies on both the How2 and our newly introduced VisSpeech dataset (Section 5.3).

Datasets, Metrics, Implementation Details

HowTo100M [START_REF] Miech | HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips[END_REF] consists of more than 1 million instructional videos associated with their automatically-extracted speech transcriptions. We only use this dataset for pre-training our model. Note that here we are not training with perfect ground truth, but using the ASR outputs of an existing model. We remove videos present in the validation and test sets of VisSpeech and How2 datasets (described next). How2 [START_REF] Sanabria | How2: a large-scale dataset for multimodal language understanding[END_REF] is an instructional video dataset created for multimodal language understanding. We use the 300 hour version. The videos are segmented into short clips (avg 5.8s), each accompanied by their user-uploaded transcript (avg 20 words). The dataset is split between a training (184,949 clips), valida-tion (2,022 clips) and test (2,305 clips). Metrics. We evaluate our models using Word Error Rate (WER). For each sentence, dynamic programming is used to align the predicted words to the ground truth. The number of word errors (deletions, substitutions and insertions) is then computed across the whole test dataset and divided by the number of ground truth words to obtain the WER. Training Implementation details. All models are trained endto-end unless otherwise specified. We use a batch size of 1,536 and 256 for pretraining and finetuning respectively. We adopt a wordpiece tokenizer [START_REF] Wu | Google's neural machine translation system: Bridging the gap between human and machine translation[END_REF] pretrained for BERT and decode using a beam search with a beam size of 4 and a brevity penalty of 0.6. We use SpecAugment with parameters adopted from [START_REF] Park | Specaugment: A simple data augmentation method for automatic speech recognition[END_REF]. We augment the visual frames using random cropping and color jittering. We use a momentum optimizer. We pretrain our model for 1M iterations with an initial learning rate of 2. The learning rate is warmed up for 1K iterations and then linearly decayed to 0. We initialize both visual and audio streams of the MBT encoder with the public ViT [START_REF] Kolesnikov | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] weights pretrained on ImageNet. For finetuning, we train for 40K iterations without warmup.

Simulated Noise for Evaluation

The ground-truth transcripts in How2 are collected by performing forced-alignment on the user uploaded transcripts, and filtering out examples with a low confidence score. Due to such a filtering process, the audio signals in How2 are inherently clean, and consequently the task is largely audio dominant. To overcome this limitation, we evaluate our models with three types of simulated audio noise: burst packet loss, environment noise and mixed noise. For the burst packet loss, we randomly drop two chunks of the input audio signal where the length of each chunk is uniformly sampled from (0, 0.1] times the video duration. To simulate environment noise, we add audio noise randomly sampled from the 'noise' and 'environmental' classes in the AudioSet dataset [START_REF] Gemmeke | Audio set: An ontology and human-labeled dataset for audio events[END_REF]. Finally, we also evaluate a combination of the two ('mixed noise'). Note that we train a single model and evaluate it under different noise configurations.

Results

Effects of Training Strategies. Table 1 compares audio-only (A) and audiovisual (A+V) models trained with different training strategies and evaluation noise settings on How2. In addition to clean audio, we report results in three degraded audio scenarios, burst loss, environment noise and mixed noise. We first note that without pretraining, our model performs well on the clean eval, but performance degrades significantly under simulated noise conditions. Adding the visual modality under these noise conditions helps performance across the board. Vanilla pretraining then improves performance significantly, however we note the gap between A and A+V also shrinks, signifying the improvement is largely from better audio encoding. In this case, the A+V model has no incentive to look at visual inputs, as the task under clean conditions is dominated by audio. In addition, the pretraining is performed on HowTo100M where the transcripts are automatically generated from the audio alone, and so our model is able to solve the task without any visual information. We find the word masking strategies to be extremely effective to mitigate this. The overall performance of both A and A+V are improved and notably, the improvements of A+V are larger. We show that with these masking schemes, adding visual inputs helps even for clean audio, with the performance improving under environment and mixed noise. Note that across the board, adding the visual modality improves per- Figure 2: Qualitative results on the VisSpeech dataset. We show the ground truth (GT), and predictions from our audio only (A) and audio-visual model (A+V). Note how the visual context helps with objects ('chain', 'eggplant', 'coin', 'dough'), as well as actions ('knit', 'fold') which may be ambiguous from the audio stream alone. Errors in the predictions compared to the GT are highlighted in red.

formance.

Comparison to the state-of-the-art. Table 2 compares AVATAR with existing state-of-the-art methods on How2. Our model trained from scratch already outperforms all existing methods and serves as a strong baseline. Our best model, which is pretrained on HowTo100M, with the random word masking technique, brings a further boost reducing the error rate by over 45% relatively compared to the existing state-of-the-art method. Evaluation on VisSpeech. We evaluate our AVATAR model trained with different strategies on VisSpeech with real-world noise (Table 3). We finetune the pretrained models for 5K iterations on How2. Our dataset effectively highlights the contribution of the visual modality without introducing any artificial noise. Once again, both masking strategies help the audiovisual (A+V) model learn to utilize the visual modality better. Content word masking improves the performance only when the visual modality is provided; providing some evidence that the A+V model uses visual inputs to correct errors on content words. To further tease apart the input of the visual modality, we compute the word error rate on content words only and on stop words only. This is because we hypothesize that visual modality should not be able to provide any useful information about stop words, and so most of the improvement should be on the content words. As expected, we find the errors on content words are reduced substantially more than those on stop words when the visual modality is incorporated with all training strategies (e.g., 1.18% vs. 0.33% absolute error rate drops with content word masking). This also confirms the contribution of the visual modality. Further evidence can be found from the qualitative examples provided in Figure 2, where it can be clearly seen that visual context helps with correcting ASR errors on objects as well as actions. End-to-end training with early audiovisual fusion. Unlike previous works on full-frame AV-ASR, AVATAR is trained (i) entirely end-to-end, and (ii) with early audiovisual fusion in the MBT encoder. To assess this effect, we test AVATAR with preextracted visual features as in [START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF], using a model pretrained on HowTo100M with NCE loss [START_REF] Nagrani | Learning audio-video modalities from image captions[END_REF]. We concatenate the audio features at the output of our MBT encoder with our preextracted visual features and provide them to the decoder. Note that in this case the audio-visual fusion happens only through the decoder. We train this model with random masking strategy and find that the end-to-end trained model outperforms the model with pre-extracted features with 9% relative improvement in the mixed noise setting and similar trends are observed in all the other noise settings. Contribution of Visual Modality. Some works show that the visual modality is simply a regularizer [START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Srinivasan | Analyzing utility of visual context in multimodal speech recognition under noisy conditions[END_REF][START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF]. As done by [START_REF] Srinivasan | Analyzing utility of visual context in multimodal speech recognition under noisy conditions[END_REF], we further investigate whether the contribution of the visual modality is simply a regularizer by replacing the visual frames of test examples with those extracted from random validation videos. Unlike [START_REF] Srinivasan | Analyzing utility of visual context in multimodal speech recognition under noisy conditions[END_REF], we observe significant degradation of A+V models in all settings (e.g., 9.11%→9.53% with random word masking and clean audio) as the models get distracted by the random visual inputs. Note that this performance (9.53%) is worse than the audio-only model in this setting (9.19%). This suggests vision in our model is not simply a regularizer contrary to what was previously reported in [START_REF] Caglayan | Multimodal grounding for sequence-to-sequence speech recognition[END_REF][START_REF] Ghorbani | Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations[END_REF].

Conclusion

In this work, we propose a novel encoder-decoder transformer architecture and training strategies based on word masking for AV-ASR. We show that our method helps the model learn to use visual inputs better and outperform the state of the art. Finally we also release VisSpeech, a new AV-ASR test benchmark, and demonstrate the effectiveness of our method under naturally occurring noise.

Figure 1 :

 1 Figure 1: AVATAR: We propose a Seq2Seq architecture for audio-visual speech recognition. Our model is trained end-toend from RGB pixels and spectrograms.

Table 1 :

 1 Audiovisual ASR vs Audio only models under various evaluation noise conditions (Clean, Burst, Environment and Mixed) and with different training masking strategies (Random and Content). Percentage Word Error Rate (%WER) is reported on the How2 test set. A: Audio-only. A+V: Audiovisual. Rel. ∆: Relative improvement of A+V over A.

	Eval Noise		Clean		Burst Loss	Environment Noise		Mixed Noise
	Training	A	A+V Rel. ∆	A	A+V Rel. ∆	A	A+V Rel. ∆	A	A+V Rel. ∆
	No Pretraining	15.72 15.62 0.64% 29.59 28.69 3.05% 50.79 47.70 6.08% 60.51 57.49 5.0%
	Vanilla	9.75 9.79 -0.33% 21.97 21.71 1.19% 25.97 25.55 1.61% 39.13 38.96 0.42%
	Random Word Masking	9.19 9.11 0.93% 15.60 15.28 2.05% 23.39 22.35 4.45% 32.43 30.64 5.50%
	Content Word Masking	9.58 9.25 3.48% 17.26 16.92 1.98% 23.77 22.67 4.65% 33.83 32.26 4.53%

Table 2 :

 2 Comparison to the state-of-the-art on How2. Our model outperforms all previous works when trained from scratch, and pretraining provides a significant boost. We report the best audio-visual numbers for all works.

	Model	%WER
	BAS [10]	18.0
	VAT [11]	18.0
	MultiRes [17]	20.5
	LLD [13]	16.7
	AVATAR (scratch)	15.6
	AVATAR (pretrained)	9.1

Table 3 :

 3 WERs of AVATAR on our newly introduced test set Vis-Speech consisting of real-world noise. The models are trained on automatic ASR from HowTo100M, and finetuned on How2. Note here we do not add any artificial audio degradation at all.

	Training Strategy	A	A+V	Rel. ∆
	No pretraining	44.57 43.41	2.61%
	Vanilla	12.69 11.91	6.11%
	Random Word Masking 12.35 11.86	3.93%
	Content Word Masking	12.72 11.28	11.30%

d model is the embedding dimension, L is the number of transformer layers, N H is the number of self-attention heads with hidden dimension d.

Acknowledgements. This work was supported in part by the ANR grant AVENUE (ANR-18-CE23-0011).