Antoine Miech

Ivan Laptev

Josef Sivic

Learnable pooling with Context Gating for video classification

Common video representations often deploy an average or maximum pooling of pre-extracted frame features over time. Such an approach provides a simple means to encode feature distributions, but is likely to be suboptimal. As an alternative, we here explore combinations of learnable pooling techniques such as Soft Bag-of-words, Fisher Vectors, NetVLAD, GRU and LSTM to aggregate video features over time. We also introduce a learnable non-linear network unit, named Context Gating, aiming at modeling interdependencies between features. We evaluate the method on the multi-modal Youtube-8M Large-Scale Video Understanding dataset using pre-extracted visual and audio features. We demonstrate improvements provided by the Context Gating as well as by the combination of learnable pooling methods. We finally show how this leads to the best performance, out of more than 600 teams, in the Kaggle Youtube-8M Large-Scale Video Understanding challenge.

Introduction

Understanding and recognizing video content is one of the major challenges in computer vision. Applications include surveillance, personal assistance, smart homes, autonomous driving, stock footage search and sports video analysis. Current methods for video analysis typically represent videos by features extracted from one or several consecutive frames, followed by feature aggregation over time. Example methods for feature extraction include deep convolutional neural network (CNN) features pre-trained on static images [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]25,36,[START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF] or short video clips [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF]12], as well as hand-crafted video features [START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Schüldt | Recognizing human actions: a local svm approach[END_REF][START_REF] Wang | Action Recognition with Improved Trajectories[END_REF]. Common methods for feature aggregation include simple temporal averaging or max-pooling as well as more sophisticated pooling techniques such as VLAD [20] as well as temporal models such as LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] and GRU [START_REF] Cho | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches[END_REF]. In this work we make the following two contributions. First we explore temporal aggregation of visual and audio features by designing new and comparing existing learnable pooling methods such as NetVLAD [START_REF] Arandjelovic | NetVLAD: CNN architecture for weakly supervised place recognition[END_REF], GRU [START_REF] Cho | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches[END_REF] and LSTM [START_REF] Hochreiter | Long short-term memory[END_REF]. We investigate each method individually and demonstrate their complementarity through combination. Second, inspired by [START_REF] Dauphin | Language modeling with gated convolutional networks[END_REF], we introduce a non-linear learnable network unit, named Context Gating (CG). CG aims at better capturing the non-linear interdependencies between features as well as among output labels.

We evaluate our method on the multi-modal Youtube-8M V2 dataset containing about 8M videos and 4716 unique tags. We use pre-extracted visual and audio features provided with the dataset [2] and demonstrate improvements obtained with the Context Gating as well as by the combination of learnable poolings. Our method obtains best performance, out of more than 600 teams, in the Kaggle Youtube-8M Large-Scale Video Understanding challenge. Figure 1 illustrates some qualitative results of the method. 1

Related work

This work is related to previous methods on video features extraction and feature aggregation reviewed below.

Feature extraction. Successful hand-crafted representations [START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Schüldt | Recognizing human actions: a local svm approach[END_REF][START_REF] Wang | Action Recognition with Improved Trajectories[END_REF] are based on local histograms of image and motion gradient orientations extracted along dense trajectories [START_REF] De Souza | Sympathy for the details: Dense trajectories and hybrid classification architectures for action recognition[END_REF][START_REF] Wang | Action Recognition with Improved Trajectories[END_REF]. More recent methods extract deep convolutional neural network activations computed from individual frames or blocks of frames using spatial [12,[START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF][START_REF] Girdhar | Actionvlad: Learning spatio-temporal aggregation for action classification[END_REF][START_REF] Wang | Action recognition with trajectory-pooled deep-convolutional descriptors[END_REF] or spatio-temporal [START_REF] Baccouche | Sequential deep learning for human action recognition[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Ji | 3D Convolutional Neural Networks for Human Action Recognition[END_REF][START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF][START_REF] Varol | Long-term Temporal Convolutions for Action Recognition[END_REF] convolutions. Convolutional neural networks can be also applied separately on the appearance channel and the pre-computed motion field channel resulting in the, so called, two-stream representations [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]12,[START_REF] Girdhar | Actionvlad: Learning spatio-temporal aggregation for action classification[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Varol | Long-term Temporal Convolutions for Action Recognition[END_REF].

Feature aggregation. Video features are typically extracted from individual frames or short video clips. The remaining question is: how to aggregate video features over the entire and potentially long videos? One way to achieve this is to employ recurrent neural networks, such as long short-term memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF] or gated recurrent unit (GRU) [START_REF] Cho | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches[END_REF]), on top of the extracted frame-level features to capture the temporal structure of video into a single representation [5,11,[START_REF] Ibrahim | A Hierarchical Deep Temporal Model for Group Activity Recognition[END_REF][START_REF] Lev | Rnn fisher vectors for action recognition and image annotation[END_REF][START_REF] Yue-Hei | Beyond short snippets: Deep networks for video classification[END_REF]. Other methods capture only the distribution of features in the video, not explicitly modeling their temporal ordering. The simplest form of this approach is the average or maximum pooling of video features [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF] over time. Other commonly used methods include bag-ofvisual-words [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF]37], Vector of Locally aggregated Descriptors (VLAD) [20] or Fisher Vector [START_REF] Perronnin | Fisher kernels on visual vocabularies for image categorization[END_REF] encoding. Application of these techniques to video include [START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Peng | Boosting VLAD with Supervised Dictionary Learning and High-Order Statistics[END_REF][START_REF] Schüldt | Recognizing human actions: a local svm approach[END_REF][START_REF] Wang | Action Recognition with Improved Trajectories[END_REF][START_REF] Xu | A Discriminative CNN Video Representation for Event Detection[END_REF]. The variants of these methods [START_REF] Lev | Rnn fisher vectors for action recognition and image annotation[END_REF]32] rely on an unsupervised learning of the codebook. However, the codebook can be also learnt in a discriminative manner [START_REF] Peng | Boosting VLAD with Supervised Dictionary Learning and High-Order Statistics[END_REF][START_REF] Peng | Action recognition with stacked fisher vectors[END_REF][START_REF] Sydorov | Deep fisher kernels and end to end learning of the Fisher kernel GMM parameters[END_REF] or the entire encoding module can be included within the convolutional neural network architecture and trained in an end-toend manner [START_REF] Arandjelovic | NetVLAD: CNN architecture for weakly supervised place recognition[END_REF]. This type of end-to-end trainable orderless aggregation has been recently applied to video in [START_REF] Girdhar | Actionvlad: Learning spatio-temporal aggregation for action classification[END_REF].

Multi-label video classification architecture

Overall architecture. Our architecture for video classification is illustrated in Figure 2 and contains three main modules. First, the input features are extracted from the video. Next, the pooling module aggregates the extracted features into a single compact (e.g. 1024-dimensional) representation for the entire video. Finally, the classification module takes the resulting video representation as input and outputs a set of labels for the video together with their scores. The three modules are described next. [START_REF] Hershey | CNN architectures for large-scale audio classification[END_REF]. PCA and whitening are then applied to reduce the dimension to 1024 for the visual features and 128 for the audio features.

Feature pooling. The pooling module has a two-stream architecture that takes the visual and audio features as input. Each modality is then processed separately by a learnable pooling method (Section 4) into a single representation. These individually pooled representations are then concatenated and fed into a fully-connected layer to reduce their dimension into a compact (1024-dimensional) vector. This compact vector is then reweighed by the Context Gating layer (Section 5) capturing non-linear interdependencies among features.

Classification. The classification module is composed of a soft Mixture-of-Experts (MoE) classifier [START_REF] Jordan | Hierarchical mixtures of experts and the em algorithm[END_REF] as described in [2]. This is followed by the Context Gating layer that reweights the output class probabilities according to learnt prior structure of the output label space (Section 5).

Learnable pooling methods

Within our video classification architecture described above, we investigate several types of learnable pooling models, which we describe next.

Pooling via clustering. We explore end-to-end trainable variants of the following three pooling techniques: Bag-ofvisual-words [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF]37], VLAD [20] and Fisher Vector [START_REF] Perronnin | Fisher kernels on visual vocabularies for image categorization[END_REF].

For VLAD encoding, we use the NetVLAD [START_REF] Arandjelovic | NetVLAD: CNN architecture for weakly supervised place recognition[END_REF] architecture proposed for place recognition [START_REF] Arandjelovic | NetVLAD: CNN architecture for weakly supervised place recognition[END_REF] and then extended to action recognition in video [START_REF] Girdhar | Actionvlad: Learning spatio-temporal aggregation for action classification[END_REF]. As opposed to the original version of NetVLAD [START_REF] Arandjelovic | NetVLAD: CNN architecture for weakly supervised place recognition[END_REF], we did not pre-train the codebook with a k-means initialization as we did not notice any improvement by doing so. We have also investigated a modification of the original NetVLAD architecture that averages the actual descriptors instead of the residuals. We call this variant NetRVLAD (for Residual-less VLAD). This is a simplification of NetVLAD that uses less parameters and needs less computing operations (about half in both cases).

For bag-of-visual-words encoding, we use softassignment of descriptors to visual word clusters [START_REF] Arandjelovic | NetVLAD: CNN architecture for weakly supervised place recognition[END_REF][START_REF] Philbin | Lost in quantization: Improving particular object retrieval in large scale image databases[END_REF] to obtain a differentiable representation. We call this representation Soft-DBoW (for Soft Deep Bag-of-visual-Words).

Finally, for Fisher Vector encoding, we modify the NetVLAD architecture to allow learning of second order feature statistics within the clusters. We will denote this as NetFV (for Net Fisher Vector) as it is an end-to-end trainable variant of the Fisher Vector [START_REF] Perronnin | Fisher kernels on visual vocabularies for image categorization[END_REF].

Recurrent models for pooling. We have investigated Long Short-Term Memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF], and Gated Recurrent Units (GRU) [START_REF] Cho | On the Properties of Neural Machine Translation: Encoder-Decoder Approaches[END_REF]. For both of these models, we stacked two networks with hidden layers of size 1024. The pooled representation is the final state of the network after processing the whole sequence of features in the video.

Context Gating

In this section we describe the Context Gating (CG) layer that transforms the input feature representation X into a new representation Y . The layer has the following form:

Y = σ(W X + b) • X, (1)
where X ∈ R n is vector of the input feature activations, σ is the element-wise sigmoid activation and • is the elementwise multiplication. W ∈ R n×n and b ∈ R n are trainable parameters. The vector of weights σ(W X + b) acts as a set of learnt gates (with values between 0 and 1) on the individual dimensions of the input feature X.

The motivation behind this transformation is two-fold. First, we wish to introduce non-linear interactions among activations of the input representation. Second, we wish to recalibrate the strengths of different activations of the input representation through a self-gating mechanism. The form of the Context Gating layer is inspired by the Gated Linear Unit (GLU) introduced recently for language modeling [START_REF] Dauphin | Language modeling with gated convolutional networks[END_REF] that considers a more complex class of transformations given by σ(W 1 X + b 1 ) • (W 2 X + b 2 ), with two sets of learnable parameters W 1 , b 1 and W 2 , b 2 . Compared to the the Gated Linear Unit [START_REF] Dauphin | Language modeling with gated convolutional networks[END_REF], our Context Gating in (1) (i) reduces the number of learnt parameters as only one set of weights is learnt, and (ii) re-weights directly the input vector X (instead of its linear transformation) and hence is suitable for situations where X has a specific meaning, such the score of a class label, that is preserved by the layer. As shown in Figure 2, we use Context Gating in the feature pooling and classification modules. First, we use CG to transform the feature vector before passing it to the classification module. Second, we use CG after the classification layer to capture the prior structure of the output label space. Details are provided below.

Capturing dependencies among features. First, we place the Context Gating layer to transform the feature vector just before the classifier. The aim is to capture the dependencies among the features. For example, the context gating can learn to suppress features likely to be on background and emphasize the foreground objects. For instance, if features corresponding to 'Trees', 'Skier' and 'Snow' have high co-occurring activations in a skiing video, context gating could learn to suppress the background features such as 'Trees' and 'Snow', which are less important for the classification.

Capturing prior structure of output space. We also place the Context Gating layer after the Mixture of Experts classifier to re-weight the output probabilities for the different classes. For instance, the output label pair 'Make Up' and 'Car' is much less likely than the label pair 'Renault' and 'Car'. Context gating aims at downweighting such unlikely label combinations in the output.

Training details

All models are trained using the Adam algorithm [24] and mini-batches with around 100 frames. The learning rate is initially set to 0.0002 and is then decreased exponentially with the factor of 0.8 every 4M samples. We use gradient clipping and batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate[END_REF] before each non-linear layer.

For the clustering-based pooling models, i.e. Soft-DBoW, NetVLAD, NetRVLAD and NetFV, we randomly sample N features with replacement from each video. N is fixed for all videos at training and testing. For training of recurrent models, i.e. LSTM and GRU, we process features in the temporal order. We have also experimented with the random sampling of frames for LSTM and GRU which performs surprisingly similarly.

All our models are trained with the cross entropy loss. We found this loss to work well for maximizing the Global Average Precision (GAP) metric. Our implementation uses the TensorFlow framework [START_REF] Abadi | Tensorflow: Large-scale machine learning on heterogeneous distributed systems[END_REF]. Each training is performed on a single NVIDIA TITAN X (12Gb) GPU.

Method

Experiments

Youtube-8M Dataset. The Youtube-8M dataset [2] is composed of approximately 8 millions videos. Visual and audio features are pre-extracted and provided with the dataset for each second of the video. Visual features are obtained by the state-of-the-art Inception CNN followed by the PCA-compression into a 1024 dimensional vector. More details on feature extraction are available in [2]. Each video is labeled with one or multiple tags referring to the main topic of the video. For instance, a video showing someone making a chocolate cake may have labels 'Food', 'Cooking', 'Cake' and 'Chocolate' from the full set of 4716 tags. Two example videos and corresponding annotations from Youtube-8M are illustrated in Figure 1.

The original dataset is divided into training, validation and test subsets with 70%, 20% and 10% of videos, respectively. In this work we keep around 20K videos for the validation, the remaining samples from the original training and validation subsets are used for training. This choice was made to obtain a larger training set and to decrease the validation time. We have noticed that the performance on our validation set was comparable (0.2%-0.3% higher) to the test performance evaluated on the Kaggle platform. As we have no access to the test labels, all results in this section are reported for our validation set. We report evaluation using the Global Average Precision (GAP) metric at top 20 as used in the Youtube-8M Kaggle competition.

Model evaluation. We evaluate the performance of individual models in Table 1. To enable a fair comparison, all pooled representations have the same size of 1024 dimensions. The "Gated" versions for the clustering-based pooling methods include CG layers as described in Section 5. Using CG layers together with GRU and LSTM has decreased performance in our experiments.

From Table 1 we can observe a significant increase of performance provided by all learnt aggregation schemes compared to the Average pooling baselines. Interestingly, the NetVLAD and NetFV representations based on the temporally-disordered feature pooling outperforms the temporal models (GRU and LSTM). Finally, we can note a consistent increase in performance provided by the CG for all clustering-based pooling methods.

Context Gating. Table 2 presents an ablation study evaluating the effect of Context Gating on the NetVLAD aggregation with 128 clusters. The addition of CG layers in the feature pooling and classification modules gives a significant increase in GAP. We have observed a similar behavior for NetVLAD with 256 clusters. We also experimented with replacing the Context Gating by the GLU [START_REF] Dauphin | Language modeling with gated convolutional networks[END_REF] after pooling. To make the comparison fair, we added a Context Gating layer just after the MoE. Despite being less complex than GLU, we observe that CG also performs better. We note that the improvement of 0.8% provided by CG is similar to the improvement of the best non-gated model (NetVLAD) over LSTM in Table 1.

Video-Audio fusion. In addition to the late fusion of audio and video streams (Late Concat) described in Section 3, we have also experimented with a simple concatenation of original audio and video features into a single vector, followed by the pooling and classification modules in a "single stream manner" (Early Concat). Results in Table 3 illustrate the effect of the two fusion schemes for different pooling methods. The two-stream audio-visual architecture with the late fusion results in improved performance for the clustering-based pooling methods (NetVLAD and NetFV). On the other hand, the early fusion scheme seems to work better for GRU and LSTM aggregations. We have also experimented with replacing the concatenation fusion of audio-video features by their outer product. We found this did not work as well as concatenation mainly due to the high dimensionality of the resulting output. To alleviate this issue, we tried to reduce the output dimension using the multi-modal compact bilinear pooling approach [START_REF] Gao | Compact bilinear pooling[END_REF] but found the resulting models underfitting the data.

Ensembling

In this section we explore the complementarity of different models and consider their combination through ensembling. Our ensemble consists of several independently trained models. The ensembling averages label prediction scores of selected models. We have observed the increased effect of ensembling when combining diverse models. The ensemble did not bring much when combining best but similar models. To choose models, we follow a simple greedy approach: we start with the best performing model and choose the next model by maximizing the GAP of the ensemble on the validation set. Our final ensemble used in the Youtube 8M challenge contains 25 models. From results in Figure 3 we observe that most of the improvements are obtained by ensembling the first seven models. A seven models ensemble is enough to reach the first place with a GAP on the private test set of 84.698. These seven models correspond to: Gated NetVLAD (256 clusters), Gated NetFV (128 clusters), Gated Soft-DBoW (4096 Clusters), Soft-DBoW (8000 Clusters), Gated NetRVLAD (256 Clusters), GRU (2 layers, hidden size: 1200) and LSTM (2 layers, hidden size: 1024). Our code to reproduce this ensemble is available from https://github. com/antoine77340/Youtube-8M-WILLOW. To obtain more diverse models for the final 25 ensemble, we also added all the non-Gated models, varied the number of clusters or varied the size of the pooled representation.

Conclusions

We have addressed the problem of large-scale video tagging and explored trainable variants of classical pooling methods (BoW, VLAD, FV) for the temporal aggregation of audio and visual features. In this context we have observed NetVLAD, NetFV and Soft-DBoW to outperform more common temporal models such as LSTM and GRU. We have also introduced the Context Gating mechanism and have shown its benefit for the trainable versions of BoW, VLAD and FV. The ensemble of our individual models have been shown to improve the performance further, enabling The performance (GAP) of video tagging in our validation set over the number of models combined in the ensemble. The GAP scores for the public and private test sets of the Youtube 8M challenge are approximatively 0.2% lower than on our validation set. our method to win the Youtube 8M Large-Scale Video Understanding Kaggle challenge. Our TensorFlow toolbox LOUPE is available for download from [28] and includes implementations of the Context Gating as well as learnable pooling modules used in our work.

  Groundtruth: Car -Vehicle -Sport Utility Vehicle -Dacia Duster -Renault -4-Wheel Drive Groundtruth: Tree-Christmas Tree -Christmas Decoration -Christmas Car -Vehicle -Sport Utility Vehicle -Dacia Duster -Fiat Automobiles Volkswagen Beetles Top 6 scores: Top 6 scores: Christmas -Christmas decoration -Origami -Paper -Tree Christmas Tree

Figure 1 :

 1 Figure 1: Two example videos from the Youtube-8M V2 dataset together with the ground truth and predicted labels. Predictions are color-coded as (green): correct, (orange): correct but missing in the ground truth, (red) incorrect.

  Figure3: The performance (GAP) of video tagging in our validation set over the number of models combined in the ensemble. The GAP scores for the public and private test sets of the Youtube 8M challenge are approximatively 0.2% lower than on our validation set.

Table 1 :

 1 Performance comparison for individual aggregation schemes. Clustering-based methods are compared with and without Context Gating.

	GAP

Table 2 :

 2 Evaluation of Context Gating for the NetVLADbased architecture with 128 clusters.

	Method	Early Concat Late Concat
	NetVLAD	81.9%	82.4%
	NetFV	81.2%	82.2%
	GRU	82.2%	82.1%
	LSTM	81.7%	81.1%

Table 3 :

 3 Evaluation of audio-video fusion methods (Early and Late Concat).

https://www.tensorflow.org/tutorials/image_ recognition

Acknowledgments

The authors would like to thanks Jean-Baptiste Alayrac and Relja Arandjelović for valuable discussions. Google for providing their Youtube-8M Tensorflow Starter Code. This work has also been partly supported by ERC grants ACTIVIA (no. 307574) and LEAP (no. 336845), CIFAR Learning in Machines & Brains program, ESIF, OP Research, development and education Project IMPACT No. CZ.02.1.01/0.0/0.0/15 003/0000468 and a Google Research Award.


Sandra Kossi

Sabrina Dadoun

Guillaume Geri

Aurore Hermet

Bruno Fautrel

Maxime Dougados

Laure Gossec
email: laure.gossec@aphp.fr

Primary inefficacy of TNF inhibitors in patients with axial spondyloarthritis: a long-term

Keywords: Spondyloarthritis, widespread pain syndrome, diagnosis, classification criteria, TNF inhibitors

INTRODUCTION

TNF inhibitors (TNFi) are extremely effective in axial spondyloarthritis (axSpA). Many studies have shown major improvement in SpA activity after TNFi therapy and currently, it is the only drug therapy approved for axSpA patients with insufficient response to non-steroidal anti-inflammatory drug (NSAIDs) [START_REF] Braun | First update of the international ASAS consensus statement for the use of anti-TNF agents in patients with ankylosing spondylitis[END_REF] However, some patients are non-responders: primary inefficacy, i.e., an initial lack of treatment response appears infrequent in axSpA with rates around 5-15% [START_REF] Lie | Effectiveness of switching between TNF inhibitors in ankylosing spondylitis: data from the NOR-DMARD register[END_REF][START_REF] Dadoun | Switching between tumour necrosis factor blockers in spondyloarthritis: a retrospective monocentre study of 222 patients[END_REF][START_REF] Coates | Real life experience confirms sustained response to long-term biologics and switching in ankylosing spondylitis[END_REF][START_REF] Paccou | Efficacy in current practice of switching between anti-tumour necrosis factor-α agents in spondyloarthropathies[END_REF].

The diagnosis of SpA is difficult and is based on a body of clinical arguments associated with laboratory tests and radiological signs [START_REF] Amor | Criteria of the classification of spondylarthropathies[END_REF][START_REF] Moll | New criteria for the diagnosis of ankylosing spondylitis[END_REF][START_REF] Rudwaleit | The Assessment of SpondyloArthritis International Society classification criteria for peripheral spondyloarthritis and for spondyloarthritis in general[END_REF]; several of the clinical criteria contributing to the diagnosis of axSpA, such as enthesitis pain, lack specificity [START_REF] Amor | Criteria of the classification of spondylarthropathies[END_REF]. Furthermore, some frequent diseases or conditions such as osteoarthritis, widespread pain syndrome and depression could interfere either with the activity of SpA (falsely heightened disease activity) or with the response to TNFi (falsely heightened inefficacy) [START_REF] Azevedo | Occurrence of fibromyalgia in patients with ankylosing spondylitis[END_REF][START_REF] Salaffi | Fibromyalgia in patients with axial spondyloarthritis: epidemiological profile and effect on measures of disease activity[END_REF][START_REF] Aloush | Fibromyalgia in women with ankylosing spondylitis[END_REF]. Thus, considering the rarity of primary inefficacy and the difficulty to diagnose axSpA, it seemed useful to explore the cases of primary inefficacy to TNFi in axSpA: who are these patients who have primary inefficacy? What are their demographic and disease characteristics and their final diagnosis? Do they have comorbidities? Is a second course of TNFi of interest in these patients?

The objective of the present study was to describe axSpA patients with primary inefficacy after the first TNFi, 5 to 10 years after their prescription, in terms of diagnosis, comorbidities and current management.

PATIENTS AND METHODS

Study design

Systematic retrospective and prospective study conducted in one tertiary care centre in Paris, France.

Population

All patients with a diagnosis of axSpA according to an expert rheumatologist Amor criteria and receiving a first TNFi in the context of usual practice between 2004 and 2009 in the centre were analysed. The available information was retrospectively collected from the medical files [START_REF] Dadoun | Switching between tumour necrosis factor blockers in spondyloarthritis: a retrospective monocentre study of 222 patients[END_REF]. For the present study, the focus was on those patients who had primary inefficacy to this first TNFi. Primary inefficacy was defined here as treatment interruption 3-4 months after treatment onset, with a rheumatologist assessment in the medical file of lack of efficacy as reason for drug interruption. The rheumatologist reason to stop the treatment was based on a mixture of efficacy (assessed by BASDAI) as well as patient preference, in the context of usual practice.

Data collected at baseline

At baseline, demographic and disease (type of disease, presence of extra articular signs, HLAB27 status, the presence of imaging sign and of elevated acute phase reactants in particular C-Reactive protein (CRP) characteristics were collected .

Long-term follow up (prospective phase)

In 2013-2014, patients with primary inefficacy were contacted by letter and seen in outpatient clinic if possible for clinical examination and medical interview. If this was not possible, they were contacted by telephone and asked to fill in a questionnaire. If the patients did not answer the letter or the 2 reminders, their general practitioner was contacted to either ask the patient to participate to the study, or give us information on the patient status.

Patients gave their consent for the use of the collected data and the study was conducted in accordance with the Declaration of Helsinski so no separate ethical approval was required Data collected at follow up Disease characteristics were collected including the type of disease (axial only, or associated with enthesitic or peripheral arthritis), the presence of extraarticular signs, the HLAB27 status, the presence of imaging signs (sacroiliitis, syndesmophytes on the most recent available imaging), and of elevated acute phase reactants in particular C-Reactive Protein (CRP). AxSpA disease activity was evaluated through an investigator global visual analog scale (VAS), the BASDAI (Bath Ankylosing Spondylitis Disease Activity Index) and the BASFI (Bath Ankylosing Spondylitis Functional Index) as well as a full clinical evaluation where possible [START_REF] Garrett | A new approach to defining disease status in ankylosing spondylitis: the Bath Ankylosing Spondylitis Disease Activity Index[END_REF][START_REF] Calin | A new approach to defining functional ability in ankylosing spondylitis: the development of the Bath Ankylosing Spondylitis Functional Index[END_REF]. ASDAS was not used in 2014 because CRP was not always available. Current treatment was collected including NSAIDs intake, TNFi and type and treatment maintenance.

The association with widespread pain syndrome was assessed with the FIRST questionnaire; this validated questionnaire comprises 6 questions and a score≥5/6 is strongly suggestive of widespread pain syndrome [START_REF] Perrot | Development and validation of the Fibromyalgia Rapid Screening Tool (FiRST)[END_REF]. Concomitant spine or lowerlimb osteoarthritis and concomitant or past depression were assessed by the physician (SK) during the visit, or by simple questions by phone and/or in the questionnaire. These data were confirmed where possible in the medical file.

The final diagnosis, i.e., confirmation of axSpA diagnosis or not, was evaluated both by applying the Assessment of SpondyloArthritis (ASAS) classification criteria for axSpA [START_REF] Garrett | A new approach to defining disease status in ankylosing spondylitis: the Bath Ankylosing Spondylitis Disease Activity Index[END_REF] and through the rheumatologist's opinion, based on the full evaluation including clinical assessment, imaging and biological results when available.

The fulfillment of the ASAS classification criteria for axSpA was also checked.

Statistical analysis

Descriptive data were summarised using means and standard deviations (SD) comparisons were performed by parametric tests. For patients who had received a second TNFi, the survival rate at one year was assessed using Kaplan Meier survival curves on SAS version 9.2.

RESULTS

SpA patients with primary inefficacy

Of 222 axSpA patients who received a first TNFi between 2004 and 2008, 27 (12%) interrupted the treatment for primary inefficacy (Table 1) [START_REF] Moll | New criteria for the diagnosis of ankylosing spondylitis[END_REF]. When compared to the 195 patients not classified as primary inefficacy, these patients were more often female (48% vs 27 % in patients without primary inefficacy, p= 0.04), were older at initiation of the TNFi (45 vs 39 years, p=0.04), had higher BASFI (68 vs 42, p=0.03) and less frequently an increased CRP (abnormal value according to the laboratory limits: 33% vs 63%, p=0.02). Among the 27 patients with primary inefficacy, fifteen patients (56%) carried the HLA B27 and 18 (67%) had radiographic sacroillitis. There was no assessment of comorbidities in patients who did not have primary inefficacy

Long term follow up

Among these 27 patients with primary inefficacy, twenty five (92%) were re-evaluated after 5 to 10 years (mean follow-up, 6.0±3.1 years); 9 with a full clinic exam, 5 through a telephone interview, 5 through a self-reported questionnaire, and 6 were re-evaluated only through the medical file. One patient refused to participate in the study and 1 patient was lost to follow up. Medical files were used to complement the findings for all patients and for 5 patients, the private practice physician also gave information. Among the 25 patients who were followed-up, thirteen (52%) were women, and mean age at the time of their follow-up was 52.6±14.6 years (Table 2).

At the long term follow up, all the patients still presented symptoms and back pain although the level of symptoms was moderate: the mean BASDAI was 42±21.

Concerning their current treatment, nine (36%) were treated with TNFi and 9 (36%) were taking NSAIDs. The other patients were prescribed analgesic drugs and/or nonpharmacological measures.

Comorbidities

Five (20%) patients had widespread pain syndrome according to the FIRST questionnaire; all were female. Ten (40%) patients had osteoarthritis of the lowerlimb peripheral joints or of the spine; 8 (32%) patients had a self-declared diagnosis of depression of whom 3 (12%) were taking anti-depressant drugs. Overall, eighteen (72%) patients had at least one of these 3 comorbidities. These patients had the following characteristics: 9 (36%) carried HLA B27, 12 (48%) had radiographic sacroiliitis, 10 (40%) had an increased CRP, 7 (28%) had a family history of spondyloarthritis and 13 (52%) were considered good responders to NSAIDs.

TNFi after primary inefficacy

In all, 16 (64% of 25) patients switched to another TNFi: 9 (36%) received 2 TNFi and 7 (28%) received 3 or more prescriptions of TNFi. Among the patients who had at least a second TNFi, the treatment was considered efficacious for 9 (56% of 16) patients whereas 7 presented with primary inefficacy to the second TNFi. The retention rate of the second TNFi at one year was 50%. At the time of follow up, 9 (36%) patients were still prescribed a TNFi. Note that overall (patients with primary inefficacy and without), 111 (57%) patients stopped the first TNF inhibitor over duration of follow up with a mean follow up of 29 months (SD: 20.1).

Final diagnosis

In all, 21 (84%) patients satisfied the ASAS classification criteria for axSpA [START_REF] Garrett | A new approach to defining disease status in ankylosing spondylitis: the Bath Ankylosing Spondylitis Disease Activity Index[END_REF]. Furthermore, the diagnosis of axSpA was confirmed according to the rheumatologist's opinion for 20 (80%) patients. Seventeen (68%) patients fulfilled both conditions.

DISCUSSION

This study confirms the infrequent nature of primary inefficacy of TNFi in axSpA (12%) and shows that most of the patients with primary inefficacy to their first TNFi had confirmed axSpA, but also that most of them had comorbidities that could affect axSpA evaluation. This is important because practitioners might consider that primary inefficacy to TNFi leads to reconsidering the diagnosis of (i.e., the notion of a TNFi prescription used as diagnostic test). We suggest here that primary inefficacy should not be considered as equivalent to a diagnostic error, and that a second prescription of TNFi may be of use in such patients, although painful comorbidities should certainly be screened for and taken into account.

This study has strength and weaknesses. We performed a systematic assessment of all patients receiving a TNFi in a single centre over a defined period of time, allowing a true assessment of the frequency of primary inefficacy. We also managed to reassess almost all the patients with initial primary inefficacy, with a long-term followup (mean, 6 years) which allows true reconsideration of the evolution and diagnosis of the patients. Wherever possible, patients were assessed fully through clinical examination, history-taking and imaging at follow up. No such data was available at baseline,in particular there was no assessment of comorbidities. However, the number of patients assessed was low, due to the relative rarity of primary inefficacy. Not all patients accepted to come into the clinic for a full physical assessment; in fact, for 6 patients we could only use the hospital medical file which is a limitation to the systematic character of the study. And finally, the definition used here for primary inefficacy, i.e., an interruption of the TNFi therapy in the 3-4 months following its introduction, with the physician putting as reason primary inefficacy can be discussed. Indeed, 3 months may be a little too short to properly assess the efficacy of TNFi; and a quantitative assessment such as the BASDAI would have been more optimal [START_REF] Rudwaleit | Prediction of a major clinical response (BASDAI 50) to tumour necrosis factor alpha blockers in ankylosing spondylitis[END_REF]. However, this is a clinical practice, real-life assessment and we believe that the patients who did interrupt their treatment at 3 months were indeed in a situation of primary inefficacy. This study showed that patients with axSpA who had primary inefficacy had slightly different clinical characteristics and above all, much comorbidity such as widespread pain syndrome, depression or osteoarthritis. Widespread pain syndrome is not rare in SpA and in particular in the absence of radiological signs of SpA, it raises the question of non-radiographic forms of axSpA [START_REF] Erbil | Nonradiographic axial spondyloarthritis background and confounding factors of this new terminology: an appraisal[END_REF]. The present study does not give new data regarding non radiographic axSpA but does confirm widespread pain syndrome should be considered when physicians are faced with primary inefficacy in axSpA. The FIRST questionnaire used in the present study is a user-friendly questionnaire for clinical practice, which may be of use for the diagnosis of widespread pain syndrome if its performances are confirmed [START_REF] Perrot | Development and validation of the Fibromyalgia Rapid Screening Tool (FiRST)[END_REF].

If the association of SpA and depressive syndrome is known [START_REF] Meesters | The risk for depression in patients with ankylosing spondylitis: a population-based cohort study[END_REF], this study reveals that depression could explain therapeutic failures in axSpA and may support the concept of systematically screening for depression. However, simple-to-use tools to diagnose depression in the setting of a rheumatology clinic, are currently lacking [START_REF] Baillet | EULAR points to consider for reporting, screening and preventing selected comorbidities in chronic inflammatory rheumatic diseases in clinical practice[END_REF].

The present results suggest the importance of a global approach to manage these patients. Widespread pain syndrome, osteoarthritis and depression are very common comorbidities and should systematically be screened and treated, for a better therapeutic management. It would be interesting to do other studies to compare the link between comorbidities and response to TNFi.

The present study indicated a switch to a second TNFi may be worthwhile, for patients who show lack of response to a first TNFi. Indeed, in the present study, sixteen patients received a second TNFi and importantly, the treatment was considered effective in 9 (56%) patients. Although previous studies had indicated the efficacy of TNFi switches in axSpA, this is the first study to specifically look at patients with primary inefficacy to a first TNFi [START_REF] Lie | Effectiveness of switching between TNF inhibitors in ankylosing spondylitis: data from the NOR-DMARD register[END_REF][START_REF] Dadoun | Switching between tumour necrosis factor blockers in spondyloarthritis: a retrospective monocentre study of 222 patients[END_REF][START_REF] Coates | Real life experience confirms sustained response to long-term biologics and switching in ankylosing spondylitis[END_REF][START_REF] Paccou | Efficacy in current practice of switching between anti-tumour necrosis factor-α agents in spondyloarthropathies[END_REF].

The fact that patients who underwent primary inefficacy to their first TNFi often have a confirmed diagnosis of axSpA confirms the TNF pathway is not the only one involved in these pathologies [START_REF] Smith | The interleukin-23/interleukin-17 axis in spondyloarthritis pathogenesis: Th17 and beyond[END_REF], explaining why some patients do not respond to TNFi therapy. Data regarding the efficacy of new therapeutics developed against the IL17/IL23 pathway in these patients, would be of interest [START_REF] Smith | The interleukin-23/interleukin-17 axis in spondyloarthritis pathogenesis: Th17 and beyond[END_REF].

In conclusion, SpA, like other chronic inflammatory diseases, justifies a holistic approach with appropriate assessment and management of potential comorbidities.

Primary inefficacy of a first TNFi should not bar a second TNFi prescription but a careful assessment of potentially painful comorbid conditions is useful in such a situation.

Funding: This study was funded by the Assistance Publique des Hôpitaux de Paris (AP-HP).