Sandeep Singh Sengar
email: sandeepsingh.s@srmap.edu.in

Motion segmentation based on structure-texture decomposition and improved three frame differencing

Keywords: Structure-texture decomposition, motion segmentation, frame difference, morphology, block

Motion segmentation from the video datasets has several important applications like traffic monitoring, action recognition, visual object tracking, and video surveillance. The proposed technique combines the structure-texture decomposition and the improved three frames differencing for motion segmentation. First, the Osher and Vese approach is employed to decompose the video frame into two components, viz., structure and texture/noise. Now, to eliminate the noise, only the structure components are employed for further steps. Subsequently, the difference between (i) the current frame and the previous frame as well as (ii) the current frame and the next frame are estimated. Next, both the difference frames are combined using pixel-wise maximum operation. Each combined difference frame is then partitioned into non-overlapping blocks, and the intensity sum as well as median of each block is computed. Successively, target objects are detected with the help of threshold and intensity median. Finally, post-processing in the form of morphology operation and connected component analysis is carried out to accurately find the foreground. Our technique has been formulated, implemented and tested on publicly available standard benchmark datasets and it is proved from performance analysis that our technique exhibit efficient outcomes than existing approaches.

Introduction

Motion segmentation is one of the basic and critical approaches of computer vision system. Motion segmentation technique is motivated by the assumption that foreground objects commonly involve changes in the intensity between successive frames. This approach is mandatory in a real environment and should be applicable to difficult circumstances, like background clutter, fake motion, and changes due to noise and illumination deviation. All of these situations lead to the false region detection for the moving targets.

Currently, background subtraction [START_REF] Elharrouss | Motion detection based on the combining of the background subtraction and the structure-texture decomposition[END_REF], optical flow [START_REF] Sengar | Detection of moving objects based on enhancement of optical flow[END_REF] and frame differencing [START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF] are commonly used techniques for motion segmentation. The background subtraction technique depends on the accuracy of generated background; if the background is not SRM University-AP, Amaravati.

accurate then there will be large segmentation error. The optical flow technique is complex and there are high amount of noise in detected result due to involvement of differential operations in it. On the other side, frame differencing is a simple and effective technique to segmenting the moving objects by calculating the pixel-wise difference between successive frames. However, there are two most common problems in two frames differencing viz., (i) background clutter (ii) foreground aperture. To reduce these problems, Sengar et al. [START_REF] Sengar | A novel method for moving object detection based on block based frame differencing[END_REF] proposed an enhancement of frame differencing technique. This is a straightforward method, which detects the moving objects by employing block based three frame differencing. However, detection accuracy of this method is low for some frames of video data due to the randomly generated noise. To improve the detection accuracy, Sengar et al. [START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF] proposed another approach for motion segmentation using block based optical flow. However, processing time of this work is high due to the local processing. In order to improve the detection accuracy, reduce the false positives and processing time, the structure-texture decomposition and three frame differencing based motion detection approach is proposed. The proposed technique first extracts the frames from the video datasets, followed by color to gray-scale processing. Next, the structure component of the gray-scale video frame is extracted using structure-texture decomposition technique so as to remove the texture/noise. Subsequently, block based improved three frame differencing algorithm is employed on extracted structure component for motion segmentation. Finally, post-processing steps using mathematical morphology operation and connected component analysis are performed to accurately detect the foreground. Our method is simple and has considerably low computational cost. We have tested the proposed method on different publically available benchmark video datasets and also compared with existing techniques. Our approach has higher detection accuracy than widely used recent techniques.

As discussed earlier, there are certain problems available in existing motion segmentation techniques. Here, in this approach we have reduced those. It is an important technique for visual surveillance, traffic monitoring and other several related applications. This scheme can be the initial step for visual object tracking, where future position of target can be predicted after detecting it in previous video frames.

The organization of remaining part of this paper is as follows. The survey on related methodologies are presented in Section 2. The proposed motion segmentation method and it's implementation details are provided in Sections 3 and 4 respectively. Experimental results with the qualitative as well as quantitative evaluations are shown in section 5. Finally, conclusion of our technique is provided in Section 6.

Related work

Several schemes for motion segmentation have been presented by image processing and computer vision groups to deal with challenges like ghosting, illumination variations, object pose variation, occlusion and out-of-plane rotation [START_REF] Sahoo | A fast valley-based segmentation for detection of slowly moving objects[END_REF][START_REF] Bouwmans | Decomposition into lowrank plus additive matrices for background/foreground separation: A review for a comparative evaluation with a large-scale dataset[END_REF][START_REF] Sengar | Moving object tracking using laplacian-dct based perceptual hash[END_REF]. One of the methods for textured image segmentation has been presented by Sandberg et al. [9]. This scheme has employed active contour model within Gabor filter environment. Elharrouss et al. [START_REF] Elharrouss | Motion detection based on the combining of the background subtraction and the structure-texture decomposition[END_REF] proposed an approach for motion detection using the structure-texture decomposition and the background subtraction. Block-wise low rank texture character-ization is employed for cartoon-texture image decomposition model [START_REF] Ono | Cartoon-texture image decomposition using blockwise low-rank texture characterization[END_REF]. This model works like a convex optimization problem. Surface normals and structure-texture decomposition are used for intrinsic image decomposition [START_REF] Jeon | Intrinsic image decomposition using structure-texture separation and surface normals[END_REF]. High quality outcomes are produced by this approach with less number of constraints. Wells et al. [START_REF] Wells | Adaptive segmentation of mri data[END_REF] presented a work for image decomposition in the context of classification and adaptive segmentation with the help of different statistical techniques. Malgouyres [START_REF] Malgouyres | Mathematical analysis of a model which combines total variation and wavelet for image restoration[END_REF][START_REF] Malgouyres | Combining total variation and wavelet packet approaches for image deblurring[END_REF], Candès et al. [START_REF] Candès | New multiscale transforms, minimum total variation synthesis: Applications to edge-preserving image reconstruction[END_REF] have employed total variation minimization framework in a wavelet domain for restoration of textured images. Casadei et al. [START_REF] Casadei | Boundary detection in piecewise homogeneous textured images[END_REF] and Zhu et al. [START_REF] Zhu | Filters, random fields and maximum entropy (frame): Towards a unified theory for texture modeling[END_REF] have presented a texture modeling with the help of statistical techniques.

Halidou et al. [START_REF] Halidou | Fast pedestrian detection based on region of interest and multi-block local binary pattern descriptors[END_REF] proposed an approach for pedestrian detection by employing multi-block local binary pattern descriptors and region of interest (ROI). Here, three frame differencing and optical flow techniques are used to compute ROI. Neural network based classification approach and frame differencing technique are employed to detect moving objects in the indoor environments [START_REF] Foresti | Detecting moving people in video streams[END_REF]. In this technique, robust classification is achieved using finite state automation. Caballero et al. [START_REF] Caballero | Optical flow or image subtraction in human detection from infrared camera on mobile robot[END_REF] presented a technique for human detection using frame differencing and optical flow. This technique is proposed for an infrared camera mounted on a mobile robot. The non-pyramidal based Lucas-Kanade and the frame differencing techniques are employed to detect target objects by using thermal signatures [START_REF] Bouguet | Pyramidal implementation of the affine Lucas kanade feature tracker description of the algorithm[END_REF]. A weighted mixture of Gaussians based adaptive background method is proposed for real time tracking [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF]. However, this technique is not good for non-static backgrounds and sudden illumination changes. The W4 and histogram based frame differencing approaches are used to detect the moving objects [START_REF] Sengar | Moving object detection based on frame difference and w4[END_REF]. Maddalena et al. [START_REF] Maddalena | The SOBS algorithm: what are the limits?[END_REF] presented a self-organizing approach for background subtraction. In this approach, the background of the frame is learned with the help of labeling decision and neural network at neighboring pixels. Principal component analysis is used to projecting the high-dimensional data into a lower dimensional subspace for the background modeling [START_REF] Oliver | Bayesian computer vision system for modeling human interactions[END_REF]. This technique is suitable for global illumination variations, but slow moving objects in local illumination variation environment cannot be detected accurately. Sengar et al. [START_REF] Sengar | Foreground detection via background subtraction and improved three-frame differencing[END_REF] proposed two methods, viz., (i) improved three frame differencing and (ii) combined improved three frame differencing and background subtraction to accurately detect the slow/fast moving objects of varying number and size in the illumination variations and background clutter environments. Quaternion is used with modified optical flow technique to increase the accuracy of information of unclear pixels [START_REF] Chen | Quaternion based optical flow estimation for robust object tracking[END_REF]. Schwarz et al. [START_REF] Schwarz | Human skeleton tracking from depth data using geodesic distances and optical flow[END_REF] employed the depth information obtained by a time of flight camera to compute the location of the foreground object. Reduced processing time and accurate results are obtained from the above techniques [START_REF] Chen | Quaternion based optical flow estimation for robust object tracking[END_REF][START_REF] Schwarz | Human skeleton tracking from depth data using geodesic distances and optical flow[END_REF], but these are not suitable for real time environments. Sengar et al. [START_REF] Sengar | Moving object area detection using normalized self adaptive optical flow[END_REF] used the normalized self adaptive optical flow technique for moving object detection. Here, self adaptive window approach and Otsu's [START_REF] Liu | Otsu method and k-means[END_REF][START_REF] Liao | A fast algorithm for level thresholding[END_REF] method are used to select the target object area and to adapt the threshold value respectively. This method accurately detects the foreground, but not suitable for moving objects of small size.

Proposed method

The steps for the proposed motion segmentation technique are as follows:

Algorithm 1: Algorithm of Osher-Vese for structure-texture decomposition [START_REF] Luminita | Modeling textures with total variation minimization and oscillating patterns in image processing[END_REF] Input : Video frame F , No of iteration, Tuning parameter λ > 0,

Step space h > 0 1. U ← F , function f1 = -Fx 2λ|∇V f | , function f2 = - Fy 2λ|∇V f | 2. for t=1 to No of iteration do for i=2 to w-1 do for j=2 to h-1 do K1 ← 1 U (i+1,j)-U (i,j) h 2 + U (i,j+1)-U (i,j-1) 2h 2 K2 ← 1 U (i,j)-U (i-1,j) h 2 + U (i-1,j+1)-U (i-1,j-1) 2h 2 K3 ← 1 U (i+1,j)-U (i-1,j) 2h 2 + U (i,j+1)-U (i,j) h 2 K4 ← 1 U (i+1,j-1)-U (i-1,j-1) 2h 2 + U (i,j)-U (i,j-1) h 2 U (i, j) ← 1 1+ 1 2λh 2 (K 1 +K 2 +K 3 +K 4 ) × F (i, j) -f 1 (i+1,j)-f 1 (i-1,j) 2h - f 2 (i,j+1)-f 2 (i,j-1) 2h + 1 2λh 2 K1U (i + 1, j) + K2U (i -1, j) + K3U (i, j + 1) + K4U (i, j -1) end end 3. V = F -U end
Output: Structure component (U) and texture/noise component (V)

• Extraction of frames from the video data.

• Conversion of video frame to gray-scale, if it is required.

• Separation of structure and texture/noise components.

• Application of three-frame differencing techniques on the structure components. Two difference frames are obtained. • Computation of pixel-wise maximum value between estimated difference frames.

• Division of computed result of previous step in non-overlapping blocks.

• Calculation of intensity sum and intensity median of each block.

• Separation of background and foreground using threshold and block's median.

• Application of post processing steps to accurately detect the foreground objects. The steps for the proposed technique is explained below:

Structure-texture image decomposition

A set of information in the form of structure, texture, and noise are contained by an image and it can be extracted using structure-texture image decomposition method. The inverse estimation problem i.e. structure-texture decomposition approach is mandatory for analyzing and understanding images. This method split a given video frame or image (F) into a structure (bounded variation) component U and a texture and/or noise (oscillating) components V i.e. F = U + V . Many models have been presented for image Fig. 1: Schematic diagram of the proposed technique decomposition, in which one method is proposed by Osher and Vese [START_REF] Luminita | Modeling textures with total variation minimization and oscillating patterns in image processing[END_REF]. This method has shown its efficiency by preserving discontinuities and removing oscillations in the image. The algorithmic steps for Osher-Vese [START_REF] Luminita | Modeling textures with total variation minimization and oscillating patterns in image processing[END_REF] method are shown in Algorithm 1.

Here in the algorithm, w and h denote the width and height of video frame.

Implementation

The sequence of steps is presented below for the implementation of our technique:

Extraction of frames from video data and conversion to gray-scale

Frames from the input video data D : Z × Z × T → Z are extracted. In case, D(x, y, t|x, y ∈ Z, t ∈ T ) = N ∈ Z. The j th frame can be represented as F j (x, y) = D(x, y, t = tj). Here in the next step, we focus on gray-scale frames in place of colored ones, due to the high complexity of three dimensional structures of color frames. Therefore, gray-scale frames can be obtained from colored one by employing [START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF]:

F (x, y) = 0.2126R(x, y) + 0.7152G(x, y) + 0.0722B(x, y) (1)
Here R, G and B represent the red, green and blue channel components of each colored pixel, and scalar multiples represent the weights of these components.

Structure-Texture decomposition

The individual video frame extracted in the preceding step can have noise, and our target is to decrease the effect of noise for detection of accurate moving objects in the subsequent step. For that, structure and texture/noise decomposition technique is employed on individual frame with the help of Algorithm 1 and for the further processing, only the structure component (U ) are being utilized.

Improved three-frame differencing

We employ the improved three frame differencing technique to find the pixel-wise difference of consecutive frames. It can be summarized in the following steps:

• Difference frame Df t,t-1 (x, y) between frames U t-1 (x, y) and U t (x, y) as well as second difference frame Df t+1,t (x, y) between frames U t+1 (x, y) and U t (x, y) are estimated:

Df t,t-1 (x, y) = |U t-1 (x, y) -U t (x, y)| (2) Df t+1,t (x, y) = |U t+1 (x, y) -U t (x, y)| (3)
Here U t-1 (x, y), U t (x, y), and U t+1 (x, y) are the structure components of three successive video frames. • Pixel-wise maximum intensity value between two difference frames is computed:

Df t (x, y) = max[Df t,t-1 (x, y), Df t+1,t (x, y)] (4)

Divide in non-overlapping blocks

Computed difference frame in previous step is partitioned into 8×8 size non-overlapping blocks. Yin [START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar [START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF], and the proposed method.

Detection of moving objects

This process can be summarized in the following steps:

• Intensity sum and median of each non-overlapping block are computed.

• Classification of background and foreground on the basis of block's median (Bm) and threshold (Th) i.e. if the value of block's intensity and block's sum are less than block's median and threshold respectively then it will be background (0), otherwise foreground [START_REF] Elharrouss | Motion detection based on the combining of the background subtraction and the structure-texture decomposition[END_REF].

F G t (x, y) = 0, Df t (x, y) < Bm and sum(B tj ) < T h 1, otherwise (5)
Here F G t , sum(B tj ) denote the foreground objects for the t th frame, sum of intensities of the j th block of the t th frame respectively. row wise, up to down: original frame, ground truth, Sukumaran [START_REF] Sukumaran | Compressed sensing based foreground detection vector for object detection in wireless visual sensor networks[END_REF], Fei [START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF], Yin [START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar [START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF], and the proposed method.

• Motion is detected by combining both the foreground and background pixels.

Post-processing

In morphology, closing is a cascaded combination of first dilation followed by erosion [START_REF] Dougherty | Hands-on morphological image processing[END_REF] and it is employed to fill the gap within the foreground object. Connected component labeling is employed to remove the isolated noisy blobs. Finally, morphological closing operations of square structuring element (SE) of size 3 × 3 (as shown in Eq. 6) followed by connected components labeling are employed to accurately segment the motion.

M V t = (F G t ⊕ SE) SE (6)
The schematic diagram of our technique is shown by Fig. 1. row wise, up to down: original frame, ground truth, Sukumaran [START_REF] Sukumaran | Compressed sensing based foreground detection vector for object detection in wireless visual sensor networks[END_REF], Fei [START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF], Yin [START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar [START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF], and the proposed method.

Experimental results and analysis

We have done qualitative and quantitative evaluation to prove the efficiency of our technique.

Qualitative evaluation

To prove the effectiveness of our method, the proposed and existing techniques have been implemented and tested on standard benchmark video datasets namely: Video 1 (Hall monitoring) [START_REF][END_REF], Video 2 (Walk) [37], and Video 3 (Daria) [START_REF]Action Recognition[END_REF]. The details of used video sequences are given in Table 1. We have iterated the structure-texture decomposition algorithm 100 times. The qualitative results of our algorithm as well as other tested methods on different video sequences are shown in Figs. 234. These above mentioned figures display the original video frames, ground truth, and the motion segmentation results in each video frame by the five different techniques independently.

For Video 1 we have displayed the experimental results on frame number 72, 191, 222, and 265. Similarly the related frame numbers are [START_REF] Sengar | A novel method for moving object detection based on block based frame differencing[END_REF][START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF]81,119), and [START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF][START_REF] Jeon | Intrinsic image decomposition using structure-texture separation and surface normals[END_REF][START_REF] Schwarz | Human skeleton tracking from depth data using geodesic distances and optical flow[END_REF][START_REF]Action Recognition[END_REF] for Video 2 and Video 3 respectively. The original frames and their ground truths are displayed by the first two rows of Figs. 2-4 respectively. Next five rows (from up to down) show the outcomes of Sukumaran [START_REF] Sukumaran | Compressed sensing based foreground detection vector for object detection in wireless visual sensor networks[END_REF], Fei [START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF], Yin [START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar [START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF] and the Proposed technique respectively. It is proved from the aforementioned results that our approach in every case has considerably higher likeness with the ground truth as compared to existing approaches. All the existing approaches mis-classify most of the foreground pixels as background and vice-versa for all the tested video datasets. Hence, target objects are detected with holes and foreground pixels cannot be separated from the background. Furthermore, we have also measured and compared the performance of our technique based on quantitative evaluation in next section.

Quantitative evaluation

Quantitative evaluation and comparison with the help of accuracy and processing time are also done in our work. The experimental results of the proposed and existing techniques are compared with the ground truth (Figs. 234). Accuracy of the results can be computed using following equation:

count = h x=1 w y=1 G t (x, y) (XOR) M V t (x, y) (7)
accuracy = 100 - count f ramesize × 100 (8)
Where, the ground truth and obtained result of t th frame are represented by G t and M V t respectively. The number of mismatched pixels between ground truth and result are denoted by variable count. The results of quantitative evaluation are given in

Conclusion

Different techniques for motion segmentation have been analyzed and we came at conclusion that detection accuracy of these techniques is not suitable because of large amount of noise in the input video dataset. For that, we have proposed a simple and efficient approach for motion segmentation from complex noise scenes. With the help of proposed structure-texture decomposition and improved three frames differencing based scheme, we eliminate almost all the background noises without loss of foreground. Therefore, optimal results are obtained finally. This approach has improvements in comparison with other recent methods in the form of enhanced accuracy and decreased processing time. Experimental results and analyses prove that the proposed algorithm can effectively handle a slow/fast moving objects, camera shake problem and variation of object size with noisy datasets. In future attempt, our technique will be improved for visual object tracking in moving camera environment.

Fig. 2 :

 2 Fig. 2: Results with Video 1 for the frames number (a) 72 th (b) 191 th (c) 222 th (d) 265 th ; row wise, up to down: original frame, ground truth, Sukumaran[START_REF] Sukumaran | Compressed sensing based foreground detection vector for object detection in wireless visual sensor networks[END_REF], Fei[START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF], Yin[START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar[START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF], and the proposed method.

Fig. 3 :

 3 Fig. 3: Results with Video 2 for the frames number (a) 4 th (b) 34 th (c) 81 th (d) 119 th ;row wise, up to down: original frame, ground truth, Sukumaran[START_REF] Sukumaran | Compressed sensing based foreground detection vector for object detection in wireless visual sensor networks[END_REF], Fei[START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF], Yin[START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar[START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF], and the proposed method.

Fig. 4 :

 4 Fig. 4: Results with Video 3 for the frames number (a) 3 th (b) 11 th (c) 28 th (d) 38 th ;row wise, up to down: original frame, ground truth, Sukumaran[START_REF] Sukumaran | Compressed sensing based foreground detection vector for object detection in wireless visual sensor networks[END_REF], Fei[START_REF] Fei | Visual tracking based on improved foreground detection and perceptual hashing[END_REF], Yin[START_REF] Yin | The infrared moving object detection and security detection related algorithms based on w4 and frame difference[END_REF], Sengar[START_REF] Sengar | Motion detection using block based bi-directional optical flow method[END_REF], and the proposed method.

Fig. 5 :Fig. 6 :

 56 Fig. 5: Comparison in the form of detection accuracy

Table 1 :

 1 Description of test video dataset

	Description of video data	Frame size	Number of	Frame	Bit
		in pixels	frames	rate	rate
	Video 1: In this dataset two men are moving in hall
	with briefcase in their hands. The color of the fore-ground and background are not much varied, but	240×352	299	30	24
	there is variation in illumination. (Color)
	Video 2 In this dataset a man is walking slowly from
	one side of the frame to other side. The color of	240×368	132	25	24
	foreground is lighter than the background. (Color)
	Video 3: In this dataset a lady is running. There
	are high amount of noise and illumination variation.	92×144	42	35	24
	(Color)

Table 2 :

 2 Average accuracy of existing and proposed schemes.

	Input video			Scheme
		Sukumaran [33] Fei [3] Yin [34] Sengar [5] Proposed
	Video 1	88.9	89.4	86.7	92.4	96.8
	Video 2	93.2	92.9	92.5	89.9	96.6
	Video 3	90.5	93.3	92.5	93.8	94.6

Table 3 :

 3 Processing time of existing and proposed schemes.Tables2 and 3, these results are computed using different approaches over different datasets. The comparable results are also shown with the help of bar chart in Figs.5 and 6for accuracy and processing time respectively. It has been agreed from accuracy and processing time based quantitative evaluation that the proposed technique gives better results in comparison with other techniques.

	Input video			Scheme
		Sukumaran [33] Fei [3] Yin [34] Sengar [5] Proposed
	Video 1	1.01	0.90	0.87	0.95	0.83
	Video 2	1.05	1.04	0.98	0.99	0.91
	Video 3	0.79	0.75	0.73	0.80	0.65


Zijian Kang

Le Wang

Ziyi Liu

Qilin Zhang

Nanning Zheng

Extracting Action Sensitive Features to Facilitate Weakly-supervised Action Localization

Keywords: Action localization, Weaky-supervised, Two-stream

Weakly-supervised temporal action localization has attracted much attention among researchers in video content analytics, thanks to its relaxed requirements of video-level annotations instead of framelevel labels. However, many current weakly-supervised action localization methods depend heavily on naive feature combination and empirical thresholds to determine temporal action boundaries, which is practically feasible but could still be sub-optimal. Inspired by the momentum term, we propose a general-purpose action recognition criterion that replaces explicit empirical thresholds. Based on such criterion, we analyze different combination of streams and propose the Action Sensitive Extractor (ASE) that produces action sensitive features. Our ASE sets temporal stream as main stream and extends with complementary spatial streams. We build our Action Sensitive Network (ASN) and evaluate on THU-MOS14 and ActivityNet1.2 with different selection method. Our network yields state-of-art performance in both datasets.

Introduction

Temporal action localization (TAL) in untrimmed videos has attracted more and more attention in recent years, many methods [START_REF] Lin | Bsn: Boundary sensitive network for temporal action proposal generation[END_REF][START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF][START_REF] Gudi | Object-extent pooling for weakly supervised single-shot localization[END_REF][START_REF] Gao | Cascaded boundary regression for temporal action detection[END_REF][START_REF] Escorcia | Daps: Deep action proposals for action understanding[END_REF][START_REF] Seybold | Rethinking the faster r-cnn architecture for temporal action localization[END_REF][START_REF] Zhao | Temporal action detection with structured segment networks[END_REF][START_REF] Xu | R-c3d: region convolutional 3d network for temporal activity detection[END_REF] that greatly enhanced performance have been developed. Because labeling action boundary in untrimmed video is expensive, some researchers [START_REF] Paul | W-talc: Weakly-supervised temporal activity localization and classification[END_REF][START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF][START_REF] Shou | Autoloc: Weaklysupervised temporal action localization in untrimmed videos[END_REF][START_REF] Singh | Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization[END_REF][START_REF] Nguyen | Weakly supervised action localization by sparse temporal pooling network[END_REF] proposed to use video-level action annotation to produce snippet level action localization results, which greatly reduced demand for human laboring and yield comparable performance. These studies combine Multiple Instance Learning (MIL) [START_REF] Dietterich | Solving the multiple instance problem with axis-parallel rectangles[END_REF] and attention mechanism [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF][START_REF] Paul | W-talc: Weakly-supervised temporal activity localization and classification[END_REF][START_REF] Nguyen | Weakly supervised action localization by sparse temporal pooling network[END_REF] with Deep Convolutional Neural Networks (DCNN) to produce clip presentations. Then, action detection criterions maps clip presentations to Class Activation Sequence (CAS) which determines which snippet includes action.

However, these weakly-supervised methods share two convenient assumptions that might be too optimistic in the real world. The first assumption is empirical thresholds to determine temporal action boundaries could be obtained in a trivial manner. This implicit assumption could be far from reality, given the diversity of datasets and applications. The second assumption is that straightforward fusion strategies are adequate in weakly-supervised TAL because of the prevailing two-stream networks [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF], where CAS is either generated separately and fused by weighted average [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF][START_REF] Shou | Autoloc: Weaklysupervised temporal action localization in untrimmed videos[END_REF][START_REF] Nguyen | Weakly supervised action localization by sparse temporal pooling network[END_REF], or generated by concatenated features [START_REF] Paul | W-talc: Weakly-supervised temporal activity localization and classification[END_REF] and [START_REF] Shou | Autoloc: Weaklysupervised temporal action localization in untrimmed videos[END_REF] regression methods. With the two-stream network, each stream is independently trained via backpropagation and no interactions happen between streams. These two strategies are straightforward to implemented but we argue that there could possibly be a better alternative.

To address these challenges, we design a general-purpose action detection criterion and an alternative stream fusion strategy. Specifically, we design the action detection criterion based on attention mechanism with momentum-inspired threshold generated in training stage. An analysis in stream combination options results in the proposed Action Sensitive Extractor(ASE) as shown in Fig. 1. Inspired by recent literature in spatial and temporal interaction [START_REF] Sun | Optical flow guided feature: a fast and robust motion representation for video action recognition[END_REF][START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF], the proposed ASE prudently selects action sensitive features between two streams and produces activations. In the ASE, we handle spatial and temporal stream asymmetrically with respect to different sensitivities in actions. With our action detection criterion and the ASE, we build Action Sensitive Network (ASN) for Weakly-supervised TAL.

Main contributions of this paper include (1) a comparative analysis on stream fusion strategies with the proposed Action Sensitive Extractor (ASE), and ( 2) a new flexible action localization criterion which generates high quality CAS. The performance gains of the proposed ASN algorithm are verified on two challenging public datasets.

Related Works

Video Action analyze has been wildly discussed in several years. Most studies focus on action recognition in trimmed videos. Many novel structures have been proposed for videos [START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF][START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF][START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF] based on DCNN neural networks [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Huang | Densely connected convolutional networks[END_REF][START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF]. Two-stream network [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF] was one design which employs RGB images and optical Flow with lateral fusion. Based on two-stream network, temporal segment network (TSN) [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF] was proposed to analyze long-term temporal data. TSN has been used as backbone in different tasks [START_REF] Zhao | Temporal action detection with structured segment networks[END_REF][START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF] with good performance. To further leverage optical Flow, [START_REF] Sun | Optical flow guided feature: a fast and robust motion representation for video action recognition[END_REF] proposed a novel structure for optical Flow. Recent proposed SlowFast Network [START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF] uses two path way to process videos similar to two-stream network. In SlowFast, a fast pathway handles wide temporal motions and a slow pathway handles rich local details.

Action localization has been greatly improved based on video action analyze. Many neural architectures and methods [START_REF] Karpathy | Largescale video classification with convolutional neural networks[END_REF][START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Ma | Learning activity progression in lstms for activity detection and early detection[END_REF][START_REF] Escorcia | Daps: Deep action proposals for action understanding[END_REF] have been developed for supervised learning. However, those studies heavy rely on data annotations of action sequences, which are expensive to acquire. To incorporate more data in training, Sun et al. [START_REF] Sun | Temporal localization of finegrained actions in videos by domain transfer from web images[END_REF] proposed to use web images and video-level annotation to handle TAL. Moreover, hide and seek [START_REF] Singh | Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization[END_REF] discovered how to force network focus on most discriminating part. UntrimmedNet [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF] designed a novel structures that trains high-quality network on untrimmed videos and proposed a method which efficiently selects action segments. UntrimmedNet not only provide a good solution for Localization but is also a good baseline model that generates local representation. Based on extracted feature representation, AutoLoc [START_REF] Shou | Autoloc: Weaklysupervised temporal action localization in untrimmed videos[END_REF] discovered an anchor generation and selection standard on feature sequences. W-TALC [START_REF] Paul | W-talc: Weakly-supervised temporal activity localization and classification[END_REF] and Nguyen et al. [START_REF] Nguyen | Weakly supervised action localization by sparse temporal pooling network[END_REF] discovered feature based networks with different auxiliary loss functions and attention mechanism.

Action Sensitive Network

In this section, our proposed Action Sensitive Network will be introduced. Section 3.1 describes the ASE we proposed. Section 3.2 describes our momentuminspired action detection criterion. In the last section, we introduce details of ASN.

Action Sensitive Extractor

In this section, we propose models to extract action sensitive features. Our proposal is to train a network that maximally leverage action sensitive features in two streams. Because actions are described in moving image in videos, spatial stream with only one frame perception unlikely to recognize action directly. While temporal stream have wider temporal perception and inherently sensitive at motion boundary [START_REF] Sevilla-Lara | On the integration of optical flow and action recognition[END_REF]. Detial analysis can be found in our experiment section. Inspired by SlowFast network [START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF], where spatial (slow) and temporal (fast) are fed into different network architectures with different channels and different temporal perception, we propose our models that treat temporal and spatial asymmetrically.

In general, we use learned action sensitive knowledge(inherit from temporal stream) as main stream. We discover different structures to extract beneficial features to reinforce main stream. We adopt strategy in DenseNet [START_REF] Huang | Densely connected convolutional networks[END_REF] that we concatenate our main stream and reinforce stream together for classification and attention calculation. We call our extraction model Action Sensitive Extractor(ASE), different settings of ASE are shown in Fig. 2. For simplicity, we still use single fully-connected layer for classification or attention branch. ASE with classification and attention branch is referred as ASE model.

Fusion with Temporal Knowledge

To leverage temporal features that are related to actions, we propose to build a network that initialized with temporal features and extended with spatial features. To achieve this goal, we adopt methods from [START_REF] Chen | Net2net: Accelerating learning via knowledge transfer[END_REF], our network on fusion(concatenated) features is initiated with pretrained temporal weights and zero spatial weights. For example, equation 1 shows the classification branch for fusioned features. To inherit knowledge in temporal classifier, we set W t and b t to pretrained temporal weights, while W s and b s are set to 0. We also apply same method to attention branch.

c = W f • x f + b f = W t , W s • x t x s + b t b s (1)
Bottleneck Model To limit overfitting with spatial features, we further study on limiting and distilling spatial features. Different from former studies that enforce loss [START_REF] Paul | W-talc: Weakly-supervised temporal activity localization and classification[END_REF], we simply use special designed network architecture. As a naive attempt, we use a bottleneck layer to extract knowledge from spatial features.

The bottleneck layer includes a dropout, a fully-connected layer and ReLU activation. The features extracted from bottleneck layer are concatenated with temporal features and feed to classification and attention branch. Our bottleneck layer extracts most expressive spatial features that help to identify actions.

Bilinear Bottleneck Model Bottleneck model will remove unnecessary spatial feature but can't introduce interactions. In recent works [START_REF] Wang | Non-local neural networks[END_REF][START_REF] Chen | Aˆ2-nets: Double attention networks[END_REF], bilinear layers are proposed to aggregate spatial-temporal features. To make use of connection between streams, we propose to use bilinear block to aggregate features. In our work, we propose to use two fully-connected bottleneck layer to aggregate features in each stream and use bilinear layer to combine temporal and spatial features. We use 0.5 dropout before bottleneck layer and bilinear layer. We use ReLU activation after fc layers and bilinear layer. The aggregated features are concatenated with temporal features as in bottleneck model. Hidden layer size of bottleneck layer and bilinear layer are set to same for simplicity.

Action Detection Criterion

Here we represent our action detection criterion. In our study, we propose to trim fixed proportion of clips as background, since proportion of background frames is relative stable in each dataset. We set our threshold as quantile of attention values during training, similar to batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], where mean and standard deviation in each batch is recorded and reused, to deal with fluctuation. Quantile level describes desired proportion of clips. Level of quantile defines how much proportion the quantile divide, e.g. quantile at 30% means around 30% of clips in each batch have lower attention than the quantile. For each batch in training, we sort attentions of each clip in this batch and sample a attention value at desired level. Current quantile is updated by a momentum factor according to equation 2. Quantile is fixed during testing.

q t+1 = αq t + (1 -α)q (2)
Our method is simple and cross modality, it is easy to apply our action detection criterion to any attention based localization problem across different settings. Note that the quantile maybe different across datasets, since proportion of background frames maybe different.

Network Details

Having explained key components, now we introduce details of our ASN as shown in Fig. 3. To efficiently look over long video, we break videos into different levels. In bottom level, each frame is represented separately as frame. We use features from our two-stream pretrained DCNN model as representation. Then the middle level, which is clip level. We average our features sampled in short temporal period as clip representation since close frames in videos should be correlated. To distill key knowledge and trim noise, we use Action Sensitive Extractor to extract features and feed to classification and attention branch. The highest level is video-level, which is aggregated by attention mechanism. This level is symmetrical to annotations.

In our study, we discover a setting based on extracted features from Untrimmed-Net [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF]. Following UntrimmedNet [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF], we randomly sample 7 clips for untrimmed videos, 1 clips for video clips. For each clip, 3 frames are sparsely sampled as in TSN [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF] and averaged as clip representation. Two fully-connected layers are used to produce classification and attention respectively. Dropout of 0.5 is used only before classification layer. To fuse clip level activations, we apply

During testing, we use strategy similarly to [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF] and [START_REF] Shou | Autoloc: Weaklysupervised temporal action localization in untrimmed videos[END_REF]. Each clip is aggregated every 15 frames. ASE model produces classification and attention activations for each clip. For video recognition, we soften our attentions by a factor (sets to 3) at first. Then, clips are fused to video representation according to their attentions as in training. For video detection, we generate CAS of size clip number × class number and feed it into selection method. Firstly, we apply softmax operation on clip classification activations. Then, we apply threshold on video-level prediction, clip activations of video unrelated class are set to 0 in CAS. Thirdly, we apply attention level threshold, clips with attentions lower than threshold are set to 0 in CAS. Finally we feed our CAS into selection method to generate action segments. on average [START_REF] Seybold | Rethinking the faster r-cnn architecture for temporal action localization[END_REF]. We train our model on training set and validation set, we test our model on testing set. ActivityNet1.2 [START_REF] Caba Heilbron | Activitynet: A large-scale video benchmark for human activity understanding[END_REF] has 100 classes for both detection and recognition. It is divided into training set, validation set and test set. In ActivityNet, 1.5 instances of actions covers 64% of video on average [START_REF] Seybold | Rethinking the faster r-cnn architecture for temporal action localization[END_REF]. We train our model on training set and test on validation set.

Experiments

Implementation Details

We train our ASN using features extracted by UntrimmedNet pretrained model, which trained on same dataset and subsets as UntrimmedNet. We train our network with Nestrov momentum [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF] of 0.9, weights decay of 0.0005. Batch size is set to 512 for THUMOS14 validation set and 8192 for THUMOS14 training set. Batch size is set to 512 for ActivityNet1.2. On THUMOS14 [START_REF] Jiang | THUMOS challenge: Action recognition with a large number of classes[END_REF], we train 80 epochs jointly on training set and validation set. Our learning rate is set to 0.1 and decay 10 times on 40th and 60th epoch. On ActivityNet1.2 [START_REF] Caba Heilbron | Activitynet: A large-scale video benchmark for human activity understanding[END_REF], we train 160 epochs jointly on training set. Learning rate is set to 0.1 and decay on 80th and 120th epoch.

Ablation study

In this section, we explore our action detection criterion at different levels of quantiles and different model settings. For simplicity and efficiency, we use naive approach in UntrimmedNet [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF] as selection method in ablation study on THU-MOS14. This method only selects consecutive activated frames in CAS. For a selected snippet from clip timestamp n to k + n with label v, confidence scores s are evaluated by video-level activation c v and average activation as shown in Equation 4. Where we use λ = 0.2 in our experiment.

s = 1 n + 1 k+n i=k c i v + λc v (4)

Evaluation of Action Detection Criterion

To demonstrate efficiency of our action detection criterion, we train our network 10 times and record performance on testing set under different quantiles. As baseline, spatial (RGB) and temporal (Flow) models are treated separately. Different levels of quantiles are recorded and tested on localization task. We train our network 10 times and quantiles are recorded at level 10%, 20%, 30% to 90%. To compare with former studies, we also apply our methods on pretrained weights provided by [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF], the quantiles of pretrained models are recorded by running on THUMOS14 validation set. CAS of two-stream model in localization are generated by two steps. First, clip level classifications after softmax are averaged. Second, attention scores of each stream are normalized by each threshold and averaged. Video-level recognition results are set to average of two streams. Results of spatial and temporal model under IoU threshold of 0.5 are shown in Fig. 4. The performance of pretrained model on different quantiles are shown in Fig. 5. For different models, performance peaks locate near 50% quantiles.

During training, we find attention quantiles are fluctuating but performances are generally stable. Notably, spatial performances are worse and more unstable than temporal. We also compare our methods with original UntrimmedNet [START_REF] Wang | Untrimmednets for weakly supervised action recognition and detection[END_REF]. Performances of our best models under different settings are shown in Table 1. Our action detection criterion can achieve high performance with only temporal stream.

Evaluation of Streams

We evaluate different combination of streams as shown in Table 1. We evaluate on spatial (RGB), temporal (Flow), two-stream and fusion stream (concatenated features of RGB and Flow). We also discover attention quality of each stream. Surprisingly, temporal stream yield the best localization performance. Streams with spatial features perform poorly. The bad behavior of spatial related streams may because of trivial details in spatial features cause overfitting. In addition, we analyze attention in each stream. For two-stream model, we fix CAS and apply only temporal or spatial attention to our criterion. We find two-stream with temporal attentions yield high performance similar to temporal stream and two-stream with spatial attentions yield low performance similar to fusion stream.

Our experiment shows differences in action sensitivity between two streams. Combining with temporal and spatial information usually yield higher performance in action recognition but lower in localization. We also find commonly used two-stream or fusion strategies are inefficient in weakly-supervised localization task, which are worse than single temporal stream. In recognition tasks, fusion model has the highest performance because it can access full information, it also proves that our bottleneck structure does restrain information. For localization, bottleneck model and bilinear bottleneck model performs much higher than fusion model. Bilinear bottleneck models perform slightly higher than bottleneck model, which denotes that bilinear layer does improve interaction. High performance of our proposed ASE models shows its ability to extract action sensitive features.

Evaluation of ASE

Experiments on AutoLoc

To evaluate our final Action Sensitive Network, we use AutoLoc [START_REF] Shou | Autoloc: Weaklysupervised temporal action localization in untrimmed videos[END_REF] as selection methods and compare with state-of-art results. AutoLoc incorporate Outer-Inter-Contrastive (OIC) loss that evaluate action snippet accurately. To further adjust performance, we increase weights for outer boundary in OIC as follow:

L OIC = λA o (φ) + A i (φ) (5)
On THUMOS14, we set λ to 2. We also increase boundary inflation rate to 0.35. These settings help AutoLoc select most distinguishable action snippets. We add more offset anchors to AutoLoc and only use AutoLoc as a selection method over CAS. We show performance of our bilinear bottleneck model with feature size 128 and inherit strategy in Table 4. For ActivityNet1.2 [START_REF] Caba Heilbron | Activitynet: A large-scale video benchmark for human activity understanding[END_REF], we set λ to 5 and boundary inflation to 0.7. We use quanile at 10% for ActivityNet1.2. Our results are shown in Table 3. Compare with other weakly-supervised TAL methods, our method have advantage especially under higher IoU and reach state-of-art level in both datasets.

Conclusion

We propose a general action detection criterion which can generate high quality CAS and can apply to different modalities. Based on this threholding method, we analyze performance of different combinations of streams. According to our experiments, spatial and temporal stream contains different information and have different sensitivity in actions. To combine two streams properly, we propose our novel Action Sensitive Network. Two-stream features are treated asymmetry to produce accurate representation without losing sensitivity in actions. We use ASE model to produce clip features and CAS that can be applied to different selection methods. Our network yields state-of-art performance with AutoLoc as selection method. In the future, we can investigate higher level relationship between different streams and apply our method to more modalities.

Fig. 1 .

 1 Fig. 1. Illustration of different strategies to combine two-stream features. (a) Lateral fusion of two-stream features. (b) Concatenating two-stream features for processing. (c) Our action sensitive extractor.

Fig. 2 .

 2 Fig. 2. Data flow of different ASE settings. We set temporal stream as main stream and spatial stream as reinforce stream. Reinforced streams are fed into classification and activation branch then. (a) Fusion Model. (b) Bottleneck Model. (c) Bilinear Bottleneck Model.

Fig. 3 .

 3 Fig. 3. Our full network for action recognition and detection. We use our ASE model to produce frame activations and attentions.Video-level classification activations are optimized with video-level annotations. CAS is generated by action detection criterion. Action segments are selected based on CAS. softmax operation on attentions x from clip 1 to t. The normalized attentions xa i = exp(x a i ) t j=1 exp(x a j ) are used to fuse clip level classifications into video level prediction x c , where x c = t i=1 xa i x c i . Next, we apply softmax operations among each dimension of prediction and optimize with multi-label cross-entropy loss.

4. 1

 1 Dataset THUMOS14 [20] has 101 classes for recognition and 20 classes out of 101 for action detection. THUMOS14 includes training set, validation set and testing set. Training set includes action video clips, validation and testing set includes untrimmed videos. In THUMOS14, 15 instances of actions covers 29% of video

Fig. 4 .

 4 Fig. 4. Localization mAP of flow and rgb model under different quantiles on THU-MOS14. mAP is recorded under 0.5 IoU threshold.

Table 1 .

 1 Comparison with different settings on THUMOS14. We compare localization mAP under common IoU threshold and recognition accuracy. UntrimmedNet use a slightly different recognition strategy.

			Localization (IoU threshold)
	Models	0.3	0.4	0.5	0.6	0.7	Recognition
	Flow pretrained	27.68	21.26	14.87	9.79	5.64	73.93%
	RGB pretrained	15.32	8.76	5.02	2.80	1.35	72.29%
	Two-stream pretrained 28.50	21.06	14.40	8.75	4.78	82.04%
	UntrimmedNet [38]	28.2	21.1	13.7	-	-	*82.2%
	Flow stream	28.67	22.43	16.60	10.42	5.63	74.15%
	RGB stream	15.32	8.76	5.02	2.80	1.35	72.29%
	Fusion stream	20.40	14.50	9.50	5.56	3.03	75.61%
	Two-stream	27.87	20.76	14.46	8.59	4.68	79.95%
	Two-stream (RGB)	21.88	14.73	9.19	4.97	2.29	-
	Two-stream (Flow)	28.57	22.23	16.40	10.04	5.62	-

Table 2 .

 2 We evaluate different ASE model settings. For inherit strategy, we our best flow model as initial weight. For fusion model, bottleneck Compare with different ASE model settings on THUMOS14. and bilinear bottleneck model, we compare training from scratch and inherit strategy with feature size of 64. We compare inherit strategy of feature size of 64 and 128 in bottleneck model and bilinear bottleneck model. Our results are shown in Table 2. Compare with training from scratch, inherit strategy greatly improves recognition and localization except for bottleneck model. For bottleneck model, only localization is slightly improved. This phenomenon may denotes that our bottleneck model has already restrained overfitting. For bottleneck models and bilinear bottleneck models, feature size from 64 to 128 slightly improves performance.

		16
		14						Flow RGB Two-stream
		12
	mAP	8 10
		6
		4
		2
		0
		10%	20%	30%	40%	50%	60%	70%	80%	90%
					levels of quantiles
	Fig. 5. Localization mAP of pretrained model under different quantiles on THU-
	MOS14. mAP is recorded under 0.5 IoU threshold.
					Localization (IoU threshold)
		Models		0.3	0.4	0.5	0.6	0.7	Recognition
		Flow ours		28.67	22.43	16.60	10.42	5.63		74.15%
		Two-stream ours	27.87	20.76	14.46	8.59	4.68		79.95%
	Fusion from scratch	20.40	14.50	9.50	5.56	3.03		75.61%
		Fusion inherit	26.21	19.38	12.72	7.45	4.02		81.39%
	Bottleneck64 inherit	32.73	24.84	17.36	11.12	6.42		78.16%
	Bottleneck64 scratch	29.35	22.61	15.91	9.94	5.39		80.92%
	Bottleneck128 inherit	32.33	25.13	17.60	10.69	5.64		79.10%
	BiBottleneck64 inherit	31.89	25.35	17.74	11.29	6.23		78.20%
	BiBottleneck64 scratch 29.12	22.84	16.27	10.30	5.99		74.73%
	BiBottleneck128 inherit 32.21	25.34	18.16	11.42	6.23		78.57%

model

Table 3 .

 3 Comparison with state-of-art methods on ActivityNet1.2 in terms of action localization mAP under different IoU. We only list weakly-supervised methods. All results in this table are based on UntrimmedNet features. We describe selection methods we used in brackets.

		Localization (IoU threshold)
	Models	0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Avg.
	UntrimmedNet [38] 7.4 6.1 5.2 4.5 3.9 3.2 2.5 1.8 1.2 0.7 3.6
	AutoLoc [30]	27.3 24.9 22.5 19.9 17.5 15.1 13.0 10.0 6.8 3.3 16.0
	W-TALC [26]	37.0 33.5 30.4 25.7 14.6 12.7 10.0 7.0 4.2 1.5 18.0
	ASN (AutoLoc)	29.8 27.1 25.0 23.1 21.2 18.6 16.1 13.1 9.6 4.4 18.8

Table 4 .

 4 Comparison with state-of-art methods on THUMOS14 in terms of action localization mAP under different IoU. All weakly-supervised results are based on UntrimmedNet features. We describe selection methods we used in brackets.

	Localization (IoU threshold)

Acknowledgement

This work was supported partly by National Key R&D Program of China Grant 2017YFA0700800, National Natural Science Foundation of China Grants 61629301 and 61773312, Young Elite Scientists Sponsorship Program by CAST Grant 2018QNRC001.


Antonios Papaleonidas

Elias Pimenidis
email: elias.pimenidis@uwe.ac.uk

Lazaros Iliadis
email: liliadis@civil.duth.gr

Hybrid Data Set Optimization in Recommender Systems Using Fuzzy T-Norms

Keywords: Recommender systems, Evaluation, Explanations, Reproducibility, Fuzzy logic, T-Norms

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The use of recommender systems is very common, especially in applications like e-Commerce and social networks. Products recommendation techniques can reduce the overall searching time of an e-shop user and increase sales. Apart from e-commerce, recommendation technology is also used in various other less known domains such as music recommendation [START_REF] Polatidis | Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations[END_REF][START_REF] Tan | Using rich social media information for music recommendation via hypergraph model[END_REF], books [START_REF] Núñez-Valdéz | Implicit feedback techniques on recommender systems applied to electronic books[END_REF], documents [START_REF] Porcel | A hybrid recommender system for the selective dissemination of research resources in a technology transfer office[END_REF], television programs [START_REF] Barragáns-Martínez | A hybrid content-based and item-based collaborative filtering approach to recommend TV programs enhanced with singular value decomposition[END_REF], people to people recommendation [START_REF] Polatidis | Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations[END_REF], applications in markets [START_REF] Costa-Montenegro | Which App? A recommender system of applications in markets: Implementation of the service for monitoring users' interaction[END_REF], e-learning [START_REF] Bobadilla | Collaborative filtering adapted to recommender systems of e-learning[END_REF] and Web search [START_REF] Mcnally | A case study of collaboration and reputation in social web search[END_REF].

The increasing importance, use and popularity of recommender systems research both in academia and in industry has led to the development of new algorithms and their experimental evaluation. Researchers are mostly focusing in creating more effective algorithms and models by trying to minimize the MAE and RMSE while at the same time they are trying to improve precision and recall of top-N recommendations [START_REF] Jalili | Evaluating Collaborative Filtering Recommender Algorithms: A Survey[END_REF][START_REF] Herlocker | Evaluating collaborative filtering recommender systems[END_REF]. While this is important to do, it should be also noted that the problem of reproducing the results exists and it is considered important [START_REF] Said | Comparative recommender system evaluation[END_REF]. In a previous work of the research team [START_REF] Nikolaos | Reproduction of experiments in recommender systems evaluation based on explanations[END_REF], offline recommender system results were successfully reproduced using an explanation-based approach.

There are different libraries that can be used for developing and testing a recommendation algorithm and include, among others, Recommender101, Apache Mahout, LensKit and MyMediaLite [START_REF] Nikolaos | Reproduction of experiments in recommender systems evaluation based on explanations[END_REF]. Polatidis et al [START_REF] Polatidis | Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations[END_REF], has shown that reproducing the experimental results of an algorithm is very difficult when using a different library because of different settings and parameters that exist between them.

In this paper we are further expanding our previous work by extending Recom-mender101 libraries, modifying the methodology that Recommender101 is selecting users with common ratings and adding all the necessary parameters in the configuration files of Recommender101.

The rest of the paper is organized as follows: Section 2 provides the relevant background and describes related work, section 3 delivers the proposed extension and algorithm of the team's previous method, section 4 describes the experiments, the results and the discussion while conclusions and future work are included in section 5.

2

Related work

Recommenders are often evaluated and compared offline using datasets collected from online platforms [START_REF] Yang | Unbiased offline recommender evaluation for missing-not-at-random implicit feedback[END_REF]. Evaluation can be done by using prediction accuracy or information retrieval metrics. However, the problem arises when in a research output of a new algorithm the source code is not made publicly available or when the exact settings for replicating the code and the experiments are missing [1][19].

Research papers that propose new recommendation algorithms will typically describe the experimental setup, the dataset used, and the framework that was [START_REF] Nikolaos | Reproduction of experiments in recommender systems evaluation based on explanations[END_REF]. A major challenge in recommender system evaluation is that there are many different libraries for evaluating algorithms and the possibility of having one single library or making all the current libraries following a universal or standardized approach is rather impossible [START_REF] Polatidis | Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations[END_REF].

The idea of a unified approach which can facilitate a common reference baseline for recommendation experiments across different frameworks and a set of guidelines to tackle a cross-industry challenge was proposed in [START_REF] Polatidis | Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations[END_REF] and was the trigger point for the proposal and development of a reproducibility framework that combines the ability to reproduce recommendation experiments as well as the support of new algorithms and methodologies.

Reproduction of experiments in recommender systems evaluation based on explanations

In [START_REF] Nikolaos | Reproduction of experiments in recommender systems evaluation based on explanations[END_REF], the problem of reproducibility in recommender systems evaluation was highlighted and the importance of the correct settings and parameters which were used within the library was indicated.

In the proposed approach we:

1. Retrieve information from the configuration file 2. Write the information in the log file along with evaluation result and re-describe the written information.

The settings retrieved from the configuration file are the following and are presented in the same way that are saved in the log file:

1. The configuration parameters and settings can be set at the configuration file recommender101.properties that be found under the conf directory of Recom-mender101. 2. The filename of the dataset is (name of the file goes here). 3. The minimum number of ratings per user to be considered is (number). 4. The minimum number of ratings per considered item is (number). 5. This experiment has used all users OR This experiment has used (number) users. 6. The minimum rating value applied is (number e.g. between 1 to 5). 7. The maximum rating value applied is (number e.g. between 1 to 5). 8. This experiment is based on a (number e.g. 5 or 10) cross fold validation OR this experiment is based on a training/test approach using (number %) for training and (number %) for testing. 9. The number of nearest neighbors used is (number). 10. The algorithm used is (name). 11. The metrics used for this experiment are (This is already implemented in rec-ommender101). 12. The results are (This is already implemented in recommender101).

The initial evaluation results were promising and triggered the idea of extending the configuration file and the necessary libraries of recommender101in order to achieve a better way of selecting the most proper users Y that could produce the best recommendations for users X.

Reproducibility of recommender systems experiments based on explanations

On [START_REF] Nikolaos | Reproduction of experiments in recommender systems evaluation based on explanations[END_REF] extension we modified the source code of Recommender101.java, Recom-mender101Impl.java and DefaultDataLoader.java in order to:

• Enrich the recommender101 configuration file (recommender101.properties) with new parameters that will help us in optimizing the use of data set.

• Improve both accuracy and quality of recommendations by providing tools to select the most proper users Y that could produce the best recommendations for users X • Strengthen our previous work by checking the offline reproduction of recommender system experiment with the use of modified source code and extended configuration file.

Five new parameters have been incorporated in the configuration file of Recom-mender101and also were defined in the source code of the platform. Those parameters are:

1. My_User_count: Takes a value from zero, up to the total number of users found in the dataset. It works in co-operation with the already existing parameter sampleNUsers and it should be grater than sampleNUsers. It indicates the total number of users which will be used in order to select the best matches for sampleNUsers. If the value is <=0 then the modified algorithm is skipped and Recommender101 runs as usual. 2. My_penalty_multiplier: This is a similarity value between two users. The higher the penalty multiplier the lower similarity between users with no identical ratings. For example, assuming that two users i and z have 5 common ratings with the values of the i user {3,4,2,5,1} and for z {3,2,2,5,2} with a penalty multiplier of 1 then the value returned is (5

-1*|3-3|)+(5-1*|4- 2|)+(5-1*|2-2|)+(5-1*|5-5|)+(5-1*|1-2|)=22
and in the case where the penalty multiplier is 2 then the value returned is (5

-2*|3-3|)+(5-2*|4-2|)+(5- 2*|2-2|)+(5-2*|5-5|)+(5-2*|1-2|)=19
, which means less similar users. 3. My_Relat_Function: Since in a dataset many users might have submitted hundreds of ratings and other users very few there might be huge differences when the penalty multiplier is applied. Therefore, to overcome this issue, AVG or SUM can be used as parameters in My_Relat_Function. AVG will normalize the differences and SUM will keep the raw calculated values. 4. My_fuzzy_norm: This parameter represents the equation that calculates the overall similarity value of a user i with all other users. Sigma Count Average [START_REF] Zadeh | Fuzzy logic[END_REF] was used in the basic implementation and in this paper Hamacher and Einstein product are introduced and used. 5. My_fuzzy_mv: After the fuzzy norm is calculated then this is the threshold that is used as a decision-making point for which users are similar and which are not.

By default, Recommender101 system uses a partial set of the available users in the data set in order to calculate top-N recommendations. Already existing sampleNUsers parameter is used to define the size of that partial set of users that will be used. The proposed algorithm uses the similarity between My_User_count number of users in order to select the best ones that will be included in the data set.

Proposed extension and Algorithm

DefaultDataLoader is the class of Recommender101 that is used to read the dataset that is described in the configuration file of the platform. We decided not to modify the recommendation algorithm but to work in DefaultDataLoader class. By doing that the main recommendation algorithm works on the same way and we can apply any type of source dataset manipulation without the need of rechecking the recommendation methodology. The core class of the platform is getting the new, processed, dataset and the main algorithm and the evaluation metrics remain the same.

The steps of the algorithm are the following:

1. Read from the configuration file the My_User_count, thus read the set of users U={U1, U2, U3, …, Ux}, the set of items UMi = {(Ui,M1), (Ui,M2), …,(Ui,My)} and the ratings for each item URi={(M1,R), (M2,R), …,(My,R) }. 2. Find all common items for each user Ux with all other users of the dataset U.

Then, for each user i create a set X with common items, thus creating X 2 sets as shown in equation 1.

∀ (𝑈 𝑖 , 𝑈 𝑧 ∈ 𝑈) , 𝐶 𝑖,𝑧 = 𝑈𝑀 𝑖 ∩ 𝑈𝑀 𝑧 (1)
3. Apply equation 2 to calculate the matching-degree between two users i and z.

In this equation y is the number of common ratings (length of 𝐶 𝑖,𝑧 ), 𝑈 𝑖 𝑀 𝑘 𝑅 is the rating of user I for an item k and 𝑈 𝑧 𝑀 𝑘 𝑅 is the rating of user z for the same item k.

∀ 𝐶 𝑖,𝑧 , 𝑀𝐷 𝑖,𝑧 = ∑(𝑀𝑎𝑥𝑅𝑎𝑡𝑖𝑛𝑔 -𝑃𝑀 * |𝑈 𝑖 𝑀 𝑘 𝑅 -𝑈 𝑧 𝑀 𝑘 𝑅| 𝑦 𝑘=1

). ( 2)

4.
Step 3 returns an X * X table with matching-degree values and this step will return and Min value, a Max value, and Average (avg) value and a standard deviation (SD) value. 5. Calculate variables F and G which will be used for the fuzzification process of the matching-degree for every user combination 𝑈 𝑖 , 𝑧. The calculation of F is shown in equation 3 and of G in equation 4.

𝐹 = { 𝐴𝑣𝑔 -(2 * 𝑆𝐷), 𝑖𝑓 (𝐴𝑣𝑔 -2(𝑆𝐷) ≥ Min) 𝑀𝑖𝑛, 𝑖𝑓 (𝐴𝑣𝑔 -2(𝑆𝐷) < Min) (3)
𝐺 = { 𝐴𝑣𝑔 + (2 * 𝑆𝐷), 𝑖𝑓 (𝐴𝑣𝑔 -2(𝑆𝐷) ≥ Max) 𝑀𝑎𝑥, 𝑖𝑓 (𝐴𝑣𝑔 -2(𝑆𝐷) < Max) (4
7. The final user matching degree is calculated based on the high fuzzy set through Sigma Count average, hybrid Hamacher or Einstein products. 8. For each user the final value is checked and if it is smaller than the My_fuzzy_mv value do not pass this user to the recommendation algorithm. 9. Start Recommender101 based on the settings and algorithms found in the configuration file.

Hybrid Hamacher and Einstein products

Hamacher and Einstein products are two well know Fuzzy T-Norms [START_REF] Iliadis | Computational intelligence and intelligent agents[END_REF] that are widely used in Fuzzy Information Aggregation [START_REF] Wang | Intuitionistic fuzzy geometric aggregation operators based on Einstein operations[END_REF][16] and they are calculated based on the equations 7 (Hamacher product) and 8 (Einstein product).

𝐴 ̅ ∩ 𝐵 ̅ = 𝜇 𝛢 ̅ (𝑥)𝜇 𝐵 ̅ (𝑥) 𝜇 𝛢 ̅ (𝑥) + 𝜇 𝐵 ̅ (𝑥) -𝜇 𝛢 ̅ (𝑥)𝜇 𝐵 ̅ (𝑥) (7)
In both cases membership values equal to Zero will result both products to be equal to zero. In that case relativity of the user will be also zero so in our hybrid version of Hamacher and Einstein products, zero membership values are not used in the calculations. This modification can be adopted due to the existence of custom variable My_Relat_Function which will "promote" users with high number of ratings even if they got some zeros in membership values.

After calculating the desired product for each user, the values are linearly normalized in [0,1] in order to be in match with My_fuzzy_mv threshold.

Experiments and results

The proposed methodology was extensively tested by measuring the performance and the accuracy of the Recommender101 under different configuration schemas. At first part of testing we used the system with the default configuration and without processing the dataset. Four different values of sampleNUsers (100, 400, 800 and 1600) were used.

At the second part of testing we used the system by enabling the source dataset processing with different values for My_User.count and My_fuzzy.norm custom variables. Once again, each case was tested with the same four different values of sampleNUsers as before. The value of My_User.count valiable was set to 300, 1200, 2400, 4600 users accordingly to sampleNUsers. Penalty multiplier was set to 1.7, Fuzzy_mv threshold was set to 0.5 and realativity function was set to AVG. All three alternatives for My_fuzzy.norm, Sigma Count average, hybrid Hamacher or Einstein products, were also used.

In total 16 different configurations were used, and the same number of results were created. In all cases both "FunkSVDRecommender" and "SlopeOneRecommender" algorithms were used. Custom variable's values were set based on the fact that this paper focuses in the importance of source dataset prospecting and not in finding the best values for achieving the maximum accuracy.

MovieLens 1 million dataset [START_REF] Harper | The MovieLens Datasets[END_REF], which consists of 6040 users, 4000 movies and 1 million ratings in a 1-5 scale was used for the experimental evaluation of the method.

All experiment took place on a Windows 10 64bit computer with 64GB ram and an i5-4570 cpu.

Evaluation metrics

The evaluation was based on well-known and widely used metrics like Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE) for error rating, and Precision and Recall for measuring the quality of the top-N recommendations [START_REF] Polatidis | Reproducibility of experiments in recommender systems evaluation 14th International Conference on Artificial Intelligence Applications and Innovations[END_REF][START_REF] Said | Comparative recommender system evaluation[END_REF][START_REF] Nikolaos | Reproduction of experiments in recommender systems evaluation based on explanations[END_REF]. MAE and RMSE are rating error prediction metrics [START_REF] Iliadis | Computational intelligence and intelligent agents[END_REF] and lower values represent better predictions, whereas Precision and Recall are information retrieval metrics and represent the quality of the retrieved recommendations and higher values are better. MAE, RMSE, precision and recall equations are following.

𝑀𝐴𝐸 = 1 𝑛 ∑ |𝑝𝑖 -𝑟𝑖| (9) 𝑛 𝑖=1 𝑅𝑀𝑆𝐸 = √ 1 𝑛 ∑(𝑝 𝑖 -𝑟 𝑖 ) 𝑛 𝑖=1 ( 10
) 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝐶𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑒𝑑 𝑖𝑡𝑒𝑚𝑠 𝑇𝑜𝑡𝑎𝑙 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑒𝑟 𝑖𝑡𝑒𝑚𝑠 (11) 𝑟𝑒𝑐𝑎𝑙𝑙 = 𝐶𝑜𝑟𝑟𝑒𝑐𝑡𝑙𝑦 𝑟𝑒𝑐𝑜𝑚𝑚𝑒𝑛𝑑𝑒𝑑 𝑖𝑡𝑒𝑚𝑠 𝑅𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑖𝑡𝑒𝑚𝑠 ( 12
)
Execution Duration is measured in seconds and was used for performance evaluation of the proposed method.

Experimental results

First four sets of results, table 1, refer to first run of the system with the default Rec-ommender101 configuration, with no dataset process and values of sampleNUsers equal to 100, 400, 800 and 1600.

Discussion

We can see from the tables above that in all the cases that the extended DefaultData-Loader class of Recommender101 was used the recommendation results have been improved. Data sparsity has improved up to 20% (from 0.041 to 0.049) in the experiments where 400 sampleNUsers were provided to Recommender101. MAE has improved by 1.45% to 5.63% and RMSE improved by 1.59% to 4.76%. While checking the quality of recommendations we can see that Precision has improved by 0.68% to 4.63% and Recall by 0.08% to 7,17%. At the same time, we can see that the execution duration of the recommendation algorithm drops up to 40% (from 632sec to 385 sec) while only in few cases it can rise only up to 6% (from 133 sec to 136 sec).

Although that, as mentioned before, the selection of the optimal values for the new custom valuables is not the main scope of this paper we run an extra experiment by using the Hybrid Einstein product as final fuzzification method and setting the fuzzy membership value (FMV) threshold to 0.55 instead of 0.50. We run the experiment only for the combination of 1600/4800 because in that case we saw that improvement was mainly reduced compared to the other three combinations. The following table 8 shows the results of that extra experiment compared with the results with the default run of Recommender101. We can see from table above that by increasing the FMV threshold, all the evaluation metrics are improved, and the sparsity of the dataset is dramatically increased. Of course, in case of a very high FMV threshold system could face the problem of not being able to select the minimum number of users required to run the recommendation algorithm.

Conclusions and future work

In this paper an extended methodology of reproducibility in recommender systems combined with source dataset optimization methods with the use or Recommender101 platform has been presented. In our previous work we have already shown that the reproducibility of results becomes achievable if the correct settings and parameters are used. This paper extends the set of parameters and achieves to increase the recommendation accuracy, to deal with the sparsity problem of big datasets and at the same time to provide the necessary tools and variables to reproduce the results of the experiments. The initial evaluation results are promising and can assist towards this direction and our approach can be straightforwardly implemented by researchers in other libraries. Furthermore, in our future work we aim to include more custom variables in the Rec-ommender101 and to extend the use of Fuzzy Norms in the user selection methodology.

) 6 .

 6 The fuzzification will consider all 𝐶 𝑖,𝑧 values based on two fuzzy sets Low and High using two symmetric semi-trapezoid function as shown in equations 5 and 6. 𝐿𝑜𝑤𝐶 𝑖,𝑧 (𝐶 𝑖,𝑧 ; 𝐹, 𝐺) = { 1, 𝐶 𝑖,𝑧 ≤ 𝐹 𝐺-𝐶 𝑖,𝑧 𝐺-𝐹 , 𝐹 < 𝐶 𝑖,𝑧 < 𝐺 0, 𝐶 𝑖,𝑧 ≥ 𝐺 (5) 𝐻𝑖𝑔ℎ𝐶 𝑖,𝑧 (𝐶 𝑖,𝑧 ; 𝐹, 𝐺) = { 0, 𝐶 𝑖,𝑧 ≤ 𝐹 𝐶 𝑖,𝑧 -𝐹 𝐺-𝐹 , 𝐹 < 𝐶 𝑖,𝑧 < 𝐺 1, 𝐶 𝑖,𝑧 ≥ 𝐺

  𝐴 ̅ ∩ 𝐵 ̅ = 𝜇 𝛢 ̅ (𝑥)𝜇 𝐵 ̅ (𝑥) 2 -[𝜇 𝛢 ̅ (𝑥) + 𝜇 𝐵 ̅ (𝑥) -𝜇 𝛢 ̅ (𝑥)𝜇 𝐵 ̅ (𝑥)]

Table 1 .

 1 Default Recommender101 execution.

	sampleNUsers	100	400	800	1600
	Items	3706	3706	3706	3706
	Ratings	14720	60796	137742	274529
	Sparsity	0,04	0,041	0,046	0,046
	Avg. Ratings/User	147,2	151,99	172,178	171,581
	Avg. Ratings/Item	3,972	16,405	37,167	74,077
	Max /User	1050	1277	1302	1743
	Max /item	59	219	469	917
	MAE FunkSVD	0,802	0,739	0,725	0,72
	MAE SlopeOne	0,792	0,733	0,711	0,692
	RMSE FunkSVD	1,024	0,94	0,921	0,915
	RMSE SlopeOne	1,004	0,933	0,904	0,882
	Precision FunkSVD	0,693	0,724	0,742	0,755
	Precision SlopeOne	0,691	0,724	0,735	0,739
	Recall FunkSVD	0,488	0,514	0,518	0,523
	Recall SlopeOne	0,483	0,517	0,509	0,509
	Execution Duration	22	133	294	632
	Next results, table 2, are showing the use of the proposed method with Sigma Count
	average as the final fuzzification method.

Table 2 .

 2 Sigma Count average as final fuzzification method results.

	sampleNUsers /	100	400	800	1600
	My_User.count	(300)	(1200)	(2400)	(4800)
	Items	2972	3418	3576	3662
	Ratings	13137	67405	134833	276467
	Sparsity	0,044	0,049	0,047	0,047
	Avg. Ratings/User	131,37	168,512	168,541	172,792
	Avg. Ratings/Item	4,42	19,721	37,705	75,496
	Max /User	608	1518	1323	1271
	Max /item	61	255	514	961
	MAE FunkSVD	0,773	0,717	0,705	0,707
	MAE SlopeOne	0,765	0,713	0,692	0,682
	RMSE FunkSVD	0,991	0,912	0,894	0,898
	RMSE SlopeOne	0,972	0,917	0,879	0,868
	Precision FunkSVD	0,714	0,737	0,751	0,763
	Precision SlopeOne	0,71	0,736	0,744	0,744

Table 3 .

 3 Hybrid Hamacher product as final fuzzification method results.

	sampleNUsers /	100	400	800	1600
	My_User.count	(300)	(1200)	(2400)	(4800)
	Items	2931	3375	3507	3642
	Ratings	12997	67302	135107	278365
	Sparsity	0,044	0,049	0,048	0,048
	Avg. Ratings/User	131,37	168,512	168,541	172,792
	Avg. Ratings/Item	4,46	19,714	37,788	79,14
	Max /User	685	1571	1334	1420
	Max /item	68	276	603	1007
	MAE FunkSVD	0,761	0,711	0,699	0,703
	MAE SlopeOne	0,759	0,711	0,689	0,679
	RMSE FunkSVD	0,981	0,91	0,899	0,897
	RMSE SlopeOne	0,972	0,918	0,861	0,86
	Precision FunkSVD	0,723	0,74	0,767	0,766
	Precision SlopeOne	0,723	0,736	0,741	0,748
	Recall FunkSVD	0,52	0,518	0,531	0,528
	Recall SlopeOne	0,512	0,519	0,525	0,519
	Execution Duration	16	142	301	410
	Hybrid Einstein product results are in table 4 below.

Table 4 .

 4 Hybrid Einstein product as final fuzzification method results.

	sampleNUsers /	100	400	800	1600
	My_User.count	(300)	(1200)	(2400)	(4800)
	Items	2972	3418	3576	3662
	Ratings	13137	67405	134833	276467
	Sparsity	0,044	0,049	0,047	0,047
	Avg. Ratings/User	131,37	168,512	168,541	172,792
	Avg. Ratings/Item	4,42	19,721	37,705	75,496
	Max /User	680	1603	1371	1407
	Max /item	72	268	530	980
	MAE FunkSVD	0,76	0,714	0,701	0,69
	MAE SlopeOne	0,756	0,709	0,671	0,677
	RMSE FunkSVD	0,983	0,904	0,89	0,891
	RMSE SlopeOne	0,972	0,917	0,879	0,868
	Precision FunkSVD	0,722	0,739	0,761	0,764
	Precision SlopeOne	0,717	0,738	0,748	0,751

Table 5 .

 5 Hybrid Einstein product as final fuzzification method results.

	sampleNUsers /	100	400	800	1600
	My_User.count	(300)	(1200)	(2400)	(4800)
	MAE FunkSVD	-3,62%	-2,98%	-2,76%	-1,81%
	MAE SlopeOne	-3,41%	-2,73%	-2,67%	-1,45%
	RMSE FunkSVD	-3,22%	-2,98%	-2,93%	-1,86%
	RMSE SlopeOne	-3,19%	-1,71%	-2,77%	-1,59%
	Precision FunkSVD	3,03%	1,80%	1,21%	1,06%
	Precision SlopeOne	2,75%	1,66%	1,22%	0,68%
	Recal FunkSVD	6,35%	0,58%	1,74%	0,08%
	Recal SlopeOne	4,35%	0,39%	1,38%	0,59%

Table 6 .

 6 Hybrid Einstein product as final fuzzification method results.

	sampleNUsers /	100	400	800	1600
	My_User.count	(300)	(1200)	(2400)	(4800)
	MAE FunkSVD	-5,11%	-3,79%	-3,59%	-2,36%
	MAE SlopeOne	-4,17%	-3,00%	-3,09%	-1,88%
	RMSE FunkSVD	-4,20%	-3,19%	-2,39%	-1,97%
	RMSE SlopeOne	-3,19%	-1,61%	-4,76%	-2,49%
	Precision FunkSVD	4,33%	2,21%	3,37%	1,46%
	Precision SlopeOne	4,63%	1,66%	0,82%	1,22%
	Recal FunkSVD	6,56%	0,78%	2,51%	0,96%
	Recal SlopeOne	6,00%	0,39%	3,14%	1,96%

Table 7 .

 7 Hybrid Einstein product as final fuzzification method results.

	sampleNUsers /	100	400	800	1600
	My_User.count	(300)	(1200)	(2400)	(4800)
	MAE FunkSVD	-5,24%	-3,38%	-3,31%	-4,17%
	MAE SlopeOne	-4,55%	-3,27%	-5,63%	-2,17%
	RMSE FunkSVD	-4,00%	-3,83%	-3,37%	-2,62%
	RMSE SlopeOne	-3,19%	-1,71%	-2,77%	-1,59%
	Precision FunkSVD	4,18%	2,07%	2,56%	1,19%
	Precision SlopeOne	3,76%	1,93%	1,77%	1,62%
	Recal FunkSVD	7,17%	0,97%	2,12%	1,53%
	Recal SlopeOne	5,80%	0,58%	4,32%	3,54%

Table 8 .

 8 Comparison of experimental results of run with FMV threshold equal to 0.55 and standard Recommender101 run.

	sampleNUsers /	Extra experi-
	My_User.count	ment	Default run	Difference
	Items	3574	3706	-132
	Ratings	337632	274529	63103
	Sparsity	0,059	0,046	28,26%
	Avg. Ratings/User	211,02	171,581	39,439
	Avg. Ratings/Item	94,469	74,077	20,392
	Max /User	1743	1743	0
	Max /item	1100	917	183
	MAE FunkSVD	0,651	0,72	-9,58%
	MAE SlopeOne	0,629	0,692	-9,10%
	RMSE FunkSVD	0,826	0,915	-9,73%
	RMSE SlopeOne	0,8	0,882	-9,30%
	Precision FunkSVD	0,787	0,755	4,24%
	Precision SlopeOne	0,773	0,739	4,60%
	Recall FunkSVD	0,531	0,523	1,53%
	Recall SlopeOne	0,524	0,509	2,95%
	Execution Duration	713	632	12,82%