Ngoc Q K Duong

Pierre Berthet
email: p.berthet@3dsoundlabs.com

Sidkièta Zabre
email: sidkieta.zabre@altran.com

Michel Kerdranvat

Alexey Ozerov

Louis Chevallier

AUDIO ZOOM FOR SMARTPHONES BASED ON MULTIPLE ADAPTIVE BEAMFORMERS

Keywords: Audio zoom on smartphone, sound capture, robust adaptive beamformer, post-processing

Some recent smartphones have offered the so-called audio zoom feature which allows to focus sound capture in the front direction while attenuating progressively surrounding sounds along with video zoom. This paper proposes a complete implementation of such function involving two major steps. First, targeted sound source is extracted by a novel approach that combines multiple adaptive beamformers having different look directions with a post-processing algorithm. Second, spatial zooming effect is created by leveraging the microphone signals and the enhanced target source. Subjective test with real-world audio recordings using a mock-up simulating an usual shape of the smartphone confirms the rich user experience obtained by the proposed system.

INTRODUCTION

Mobile devices such as smartphones and tablets have become very popular nowadays for many users. Their hardware and processing power has also been improved day by day, that makes them able to offer more enhanced applications with richer user experience. This paper considers a so-called audio zoom application [START_REF] Avendano | Audio zoom[END_REF][START_REF] Lee | Mobile terminal and audio zooming method thereof[END_REF] where mobile devices can focus the sound capture on a desired direction while attenuating progressively surrounding sounds 1 . Audio zoom has been commercialized in recent smartphones (e.g., Samsung Galaxy S5 and LG G2), and it would be even more powerful in future products owning larger microphone array.

In order to perform audio zoom, the target sound source needs to be isolated from other surrounding sounds (i.e., interferences originated from unwanted spatial directions) first. Thanks to the hardware improvement where most smartphone nowadays possesses two or more microphones (e.g., Apple iPhone 5 showed up with not one or two, but even three microphones 2 ), research in microphone array processing field can be well-applied to this considered problem. Specifically, beamforming [START_REF] Veen | Beamforming: a versatile approach to spatial filtering[END_REF][START_REF] Li | Robust adaptive beamforming[END_REF] and audio source separation (ASS) [START_REF] Makino | Blind Speech Separation[END_REF][START_REF] Vincent | The Signal Separation Campaign (2007-2010): Achievements and remaining challenges[END_REF] can be considered as the most appropriate approaches. As ASS generally requires higher computation cost than beamforming algorithms since it usually involves, in addition, the advanced spectral This work has been done while the second the third authors were with Technicolor.

1 https://www.youtube.com/watch?v=7DEyuapmRCs 2 http://www.idownloadblog.com/2012/09/12/iphone-5-three-mics/ modelling of the audio sources [START_REF] Thiemann | An experimental comparison of source separation and beamforming techniques for microphone array signal enhancement[END_REF]. Thus, by considering the critical constraint of limited processing power in mobile devices, we design our signal enhancement algorithm for the target source grounded on beamforming technique3 . However, since beamforming usually requires a large microphone array in order to create a narrow beam capturing sound from a desired direction, we propose in this paper a novel approach that combines multiple robust adaptive beamformers [START_REF] Bitzer | Superdirective microphone arrays[END_REF][START_REF] Gu | Robust adaptive beamforming based on interference covariance matrix reconstruction and steering vector estimation[END_REF] with a derived post-processing algorithm taking into account outputs of the beamformers so as to greatly enhance the targeted sound source. Once the target sound source is extracted, we further propose the creation of zooming effect as second step of the audio zoom system. Note that in the considered beamforming implementation, one beamformer has directivity pattern that emphasizes the target source while, on the contrary, the other beamformers suppress the target source. Similar strategy has been presented in [START_REF] Takada | Sound source separation using null-beamforming and spectral subtraction for mobile devices[END_REF] with the use of two fixed null beamformers, instead of multiple adaptive beamformers as considered in this paper, and spectral substraction as post-processing algorithm.

The paper aims to design a complete audio zoom system which can be implemented in mobile devices as an emerging application with reasonable processing cost. Yet, to the best of our knowledge, non of the scientific publications has been described such a similar system. It is also worth noting that the proposed approach has been implemented as part of a MediaPlayer running real-time on Android smartphones 4 .

The rest of the paper is organized as follows. In Section 2 we present the global workflow as well as the detail steps of the proposed audio zoom system. We conduct experiment with subjective test on real-world sound scene recordings to validate the effectiveness of the proposed approach in Section 3. Finally we conclude in Section 4.

PROPOSED AUDIO ZOOM SYSTEM

General workflow of the proposed audio zoom approach is shown in Fig. 1. It consists in two major steps: (1) target sound source extraction and (2) zooming effect creation. These steps will be described in detail in Section 2.1 and Section 2.2, respectively. Let us denote by x f m ∈ C P ×1 the complex-valued STFT coefficients in time frame m and frequency bin f of the mixture signal recorded by P microphones. Beamforming isolates sound coming from a target spatial direction θt by deriving a frequency dependent weight vector w θ t ,f such that its output is given by

ŝθ t ,f m = w H θ t ,f x f m , (1)
where (.) H denotes Hermitian transpose. As relevant to the considered audio zoom application, where users usually focus sound capture in the front direction that is perpendicular to the device's surface, in the rest of the paper we consider θt = 90 0 . The optimal weight vector w θ t ,f can be obtained by minimizing the energy of the interfering sources and noise under the constraint to keep unit response in the target direction. In this derivation, the beamformer is known as Minimum Variance Distortionless Response (MVDR) [START_REF] Bitzer | Superdirective microphone arrays[END_REF] -a well-known one in the literature and the resulting weight vector is given by

w θ t ,f = R -1 i+n,f m d θ t ,f d H θ t ,f R -1 i+n,f m d θ t ,f , (2)
where R i+n,f m is the interference-plus-noise covariance matrix and d θ t ,f is the steering vector which accounts for the time differences of arrival from the target source to the microphones and is computed as

d θ t ,f = [1, e i2πf (τ θ t 2 -τ θ t 1 ) , ..., e i2πf (τ θ t P -τ θ t 1 ) ], (3)
where τ θ t p is the the time it takes for the sound to travel from target source at direction θt to microphone p (p = 1, ..., P ) on a direct path.

In practice since R i+n,f m is unknown, it is often replaced by the sample covariance matrix R

x,f m = E[x f m x H f m ] [8]
. A more advanced approach [START_REF] Gu | Robust adaptive beamforming based on interference covariance matrix reconstruction and steering vector estimation[END_REF], known as robust adaptive beamforming, proposes to estimate this interference-plus-noise covariance matrix by integrating the spatial spectrum distribution over all possible direc-tions containing unwanted signals Θi+n as

R i+n,f m = Θ i+n d θ,f d H θ,f d H θ,f R -1 x,f m d θ,f dθ, (4)
where d θ,f is computed similarly to (3) for direction θ.

Proposed implementation of multiple beamformers

In our implementation, in order to reduce the computation cost for smartphone application we first replace the integration (4) by the sum over several unwanted directions (e.g., Θi+n = 0 0 , 45 0 , 135 0 , 180 0 when the target direction is θt = 90 0 ). Additionally, we incorporate the diagonal loading technique investigated in [START_REF] Mestre | On diagonal loading for minimum variance beamformers[END_REF] to enhance the directivity pattern design. The resulting weight of the proposed beamforming implementation, named robust MVDR (RMVDR), is estimated as

w θ t ,f = ( R i+n,f m + γI) -1 d θ t ,f d H θ t ,f ( R i+n,f m + γI) -1 d θ t ,f , (5)
where I is the P × P identity matrix, γ is a loading factor preventing instability [START_REF] Mestre | On diagonal loading for minimum variance beamformers[END_REF], and the interference-plus-noise covariance matrix is computed by

R i+n,f m = θ∈ Θ i+n d θ,f d H θ,f d H θ,f R -1 x,f m d θ,f . (6)
Since equipping a large microphone array for a smartphone so that beamforming can isolate well the target source is not feasible in practice, we further propose to use multiple RMVDR where one of them enhances the target source (i.e., RMVDR 90 0 ) and the others enhance unwanted sound coming from other directions (e.g., RMVDR 0 0 and RMVDR 180 0 , these beamformers have look directions perpendicular to the desired one). This implementation is depicted in Fig. 1 for the case when three RMVDRs are used. Note that the overall computational cost does not increase linearly with respect to the number of RMVDRs used since the sample covariance matrix R x,f m needs to be computed once, and similarly steering vectors d θ,f needed in ( 5) and ( 6) can be shared between RMVDRs.

We will discuss the post-processing of the outputs of these beamformers so as to further isolate the target sound source, compared to the conventional case where only RMVDR 90 0 is used, in Section 2.1.3.

Proposed post-processing algorithm

Denoting by ŝθ t ,f m and ŝθ,fm the output of RMVDRs looking at the target direction θt and other directions θ = θt, respectively. As example in our setting shown in Fig. 1, θt = 90 0 while θ = 0 0 or 180 0 . However, one can easily extend the algorithm with the use of more RMVDRs and any desired direction than the 90 0 . We propose to compute the STFT coefficients of the post-processed output signal for audio zoom as

ŝfm = x p,f m if |ŝ θ t ,f m | > α max{|ŝ θ,f m |, ∀θ = θt} β f m ŝθ t ,f m otherwise (7)
where |.| denotes the absolute value, p denotes a reference microphone signal such as p = 2 for a front microphone in our setting, α > 1 is a tuning constant, and

β f m = 1 + max{|ŝ θ,f m |,∀θ =θ t } |ŝ θ t ,f m | (8)
where is a constant (e.g., = 1).

Our derivation to equations ( 7) and ( 8) is motivated by the well-known observation that the sound sources are usually nonoverlapped in the time-frequency (T-F) domain. As can be seen from the first line of ( 7), for time-frequency (T-F) points where the estimated target source is really dominant than the others, we take signal from a front microphone x p,f m as the final output so as to maximize the sound quality 5 . In this case, a reference microphone signal is a good estimate of the target source since other sources are considered to be inactive. Otherwise, the estimated target STFT coefficients ŝθ t ,f m will be considered. The derivation to equation ( 8) can be explained by the fact that in T-F points where sound from non-desired directions is really dominant (i.e., max{|ŝ θ,f m |, ∀θ = θt} >> ŝθ t ,f m ), the target source ŝfm should be considered as inactive. Thus its value should close to 0 as β f m will be very small. In neutral case where none of the estimated sources is really dominant, the smaller ŝθ t ,f m compared to the other sources, the more amplification it should be, as β f m increase, in order to further improve the designed zooming effect as presented in Section 2.2. Finally, the time domain signal ŝ(t) of the enhanced target source is obtained by the inverse STFT of ŝfm .

Proposed audio zoom effect creation

Let us denote by z ∈ [0, 1] the zooming factor where the higher value of z the more target sound source is focused, and z = 1 corresponds to the maximum zoom (i.e., 100%). In order to maintain spatial effect of the perceived stereo output signal, we propose to mix the estimated target source after the post-processing ŝ(t) with the original signals recorded by left and right microphones, denoted as x l (t) and xr(t), respectively. The final left and right channels of the output signal, denoted by sl (t), and sl (t), respectively, are computed as

sl (t) = z * ŝ(t) + (1 -z) * x l (t), (9) sr
(t) = z * ŝ(t) + (1 -z) * xr(t). ( 10
)
It can be seen that there is no zooming effect when z = 0, and when z increases the estimated target source ŝ(t) contributes more to the output signal as it should be more progressively focused. In case of maximum zoom with z = 1, both output channels take the same value (i.e., sl (t) = sr(t) = ŝ(t)) so that the user can experience spatial effect of the isolated sound as if it comes from the front direction (θt = 90 0 ) and the target sound source is most focused.

EXPERIMENTS

We fist describe the recording setup in Section 3.1. We then present the algorithm implementation and result of the subjective test where different users experienced audio zooming effect created by the proposed approach in Section 3.2.

Experiment setup

In order to make a test close to the real situation, we built a mock-up containing four microphones mimicking a smartphone as shown in Fig. 2. In this setting, two microphones are located at the top and bottom of the mock-up as usual with most available smartphones, two other microphones are located at the back side so as to ease sound capture during the video recording. The detail (x,y,z) coordinates of these microphones, measured in centimeter, are (6.5, 2, 0.5); (3.3, 0, 0); (-0.033, 0, 0); (-6.5, 2, 0.5), respectively. We performed two 40 second length indoor audio recordings without video capture. The setups are shown in Fig. 3(a) and Fig. 3(b), respectively, where M1 and M2 are two musical instruments while S1 and S2 are two speeches. In both cases, audio zoom algorithm aims to enhance two sound sources located near the center while progressively attenuating two other unwanted sources. For a more realistic evaluation of the user perception when audio zoom is performed together with video zoom, we made an additional outdoor recording in a park as shown in Fig. 3(c) where audio and video is captured together. The recording duration is 90 seconds and audio zoom algorithm aims to focus on the bird song while canceling surrounding sounds including human walking, speech, environmental wind, etc.,. Table 1. Results with subjective listening tests performed by 13 users.

Result with subjective test

We developed an application with a friendly graphical user interface (GUI) so as user can perform audio zoom and experience the audio quality obtained by different implementations via a headphone. It is worth noting that we prefer the real-world user test than the objective evaluation since the former case is more relevant to the target application. We invited 13 people at different ages to participate in three different listening tests where each of them was asked to indicate which algorithm yields better zooming experience, or they offer the similar quality in his/her opinion. The results for three test cases, performed in double-blind fashion, and for each recording condition are shown in Table 1 where the value means the number of rated users in each option. Note that in all tests, the second step for creating audio zooming effect is implemented similarly for all other approaches under comparison. We first validate whether the state-of-the-art post filtering technique brings some benefit when it is implemented after beamforming as a standard way [START_REF] Thiemann | An experimental comparison of source separation and beamforming techniques for microphone array signal enhancement[END_REF] in the "Test 1". For this purpose, we asked users' opinion when they experienced results obtained by the baseline robust adaptive beamformer (RMVDR 90 0 ) and that obtained by the RMVDR 90 0 followed by the well-known Optimal Modified Minimum Mean-Square Error Log-Spectral Amplitude (OMLSA) 6post-filtering algorithm (named "B+OMLSA" in Table 1). Note that other post-filters (e.g., Zelinskis [START_REF] Zelinski | A microphone array with adaptive post-filtering for noise reduction in reverberant rooms[END_REF] and McCowans [START_REF] Mccowan | Microphone array post-filter for diffuse noise field[END_REF]) can also be tested, but as observed in [START_REF] Thiemann | An experimental comparison of source separation and beamforming techniques for microphone array signal enhancement[END_REF] that they did not bring benefit compared to OMLSA, we consider OMLSA as the state-of-the-art post filter in our implementation and test. As can be seen, for indoor recording more users prefer not to use OMLSA since it brings additional signal distortion. For outdoor recording, even though OMLSA really suppresses more background diffuse noise, it still does not bring benefit in the test. This listening test is actually coherent with the observation in [START_REF] Thiemann | An experimental comparison of source separation and beamforming techniques for microphone array signal enhancement[END_REF] that MVDR+OMLSA adds further signal distortion compared to MVDR alone so as user perceives more artifact.

The "Test 2" aims to compare the proposed approach, i.e. three RMVDR having look directions of 0 0 , 90 0 , and 180 0 , respectively, and post-processing as shown in Fig. 1, (named "Proposed"), with the state-of-the-art beamforming approach using one RMVDR 90 0 . Note that we did not compare to the case where RMVDR 90 0 is followed by OMLSA here since it has been shown in the Test 1 that users prefer RMVDR 90 0 alone. We also implemented a method using null-beamformers and post-processing algorithm described in [START_REF] Takada | Sound source separation using null-beamforming and spectral subtraction for mobile devices[END_REF], but subjectively observed that it performs poorer than the two considered algorithms, so we did not formally perform user test with it in order to avoid too much listening for users. As can be seen in Table 1, most users prefer the audio zoom quality obtained by the proposed approach in all three recording conditions. As example, for the real-world outdoor recording where audio zoom was performed together with video zoom to maximize the user experience, 10 users prefer the result of the proposed approach while only 2 users prefer the result of the baseline. It is also worth noting that our informal listening test in case of using two microphones, instead of four, also shares the same experience that the proposed approach performs better than the others.

The final test was devoted to the zooming effect only where we want to validate if increasing the volume of the enhanced signal can improve overall user experience. Thus we compare the "proposed" with a case where the enhanced signal after beamforming and post processing ŝ(t) is boosted by 6dB energy before mixing with the original microphone signals in the zooming creation step. The result is shown in "Test 3". Surprisingly, overall performance for three recording conditions shows that user experience is generally not improved as expected when increasing volume of the target sound. This can be explained by the fact that ŝ(t) still contains noticeable distortion so that when its volume increases users also perceive more artifacts.

CONCLUSION

In this paper, we have presented a novel approach for performing audio zoom, an emerging application, in mobile devices with low computation cost. The proposed implementation combines several robust adaptive beamformers with a derived post-processing algorithm to further enhance the targeted sound source. We also describe the design of zooming effect so as to improve the user perceptual experience. Subjective tests with both real-world indoor and outdoor recordings confirm the effectiveness of the derived approach. Future research would be devoted to perform a formal objective evaluation where ground truth is available. Additionally, the investigation of audio source separation based approach where the target direction can be taken into account as prior information [START_REF] Duong | Spatial location priors for gaussian model based reverberant audio source separation[END_REF] would be potential.

Fig. 1 .

 1 Fig. 1. General workflow of the proposed audio zoom implementation.

2. 1 .

 1 Target sound source enhancement 2.1.1. Robust adaptive beamforming

Fig. 2 .

 2 Fig. 2. Mock-up with 4 microphones for the experiment.

Fig. 3 .

 3 Fig. 3. Experiment setup for user test on audio zoom feature.

Note that, preliminary study in[START_REF] Thiemann | An experimental comparison of source separation and beamforming techniques for microphone array signal enhancement[END_REF] did not show remarkable advantage of ASS compared to beamforming in some specific setups such as a single target source in noise field.

The demostration will be presented at the Show and Tell session of the 41st IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP 2016)

Note that in the output of RMVDR there is usually some artifact due to the nonlinear processing, and the signal distortion is more severe at high frequencies where the array's geometry error has more impact.

Matlab code is available at: http://webee.technion.ac.il/Sites/People/IsraelCohen/Download/omlsa.m