Nicolas Scaringella
email: nicolas.scaringella@epfl.ch

Giorgio Zoia
email: giorgio.zoia@epfl.ch

A REAL-TIME BEAT TRACKER FOR UNRESTRICTED AUDIO SIGNALS

Analysis of real music content, not available in symbolic form, still remains a very challenging problem. Promising results can be obtained combining signal processing techniques with intelligent agents, in order to support the often ambiguous results of the analytic phase with smart decision systems, trained by a consistent preliminary knowledge or characterized by forms of learning. In this paper we propose a multiagent algorithm for beat and tempo analysis and induction for unrestricted audio signals; it is based on the combination of lossy onset detection, note accentuation evaluation to estimate metrically essential events, and a multi-agent mechanism to allow dynamic beat tracking. Each agent maintains a self-confidence attribute to rate the confidence for the theory it supports. Consistent test criteria have been used. Experimental results are reported for a database of musical samples from different styles and genres; these results are quite promising. The integration with a harmony analyzer for mutual consolidation is envisaged as next step.

INTRODUCTION

Multimedia music applications are nowadays rapidly moving from simple content related scenarios to more complex and sophisticated domains including content, interaction, related descriptions and annotations, item identification. The creation of huge databases coming from both restoration of existing analog content and new digital content is requiring more and more reliable and fast tools for content analysis and description, to be used for searches, content queries and interactive access. Another requirement that is gaining importance in the domain is the automatic control of signal processing parameters according to content features.

Analysis of real music content, not necessarily available in symbolic form, has always been regarded as a very challenging problem, and it still remains such in spite of several years of careful and interesting research in the area. One of the most promising analysis directions consists of the combination of signal processing techniques with some form of intelligent agent, in order to support the often ambiguous results of the first phase with smart decision systems based on a consistent preliminary knowledge or on some learning algorithm. This approach necessarily introduces an approximation in the results, but very often this is acceptable in real life applications as far as the performance is able to meet a quality of service that can be considered functional by users.

We are interested in providing real-time tempo and rhythm analysis on audio content, possibly without any kind of symbolic or metadata information being preliminary available; this corresponds in practice to cases in which the musical content is presented to the tool in a flat, digital sample format, without any other kind of information about it. In these cases, which we consider to correspond better than others to real world applications, extremely high precision and confidence in the results are very difficult to obtain; however, it will be shown that results that can be achieved are indeed interesting and useful for several practical purposes. Integration with other forms of analysis (chords, timbre, etc.) for mutual consolidation is envisaged.

In particular our research aims at real-time human machine synchronization in the music domain and especially at content identification, description and classification (including genre recognition) in the multimedia domain; real-time automatic control of content (equalization, track sequencing, etc.) will be another area of interest and possible exploitation.

This paper is organized as follows: the second section shortly presents the state of the art in the domain of our research; the third section introduces and explains the new multi-agent algorithm for beat and tempo induction, first describing the basic building blocks and their rationales, and then making some remarks on confidence evaluation of the obtained results; the fourth section presents experimental results whereas the last section concludes the paper with final remarks and future work to be done.

RELATED WORK

Several approaches have been investigated to perform tempo and beat induction in the last decade. We pay a special attention on multi-agent models as the proposed approach falls in this category. For a more complete review of existing models see [START_REF] Gouyon | Towards rhythmic content processing of musical signalsfostering complementary approaches[END_REF].

One advantage of multi-agent models is their ability to predict the position of future beats. This is coherent with [START_REF] Desain | Computational models of beat induction: the rule-based approach[END_REF], which argues that human perception of rhythm is based on two "diatomic" processes: a bottom-up process that permits to rapidly obtain a perception of rhythm from scratch and a top-down process that induces the organization of incoming events. In [START_REF] Roberts | Interpreting rhythmic structures using artificial neural networks[END_REF] a self-organizing neural network (SONNET1) is exploited to infer beat times and to extract temporal patterns. While the neural network formalism is used, the system is similar to other multi-agent models.

Dixon [START_REF] Dixon | Automatic extraction of tempo and beats from expressive performance[END_REF] proposes a non real-time multi-agent model which uses musical knowledge not related to any style. The model works with audio or MIDI data but the musical knowledge only applies to MIDI data.

In [START_REF] Goto | An audio-based real-time beat tracking system for music with or without drum-sounds[END_REF] a real-time audio based multi-agent beat tracker is developed that uses three kinds of musical knowledge: note onset times, chord changes, drum patterns.

Meudic [START_REF] Meudic | A causal algorithm for beat tracking[END_REF] gives a real-time version of Dixon's model restricted to the case of MIDI data.

The proposed algorithm is based on a multi-agent architecture and is applicable to unrestricted polyphonic audio waveforms. It is causal, so that it can be run in real-time. It replaces Dixon's notion of musical knowledge adapted to MIDI files by a salience [START_REF] Dixon | Automatic extraction of tempo and beats from expressive performance[END_REF] calculation directly computed from low-level signal descriptors.

THE BEAT-TRACKER ALGORITHM

This section gives an overview of the complete tempo and beat tracking system. The block diagram of Figure 1 above summarizes the complete algorithms and its main blocks, which are described in further details in the next subsections.

Onset detection

The first step is the extraction of onset times of musical notes from the audio waveform.

While highly sophisticated algorithms exist for note onset detection of unrestricted polyphonic music (see [START_REF] Duxbury | Complex domain onset detection for musical signals[END_REF] or [START_REF] Klapuri | Sound onset detection by applying psychoacoustic knowledge[END_REF]), Dixon states [START_REF] Dixon | Automatic extraction of tempo and beats from expressive performance[END_REF] that a lossy onset detection algorithm is not a problem for a beat tracker since it filters out the less salient onsets, which would less probably correspond to beats. We agree with this position and use simple onset detection.

The energy envelope of the signal is extracted and peak detection is performed on this representation to locate note onsets. Peak detection is performed within a simple sliding window which size corresponds to the minimum discriminable inter-onset distance (depending on different cases, this distance may vary between 50 ms and 70 ms).

Phenomenal accent evaluation

There is a good agreement in literature about the fact that musical events with greater accentuation tend to occur in stronger metrical positions [START_REF] Parncutt | A perceptual model of pulse salience and metrical accent in musical rhythms[END_REF]. The factors influencing perceived accentuation can be divided into four categories: 1. Phenomenal accent: the note is stressed because played in a sharper or louder manner, or with a slight delay, or it involves sudden changes in dynamics or timbre or leaps to relatively high or low pitched notes; 2. Metrical accent: the note is stressed because of its metrically strong position; 3. Structural accent: stress caused by a profound harmonic or melodic effect (such as cadences); 4. Durational accent: notes that are longer than the surrounding notes In the case of a multi-agent model dealing with discrete events, the use of musical accentuation is crucial since it can be used to emphasize the most important agents and thus to limit the number of interpretations of a sequence. Accentuation is used here as a prior element for the probability that a note is a beat.

Various factors are used to account for note accentuation.

Some experiments have been done using a multilayer perceptron and a set of low level audio descriptors to model the local physical characteristics of beats. While we obtained similar results as Seppänen who worked on a close problem with a Bayesian classifier [START_REF] Seppännen | Computational models of musical meter recognition[END_REF], our tracking results were not significantly improved using the neural network compared to the simple use of the energy of the onset as a single descriptor.

We agree with Gouyon et al. [START_REF] Gouyon | Determination of the Meter of musical audio signals: Seeking recurrences in beat segment descriptors[END_REF] who state that emphasis should be put on recurrence of low-level features rather on their local values. In this sense, we experimented with timbral similarities based on the assumption that an onset is more likely to be a beat if it shares some timbral similarities with the onsets previously considered as beats.

The time proximity between an onset and a beat prediction is also considered as beats tend to be played with less deviation than other notes [START_REF] Dixon | Automatic extraction of tempo and beats from expressive performance[END_REF]. Some higher level information can be considered, if available, to confirm the fact that an event is indeed a beat. Some preliminary experiments have been done with a chord detector developed in our laboratory for polyphonic audio signals [START_REF] Zhou | A multitimbre chord/harmony analyzer based on signal processing and neural networks[END_REF]. We are using the common assumption [START_REF] Goto | An audio-based real-time beat tracking system for music with or without drum-sounds[END_REF] that chord changes positions are correlated with beats. Consolidated joint results will be available in future publications.

Tempo estimator

The tempo estimator stage outputs a ranked list of tempi based on the observation of every possible inter-onset intervals (IOIs) in a limited memory of 5 seconds (which is an estimate of human short-term memory length). Each IOI is reduced to fit in the range 0.3 to 1.5 ms (that is 200 to 40 beats per minute).

The list of possible IOIs is clustered with Dixon's algorithm described in [START_REF] Dixon | Real time tracking and visualisation of musical expression[END_REF] to give a limited ranked list of tempi.

Beat tracking agent: associating events to beats

A rhythm interpretation is supported by an agent. Each agent is characterized by a state (its current tempo, his predictions for the next beat times and his current confidence) and a history (the list of accepted beat times). Agents analyse the stream of events by checking if the current event is a beat according to the rhythm interpretation it supports.

A new event is associated to an agent's prediction if the event fits in a window centered on the prediction and which size is proportional to the period of the agent (see Figure 2). If multiple events fit in the window, the one with the largest accentuation (the highest arrow in the picture) is considered as the beat.

Each time an agent considers the current event as a beat, it must update its internal state and add the new beat to its history. The system being causal, an agent may consider an event as a beat at a date n and find a better candidate at date n+1. In such cases, the agent recovers its previous state before updating.

When a prediction of an agent can not be fulfilled, an induced beat is added to its history. His confidence is updated (downwards) but its beat rate and set of predictions are left unchanged.

Beat tracking: confidence evaluation

Each agent maintains a self-confidence attribute that rates the confidence for the interpretation it supports. The evaluation of confidence is a critical point of our algorithm as it implements the competition between agents.

The confidence property is basically evaluated by comparing the measured events and the predicted events associated to the agent and by integrating the accentuation of beats actually observed.

More formally, the confidence C t+1 of an agent at time t+1 is evaluated as:

) ) 1 ( ) ( ( 2 1 1 t t t t t t t t C C I C k E R C C + - - + = + (1)
where R t is the beat rate of the agent at time t, k 1 is a constant greater than unity that fixes the maximum possible confidence, E t is the excitatory component at time t and I t is the inhibitory component at time t.

The excitatory component E t is evaluated as:

notInduced R ACC k E t t t ⋅ ℘ ⋅ ⋅ = ) ( 2 (2)
where k 2 is a constant, the function ℘is Parncutt's preferred tempo function [START_REF] Parncutt | A perceptual model of pulse salience and metrical accent in musical rhythms[END_REF], notInduced is a boolean value equal to 0 if the considered beat was induced and equal to 1 otherwise. The term ACC t accounts for the total accentuation of the event at date t. This accentuation term is itself customisable (see section 3.2.).

The inhibitory component I t equals:

t t C k Induced k I max, 4 3 + ⋅ = (3)
where k 3 and k 4 are constants, C max, t is the confidence of the most confident agent at date t and Induced is a boolean value equal to 1 if the considered beat was induced and equal to 0 if the beat was observed.

The normalization by the beat rate R t is used to make sure that agents with shorter period (that will encounter more beats) will not be favoured.

Multi-agent beat tracking

The main algorithm is basically an administrator of beat tracking agents. It rules the creation of agents, their relations, their destruction and it is in charge of choosing which one supports the correct interpretation.

The first agent whose confidence reaches a high threshold is considered as the winner (i.e. it is considered to follow the correct beats at the correct tempo). Another agent may replace the first winner if it fulfills the following conditions for a minimum specified amount of time:

1. it has the highest confidence 2. its confidence is above a high threshold 3. its confidence is sufficiently higher than the confidence of the previous winner. This mechanism allows avoiding spurious changes of interpretation and is coherent with what Lerdhal and Jackendoff refer to as conservative hearing [START_REF] Lerdahl | A generative theory of tonal music[END_REF]. It also favours the beginning of the excerpt and reflects the primacy effect of Parncutt [START_REF] Parncutt | A perceptual model of pulse salience and metrical accent in musical rhythms[END_REF].

To minimize complexity, agents supporting an improbable theory must be removed as soon as possible. If an agent runs out of prediction (i.e. it has not been updated for a while), it is erased. If its confidence is under a low threshold while created some time before, it is erased as its interpretation is not probable. If its interpretation has converged towards the interpretation of a more confident agent, it is also erased as the two agents will follow the same beats.

EXPERIMENTAL RESULTS

Measuring the quality of a tempo and beat tracking algorithm often remains an open issue. Comparing the output of the algorithm to a score is not necessarily a good solution since musicians do not always play straight on the beat in real performance. Furthermore, in the case of audio data there is no score to use as an exact reference.

Since the proposed algorithm extracts the beat as played in the performance, an intuitive and easy way to evaluate its quality is to add a percussive sound on each beat of the analysed song. Listening to the results, one can easily decide if the extraction is coherent with human natural foot tapping.

Results on the music genre database

To obtain a first automated evaluation of the algorithm, we annotated a test database of 84 songs of various musical genres [START_REF] Goto | RWC Music Database: Music genre database and musical instrument sound database[END_REF] with the beats tapped by a trained human listener and refer to them as the correct beat positions (giving the correct tempo). We use Cemgil's performance metric [START_REF] Cemgil | On tempo tracking: tempogram representation and Kalman filtering[END_REF] to evaluate the quality of our beat tracker. This metric computes a scalar value ρ that can be interpreted as a percentage of correct beats (a value of ρ = 100% for a song means that all of its beats were correctly tracked). The average value of ρ, identified as ρ, is evaluated for the complete corpus and for each major style with the collective performance measure of Seppänen [START_REF] Seppännen | Computational models of musical meter recognition[END_REF].

On the complete corpus of 84 songs, we obtained a performance measure ρ = 72.52 %. It is interesting to briefly detail the quality of the tracking for the tested musical genres:

1. Pop, Ballads, Rock, Heavy-Metal -12 songs, ρ = 89.66 %: all complete songs were correctly tracked except for one heavy metal song, for which a pattern at 80 bpm embedded in the main rhythm structure at 120 bpm was tracked instead; 2. Rap, House, Techno, Funk, Soul-R\&B -15 songs, ρ = 95.91 %: all songs were correctly tracked; the measure is not equal to 100 % because the first beats (a variable number according to the specific song features) are usually missed as the system needs a small amount of time to lock on an interpretation of rhythm; 3. Jazz big band, Modern jazz, Jazz fusion -9 songs, ρ = 66.94 %: most big band and fusion songs were correctly tracked as the rhythmic structure in the examples was well defined; on the contrary, the modern jazz excerpts were badly tracked and are easier to track at the tatum level (the tatum can be defined as the smallest time interval between successive notes in a rhythmic phrase [START_REF] Bilmes | Timing is of the essence: perceptual and computational techniques for representing, learning, and reproducing expressive timing in percussive rhythm[END_REF]); 4. Bossa nova, Samba, Reggae, Tango -12 songs, ρ = 64.53 %: the samba and reggae songs were mostly correctly tracked, the bossa nova was harder to follow because of its syncopated structure; the tango pieces were badly tracked because the onset detection scheme proved to be inaccurate on excerpts based on violins and accordions with soft and long attacks; 5. Baroque, Classic, Romantic, Contemporary, Brass Band -12 songs, ρ = 44.60 %: the songs correctly tracked include small ensembles or solo piano or harpsichord; on the contrary large ensembles are difficult to follow, again because the onset detection scheme does not provide a robust basis for rhythm tracking. It is interesting to notice that some difficult pieces (such as large classical ensembles in this group) are correctly tracked if some parameters of the algorithm are slightly modified (see section 4.2.); 6. Blues, Folk, Country, Gospel -12 songs, ρ = 73.08 %: the results were pretty good except for some folk and country excerpts without drums; 7. African, Indian, Flamenco, Japanese Traditional and Folk -12 songs, ρ = 60.59 %: again the onset detection proved to be a weakness: in the case of Indian music played on zither, note onsets are difficult to extract while the repetitive character of music is easy to perceive. More detailed results for the complete analysed database are reported in Table 1.

Genre

Remarks on the previous results

The previous results were obtained with the system tuned with general parameters. In that case, with popular music (rock, pop, rap, techno…) very good tracking results are obtained, while they are a little disappointing with classical music or modern jazz. As discussed above, the onset detection scheme is probably the main reason explaining the difficulty to track some of the pieces.

Yet, some experiments have been made with difficult pieces and various tuning of the algorithm. A classical piece for orchestra that was previously incorrectly tracked (Egmont, Overture, op. 84 by Beethoven ) has been used. This piece has a rather slow tempo. If we get rid of the weighting by Parncutt's preferred tempo in equation ( 2) and prediction windows (section 3.4.) slightly larger are used, allowing for stronger tempo variations, the piece is perfectly tracked.

However, with such a tuning, some ambiguities arise for some of the easier pieces and the global performance measure is lower. Some fine tuning of the parameters or better some dynamic tuning, depending on the musical genre, may be used to improve the quality of the tracker. Anyhow, we think that this example prove the generality of the algorithm and its ability to track a large variety of musical genre.

Comparison with other results in literature

Beat trackers operating non-causally on MIDI input [START_REF] Cemgil | On tempo tracking: tempogram representation and Kalman filtering[END_REF] usually report a performance value ρ around 90%.

Seppännen has evaluated, with the same measure ρ, the quality of his causal audio beat tracker on a database of 330 music signals of any genre [START_REF] Seppännen | Computational models of musical meter recognition[END_REF]. He has also evaluated the performance of Scheirer's causal audio beat tracker on the same database. He reports a collective performance of 40% for his own tracker and of 51% for Scheirer's model.

As it always happens in other cases of known literature, it is rather difficult to compare our results with those reported by Seppännen, as the used databases are different (and his database is actually 3 times larger); with such a large database the reported results would much probably not be as good as with the analyzed test cases. Nevertheless, considering the detailed result table and the overall collective performance of 72.52%, it is legitimate to assume that the proposed model is, if not better, at least equivalent to the two other mentioned above.

CONCLUSION

In this paper we presented a multi-agent algorithm for beat and tempo analysis and induction; the results are quite promising in comparison with the state of the art, but they still present some flaws when onset detection scheme does not provide a robust basis for rhythm tracking.

This algorithm was initially designed to deal with discrete events extracted by an onset detector. As Scheirer argues in [START_REF] Scheirer | Tempo and beats analysis of acoustic musical signals[END_REF], beat-trackers should work directly with audio data when dealing with acoustic signals rather than relying on some score-like representations as state-of-the-art onset detector have difficulties with soft and non-percussive onsets. We indeed observed the bad tracking results exactly for those signals where onset detection was inaccurate.

One solution would be to avoid actual onset detection and to use a stream of continuous descriptors of the signal rather than some discrete events. As a matter of fact, the multi-agent mechanism is a mean for exploring a space of possible interpretations, be it a discrete or continuous space. The capacity of prediction of the agents would allow putting emphasis on small chunks of the continuous descriptors. Agents would integrate these description functions with a lower complexity than the comb filterbank of Scheirer.

The stream of continuous descriptors might include energy envelope in frequency bands or some typical low-level descriptors but also some higher level descriptors from other modules such as a chord change probability function or a pitch deviation function.

In this design, be it continuous or discrete, the task of beat tracking reduces essentially to a good choice of descriptors of the signal. The system has the advantage of being modular allowing the input of any combination of descriptors of the signal.

Figure 1 .

 1 Figure 1. Overall system block diagram.

Figure 2 .

 2 Figure2. Tolerance window around a prediction. At date n, event e n is associated to the prediction, but at date n+1, event e n+1 replace e n as it has a more important accentuation (the height of the arrow meaning here the importance of the overall accentuation of the event).

Table 1 .

 1 Detailed results on the most relevant music database (RWC) used in this paper

		Songs	ρ
	Complete corpus	90	72.52%
	Popular	3	91.66%
	Ballads	3	96.69%
	Rock	3	96.64%
	Heavy-metal	3	73.66%
	Rap/Hip-hop	3	98.55%
	House	3	95.27%
	Techno	3	89.52%
	Funk	3	97.16%
	Soul/R&B	3	99.05%
	Big band	3	86.87%
	Modern jazz	3	26.37%
	Fusion	3	87.59%
	Bossa nova	3	53.18%
	Samba	3	100.00%
	Reggae	3	73.60%
	Tango	3	31.33%
	Baroque	4	53.03%
	Classic	3	25.21%
	Romantic	3	51.53%
	Modern	2	36.68%
	Brass band	3	56.56%
	Blues	3	83.28%
	Folk	3	48.40%
	Country	3	75.44%
	Gospel	3	85.19%
	African	2	17.28%
	Indian	2	66.79%
	Flamenco	3	59.29%
	Chanson	1	28.41%
	Traditional Japanese	3	95.22%
	Japanese Folk Min'you	2	64.37%
	Ancient Jap. Gagaku	1	39.08%
	A cappella	1	22.46%


Rosemary A Fitzgerald
email: r.a.fitzgerald@lancaster.ac.uk

Adam T Lindsay

TYING SEMANTIC LABELS TO COMPUTATIONAL DESCRIPTORS OF SIMILAR TIMBRES

This paper asserts the importance of using multiple representations when examining computational descriptions of any musical percept. We examine the existing stateof-the-art among the MPEG-7 timbre descriptors, and by choosing oboe timbre as our domain we investigate their application to timbres from the same instrument. Using a dual approach, correlating perceptual information with a wide range of computational descriptors, we propose potential extensions to MPEG-7 representations. By eliminating the need for generality across instruments, we believe we can reach a higher level of semantic representation within timbre domains from a single instrument class.

INTRODUCTION

Despite past efforts from standards bodies, MPEG-7, the first edition of which was published by ISO in 2002, standardises some aspects of musical instrument timbre. We look at the details of such an effort, examining the representation space that the standard spans. We then outline research to expand that representation space, based on psychological experiments and a wider range of signal processing. We expect to be able to reach a wider, more flexible range of representations than are currently available, and to allow for more sophisticated reasoning on musical timbre.

TIMBRE DESCRIPTION

Due to its complex multidimensional nature, timbre is still poorly understood and difficult to manipulate in a controlled, scientific way. The principal reason for this is that there is no universal definition of timbre. There are no standard units from which timbre may be quantified; any description given is verbal or phenomenological. Such qualitative descriptions have been used to compare the verbal attributes of orchestral instruments [START_REF] Kendall | Verbal attributes of simultaneous wind instrument timbres: II. Adjectives induced from Piston's Orchestration[END_REF], [START_REF] Kendall | Perceptual and acoustical features of natural and synthetic orchestral instrument tones[END_REF] in attempts to gain greater understanding of timbre.

Many definitions of timbre have centred upon trying to construct a constitutive definition. This has been found as far back as early Chinese civilizations who developed sophisticated written definitions of timbre, based on a classification of material sources such as metal, stone, clay, skin, silk threads, wood, gourd and bamboo [START_REF] Hornbostel | Classification of musical instruments[END_REF]. In a musical sense, the tonal quality characterising a sound can have many forms. Timbre may either refer to a specific family of tones (e.g., woodwind or double reed), or to an instrument (e.g., oboe). It may also be applicable to the perceived quality of a specific tone (e.g., a dark or a nasal oboe tone) and is sometimes used when describing different instrumental combinations when detailing the orchestration of a musical work.

The large number of variables involved in describing timbre leads to definitional vagueness. This is perhaps most evident in the frequently quoted statement by the American National Standards Institute (ANSI) in which timbre is described as "… an attribute of auditory sensation in terms of which a listener can judge that two sounds are similarly presented and having the same loudness and pitch are dissimilar'' [START_REF] Ansi | American national standard: Psychoacoustical terminology. timbre[END_REF]. It has been suggested that the vagueness of the definition may, perhaps, be related to the multidimensionality of the phenomenon [START_REF] Plomp | Aspects of Tone Sensation: A Psychophysical Study[END_REF], which can perhaps be related to a number of perceptual attributes, described in the footnote to the ANSI definition. The footnote states that, "… timbre depends primarily upon the spectrum of the stimulus but it also depends upon the waveform, the sound pressure the frequency location of the spectrum and the temporal characteristics of the stimulus." [START_REF] Ansi | American national standard: Psychoacoustical terminology. timbre[END_REF].

A PROGRESSIVE APPROACH TO TIMBRE

Staircase model of perception

When studying the computational interpretation of multimedia, we have found it helpful to consider the progression of the computational process from expression (signal) to description (meaning) by imagining several steps on a staircase proceeding upwards from a signal towards meaning. This is very much inspired by Marr's [START_REF] Marr | Vision[END_REF] representational framework laid out in his seminal work. We abstract from his reliance on the primal and 2-½ D sketches to a more extensive survey of what may plausibly happen along the lines of human perception and understanding. One instance is illustrated in Figure 1.

We use this staircase as an instructional model to reveal multiple representations inherent in the computational analysis of timbre, the assumptions involved in doing such analysis, and to point the way forward for refined processing. The model, in brief, attempts to expose the steps -whether perceptual or computational, human or computer -taken in response to an external stimulus. Each step is conceivably a representation slightly more abstract than the previous one. At each step, new knowledge, whether explicit, implicit, or algorithmic, is added to the previous representation. Once the representation reaches the top level, the model may be iterated, whether refining symbolic information (the jump to segmentation) or reinterpreting (potentially a segment of) the signal (the jump to the signal). It is hoped that the example below will further clarify the model. Figure 1. A generic series of steps taken to extract "meaning" from a signal. An interpretation may be iteratively re-segmented and "chunked" into larger items of meaning.

Musical Instrument Timbre Comparisons in MPEG-7

The MPEG-7 activities represent an effort to establish a standard for computational descriptions of multimedia content. For audio in particular, there is an aspiration to derive meaning via signal processing. Although the official stance throughout the developing standard is that there is no preference as to whence a description arises (e.g., hand-annotated or computationally derived) it is clear throughout the Audio and Visual parts of the standard that the chosen representation favours a signalprocessing method of feature extraction. We examine the timbre descriptors from the audio part of the standard to see where the processing assumptions are made, and how they fit the above staircase model of content understanding.

The application-oriented description schemes concerned with musical instrument timbre within the Audio part of the MPEG-7 standard draw upon research in musical perception and psychophysics that attempts to determine what features of a given musical sound distinguish it from other sounds at the same loudness and pitch [START_REF]Multimedia Content Description Interface Part 4: Audio," International Standard 15938-4[END_REF]. The scheme draws upon low-level descriptors that have direct analogues within signal processing. The relationship between these schemes and the description output from a system implementing the descriptors is worthy of comment, but it is more suitable to begin by examining of the types of multimedia content that are expected to be input to the system, and what that the implications of these are.

The MPEG-7 Timbre tools describe perceptual features of "monophonic, non-mixed, non-layered instrument sounds'' [START_REF]Multimedia Content Description Interface Part 4: Audio," International Standard 15938-4[END_REF]. This places an explicit limit on the types of signal they may describe. The range of all possible signals is first limited to musical instrument sounds, and is further constrained to be of a solo instrument playing a note in isolation. Thus, a perceiver (or possibly another computer system) is required to ascend the staircase via the physical and perceptual steps to determine that this is indeed a monophonic signal. Then, that note must be either isolated from its neighbours in a temporal stream, or it must be determined that this has already been done (segmentation). At this point, the instrumental note will have been labelled with a token indicating that it is a sound suitable for timbre description.

Following the initial assumptions discussed immediately above, the clause in the standard posits four classes of musical instrumental sounds that may be described: non-sustained sounds; sustained, harmonic, coherent sounds; sustained, non-harmonic, coherent sounds; and sustained non-coherent sounds. The standard currently accommodates only the first two of these classes of sounds, which then form perceptual "spaces" in which sounds are compared. Thus, the sounds that have been segmented are interpreted and placed into a context (and reaching the top of the staircase) before they are even analysed by a computer for their intended use -"lowlevel" timbral descriptors.

By deciding which class of musical sound the signal belongs to, one decides which of the seven possible lowlevel temporal and spectral features are used to describe the sound. Two of the possible descriptors applicable to sustained, harmonic, coherent sounds are log attack time and harmonic spectral centroid. These are physical features of signals that are proxies for the perceptual features of "attack" and "brightness", respectively. The computational details are irrelevant here, but it can be observed that both features rely on another ascent up the staircase (e.g., segment through the temporal signal to note the beginning and the loudest initial part of the sound) and then yield a physical measure that finally approximates a perceptual comparison. The final representation consists of four or five quantitative values. These values may then be compared in a perceptually scaled space to judge the perceptual similarity between two sounds.

The above description is not intended to denigrate the technology behind the MPEG-7 Timbre descriptors; those descriptors are the state of the art, backed up by experimental evidence. The intention here is rather to expose the series of assumptions that underpin the simple application of a pre-defined group of descriptors for a sound. A supervisory system (a human, in most cases) applies various analytical processes at each step up the staircase, to arrive at a classified segmented sound. In other words, the computational techniques that purport simply to traverse the space between signal and perception actually embody techniques that incorporate segmentation and tokenisation as intermediate steps in order to compute their corresponding features.

Multiple representations are key

We see that although the final product is rather terse, typically with five parameters describing a note, there are many points along the way to that product that are plausible representations. We believe that further representations may be derived from the MPEG-7 timbre descriptors, both supplementing them at the same level and building upon them at higher levels. This is entirely consistent with MPEG-7 audio, trading off generality for descriptive power. With a wider range of salient representations, there is more flexibility in processing, as well as the possibility of a more intuitive interface for a user navigating timbre-space than navigating along axes such as "Harmonic Spectral Variation."

A PROGRAMME FOR EXPANDED TIMBRE DESCRIPTION

In order to prise open the series of perceptual and computational assumptions for ourselves we examined how musically-trained listeners tie perceptual similarities to descriptive labels. We now describe a perceptual experiment that explores perceptual similarities and verbal attribute magnitude estimation (VAME) of oboe tones from two different performers.

Psychoacoustic experiments

To obtain the perceptual dissimilarity, 32 musicallytrained subjects were asked to make judgements of dissimilarity on a scale from 0-100. Twenty-four isolated tones digitally recorded at six different pitches (C4, F4, A4, C♯5, A♯5, F6) and two different dynamic levels mf, ff) by two oboists (A and B) from different schools of playing (British and American) were used as a primary data set. The recordings were premised on the use of real-world sounds, which involves treating the oboist, reed and instrument as a whole mechanism and recording the tones in an acoustically live room. The tones in the data set were equalised for duration (by adding a false decay) and amplitude as they would be compared against each other experimental conditions. Eight adjectives, taken from the principal components analysis by Kendall and Carterette [START_REF] Kendall | Verbal attributes of simultaneous wind instrument timbres: II. Adjectives induced from Piston's Orchestration[END_REF] in their studies on wind instrument dyads, were used as a basis to assess their suitability for describing oboe timbre. Kendall and Carterette [2] also used the same group of eight adjectives to assess the verbal characteristics of natural and synthetic single instrument tones. In their study these adjectives were found to describe four different factors/dimensions of the wind timbres: strong; tremulous; light (factor 1); nasal; rich (factor 2); brilliant; ringing (factor 3); reedy (factor 4).

To collect the VAME data for the study of oboe timbre subjects were asked to rate the magnitude of the verbal attributes for each tone after hearing it played once. They achieved this in the same manner as the perceptual similarity scaling by means of a computer-based moving slider, which converted the positioning by the subject to a value scale of 0-100. The poles of the rating scale were labelled "not adjective" -"adjective". The order of the presentation of the tones within each set of verbal attributes and dissimilarity comparison was randomly assigned.

Results

Perceptual differences are revealed between most of the tones played by the two oboists as shown by the Multidimensional Scaling (MDS) representation in figure 2. There are significant differences between tones for different performers across both the same loudness level (although some confusion occurs for pitch C4) and different loudness levels. Perceptual differences are also revealed within a performer: there are significant differences between tones across different loudness levels (although some confusion occurs for pitch C♯5). Using analysis of variance (ANOVA), individual VAME relationships are revealed for each performer as there are significant differences in the data between the two oboists. This is evident for tones at the same loudness level and different loudness levels. VAME relationships are revealed within a performer as there are differences in judgments between tones for the same performer across different loudness levels, (except for pitch F6 for oboist B where VAME judgments were almost identical.)

Principal components analysis (PCA) with Varimax rotation (Kaiser normalisation) was performed on the VAME ratings for all tones. Figure 3 shows the threedimensional solution for the PCA loadings for the mean verbal attribute ratings across pitches. Three factors, those with eigenvalues over 1, account for 54.055% of the variance. Factor 1, which accounts for 20.586% of the variance could, perhaps, be named the 'Power' factor as the attributes Strong, Rich and Brilliant rate the most positively, whilst the attributes Light and Nasal load negatively. The 'Power' factor was found by Kendall and Carterette [START_REF] Kendall | Verbal attributes of simultaneous wind instrument timbres: II. Adjectives induced from Piston's Orchestration[END_REF] onto which the attribute Strong loaded positively. Factor 2 accounts for 17.025% of the variance and could, perhaps, be named as the 'Vibrancy' factor for the attributes Tremulous, Ringing and Brilliant rate the most positively, whilst Rich and Reedy load negatively. Factor 3 accounts for 16.444% of the variance and could, perhaps, be labelled the 'Pinched' factor. Attributes Nasal and Reedy rate the most positively, whilst Light and Brilliant load negatively. The averaged VAME ratings suggest that subjects are not confusing the attributes Nasal and Reedy and that the negative loadings of Light and Brilliant are being judged almost as opposites. Although only two dimensions are found for the dissimilarity scaling solutions (see figure 2), it is suggested that the first two factors of 'Power' and 'Vibrancy' may account for these dimensions. It is possible that the 'Power' factor could be used as the label on the dimension differentiating the tones by dynamic, whereas the 'Vibrancy' factor is used to differentiate between oboist. This is reflected in the results of the ANOVA (as oboist A's tones were judged to be higher over all pitches on ratings of Tremulous and Ringing than those from oboist B).

The Kendall and Carterette adjectives are suitable for describing oboe timbre as subjects seem to be using each adjective differently. One drawback to having a limited number of results for the dissimilarity rating experiment is that only a two-dimensional solution could be plotted, whereas the PCA analysis of the VAME data suggested that three or four dimensions are needed to differentiate results. In summary, VAME ratings for each attribute generally distinguish between oboists at all dynamic levels. At extremes of pitch the VAME ratings are more similar for each performer suggesting that their tones are being perceived as being more alike.

Signal Processing

To quantitatively examine the dimensions of the timbre space obtained from the perceptual experiments, we have developed a timbre analysis toolbox to extract spectral and temporal features of tones. (For details of the descriptors please see [START_REF] Fitzgerald | Performer-dependent dimensions of timbre: identifying acoustic cues for oboe tone discrimination[END_REF].) The toolbox is implemented in Scilab (an open source signal processing environment) [10]. The analysis functions that extract both the spectral and temporal timbral features have their origins in many previous studies on timbre. Vibrato tracking features using a method of autocorrelation are also implemented. Also included in the toolbox are implementations of the timbre descriptors used in the MPEG-7 specification [START_REF] Peeters | Instrument sound description in the context of MPEG-7[END_REF] and the Kendall and Carterette studies [START_REF] Kendall | Verbal attributes of simultaneous wind instrument timbres: II. Adjectives induced from Piston's Orchestration[END_REF][2].

Results

The spectral descriptors correlate the above perceptual results with the results of the toolbox, thereby evaluating the performance of the MPEG-7 descriptors. Three main results can be summarised thus: spectral centroid and spectral deviation (from MPEG-7 descriptors) correlated best with the 'Power' factor, spectral flux and centroid variability (from Kendall and Carterette descriptors) having less significant correlations; spectral spread and spectral variation (from MPEG-7 descriptors) with the 'Vibrancy' factor, and spectral centroid and spectral variation with the 'Pinched' factor. Significant attribute correlations are Rich with spectral centroid and spectral deviation (positive), Ringing and Tremulous with spectral spread (negative), Light for spectral flux (positive), Rich and Reedy for spectral flux (negative), Rich for the centroid variability (negative).

Further investigation is recommended using both oboe-specific verbal and computational descriptors, e.g., vibrato variations, and formant structure, to obtain more accurate features of timbre relating to the oboe that correlate with instrument-specific verbal descriptors. Future work, leading to expanding the MPEG-7 timbre descrip-tor set, needs to examine timbres from a number of different instrumen.

CONCLUSION

We have seen that multiple representations of timbre are extremely helpful in terms of analysis, and that one should be aware of all of the assumptions going into a computational signal processing system. The semantic descriptions in this current work were used to describe all instruments and were taken, by Kendal and Carterette [START_REF] Kendall | Verbal attributes of simultaneous wind instrument timbres: II. Adjectives induced from Piston's Orchestration[END_REF], from Piston's work on Orchestration [START_REF] Piston | Orchestration[END_REF]. We have seen a way to move beyond the current way of thinking within the MPEG-7 standard, by providing more semantic 'axes' for navigating within the representational space formed by computational timbre descriptors. Most significantly this research highlights the importance of exploring timbres from the sameinstrument class to further examine relationships between perceptual and computational descriptors.

We believe that, with further verification, the 'Power,' 'Vibrancy,' and 'Pinched' groupings of descriptors could be used as a higher-level, oboe-specific description, atop the existing MPEG-7 timbre descriptors. This layering of descriptions is entirely consistent with both the MPEG-7 approach and the staircase model described above. The adjective groupings could be used to make a user interface more intuitive.

Future Work

With the continuation of the psychological statistics and signal processing analyses, there are many potential directions this research could take. The techniques used in this very restricted, single-instrument domain could be examined to see if they could be transferred to other instruments, or made more general again. The MPEG-7 standard may be enhanced with this further research, potentially included in a second version of the standard. In any case, we believe this research to add to the reper-toire of representations for anyone attempting to work with timbre -whether perceptually or computationally.

Figure 2 .

 2 Figure 2. Two-dimensional MDS solution for the oboist/dynamic tones (by pitch)

Figure 3 .

 3 Figure 3. Three-dimensional configuration of the verbal attributes across all tones and pitches for the principal components analysis.


Hui Peng
email: davidpeng1749@gmail.com

Extended-Delayed-Choice Experiments Supporting Causality

Keywords: delayed-choice experiment, causality, Mach-Zehnder-Interferometer, double-slit experiment, cross-double slit experiments, complementarity principle

The interpretations of the delayed-choice experiments suggest that photons "retroactively decide" to travel as particles or as wave from the time of its emission. To test the interpretations thoroughly, we propose and perform extended-delayed-choice experiments. The experiments are designed for easier to set up, carry out and reach meaningful results without ambiguous. The experimental results show the following: (A) the causality holds; (B) Bohr's complementarity principle is violated; (C) photons behave as particle before striking at output beam splitter/double-slit/crossdouble-slit; (D) the present does not determine the past retroactively, but determines the future.

of quantum mechanics. In reality, it contains the only mystery [of quantum mechanics]" [START_REF] Feynman | The Feynman Lectures on Physics[END_REF]. Moreover, the nature of photons truly puzzled Einstein. He wrote to M. Besso: "All these 50 years of conscious brooding have brought me no nearer to the answer to the question: What are light quanta?" [START_REF] Rashkovskiy | Is a rational explanation of wave-particle duality possible?[END_REF].

To study the behavior of the light, in 1978 and 1984, Wheeler proposed a series of thought experiments, delayed-choice experiments, which was designed to resolve the fundamental issues in quantum physics [START_REF] Marlow | Mathematical Foundations of Quantum Theory[END_REF]. One version is to perform experiments with Mach-Zehnder-Interferometer (denoted as MZI) by removing/inserting an output device, beam splitter (BS). Another version is to utilize double-slit apparatus. Those experiments attempt to decide whether a photon "senses" the experimental apparatus in experiment it travels through and adjusts its previous behavior to fit for any final experimental configurations. Wheeler's question is whether a time/position could be determined at which the photon made its "decision" to act as wave or as particle. A photon is described as a "person" who can "sense", "make decision", even "retroactively decide" that challenges the causality. The MZI version has three configurations:

First configuration (Figure 1a): without an output BS2 in place. Each photon behaves as particle detected by D1 and D2 respectively. Second configuration (Figure 1b): with BS2 in place. The light beam creates wave distribution on D2. Third configuration (Figure 1c): inserting/removing BS2, while photon is flying and before arriving at the position of BS2.

The regular interpretations are: [START_REF] Ananthaswamy | Through Two Doors at Once[END_REF] for first configuration: photon should behave the same, as particle, from the time of its emission to the time of its detection; [START_REF] Robinson | The Last Man Who Knew Everything[END_REF] for second configuration: before arriving at and after passing through BS2, photons should behave the same, as wave, from the time of its emission to the time of its detection; (3) for third configuration: photons would behave as wave before arriving at the position of BS2 as BS2 were there; namely, photons would behave as wave from the time of its emission to the time of its detection, even there was no BS2 at the time of its emission, which challenges the causality; it is stated that photons, as a "person", made "retroactive decision".

Moreover, there are similar interpretations of delayed-choice-double-slit experiments [START_REF] Greenstein | The Quantum Challenge: Modern Research on the Foundations of Quantum Mechanics[END_REF]: [START_REF] Ananthaswamy | Through Two Doors at Once[END_REF] when there is no double-slit, photon behaves the same, as particle, from the time of its emission to the time of its detection; [START_REF] Robinson | The Last Man Who Knew Everything[END_REF] when there is a double-slit, before arriving at and after passing through the double-slit, photons would behave the same, as wave, from the time of its emission to the time of its detection; the experimenters conclude that each photon "decides" to travel as a wave; (3) inserting the double-slit, when a photon is flying, then the photon "decides" to travel as a wave distribution, from the time of its emission to the time of its detection. The standard interpretation of the delayed-choice-double slit experiment states that photons made "retroactive decisions", which challenges causality.

Although the experimental setups have been difficult, the intensive efforts focus on both experiment and interpretation, for recent articles, see, for example, [7] [8].

To test the above interpretations of MZI-delayed-choice experiments, the variety of delayed-choice experiments have been proposed and performed, such as quantum eraser delayed-choice experiment [START_REF] Kim | A Delayed-choice Quantum Eraser[END_REF] [10], which challenges the causality. Recently, the extended-MZI-delayed-choice cross-double slit experiments were carried out [11] [12]. The basic structure of the extended-MZI apparatuses is to combine normal MZI apparatus with double-slit and/or cross-double-slit apparatuses [START_REF] Peng | Observation of Cross-double-Slit Experiments[END_REF], which supports the causality. In this article more experiments are presented, which supports the causality.

2D-Cross-double Slit Experiments

Cross-Double Slit Apparatus

The cross-double slit apparatuses are utilized throughout the article, let us briefly review the cross-double slit apparatuses and experiments. To study the mystery of the double slit experiments and wave-particle duality, cross-double slit experiments have been proposed/performed [11] [13]. Figure 2 shows the diaphragms containing double slit crossing double slit with same/different spacing between paired slits, hereafter, denoted all as "cross-double slit". Experiment-1 (Figure 4): Double slit crossing double slit and its interference pattern.

Figure 4 Figure 5 Experiment-2 (Figure 5): Tilt double slit crossing double slit and its interference pattern.

Experiment-3 (Figure 6): Three double slits crossing at the same spot and its interference pattern. The angle between the tilt double slit and the vertical double slit is 45 degree.

Figure 6

Figure 7 Experiment-4 (Figure 7): Three double slits crossing at the same spot and its interference pattern.

The angle between the tilt double slit and the vertical double slit is 15 degree.

Experiment-5 (Figure 8): Three double slits crossing and its interference pattern.

Figure 8 Figure 9 Experiment-6 (Figure 9): Three crossing double slits and its interference pattern. The angles of tilted double slits are 15 and 30 degrees respectively.

Experiment-7 (Figure 10): Four crossing double slits and its interference pattern. Experiment-8 (Figure 11): Five crossing double slits and its interference pattern.

Experiment-9 (Figure 12): Six crossing double slits and its interference pattern.

Figure 13 Experiment-10 (Figure 13): Rotation Invariance of double slit/cross-double slit.

The combination of Figures 10, 11 and 12 suggests that the double slit and cross-double slit have rotation invariance around their normal vectors. If we increase the number of double slits that intersect at the same spot, the shape of the intersection will approach a circular disc. To rotate either a double slit or cross-double slit around its normal vector, each slit is tangent to the intersection and forms disc-3, which is surrounded by ring-2. The final pattern can be obtained from the apparatus of disc-3 with a diameter that is equal to the spacing between two slits, and ring-2 of the width that is equal to the width of a single slit. We argue that Figure 13 indicates that double slit and cross-double slit have rotationinvariance around their normal vector.

Double Slit Crossing Double Slit: Different Spacing

Experiment-11 (Figure 14). Double slit crossing double slit of different spacing (the left of Figure 14)

and its interference pattern (the right of Figure 14). It is challenge to interpret the experiments in Section 2 consistently.

Delayed-choice Double Slit/Cross-double slit Experiments: Supporting Causality

The standard MZI version of the delayed-choice experiment is shown in Figure 1. The standard interpretation states that photons made "retroactive decision", which challenges the causality. Next, we perform the experiments to test the interpretation of "retroactive decision".

Rule-1, Rule-2 and Postulate

To analyze delayed-choice phenomena, we propose and test two rules and postulate.

Rule-1: When a single input beam of photons travelling as particles strikes a BS, reflection and transmission attributed to the BS do not change the behavior of photons, namely, before and after reflection and transmission, photons behave as particles.

Example: when a beam of photons outputted from a BS behaves as particles, then the input beam of photons behaves as particles, while the other output beam of photons behaves as particles.

Rule-2: When a beam of photons travelling as waves strike a BS, reflection and transmission attributed to the BS do not change the behavior of photons, namely, before and after reflection and transmission, photons distribute as waves.

Postulate: Before passing through a diaphragm of double slit/cross-double slit, photons emitted from the source behave as particles.

Note that hereafter, to clearly show the results of experiments and draw conclusions, we arrange apparatuses to show the patterns created by photons passing through different paths on the same large screen/detector, a long white board.

We demonstrate Rule-1, Rule-2 and Postulate experimentally first.

Experiment-13 (Figure 16): Testing Rule-1

Experimental setup: A laser source emits photons that are partially reflected by BS1 and arrive at detector 1 (D1) and partially transmitted though BS1 and arrive at D2 (Figure 16a).

Conclusion:

Rule-1 is proved. Photons detected on both D1 and D2 have the same particle nature.

Namely the particle nature is not changed by either being reflected by BS or passing through BS.

Experiment-14 (Figure 17): Testing Rule-2

Experimental setup: A laser source emits photons that travel through a slide of cross-double slit.

Then, the photons are partially reflected by BS1 and partially transmitted through BS1, and arrive at D1 and D2 respectively (Figure 17a). The slide is a cross-double slit. Conclusion: Thus, according to Rule-1, photons reflected by BS1 and BS2/M1/M3 traveling towards slide-1 and slide-3, respectively, behave as particles. Namely, photons behave as particles before arriving double slit/cross-double slits. Postulate is proved.

On the other hand, D1 and D3 show interference patterns. Particle nature and wave distribution coexist in "the same experiment". To study wave-particle duality and complementarity principle, let us define the term, "the same experiment", as followings: when there is "only one source" emitting light/photons, regardless of the configurations of the experimental apparatus, the experiment is defined as the same experiment. (1) Based on Rule-1, photons passing through B1 and B3 behave as particles, thus, before arriving at slide-2 and slide-4, photons behave as particles.

(2) Slide-2 and slide-4 convert photons' particle behavior (before arriving) to wave distribution (after passing through) respectively. The slide determines the behavior of photons only when photons pass through it but not the behavior before passing it. Thus, slide-2 and slide-4 can be either removed or inserted any time during the experiment, which makes the delayed-choice experiments easier to perform.

(3) The particle nature and wave distribution coexist in the same experiment.

(4) There is no "retroactive decision" phenomenon. The standard interpretation of the delayed-choice experiments is challenged at the macroscopic level.

It would be interesting to perform the experiment by emitting photons one at a time.

Extended-MZI Version of Delayed-choice Double Slit/Cross-Double Slit Experiments

In the interpretations of delayed-choice experiments, photons are described as a "person" to "decide" their own behavior. To explore the interpretations further by MZI configurations, we combine the MZI and double slit/cross-double slit in the same delayed-choice experiment. The purposes of the experiments in this section are to test (1) whether photons behave differently when traveling along different paths; [START_REF] Robinson | The Last Man Who Knew Everything[END_REF] what determines the behaviors of photons under the circumstances of with and without an output slide of the double slit/cross-double slit.

To be able to visually observe the experimental result, we modify the MZI delayed-choice experiment by replacing BS2 with a slide of double slit and/or cross-double slit.

Experiment-20-open (Figure 23): Extended MZI delayed-choice experiment (1)

Experimental

Step: Figure 23a is a standard open-MZI. We Extended MZI as shown in Figure 23b. Experimental Setup: Adding slide-4 between BS4 and M4 (Figure 25a). Observation (Figure 28b): D3 and D4 still show the same wave distribution patterns, while D1 and D2

show the wave distribution due to slide 1.

Conclusion:

Remove/inserting of slide-1 does not affect the behavior of photons detected by D3/D4.

Thus, experiments-21 to 22 challenge the retroactive interpretation of delayed-choice experiments.

Experiment-23-closed (Figure 29): Placing slide2 between BS3 and M2, slide1 between BS4 and D1, slide4 between BS4 and M4 (Figure 29a). We referred it as Extended-closed-MZI. Therefore, the present does not determine the past retroactively but determines the future.

The Extended-MZI delayed-choice double slit/cross-double slit experiments can be performed without sophisticated equipment. The conclusion of the experiments can be observed by the naked eye.

The double slit experiments are extended to the cross-double slit experiments, which are more complex. We suggest that an interpretation must be able to explain all double slit and cross-double slit experiments consistently.

Testing Interpretation of Delayed-Choice Double Slit/Cross-Double Slit Experiment

Now, we modify the closed-MZI to test the interpretation that states that when a photon is flying, inserting BS2, photons would behave as waves from the time of its emission to the time of its detection, even if there was no BS2 at the time of its emission. The key is to test whether photons behave the same before arriving at the position of BS2 for the situations of with and without BS2.

For visual observation, we replaced BS2 with two slides of double slit/cross-double slit.

Experiment-24-open (Figure 32): Extended-open-MZI delayed-choice cross-double slit experiments.

Experimental Setup: we insert BS3/slide3/D3 and BS4/D4 between the source and BS1 (Figure 32a).

Summary and Conclusion

First, one might question: whether an output BS, BS2, and a slide of double-slit/cross-double-slit are equivalent in a delayed-choice experiment? To answer, let's analyze their effects in a delayedchoice experiment. Consider a regular MZI-delayed-choice experiment, as shown by Figure 1. With BS2, D1 and D2 detect the wave-like distribution, while without BS2, D1 and D2 detect the particle nature of photons.

With a slide of double-slit, instead of BS2, D1 and D2 detect the wave-like distribution, while without the slide of double-slit, D1 and D2 detect the particle nature of photons.

Therefore, for the purpose of the delayed-choice experiments, an output BS, BS2, is equivalent to a slide of double-slit/cross-double-slit.

With and without an output device, either a BS or a double-slit, photons do not retroactively decide their behavior. Thus, during delayed-choice experiments, an output device, either a BS or a double-slit, can be removed or inserted any time without affecting the conclusion of experiments.

The function of an output device is to affect the behavior of photons after passing through it, but not before.

The causality holds.

The photons behave as particle before arriving a double-slit/cross-double-slit.

Bohr's complementarity principle is challenged.

We suggest that the particle nature of photons always exists in experiments, thus is primary nature. The wave distribution exists only for special situations, thus is secondary.

It is a challenge to consistently interpret the experiments in this article.

Appendix (A-1) Photons' Behavior Depending on Angle of BS: A Challenge to Photons

The delayed-choice experiments attempt to decide whether a photon "senses" the experimental apparatus and adjusts its previous behavior to fit for any final experimental configurations. In a standard MZI version of delayed-choice experiments, "the final experimental configurations" are two situations:

either with BS2 or without BS2. The photon's decision is based on these two situations.

However, we emphasize that the photon's decision must also base on the angle of BS2. For clearly showing, we schematically draw different cases (Figure A1). Figure A1a shows that BS2 is at 0 0 and Figure A1b shows that BS2 is at 90 0 . For these two situations, photons directly pass through BS2 without reflection and land on D1 and D2 respectively, thus no interference pattern shown on D2, i.e., the photons behave as particle, even there is BS2. Figure A1c shows that the angle of BS2 is between 0 0 and 90 0 , but not 45 0 . The standard MZI delayed-choice experiments studied the situation of BS2 at 45 0 , for which there is the interference pattern, the photons distribute as wave; Figure A1d shows that an experimenter may place BS2 at many different angles, and, at those angles, the photons behave as particle.

A photon travels along one of two paths randomly. BS2 should be placed with a random angle, i.e., BS2 can be placed at any angle between 0 0 and 90 0 . If, in a series of experiments, gradually reducing the angle of BS2 from 45 0 to 0 0 in each experiment, for example, placing BS2 at angles of 45 0 , 35 0 , 25 0 , 15 0 , 5 0 and 0 0 respectively, a perfect interference pattern gradually becomes the pattern that shows particle nature of photon.

Surprisingly, the nature of photon is determined also by the angle of BS2. Namely the experimenter determines the behavior of photons.

As a "person", photon "knows" that when the angle of BS2 is at 45 0 , it should behave as wave.

Photon faces two "tasks": First, a photon needs to "sense", real time, whether there is BS2; Second, if the photon senses BS2, then the photon needs to "measure" the angle of BS2. After completing those two tasks, the photon can "make decision" on its behavior: either as particle or as wave distribution.

Let us consider a simple experiment, an experimenter blindly places BS2 with an arbitrary angle.

After BS2 is placed, repeating the experiment many times, then the final pattern emerges, which is determined by the angle of BS2. Only when the experiment is completed, the experimenter knows whether the angle of BS2 is equal or not equal to 45 0 .

The experiment with BS2 at a random angle processes as the following:

1) Emitting photon-1 that lands at a point either on D1 or on D2 without forming a pattern; photon-1 behaves as it knows the angle and behaves accordingly. 2) After photon-1 landed either on D1 or on D2, emitting photon-2 that lands at a point either on D1 or on D2 without forming a pattern; photon-2 behaves as it knows the angle.

3) Emitting photon-3, photon-4, and so on.

After repeating many times, pattern (s) emerges gradually either on D2 or on both D1 and D2. If the experimenter observes a standard interference pattern, then he/she knows that the angle of BS2 is 45 0 ; if observes two groups of dots on D1 and D2 respectively, the angle is not 45 0 . Let us define the "standard interference pattern": the interference pattern can be described by the standard optical formular.

It seems that each of photons "knows" not only the existence of BS2, but also BS2's angle. So, if a photon, was described as a person, can "sense" experimental apparatus and decide its behavior retroactively, then the photon must be able to "measure" the angle of BS2.

The question is how would a photon "measure" the angle of BS2 in the entire experiment and behaves accordingly?

Figure A2 Extend-MZI with Entangled Photons

The apparatus is similar to that of experiment-1, except that source emits a pair of two entangled photons at a time. One of entangled photons, denoted as photon-1, travels to either double-slit D4 or detector D1, while another entangled photon, denoted as photon-2 goes to either D2 or double-slit D3.

We have six possible results:

(1) Photon-1 and photon-2 behave as particle, when they arrive D1 and D2 respectively;

(2) Photon-1 and photon-2 behave as particle, before they passing through D4 and D3, e.g., double-slit, respectively; (3) Photon-1 behaves as particle, when it arrives D1; while photon-2 behaves as wave, when it arrives D3; the question is whether entangled photon-1 and photon-2 should have the same nature? (4) Photon-1 behaves as wave, when it arrives D4; while photon-2 behaves as particle, when it arrives D2; the same question is whether entangled photon-1 and photon-2 should have the same nature?

(5) Photon-1 behaves as wave, when it arrives D4, and behaves as particle, when it arrives D1; the question is whether entangled photon-1 have the same nature, when travels along the common path: source-BS4? (6) Photon-2 behaves as wave, when it arrives D3, and behaves as particle, when it arrives D2; the question is whether entangled photon-2 have the same nature, when travels along the common path source-BS3? Thought experiment-A1 implies that Photons behave differently along different paths. Photons behave differently along the same path.

We propose Extended-EPR paradox: should the natures of entangle photons be the same or not?

If the natures of entangle photons should be the same, then the interpretation of thought experiment-A1 leads to a paradox.

If the natures of entangle photons can be different, then thought experiment-A1 discloses a novel property of entangled photons and test the original EPR paradox at a fundamental level. (2) Bell/GHZ experiments test EPR paradox at the level of intrinsic property [START_REF] Bell | On the Einstein-Podolsky-Rosen paradox[END_REF] [16] [START_REF] Greenberger | Going beyond Bell's Theorem[END_REF]; and

(3) thought experiment-A1 of Extended-EPR tests the EPR paradox at the level of physical nature, a fundamental level [START_REF] Peng | Novel Quantum Experiments[END_REF]. The thought experiment-A2 is similar to thought experiment-A1, except that D2 is double-slit, and D3 is a detector. The discussion should be similar accordingly.

Thought experiment-A3 (Figure A4): The apparatus consists of source emitting a pair of two entangled photons at a time. One of entangled photons, denoted as photon-1, travels to either detector D4 or D1 or D2, while another entangled photon, denoted as photon-2, goes to either D3 or D1 or D2.

Note there is phase shifter (not show in Figure A4) to change phase so that photon-1 and photon-2 would interfere at D2.

We would observe the following: Photon-1 and photon-2 behave as wave, when they arrive at D1 and D2 respectively. Namely, according to the regular interpretation of delayed-choice experiment, D2 and D1 demand that photon-1 and photon-2 behave as wave from the time of its emission to the time of its detection.

While photon-1 and photon-2 behave as particle, when they arrive at D4 and D3 respectively.

The thought experiment-A3 will test whether photon-1 and photon-2 behave as particle or as wave or as both?

The same question arises whether entangled photons should have the same physical nature?

The possible observation of this thought experiment would test EPR/Extended-EPR paradox at physical nature level.

Figure 1

 1 Figure 1 Normal Setup of Delayed-Choice Experiment

Fig. 2 2 . 2 .

 222 Fig. 2 Cross-double slit with same/different spacing

Figure 3

 3 Figure 3 Young's double slit experiment

Figure 14 Figure 15 Experiment- 12 (

 141512 Figure 14Figure15Experiment-12 (Figure15): Double slit crossing double slit of different spacing and its pattern.

Figure 16 Testing Rule- 1 Observation

 161 Figure 16 Testing Rule-1 Observation (Figure 16b): both D1 and D2 show the images of the source.

Figure 17 Testing Rule- 2 Observation

 172 Figure 17 Testing Rule-2 Observation (Figure 17b): D1 and D2 show the same interference patterns.

  Figure 4 Observation

Figure 18 .

 18 Figure 18. Testing Postulate (1) Observations (Figure 18b): (1) D1 shows the image of the source; (2) D2 shows an interference pattern of the double slit. Conclusion: D1 shows the particle nature of photons. Based on Rule-1, photons passing through BS1 towards the slide behave as particles as well, namely before arriving at the slide, photons behave as particles. Postulate is proved. Experiment-16 (Figure 19): Testing Postulate Experimental setup (Figure 19a): Photons travelling through BS1 and BS2, reflected by M2 and strike at D2. Photons reflected by BS1 and passing through slide-1 strike at D1. Photons reflected by BS2/M1/M3 and then passing through slide-3 strike at D3. To show the difference, we use a crossdouble slit for slide-1 and a double slit for slide-3.

Figure 19

 19 Figure 19 Testing Postulate (2) Observations (Figure 19b): D2 displays the image of the source, i.e., photons passing through BS1 and BS2 behave as particles.

  Figure 10 Schematic drawing Figure 11 Slide-2

3. 2 .

 2 Figure 20 Double slit version of delayed-choice experiment (1) Observations (Figure 20b): D1 and D3 show the image of the source; D2 displays the wave distribution; D4 displays the wave distribution. Experiment-18 (Figure 21): Delayed-choice double slit/cross-double slit experiment (2) Experimental setup (Figure 21a): removing slide-4 (denoted as dashed slide-4 and dashed arrow).

Figure 21 Figure 22

 2122 Figure 21 Double slit version of delayed-choice experiment (2) Observations (Figure 21b): D4 shows the image of the source; D1 and D3 still show the particle patterns; D2 still shows the wave distribution. Experiment-19 (Figure 22): Delayed-choice double slit/cross-double slit experiment (3) Experimental setup (Figure 22a): removing slide-2

Figure 23 -Figure 24

 2324 Figure 23-open Extended MZI delayed-choice experiment (1) To study the behavior of photons travelling along the path of source-BS1-M1-D2 and source-BS1-M2-D1, M1 and M2 are replaced by BS3/D3 and BS4/M4/D4 respectively (Figure 23b). Observation (Figure 23c): All detectors show the images of the source. Experiment-20-close (Figure 24): Extended MZI delayed-choice experiment (1) Experimental Setup (Figure 24a): inserting slide-1.

Figure 25

 25 Figure25Extended MZI delayed-choice experiment[START_REF] Robinson | The Last Man Who Knew Everything[END_REF]

Figure 27

 27 Figure 27 Extended MZI delayed-choice experiment (3) Observation (Figure 27b): D1 and D2 show the image of the source, while D3 and D4 show the wave distributions due to slide-3 and slide-4, respectively. Experiment-22-closed (Figure 28): Extended closed-MZI delayed-choice experiment (3)

.

Figure 29

 29 Figure 29 Extended-closed-MZI and patterns Observation (Figure 29b): D3 shows the image of the source, while D1, D2 and D4 show the interference patterns created by slide 1, slid 2 and slide 4, respectively, so photons distribute as waves. Experiment-23-half-open-1 (Figure 30a): remove slide 2. We referred it as

Figure 32

 32 Figure 32 Extended MZI delayed-choice experiment (4) Observation (Figure 32b): D3 shows the wave distribution due to slide 3, D4 shows the image of the source, while D1 and D2 show the images of the source. Experiment-24-closed (Figure 33): Extended-closed-MZI delayed-choice experiment.

Figure 33

 33 Figure 33 Extended-closed-MZI and patterns Observation (Figure 33b): D3 still shows the interference pattern created by slide 3, while D4 still shows the image of the source. D1 and D2 show the interference pattern created by slide 2. The patterns on D1/D2 are changed from the images of the source to wave distributions. Conclusion: Removing/inserting silde 2 does not determine the behavior of photons from the source to BS1, namely, it does not retroactively determine the behaviors of photons from the time of its emission to the time of its detection. However, removing/inserting slide 2 does change the behaviors of photons after passing through slide 2. Taking into account Postulate, slide 2 can be removed/inserted at any time during the experiment, and there is no need to emit photons one at a time. The particle nature and wave distribution coexist in the same experiment. Experiment-25-closed (Figure 34): Extended-closed-MZI delayed-choice experiment. Experimental Setup: An alternative configuration is shown in Figure 34. We place slide 2 between M2 and M3 and slide 1 between M1 and D1.

Figure 34

 34 Figure 34 Extended-closed-MZI and patterns Figure 34b shows that each pattern is created independently and thus, is not affected by what happens in other paths.

.

Figure A1

 A1 Figure A1 Different angles of BS2 cause different patterns

(A- 2 )

 2 photons.

  Figure A3 Extended-MZI with Entangled Photons

Figure A4 .

 A4 Figure A4. Extended-MZI with Entangled Photons

Table 1 compares

 1 EPR paradox, Bell theorem and Extend-EPR paradox, and show that EPR Paradox is tested at different levels:

Table 1 .

 1 Comparison(1) EPR proposed thought experiments to test EPR paradox at the level of Kinematic/dynamic variables[START_REF] Einstein | Can quantum-mechanical description of physical reality be considered complete?[END_REF];