4114 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 62, …mallat/papiers/AudioScatSpectrum.pdf · 4114 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 62, NO. 16, AUGUST 15, 2014 Deep

4114 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 62, NO. 16, AUGUST 15, 2014

Deep Scattering SpectrumJoakim Andén, Member, IEEE, and Stéphane Mallat, Fellow, IEEE

Abstract—A scattering transform defines a locally translationinvariant representation which is stable to time-warping deforma-tion. It extends MFCC representations by computing modulationspectrum coefficients of multiple orders, through cascades ofwavelet convolutions and modulus operators. Second-order scat-tering coefficients characterize transient phenomena such asattacks and amplitude modulation. A frequency transpositioninvariant representation is obtained by applying a scatteringtransform along log-frequency. State-the-of-art classificationresults are obtained for musical genre and phone classification onGTZAN and TIMIT databases, respectively.

Index Terms—Audio classification, deep neural networks,MFCC, modulation spectrum, wavelets.

I. INTRODUCTION

A MAJOR difficulty of audio representations for classifi-cation is the multiplicity of information at different time

scales: pitch and timbre at the scale of milliseconds, the rhythmof speech and music at the scale of seconds, and the music pro-gression over minutes and hours. Mel-frequency cepstral coef-ficients (MFCCs) are efficient local descriptors at time scales upto 25 ms. Capturing larger structures up to 500 ms is howevernecessary in most applications. This paper studies the construc-tion of stable, invariant signal representations over such largertime scales. We concentrate on audio applications, but introducea generic scattering representation for classification, which ap-plies to many signal modalities beyond audio [1].Spectrograms compute locally time-shift invariant descrip-

tors over durations limited by a window. However, Section IIshows that high-frequency spectrogram coefficients are notstable to variability caused by time-warping deformations,which occur in most signals, particularly in audio. Stabilitymeans that small signal deformations produce small modi-fications of the representation, measured with a Euclideannorm. This is particularly important for classification. Mel-fre-quency spectrograms are obtained by averaging spectrogramvalues over mel-frequency bands. It improves stability to

Manuscript received May 14, 2013; revised September 27, 2013 and De-cember 31, 2013; accepted January 12, 2014. Date of publicationMay 29, 2014;date of current version July 18, 2014. The associate editor coordinating the re-view of this manuscript and approving it for publication was Prof. Xiao-PingZhang. This work was supported by the ANR 10-BLAN-0126 and ERC In-variant Class 320959 Grants.J. Andén was with the was with Centre de Mathématiques Appliquées, Ecole

Polytechnique, Route de Saclay, 91128 Palaiseau, France. He is now with theProgram in Applied and Computational Mathematics, Princeton University,Princeton, NJ 08544 USA (e-mail: [email protected]).S.Mallat is with the Département d’Informatique, Ecole Normale Supérieure,

75005 Paris, France (e-mail: [email protected]).Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TSP.2014.2326991

time-warping, but it also removes information. Over timeintervals larger than 25 ms, the information loss becomes tooimportant, which is why mel-frequency spectrograms andMFCCs are limited to such short time intervals. Modulationspectrum decompositions [2]–[10] characterize the temporalevolution of mel-frequency spectrograms over larger timescales, with autocorrelation or Fourier coefficients. How-ever, this modulation spectrum also suffers from instabilityto time-warping deformation, which degrades classificationperformance.Section III shows that the information lost by mel-frequency

spectrograms can be recovered with multiple layers of waveletcoefficients. In addition to being locally invariant to time-shifts,this representation is also stable to time-warping deformation.Known as a scattering transform [11], it is computed through acascade of wavelet transforms and modulus non-linearities. Thecomputational structure is similar to a convolutional deep neuralnetwork [12]–[19], but involves no learning. It outputs time-averaged coefficients, providing informative signal invariantsover potentially large time scales.A scattering transform has striking similarities with phys-

iological models of the cochlea and of the auditory pathway[20], [21], also used for audio processing [22]. Its energyconservation and other mathematical properties are reviewedin Section IV. An approximate inverse scattering transform isintroduced in Section V with numerical examples. Section VIrelates the amplitude of scattering coefficients to audio signalproperties. These coefficients provide accurate measurementsof frequency intervals between harmonics and also characterizethe amplitude modulation of voiced and unvoiced sounds. Thelogarithm of scattering coefficients linearly separates audiocomponents related to pitch, formants and timbre.Frequency transpositions form another important source of

audio variability, which should be kept or removed dependingupon the classification task. For example, speaker-independentphone classification requires some frequency transposition in-variance, while frequency localization is necessary for speakeridentification. Section VII shows that cascading a scatteringtransform along log-frequency yields a transposition invariantrepresentation which is stable to frequency deformation.Scattering representations have proved useful for image clas-

sification [23], [24], where spatial translation invariance is cru-cial. In audio, the analogous time-shift invariance is also im-portant, but scattering transforms are computed with very dif-ferent wavelets. They have a better frequency resolution, whichis adapted to audio frequency structures. Section VIII explainshow to adapt and optimize the frequency invariance for eachsignal class at the supervised learning stage. A time and fre-quency scattering representation is used for musical genre clas-sification over the GTZAN database, and for phone segment

1053-587X © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

ANDÉN AND MALLAT: DEEP SCATTERING SPECTRUM 4115

Fig. 1. (a) Spectrogram for a harmonic signal (centeredin ) followed by for (centered in), as a function of and . The right graph plots (blue)

and (red) as a function of . Their partials do not overlapat high frequencies. (b) Mel-frequency spectrogram fol-lowed by . The right graph plots (blue) and

(red) as a function of . With a mel-scale frequency aver-aging, the partials of and overlap at all frequencies.

classification over the TIMIT corpus. State-of-the-art results areobtained with a Gaussian kernel SVM applied to scattering fea-ture vectors. All figures and results are reproducible using aMATLAB software package, available at http://www.di.ens.fr/data/scattering/.

II. MEL-FREQUENCY SPECTRUM

Section II.A shows that high-frequency spectrogram coeffi-cients are not stable to time-warping deformation. The mel-fre-quency spectrogram stabilizes these coefficients by averagingthem along frequency, but loses information. To analyze this in-formation loss, Section II.B relates the mel-frequency spectro-gram to the amplitude output of a filter bank which computes awavelet transform.

A. Fourier Invariance and Deformation Instability

Let be the Fourier transform of .If then . The Fouriertransform modulus is thus invariant to translation:

A spectrogram localizes this translation invariance with awindow of duration such that . It is definedby

(1)

If then one can verify that .However, invariance to time-shifts is often not enough. Sup-

pose that is not just translated but time-warped to givewith . A representation is said to

be stable to deformation if its Euclidean normis small when the deformation is small. The deformation sizeis measured by . If it vanishes then it is a “pure”translation without deformation. Stability is formally defined

as a Lipschitz continuity condition relatively to this metric. Itmeans that there exists such that for and all with

(2)

The constant is a measure of stability.This Lipschitz continuity property implies that time-warping

deformations are locally linearized by . Indeed, Lipschitzcontinuous operators are almost everywhere differentiable. It re-sults that can be approximated by a linear op-erator if is small. A family of small deformationsthus generate a linear space. In the transformed space, an in-variant to these deformations can then be computed with a linearprojector on the orthogonal complement to this linear space. InSection VIII we use linear discriminant classifiers to become se-lectively invariant to small time-warping deformations.A Fourier modulus representation is not stable to

deformation because high frequencies are severely distorted bysmall deformations. For example, let us consider a small dilation

with . Since , the Lipschitzcontinuity condition (2) becomes

(3)

The Fourier transform of is. This dilation shifts a fre-

quency component at by . For a harmonic signal, the Fourier transform is a sum of

partials

After time-warping, each partial is translated by ,as shown in the spectrogram of Fig. 1(a). Even though is small,at high frequencies becomes larger than the bandwidth of. Consequently, the harmonics of donot overlap with the harmonics of . The Euclideandistance between and thus does not decrease propor-tionally to if the harmonic amplitudes are sufficiently largeat high frequencies. This proves that the deformation stabilitycondition (3) is not satisfied for any .The autocorrelation is also

a translation invariant representation which has the same de-formation instability as the Fourier transform modulus. Indeed,

so .

B. Mel-Frequency Deformation Stability and Filter Banks

A mel-frequency spectrogram averages the spectrogram en-ergy with mel-scale filters , where is the center frequencyof each :

(4)

The bandpass filters have a constant- frequency bandwidthat high frequencies. Their frequency support is centered atwith a bandwidth of the order of . At lower frequencies,


instead of being constant-Q, the bandwidth of remains equalto .The mel-frequency averaging removes deformation insta-

bility created by large displacements of high frequencies underdilations. If then we saw that the frequencycomponent at is moved by , which may be large if

is large. However, the mel-scale filter coveringthe frequency has a frequency bandwidth of the order of

. As a result, the relative error after averagingby is of the order of . This is illustrated by Fig. 1(b)on a harmonic signal . After mel-frequency averaging, thefrequency partials of and overlap at all frequencies. Onecan verify that , where isproportional to , and does not depend upon or . Unlike thespectrogram (1), the mel-frequency spectrogram (4) satisfiesthe Lipschitz deformation stability condition (2).Mel-scale averaging provides time-warping stability but

loses information. We show that this frequency averaging isequivalent to a time averaging of a filter bank output, whichwill provide a strategy to recover the lost information. Since

in (1) is the Fourier transform of ,applying Plancherel’s formula gives

If then is approximately constant on the supportof , so , and hence

(5)

The frequency averaging of the spectrogram is thus nearlyequal to the time averaging of . In this formulation,the window acts as a lowpass filter, ensuring that the rep-resentation is locally invariant to time-shifts smaller than .Section III.A studies the properties of the constant-Q filter bank

, which defines an analytic wavelet transform.Figs. 2(a) and (b) display and , respec-

tively, for a musical recording. The window duration isms. This time averaging removes fine-scale information suchas vibratos and attacks. To reduce information loss, a mel-fre-quency spectrogram is often computed over small timewindowsof about 25 ms. As a result, it does not capture large-scale struc-tures, which limits classification performance.To increase without losing too much information, it is nec-

essary to capture the amplitude modulations of atscales smaller than , which are important in audio percep-tion. The spectrum of these modulation envelopes can be com-puted from the spectrogram [2]–[5] of , or representedwith a short-time autocorrelation [6], [7]. However, these mod-ulation spectra are unstable to time-warping deformation. In-deed, a time-warping of induces a time-warping of ,

Fig. 2. (a) Scalogram for a musical signal, as a function ofand . (b): Averaged scalogram with a lowpass filterof duration ms.

and Section II.A showed that spectrograms and autocorrela-tions suffer from deformation instability. Constant-Q averagedmodulation spectra [9], [10] stabilize spectrogram representa-tions with another averaging along modulation frequencies. Ac-cording to (5), this can also be computed with a second con-stant-Q filter bank. The scattering transform follows this latterapproach.

III. WAVELET SCATTERING TRANSFORM

A scattering transform recovers the information lost by a mel-frequency averaging with a cascade of wavelet decompositionsand modulus operators [11]. It is locally translation invariantand stable to time-warping deformation. Important propertiesof constant-Q filter banks are first reviewed in the framework ofa wavelet transform, and the scattering transform is introducedin Section III.B.

A. Analytic Wavelet Transform and Modulus

Constant-Q filter banks compute a wavelet transform. We re-view the properties of complex analytic wavelet transforms andtheir modulus, which are used to calculate mel-frequency spec-tral coefficients.A wavelet is a bandpass filter with . We con-

sider complex wavelets with quadrature phase such thatfor . For any , a dilated wavelet of center fre-

quency is written

(6)

The center frequency of is normalized to 1. In the fol-lowing, we denote by the number of wavelets per octave,which means that for . The bandwidth of is ofthe order of , to cover the whole frequency axis with thesebandpass wavelet filters. The support of is centered inwith a frequency bandwidth whereas the energy ofis concentrated around 0 in an interval of size . To guar-antee that this interval is smaller than , we define with (6)only for . For , the lower-frequency in-terval is covered with about equally-spacedfilters with constant frequency bandwidth . For sim-plicity, these lower-frequency filters are still calledwavelets.Wedenote by the grid of all wavelet center frequencies .


The wavelet transform of computes a convolution of witha lowpass filter of frequency bandwidth , and convolu-tions with all higher-frequency wavelets for :

(7)

This time index is not critically sampled as in wavelet basesso this representation is highly redundant. The wavelet andthe lowpass filter are designed to build filters which cover thewhole frequency axis, which means that

satisfies, for all :

(8)

This condition implies that the wavelet transform is a stableand invertible operator.Multiplying (8) by and applyingthe Plancherel formula [25] gives

(9)

where and the squared norm of is thesum of all coefficients squared:

The upper bound (9) means that is a contractive operatorand the lower bound implies that it has a stable inverse. Onecan also verify that the pseudo-inverse of recovers withthe following formula

(10)

with reconstruction filters defined by

(11)

where is the complex conjugate of . If in(8) then is said to be a tight frame operator, in which case

and .One may define an analytic wavelet with an octave reso-

lution as and hencewhere is the transfer function of a lowpass filter whose band-width is of the order of . If then we define

, which guarantees that. If is a Gaussian then is called a Morlet wavelet,

which is almost analytic because is small but not strictlyzero for . Fig. 3 shows Morlet wavelets with .In this case is also chosen to be a Gaussian. For , tightframe wavelet transforms can also be obtained by choosingto be the analytic part of a real wavelet which generates an or-thogonal wavelet basis, such as a cubic spline wavelet [11]. Un-less indicated otherwise, wavelets used in this paper are Morletwavelets.

Fig. 3. Morlet wavelets with wavelets per octave, for different. The low-frequency filter (in red) is a Gaussian.

Following (5), mel-frequency spectrograms can be approxi-mated using a non-linear wavelet modulus operator which re-moves the complex phase of all wavelet coefficients:

Taking the modulus of analytic wavelet coefficients can beinterpreted as a subband Hilbert envelope demodulation.Demodulation is used to separate carriers and modulationenvelopes. When a carrier or pitch frequency can be detected,then a linear coherent demodulation is efficiently implementedby multiplying the analytic signal with the conjugate of thedetected carrier [26]–[28]. However, many signals such asunvoiced speech are not modulated by an isolated carrierfrequency, in which case coherent demodulation is not welldefined. Non-linear Hilbert envelope demodulations apply toany bandpass analytic signals, but if a carrier is present thenthe Hilbert envelope depends both on the carrier and on theamplitude modulation. Section VI.C explains how to isolateamplitude modulation coefficients from Hilbert envelope mea-surements, whether a carrier is present or not.Although a wavelet modulus operator removes the complex

phase, it does not lose information because the temporal vari-ation of the multiscale envelopes is kept. A signal cannot bereconstructed from the modulus of its Fourier transform, but itcan be recovered from the modulus of its wavelet transform.Since the time variable is not subsampled, a wavelet transformhas more coefficients than the original signal. These coefficientsare highly redundant when filters have a significant frequencyoverlap. For particular families of analytic wavelets, one canprove that is an invertible operator with a continuous in-verse [29]. This is further studied in Section V.The operator is contractive. Indeed, the wavelet trans-

form is contractive and the complex modulus is contractivein the sense that for any so

If is a tight frame operator thenso preserves the signal norm.

B. Deep Scattering Network

We showed in (5) that mel-frequency spectral coefficientsare approximately equal to averaged squared wavelet

coefficients . Large wavelet coefficients areconsiderably amplified by the square operator. To avoid ampli-fying outliers, we remove the square and calculateinstead. High frequencies removed by the lowpass filter arerecovered by a new set of wavelet modulus coefficients. Cas-cading this procedure defines a scattering transform.


A locally translation invariant descriptor of is obtained witha time-average , which removes all high fre-quencies. These high frequencies are recovered by a waveletmodulus transform

It is computed with wavelets having an octave frequencyresolution . For audio signals we set , which de-fines wavelets having the same frequency resolution as mel-fre-quency filters. Audio signals have little energy at low frequen-cies so . Approximate mel-frequency spectral coef-ficients are obtained by averaging the wavelet modulus coeffi-cients with :

These are called first-order scattering coefficients. They arecomputed with a second wavelet modulus transformapplied to each , which also provides complementaryhigh-frequency wavelet coefficients:

The wavelets have an octave resolution which may bedifferent from . It is chosen to get a sparse representationwhich means concentrating the signal information over as fewwavelet coefficients as possible. These coefficients are averagedby the lowpass filter of size , which ensures local invari-ance to time-shifts, as with the first-order coefficients. It definessecond-order scattering coefficients:

These averages are computed by applying a third wavelet mod-ulus transform to each . It computes theirwavelet modulus coefficients through convolutions with a newset of wavelets having an octave resolution . Iteratingthis process defines scattering coefficients at any order .For any , iterated wavelet modulus convolutions are

written:

where th-order wavelets have an octave resolution ,and satisfy the stability condition (8). Averaging withgives scattering coefficients of order :

Applying on computes both and :

(12)

A scattering decomposition of maximal order is thus definedby initializing , and recursively computing (12) for

Fig. 4. A scattering transform iterates on wavelet modulus operators tocompute a cascade of wavelet convolutions and moduli stored in , andto output averaged scattering coefficients .

. This scattering transform is illustrated in Fig. 4.The final scattering vector aggregates all scattering coefficientsfor :

The scattering cascade of convolutions and non-linearitiescan also be interpreted as a convolutional network [12], where

is the set of coefficients of the th internal network layer.These networks have been shown to be highly effective foraudio classification [13]–[19]. However, unlike standard con-volutional networks, each such layer has an output

, not just the last layer. In addition, all filters are pre-defined wavelets and are not learned from training data. A scat-tering transform, like MFCCs, provides a low-level invariantrepresentation of the signal without learning. It relies on priorinformation concerning the types of invariants that need to becomputed, in this case relatively to time-shifts and time-warpingdeformations, or in Section VII relatively to frequency transpo-sitions. When no such information is available, or if the sourcesof variability are much more complex, it is necessary to learnthem from examples, which is a task well suited for deep neuralnetworks. In that sense both approaches are complementary.The wavelet octave resolutions are optimized at each layerto produce sparse wavelet coefficients at the next layer.

This better preserves the signal information as explained inSection V. Sparsity seems also to play an important role forclassification [30], [31]. For audio signals , choosingwavelets per octave has been shown to provide sparse rep-resentations of a mix of speech, music and environmentalsignals [32]. It nearly corresponds to a mel-scale frequencysubdivision.At the second order, choosing defines wavelets with

more narrow time support, which are better adapted to charac-terize transients and attacks. Section VI shows that musical sig-nals including modulation structures such as tremolo may how-ever require wavelets having better frequency resolution, andhence . At higher orders we always set ,but we shall see that these coefficients can often be neglected.The scattering cascade has similarities with several neuro-

physiological models of auditory processing, which incorporatecascades of constant-Q filter banks followed by non-linearities[20], [21]. The first filter bank with models the cochlear


filtering, whereas the second filter bank corresponds to later pro-cessing in the models with filters that have [20], [21].

IV. SCATTERING PROPERTIES

We briefly review important properties of scattering trans-forms, including stability to time-warping deformation, energyconservation, and describe a fast computational algorithm.

A. Time-Warping Stability

Stability to time-warping allows one to use linear operatorsfor calculating descriptors invariant to small time-warping de-formations. The Fourier transform is unstable to deformationbecause dilating a sinusoidal wave yields a new sinusoidal waveof different frequency which is orthogonal to the original one.Section II explains that mel-frequency spectrograms becomestable to time-warping deformation with a frequency averaging.One can prove that a scattering representation satis-fies the Lipschitz continuity condition (2) because wavelets arestable to time-warping [11]. Let us write .One can verify that there exists such that

, for all and all . This property is at thecore of the scattering stability to time-warping deformation.The squared Euclidean norm of a scattering vector is the

sum of its coefficients squared at all orders:

We consider deformations withand , which means that the maximum displace-ment is small relatively to the support of . One can prove thatthere exists a constant such that for all and any such [11]:

up to second-order terms. As explained for mel-spectral decom-positions, the constant is inversely proportional to the octavebandwidth of wavelet filters. Over multiple scattering layers, weget . For Morlet wavelets, numerical ex-periments on a broad range of examples give .

B. Contraction and Energy Conservation

We show that a scattering transform is contractive and canpreserve energy. We denote by the squared Euclideannorm of a vector of coefficients , such asor . Since is computed by cascading wavelet modulusoperators , which are all contractive, it results that isalso contractive:

A scattering transform is therefore stable to additive noise.

TABLE IAVERAGED VALUES COMPUTED FOR SIGNALS IN THE TIMITSPEECH DATASET [33], AS A FUNCTION OF ORDER AND AVERAGING SCALE. FOR IS CALCULATED BY MORLET WAVELETS WITH ,AND FOR BY CUBIC SPLINE WAVELETS WITH

If each wavelet transform is a tight frame, that is in (8),each preserves the signal norm. Applying this property to

yields

Summing these equations proves that

Under appropriate assumptions on the mother wavelet , onecan prove that goes to zero as increases, which im-plies that for [11]. This property comesfrom the fact that the modulus of analytic wavelet coefficientscomputes a smooth envelope, and hence pushes energy towardslower frequencies. By iterating on wavelet modulus operators,the scattering transform progressively propagates all the energyof towards lower frequencies, which is captured by thelowpass filter of scattering coefficients .One can verify numerically that converges to zero

exponentially when goes to infinity and hence that con-verges exponentially to . Table I gives the fraction of en-ergy absorbed by each scattering order. Sinceaudio signals have little energy at low frequencies, is verysmall and most of the energy is absorbed by forms. This explains why mel-frequency spectrograms are typi-cally sufficient at these small time scales. However, as in-creases, a progressively larger proportion of energy is absorbedby higher-order scattering coefficients. For s, about47% of the signal energy is captured in . Section VI showsthat at this time scale, important amplitude modulation informa-tion is carried by these second-order coefficients. For s,

carries 26% of the signal energy. It increases as increases,but for audio classification applications studied in this paper,remains below 1.5 s, so these third-order coefficients are lessimportant than first- and second-order coefficients. We there-fore concentrate on second-order scattering representations:

C. Fast Scattering Computation

Subsampling scattering vectors provide a reduced represen-tation, which leads to a faster implementation. Since the aver-aging window has a duration of the order of , we computescattering vectors with half-overlapping windows atwith .


We suppose that has samples over each frame of du-ration , and is thus sampled at a rate . For each timeframe , the number of first-order wavelets isabout so there are about first-order coef-ficients . We now show that the number of non-neg-ligible second-order coefficients which needs tobe computed is about .The wavelet transform envelope is a de-

modulated signal having approximately the same frequencybandwidth as . Its Fourier transform is mostly supportedin the interval for , and in

for . If the support ofcentered at does not intersect the frequency support of

, then

One can verify that non-negligible second-order coefficientssatisfy

For a fixed , a direct calculation then shows that there are ofthe order of second-order scattering coeffi-cients. Similar reasoning extends this result to show that thereare about non-negligible th-orderscattering coefficients.To compute and we first calculate and

and average them with . Over a time frame of duration ,to reduce computations while avoiding aliasing, issubsampled at a rate which is twice its bandwidth. The familyof filters covers the whole frequency domain andis chosen so that filter supports barely overlap. Over a time

frame where has samples, with the above subsamplingwe compute approximately first-order wavelet coefficients

. Similarly, is sub-sampled in time at a rate twice its bandwidth. Over the sametime frame, the total number of second-order wavelet coeffi-cients for all and stays below . With a fast Fouriertransform (FFT), these first- and second-order wavelet moduluscoefficients are computed using operations. Theresulting scattering coefficients andare also calculated with operations using FFT con-volutions with .

V. INVERSE SCATTERING

To better understand the information carried by scatteringcoefficients, this section studies a numerical inversion ofthe transform. Since a scattering transform is computed bycascading wavelet modulus operators , it can be approx-imately inverted by inverting each for . At themaximum depth , the algorithm begins with a deconvo-lution, estimating at all on the sampling grid of ,from .Because of the subsampling, one cannot compute fromexactly. This deconvolution is thus the main source of error.

To take advantage of the fact that , the deconvolutionis computed with the Richardson-Lucy algorithm [34], whichpreserves positivity if .We initialize by interpolating

linearly on the sampling grid of , which introduceserror because of aliasing. The Richardson-Lucy deconvolutioniteratively computes

with . Since it converges to the pseudo-inverse ofthe convolution operator applied to , it blows up when in-creases because of the deconvolution instability. Deconvolutionalgorithms thus stop after a fixed number of iterations, which isset to 30 in this application. The result is then our estimate of

.Once an estimation of is calculated by deconvolution,

we compute an estimate of by inverting each for. The wavelet transform of a signal of size

is a vector of about co-efficients, where is the number of wavelets per octave.These coefficients live in a subspace of dimension . To re-cover from , we search for avector in whose modulus values are specified by . Thisa non-convex optimization problem. Recent convex relaxationapproaches [35], [36] are able to compute exact solutions, butthey require too much computation and memory for audio ap-plications. Since the main source of errors is introduced at thedeconvolution stage, one can use an approximate but fast inver-sion algorithm. The inversion of is typically more stablewhen is sparse because there is no phase to recover if

. This motivates using wavelets which providesparse representations at each order .Griffin & Lim [37] showed that alternating projections re-

cover good-quality audio signals from spectrogram values, butwith large mean-square errors because the algorithm is trappedin local minima. The same algorithm inverts by alternatingprojections on the wavelet transform space and on the mod-ulus constraints. An estimation of is calculated from ,by initializing to be Gaussian white noise. For any

is then computed from by first adjusting the modulusof its wavelet coefficients with a non-linear projector

Applying the wavelet transform pseudo-inverse (10) yields

The dual filters are defined in (11). One can verify thatis the orthogonal projection of onto . Numer-ical experiments are performed with iterations, and weset .When , an approximation of is computed from

by first estimating from withthe Richardson-Lucy deconvolution algorithm. We then com-pute from and this estimation of by approximatelyinverting using the Griffin & Lim algorithm. When isabove 100 ms, the deconvolution loses too much information,and audio reconstructions obtained from first-order coefficientsare crude. Fig. 5(a) shows the scalograms of a


Fig. 5. (a) Scalogram for recordings of speech (top) and acello (bottom). (b), (c) Scalograms of reconstructions fromfirst-order scattering coefficients in (b), and from first- and second-ordercoefficients in (c). Scattering coefficients were computed with

ms for the speech signal and ms for the cello signal.

speech and a music signal, and the scalograms oftheir approximations from first-order scattering coefficients.When , the approximation is calculated from

by applying the deconvolution algorithm toto estimate , and then by successively

inverting and with the Griffin & Lim algorithm.Fig. 5(c) shows for the same speech andmusic signals. Amplitude modulations, vibratos and attacks arerestored with greater precision by incorporating second-ordercoefficients, yielding much better audio quality comparedto first-order reconstructions. However, even with ,reconstructions become crude for ms. Indeed, thenumber of second-order scattering coefficientsis too small relatively to the number audio samples ineach audio frame, and they do not capture enough infor-mation. Examples of audio reconstructions are available athttp://www.di.ens.fr/data/scattering/audio/.

VI. NORMALIZED SCATTERING SPECTRUM

To reduce redundancy and increase invariance, Section VI.Anormalizes scattering coefficients. Section VI.B shows that nor-malized second-order coefficients provide high-resolution spec-tral information through interferences. Section VI.C also provesthat they characterize amplitude modulations of audio signals.

A. Normalized Scattering Transform

Scattering coefficients are renormalized to increase their in-variance. It also decorrelates these coefficients at different or-ders. First-order scattering coefficients are renormalized so thatthey become insensitive to multiplicative constants:

(13)

The constant is a silence detection threshold so thatwhen , and may be set to 0.The lowpass filter can be wider than the one used in the

scattering transform. Specifically, if we want to retain local am-plitude information of below a certain scale, we can nor-malize by the average of over this scale, creating invarianceonly to amplitude changes over larger intervals.At any order , scattering coefficients are renormalized

by coefficients of the previous order:

A normalized scattering representation is defined by. We shall mostly limit ourselves to

.For ,

Let us show that these coefficients are nearly invariant to a fil-tering by if is approximately constant on the supportof . This condition is satisfied if

It implies that , and hence. It results that

Similarly, , so afternormalization

Normalized second-order coefficients are thus invariant to fil-tering by . One can verify that this remains valid at any order

.

B. Frequency Interval Measurement From Interference

A wavelet transform has a worse frequency resolution than awindowed Fourier transform at high frequencies. However, weshow that frequency intervals between harmonics are accuratelymeasured by second-order scattering coefficients.Suppose has two frequency components in the support of. We then have

whose modulus squared equals

We approximate with a first-order expansion of thesquare root, which yields


If has a support of size , then, so satisfies

(14)

These normalized second-order coefficients are thus non-neg-ligible when is of the order of the frequency interval .This shows that although the first wavelet does not haveenough resolution to discriminate the frequencies and ,second-order coefficients detect their presence and accuratelymeasure the interval . As in audio perception, scatteringcoefficients can accurately measure frequency intervals but notfrequency location. The normalized second-order scattering co-efficients (14) are large only if and have the same orderof magnitude. This also conforms to auditory perception wherea frequency interval is perceived only when the two frequencycomponents have a comparable amplitude.If has more frequency compo-

nents, we verify similarly that is non-negligiblewhen is of the order of for some . These co-efficients can thus measure multiple frequency intervals withinthe frequency band covered by . If the frequency resolu-tion of is not sufficient to discriminate between two fre-quency intervals and , these intervals willinterfere and create high-amplitude third-order scattering coef-ficients. A similar calculation shows that third-order scatteringcoefficients detect the presence of two suchintervals within the support of when is close to

. They thus measure “intervals of intervals.”Fig. 6(a) shows the scalogram of a signal

containing a chord with two notes, whose fundamental fre-quencies are Hz and Hz, followed byan arpeggio of the same two notes. First-order coefficients

in Fig. 6(b) are very similar for the chord andthe arpeggio because the time averaging loses time local-ization. However they are easily differentiated in Fig. 6(c),which displays for Hz, as afunction of . The chord creates large amplitude coefficientsfor Hz, which disappear for the arpeggiobecause these two frequencies are not present simultaneously.Second-order coefficients have also a large amplitude at lowfrequencies . These arise from variation of the note en-velopes in the chord and in the arpeggio, as explained in thenext section.

C. Amplitude Modulation Spectrum

Audio signals are usually modulated in amplitude by an enve-lope, whose variations may correspond to an attack or a tremolo.For voiced and unvoiced sounds, we show that amplitude modu-lations are characterized by normalized second-order scatteringcoefficients.Let be a sound resulting from an excitation filtered

by a resonance cavity of impulse response , which is mod-ulated in amplitude by to give

Fig. 6. (a) Scalogram for a signal with two notes, of funda-mental frequencies Hz and Hz, first played as a chordand then as an arpeggio. (b) First-order normalized scattering coefficients

for ms. (c) Second-order normalized scatteringcoefficients with as a function of and . Thechord interferences produce large coefficients for .

We shall start by taking to be a pulse train of pitch givenby

representing a voiced sound. The impulse response is typ-ically very short compared to the minimum variation interval

of the modulation term and is smaller than.

We consider whose time support is short relatively toand to the averaging interval , and whose fre-

quency bandwidth is smaller than the pitch and the minimumvariation interval of . These conditions are satisfied if

(15)

After the normalization , theAppendix shows that

(16)

where and is an integer such that. First-order coefficients are thus proportional to the

spectral envelope if is close to a harmonicfrequency.Similarly, for , the Appendix

shows that

(17)

Normalized second-order coefficients thus do not depend uponand but only on the amplitude modulation provided that

is non-negligible.


Fig. 7. (a) Scalogram for a signal with three voiced soundsof same pitch Hz and same but different amplitude modula-tions : first a smooth attack, then a sharp attack, then a tremolo of fre-quency . It is followed by three unvoiced sounds created with the sameand same amplitude modulations as the first three voiced sounds. (b) First-order scattering with ms. (c) Second-order scattering

displayed for , as a function of and .

Fig. 7(a) displays for a signal having threevoiced and three unvoiced sounds. The first three are producedby a pulse train excitation with a pitch of Hz.Fig. 7(b) shows that has a harmonic structure,with an amplitude depending on . The averaging byand the normalization remove the effect of the different mod-

ulation amplitudes of these three voiced sounds.Fig. 7(c) displays for the fourth partial

as a function of . The modulation envelopeof the first sound has a smooth attack and thus produces largecoefficients only at low frequencies . The envelope ofthe second sound has a much sharper attack and thus produceslarge-amplitude coefficients for higher frequencies . Thethird sound is modulated by a tremolo, which is a periodic os-cillation . According to (17), this tremolocreates large amplitude coefficients when , as shown inFig. 7(c).Unvoiced sounds are modeled by excitations which are

realizations of Gaussian white noise. The modulation amplitudeis typically non-sparse, which means the square of the averageof on intervals of size is of the order of the average of

. The Appendix shows that

(18)

Similarly to (16), is proportional to but doesnot have a harmonic structure. This is shown in Fig. 7(b) by thelast three unvoiced sounds. The fourth, fifth, and sixth soundshave the same filter and envelope as the first, second,and third sounds, respectively, but with a Gaussian white noiseexcitation .

Similarly to (17), the Appendix also shows that

where is less thanwith . For voiced and unvoiced sounds,mainly depends on the amplitude modulation . This is il-lustrated by Fig. 7(c), which shows that the fourth, fifth, andsixth sounds have second-order coefficients similar to those ofthe first, second, and third sounds, respectively. The stochasticerror term produced by unvoiced sounds appears as randomlow-amplitude fluctuations in Fig. 7(c).

VII. FREQUENCY TRANSPOSITION INVARIANCE

Audio signals within the same class may be transposed infrequency. For example, frequency transposition occurs when asingle word is pronounced by different speakers. It is a complexphenomenon which affects the pitch and the spectral envelope.The envelope is translated on a logarithmic frequency scale butalso deformed. We thus need a representation which is invariantto frequency translation on a logarithmic scale, and which alsois stable to frequency deformation. After reviewing the mel-fre-quency cepstral coefficient (MFCC) approach through thediscrete cosine transform (DCT), this section defines such arepresentation with a scattering transform computed alonglog-frequency.MFCCs are computed from the log-mel-frequency spectro-

gram by calculating a DCT along the mel-fre-quency index for a fixed [38]. This is linear in for lowfrequencies, but is proportional to for higher frequencies.For simplicity, we write and , although thisshould be modified at low frequencies.The frequency index of the DCT is called the “quefrency”

parameter. In MFCCs, high-quefrency coefficients are often setto zero, which is equivalent to averaging alongand provides some frequency transposition invariance. The

more high-quefrency coefficients are set to zero, the bigger theaveraging and hence themore transposition invariance obtained,but at the expense of losing potentially important information.The loss of information due to averaging along can be

recovered by computing wavelet coefficients along . Wethus replace the DCT by a scattering transform along . Afrequency scattering transform is calculated by iteratively ap-plying wavelet transforms and modulus operators. An analyticwavelet transform of a log-frequency dependent signal isdefined as in (7), but with convolutions along the log-frequencyvariable instead of time:

Each wavelet is a bandpass filter whose Fourier transformis centered at “quefrency” and is an averaging filter.

These wavelets satisfy the condition (8), so is contractiveand invertible.Although the scattering transform along can be computed at

any order, we restrict ourself to zero- and first-order scattering


coefficients, as this seems to be sufficient for classification. Afirst-order scattering transform of is calculated from

(19)

by averaging these coefficients along with :

(20)

These coefficients are locally invariant to log-frequency shifts,over a domain proportional to the support of the averaging filter. This frequency scattering is formally identical to a time

scattering transform. It has the same properties if we replace thetime by the log-frequency variable . Numerical experimentsare implemented using Morlet wavelets with .Similarly to MFCCs, we apply a logarithm to normalized

scattering coefficients so that multiplicative components be-come additive and can be separated by linear operators. Thiswas shown to improve classification performance. The loga-rithm of a second-order normalized time scattering transformat frequency and time is

This is a vector of signals , where depends on and .Let us transform each by the frequency scattering opera-tors or , defined in (19) and (20). Letand stand for the concatenation of these trans-formed signals for all and . The representation iscalculated by cascading a scattering in time and a scattering inlog-frequency. It is thus locally translation invariant in time andin log-frequency, and stable to time and frequency deformation.The interval of time-shift invariance is defined by the size of thetime averaging window , whereas its frequency-transpositioninvariance depends upon the width of the log-frequency aver-aging window .Frequency transposition invariance is useful for certain tasks,

such as speaker-independent speech recognition or transposi-tion-independent melody recognition, but it removes informa-tion important to other tasks, such as speaker identification.The frequency transposition invariance, implemented by the fre-quency averaging filter , should thus be adapted to the classi-fication task. Next section explains how this can be done by re-placing by and optimizing thelinear averaging at the supervised classification stage.

VIII. CLASSIFICATION

This section compares the classification performanceof support vector machine classifiers applied to scatteringrepresentations with standard low-level features such as-MFCCs or more sophisticated state-of-the-art represen-

tations. Section VIII.A explains how to automatically adaptinvariance parameters, while Sections VIII.B and VIII.Cpresent results for musical genre classification and phoneclassification, respectively.

A. Adapting Frequency Transposition Invariance

The amount of frequency-transposition invariance dependson the classification problem, and may vary for each signal

Fig. 8. A time and frequency scattering representation is computed by applyinga normalized temporal scattering on the input signal , a logarithm, and ascattering along log-frequency without averaging.

class. This adaptation is implemented by a supervised classifier,applied to the time and frequency scattering transform.Fig. 8 illustrates the computation of a time and frequency

scattering representation. The normalized scattering trans-form of an input signal is computed along time, overhalf-overlapping windows of size . The log-scattering vectorfor each time window is transformed along frequencies by thewavelet modulus operator , as explained in Section VII.Since we do not know in advance how much transpositioninvariance is needed for a particular classification task, thefinal frequency averaging is adaptively computed by the super-vised classifier, which takes as input the vector of coefficients

, for each time frame indexed by .The supervised classification is implemented by a support

vector machine (SVM). A binary SVM classifies a featurevector by calculating its position relative to a hyperplane,which is optimized to maximize class separation given a setof training samples. It thus computes the sign of an optimizedlinear combination of the feature vector coefficients. With aGaussian kernel of variance , the SVM computes differenthyperplanes in different balls of radius in the feature space.The coefficients of the linear combination thus vary smoothlywith the feature vector values. Applied to ,the SVM optimizes the linear combination of coefficients along, and can thus adjust the amount of linear averaging to createfrequency-transposition invariant descriptors which maximizeclass separation. A multi-class SVM is computed from binaryclassifiers using a one-versus-one approach. All numericalexperiments use the LIBSVM library [39].The wavelet octave resolution can also be adjusted at the

supervised classification stage, by computing the time scatteringfor several values of and concatenating all coefficients in asingle feature vector. A filter bank with has enoughfrequency resolution to separate harmonic structures, whereaswavelets with have a smaller time support and canthus better localize transients in time. The linear combinationoptimized by the SVM is a feature selection algorithm, whichcan select the best coefficients to discriminate any two classes.In the experiments described below, adding more values ofbetween 1 and 8 provides marginal improvements.

B. Musical Genre Classification

Scattering feature vectors are first applied to a musical genreclassification problem on the GTZAN dataset [40]. The datasetconsists of 1000 thirty-second clips, divided into 10 genres of100 clips each. Given a clip, the goal is to find its genre.Preliminary experiments have demonstrated the efficiency of

the scattering transform for music classification [41] and for en-vironmental sounds [42]. These results are improved by lettingthe supervised classifier adjust the transform parameters to thesignal classes. A set of feature vectors is computed over half-


TABLE IIERROR RATES (IN PERCENT) FOR MUSICAL GENRE CLASSIFICATION ONGTZAN AND FOR PHONE CLASSIFICATION ON THE TIMIT DATABASE FORDIFFERENT FEATURES. TIME SCATTERING TRANSFORMS ARE COMPUTEDWITH MS FOR GTZAN AND WITH MS FOR TIMIT

overlapping frames of duration . Each frame of a clip is clas-sified separately by a Gaussian kernel SVM, and the clip is as-signed to the class which is most often selected by its frames. Toreduce the SVM training time, feature vectors were only com-puted every 370 ms for the training set. The SVM slack param-eter and the Gaussian kernel variance are determined throughcross-validation on the training data. Table II summarizes re-sults with one run of ten-fold cross-validation. It gives the av-erage error and its standard deviation.Scattering classification results are first compared with results

obtained for MFCC feature vectors. A -MFCC vector aug-ments an MFCC vector at time by estimates of its first andsecond derivatives derived from vectors centered atand .When computed for ms, the -MFCC erroris 20.2%, which is reduced to 18.0% by increasing to 740 ms.Further increasing does not reduce the error. State-of-the-artalgorithms provide refined feature vectors to improve classifi-cation. Combining MFCCs with stabilized modulation spectraand performing linear discriminant analysis, [8] obtains an errorof 9.4%, the best non-scattering result so far. A deep beliefnetwork trained on spectrograms [18], achieves 15.7% errorwith an SVM classifier. A sparse representation on a constant-Qtransform [30], gives a 16.6% error with an SVM.Table II gives classification errors for different scattering fea-

ture vectors. For , they are composed of first-order timescattering coefficients computed with and ms.These vectors are similar to MFCCs as shown by (5). As a re-sult, the classification error of 19.1% is close to that of MFCCsfor the same . For , we add second-order coefficientscomputed with . It reduces the error to 10.7%. This40% error reduction shows the importance of second-order co-efficients for relatively large . Third-order coefficients are alsocomputed with . For , including these coefficientsreduces the error marginally to 10.6%, at a significant computa-tional and memory cost.We therefore restrict ourselves to .Musical genre recognition is a task which is partly invariant

to frequency transposition. Incorporating a scattering along thelog-frequency variable for frequency transposition invariancereduces the error by about 15%. These errors are obtained with afirst-order scattering along log-frequency. Adding second-ordercoefficients only improves results marginally.Providing adaptivity for the wavelet octave bandwidth by

computing scattering coefficients for both andfurther reduces the error by almost 10%. Indeed, music signals

include both sharp transients and narrow-bandwidth frequencycomponents. We thus have an error rate of 8.6%, which com-pares favorably to the non-scattering state-of-the-art of 9.4%error [8].Replacing the SVM with more sophisticated classifiers can

improve results. A sparse representation classifier applied tosecond-order time scattering coefficients reduces the error ratefrom 10.7% to 8.8%, as shown in [44]. Let us mention that theGTZAN database suffers from some significant statistical issues[45], which probably does not make it appropriate to evaluatefurther algorithmic refinements.

C. Phone Segment Classification

The same scattering representation is tested for phone seg-ment classification with the TIMIT corpus [33]. The datasetcontains 6300 phrases, each annotated with the identities, loca-tions, and durations of its constituent phones. This task is sim-pler than continuous speech recognition, but provides an evalu-ation of scattering feature vectors for representing phone seg-ments. Given the location and duration of a phone segment,the goal is to determine its class according to the standard pro-tocol [46], [47]. The 61 phone classes (excluding the glottal stop/q/) are collapsed into 48 classes, which are used to train andtest models. To calculate the error rate, these classes are thenmapped into 39 clusters. Training is achieved on the full 3696-phrase training set, excluding “SA” sentences. The Gaussiankernel SVM parameters are optimized by validation on the stan-dard 400-phrase development set [48]. The error is then calcu-lated on the core 192-phrase test set.An audio segment of length 192 ms centered on a phone can

be represented as an array of MFCC feature vectors with half-overlapping time windows of duration . This array, with thelogarithm of the phone duration added, is fed to the SVM. Inmany cases, hidden Markov models or fixed time dilations areapplied when matching different MFCC sequences to accountfor the time-warping of the phone segment [46], [47]. Table IIshows that ms yields a 18.5% error which is much lessthan the 60.5% error for ms. Indeed, many phoneshave a short duration with highly transient structures and arenot well-represented by wide time windows.A lower error of 17.1% is obtained by replacing the SVM

with a sparse representation classifier on MFCC-like spectralfeatures [49]. CombiningMFCCs of different window sizes andusing a committee-based hierarchical discriminative classifier,[43] achieves an error of 16.7%, the best so far. Finally, convo-lutional deep-belief networks cascades convolutions, similarlyto scattering, on a spectrogram using filters learned from thetraining data. These, combined with MFCCs, yield an error of19.7% [13].Rows 4 through 6 of Table II gives the classification results

obtained by replacing MFCC vectors with a time scatteringtransform computed using first-order wavelets with .In order to retain local amplitude structure while creatinginvariance to loudness changes, first-order coefficients arerenormalized in (13) using averaged over a window thesize of the whole phone segment. Second- and third-orderscattering coefficients are calculated with . Thebest results are obtained with ms. For , we only


keep first-order scattering coefficients and get a 19.0% error,similar to that of MFCCs. The error is reduced by about 10%with , a smaller improvement than for GTZAN becausescattering invariants are computed on smaller time interval

ms as opposed to 740 ms for music. Second-ordercoefficients carry less energy when is smaller, as shown inTable I. For the same reason, third-order coefficients provideeven less information compared to the GTZAN case, and donot improve results.Note that no explicit time-warping is needed in this model.

Thanks to the scattering deformation stability, supervised linearclassifiers can indeed compute time-warping invariants whichremain sufficiently informative.For , cascading a log-frequency transposition invariance

computed with a first-order frequency scattering transform ofSection VII reduces the error by about 5%. Computing a second-order frequency scattering transform only marginally improvesresults. Allowing to adapt the wavelet frequency resolution bycomputing scattering coefficients with and alsoreduces the error by about 5%Again, these results are for the problem of phone classifica-

tion, where boundaries are given. Future work will concentrateon the task of phone recognition, where such information is ab-sent. Since this task is more complex, performance is generallyworse, with the state-of-the-art achieved at a 17.7% error rate[16].

IX. CONCLUSION

The success of MFCCs for audio classification can partiallybe explained by their stability to time-warping deformation.Scattering representations extend MFCCs by recovering losthigh frequencies through successive wavelet convolutions.Over windows of ms, signals recovered from first-and second-order scattering coefficients have good audioquality. Normalized scattering coefficients characterizes ampli-tude modulations, and are stable to time-warping deformation.A frequency transposition invariant representation is obtainedby cascading a second scattering transform along frequen-cies. Time and frequency scattering feature vectors yieldstate-of-the-art classification results with a Gaussian kernelSVM, for musical genre classification on GTZAN, and phonesegment classification on TIMIT.

APPENDIX A

Following (15), is nearly constant over the time supportof and is nearly constant over the frequency supportof . It results that

(21)

Let be a harmonic excitation. Since we supposed thatcovers at most one harmonic whose frequency

is close to . It then results from (21) that

(22)

Computing gives

(23)

Let us compute

Since and are approximately constant over intervalsof size , and the support of is smaller than , one canverify that

This approximation together with (23) verifies (16).It also results from (22) that

which, combined with (23), yields (17).Let us now consider a Gaussian white noise excitation .

We saw in (21) that

(24)

Let us decompose

(25)

where is a zero-mean stationary process. If is a normal-ized Gaussian white noise then is a Gaussian randomvariable of variance . It results that andhave a Rayleigh distribution, and since is a complex waveletwith quadrature phase, one can verify that

Inserting (25) and this equation in (24) shows that

(26)

When averaging with , we get

Suppose that is not sparse, in the sense that

(27)

It means that ratios between local and norms of is ofthe order of 1. We are going to show that if then

(28)

which implies

(29)


We give the main arguments to compute the order of magnitudesof the stochastic terms, but it is not a rigorous proof. For a de-tailed argument, see [50]. Computations rely on the followinglemma.Lemma 1: Let be a zero-mean stationary process of

power spectrum . For any deterministic functionsand

Proof: Let . Nowis given by

and hence

Since is the kernel of a positive symmetric operator whosespectrum is bounded by it results that

because .Because is normalized white noise, one can verify using a

Gaussian chaos expansion [50] that ,where . Applying Lemma 1 to andgives

Since has a duration , it can be written asfor some of duration 1. As a result, if

(27) holds then

(30)

The frequency support of is proportional to , so wehave . Together with (30), ifit proves (28) which yields (29).We approximate similarly. First, we write

where is a zero-mean stationary process. Since isnormally distributed in has distribution and

which then gives

One can show that [50], soapplying Lemma 1 gives

Now (30) implies that

since is non-sparse and has a support much smaller thanso . Consequently,

which, together with (29), gives (18).Let us now compute .

If then (29) together with (26) shows that

where

(31)

Observe that

Lemma 1 applied to and gives the followingupper bound:

(32)

One can write where sat-isfies . Similarly to (30), if (27) holds over timeintervals of size , then

(33)

Since and when, it results from (31), (32), (33) that

with .

REFERENCES[1] V. Chudáček, J. Andén, S. Mallat, P. Abry, and M. Doret, “Scattering

transform for intrapartum fetal heart rate characterization and acidosisdetection,” presented at the IEEE Int. Conf. Eng.Med. Biol. Soc., 2013.

[2] H. Hermansky, “The modulation spectrum in the automatic recogni-tion of speech,” inProc. IEEE Autom. Speech Recognit. UnderstandingWorkshop, 1997, pp. 140–147.

[3] M. S. Vinton and L. E. Atlas, “Scalable and progressive audio codec,”in Proc. 2001 IEEE Int. Conf. Acoust., Speech, Signal Process.(ICASSP’01), 2001, vol. 5, pp. 3277–3280.

[4] J. McDermott and E. Simoncelli, “Sound texture perception viastatistics of the auditory periphery: Evidence from sound synthesis,”Neuron, vol. 71, no. 5, pp. 926–940, 2011.

[5] M. Ramona and G. Peeters, “Audio identification based on spectralmodeling of bark-bands energy and synchronization through onset de-tection,” in Proc. IEEE ICASSP, 2011, pp. 477–480.

[6] M. Slaney and R. Lyon, , M. Cooke, S. Beet, andM. Crawford, Eds., Vi-sual Representations of Speech Signals. Hoboken, NJ, USA: Wiley,1993, pp. 95–116.

[7] R. D. Patterson, “Auditory images: How complex sounds are repre-sented in the auditory system,” J. Acoust. Soc. Japan (E), vol. 21, no.4, pp. 183–190, 2000.

[8] C. Lee, J. Shih, K. Yu, and H. Lin, “Automatic music genre classifi-cation based on modulation spectral analysis of spectral and cepstralfeatures,” IEEE Trans. Multimedia, vol. 11, no. 4, pp. 670–682, Jun.2009.


[9] D. Ellis, X. Zeng, and J. McDermott, “Classifying soundtracks withaudio texture features,” in Proc. IEEE ICASSP, Prague, Czech Re-public, May 22–27, 2011, pp. 5880–5883.

[10] J. K. Thompson and L. E. Atlas, “A non-uniform modulation transformfor audio coding with increased time resolution,” in Proc. 2003 IEEEInt. Conf. Acoustics, Speech, Signal Process. (ICASSP’03), 2003, vol.5, pp. V–397.

[11] S. Mallat, “Group invariant scattering,” Commun. Pure Appl. Math.,vol. 65, no. 10, pp. 1331–1398, 2012.

[12] Y. LeCun, K. Kavukvuoglu, and C. Farabet, “Convolutional networksand applications in vision,” presented at the IEEE ISCAS, 2010.

[13] H. Lee, P. Pham, Y. Largman, and A. Ng, “Unsupervised featurelearning for audio classification using convolutional deep beliefnetworks,” presented at the NIPS, 2009.

[14] G. Hinton et al., “Deep neural networks for acoustic modeling inspeech recognition: The shared views of four research groups,” IEEESignal Process. Mag., vol. 29, no. 6, pp. 82–97, Dec. 2012.

[15] L. Deng, O. Abdel-Hamid, and D. Yu, “A deep convolutional neuralnetwork using heterogeneous pooling for trading acoustic invariancewith phonetic confusion,” presented at the ICASSP, 2013.

[16] A. Graves, A.-R. Mohamed, and G. Hinton, “Speech recognition withdeep recurrent neural networks,” presented at the ICASSP, 2013.

[17] E. J. Humphrey, T. Cho, and J. P. Bello, “Learning a robust ton-netz-space transform for automatic chord recognition,” in Proc. IEEEICASSP, 2012, pp. 453–456.

[18] P. Hamel and D. Eck, “Learning features from music audio with deepbelief networks,” presented at the ISMIR, 2010.

[19] E. Battenberg and D. Wessel, “Analyzing drum patterns using condi-tional deep belief networks,” presented at the ISMIR, 2012.

[20] T. Dau, B. Kollmeier, and A. Kohlrausch, “Modeling auditory pro-cessing of amplitude modulation. I. Detection and masking withnarrow-band carriers,” J. Acoust. Soc. Amer., vol. 102, no. 5, pp.2892–2905, 1997.

[21] T. Chi, P. Ru, and S. Shamma, “Multiresolution spectrotemporalanalysis of complex sounds,” J. Acoust. Soc. Am., vol. 118, no. 2, pp.887–906, 2005.

[22] N. Mesgarani, M. Slaney, and S. A. Shamma, “Discrimination ofspeech from nonspeech based on multiscale spectro-temporal modu-lations,” IEEE Audio, Speech, Language Process., vol. 14, no. 3, pp.920–930, 2006.

[23] J. Bruna and S. Mallat, “Invariant scattering convolution networks,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1872–1886,Aug. 2013.

[24] L. Sifre and S. Mallat, “Rotation, scaling and deformation invariantscattering for texture discrimination,” presented at the CVPR, 2013.

[25] S.Mallat, A Wavelet Tour of Signal Processing. NewYork, NY, USA:Academic Press, 1999.

[26] S. Schimmel and L. Atlas, “Coherent envelope detection for modula-tion filtering of speech,” in Proc. ICASSP, 2005, vol. 1, pp. 221–224.

[27] R. Turner and M. Sahani, “Probabilistic amplitude and frequency de-modulation,” Adv. Neural Inf. Process. Syst., pp. 981–989, 2011.

[28] G. Sell and M. Slaney, “Solving demodulation as an optimizationproblem,” IEEE Trans. Audio, Speech, Language Process., vol. 18,no. 8, pp. 2051–2066, Aug. 2010.

[29] I. Waldspurger and S. Mallat, “Phase Retrieval for the Cauchy wavelettransform,” J. Fourier Anal. Appl. [Online]. Available: http://arxiv.org/abs/1404.1183, submitted for publication

[30] M. Henaff, K. Jarrett, K. Kavukcuoglu, and Y. LeCun, “Unsupervisedlearning of sparse features for scalable audio classification,” presentedat the ISMIR, 2011.

[31] J. Nam, J. Herrera, M. Slaney, and J. Smith, “Learning sparse featurerepresentations for music annotation and retrieval,” presented at theISMIR, 2012.

[32] E. C. Smith and M. S. Lewicki, “Efficient auditory coding,” Nature,vol. 439, no. 7079, pp. 978–982, 2006.

[33] W. Fisher, G. Doddington, and K. Goudie-Marshall, “The DARPAspeech recognition research database: Specifications and status,” inProc. DARPA Workshop Speech Recognit., 1986, pp. 93–99.

[34] L. Lucy, “An iterative technique for the rectification of observed dis-tributions,” Astron. J., vol. 79, p. 745, 1974.

[35] E. J. Candès, Y. C. Eldar, T. Strohmer, and V. Voroninski, “Phase re-trieval via matrix completion,” SIAM J. Imaging Sci., vol. 6, no. 1, pp.199–225, 2013.

[36] I. Waldspurger, A. d’Aspremont, and S. Mallat, “Phase Recovery,Maxcut and Complex Semidefinite Programming,”Math. Programm.,pp. 1–35, 2013.

[37] D. W. Griffin and J. S. Lim, “Signal estimation from modified short-time Fourier transform,” IEEE Trans. Acoust., Speech, Signal Process.,vol. 32, no. 2, pp. 236–243, Feb. 1984.

[38] S. Davis and P. Mermelstein, “Comparison of parametric representa-tions for monosyllabic word recognition in continuously spoken sen-tences,” IEEE Trans. Acoust., Speech, Signal Process., vol. 28, no. 4,pp. 357–366, Apr. 1980.

[39] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vectormachines,” ACM Trans. Intell. Syst. Technol. vol. 2, pp. 27:1–27:27,2011 [Online]. Available: http://www.csie.ntu. edu.tw/~cjlin/libsvm,software available at

[40] G. Tzanetakis and P. Cook, “Musical genre classification of audio sig-nals,” IEEE Trans. Speech Audio Process., vol. 10, no. 5, pp. 293–302,Jul. 2002.

[41] J. Andén and S. Mallat, “Multiscale scattering for audio classification,”in Proc. ISMIR, Miami, FL, USA, Oct. 24–28, 2011, pp. 657–662.

[42] C. Baugé, M. Lagrange, J. Andén, and S. Mallat, “Representing envi-ronmental sounds using the separable scattering transform,” presentedat the IEEE ICASSP, 2013.

[43] H.-A. Chang and J. R. Glass, “Hierarchical large-margin Gaussianmix-ture models for phonetic classification,” in Proc. IEEE ASRU , 2007,pp. 272–277.

[44] X. Chen and P. J. Ramadge, “Music genre classification using mul-tiscale scattering and sparse representations,” presented at the CISS,2013.

[45] B. L. Sturm, “An analysis of the GTZANmusic genre dataset,” inProc.2nd Int. ACMWorkshop Music Inf. Retrieval With User-Centered Mul-timodal Strategies, 2012, pp. 7–12.

[46] K.-F. Lee and H.-W. Hon, “Speaker-independent phone recognitionusing hidden Markov models,” IEEE Trans. Acoust., Speech, SignalProcess., vol. 37, no. 11, pp. 1641–1648, Nov. 1989.

[47] P. Clarkson and P. J. Moreno, “On the use of support vector machinesfor phonetic classification,” IEEE Trans. Acoust., Speech, SignalProcess., vol. 2, pp. 585–588, Feb. 1999.

[48] A. K. Halberstadt, “Heterogeneous Acoustic Measurements and Mul-tiple Classifiers for Speech Recognition,” Ph.D. dissertation, Massa-chusetts Inst. Technol., Cambridge, MA, USA, 1998.

[49] T. N. Sainath, D. Nahamoo, D. Kanevsky, B. Ramabhadran, and P.Shah, “A convex hull approach to sparse representations for exemplar-based speech recognition,” in Proc. IEEE ASRU, 2011, pp. 59–64.

[50] J. Andén, “Time and Frequency Scattering for Audio Classification,”Ph.D. dissertation, Ecole Polytechnique, Palaiseau, France, 2014.

Joakim Andén (M’14) received the M.Sc. degreein mathematics from the Université Pierre et MarieCurie, Paris, France, in 2010 and the Ph.D. degreein applied mathematics from Ecole Polytechnique,Palaiseau, France, in 2014. His doctoral work con-sisted of studying the invariant scattering transformapplied to time series, such as audio and medicalsignals, in order to extract information relevant toclassification and other tasks.He is currently a postdoctoral researcher with the

Program in Applied and Computational Mathematicsat Princeton University, Princeton, USA. His research interests include signalprocessing, machine learning, and statistical data analysis. He is a member ofthe IEEE.

Stéphane Mallat (M’91–SM’02–F’05) received anengineering degree from Ecole Polytechnique, Paris,a Ph.D. in electrical engineering from the Universityof Pennsylvania, Philadelphia, in 1988, and an habil-itation in applied mathematics from Université Paris-Dauphine.In 1988, he joined the Computer Science Depart-

ment of the Courant Institute of Mathematical Sci-ences where he was Associate Professor in 1994 andProfessor in 1996. From 1995 to 2012, he was a fullProfessor in the Applied Mathematics Department at

Ecole Polytechnique, Paris. From 2001 to 2008 he was a co-founder and CEOof a start-up company. Since 2012, he joined the computer science departmentof Ecole Normale Supérieure, in Paris.Dr. Mallat is an IEEE and EURASIP fellow. He received the 1990 IEEE

Signal Processing Society’s paper award, the 1993 Alfred Sloan fellowship inMathematics, the 1997 Outstanding Achievement Award from the SPIE OpticalEngineering Society, the 1997 Blaise Pascal Prize in applied mathematics fromthe French Academy of Sciences, the 2004 European IST Grand prize, the 2004INIST-CNRS prize for most cited French researcher in engineering and com-puter science, and the 2007 EADS prize of the French Academy of Sciences.His research interests include computer vision, signal processing and har-

monic analysis.

Documents

4114 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 62, …mallat/papiers/AudioScatSpectrum.pdf · 4114 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 62, NO. 16, AUGUST 15, 2014 Deep