Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Formant Based Synthesizer
Qin Yan
Communication & Multimedia Signal Processing Group
Dept of Electronic & Computer Engineering, Brunel University
28 July, 2004
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Main Progress
• Kalman filter based formant tracking system in clean speech
• Speech Synthesis via formant tracks
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Formant Candidate Estimation
LP PoleAnalysis
Kalman Filter
NoisySpeech
Restored Formant & Bandwidth
tracks
Formant Candidate
Estimation
Kalman Filter
Vowel/Consonant
Classification
Voiced?Yes
No
Noise Model
LP-basedSpectral
Subtraction
VAD
Pos.& neg. PolesReconstruction
LP Spectrum Reconstruction
Residual
Real Pole
Speech Reconstruction
Enhanced Speech
Formant Track Restoration Module
Formant based Speech Enhancement System
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
ConfidenceScore
Calculation
LP Pole Analysis
Kalman Filter
Clean Speech
Formant & Bandwidth
tracks
Real Poles
Speech Reconstruction
Output Speech
Residual
Confidence Score
Calculation
Kalman Filter
Positive Poles
Vowel/Consonant
ClassificationVowel?
Yes
No
Formant Candidate
Interpolation
Formant Candidate
Interpolation
Speech Synthesis System
Kalman Filter based Formant Tracker for Clean Speech
Speech Synthesizer via Formant Tracks
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Vowel/Consonant Classification
• Discriminant feature used is the slope coefficient of a 1st order polynomial of LP spectrum;• Positive slope: Consonant; Negative slope: Vowel
Confidence Scores of Formant Candidates
• The score quantifies how significant a pole is • Score for Vowels: Mag(m) /BW(m)• Score for Consonant: m*Mag(m) / BW(m)• The candidate with highest score is interpolated with the closest formant candidate. The rest of formant candidates are sorted in ascending order.• Interpolation function:
Where W(m) is the weights
Parallel Kalman Filters
• Two kalman filters: One for vowel segments, the other for consonant segments.
Kalman Filter based Formant Track in Clean Speech
22 )),1(),((*),(
1)(
ktFkmPkmPmW
FBW
Nk
m
N
mF
mW
mWkmPk
1
1
)(
)(*),(
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Performance
Red : Formant tracks from 2D-HMM; Green : Formant tracks from Kalman filter
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Speech Synthesis via Formant tracks
Pos.& neg. PolesReconstruction
Noisy Speech
Real Pole
Speech Reconstruction
Enhanced Speech
Residual
RestoredFormant track
LP Pole Analysis
Real poles are included to adjust the slope of LP spectrum
LP order = Number of formant tracks + 1
HMM based Formant tracks
Kalman Filter based Formant Tracks
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
Performance Evaluation
-1
4
9
14
19
24
0 5 10 15 20Global SNR(dB)
snris1kl/mean(kl)IS2
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
ConfidenceScore
Calculation
LP Pole Analysis
Kalman Filter
Clean Speech
Formant & Bandwidth
tracks
Real Poles
Speech Reconstruction
Output Speech
Residual
Confidence Score
Calculation
Kalman Filter
Positive Poles
Vowel/Consonant
ClassificationVowel?
Yes
No
Formant Candidate
Interpolation
Formant Candidate
Interpolation
Kalman Filter based Formant Tracker for Clean Speech
Speech Synthesizer via Formant Tracks
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
SignificanceScore
Calculation
LP PoleAnalysis
Kalman Filter
NoisySpeech
Formant & Bandwidth
tracks
Significance Score
Calculation
Kalman Filter
Vowel/Consonant
Classification
Voiced? Yes
No
Formant Candidate
Interpolation
Formant Candidate
Interpolation
Noise Model
LP-basedSpectral
Subtraction
VAD
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing Source
Speech
Cepstral Feature Analysis
LP Pole Analysis
Speech HMMs
Training
FormantFeatures
Extraction
Speech Labelling &Segmentation
Formant HMMsTraining
Formant candidates classification
Formant Candidates Interpolation
Formant Tracks
State-dependentKalman Filter
R
Fi , BWi
Co
mm
un
icat
ion
s &
Mu
ltim
edia
Sig
nal
Pro
cess
ing
LP PoleAnalysis
NoisySpeech
Restored Formant & Bandwidth
tracks
Formant Candidate
Estimation
Kalman Filter
Vowel/Consonant
ClassificationLP ModelOf Noise
LP-Analysisand
LP-Spectral Subtraction
VAD
Pos.& neg. PolesReconstruction
LP Spectrum Reconstruction
ResidualSpeech
ReconstructionEnhanced
Speech
Formant Track Restoration Module