1. Models and Methods - uni-bielefeld.degibbon/Guangzhoulectures2016/... · 2016. 11. 4. · Dafydd Gibbon, Guangzhou Prosody Lectures, November 2016 Lecture 01: Models and Methods

Prosody: speech rhythms and melodies

1. Models and Methods

Dafydd Gibbon

Guangzhou Prosody Lectures, November 2016

Dafydd Gibbon, Guangzhou Prosody Lectures, November 2016 Lecture 01: Models and Methods 2

Schedule

Week 1:01 Forms and functions of prosody: models and methods

Nov. 2 (Wednesday) 4:30pm--6:00pm02 Forms and functions of prosody: prosodic semantics

Nov. 3 (Thursday) 8am—9:40am

Week 2:03 Basics of digital phonetics

Nov. 8 (Tuesday) 10am--12am04 Pitch Stylisation: tone and intonation

Nov. 8 (Tuesday) 2:30pm--4:30pm

Week 3:05 Computational modelling of syllables

Nov.15 (Tuesday) 10am--12am06 SpeechTiming: durations and rhythm

Nov.15 (Tuesday) 2:30pm--4:30pm


Prosody: Rhythm and Melody

PITCH TRACE

WAVEFORM

Aix-MARSEC corpus, extract from A0101B, visualisation with PViz






WAVEFORM

syllable annotation tier pitch accent annotation tierannotation labels

Aix-MARSEC corpus, extract from A0101B, visualisation with Praat






PITCH TRACEwith linear regression line

showing the overall contour trend



Pitch accent notations – three kinds of information:1. Shape of pitch accent contour2. Main pitch accent tone associated with a stressed syllable3. Height of pitch accent on a frequency scale

Level and Contour Tone Lexicon: Notations (English)




H* L* L*H LH* H*L HL* H*HTranscription symbol notation:




Icon notation:




shape=pointplace=H

shape=riseanchor=startplace=L

shape=riseanchor=endplace=L

shape=fallanchor=startplace=H

shape=fallanchor=endplace=L

shape=levelanchor=startplace=H

Feature notation:

shape=pointplace=L




H* L* L*H LH* H*L HL* H*H

shape=pointplace=H

shape=riseanchor=startplace=L

shape=riseanchor=endplace=L

shape=fallanchor=startplace=H

shape=fallanchor=endplace=L

shape=levelanchor=startplace=H

Feature notation:

shape=pointplace=L

Icon notation:

Transcription symbol notation:


All you need to know about prosody

Prosody is physical actionProsody must be practised – that is, observed and performed like music, sports, cooking, ...



Prosody is physical actionProsody must be practised – that is, observed and performed like music, sports, cooking, ...

Prosody is semiotic actionLike other components of discourse, prosody must be communicated and understood



Prosody is physical action

Prosody must be practised – that is, observed and performed like music, dancing, sports, cooking, ...

Prosody is semiotic actionLike other components of discourse, prosody must be communicated and understood

Prosody is structured actionLike word and sentence structure, prosody must be systematically generated and parsed.



Prosody is physical action:Melody – Rhythm - Intensity

Prosody is semiotic action:Meaning – Intentions - Attitudes

Prosody is structured action:Starts – Accents/Tones - Ends


Like other components of discourse, prosody must be communicated and understood

Like word and sentence structure, prosody must be systematically generated and parsed.



Prosody is physical action:Melody – Rhythm - Intensity


Prosody is semiotic action:Meaning – Intentions -

Attitudes

Like other components of discourse, prosody must be communicated and understood

Prosody is structured action:Starts – Accents/Tones -

Ends

Like word and sentence structure, prosody must be systematically generated and parsed.

These three points applyto language learners,language teachers,

translators, interpreters,linguists – and of course

to “real people”!


Models and Methods


Models


Models

What do all these have in common?What do they share?

So how would you define “model”?

(A picture is also a model, so these are actually models of models!)


Models

A model is a simplified representation of reality.

Models resemble reality, in colour, shape, ...

Models differ from reality, in material, size, dimensionality, ...


REALITY(or rather: a picture model of reality)

Scientific Models – Linguistic Models

MODEL

Conceptual Physical

• analog• digital

Statistical Mathematical Visualisation

complex, multidimensional, ...

simplification

approximation


REALITY(or rather: a picture model of reality)

Scientific Models – Linguistic ModelsMODEL

complex, multidimensional, ...

simplification

approximation


Three dimensions of scientific method

Scientific Method

● A scientific method is a technique for creating scientific models of selected aspects of reality– formalism:

● informal text● formalism – logic, mathematics

– observation● intuition● measurement

– corpus– experiment

● generalisation– hypothetico-deductive– inductive– interpretative

– domain● empirical basis● simplfied reality

formalism

domain

observation


Scientific Methodology

● Scientific method: a technique for creating scientific models of selected aspects of reality

● Scientific theory: a logical system for● describing models of language and speech● predicting new models of language and speech● and so (indirectly) describing and predicting reality

● Scientific framework: shared conventions– theories as predictors for models– models as simplifications of reality– legitimate methods for creating models


Scientific Methodology

● Scientific method: a technique for creating scientific models of selected aspects of reality

● Scientific theory: a logical system for● describing models of language and speech● predicting new models of language and speech● and so (indirectly) describing and predicting reality

● Scientific framework: shared conventions– theories as predictors for models– models as simplifications of reality– legitimate methods for creating models

BEWARE

The terms ‘theory’, ‘model’ and ‘framework’are often confusingly used as synonyms

in linguistics.

The definitions given here are reasonablyconsensual in studies of scientific methodology.


Major issues in prosody


Some questions on prosody to think about

● Basic ontology:– Are prosodic properties categorial or continuous?– By analogy with the phoneme and the morpheme, are

there prosodemes? intonemes? tonemes? accenteme?– Are there clear relations with categories of written text

such as character, word, phrase, clause, sentence, paragraph, text? Are there paratones?

– Is there a prosodic grammar?– Is there a prosodic lexicon

● Basic epistemology:– Is prosody teachable or only learnable?– Do we share prosody with animals, e.g. apes?


A Practical Illustration of Prosodic Model-Fitting

(Aix-MARSEC corpus, extract from A0101B)



... and of course .

..

Aix-MARSEC corpus, extract from A0101B, visualisation with Praat



... and of course ..

.

● Trying to fit a model to the pitch pattern:– First, straight line– Second, quadratic regression– Then higher order polynomial regression



Orientation: the architecture of language

functional models

grammar-based models

processing models


Functional declarative architectures

GRAMMAR

SEMANTICCONSTRUCTION

PHONETICS


Grammar-based declarative architectures

GRAMMAR

SEMANTICINTERPRETATION

PHONETICINTERPRETATION



GRAMMAR



CLASSIC GENERATIV

E

inventory+

rules+

derivation



GRAMMAR



OPTIMALITYTHEORY

inventory+

constraints+

resolution



GRAMMAR



CLASSIC GENERATIV

E

inventory+

rules+

derivation

OPTIMALITYTHEORY

inventory+

constraints+

resolution



GRAMMAR



CLASSIC GENERATIV

E

inventory+

rules+

derivation

OPTIMALITYTHEORY

inventory+

constraints+

resolution

algebra plus logical inference

algebra plus set theoretic

inference

FORMAL FOUNDATIONS


But phoneticians like a challenge!

Communication based procedural architectures

● Empirical speech modelling is not independent of– processing constraints:

– sensory constraints (perception)● motor constraints (production)● cognitive constraints:

– processing time + complexity– memory space + structure

– channel constraints● physical

– auditory, visual, text, electronics● social

– face-to-face, distal, unidirectional, multidirectional● noise

– errors, interference

The old ‘competence’ idea is far too simple to do

justice to the complexity of language and language processing constraints

Maybe ‘performance’ is simply too difficult?


Communication is bidirectional

GRAMMAR




Processing is bidirectional

UNDERSTANDING

FORMULATION

INPUT

Cf. Levelt and many others...

PLANNING

OUTPUT

PARSING

PHYSICAL CHANNEL

The model

applies to

individals as

well as to

interlocutors


Semiotic architecture: the Rank Interpretation Model

SEMANTICand

PRAGMATICCONSTRUCTION

PHONETIC and VISUAL

INTERPRETATION

LE

XIC

ON

indices & icons

GRAMMAR

parsing

understanding

perception



SEMANTICand


PHONETIC and VISUAL

INTERPRETATION

LE

XIC

ON

indices & icons

GRAMMAR

planning

generation production



SEMANTICand


PHONETIC and VISUAL

INTERPRETATIONGRAMMAR



SEMANTIC and PRAGMATIC

INTERPRETATION

PHONETIC and VISUAL

INTERPRETATION



PHONETIC and VISUAL

INTERPRETATION



SEMANTIC and PRAGMATIC

INTERPRETATION





PROSODY


Empirical methods: Illustration

listening + descriptiontranscriptionannotation

data-miningsignal analysis


The approach

● Primarily:● acoustic phonetics● empirical, data-driven, authentic data● semiotic: relations of form and function● inductive/interpretive: recognition of patterns in data● descriptive: no adoption of ready-made paradigms● computational

● Secondarily:● rational, theory-driven – but beware: don’t invent data!!!● structural: patterns and rules● hypothetico-deductive: prediction of patterns in data● evaluative: examination of existing paradigms● comparison with other approaches

– generative: derivational (SPE, AMP), constraint-based (OT)


The domain

● Phonetic phases:– sender: articulatory domain– channel: acoustic domain– receiver: auditory domain

● Empirical methods:– observational

● perceptual● instrumental

– experimental● production● perceptionl


Empirical procedure: analytic listening

An example from German

from the fable

“Der Nordwind und die Sonne”(The North Wind and the Sun)


Empirical procedure: transcription

● Transcription– IPA phonetic transcription:

[ ˈɛntlɪç ɡaːp dɐ ˈnɔɐ̯tvɪn dəŋ ˈkamp͡f ˈaʊf̯ ]

– IPA phonemic transcription:– / ˈɛntlɪç ɡaːp deɐ ˈnɔɐtvɪnt dən ˈkamp͡f ˈaʊf /

● Two questions:– How do we make a detailed phonetic description? -

Annotation!– How do we transcribe the prosody? - This will come later!

Endlich gab der Nordwind den Kampf auf.



Empirical procedure: annotation (Praat screenshot)



Empirical procedure: annotation (Praat screenshot)



Empirical procedure: annotation (Praat drawing)



Empirical procedure: annotation

● Annotation:– Creation of pairs of labels and time-stamps

<vint, 1.309, 1.538>

– In Praat TextGrid format: intervals [8]: xmin = 1.3087321400301957 xmax = 1.537624753139184 text = "vint"


Domains of Prosody


Forms of prosody: parameter domains

● The classic three domains of Prosody:– melody:

● Phonetics: variation of fundamental frequency in time● Grammar: intonation, pitch accent, tone

– volume:● Phonetics: variation of intensity in time● Grammar: accent – focus / contrast / emphasis

– timing:● Phonetics: variation of unit durations in time

– Note that duration has three temporal dimensions:● duration of units (syllables, phrases, ...), rhythm● rate (speed, tempo) of utterance● acceleration, deceleration

● Grammar: phonemic; accent – focus / contrast / emphasis


Forms of prosody: the two main systems

● Melody– complex function of f0 at different rank levels

● phones (f0 perturbations by consonants and vowels)● syllables (contrastive functions of tones)● words (morphemic functions of tones)● phrases (structuring, attitudinal, iconic meanings tones)

● Rhythm– complex epiphenomenal effect of

● duration of phonetic units at different rank levels– phones, syllables, words, phrases, ...

● intensity variation at different rank levels– phones, syllables, words, phrases, …

● melody variation at different rank levels● top-down prediction by listener


Summary: Forms of Prosody – Definitions

● Basic perspective:● Prosodic features are temporal properties of sequences of

speech units

● Alternative perspective:● Prosodic properties (features) are time functions of

phonetic features– typically of, melody, volume, durations– but also of other language specific features including nasality, vowel

harmony

● Traditional perspective:Prosodic features are phonetic features whose temporal functional domain is larger than the phoneme

(hence one of the tems. suprasegmental features)


Forms of prosody: paradigmatic structures

● Paradigmatic structures: classification– inventories of items which contrast with each other and

enter into syntagmatic structures● tones:

– African tone languages: high / low; high / mid / low, ...– Mandarin Chinese: high, rise, fall-rise, fall

● pitch accents:– English:

● stress position: primary stress, secondary stress, unstressed● phonetic interpretation as pitch accent:

high, low, rise, fall, rise-fall, fall-rise, rise-fall-rise● intonation units:

– global contour: high / low; rising / falling; final accent / tone– local contour:

● focus / contrast / emphatic accents● paratones:

– falling / rising; wider / narrower range


Forms of prosody: syntagmatic structure

● Syntagmatic structure: composition– small units join into larger units of a different type

● linear sequence of consecutive units● hierarchical grouping of smaller into larger units● parallel synchronisation with overlapping association

● Requirements for processing models:– To model prosody, two kinds of compositional operation

are needed:● consecutive, sequential, temporal relations between speech

units– vocabularies of units at each rank of the language hierarchy– combinatorial principles for these units

● concurrent, parallel temporal relations between speech units – synchronisation, association


Forms of prosody: syntagmatic structure

● In the Rank Interpretation Architecture:– consecutive and concurrent processing operations

apply to● the phonetic gestures of speech● but also to the gestures used in writing

– handwriting– typing

● conversational gestures:– face and arm gesture, posture, body language– beats, deictics, metaphorics, ...

● sign language:– different languages of the deaf– in many different communities


Phonetic correlates – pitch, intensity, timing


Phonetic correlates: acoustic measurements




amplitude; intensity = f(amplitude2)




voicing

pitch track




global downtrend model,declination model


smooth(f0) – f0 =

microprosody



global downtrend modeldeclination model

pitch perturbation model microprosody model


smooth(f0) – f0 =

microprosody



boundary tone



local pitch accents




local pitch accents boundary tone




amplitude; intensity = f(amplitude2)

local pitch accents

global downtrend modeldeclination model

boundary tonepitch perturbation model microprosody model

voicing

pitch trackf0



Summary of pitch-based prosodic categories

● Global prosody model:– global downtrend / declination / inclination– boundary tones

● Local prosody model:– pitch accents– tones

● Microprosody model:– intrinsic pitch of vowels– pitch perturbations by consonants


Summary in terms of the RIA approach





GLOBALPROSODY

MODEL

LOCALPROSODY

MODEL


Summary: what you should know about by now

● The place of prosody in architectures of language:– functional, grammar-based, procedural architectures– the Rank Interpretation Architecture

● Methods of prosodic analysis– listening, transcription, annotation– prosodic domains, parameters, methods– acoustic phonetic analysis

Documents

1. Models and Methods - uni-bielefeld.degibbon/Guangzhoulectures2016/... · 2016. 11. 4. · Dafydd Gibbon, Guangzhou Prosody Lectures, November 2016 Lecture 01: Models and Methods