1. Models and Methods - uni-bielefeld.degibbon/Guangzhoulectures2016/... · 2016. 11. 4. · Dafydd...

Prosody: speech rhythms and melodies

1. Models and Methods

Dafydd Gibbon

Guangzhou Prosody Lectures, November 2016

Dafydd Gibbon, Guangzhou Prosody Lectures, November 2016 Lecture 01: Models and Methods 2

Schedule

Week 1:01 Forms and functions of prosody: models and methods

Nov. 2 (Wednesday) 4:30pm--6:00pm02 Forms and functions of prosody: prosodic semantics

Nov. 3 (Thursday) 8am—9:40am

Week 2:03 Basics of digital phonetics

Nov. 8 (Tuesday) 10am--12am04 Pitch Stylisation: tone and intonation

Nov. 8 (Tuesday) 2:30pm--4:30pm

Week 3:05 Computational modelling of syllables

Nov.15 (Tuesday) 10am--12am06 SpeechTiming: durations and rhythm

Nov.15 (Tuesday) 2:30pm--4:30pm

Prosody: Rhythm and Melody

PITCH TRACE

WAVEFORM

Aix-MARSEC corpus, extract from A0101B, visualisation with PViz

WAVEFORM

syllable annotation tier pitch accent annotation tierannotation labels

Aix-MARSEC corpus, extract from A0101B, visualisation with Praat

PITCH TRACEwith linear regression line

showing the overall contour trend

Pitch accent notations – three kinds of information:1. Shape of pitch accent contour2. Main pitch accent tone associated with a stressed syllable3. Height of pitch accent on a frequency scale

Level and Contour Tone Lexicon: Notations (English)

H* L* L*H LH* H*L HL* H*HTranscription symbol notation:

Icon notation:

shape=pointplace=H

shape=riseanchor=startplace=L

shape=riseanchor=endplace=L

shape=fallanchor=startplace=H

shape=fallanchor=endplace=L

shape=levelanchor=startplace=H

Feature notation:

shape=pointplace=L

H* L* L*H LH* H*L HL* H*H

shape=pointplace=H

shape=riseanchor=startplace=L

shape=riseanchor=endplace=L

shape=fallanchor=startplace=H

shape=fallanchor=endplace=L

shape=levelanchor=startplace=H

Feature notation:

shape=pointplace=L

Icon notation:

Transcription symbol notation:

All you need to know about prosody

Prosody is physical actionProsody must be practised – that is, observed and performed like music, sports, cooking, ...

Prosody is semiotic actionLike other components of discourse, prosody must be communicated and understood

Prosody is physical action

Prosody must be practised – that is, observed and performed like music, dancing, sports, cooking, ...

Prosody is semiotic actionLike other components of discourse, prosody must be communicated and understood

Prosody is structured actionLike word and sentence structure, prosody must be systematically generated and parsed.

Prosody is physical action:Melody – Rhythm - Intensity

Prosody is semiotic action:Meaning – Intentions - Attitudes

Prosody is structured action:Starts – Accents/Tones - Ends

Like other components of discourse, prosody must be communicated and understood

Like word and sentence structure, prosody must be systematically generated and parsed.

Prosody is physical action:Melody – Rhythm - Intensity

Prosody is semiotic action:Meaning – Intentions -

Attitudes

Like other components of discourse, prosody must be communicated and understood

Prosody is structured action:Starts – Accents/Tones -

Like word and sentence structure, prosody must be systematically generated and parsed.

These three points applyto language learners,language teachers,

translators, interpreters,linguists – and of course

to “real people”!

Models and Methods

Models

What do all these have in common?What do they share?

So how would you define “model”?

(A picture is also a model, so these are actually models of models!)

Models

A model is a simplified representation of reality.

Models resemble reality, in colour, shape, ...

Models differ from reality, in material, size, dimensionality, ...

REALITY(or rather: a picture model of reality)

Scientific Models – Linguistic Models

Conceptual Physical

• analog• digital

Statistical Mathematical Visualisation

complex, multidimensional, ...

simplification

approximation

REALITY(or rather: a picture model of reality)

Scientific Models – Linguistic ModelsMODEL

complex, multidimensional, ...

simplification

approximation

Three dimensions of scientific method

Scientific Method

● A scientific method is a technique for creating scientific models of selected aspects of reality– formalism:

● informal text● formalism – logic, mathematics

– observation● intuition● measurement

– corpus– experiment

● generalisation– hypothetico-deductive– inductive– interpretative

– domain● empirical basis● simplfied reality

formalism

domain

observation

Scientific Methodology

● Scientific method: a technique for creating scientific models of selected aspects of reality

● Scientific theory: a logical system for● describing models of language and speech● predicting new models of language and speech● and so (indirectly) describing and predicting reality

● Scientific framework: shared conventions– theories as predictors for models– models as simplifications of reality– legitimate methods for creating models

Scientific Methodology

● Scientific method: a technique for creating scientific models of selected aspects of reality

● Scientific theory: a logical system for● describing models of language and speech● predicting new models of language and speech● and so (indirectly) describing and predicting reality

● Scientific framework: shared conventions– theories as predictors for models– models as simplifications of reality– legitimate methods for creating models

BEWARE

The terms ‘theory’, ‘model’ and ‘framework’are often confusingly used as synonyms

in linguistics.

The definitions given here are reasonablyconsensual in studies of scientific methodology.

Major issues in prosody

Some questions on prosody to think about

● Basic ontology:– Are prosodic properties categorial or continuous?– By analogy with the phoneme and the morpheme, are

there prosodemes? intonemes? tonemes? accenteme?– Are there clear relations with categories of written text

such as character, word, phrase, clause, sentence, paragraph, text? Are there paratones?

– Is there a prosodic grammar?– Is there a prosodic lexicon

● Basic epistemology:– Is prosody teachable or only learnable?– Do we share prosody with animals, e.g. apes?

A Practical Illustration of Prosodic Model-Fitting

(Aix-MARSEC corpus, extract from A0101B)

... and of course .

Aix-MARSEC corpus, extract from A0101B, visualisation with Praat

... and of course ..

● Trying to fit a model to the pitch pattern:– First, straight line– Second, quadratic regression– Then higher order polynomial regression

Orientation: the architecture of language

functional models

grammar-based models

processing models

Functional declarative architectures

GRAMMAR

SEMANTICCONSTRUCTION

PHONETICS

Grammar-based declarative architectures

GRAMMAR

SEMANTICINTERPRETATION

PHONETICINTERPRETATION

GRAMMAR

CLASSIC GENERATIV

inventory+

rules+

derivation

GRAMMAR

OPTIMALITYTHEORY

inventory+

constraints+

resolution

GRAMMAR

CLASSIC GENERATIV

inventory+

rules+

derivation

OPTIMALITYTHEORY

inventory+

constraints+

resolution

GRAMMAR

CLASSIC GENERATIV

inventory+

rules+

derivation

OPTIMALITYTHEORY

inventory+

constraints+

resolution

algebra plus logical inference

algebra plus set theoretic

inference

FORMAL FOUNDATIONS

But phoneticians like a challenge!

Communication based procedural architectures

● Empirical speech modelling is not independent of– processing constraints:

– sensory constraints (perception)● motor constraints (production)● cognitive constraints:

– processing time + complexity– memory space + structure

– channel constraints● physical

– auditory, visual, text, electronics● social

– face-to-face, distal, unidirectional, multidirectional● noise

– errors, interference

The old ‘competence’ idea is far too simple to do

justice to the complexity of language and language processing constraints

Maybe ‘performance’ is simply too difficult?

Communication is bidirectional

GRAMMAR

Processing is bidirectional

UNDERSTANDING

FORMULATION

Cf. Levelt and many others...

PLANNING

OUTPUT

PARSING

PHYSICAL CHANNEL

The model

applies to

individals as

well as to

interlocutors

Semiotic architecture: the Rank Interpretation Model

SEMANTICand

PRAGMATICCONSTRUCTION

PHONETIC and VISUAL

INTERPRETATION

indices & icons

GRAMMAR

parsing

understanding

perception

SEMANTICand

PHONETIC and VISUAL

INTERPRETATION

indices & icons

GRAMMAR

planning

generation production

SEMANTICand

PHONETIC and VISUAL

INTERPRETATIONGRAMMAR

SEMANTIC and PRAGMATIC

INTERPRETATION

PHONETIC and VISUAL

INTERPRETATION

PHONETIC and VISUAL

INTERPRETATION

SEMANTIC and PRAGMATIC

INTERPRETATION

PROSODY

Empirical methods: Illustration

listening + descriptiontranscriptionannotation

data-miningsignal analysis

The approach

● Primarily:● acoustic phonetics● empirical, data-driven, authentic data● semiotic: relations of form and function● inductive/interpretive: recognition of patterns in data● descriptive: no adoption of ready-made paradigms● computational

● Secondarily:● rational, theory-driven – but beware: don’t invent data!!!● structural: patterns and rules● hypothetico-deductive: prediction of patterns in data● evaluative: examination of existing paradigms● comparison with other approaches

– generative: derivational (SPE, AMP), constraint-based (OT)

The domain

● Phonetic phases:– sender: articulatory domain– channel: acoustic domain– receiver: auditory domain

● Empirical methods:– observational

● perceptual● instrumental

– experimental● production● perceptionl

Empirical procedure: analytic listening

An example from German

from the fable

“Der Nordwind und die Sonne”(The North Wind and the Sun)

Empirical procedure: transcription

● Transcription– IPA phonetic transcription:

[ ˈɛntlɪç ɡaːp dɐ ˈnɔɐ̯tvɪn dəŋ ˈkamp͡f ˈaʊf̯ ]

– IPA phonemic transcription:– / ˈɛntlɪç ɡaːp deɐ ˈnɔɐtvɪnt dən ˈkamp͡f ˈaʊf /

● Two questions:– How do we make a detailed phonetic description? -

Annotation!– How do we transcribe the prosody? - This will come later!

Endlich gab der Nordwind den Kampf auf.

Empirical procedure: annotation (Praat screenshot)

Empirical procedure: annotation (Praat drawing)

Empirical procedure: annotation

● Annotation:– Creation of pairs of labels and time-stamps

<vint, 1.309, 1.538>

– In Praat TextGrid format: intervals [8]: xmin = 1.3087321400301957 xmax = 1.537624753139184 text = "vint"

Domains of Prosody

Forms of prosody: parameter domains

● The classic three domains of Prosody:– melody:

● Phonetics: variation of fundamental frequency in time● Grammar: intonation, pitch accent, tone

– volume:● Phonetics: variation of intensity in time● Grammar: accent – focus / contrast / emphasis

– timing:● Phonetics: variation of unit durations in time

– Note that duration has three temporal dimensions:● duration of units (syllables, phrases, ...), rhythm● rate (speed, tempo) of utterance● acceleration, deceleration

● Grammar: phonemic; accent – focus / contrast / emphasis

Forms of prosody: the two main systems

● Melody– complex function of f0 at different rank levels

● phones (f0 perturbations by consonants and vowels)● syllables (contrastive functions of tones)● words (morphemic functions of tones)● phrases (structuring, attitudinal, iconic meanings tones)

● Rhythm– complex epiphenomenal effect of

● duration of phonetic units at different rank levels– phones, syllables, words, phrases, ...

● intensity variation at different rank levels– phones, syllables, words, phrases, …

● melody variation at different rank levels● top-down prediction by listener

Summary: Forms of Prosody – Definitions

● Basic perspective:● Prosodic features are temporal properties of sequences of

speech units

● Alternative perspective:● Prosodic properties (features) are time functions of

phonetic features– typically of, melody, volume, durations– but also of other language specific features including nasality, vowel

harmony

● Traditional perspective:Prosodic features are phonetic features whose temporal functional domain is larger than the phoneme

(hence one of the tems. suprasegmental features)

Forms of prosody: paradigmatic structures

● Paradigmatic structures: classification– inventories of items which contrast with each other and

enter into syntagmatic structures● tones:

– African tone languages: high / low; high / mid / low, ...– Mandarin Chinese: high, rise, fall-rise, fall

● pitch accents:– English:

● stress position: primary stress, secondary stress, unstressed● phonetic interpretation as pitch accent:

high, low, rise, fall, rise-fall, fall-rise, rise-fall-rise● intonation units:

– global contour: high / low; rising / falling; final accent / tone– local contour:

● focus / contrast / emphatic accents● paratones:

– falling / rising; wider / narrower range

Forms of prosody: syntagmatic structure

● Syntagmatic structure: composition– small units join into larger units of a different type

● linear sequence of consecutive units● hierarchical grouping of smaller into larger units● parallel synchronisation with overlapping association

● Requirements for processing models:– To model prosody, two kinds of compositional operation

are needed:● consecutive, sequential, temporal relations between speech

units– vocabularies of units at each rank of the language hierarchy– combinatorial principles for these units

● concurrent, parallel temporal relations between speech units – synchronisation, association

Forms of prosody: syntagmatic structure

● In the Rank Interpretation Architecture:– consecutive and concurrent processing operations

apply to● the phonetic gestures of speech● but also to the gestures used in writing

– handwriting– typing

● conversational gestures:– face and arm gesture, posture, body language– beats, deictics, metaphorics, ...

● sign language:– different languages of the deaf– in many different communities

Phonetic correlates – pitch, intensity, timing

Phonetic correlates: acoustic measurements

amplitude; intensity = f(amplitude2)

voicing

pitch track

global downtrend model,declination model

smooth(f0) – f0 =

microprosody

global downtrend modeldeclination model

pitch perturbation model microprosody model

smooth(f0) – f0 =

microprosody

boundary tone

local pitch accents

local pitch accents boundary tone

amplitude; intensity = f(amplitude2)

local pitch accents

global downtrend modeldeclination model

boundary tonepitch perturbation model microprosody model

voicing

pitch trackf0

Summary of pitch-based prosodic categories

● Global prosody model:– global downtrend / declination / inclination– boundary tones

● Local prosody model:– pitch accents– tones

● Microprosody model:– intrinsic pitch of vowels– pitch perturbations by consonants

Summary in terms of the RIA approach

GLOBALPROSODY

LOCALPROSODY

Summary: what you should know about by now

● The place of prosody in architectures of language:– functional, grammar-based, procedural architectures– the Rank Interpretation Architecture

● Methods of prosodic analysis– listening, transcription, annotation– prosodic domains, parameters, methods– acoustic phonetic analysis

1. Models and Methods - uni-bielefeld.degibbon/Guangzhoulectures2016/... · 2016. 11. 4. · Dafydd...

Documents

Bywyd Dafydd 2

MST209 Mathematical Methods and Models

Upper GI for Finals - Dafydd Loughran

Illumination Models & Surface-Rendering Methods

Dafydd Dil Photography

Predictive models and methods - Logistics

Clinical surface anatomy - Dafydd Loughran

Dafydd y Garreg Wen

Bayesian Methods for Adaptive Models

Methods & Models of Business Strategy

Wrist fractures - Dafydd Loughran

Dafydd ap Gwilym: Capers and Caricatures

Biofilm Models and Methods Of

Dafydd ap Gwilym - ‘Yr Wylan ’

Dynamical Models James Binney Oxford University. Outline Importance of models Importance of models Modelling methods Modelling methods Schoenrich’s model

Information Retrieval: Models and Methods

mhtml:file://H:ERepositoryPainting the World Green Dafydd Iwaorca.cf.ac.uk/5364/1/EWynJamesPaintingtheWorldGreen.pdf · Painting the World Green: Dafydd Iwan and the Welsh Protest

Dafydd ap Gwilym: Reflections and Refractions

Review of Existing Methods, Models

DAFYDD JONES: THE LAST HURRAH · 2018-08-25 · DAFYDD JONES: THE LAST HURRAH Snoggers, Hammersmtih Palais, London, 1981 British photographer Dafydd Jones (b.1956) has worked as social