Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
DT2119: Speech and Speaker RecognitionCourse Analysis VT2018
Course analysis carried out by (name, e-mail):
Giampiero Salvi, [email protected]
COURSE DESIGN
Briefly describe the course design (learning activities, examinations) and any changesthat have been implemented since the last course offering.
In VT2018 the course consisted of 12 lectures, 3 computer labs each evaluated with an oralpresentation and a final project evaluated with a written report, peer review and a poster session.Special care was put into aligning the learning outcomes with the teaching, learning activitiesand assessment. This is a course that, besides the specific topic, intends to teach the basicresearch paradigm, based on performing scientific studies, peer review, and public presentations.
The main differences with previous years were the following:
• the third lab on deep learning was completely rewritten and now is based on Pythonand TensorFlow. This improves the pedagogical value of the lab because it lets studentsimplement most of the methods used,
• improvement were made to lab 1 and 2 to address some of the feedback received in VT2017,
• grading criteria were improved,
• some improvements were made to the software available at Parallel Data Centre machinesto simplify use for the students
• credit to the Google Cloud Platform was obtained under an academic licence and theservice was made available to the students
THE STUDENT’S WORKLOAD
Does the students’ workload correspond to the expected level (40 hours/1.5 credits)?If there is a significant deviation from the expected, what can be the reason?
The students who participated in this course belonged to eight different programs (CDATE,CINTE, CMIEL, D, TMAIM, TMETM, TMLEM, TSCRM) and had, therefore different back-grounds. Consequently, the workload exhibits large variations, Besides one student who reported33–35 hours/week load, most students range from 3–5 to 18–20 hours/week. On average, thework load is 12.5–14.5 hours/week which is low compared to the 20 hours/week required for thiskind of courses. There is still room to include more teaching and learning activities, for examplea fourth lab.
Page 1 (of 22)
DT2119 Speech and Speaker Recognition • Vårterminen 2018Giampiero Salvi
THE STUDENTS’ RESULTS
How well have the students succeeded on the course? If there are significant differ-ences compared to previous course offerings, what can be the reason?
There were 55 students applying and registering to the course (source Canvas). Of them, 43were active during the course and passed at the end. Below you can see the evolution in time forthe course (until 2016 the course code was DT2118). The increase in number of students seemto have stopped in 2018. This may be due to the course reaching its limit, also considering theever increasing course offering on Machine Learning topics at KTH.
2011
2012
2013
2014
2015
2016
2017
2018
10
20
30
40
50
Year
#stud
ents
# students in DT2118/9
registeredactivepassed
The milestones during the years are:
VT2015: the matlab exercises and computer lab based on the HTK software package weresubstituted for three python based labs on Feature Extraction, HMMs and ContinuousSpeech Recognition in order to improve the alignment between the learning outcomes, theteaching/learning activities and the assessment. However, Lab 3 was still based on theHTK software package.
VT2016: An alternative version of Lab 3 was introduced to include the Deep Learning modelsthat became state-of-the art in speech recognition. However, this lab was still based onpre-made software packages and offered limited learning opportunities for the students
VT2017: An introduction to signal processing was added to the course to help student lackingthat background (mainly computer science students).
VT2018: Lab 3 was completely redesigned in Python/TensorFlow to allow the students toimplement most of the methods, giving them a better opportunity to learn.
Page 2 (of 22)
DT2119 Speech and Speaker Recognition • Vårterminen 2018Giampiero Salvi
OVERALL IMPRESSION OF THE LEARNING ENVIRONMENT
What is your overall impression of the learning environment in the polar diagram?If there are significant differences between different groups of students, what can bethe reason?
Only 13 students responded to the survey. The overall impression is that the students highlyenjoyed the course and found it stimulating (Q1=5.6). They find the course less challengingcompared to previous years (Q4=4.8). This probably is due to the different distribution ofstudents that joined the course this year that are mainly from the machine learning master. Itis, therefore possible, to design more challenging tasks in the course. Similarly the backgroundof participants seems to be adequate (Q17=5.8) and the students thought that the assessmentwas fair and honest (Q16=5.6). The lowest score (4.4) was for Q13: “I understood what I wasexpected to learn in order to obtain a certain grade”, which indicates that the grading criteriacan still be improved. It is to note that 4.4 is not a low score.
The system identified several subgroups. The distinction between men and women is ratherinteresting and shows that women are much more positive than man in their feedback. Thequestions with the larges difference were Q1 (interest), Q6 (“The atmosphere on the coursewas open and inclusive”), Q7–Q8 (clear goals and organisation), Q9–Q11 (understanding ofsubject matter), Q12: “The course activities helped me to achieve the intended learning outcomesefficiently”, Q21 (collaboration) and Q22 (support). This is very encouraging. The distinctionbetween international master students and total, does not show interesting differences.
ANALYSIS OF THE LEARNING ENVIRONMENT
Can you identify some stronger or weaker areas of the learning environment in thepolar diagram - or in the response to each statement - respectively? Do they havean explanation?
From the numeric results of the survey, it seems that the course does a good job in most ofthe areas. Compared to previous years, the students who apply to this course seem to have amore fitting background resulting in lower work loads and lower perception of the course beingchallenging. This means that the assignments can be extended both in terms of coverage ofthe course topics and in terms of difficulty. Also, in the past years, when the course was runon KTH Course Web, one of the most appreciated feature was the availability of the teacherto answer question through the forum. Now, since Canvas has been introduced, the score onquestions related to feedback have decreased. I intend to investigate if this is due to the forumfunctionality in Canvas that is not as user friendly as in Course Web, or if I have been less presentdue to other engagements (the Machine Learning Master Admission process takes a significantamount of my time during the course). However, the open questions on Canvas and feedbackfrom the teacher were extremely positive, making me think that the lower numerical scores aresimply due to the different scoring traditions of international students.
Throughout the years I have been meaning to find the resources to organise meetings withthe students during the project to check progress and provide some form of supervision. PerhapsI can achieve this by recruiting some teaching assistants with a good background in the subject.
Page 3 (of 22)
DT2119 Speech and Speaker Recognition • Vårterminen 2018Giampiero Salvi
ANSWERS TO OPEN QUESTIONS
What emerges in the students’ answers to the open questions? Is there any goodadvice to future course participants that you want to pass on?
There is a majority of very positive answers to open questions which is very encouraging. Thelab material was in general considered very positively, and students were extremely happy aboutthe way the course was organised on Canvas. Many had difficulties using the computationalresources at PDC, and were glad that the alternative of Google Cloud Platform was madeavailable in the course. Some improvements are required for lab 3, and the students would liketo have TA sessions during the labs and project to get assistance.
PRIORITY COURSE DEVELOPMENT
What aspects of the course should primarily be developed? How could these aspectsbe developed in the short or long term?
The course has reached maturity. However there is still room for improvements:
• The instructions of lab 3 should be revised.
• Also, writing lab 3, I realised that the instructions of some aspects of lab 2 could be mademore general so to fit both labs and simplify the work for the students.
• I will consider adding a fourth lab on continuous speech recognition. This is an importantaspect in the course and it would be good if the students dedicated some extra time onit. The problem so far has been making sure that the students have enough time to carryout four labs without affecting the results of the project. However, the reported work loadindicates that there is room for adding more tasks in the course.
• The grading criteria need to be improved as some students reported they were not surewhat they were expected to do to obtain a certain grade.
• I will try to define more Canvas quizzes to cover those topics that are not covered by labsand project. This has also been requested by the students.
RESULT OF THE STUDENT SURVEY
In the next pages I report the result of the student survey.
Page 4 (of 22)
DT2119 Speech and Speaker Recognition • Vårterminen 2018Giampiero Salvi
DT2119 - 2018-06-06
Antal respondenter: 46
Antal svar: 13
Svarsfrekvens: 28,26 %
ESTIMA
TED W
OR
KLO
AD
Com
ments
Com
ments (I w
orked: 9-11 timmar/vecka)
I can't really remem
ber the hours/week but it felt like a good w
ork load/credits ratio.
Com
ments (I w
orked: 33-35 timmar/vecka)
There was no w
ay near enough time to read the reading m
aterial associated with the lectures. (B
ut maybe this w
as not the idea)
LEAR
NIN
G EXPER
IENC
E
The polar diagrams below
show the average response to the LE
Q
statements for different groups of respondents (only valid responses are
included). The scale that is used in the diagrams is defined by:
1 = No, I strongly disagree w
ith the statement
4 = I am neutral to the statem
ent7 = Y
es, I strongly agree with the statem
ent
Note! A
group has to include at least 3 respondents in order to appear ina diagram
.
KTH
Learning Experience Questionnaire v3.1.3
Meaningfulness - em
otional level
Stim
ulating tasks
1. I worked w
ith interesting issues (a)
Exploration and ow
n experience
2. I explored parts of the subject on my ow
n (a)3. I w
as able to learn by trying out my ow
n ideas (b)
Challenge
4. The course was challenging in a stim
ulating way (c)
Belonging
5. I felt togetherness with others on the course (d)
6. The atmosphere on the course w
as open and inclusive (d)
Com
prehensibility - cognitive level
Clear goals and organization
7. The intended learning outcomes helped m
e to understand what I w
as expected to achieve (e)8. I understood how
the course was organized and w
hat I was expected
to do (e)
Understanding of subject m
atter
9. I understood what the teachers w
ere talking about (f)10. I w
as able to learn from concrete exam
ples that I could relate to (g)11. U
nderstanding of key concepts had high priority (h)
Constructive alignm
ent
12. The course activities helped me to achieve the intended learning
outcomes efficiently (i)
13. I understood what I w
as expected to learn in order to obtain a certaingrade (i)
Feedback and security
14. I received regular feedback that helped me to see m
y progress (j)15. I could practice and receive feedback w
ithout being graded (j)16. The assessm
ent on the course was fair and honest (k)
Manageability - instrum
ental level
Sufficient background know
ledge
17. My background know
ledge was sufficient to follow
the course (f)
Time to reflect
18. I regularly spent time to reflect on w
hat I learned (l)
Variation and choices
19. I was able to learn in a w
ay that suited me (m
)20. I had opportunities to choose w
hat to do (m)
Collaboration
21. I was able to learn by collaborating and discussing w
ith others (n)
Support
22. I was able to get support if I needed it (c)
Learning factors from the literature that LEQ
intends to examine
We tend to learn m
ost effectively (in ways that m
ake a sustained, substantial, and positive influence on the w
ay we think, reflect, act or
feel) when:
a) We are trying to answ
er questions, solve problems or acquire skills
that we find interesting, intriguing or im
portant
b) We can speculate, try out ideas (intellectually or practically) and learn
from experience, even before w
e know much about the subject
c) We are able to do so in a challenging yet supportive environm
ent
d) We feel that w
e are part of a community and believe that other people
have faith in our ability to learn
e) We understand the m
eaning of the intended learning outcomes, how
the environm
ent is organized and what is expected of us
f) We have sufficient background know
ledge to manage the present
learning situation
g) We can learn inductively by m
oving from specific exam
ples and experiences to general principles, rather than the other w
ay around
h) We are challenged to develop a proper understanding of key
concepts and successively create a coherent whole of the content
i) We believe that the w
ork we are expected to do w
ill help us to reach the intended learning outcom
es
j) We can try, fail, and receive feedback in advance of and separate from
any summative judgm
ent of our efforts
k) We believe that our w
ork will be considered fairly and honestly
l) We have sufficient tim
e to learn and devote the time necessary to do
so
m) W
e believe that we are in control of our ow
n learning, not manipulated
n) We can w
ork collaboratively with other learners struggling w
ith the sam
e problems
Literature
Bain, K
. (2004). What the B
est College Teachers D
o, Chapter 5, pp.
98-134. Cam
bridge: Harvard U
niversity Press.
Biggs J. &
Tang, C. (2011). Teaching for Q
uality Learning at University,
Chapter 6, pp. 95-110. M
aidenhead: McG
raw Hill.
Elmgren, M
. & Henriksson, A
-S. (2014). A
cademic Teaching, C
hapter 3, pp. 57-72. Lund: S
tudentlitteratur.
Kem
ber, K. &
McN
aught, C. (2007). E
nhancing University Teaching:
Lessons from R
esearch into Aw
ard-Winning Teachers, C
hapter 5, pp. 31-40. A
bingdon: Routledge.
Ram
sden, P. (2003). Learning to Teach in H
igher Education, C
hapter 6, pp. 84-105. N
ew York: R
outledgeFalmer.
Com
ments
Com
ments
Com
ments (I am
: Svensk student i årskurs 1-3)
Thanks for an interesting course! :)
GEN
ERA
L QU
ESTION
S
What w
as the best aspect of the course?
What w
as the best aspect of the course? (I worked: 3-5 tim
mar/vecka)
The labsThe labs and the final project
What w
as the best aspect of the course? (I worked: 9-11 tim
mar/vecka)
Very relevant and w
ell written assignm
ents throughout the course.The project, allow
ing for us to do something w
e are interested in within the field.
Good labs. C
lear lab instructions on the first two labs.
What w
as the best aspect of the course? (I worked: 15-17 tim
mar/vecka)
The freedom to choose alm
ost any speech related subject for the final projectFöreläsningarna, att det var m
ycket arbete i grupp, att projektet var så fritt och redovisningsformatet m
ed posters.
What w
as the best aspect of the course? (I worked: 18-20 tim
mar/vecka)
the projectCom
ing into contact with speech preprocessing
What w
as the best aspect of the course? (I worked: 33-35 tim
mar/vecka)
That it covered a lot of topics
What w
ould you suggest to improve?
What w
ould you suggest to improve? (I w
orked: 3-5 timmar/vecka)
Maybe propose subjects for projects
It suppose to have course quiz
What w
ould you suggest to improve? (I w
orked: 9-11 timmar/vecka)
Maybe em
phasize a bit on speech characteristics when m
odels are presented. More guidance during the project
The lab instructions on the third lab.
What w
ould you suggest to improve? (I w
orked: 15-17 timmar/vecka)
Improve the assignm
ent descriptions. Som
e people including me lost a lot of tim
e trying to understand what the description text w
ants us to do and how
. There w
ere cases where the description w
as apparently not very carefully written and these things w
ere fixed only after someone
poses question on the discussions.
- Det hade varit trevligt att ha alla quizzar.
- Labb 3 tog väldigt mycket tid och inkräktade därm
ed på projektet. Mycket av det berodde på att m
ängden data var såpass stor så våra datorerhade problem
med att hantera den.
What w
ould you suggest to improve? (I w
orked: 18-20 timmar/vecka)
better defined assignments w
ithout bugs in the skeletons T
he signal processing lecture was not sufficient to understand the reasons behind the different m
anipulations of the speech signal.
Less focus on recreation of historic techniques, more critical analysis of them
(what are their draw
backs, and where do they com
e from), and
more contact w
ith modern techniques and how
they try to solve some of the draw
backs.
What w
ould you suggest to improve? (I w
orked: 33-35 timmar/vecka)
Add prerequisite courses
Add TA
help sessions for the labs and the project
What advice w
ould you like to give to future participants?
What advice w
ould you like to give to future participants? (I worked: 3-5 tim
mar/vecka)
pay more attention to the final project
What advice w
ould you like to give to future participants? (I worked: 9-11 tim
mar/vecka)
Try to start the project as early as possible.Learn about sound theory to better understand the labs
What advice w
ould you like to give to future participants? (I worked: 15-17 tim
mar/vecka)
Hitta en labb-/projektpartner du trivs m
ed för ni kommer att tillbringa m
ycket tid ihop.
What advice w
ould you like to give to future participants? (I worked: 18-20 tim
mar/vecka)
start the lab ASAP as you w
ill find a lot of bugsGet quite fam
iliar with the preprocessing pipelineIs there anything else you w
ould like to add?
Is there anything else you would like to add? (I w
orked: 3-5 timmar/vecka)
noIs there anything else you would like to add? (I w
orked: 9-11 timmar/vecka)
No
Is there anything else you would like to add? (I w
orked: 15-17 timmar/vecka)
Jag använde inte kurslitteraturen alls men hade viss nytta av Jurafsky &
Martins S
peech and language processing.
Is there anything else you would like to add? (I w
orked: 18-20 timmar/vecka)
please pay attention to the lab specsModern A
SR still seem
s like it's just an interface above a really bad UX design, and the course did not help change that view
. It may not have
been in the course's aims though. In any case, I do not feel I have any greater appreciation for the technology and its applications, it still seem
slike a terrible w
ay to interact with com
puters
SPECIFIC
QU
ESTION
S
Arbetsinsatsen var jäm
nfördelad under kurstiden
Arbetsinsatsen var jäm
nfördelad under kurstidenYes
No, there w
as too much w
ork in a short period of time for Lab2 and Lab3 (given that w
e were doing the project at the sam
e time). O
n the other hand there w
as too much tim
e given for Lab1. How
ever, I assume the labs 2 and 3 w
ould take far less time to im
plement if w
e had better instructions in the assignm
ent description.Yes that is som
ething I like about the course !Y
es, it was evenly distributed and felt fair
Yes, m
aybe except for the last projectYes
Yes, a part from
lab3 being to heavyyesyesDet m
esta av arbetet lades på labb 3 och projektet som båda var på slutet.
Laborationerna hjälpte mig förstå de teoretiska aspekterna
Laborationerna hjälpte mig förstå de teoretiska aspekterna
Significantly
Yes but I learned a lot of stuff during the actual presentation from
the TA
s. I believe all that stuff should be included in the description or at least hints should be given so that w
e do not squander our time.
Yes definitely but it w
as sometim
e hard to meet the deadlines w
ith my 2-3 other courses
Yes
Yes
Yes
to an extend. They were not alw
ays clear on what their aim
was, or w
hat particular part of the whole recognition process w
e were dealing w
ith in a concrete and focused w
ay. Lab 3 was particularly badly form
ed, with the task being unclear both in its nature and in its necessity, since w
e already w
ere doing the same thing w
ith HMMs. It w
as a good discussion on the pros and cons of each method at least
yesJa
Informationsflödet i C
anvas var tidsenlig och användbar
Informationsflödet i C
anvas var tidsenlig och användbarYes
Yes
Yes
To some extend yes
Very helpful
Yes
Yes
Yes, the teacher did a very good job w
ith canvas. He even added a deadline for us to apply for pdc accounts so w
e wouldn't forget
yesyesJa, C
anvas funkade överlag väldigt bra
Beräkningsresurserna var adekvat (PD
C/G
CP/A
WS)
Beräkningsresurserna var adekvat (P
DC/GCP/AWS)
PD
C w
as unusable in the end, AW
S w
as very helpful and easy to use. P
s : I used AW
S too m
uch and consumed m
ore than my free credit. T
he billing dashboard is not clear, and seems to take tim
e to refresh. We did not run into any issues but I w
as afraid that running certain stuff on GCP would m
ake us run out of credits. Yes but kind of hard to use som
etimes.
Yeah, I like that you added google clouds. They are so easy to use
They were a bit com
plicatedPDC was very hard to use and w
e spent many hours trying to get it to w
ork. Once it w
as working it w
as extremely hard to get a tim
e slot for running our jobs. The google cloud platform
which w
e used instead, was how
ever very convenient and user friendly! GCP was. W
e were unable to use P
DC after correctly setting everything up
GCP
Föreläsningen om PDC kändes inte alls anpassad efter oss och vad vi behövde veta och klargjorde inte särskilt m
ycket. Vi använde istället
Google C
loud och det fungerade utmärkt, lätt att kom
ma igång och kvoten räckte m
er än väl. Vi hade dock behövt m
er minne än förslaget i
instruktionerna, men det visste vi ju nte när vi skapade vår V
M och det gjorde att vi fick hoppa över vissa körningar.
RESPO
NSE D
ATA
The diagrams below
show the detailed response to the LE
Q statem
ents.The response scale is defined by:
-3 = No, I strongly disagree w
ith the statement
0 = I am neutral to the statem
ent+3 = Y
es, I strongly agree with the statem
ent
X = I decline to take a position on the statem
ent
Com
ments
Com
ments
Com
ments
Com
ments
Com
ments (M
y response was: +2)
Yes, but som
etimes frustrating, especially lab3 since this lab w
as pretty extensive, stressful and a bit unclear
Com
ments
Com
ments (M
y response was: +3)
Students w
here very helpful! And there w
as a very open and inclusive atmosphere in the class!
Com
ments
Com
ments
Com
ments
Com
ments
Com
ments (M
y response was: +1)
Most of the tim
e
Com
ments
Com
ments
Com
ments (M
y response was: +3)
Very w
ell structured powerpoints and lectures!
Com
ments
Com
ments
Com
ments (M
y response was: +1)
Lite oklart hur projektet kommer att betygsättas.
Com
ments
Com
ments (M
y response was: +2)
How
ever, I think it could be good with helpsessions w
ith TAs as w
e had in the DD2424
Com
ments
Com
ments
Com
ments (M
y response was: X
)No assesm
ent yet, still waiting for project grades
Com
ments
Com
ments (M
y response was: -2)
Without any background in m
achine learning, advanced machine and signal processing it w
as sometim
es very challenging to follow the course.
I would recom
mend adding them
as prerequisites or recommended prerequisites on the course page
Com
ments
Com
ments
Com
ments
Com
ments (M
y response was: -2)
During the project only
Com
ments
Com
ments