1

Automatic Neural Lyrics and Melody CompositionGurunath Reddy M, Yi Yu, Florian Harscoet, Simon Canales, Suhua Tang

Abstract—In this paper, we propose a technique to addressthe most challenging aspect of algorithmic songwriting process,which enables the human community to discover original lyrics,and melodies suitable for the generated lyrics. The proposedsongwriting system, Automatic Neural Lyrics and Melody Com-position (AutoNLMC) is an attempt to make the whole process ofsongwriting automatic using artificial neural networks. Our lyricto vector (lyric2vec) model trained on a large set of lyric-melodypairs dataset parsed at syllable, word and sentence levels are largescale embedding models enable us to train data driven modelsuch as recurrent neural networks for popular English songs.AutoNLMC is a encoder-decoder sequential recurrent neuralnetwork model consisting of a lyric generator, a lyric encoderand melody decoder trained end-to-end. AutoNLMC is designedto generate both lyrics and corresponding melody automaticallyfor an amateur or a person without music knowledge. It can alsotake lyrics from professional lyric writer to generate matchingmelodies. The qualitative and quantitative evaluation measuresrevealed that the proposed method is indeed capable of generat-ing original lyrics and corresponding melody for composing newsongs.

Impact Statement—Recently, we can see a surge in growingresearch interest in automatically generating music after thesuccessful application of deep neural networks for sequentialmodeling. Most of the works focus on either generating melodiesor lyrics. However, to the best of our knowledge, there are noworks which focus on automating the song creation process whichhas potential application in cinema and gaming industries, andeducational pedagogy. The proposed AutoNLMC can be used byboth amateur and professionals to create songs with both lyricsand melodies automatically. Further, generated lyric melody pairscan be used to synthesize songs using synthesizers which canbe readily used in gaming and cinema applications. Also, usingAutoNLMC one can discover new melodies for already existinglyrics without melodies which enables a faster song creationprocess.

Index Terms—Lyrics, Melody, Notes, Singing voice

I. INTRODUCTION

MUSIC composition is a human creative process thatrequires a wide range of strong musical knowledge and

expertise to create soothing music which continues to remainin our heart forever. Given the vast majority of music loversand the limited availability of professional music composers,there is a strong need for machines to assist human creativity.Recent advancement in the software based music creationtechnology helped the professional and amateur music creators

Gurunath Reddy M is with Indian Institute of Technology, Kharagpur, India(email: mgurunathreddy@sit.iitkgp.ernet.in)

Yi Yu is with National Institute of Informatics, Japan, (email:yiyu@nii.ac.jp)

Florian Harscoet is with ISIMA, France, (email: flo-rian.harscoet@etu.uca.fr)

Suhua Tang is with The University of Electro-Communications, Japan,(email: shtang@uec.ac.jp)

Simon Canales is with EPFL, Switzerland, (email: simon.canales@epfl.ch)This paragraph will include the Associate Editor who handled your paper.

to produce music with great joy and ease of production inmasses to be consumed by the music consumers with personalcomputers and hand-held devices. Though there exists a plentyof machine assistance to create high quality music withrelative ease of production, the process of songwriting thatis automatically generating lyrics, composing melody corre-sponding to the generated lyrics and synthesizing singing voicecorresponding to the generated melody and lyrics remainedas mutually exclusive tasks. Till date, the construction ofnovel/original songs is limited to the individuals who possessthe following skills: the ability to create lyrics, composemelody and combine lyrics and melody to create a rational,relevant and soothing final complete songs [1].

In literature, we can find considerable amount of researchwork published on automatic music generation (without condi-tioning on lyrics). Early machine assisted music generation ismostly based on music theory and expert domain knowledge tocreate novel works. With the advent of data driven approachesand exploded public music collections in the internet, datadriven methods such as Hidden Markov models [2], graphicmodels [3] and deep learning models [4], [5], [6], [7], [8],[9] showed a potential for music creation. Though thereexists substantial amount of research on unconditional musicgeneration, there exists considerably less amount of work doneso far on generating melody from lyrics given in the form oftext, which we call conditional melody/song generation fromlyrics. The primary reasons for substantially less research onconditional melody generation can be attributed to i) the non-availability of the direct source for lyrics-melody pair datasetto train the data driven models, ii) a lyrics composition canhave multiple melodic representations, which makes it hardto learn the correlation between the lyrics and melodies, andiii) it is hard to evaluate the generated melodies by objectivemeasures.

This paper focuses on the most challenging aspect of algo-rithmic songwriting process which enables the human com-munity to discover original lyrics, and melodies suitable forthe generated lyrics. To the best our knowledge, the proposedAutoNLMC is the first attempt to make the whole processof songwriting automatic using artificial neural networks. Wealso present the lyrics to vector model which is trained ona large dataset of popular English songs to obtain the denserepresentation of lyrics at syllables, words and sentence levels.The proposed AutoNLMC is an attention based encoder-decoder sequential recurrent neural network model consistsof a lyric generator, lyric encoder and melody decoderstrained end-to-end. We train several encoder-decoder modelson various dense representations of the lyric tokens to learnthe correlation between lyrics and corresponding melodies.Further, we prove the importance of dense representationof lyrics by various qualitative and quantitative measures.

arX

iv:2

011.

0638

0v1

[cs

.SD

] 1

2 N

ov 2

020

2

Fig. 1: Example music score

AutoNLMC is designed in such a way that it can generateboth lyrics and corresponding melodies automatically for anamateur or a person without music knowledge by acceptinga small piece of initial seed lyrics as input. It can also takelyrics from professional lyrics writer to generate the matchingmeaningful melodies.

II. RELATED WORK

Here, we briefly present closely related work on lyrics-conditional music generation frameworks developed by re-searchers in past years. Orpheus [10] is a dynamic program-ming based melody composition algorithm for Japanese lyrics.Orpheus is designed as an optimal melody search problem forthe given lyrics under the prosodic constraints of the Japaneselyrics. The authors design two individual models such asrhythm and probabilistic pitch inference model to generatemelodies from the given lyrics. A Finish song generatingsystem called Sucus-Apparatusf is developed by Toivanen etal. [11]. Sucus-Apparatusf is designed to randomly chooserhythm from the rhythm patterns actually found in the Finishart songs. Further, a second order Markov model is designedto generate the chord progression for the given lyrics. Thepitches are generated from the joint probabilistic distributionof previously generated notes and the chords. In [12], theauthors propose a system for automatically generating melodicaccompaniments from a given lyrical text. The system isdesigned to generate the pitches modeled as n-gram modelsfrom the melodies of songs with similar style. Further, therhythm for the melodic accompaniment is derived from thecadence information present in the text. The proposed methodgenerated hundreds of melodies by giving random optionsdriven by set of rules, followed by selecting among the gener-ated options with an objective measure that incorporates expertmusic knowledge. Both melody and rhythm are generated bythe same process for a given lyrics. Rhythm suggestion fromlyrics by using set of rules is studied by Nichols [13] whileOliveira [14] proposed the inverse process of lyrics generationfrom the rhythm of melody. ALYSIA [1] is the first fully datadriven model based on random forests to generate melody fromthe lyrics. ALYSIA is trained on a large set of features manu-ally extracted from Music-XML files. ALYSIA is designed tosuggest multiple melodies as output for a given lyrical piecethus giving user the ability to choose more pleasing melodyfor the given lyrics. ALYSIA consists of two independentmelody prediction models to predict duration of the noteand scale of the note independently. An encoder-decoderbased RNN sequential model for lyrics-conditional melodygeneration for Chinese pop songs is presented in [15]. Thesequential model called Songwriter consists of two encoders

and one hierarchical decoder. The encoders are designed toencode the lyric syllables and context melody of the priorgenerated melody. The hierarchical decoder is designed todecode the note attributes such as pitch, duration and syllable-note alignment labels since most of the syllables in Chinesesongs had more than one note.

III. MUSIC BACKGROUND

Melody can be defined as a sequence of meaningful musicalnotes and musical rests. Let N be the number of notes of agiven melody. Each note of the melody has two attributes:

1) its pitch – called note or tone in common musicallanguage – which is represented by a letter followedby some number (e.g. C4) (and eventually a [ (flat) or] (sharp) symbol (e.g. F]5));

2) its duration, which depends on the note type.The common unit used for the duration is the beat. In

most modern Western music, four beats form a measure.A note which lasts for one beat (or a quarter measure) iscalled a quarter-note. Accordingly other notes can be created:sixteenth, eighth, etc. Then, between successive notes there iseither a rest or no rest. Again, the rest duration unit is thebeat.

Therefore, a melody of length N can be defined as Z ={z1, z2, . . . , zN}, where each zi’s are {note pitch, note dura-tion, rest duration} triplets (which is the representation usedfor this work). The i-th rest duration value is the duration ofthe rest before the i-th note. A null rest duration means norest before the note.

We can define lyrics as sequence of natural language tokenswhich can eventually make up song consisting of chorusesand verses. The lyrics token representation adopted in ourframework is a syllable. A syllable is a part of word or asingle-syllable word. Each syllable is associated with melodyattributes: pitch, duration and rest to from the lyric-melodypairs.

We can represent lyrics-melody pair graphically via musicscores. Each syllable is associated with the corresponding note(see Figure 1). The shape of the notes gives their duration, andtheir vertical position their pitch. The duration of a rest is givenby its shape. Most of the time, melodies have perfect scaleconsistency, meaning that the pitches of the notes composingthe melody all belong to the same scale. A scale is a subsetof pitches which have properties such that they sound goodwhen consecutively played.

IV. PROBABILISTIC MODELING OF AUTONLMCThe proposed AutoNLMC is designed to first generate lyrics

given a piece of seed lyrics in the form of text and then

3

composes the melody for the generated lyrics sequentially onesentence at a time. We can model the AutoNLMC as a proba-bilistic model with two conditionally dependent components:a lyrics generator and a melody composer sequential model.Lyrics generator is modeled as a conditional distribution topredict the next lyrical token given the previous tokens ofthe lyric sequence modeled as p(st|s1, s2, ..., st−1). Here, thetokens are the sequence of syllables of the lyrics given byS = {s1, s2, ..., sT }. We can learn a probability distributionsuch as

p(S) =

T∏t=1

p(st|s1, s2, ..., st−1) (1)

the learned probability distribution is sampled one token atevery time to generate the full sequence lyrics.

We can model the Lyrics generator using recurrent neuralnetwork (RNN) to implicitly learn the conditional probabilitydistribution of the lyrics generator. RNN can be trained to learnthe probability distribution of a sequence to predict the nexttoken of the sequence at time t. RNN is essentially an artificialneural network consisting of a hidden state h, an output y,operates on a sequence of tokens to update the hidden statesat each time step t given by

ht = g(ht−1, st) (2)

where g is a non-linear function mapping from input and pre-vious state to the present state. The non-linear function g canbe a simple sigmoid function [16] or it can be a more complexgated unit such as long short-term memory (LSTM) [17].Melody composer is modeled as a conditional distributionmodel over melody sequence on a lyrics sequence given byp(m1,m2, ..,mT |s1, s2, ..., sT ) where M = {m1,m2, ..,mT }is a melody sequence. The melody composer is modeled asa sequential encoder-decoder model [18]. The encoder is anRNN network which takes the lyrics tokens generated bylyrics generator model Eq. 1 at each time step sequentially toproduce the encoder hidden states at time t defined in Eq. 3.The encoder also generates a variable context vector Ci whichencodes the parts of the lyrics for which most attention to bepaid during melody decoding. The decoder is another RNNtrained to predict the melody token at time t given the hiddenstate of the decoder RNN. The hidden state of the decoder iscomputed recursively by

ht = g(ht−1,mt−1, Ci) (3)

The next token distribution of the decoder can be similarlydefined as

p(mt|m1,m2, ....,mt−1, Ci) = f(ht,mt−1, Ci) (4)

The melody composer can be jointly trained to maximize thelikelihood

maxθ

1

N

N∑i=1

log pθ(mi|si) (5)

where θ is the model parameters and the pair mi, si is theinput and output to the melody composer.

Fig. 2: Skip-gram model to extract lyrics embedding.

V. LYRIC TO VECTOR (LYRIC2VEC)

We train continuous skip-gram models to obtain the denserepresentation of the input lyrics text. This enables us to learnhigh-quality real valued vector representation of text tokensin an efficient way from large corpus of text data [19]. Thevector representations computed from the skip-gram modelexplicitly encode many linguistic regularities and patternswhich can be represented as linear translations. We exploit thelinear translation property of embeddings to obtain the lyricsembedding representation to train deep generative models.

The objective of the skip-gram model is to predict thesurrounding context tokens given a token at position t in apiece of input text. We can define the objective function moreformally given a sequence of tokens such as s1, s2, ..., sT . Wewant the model to maximize the log probability [20] of thesurrounding tokens given by

1

T

T∑t=1

∑−c≤i≤c,i6=0

log p(st+i|st) (6)

where c is the length of the context of tth token st.The vanilla skip-gram model formulates the objective func-

tion as softmax function given by

p(sO|sI) =exp(vTs0vsI )∑Tt=1 exp(vTs0vsI )

(7)

where vsI and vs0 are input and output vector representationsof token s, and T is the number of unique tokens in thecorpus. This formulation is impractical given the large numberof unique tokens/vocabulary of the dataset. Hence, we usenegative sampling [19] defined by

4

Pitch decoder

LSTMLSTMLSTM ……

Attention 𝑐

ℎ ……ℎ ℎ

[𝑐 , ℎ ]

Duration decoder

LSTMLSTMLSTM ……

ℎ ……ℎ ℎ

[𝑐 , ℎ ]

Rest decoder

LSTMLSTMLSTM ……

ℎ ……ℎ ℎ

[𝑐 , ℎ ]

Lyrics predictor Melody generator

LSTMLSTMLSTM ……

ℎ ……ℎ ℎ

LSTMLSTMLSTM ……

…………

LSTM output

LSTM output

LSTM output

Dense layer

Dense layer

Dense layer

SoftmaxSoftmaxSoftmax

Lyrics encoder

“I love machine learning. It is my passion.”

Embedding Embedding Embedding

Fig. 3: Figure shows the proposed Neural lyrics and melody composer (AutoNLMC) model. AutoNLMC consists of a lyricsprediction module, encoder and melody decoders. During inference, lyrics are predicted from the lyrics prediction LSTMmodule shown in the rectangular red box in the lower half part of the lyrics predictor with the help of softmax layers. Whilemelody is predicted by bypassing the softmax layers to predict pitch, duration and rest with individual decoders as shown infigure.

log σ(vTsOvsI ) +

j=k∑j=1

Esj∼pn(s)[log σ(vTsjvsI )] (8)

Here the objective of the negative sampling function is todiscriminate the actual token sO from the k negative tokensdrawn from the smoothed noise distribution [21] pn(s) givenby

pn(s) =f(s)α∑c f(s)α

(9)

where f(s) is token frequency s, and α is real valued dis-tribution smoothing parameter. We train skip-gram models toobtain the embedding vectors of lyrics at syllable and wordlevel. We divide the lyrics of each song into sentences, eachsentence into words and each word into syllables. We treatsyllables S = {s1, s2, ..., sn} as tokens to train the syllablelevel embedding model and words W = {w1, w2, w3, ..., wn}as tokens for training word level embedding model. We trainskip-gram model as a logistic regression with stochastic gradi-ent decent as optimizer, learning rate with an initial value 0.03is gradually decayed every epoch until 0.0007. We use c = 7tokens context window and negative sampling distributionparameter α is set to 0.75. We trained the models to obtainthe syllable and world level embedding vectors of dimensionsv = 50. The skip-gram model to extract the lyric embeddingsis shown in Fig. 2. The input one-hot vector representing thesyllable or word is compressed to a low dimensional denserepresentation by linear units to predict the context tokens. Thefollowing embedding representations are explored individuallyto train the lyrics prediction and melody composition model: 1)

syllable embeddings (SE) where each syllable si is encodedwith v dimensional vector from syllable embedding model.2) syllable and corresponding word embedding concatenation(SWC) where a syllable si is concatenated with correspondingword embedding wi of v dimension. 3) addition of syllable andword vector (ASW) where syllable vector is added elementwise to the word vector. 4) concatenated syllable, word andsyllable projected word vector (CSWP). In CSWP, we projectthe syllable embedding vector si onto the corresponding wordvector wi to give the wordness to syllable si by

projwisi =si.wi|wi|2

wi (10)

finally, we concatenate si, wi and projwisi to form anembedding vector for syllable si.

VI. LYRICS AND MELODY COMPOSER

The proposed lyrics and melody composer is a sequentialencoder-decoder model trained end-to-end to predict lyricsand compose the melody for the predicted lyrics. Initially,AutoNLMC takes a piece of seed lyrics as input and startsgenerating the lyrics one sentence at a time with syllables astokens. For each generated lyrics, the melody is composeduntil the last sentence. At the end, all pairs of generated lyricsand melody are stitched together to form the complete song.

A. Neural Lyrics Generator and Melody Composer

The proposed neural lyrics and melody composer (Au-toNLMC) is shown in Fig. 3. The AutoNLMC is a sequentialRNN encoder-decoder network consists of a lyric generator,encoder and melody decoder. The RNN sequential lyrics

5

generator and encoder are clubbed together during training asshown in Fig. 3. The melody decoder is designed to decodethe melody attributes in the form of MIDI. Specifically, thelyric generator takes the sequence of lyric tokens S as inputwhere each token is embedded into a fixed size dense vectorby the skip-gram model discussed in section V to generatethe next lyrics token. Each generated lyrics token is encodedby a RNN encoder to a fixed dimensional hidden vector ht ateach time step. The melody decoder uses the encoded hiddenvector ht and the dynamic context vector Ct to generate themelody decoder hidden state at each time step to generate themelody sequence. The decoder consists of three independentMIDI decoders for each attribute receives the lyric contextvector Ct and decoder state to generate the pitch, duration andrest MIDI attributes for each syllable. We design independentdecoder for each attribute to reduce the number of classesand hence to increase the accuracy of the predicted output,which is partially motivated by the separate pitch and rhythmmodels in [1]. Each unit in the RNN is modeled with gatedLSTM [16]. The hidden state ht of the lyrics generator iscomputed by

ht = (1− zt) ◦ h(t−1) + zt ◦ ht (11)

ht = tanh(Wst − 1 + U [ri ◦ ht−1]) (12)

zt = σ(Wzst + Uzht−1) (13)

rt = σ(Wrst + Urht−1) (14)

where zt and rt are the reset and update gates of the LSTMunit. σ is the non-linear sigmoid function. The reset gatemakes the hidden state to forgot the past sequence informationirrelevant to predict the future sequence. While, the updategate controls the information flow from previous hiddenstates to current state, and further acts as memory cell toremember long term sequential dependencies which helps tolearn the semantic information of sequences. The variablesW,Wz,Wr, U, Uz, Ur are the weights of the LSTM unit. Theweights are learned automatically during training. The lyricsgenerator is trained to maximize the log conditional probability

maxθ

1

N

N∑i=1

log pθ(si|s<i) (15)

where si denotes the target lyrics token of the ith exampleand s<i is shorthand notation for {s1, s2, ..., st−1}.

The melody decoder decodes the melody attributes mi ={pi, di, ri} i.e., pitch, duration, and rest independently bymaximizing the log conditional probability

maxθ

1

N

N∑i=1

log pθ(mi|si) (16)

The decoder states ht where ht = {hpt, hdt, hrt} are the statesof individual decoders given the encoder hidden state ht iscomputed by

ht = (1− zt) ◦ ht−1 + zt ◦ ¯ht (17)

¯ht = tanh(Wmt−1 + U [ri ◦ ht−1] + Cct) (18)

zt = σ(Wzmt−1 + Uzht−1 + Czct) (19)

rt = σ(Wrmt−1 + Urht−1 + Crct) (20)

The free variables W,Wz,Wr, U, Uz, Ur, C, Cz, Cr are alllearnable parameters.

The attention vector ct to align the input lyrics with themelody by computing alignment model [22]

ct =

T∑j=1

αtjht (21)

αtj =exp(etj)∑Tk=1 exp(etk)

(22)

etj = V Ta tanh(Waht−1 + Uaht) (23)

the parameters V,Wa, Ua are the learnable weights.The neural lyric and melody composition model is trained

end-to-end to minimize the total loss function defined by

L = − 1

N

N∑i=1

|M(i)|∑j=1

log p(M (i)|θ, S(i),m(i)<k, c

(i))

− 1

N

N∑i=1

|S(i)|∑k=1

log p(S(i)k|S(i)<k)

(24)

L = − 1

N

N∑i=1

|M(i)|∑k=1

log p(p(i)k, d(i)k, r(i)k|θ, S(i),m(i)<k, c

(i))

− 1

N

N∑i=1

|S(i)|∑k=1

log p(S(i)k|S(i)<k)

(25)

L = − 1

N

N∑i=1

( |M(i)|∑j=1

log p(p(i)k|θ, S(i), p(i)<k, c

(i))

+

|M(i)|∑j=1

log p(d(i)k|θ, S(i), d(i)<k, c

(i))

+

|M(i)|∑j=1

log p(r(i)k|θ, S(i), r(i)<k, c

(i))

)

− 1

N

N∑i=1

|S(i)|∑k=1

log p(S(i)k|S(i)<k)

(26)

where N is the set of (S(i),M (i)) lyric melody pairs, S(i) =

{s(i)1, s(i)2, ...., s|S(i)|}, M (i) = {m(i)1,m(i)2, ....,m|M(i)|}

and m(i)1 = {p(i)1, d(i)1, r(i)1}.

6

The number of LSTM units used for all the models is 128.The initial state of lyrics generator model is initialized withzero vector, where as the encoder initial state is initializedwith the last hidden state of lyrics generator model. The pitchdecoder, duration decoder and rest decoder of the melodyprediction model is initialized with the last hidden state ofencoder model. The loss function is minimized by Adamoptimizer with initial learning rate of 0.0001 and linearlydecayed after every 10 epochs. The model is trained to min-imize the cross entropy loss function with 32 batch size. Allweight matrices are initialized from zero mean, 0.02 varianceGaussian distribution.

B. Lyrics and Melody Inference

Lyrics inference is a process of generating new lyrics bycombining the previously generated lyrics from the condi-tional probability distribution to generate the whole lyrics.Specifically, we obtain the next lyric token by multinomialdistribution output by the softmax non-linear activation func-tion as shown in the lower part of lyrics predictor of Fig. 3.At each time step, we generate a lyrics token and feedbackto the RNN unit at next time step to generate new token.We use the following strategy to generate the lyrics from thelyrics prediction model and evaluate the effectiveness of eachstrategies in the evaluation section. i) Greedy search: at eachtime step, we pick the most probable token from the softmaxprobability distribution given by

arg maxj

p(s(t,j) = 1|s(t−1), ..., s1) =exp(wjht)∑K

j′=1 exp(wj′ht)(27)

where j = 1, ...,K are the unique lyric tokens, wj arethe rows of the LSTM weight matrix W . ii) Temperaturesampling: The probability of each lyric token is transformedto a freezing function fτ [23] with a controllable parameter τbefore sampling most likely lyric token given by

pj = fτ (pj) =p

1τj∑K

j=1 p1τj

(28)

the value of τ ∈ [0, 1] makes the model to predict more robustand diverse lyrics.

The melody inference to predict the most probable melodysequence of the generated lyrics is done at two modes: i)composing melody from the lyrics generated from the lyricsgenerator and ii) composing melody from the original lyricscreated by the human lyrics writer. In the former case, weobtain the lyrics from the lyrics generator. For each lyric token,we obtain the dense representation from the skip-gram model.The attention vector at each time step which aligns the inputlyrics to the output pitch is computed from the encoder hiddenvectors. The initial state of the pitch decoder is initializedwith the last state of the encoder. At each time step, we feedthe variable attention vector and the decoder hidden state tothe softmax probability distribution function to predict thepitch corresponding to the lyric token. To generate the fulllength sequence at each time step, we feed the previously

generated pitch tokens to generate the next tokens. We iteratethis process until we generate all pitch tokens correspondsto lyrics. Similarly, we follow the same steps to decode theduration and rest attributes of all lyric tokens. Finally, weclub the predicted melody triplets i.e., pitch, duration and restand assign to the corresponding lyric syllable to form thelyrics melody pairs. In case of composing melody from theoriginal lyrics created by the human lyrics writer, we feed themelody inference model with the lyrics created by the humanwriter (here, lyrics prediction model is turned off.), where onesentence at a time is generated to match the correspondingmelody form the model.

0 20 40 60 80 100 120Pitch

0

50000

100000

150000

200000

250000

300000

350000

Cou

nt

Fig. 4: Pitch distribution.

−30 −20 −10 0 10 20 30Pitch interval

0

200000

400000

600000

800000

Cou

nt

Fig. 5: Pitch interval distribution.

0 10 20 30 40 50 60 70 80Pitch range

0

200

400

600

800

1000

1200

1400

Cou

nt

Fig. 6: Pitch range distribution.

7

0 5 10 15 20 25 30Duration

0

200000

400000

600000

800000

1000000

1200000

1400000C

ount

Fig. 7: Histogram distribution of duration.

0 5 10 15 20 25 30Rest

0

250000

500000

750000

1000000

1250000

1500000

1750000

2000000

Cou

nt

Fig. 8: Histogram distribution of rest.

VII. DATASET DESCRIPTION

The dataset used in our experiments initially created in [9],which come from two different sources: a) the ”LMD-fulldataset” of the Lakh MIDI Dataset v0.1 1 and a datasetfound on a reddit thread called ”the largest midi collectionon the internet”2. The LMD-full dataset contains a total of176,581 different MIDI files but, after taking only MIDI fileswith sufficient English lyrics, only 7,998 files could be used.The largest midi collection on the internet dataset contains130,000 different MIDI files but only 4,025 with enoughEnglish lyrics. These two datasets were merged together toget a total of 12,023 MIDI files. For our experiments, weparsed the dataset into three different formats: a) The syllablelevel: This format is the lowest level that pair together everynotes and the corresponding syllable and its attributes. b) Theword level: This format regroups every notes of a word andgives the attributes of every syllables that makes the word. c)The Sentence level: Similarly, this format put together everynotes that forms a syllable (or in most case, a lyric line) andits corresponding attributes. Each syllable is represented withdiscrete attributes: a) The Pitch of the note: In music, the pitchis what decide of how the note should be played. We use Midinote number as an unit of pitch, it can take any integer valuebetween 0 and 127. b) The duration of the note: The durationof the note in number of staves. It can be a quarter note, a

1https://colinraffel.com/projects/lmd/2https://https://www.reddit.com/r/datasets/comments/3akhxy/the largest

midi collection on the internet/

half note, a whole note or more. The exhaustive set of valuesit can take in our parsing is [0.125, 0.25, 0.5, 0.75, 1, 1.5, 2,4, 8, 16, 32]. and c) The duration of the rest before the note:This value can take the same numerical values as the durationbut it can also be null (or zero).

The algorithm to parse the MIDI files to extract the lyricsyllables and the corresponding melody attributes is given inAlgorithm 1. For the word and sentence level, the algorithmis extremely similar, with just the part about recognizing whatis a word and what is a sentence that is added.

open a midi file;i← 0;TrackNumber ← 0;while note’s timestamp and first lyric’s timestamp notalmost equal do// The first 5 notes of every

tracks are checked until thereis an equality

if i = 5 theni← 0;TrackNumber ← TrackNumber + 1;

elsei← i+ 1;

endend // Here the TrackNumber is foundGo on i-th first note of the found Track;ListNote← [ ];// The iteration and parsing of the

song begins herewhile The song and the lyrics has not reached theirend do

if The next note and lyric’s timestamp match thenListNote← ListNote+ CurrentNote;Iterate to the next note and next lyric;

elseif The note is a lot before the lyrics’s start then

Iterate to the next note;else

Iterate to the next lyric;end

endendclose the file;

Algorithm 1: Syllable level parsing

The pitch distribution of the dataset is shown in Fig. 4.From distribution plot in Fig. 4, we can observe that most ofour songs have pitch range between 60 and 80 midi values.With mean pitch value of approximately 75. The histogramdistribution of the duration of all our notes is shown in Fig. 7.From Fig 7, we can observe that most of our notes are quarternotes followed by half notes and other notes. Similarly, we canobserve the distribution of the rest attribute of our dataset inFig. 8. From the rest distribution in Fig. 8, we can observethat almost 90% of our songs have zero rest between thenotes. The only significant rest attribute between the notes is

8

45 50 55 60 65 70 75

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(a) Greedy search

45 50 55 60 65 70 75

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(b) τ = 0.5

45 50 55 60 65 70 75

Pitch

0

100

200

300

400

500

Cou

nt

(c) τ = 0.6

45 50 55 60 65 70 75

Pitch

0

100

200

300

400

Cou

nt

(d) τ = 0.7

45 50 55 60 65 70 75

Pitch

0

100

200

300

400

Cou

nt

(e) τ = 0.8

45 50 55 60 65 70 75

Pitch

0

100

200

300

400

Cou

nt

(f) τ = 0.9

45 50 55 60 65 70 75

Pitch

0

100

200

300

400

Cou

nt

(g) τ = 1.0

Fig. 9: Pitch histogram distribution of the generated lyrics predicted from melody decoder for greedy and temperature samplingmethods with lyrics encoded only with syllable embeddings (SE).

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

Cou

nt

(a) Greedy search

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

700

Cou

nt

(b) τ = 0.5

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

700

Cou

nt

(c) τ = 0.6

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

Cou

nt

(d) τ = 0.7

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

Cou

nt

(e) τ = 0.8

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

Cou

nt

(f) τ = 0.9

60 62 64 66 68

Pitch

0

100

200

300

400

500

600

Cou

nt

(g) τ = 1.0

Fig. 10: Pitch histogram distribution of the generated lyrics predicted from melody decoder for greedy and temperature samplingmethods with lyrics encoded by concatenating syllable and word embeddings (SWC).

9

40 45 50 55 60 65 70 75 80

Pitch

0

50

100

150

200

250

300

Cou

nt

(a) Greedy search

40 50 60 70 80

Pitch

0

50

100

150

200

250

Cou

nt

(b) τ = 0.5

40 50 60 70 80

Pitch

0

50

100

150

200

250

Cou

nt

(c) τ = 0.6

40 50 60 70 80

Pitch

0

50

100

150

200

Cou

nt

(d) τ = 0.7

40 50 60 70 80

Pitch

0

50

100

150

200

250

Cou

nt

(e) τ = 0.8

45 50 55 60 65 70 75 80

Pitch

0

25

50

75

100

125

150

175

200

Cou

nt

(f) τ = 0.9

50 60 70 80

Pitch

0

50

100

150

200

250

Cou

nt

(g) τ = 1.0

Fig. 11: Pitch histogram distribution of the generated lyrics predicted from melody decoder for greedy and temperature samplingmethods with lyrics encoded by adding syllable and word embeddings (ASW).

45 50 55 60 65 70 75 80 85

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(a) Greedy search

40 50 60 70 80

Pitch

0

50

100

150

200

250

300

350

Cou

nt

(b) τ = 0.5

45 50 55 60 65 70 75 80

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(c) τ = 0.6

45 50 55 60 65 70 75 80

Pitch

0

50

100

150

200

250

300

350

Cou

nt

(d) τ = 0.7

45 50 55 60 65 70 75 80 85

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(e) τ = 0.8

40 50 60 70 80

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(f) τ = 0.9

50 55 60 65 70 75 80

Pitch

0

50

100

150

200

250

300

350

400

Cou

nt

(g) τ = 1.0

Fig. 12: Pitch histogram distribution of the generated lyrics predicted from melody decoder for greedy and temperature samplingmethods with lyrics encoded by concatenating syllable, word and syllable projected word vector embedding (CSWP).

10

57.5 60.0 62.5 65.0 67.5 70.0 72.5

Pitch

0

100

200

300

400

500

600

Cou

nt

(a) Syllable embeddings (SE)

60 62 64 66 68 70

Pitch

0

100

200

300

400

500

600

700

Cou

nt

(b) syllable and word embeddingconcatenation (SWC)

50 60 70 80

Pitch

0

50

100

150

200

250

300

Cou

nt

(c) addition of syllable and wordembeddings (ASW)

40 50 60 70 80

Pitch

0

50

100

150

200

250

300

Cou

nt

(d) concatenated syllable, wordand syllable projected word vector(CSWP)

Fig. 13: Pitch histogram distribution of melody generated from the lyrics from test dataset for various lyrics embeddingrepresentations.

quarter, half and full duration rests. We also plot the histogramdistribution of the pitch interval in the dataset which is showin Fig. 5. From Fig. 5, we can observe that the pitch intervaldistribution can be approximated with a Gaussian with zeromean and standard deviation of approximately 5. i.e., mostof our consecutive note sequences have smooth pitch changesand they span form 0 to 5 midi pitch values. This also signifiesthat we have a sufficient data which can be modeled to fit thedistribution. The pitch range distribution of all songs in thedataset is shown Fig. 6. We can observe that an average, wehave approximately 15 tones gap between the minimum andthe maximum note value in each song with the variation of±5 tones.

VIII. EXPERIMENTS

We have experimented with greedy search and temperaturesampling as discussed in subsection VI-B methods to generatethe lyrics from the lyrics prediction model. We generate 10 fulllength songs by constraining the number of syllables generatedin each song to be 100 syllable length. We use ten variablelength seed lyrics i.e., lyrics to start with generating full songlyrics form the test dataset. Few examples of seed lyrics usedto generate the full length song lyrics and the generated lyricsare shown in Table I. We evaluate the quality of the generatedlyrics qualitatively by comparing the distributions of themelody composed from the generated lyrics and the datasetmelody distributions. That is, for all the generated lyrics, wecompose the melody by the melody composition model andcompare the individual melody attribute distributions: pitch,duration and rest with the dataset distributions described insection VII. Also, as discussed in section V, we representthe lyrics tokens in various encoding forms and train theproposed lyrics generator and melody decoder model onefor each representation. The histogram distribution of thepitch decoded from the generated lyrics for greedy searchand various temperature values τ of temperature samplingis shown in Figs. 9, 10, 11, 12 for lyrics encoded withsyllable embedding (SE), addition of syllable and word vector(ASW), syllable and word embedding concatenation (SWC)and concatenated syllable, word and syllable projected wordvector (CSWP) respectively. From Fig. 9, we can observe thatthe model trained with syllable embeddings alone shows quite

interesting distribution similar to the dataset. But a carefulobservation reveals that the predicted pitch is biased towardsthe lower pitch range. This signifies that the syllables alonecannot capture the whole dataset distribution. The modeltrained by concatenating syllable and word embeddings withan anticipation to improve the distribution range does notsucceeded in capturing underlying distribution shown inFig. 10. Instead, it confines the distribution to a small rangethus the model helps in preventing accidental abrupt changein pitch between notes. Though this type of model is quitestable in generating stable lyrics with no sudden jumps inpitch values between notes, it will be less interesting becausethe generated songs will have very low pitch span and lessnumber of unique notes results in mostly monotonous melodyfor any given lyrics. We found that the model trained with thesum of syllable and word embedding vectors (ASW) showsinteresting results with dense distribution round the meanpitch value of the dataset distribution. Unlike the distributionsof models trained with SE and CSWP, ASW shows a morebell like distribution similar to dataset distribution resultsin picking other pitches with more likelihood. The pitchdistribution for the model trained with CSWP embeddingvectors is shown in Fig. 12. We have concatenated theprojection syllable vector on world vector in order to givethe wordness to the syllable vector since syllable and wordembedding models trained separately. The pitch distributionof the model trained with CSWP captures the distributionrange similar to the dataset distribution but the model mostlikely predicts the mean pitch of the dataset distributionresulting in predicting average melody of the dataset. We canconclude that ASW and CSWP embeddings are most likelysuitable embeddings for generating meaningful melodiesqualitatively. We can also note that greedy search cangenerate at most one lyric for a seed lyric but temperaturebased sampling can generate original lyric and melody pairfor each τ . It should be noted that for various τ , the rangeof the distribution remains same with slight changes inthe shape of the distribution that signifies that the lyricsgenerated by temperature sampling is not random and wecan generate novel lyrics-melody pairs with each valueof τ . We also noted that the τ ∈ [0.5 − 1.0] is the bestrange to generate the lyrics-melody pairs which follows

11

10

15

20

25

Unique tones

70

75

80

85

Maximum tone value

SE

SWC

ASW

CSWP

0.5 0.6 0.7 0.8 0.9 1.0

τ

40

50

60

Minimum tone value

0.5 0.6 0.7 0.8 0.9 1.0

τ

10

20

30

40

50

Tone span

Fig. 14: Objective measures for various τ values and embedding representations for the melody composed from generatedlyrics.

TABLE I: Seed lyrics and the generate full length songs (τ = 0.6).

Seed lyrics Generated lyrics

i never needed you to be strong

i never needed you to be strong i neversaw you you any time i could we be got

i believe you i will always will be the onei love you if you know it’s just a girl i can stop

the music in the night you can make mefeel the way now i know you will see something

that i need i feel alone i wanna be alone without a love to be a part how am i supposed to

chances are find me some where on your road

chances are find me some where on your roadso the morning comes before i believe katy she

walks the one is me pa pa pa to three yearsto rock me pretending let me be the one we flyi will be there when we have to let the musicwe belong together i don’t know now i know

i know but i know i know i need i gotto believe i would n’t cry i would n’t let me

our world is miles of endless roads

our world is miles of end less roads a drummerwalks we always be mellow nun y happyit’s i never mind in love and in your heart

that’s that ing i don’t need to know if are i findyour self i can never be my baby but i

can’t believe that i’m not the one i’m waiting forthe rain on the rain once makes me cry on t

he rain on the rain looking out that i still have

12

Unique tones Max. tone Min. tone Tone span

10

20

30

40

50

60

70

80

SE

SWC

ASW

CSWP

Fig. 15: Objective measures for lyric embedding representa-tions for test set lyrics.

dataset distribution. Further, we can also conclude that ASWrepresentation of lyrics can learn the correlation between thegenerated lyrics and melody in an effective manner. Sincewe can also generate the melody by feeding the originalhuman generated lyrics to our model, we have composed themelody for 10 randomly chosen full length lyrics. The pitchhistogram distribution shown in Fig. 13 also confirms that theASW and CSWP lyric embedding representations are betterrepresentations for capturing the lyric-melody correlations.We also investigate the musical note attribute related objectivemeasures proposed in [6]: 1) unique tones, which measuresthe total number of unique tones generated by the model,2) maximum tone value, which gives the highest tone valuegenerated, 3) minimum tone value, which represents theminimum tone value generated and 4) tone span, which isthe range of the pitch values present in the generated melodyto validate the claim. Fig. 14 shows the tone related objectivemeasures for the melody composed from generated lyricsfor various τ values and lyric embedding representations.From the left top Fig. 14, we can observe that ASW andCSWP embedding representations generate a large numberof unique tones compare to other representations. Similarly,ASW and CSWP representations are capable of generatingvery low and very high tone values which is consistent withthe dataset distribution which can be observed from leftbottom and right top of Fig. 14. The tone span plotted in thebottom right of Fig. 14 also shows that ASW and CSWPrepresentations are capable of generating larger tone spanmelodies compared to other representations. The tone relatedobjective measures in Fig. 14 also reveal that irrespectiveof τ value, the objective measures remains constant for SEand SWC which indicated that these representations arenot capable of generating expressive melodies for differentvalues of τ even though generated lyrics are quite differentfor each τ value. Whereas ASW and CSWP are capableof generating interesting tones for various values of τ . Theobjective measures for the pitch generated from the test setlyrics is shown in Fig. 14 also confirms that ASW and CSWPare indeed good representations for generating melodies.In [15], authors proposed to use BLEU (bilingual evaluationunderstudy) [24] as a metric for evaluating the predictedpitch from lyrics. We also evaluate our model with BLEUscore for predicted pitch, duration and rest. BLEU is a singlereal number value to evaluate the translation quality of the

model from input to output. Here, we compute the BLEUscores for the melodies generated from the test set lyricsand corresponding ground truth melodies. Higher BLEUscore indicates better correlation between the generated andground truth melodies. The 1-gram, 2-gram, 3-gram, 4-gramand 5-gram BLEU scores of pitch, duration and rest melodyattributes for each embedding representations are shownin Fig. 16. From Fig. 16, we can observe that ASW andCSWP representations out performs other representations.Few samples of music sheet scores for the generatedlyrics and melody pairs by the proposed AutoNLMC isshown in Fig. 17. More samples can be found at https://drive.google.com/file/d/1NTvo19CRzUqiUZokJaXM4WVogXj1sdb/view?usp=sharing.

IX. SUBJECTIVE EVALUATION

Although statistical and objective measures indicates thatthe model succeeds in capturing the underlying data distri-bution for generating novel lyrics and melodies, it is stilldifficult to conclude that the automatically generated compo-sition pleases human ears. Further, lyric writing and melodycomposition are human creative process hence, it is verychallenging to precisely quantify the automatically gener-ated lyrics and melodies objectively and statistically. Hence,we adopt the subjective evaluation method proposed in [9]and [25] for evaluating generated lyrics and melodies by ourAutoNLMC. We ask the following questions to participantsduring subjective evaluation• How meaningful are the predicted lyrics?• How well does the melodies fit the lyrics?

Participants rate the given samples on a five point discretescale from 1 to 5 (where 1 corresponds to ”very bad”, 2to ”bad”, 3 to ”ok”, 4 to ”good”, and 5 to ””very good”).We consider five full length lyrics and melodies generatedfrom AutoNLMC for subjective evaluation. We evaluate thegenerated lyrics and melodies with baseline method and theground truth human compositions. We follow [9] and [25]to create baseline lyrics and melodies. The attributes of thebaseline melodies i.e., pitch, duration and rest are sampledfrom the respective data distributions except for pitch whoseMIDI values sampled between 55-80, as most of the pitchdistribution is concentrated in this range in our dataset. Thebaseline lyrics are sampled from the most frequent sylla-ble lyrics vocabulary. We separately conduct the subjectiveevaluation for generated lyrics and lyric-melody pairs. Thesubjective evaluation is conducted through the Google sheetswhere Google sheets consists of the samples understudy, radiobuttons indicating scores to choose from, and clear instructionson how to rate the samples based on 5 point scale. Anexample evaluation sheet used for lyrics evaluation can beaccessed from here https://forms.gle/vZ6KnPDR6FgStizG9.We requested 20 adults who had at least basic knowledgeabout lyrics, melodies and specifically about the music compo-sition for subjective evaluation. Out of 20 request participants,we have obtained response from about 11 participants overthe two weeks survey period. We randomize 5 full lengthgenerated lyrics from the proposed AutoNLMC, baseline and

13

1 2 3 4 5

0.05

0.10

0.15

0.20

0.25

pitchSESWCASWCSWP

1 2 3 4 5

0.15

0.20

0.25

0.30

0.35

duration

1 2 3 4 5

0.15

0.16

0.17

0.18

0.19

0.20

0.21

0.22

rest

Fig. 16: BLEU score evaluation of pitch, duration and rest melody attributes for SE, SWC, ASW and CSWP embeddingrepresentations. Abscissa represents the 1, 2, 3, 4, and 5 gram BLEU scores respectively.

(a)

(b)

(c)

(d)

Fig. 17: Music sheet scores for generated lyrics-melody pairs for τ = 0.8 for SE, SWC, ASW and CSWP are show in Figs.a), b), c) and d) respectively.

14

Fig. 18: Subjective evaluation results for lyrics and lyric-melody pairs. Lyrics and melodies are generated from ASW andCSWP of the proposed AutoNLMC for τ = 0.6.

the ground truth lyrics. We ask the subjects to rate for thequestion ”how meaningful are the presented lyrics”. Similarly,for lyrics-melody pairs, we randomly choose lyrics-melodypairs from AutoNLMC, baseline and the ground truth humancompositions. We ask the raters to rate for the question ”howwell does the melodies fit the lyrics” on a five point scale. Thesubjective evaluation results for ASW and CSWP for τ = 0.6are shown in Fig. 18. We can see that ASW is marginally betterthan CSWP. We can also observe that AutoNLMC is close tothe human compositions. The baseline performs worst thanthe other methods without any surprise. We also see that theASW performs better than CSWP which is in correlation withthe objective measures of Section VIII. From the subjectiveevaluation measures, we can find that there is a gap betweenthe human compositions and generated lyrics and melodiesby the proposed model. Which indicated that there is a lot ofscope to explore injecting prior musical knowledge to improvethe current model.

X. SUMMARY

In this paper, we proposed Automatic Neural Lyrics andMelody Composition (AutoNLMC) to enable the human com-munity to discover the original lyrics, and the correspondingmatching melodies. We trained lyrics to vector (lyric2vec)models on a large set of lyrics-melody parallel dataset parsedat syllable, word and sentence level to extract the lyricembeddings. The proposed AutoNLMC is a encoder-decodersequential recurrent neural network model consisted of lyricgenerator, lyrics encoder and melody decoder for each noteattribute trained end-to-end to learn the correlation between thelyrics and melody pairs with attention mechanism. AutoNLMCis designed to operate in two modes such that it can generateboth lyrics and corresponding melody automatically for anamateur or person with no music knowledge, or it can takeseed lyrics from professional lyric writer to generate the corre-sponding melodies. The qualitative and quantitative evaluationmeasures reveled that the proposed method indeed capableof generating original lyrics and corresponding melody forgenerating new songs.

REFERENCES

[1] M. Ackerman and D. Loker, “Algorithmic songwriting with alysia,”in International Conference on Evolutionary and Biologically InspiredMusic and Art. Springer, 2017, pp. 1–16.

[2] F. Pachet and P. Roy, “Markov constraints: steerable generation ofmarkov sequences,” Constraints, vol. 16, no. 2, pp. 148–172, 2011.

[3] F. Pachet, A. Papadopoulos, and P. Roy, “Sampling variations ofsequences for structured music generation.” in ISMIR, 2017, pp. 167–173.

[4] E. Waite et al., “Generating long-term structure in songs and stories,”Magenta Bolg, 2016.

[5] H. Chu, R. Urtasun, and S. Fidler, “Song from pi: A musically plausiblenetwork for pop music generation,” arXiv preprint arXiv:1611.03477,2016.

[6] O. Mogren, “C-rnn-gan: Continuous recurrent neural networks withadversarial training,” arXiv preprint arXiv:1611.09904, 2016.

[7] H.-W. Dong, W.-Y. Hsiao, L.-C. Yang, and Y.-H. Yang, “Musegan:Multi-track sequential generative adversarial networks for symbolic mu-sic generation and accompaniment,” in Thirty-Second AAAI Conferenceon Artificial Intelligence, 2018.

[8] Y. Yu, S. Tang, F. Raposo, and L. Chen, “Deep cross-modal correlationlearning for audio and lyrics in music retrieval,” ACM Transactions onMultimedia Computing, Communications, and Applications (TOMM),vol. 15, no. 1, p. 20, 2019.

[9] Y. Yu and S. Canales, “Conditional lstm-gan for melody generation fromlyrics,” arXiv preprint arXiv:1908.05551, 2019.

[10] S. Fukayama, K. Nakatsuma, S. Sako, T. Nishimoto, and S. Sagayama,“Automatic song composition from the lyrics exploiting prosody ofthe japanese language,” in Proc. 7th Sound and Music ComputingConference (SMC), 2010, pp. 299–302.

[11] J. Toivanen, H. Toivonen, A. Valitutti et al., “Automatical composition oflyrical songs,” in The Fourth International Conference on ComputationalCreativity, 2013.

[12] K. Monteith, T. R. Martinez, and D. Ventura, “Automatic generation ofmelodic accompaniments for lyrics.” in ICCC, 2012, pp. 87–94.

[13] E. Nichols, “Lyric-based rhythm suggestion,” in ICMC, 2009.[14] H. G. Oliveira, “Tra-la-lyrics 2.0: Automatic generation of song lyrics

on a semantic domain,” Journal of Artificial General Intelligence, vol. 6,no. 1, pp. 87–110, 2015.

[15] H. Bao, S. Huang, F. Wei, L. Cui, Y. Wu, C. Tan, S. Piao, andM. Zhou, “Neural melody composition from lyrics,” arXiv preprintarXiv:1809.04318, 2018.

[16] K. Cho, B. Van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares,H. Schwenk, and Y. Bengio, “Learning phrase representations usingrnn encoder-decoder for statistical machine translation,” arXiv preprintarXiv:1406.1078, 2014.

[17] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neuralcomputation, vol. 9, no. 8, pp. 1735–1780, 1997.

[18] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learningwith neural networks,” in Advances in neural information processingsystems, 2014, pp. 3104–3112.

[19] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation ofword representations in vector space,” arXiv preprint arXiv:1301.3781,2013.

[20] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean,“Distributed representations of words and phrases and their composi-tionality,” in Advances in neural information processing systems, 2013,pp. 3111–3119.

[21] H. Caselles-Dupre, F. Lesaint, and J. Royo-Letelier, “Word2vec appliedto recommendation: Hyperparameters matter,” in Proceedings of the 12thACM Conference on Recommender Systems. ACM, 2018, pp. 352–356.

15

[22] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation byjointly learning to align and translate,” arXiv preprint arXiv:1409.0473,2014.

[23] R. Stewart, “Maximum likelihood decoding with rnns - the good, thebad, and the ugly,” https://nlp.stanford.edu/blog/maximum-likelihood-decoding-with-rnns-the-good-the-bad-and-the-ugly/, 2016.

[24] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a methodfor automatic evaluation of machine translation,” in Proceedings ofthe 40th annual meeting on association for computational linguistics.Association for Computational Linguistics, 2002, pp. 311–318.

[25] H.-P. Lee, J.-S. Fang, and W.-Y. Ma, “icomposer: An automatic song-writing system for chinese popular music,” in Proceedings of the 2019Conference of the North American Chapter of the Association forComputational Linguistics (Demonstrations), 2019, pp. 84–88.