Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps

www.AgeofMobility.com

http://www.bigbaldapps.com


@ActiveNick 2005-2014

Computer Speech is the Dream

Mobile Speech is a Responsibility

AccessibilityMedical

Conditions

Hands &

Eyes Free

Driving Apps

& Games

Learning &

Therapy

Travel Apps

with

Translation

Speech SDK

Features for DevsWindows Phone Windows Store

iOS

(iPhone/iPad)

Android

(Phone/Tablet)

Built-in Personal

Assistant with

Speech

Cortana* N/A Siri* Google Now

Launcher*

Personal Assistant

Extensions in third-

party apps using

Voice Commands

Yes N/A No

(first-party apps

only)

No

(first-party apps

only)

Speech Synthesis

SDK for Devs

(Text-to-Speech)

Yes Yes Yes Yes

In-app Speech

Recognition SDK

for Devs

Yes Yes* No* Yes

<?xml version="1.0" encoding="utf-8"?><VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1">

<CommandSet xml:lang="en-us" Name="englishCommands">

<CommandPrefix>MSDN</CommandPrefix><Example>How do I add Voice Commands to my application</Example>

<Command Name="FindText"><Example>Find Install Voice Command Sets</Example><ListenFor>Search</ListenFor><ListenFor>Search for {dictatedSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {dictatedSearchTerms}</ListenFor><Feedback>Search on MSDN</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<Command Name="nlpCommand"><Example>How do I add Voice Commands to my application</Example><ListenFor>{dictatedVoiceCommandText}</ListenFor><Feedback>Starting MSDN...</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<PhraseTopic Label="dictatedVoiceCommandText" Scenario="Dictation"><Subject>MSDN</Subject>

</PhraseTopic>

<PhraseTopic Label="dictatedSearchTerms" Scenario="Search"><Subject>MSDN</Subject>

</PhraseTopic>

</CommandSet>

</VoiceCommands>





</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

<CommandPrefix>MSDN</CommandPrefix>

http://schemas.microsoft.com/voicecommands/1.1





</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Example>How do I add Voice Commands to my application</Example>






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>


</Command>


/Command>






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Example>Find Install Voice Command Sets</Example>






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

<ListenFor>Search</ListenFor><ListenFor>Search for {dictatedSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {dictatedSearchTerms}</ListenFor>






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Feedback>Search on MSDN</Feedback>






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

<Navigate Target="MainPage.xaml" />






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>


</PhraseTopic>


</PhraseTopic>




<CommandPrefix>MSDN</CommandPrefix><Example>Find Voice Commands</Example>

<Command Name="FindText"><Example>Find Windows Phone</Example><ListenFor>Search</ListenFor><ListenFor>Search {*}</ListenFor><ListenFor>Search for {listSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {*}</ListenFor><ListenFor>Find {listSearchTerms}</ListenFor><Feedback>Search on MSDN</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<PhraseList Label="listSearchTerms" Disambiguate="false"><Item>Voice Commands</Item><Item>Windows Phone</Item>

</PhraseList>

</CommandSet>

</VoiceCommands>





</Command>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

Windows Phone 8.0 Windows Phone 8.1

Simple Scenarios, Limited Accuracy More Powerful, Easier, More Accurate





<CommandPrefix>MSDN</CommandPrefix><Example>Find Voice Commands</Example>

<Command Name="FindText"><Example>Find Windows Phone</Example><ListenFor>Search</ListenFor><ListenFor>Search {*}</ListenFor><ListenFor>Search for {listSearchTerms}</ListenFor><ListenFor>Find</ListenFor><ListenFor>Find {*}</ListenFor><ListenFor>Find {listSearchTerms}</ListenFor><Feedback>Search on MSDN</Feedback><Navigate Target="MainPage.xaml" />

</Command>

<PhraseList Label="listSearchTerms" Disambiguate="false"><Item>Voice Commands</Item><Item>Windows Phone</Item>

</PhraseList>

</CommandSet>

</VoiceCommands>





</Command>


</PhraseTopic>

</CommandSet>

</VoiceCommands>

Windows Phone 8.0 Windows Phone 8.1

Simple Scenarios, Limited Accuracy … and enables Natural Language






</Command>


</Command>


</PhraseTopic>


</PhraseTopic>

</CommandSet>

</VoiceCommands>


private async void

// SHOULD BE PERFORMED UNDER TRY/CATCH

Uri new ms-appx:///vcd.xml UriKind.Absolute

await

Windows Phone Silverlight Apps on Windows Phone 8.1

Windows Runtime Apps on Windows Phone 8.1

private async void

// SHOULD BE PERFORMED UNDER TRY/CATCH

Uri uriVoiceCommands = new Uri("ms-appx:///vcd.xml", UriKind.Absolute);

StorageFile file = await StorageFile.GetFileFromApplicationUriAsync(uriVoiceCommands);

await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(file);

❶

❷

// Windows Phone Silverlight App, in MainPage.xaml.cs

protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)

{

base.OnNavigatedTo(e);

if (e.NavigationMode == System.Windows.Navigation.NavigationMode.New)

{

string recoText = null; // What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"

NavigationContext.QueryString.TryGetValue("reco", out recoText);

string voiceCommandName = null; // Which command was recognized in the VCD.XML file? e.g. "FindText"

NavigationContext.QueryString.TryGetValue("voiceCommandName", out voiceCommandName);

string searchTerms = null; // What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"

NavigationContext.QueryString.TryGetValue("dictatedSearchTerms", out searchTerms);

switch (voiceCommandName) // What command launched the app?

{

case "FindText":

HandleFindText(searchTerms);

break;

case "nlpCommand":

HandleNlpCommand(recoText);

break;

}

}

}



{



{








{

case "FindText":


break;

case "nlpCommand":


break;

}

}

}





{



{








{

case "FindText":


break;

case "nlpCommand":


break;

}

}

}

// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"string recoText = null;NavigationContext.QueryString.TryGetValue("reco", out recoText);

// Which command was recognized in the VCD.XML file? e.g. "FindText"string voiceCommandName = null; NavigationContext.QueryString.TryGetValue("voiceCommandName", out voiceCommandName);

// What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"string searchTerms = null;NavigationContext.QueryString.TryGetValue("dictatedSearchTerms", out searchTerms);



{



{








{

case "FindText":


break;

case "nlpCommand":


break;

}

}

}

switch (voiceCommandName) // What command launched the app?{

case "FindText":HandleFindText(searchTerms);break;

case "nlpCommand":HandleNlpCommand(recoText);break;

}

❶

❷

❸

// Windows Runtime App on Windows Phone 8.1, inside OnActivated override in App class

if (args.Kind == ActivationKind.VoiceCommand)

{

VoiceCommandActivatedEventArgs vcArgs = (VoiceCommandActivatedEventArgs)args;

string voiceCommandName = vcArgs.Result.RulePath.First(); // What command launched the app?

switch (voiceCommandName) // Navigate to right page for the voice command

{

case "FindText": // User said "find" or "search"

rootFrame.Navigate(typeof(MSDN.FindText), vcArgs.Result);

break;

case "nlpCommand": // User said something else

rootFrame.Navigate(typeof(MSDN.NlpCommand), vcArgs.Result);

break;

}

}



{




{



break;



break;

}

}

// Windows Runtime App on Windows Phone 8.1, inside OnActivated override// in App class




{




{



break;



break;

}

}


// What command launched the app?string voiceCommandName = vcArgs.Result.RulePath.First();

// Navigate to right page for the voice command switch (voiceCommandName)



{




{



break;



break;

}

}

// Navigate to right page for the voice command switch (voiceCommandName) {

case "FindText": // User said "find" or "search"rootFrame.Navigate(typeof(MSDN.FindText), vcArgs.Result);break;

case "nlpCommand": // User said something elserootFrame.Navigate(typeof(MSDN.NlpCommand), vcArgs.Result);break;

}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in FindText.xaml.cs

protected override void OnNavigatedTo(NavigationEventArgs e)

{

// Get recognition result from parameter passed in frame.Navigate call

SpeechRecognitionResult vcResult = e.Parameter as SpeechRecognitionResult;

if (vcResult!=null)

{

// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"

string recoText = vcResult.Text;

// Store the semantics dictionary for later use

IReadOnlyDictionary<string,IReadOnlyList<string>> semantics = vcResult.SemanticInterpretation.Properties;

string voiceCommandName = vcResult.RulePath.First();

if (voiceCommandName == "FindText")

{

// What did the user say, for named phrase topic or list "slots"? e.g. "Windows Phone Voice Commands"

if (semantics.ContainsKey("dictatedSearchTerms"))

{

HandleFindTextWithSearchTerms(semantics["dictatedSearchTerms"][0]);

}

else

{

HandleNoSearchTerms();

}

}

// Else handle other voice commands

} navigationHelper.OnNavigatedTo(e)

}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In FindText.xaml.cs


{



if (vcResult!=null)

{







{



{


}

else

{


}

}



}

// Windows Runtime App on Windows Phone 8.1, inside // OnNavigatedTo in FindText.xaml.csprotected override void OnNavigatedTo(NavigationEventArgs e){

// Get recognition result from parameter passed in frame.Navigate callSpeechRecognitionResult vcResult = e.Parameter as SpeechRecognitionResult;

if (vcResult!=null){

// What did the user say? e.g. MSDN, "Find Windows Phone Voice Commands"string recoText = vcResult.Text;

// Store the semantics dictionary for later useIReadOnlyDictionary<string,IReadOnlyList<string>> semantics =

vcResult.SemanticInterpretation.Properties;


// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In FindText.xaml.cs


{



if (vcResult!=null)

{







{



{


}

else

{


}

}



}

if (voiceCommandName == "FindText"){

// What did the user say, for named phrase topic or list "slots"?// e.g. "Windows Phone Voice Commands"if (semantics.ContainsKey("dictatedSearchTerms")){

HandleFindTextWithSearchTerms(semantics["dictatedSearchTerms"][0]);}else{

HandleNoSearchTerms();}

}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo in NlpCommand.xaml.cs


{




// Check for null!

string commandMode = vcResult.SemanticInterpretation.Properties["commandMode"][0];

if (commandMode == "voice") // Did the user speak or type the command?

{

SpeakText(audioPlayer, String.Format("MSDN app heard you say {0}", vcResult.Text));

HandleNlpCommand(vcResult);

}

else if(commandMode=="text")

{

messageTextBox.Text = string.Format("Working on your request \"{0}\"", vcResult.Text);


}

}

// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In NlpCommand.xaml.cs


{




// Check for null!



{



}


{



}

}

// Windows Runtime App on Windows Phone 8.1, in OnNavigatedTo in NlpCommand.xaml.cs


{




// Windows Runtime App on Windows Phone 8.1, inside OnNavigatedTo In NlpCommand.xaml.cs


{




// Check for null!



{



}


{



}

}



{

SpeakText(audioPlayer, String.Format("MSDN voice heard you say {0}", vcResult.Text));


}


{

msgTextBox.Text= string.Format("Working on your request \"{0}\"", vcResult.Text);


}

private void HandleNlpCommand(string recoText, bool actSilently)

{

string action = null;

string navigateTo = null;

string searchFor = null;

recoText = recoText.ToLower();

if (recoText.Contains("go to ") || recoText.Contains("goto ") ||

recoText.Contains("find ") || recoText.Contains("search ") ||

recoText.Contains("show me "))

{

action = "navigate";

if (recoText.Contains("windows phone dev center"))

{

navigateTo = "http://dev.windowsphone.com";

}

}

else if (recoText.Contains("learn how to "))

{

action = "find";

searchFor = recoText.Substring(recoText.IndexOf("learn how to ") + 13);

}

else

{

action = "find";

searchFor = recoText;

}

switch (action)

{

case "find":

// ...


{








{



{


}

}


{

action = "find";


}

else

{

action = "find";


}

switch (action)

{

case "find":

// ...

if (recoText.Contains("go to ") || recoText.Contains("goto ") ||recoText.Contains("find ") || recoText.Contains("search ") ||recoText.Contains("show me "))

{action = "navigate";if (recoText.Contains("windows phone dev center")){

navigateTo = "http://dev.windowsphone.com";}

}


{








{



{


}

}


{

action = "find";


}

else

{

action = "find";


}

switch (action)

{

case "find":

// ...

else if (recoText.Contains("learn how to ")){

action = "find";searchFor = recoText.Substring(recoText.IndexOf("learn how to ") + 13);

}


{








{



{


}

}


{

action = "find";


}

else

{

action = "find";


}

switch (action)

{

case "find":

// ...

else{

action = "find";searchFor = recoText;

}

// Windows Phone Silverlight App

// Synthesis

private async void SpeakText(string textToSpeak)

{

SpeechSynthesizer synthesizer = new SpeechSynthesizer();

await synthesizer.SpeakTextAsync(textToSpeak);

}

// Recognition

private async Task<SpeechRecognitionUIResult> RecognizeSpeech()

{

SpeechRecognizerUI recognizer = new SpeechRecognizerUI();

// One of three Grammar types available

recognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(

"key1", SpeechPredefinedGrammar.WebSearch);

await recognizer.Recognizer.PreloadGrammarsAsync(); // Optional but recommended

// Put up UI and recognize user's utterance

SpeechRecognitionUIResult result = await recognizer.RecognizeWithUIAsync();

return result;

}

// Calling code uses result.RecognitionResult.Text or result.RecognitionResult.Semantics


// Synthesis


{



}

// Recognition


{








return result;

}


// Synthesisprivate async void SpeakText(string textToSpeak){

SpeechSynthesizer synthesizer = new SpeechSynthesizer();await synthesizer.SpeakTextAsync(textToSpeak);

}

// Windows Phone Silverlight App// Synthesisprivate async void SpeakText(string textToSpeak){

SpeechSynthesizer synthesizer = new SpeechSynthesizer();await synthesizer.SpeakTextAsync(textToSpeak);

}

// Recognitionprivate async Task<SpeechRecognitionUIResult> RecognizeSpeech(){


// One of three Grammar types availablerecognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(



// Put up UI and recognize user's utteranceSpeechRecognitionUIResult result = await recognizer.RecognizeWithUIAsync();

return result;}// Calling code uses result.RecognitionResult.Text or result.RecognitionResult.Semantics

// Recognitionprivate async Task<SpeechRecognitionUIResult> RecognizeSpeech(){


// One of three Grammar types availablerecognizer.Recognizer.Grammars.AddGrammarFromPredefinedType(


// Optional but recommendedawait recognizer.Recognizer.PreloadGrammarsAsync();


// Synthesis


{



}

// Recognition


{








return result;

}


// Put up UI and recognize user's utteranceSpeechRecognitionUIResult result

= await recognizer.RecognizeWithUIAsync();

return result;

// Windows Phone Store App

// Synthesis



<MediaElement Name="audioPlayer" AutoPlay="True" .../>

// C# code behind

// Function to speak a text string

private async void SpeakText(MediaElement audioPlayer, string textToSpeak)

{


SpeechSynthesisStream ttsStream = await synthesizer.SynthesizeTextToStreamAsync(textToSpeak);

audioPlayer.SetSource(ttsStream, ""); // This starts the player because AutoPlay="True"

}


// Synthesis



// C# code behind



{




}

// Synthesis<MediaElement Name="audioPlayer" AutoPlay="True" .../>


// Synthesis



// C# code behind



{




}

// C# code behind// Function to speak a text stringprivate async void SpeakText(MediaElement audioPlayer, string textToSpeak){


SpeechSynthesisStream ttsStream= await synthesizer.SynthesizeTextToStreamAsync(textToSpeak);


}


// Recognition

private async Task<SpeechRecognitionResult> RecognizeSpeech()

{

SpeechRecognizer recognizer = new SpeechRecognizer();

// One of three Constraint types available

SpeechRecognitionTopicConstraint topicConstraint

= new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.WebSearch, "MSDN");

recognizer.Constraints.Add(topicConstraint);

await recognizer.CompileConstraintsAsync(); // Required


SpeechRecognitionResult result = await recognizer.RecognizeWithUIAsync();

return result;

}

// Calling code uses result.RecognitionResult.Text or// result.RecognitionResult.SemanticInterpretation


// Recognition


{









return result;

}



// One of three Constraint types availableSpeechRecognitionTopicConstraint topicConstraint





// Recognition


{









return result;

}


// Put up UI and recognize user's utteranceSpeechRecognitionResult result

= await recognizer.RecognizeWithUIAsync();

return result;}// Calling code uses result.RecognitionResult.Text or// result.RecognitionResult.SemanticInterpretation

❶

❷

❶

❷

❸

❹

❺

❻

❶

❷

http://aka.ms/CortanaDev

http://channel9.msdn.com/Events/Build/2014/2-530

http://www.bing.com/blogs/site_blogs/b/search/archive/2014/04/14/integrating-store-apps-with-cortana-in-windows-phone-8-1.aspx

http://www.bing.com/dev/en-us/speech

http://msdn.microsoft.com/en-us/magazine/jj721592.aspx

http://blogs.msdn.com/b/robch

http://www.windowsphone.com/en-us/store/app/msdn-voice-search/aed09797-b424-4030-a04d-5ec8ec63847f

http://code.msdn.microsoft.com/wpapps/MSDN-Voice-Search-for-95c16d92

https://voicecommands.codeplex.com

http://code.msdn.microsoft.com/wpapps/Voice-Commands-for-Windows-2f808ac6

http://code.msdn.microsoft.com/windowsapps/Speech-synthesis-sample-6e07b218

http://code.msdn.microsoft.com/windowsapps/Bing-Speech-Recognition-210c8978

http://code.msdn.microsoft.com/windowsapps/Speech-Recognition-19c452b4

https://developer.apple.com/technologies/ios/accessibility.html

https://developer.apple.com/library/ios/documentation/AVFoundation/Reference/AVSpeechSynthesizer_Ref/Reference/Reference.html

http://nshipster.com/avspeechsynthesizer

DEVFRIGHT Blog Post: http://www.devfright.com/quick-look-avspeechsynthesizer-class/

https://play.google.com/store/apps/details?id=com.google.android.launcher

http://developer.android.com/reference/android/speech/package-summary.html

http://developer.android.com/reference/android/speech/tts/package-summary.html

http://channel9.msdn.com/Events/Build/2014/

http://channel9.msdn.com/Events/Build/2014

Get Ready to Become a Windows Phone Developerdev.windowsphone.com

Learn More About Windows Phone Dev via Official Microsoft Videoshttp://aka.ms/wp81js

http://aka.ms/wp81devbegin

Check Out Additional Learning Resourceswww.pluralsight.com/training/Courses#windows-phone

www.developer.nokia.com

Download Additional Resources & Become an Expertphone.codeplex.com

http://bit.ly/nokiadevoffers

68

1

2

3

4


dev.windowsphone.com

dev.windows.com

blogs.windows.com/windows_phone/b/wpdev

blogs.windows.com/windows_phone/b/windowsphone

http://developer.nokia.com

http://bit.ly/godvlup

http://conversations.nokia.com

http://appstudio.windows.com

http://flip.it/95YFG

www.geekchamp.com

www.wpcentral.com

Thank You!Slides will be posted on my Slideshare account.

Slideshare: www.slideshare.net/ActiveNick

Blog: www.AgeofMobility.com

Twitter: @ActiveNick

Mobile Apps: www.bigbaldapps.com

LinkedIn: www.linkedin.com/in/activenick

Website: www.AgeofMobility.com

Email: [email protected]

Technology

Beyond Cortana & Siri: Using Speech Recognition & Speech Synthesis for the Next Generation of Mobile Apps