Artificial, Artificial Artificial, Artificial IntelligenceIntelligenceUsing Amazon Mechanical Turk and .NET to Using Amazon Mechanical Turk and .NET to Create a New Breed of Web AppCreate a New Breed of Web App

Jeff BarrJeff BarrWeb Services EvangelistWeb Services EvangelistAmazon Web ServicesAmazon Web Services

NGW012NGW012

AgendaAgenda

Amazon Web Services OverviewAmazon Web Services Overview

Artificial, Artificial IntelligenceArtificial, Artificial Intelligence

DemosDemos

Writing Mechanical Turk ApplicationsWriting Mechanical Turk Applications

Q&AQ&A

Introduction And Introduction And BackgroundBackground

Software development backgroundSoftware development background

Veteran of several startupsVeteran of several startups

Visual Studio team at MicrosoftVisual Studio team at Microsoft(DHTML, XML, Web Services) (DHTML, XML, Web Services)

3.5 Years with Amazon3.5 Years with Amazon

Amazon Web Services EvangelistAmazon Web Services Evangelist

What Is Amazon?What Is Amazon?

Online RetailerOnline RetailerOver 55 million active customer accountsOver 55 million active customer accounts

Seven countries: US, UK, Germany, Japan, France, Seven countries: US, UK, Germany, Japan, France, Canada, ChinaCanada, China

Technology ConsumerTechnology ConsumerMulti-National Web SitesMulti-National Web Sites

Vast Data Warehouse – 25 TBVast Data Warehouse – 25 TB

World-Class Logistics – 21 fulfillment centers; 9 million ft2World-Class Logistics – 21 fulfillment centers; 9 million ft2

Technology ProviderTechnology ProviderHundreds of thousands of Amazon AssociatesHundreds of thousands of Amazon Associates

Over 1,050,000 active seller accountsOver 1,050,000 active seller accounts

Over 150,000 software developers registered to use Over 150,000 software developers registered to use Amazon Web ServicesAmazon Web Services

Web Service ArchitectureWeb Service Architecture

Operating SystemOperating System

Libraries / DLLs / AssembliesLibraries / DLLs / Assemblies

ApplicationApplication

InternetInternet

Web ServicesWeb Services

ApplicationApplication

What Is Amazon Web What Is Amazon Web Services? Services?

APIs that give developers programmatic APIs that give developers programmatic access to Amazon’s data and technologyaccess to Amazon’s data and technology

Building-block web servicesBuilding-block web services

Web-scale infrastructureWeb-scale infrastructure

E-commerce capabilityE-commerce capability

Content, data, and informationContent, data, and information

New business modelsNew business models

Customer-created contentCustomer-created content

AWS Product FamilyAWS Product Family

Amazon E-Commerce ServiceAmazon E-Commerce ServiceComplete access to Amazon’s Complete access to Amazon’s product catalogproduct catalogFree + Associates Free + Associates commissions paidcommissions paid

Amazon Historical PricingAmazon Historical PricingData warehouse access for Data warehouse access for product pricingproduct pricingMonthly FeeMonthly Fee

Amazon Mechanical Turk Amazon Mechanical Turk Artificial Artificial IntelligenceArtificial Artificial Intelligence10% Commission 10% Commission Paid workforcePaid workforce

Amazon Simple Queue ServiceAmazon Simple Queue ServiceIT building blockIT building blockIn betaIn beta

Amazon S3Amazon S3Storage for the internetStorage for the internetCharge by storage/bandwidth Charge by storage/bandwidth usageusage

Alexa Web Information ServiceAlexa Web Information ServiceData warehouse access for web Data warehouse access for web crawl datacrawl data10K calls per month free, then 15 10K calls per month free, then 15 cents per 1000 callscents per 1000 calls

Alexa Web Search PlatformAlexa Web Search PlatformRoll your own search engineRoll your own search enginePay for time, storage, bandwidthPay for time, storage, bandwidth

Alexa Top SitesAlexa Top SitesTop sites by Alexa traffic rankTop sites by Alexa traffic rankCharges by URLCharges by URL

Amazon S3Amazon S3Simple storage serviceSimple storage service

Storage for the internet - web service to read Storage for the internet - web service to read and write dataand write data

15 cents per Gigabyte-Month to store data15 cents per Gigabyte-Month to store data

20 cents per Gigabyte to access data20 cents per Gigabyte to access data

Private and public storagePrivate and public storage

Scalable, reliable, cost-effective, and simple!Scalable, reliable, cost-effective, and simple!

Amazon Amazon Mechanical Mechanical

TurkTurk

Man Versus MachineMan Versus Machine

But Not ReallyBut Not Really

Chair Or Table?Chair Or Table?

Survey Says…Survey Says…

Amazon Mechanical TurkAmazon Mechanical Turk

Provides Web Services API for Provides Web Services API for computers to integrate artificial artificial computers to integrate artificial artificial

intelligence directly into their intelligence directly into their processing by making requests processing by making requests

of humansof humans

Put The Human In The LoopPut The Human In The Loop

while (read (photo))while (read (photo))

{{

ContainsHuman = CallMechanicalTurk(photo);ContainsHuman = CallMechanicalTurk(photo);

if (ContainsHuman)if (ContainsHuman)

{{    acceptPhoto(photo);    acceptPhoto(photo);  }  } else else

{{    rejectPhoto(photo);    rejectPhoto(photo); } }

} }

How It WorksHow It Workswww.mturk.com

WorkersArtificial, Artificially

Intelligent Software

Requester(Developer)

Human Intelligence Tasks (HITs)

Completed HITs

Worker Qualifications

What This MeansWhat This Means

For Software DevelopersFor Software DevelopersSolves problem of building applications that until now have not Solves problem of building applications that until now have not worked well without human intelligenceworked well without human intelligence

For BusinessesFor BusinessesEfficiently, effectively and inexpensively complete millions of Efficiently, effectively and inexpensively complete millions of tasks that require human intelligencetasks that require human intelligence

For For AnyoneAnyoneMake money using the skills you already have whenever your Make money using the skills you already have whenever your schedule allowsschedule allowsPreviously unimaginable business ideas are now feasible…Previously unimaginable business ideas are now feasible…

High Quality, Inexpensive, High Quality, Inexpensive, Massive Scale Image Clean-UpMassive Scale Image Clean-Up

Which picture best represents this Which picture best represents this business?business?

Easy-to-use application to build HITs Easy-to-use application to build HITs for Mturkfor Mturk

Requester-oriented interface:Requester-oriented interface: Create and issue a series of HITs (Human Create and issue a series of HITs (Human Intelligence Tasks)Intelligence Tasks)

There's no need to deal with source code or There's no need to deal with source code or with raw XML datawith raw XML data

Free Use of HIT-Builder Functions during Beta Test period Free Use of HIT-Builder Functions during Beta Test period (requires Amazon Requester account)(requires Amazon Requester account)

Building A HITBuilding A HIT

Hit-Hit-Builder.comBuilder.com

Building Applications Building Applications WithWith

Amazon Mechanical Amazon Mechanical TurkTurk

API StatusAPI Status

RequestersRequestersMust use APIsMust use APIs

Must have development skillsMust have development skills

Expect some small tweaks during the betaExpect some small tweaks during the beta

WorkersWorkersMust use mturk.comMust use mturk.com

Worker API in the futureWorker API in the future

API Basics - #1API Basics - #1

Use REST or SOAP, your choiceUse REST or SOAP, your choice

Be sure to check for IsValid in Be sure to check for IsValid in each responseeach response

URL encode Question and Answer URL encode Question and Answer XML dataXML data

Include proper namespace in XML dataInclude proper namespace in XML data

API Basics - #2API Basics - #2

Sign each request using your Sign each request using your Secret Key:Secret Key:

HMAC(Hashing Function)

Service

Operation

Timestamp

Secret Key

Base 64Encode

SOAPRequest

RESTRequest

URLEncode

System Design ProcessSystem Design Process

Design Great HITsDesign Great HITs

Identify Worker QualificationsIdentify Worker Qualifications

Determine approval strategyDetermine approval strategy

Build ApplicationBuild Application

Deposit funds into Amazon accountDeposit funds into Amazon account

Load QualificationsLoad Qualifications

Load HITsLoad HITs

Approve completed AssignmentsApprove completed Assignments

Retrieve resultsRetrieve results

Designing A Good HIT - #1Designing A Good HIT - #1

Question elementsQuestion elementsTitleTitle

Text BlockText Block

ListList

BinaryBinary

Answer elementsAnswer elementsFree TextFree Text

Selection (1 or multiple)Selection (1 or multiple)

Build XML using the Build XML using the Question BuilderQuestion Builder

Designing A Good HIT - #2Designing A Good HIT - #2

HIT design considerationsHIT design considerationsBe clearBe clearBe verbose, if necessaryBe verbose, if necessaryBe specific about needs & expectationsBe specific about needs & expectationsLeave no ambiguitiesLeave no ambiguitiesTest drive and monitor before going Test drive and monitor before going full-borefull-boreAlways have an appropriate “none of the Always have an appropriate “none of the above” choiceabove” choicePunctuation, capitalization, etc.Punctuation, capitalization, etc.Approval strategyApproval strategy

Identify Worker Identify Worker QualificationsQualifications

Create a test to admit only workers with the Create a test to admit only workers with the skills or other qualifications that are needed skills or other qualifications that are needed for your particular type of HITsfor your particular type of HITs

Qualification often looks very much like a HITQualification often looks very much like a HIT

Objective (multiple choice) or subjective Objective (multiple choice) or subjective (free text)(free text)

If necessary, use “Secret Code” If necessary, use “Secret Code” qualificationsqualifications

Use or build on existing QualificationsUse or build on existing Qualifications

Application FlowApplication Flow

LoadQualifications

Load HITs

AnyCompleted

Assignments?

Retrieve and Approve

Do LocalProcessing

More Work?

Y

YDone!

Do LocalProcessing

N

Load QualificationsLoad Qualifications

Relevant API CallsRelevant API CallsCreateQualificationTypeCreateQualificationType

GetQualificationRequestsGetQualificationRequests

GrantQualificationGrantQualification

NotesNotesQualifications which include an Answer Key are Qualifications which include an Answer Key are automatically granted when passedautomatically granted when passed

Application must explicitly check and grant others Application must explicitly check and grant others using GrantQualificationusing GrantQualification

You cannot currently attach a Qualification to a You cannot currently attach a Qualification to a worker without a prior requestworker without a prior request

Load HITsLoad HITs

Relevant API CallsRelevant API CallsCreateHIT - Returns HIT identifierCreateHIT - Returns HIT identifier

DisableHIT – Turn it offDisableHIT – Turn it off

ExtendHIT – Add additional timeExtendHIT – Add additional time

NotesNotesMultiple assignments per HIT are importantMultiple assignments per HIT are important

Auto approval timeAuto approval time

Qualification requirements can be attached Qualification requirements can be attached to HITsto HITs

Attach application data using Attach application data using RequesterAnnotationRequesterAnnotation

Poll And Approve WorkPoll And Approve Work

Relevant API CallsRelevant API CallsGetReviewableHITsGetReviewableHITs

GetHITsGetHITs

GetAssignmentsForHITGetAssignmentsForHIT

ApproveAssignment – Pay the WorkerApproveAssignment – Pay the Worker

RejectAssignment – Don’t pay the WorkerRejectAssignment – Don’t pay the Worker

DisposeHITDisposeHIT

ExtendHITExtendHIT

NotifyWorkers – Send e-mail notificationNotifyWorkers – Send e-mail notification

Retrieve ResultsRetrieve Results

Relevant API callsRelevant API callsGetAssignmentsForHIT – Get the GetAssignmentsForHIT – Get the actual answersactual answers

NotesNotesGetReviewableHITs returns list of non-GetReviewableHITs returns list of non-assignable HITsassignable HITs

Dispose or Extend (or they’ll keep Dispose or Extend (or they’ll keep coming back)coming back)

Perform Local ProcessingPerform Local Processing

Specific to your applicationSpecific to your application

TypesTypesIncremental (per Assignment)Incremental (per Assignment)

Batch (per group of related HITs)Batch (per group of related HITs)

Connect application logic back to TurkConnect application logic back to TurkIf application’s needs change, Dispose of If application’s needs change, Dispose of Associated HITAssociated HIT

Adjust pricing and timing dynamicallyAdjust pricing and timing dynamically

Adjust necessary QualificationsAdjust necessary Qualifications

Next Steps…Next Steps…

Find potential HITs:Find potential HITs:Within your organizationWithin your organization

Unmet or underserved marketUnmet or underserved market

Design workflow process around the Design workflow process around the Mechanical TurkMechanical Turk

Start writing code and loading HITsStart writing code and loading HITs

Process resultsProcess results

Get your work done!Get your work done!

Q & AQ & A

Thanks!Thanks!

© 2006 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

For More Information:For More Information:

Mechanical Turk: Mechanical Turk: www.mturk.comwww.mturk.com

AWS Blog: AWS Blog: aws.typepad.comaws.typepad.com

Amazon Web Services: Amazon Web Services: aws.amazon.comaws.amazon.com

© 2006 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.