Web Search

Module 6

INST 734

Doug Oard

Agenda

The Web

• Crawling

• Web search

Washington Post, February 10, 2011

HTML(data/display)

Internetcommunication

protocols

RTSPFTPEmail

WebServer

HTTP(transfer)

File System

URL(e.g.,http://www.foo.org/snarf.html)

HTMLHTTPURL

“The Web”

Internet Web

• Internet: collection of global networks

• Web: way of managing information exchange

• There are many other uses for the Internet– File transfer (FTP)– Email (SMTP, POP, IMAP)

What “Caused” the Web?

• Affordable storage– 300,000 (typed) words/$ by 1995

• Adequate network capacity– 25,000 simultaneous transfers by 1995– 1 second/screen (of text) by 1995

• Display capability– 10% of US population could see images by 1995

• Effective search capabilities– Lycos and Yahoo! achieved useful scale in 1994-1995

Internet Hosts

Jan-81 Jan-86 Jan-91 Jan-96 Jan-01 Jan-06 Jan-110

100,000,000

200,000,000

300,000,000

400,000,000

500,000,000

600,000,000

700,000,000

800,000,000

900,000,000

1,000,000,000

Internet Users

Jan-94 Jan-96 Jan-98 Jan-00 Jan-02 Jan-04 Jan-06 Jan-08 Jan-10 Jan-120%

5%

10%

15%

20%

25%

30%

35%

Por

tion

of

the

Glo

bal

Pop

ula

tion

http://www.internetworldstats.com/

64%5%

4%

5%

2%

8%

2%4%

5% 0%

33%

28%

9%

6%

5%

5%

4%

4%

4%2%

EnglishChineseSpanishJapanesePortuguese GermanArabicFrenchRussianKorean

Billions of Queries per Month

Google; 114.7

Baidu; 14.5

Bing+Yahoo; 13.1

Yandex; 4.8

Other; 28.7

December, 2012

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

18,000,000

20,000,000

Mar

-03

Apr-0

3

May

-03

Jun-

03

Jul-0

3

Aug-0

3

Sep-

03

Oct

-03

Nov-0

3

Dec-0

3

Jan-

04

Feb-

04

Mar

-04

Apr-0

4

May

-04

Jun-

04

Jul-0

4

Aug-0

4

Sep-

04

Oct

-04

Nov-0

4

Dec-0

4

Jan-

05

Feb-

05

Mar

-05

Apr-0

5

May

-05

Jun-

05

Jul-0

5

Aug-0

5

Sep-

05

Oct

-05

Doubling

Doubling

Doubling

18.9 Million Weblogs TrackedDoubling in size approx. every 5 monthsConsistent doubling over the last 36 months

An Adoption Curve: BlogsDoubling

November 2013http://www.searchenginejournal.com/growth-social-media-2-0-infographic/77055/

Agenda

• The Web

Crawling

• Web search