Upload
bernice-mcdowell
View
216
Download
1
Tags:
Embed Size (px)
Citation preview
Scalable Web Crawling and Basic Transactions
Zachary G. IvesUniversity of Pennsylvania
CIS 455 / 555 – Internet and Web Systems
April 21, 2023
2
Administrivia
Emailed list of project partners due Friday
… For those without 4-person groups, I will try to assign / merge groups over the weekend … This might result in breaking up some 3
person groups
3
Mercator: Scalable Web Crawler
Expands a “URL frontier” Avoids re-crawling same URLs
Also considers whether a document has been seen before Every document has signature/checksum info
computed as it’s crawled
4
Mercator Web Crawler
1. Dequeue frontier URL2. Fetch document3. Record into RewindStream
(RIS)4. Check against fingerprints
to verify it’s new
5. Extract hyperlinks6. Filter unwanted links7. Check if URL repeated
(compare its hash)8. Enqueue URL
5
Mercator’s Polite Frontier Queues
Tries to go beyond breadth-first approach – want to have only one crawler thread per server
Distributed URL frontier queue: One subqueue per worker thread The worker thread is determined by hashing
the hostname of the URL Thus, only one outstanding request per web server
6
Mercator’s HTTP Fetcher
First, needs to ensure robots.txt is followed Caches the contents of robots.txt for various web
sites as it crawls them
Designed to be extensible to other protocols Had to write own HTTP requestor in Java –
their Java version didn’t have timeouts Today, can use setSoTimeout()
Can also use Java non-blocking I/O if you wish: http://www.owlmountain.com/tutorials/NonBlockingIo.htm But they use multiple threads and synchronous I/O
7
Other Caveats
Infinitely long URL names (good way to get a buffer overflow!)
Aliased host names Alternative paths to the same host Can catch most of these with signatures of
document data (e.g., MD5) Crawler traps (e.g., CGI scripts that link to
themselves using a different name) May need to have a way for human to override
certain URL paths – see Section 5 of paper
Mercator Document Statistics
PAGE TYPE PERCENTtext/html 69.2%image/gif 17.9%image/jpeg 8.1%text/plain 1.5%pdf 0.9%audio 0.4%zip 0.4%postscript 0.3%other 1.4%
Histogram of document sizes
(60M pages)
9
Further Considerations
May want to prioritize certain pages as being most worth crawling Focused crawling tries to prioritize based on
relevance
May need to refresh certain pages more often
10
Web Search Summarized
Two important factors: Indexing and ranking scheme that allows most
relevant documents to be prioritized highest Crawler that manages to be (1) well-
mannered, (2) avoid traps, (3) scale
We’ll be using Pastry to distribute the work of crawling and to distribute the data (what Google calls “barrels”)
11
We Need More Than Synchronization
What needs to happen when you… Click on “purchase” on Amazon?
Suppose you purchased by credit card?
Use online bill-paying services from your bank? Place a bid in an eBay-like auction system? Order music from iTunes?
What if your connection drops in the middle of downloading?
Is this more than a case of making a simple Web Service (-like) call?
12
Transactions Are a Means of Handling Failures
There are many (especially, financial) applications where we want to create atomic operations that either commit or roll back
This is one of the most basic services provided by database management systems, but we want to do it in a broader sense
Part of “ACID” semantics…
13
ACID Semantics
Atomicity: operations are atomic, either committing or aborting as a single entity
Consistency: the state of the data is internally consistent
Isolation: all operations act as if they were run by themselves
Durability: all writes stay persistent!
14
A Problem Confronted by eBay
eBay wants to sell an item to: The highest bidder, once the auction is over, or The person who’s first to click “Buy It Now!”
But: What if the bidder doesn’t have the cash?
A solution: Record the item as sold Validate the PayPal or credit card info with a 3rd
party If not valid, discard this bidder and resume in prior
state
15
“No Payment” Isn’t the Only Source of Failure
Suppose we start to transfer the money, but a server goes down…
Purchase:sb = Seller.balbb = Buyer.balWrite Buyer.bal= bb - $100
Write Item.sellTo = Buyer
Write Seller.bal= sb + $100
CRASH!
16
Providing Atomicity and Consistency
Database systems provide transactions with the ability to abort a transaction upon some failure condition Based on transaction logging – record all
operations and undo them as necessary
Database systems also use the log to perform recovery from crashes Undo all of the steps in a partially-complete
transaction Then redo them in their entirety This is part of a protocol called ARIES
17
The Need for Isolation
Suppose eBay seller S has a bank account that we’re depositing money into, as people buy:
What if two purchases occur simultaneously, from two different servers on different continents?
S = Accounts.Get(1234)Write S.bal = S.bal + $50
18
Concurrent Deposits
This update code is represented as a sequence of read and write operations on “data items” (which for now should be thought of as individual accounts):
where S is the data item representing the seller’s account # 1234
Deposit 1 Deposit 2Read(S.bal) Read(S.bal)S.bal := S.bal + $50 S.bal:= S.bal + €10Write(S.bal) Write(S.bal)
19
A “Bad” Concurrent Execution
Only one action (e.g. a read or a write) can actually happen at a time for a given database, and we can interleave deposit operations in many ways:
Deposit 1 Deposit 2Read(S.bal) Read(S.bal)S.bal := S.bal + $50 S.bal:= S.bal + €10Write(S.bal) Write(S.bal)
time
BAD!
20
A “Good” Execution
Previous execution would have been fine if the accounts were different (i.e. one were S and one were T), i.e., transactions were independent
The following execution is a serial execution, and executes one transaction after the other:
Deposit 1 Deposit 2Read(S.bal) S.bal := S.bal + $50 write(S.bal) Read(S.bal) S.bal:= S.bal + $10 Write(S.bal)
time
GOOD!
21
Good Executions
An execution is “good” if it is serial (transactions are executed atomically and consecutively) or serializable (i.e. equivalent to some serial execution)
Equivalent to executing Deposit 1 then 3, or vice versa Why would we want to do this instead?
Deposit 1 Deposit 3read(S.bal) read(T.bal)S.bal := S.bal + $50 T.bal:= T.bal + €10write(S.bal) write(T.bal)
22
Concurrency Control
A means of ensuring that transactions are serializable
There are many methods, of which we’ll see one Lock-based concurrency control (2-phase
locking) Optimistic concurrency control (no locks –
based on timestamps) Multiversion CC …