Population control for multi-agent based topicalcrawlers

Alban MoutonVALORIA

Universite de Bretagne SudUniversite Europeenne de Bretagne

Vannes, FranceEmail: alban.mouton@univ-ubs.fr

Pierre-Francois MarteauVALORIA

Universite de Bretagne SudUniversite Europeenne de Bretagne

Vannes, FranceEmail: pierre-francois.marteau@univ-ubs.fr

Abstract—The use of multi-agent topical Web crawlers basedon the endogenous fitness model raises the problem of controlingthe population of agents. We tackle this question through anenergy based model to balance the reproduction/life expectencyof agents. Our goal is to simplify the tuning of parameters andto optimize the use of ressources available for the crawling.We introduce an energy based model designed to control thenumber of agents according to the precision of the crawling.We present some experiments that show that the size of thepopulation remains under control during the crawling.

I. INTRODUCTION

Population based multi-agent models for focused crawlingon the Web have been quite intensively introduced in previousworks [1]. Most systems based on artificial life involve pop-ulations of agents whose evolution is difficult to control: theconsequence is mainly an increase of required ressources forefficient processing.

The aim of this article is to propose automatic regulationtechniques for the endogenous fitness model applied to focusedcrawling on the Web. Firstly, we re-define the notion ofenergy in the scope of the endogenous fitness model: themain objectives are to simplify the parameters of the modeland to automatically preserve a viable population of agents.Secondly, we introduce a concept of energy conservation oncrawled pages that improves our control of the size of thepopulation of agents and its activity. We, then, present someexperiments that detail the behaviour of our model and showits effectiveness.

II. RELATED WORKS

A. Focused crawling and artificial life

The notion of focused crawling, also known as topical ortopic-driven crawling, was first introduced by Chakrabarti &al. [2] to refer to Web crawlers specialized in topic-specificinformation discovery. Focused crawling raises many inter-esting topics. For example, it generally leans heavily on thevast field of automatic text processing initiated by Salton [3]for topical judgments. Automatic extraction of link contextsin semi-structured documents is another recurrent problemaddressed notably by Pant & al. [4]. The study of the topologyof the topical graphs in the Web has consequences on focused

crawling and free exploration of Web pages or exploitation ofalready discovered resources [5]. Interesting works attempt toformulate the numerous variants involved in the design of thiskind of system and to propose suited evaluation frameworksand metrics to research teams working in the field [6] [7].

Quite early, some artificial life models were studied fordevelopping efficient topical Web crawlers. Firstly, Menczer& al. [8] proposed to apply the endogenous fitness artifi-cial life model to focused crawling. They also studied theinteresting association of multi-agent models with contextualmachine learning mechanisms and showed that adaptivity ofagents could scale up to the size of the environment throughcloning and deployment of the population [1]. This workled to the projects InfoSpiders and MySpiders [9]. Otherapproaches based on different models were proposed, some inparticular were built upon the ant colony paradigm [10] [11].Kushchu [12] proposed a survey of evolutionary and adaptiveapproaches applied to Information Retrieval on the Web.

B. The core of the original Endogenous Fitness model forfocused crawling

In this paper, we focus on the endogenous fitness modelapplied to focused crawling as described by Filippo Menczer[8]. The core of this model is presented in Algorithm 1. In thismulti-agent model, agents independently visit Web pages andpick up new links searching for relevant information. Agentsclone themselves or die according to their success that isevaluated by converting relevant information into energy.

III. NEW STRATEGY FOR POPULATION CONTROL IN THEENDOGENOUS FITNESS MODEL

A. Energy balance

Controlling the endogenous fitness model through its energyparameters is a quite difficult task to achieve. Depending onthe values of these parameters the population of agents canundergo important variations of size that make resource con-sumption unpredictable. Excessive variations of the populationof agents can also decrease the quality of the results becausetoo many agents may imply a dispersed and insufficientlyselective crawler.

Eighth International Conference on Intelligent Systems Design and Applications

978-0-7695-3382-7/08 $25.00 © 2008 IEEE

DOI 10.1109/ISDA.2008.111

133

Algorithm 1 Endogenous fitness based crawlerInitialize system with parameters:- energy cost and gain rate: C and ρ- number of agents: N- seeds pages: S- query QInitialize N agents with energy E = 1 and current documentD in Sloop

For each agent alive:D = Pick link in DE = E − C + Similarity(Q,D) ∗GApply machine learning technique using D, Q andSimilarity(Q,D) (optional)if E > 1 then

Clone agent and share E between cloneselse if E < 0 then

Death of agentend if

end loop

The idea we suggest is to dynamically estimate relevancevalues of the crawled pages and to try to establish a relationbetween relevance and energy to ensure a better stability ofthe population of agents.

We use a simple tf*idf similarity model in order to evaluatepage relevance according to a topical request, but indeed otherand richer relevance models, that are not in the scope of thispaper, can be used. Independently from the used similaritymodel the endogenous fitness model requires relevance valuesto be converted into energy in order to control the activity ofthe crawler.

The original energy model [8] can be written as the follow-ing equation:

∆e = f (r) = ρ · r − C

Where:• r is the relevance of the crawled page. ∆e is the energy

update endorsed by the crawler.• C is the cost in energy for the visit of a single page by

an agent. It contributes to determine how selective thecrawler is related to the success of its agents and howtolerant it is to the traversal of minimal relevance pages.

• ρ is the energy gain rate used to convert relevant infor-mation in energy. The range of ρ depends on the order ofthe similarity metric used, and on the concrete relevanceof collected pages.

To better regulate the population of agents we would like tobe able to control the reproduction rate and we have chosento introduce a constant gain parameter that defines the energygain for top relevant pages. Sigmoidal functions are often usedin artificial life systems to model state transitions. An energyvariation function inspired by a sigmoid function allows bothcost and gain normalization for extreme values. Figure 1 showshow the energy varies according to our sigmoidal model.

Fig. 1. Sigmoidal function between textual relevance of the documents andenergy variations

∆e = f (r,R) =(

(G+ C) · 11 + e−λ(r+R)

)− C

Energy balance in the system when n pages have beencollected is ensured if the following constraint is satisfied:

n∑i=0

f (ri, R) = 0

where ri is the relevance of the ith visited page and R isto be determined.

To calculate a satisfying R we use an approximation off (r,R).

We define g (r,R) as:

r ≤ R→ g (r,R) = −C

r > R→ g (r,R) = G

For a high value of λ the slope in the intermediate stateof the sigmoid function is greater and over a large number ofvalues g (r,R) is a satisfying approximation of f (r,R).

We can use a probabilistic interpretation to get the expectedvalue of ∆e.

E(∆e) =∫r≤R

f(r,R) · p(r) · dr +∫r>R

f(r,R) · p(r) · dr

Where p(r) is the probability density that the collected pagehas a relevance of r.

When approximating f by g we get:

E(∆e) '∫r≤R−|C| · p(r) · dr +

∫r>R

G · p(r) · dr

E(∆e) ' −|C| · Prob(r ≤ R) +G · Prob(r > R)

Where Prob is the distribution associated to p. Prob(r >R) is the probability that a page’s relevance is higher than R.

We seek a null mathematical expectation for ∆e:

E(∆e) ' 0

Therefore:Prob(r > R)Prob(r ≤ R)

=G

|C|To estimate a satisfying value for R we sort the previously

visited pages by their values of relevance. Let n be the total

134

number of pages and n∗ be the rank of the page whoserelevance is the closest to the ideal R.

We can use n∗ and n to estimate Prob(r ≤ R) andProb(r > R):

Prob(r ≤ R) ' n∗

n

Prob(r > R) ' n− n∗

n

Prob(r > R)Prob(r ≤ R)

' n∗

n− n∗

n∗

n− n∗' G

|C|

n∗ ' n

C/G+ 1

In practice, we calculate n∗, on the basis of the history ofthe n collected pages, then we set R = rn∗

In order for the approximation of f (r,R) by g (r,R) tomake sense we need to maintain a sufficiently large value forλ.

Through experience we defined h (rm, rM , R), where rm isthe smallest and rM the greatest relevance values respectively,such as:

8R− rm

≤ rM −R→ h (rm, rM , R) =8

R− rm

8R− rm

> rM −R→ h (rm, rM , R) =8

rM −R

We use λ = h (rm, rM , R).The user’s parameters at this point are C and G. They allow

to tune the tolerance of the crawler to the traversal of lowrelevance zones, the reactivity of the agents to highly relevantpages and indirectly the selectivity of the crawler. In practice,a value of C close to 0 means that the crawler tolerates that itsagents traverse many low relevance pages before dying. On thecontrary, a value of C greater than 1 ensures that every agentwho visited a low relevance page dies instantly. If G is set to10 then an agent visiting a very relevant page creates aboutten clones of himself. If C is set to 0.5 and G to 10 then aboutone page in twenty is considered relevant by the system andrewarded by an energy gain.

While experimenting with the crawler, studying the graphof the Web, and more specifically topical sub-graphs of theWeb, it is possible to define static values for these parametersthat should offer satisfying results for all use cases. Thiswould provide an auto-adaptive system and leave the crawlingprocess unsupervised.

B. Energy conservation

In order for the crawler to be highly selective and reactiveto the discovery of relevant information it is necessary to usequite high values of the energy parameter G. Even if ourglobal energy balance computation provides the main stabilityof the system, it does not prevent coarse variations of thepopulation of agents. We want to dissociate in time energydiscovery variations and population variations by implement-ing an energy buffer on the pages. If the number of agents istoo high the energy of relevant pages is not used and is leftbehind for future use. Conversely when the number of agentsis low, dying agents are ”teleported” at no cost on pages withenergy remnants. To not leave some pages behind for too longwith energy remnants the stochastic process is modified andincludes random walk toward these pages. The modified modelis described in algorithm 2.

IV. EXPERIMENTS

A. Experimental settings

Although frameworks exist for topical crawling evaluation(e.g. Srinivasan & al. [7]), they are not fully adapted to thespecific scope of this paper. Instead we designed the followingdedicated set of experiments.

To validate the regulating effect of the sigmoidal model andthe energy buffer and to study the influence of the parameters,we ran the crawler many times, with varying models, topicsand parameter values.

The target topics are described as simple text queries thatare manually created by stripping introduction paragraphs ofwikipedia pages of irrelevant terms. For each topic we manu-ally pick as seeds, one, two or three relevant pages returned bygoogle. Link contexts in the pages are extracted according tothe recommendation of Pant & al. [4]. Our prototype uses thetf*idf metric to evaluate the relevance of crawled pages and oflink contexts. The idf values are estimated before initializationby using the number of results returned by google queries onthe keywords of the topic. And finally, the tf*idf values arealso used to draw the precision curves.

B. Impact of the new energy function

Figure 2 shows the variations of population during thetime of three crawlers based on the original energy model.Those crawlers were ran five times each and differ only onthe energy gain rate parameter. The curves clearly illustratethat depending on this parameter the population of agents caneither die very quickly, increase brutally or remain usable.

Figure 3 shows the variations of population during the timeof three other crawlers based on the new sigmoidal energymodel. These crawlers were also ran five times with differentvalues of C and G. In the three cases, the population of agentsincrease in time because of the corresponding increase of pre-cision of the system, however, unlike in Figure 2 the evolutionof the population does not variate drastically according tothe user’s parameters. Therefore, population variations remaincontrollable and coherent with the activity of the crawler and

135

Algorithm 2 New model with energy conservationThe crawler:Initialize system with parameters:- maximal energy cost and gain : C and G- initial number of agents : N- upper and lower boundaries of the number of agents : UBand LB- seeds pages : S- query QInitialize documents base DB (empty or from precedentexecution)Initialize N agents with energy E = 1 and current documentD in Sloop

For each agent alive :Pick link in D or in DB {only documents with energyremnants can be picked in DB}Fetch D from the Web or from DBupdate energy()reproduce()

end loop———————————————————————–Agent.update energy():r ⇐ content relevance(D,Q)if D not in DB thenDB.set document(D,r)R⇐ DB.get R()DB.set doc energy(D,energy function(G,C,R,r))

end ifif System.number of agents() > UB thenE ⇐ E − ‖C‖

elseE ⇐ E +DB.get document energy()DB.set doc energy(D, 0)

end if———————————————————————–Agent.reproduce():if E > 1 then

Create bEc clones with E ÷ dEe energyE ⇐ E ÷ dEe

else if E < 0 thenif System.number of agents() < LB then

Force to pick D in DB at next iterationelse

die()end if

end if

Fig. 2. Variations of population for various parameters of the original model

the energy buffer will be able to erase local variations of thepopulation.

Fig. 3. Variations of population for various parameters of the sigmoidalmodel

Fig. 4. Comparison of the precision of the two models

Figure 4 shows the precision results of two crawlers alreadyused in Figures 2 and 3. The first one is based on the simpleoriginal energy model while the other one uses the newsigmoidal energy model. Values of C are identical, but thecrawler based on the new model uses a value of G set to8. Figures 2 and 3 show that the two crawlers have a quitesimilar evolution of their population in time, but the differenceof precision seen in Figure 4 clearly illustrates the positiveimpact of the introduction of parameter G.

136

C. Impact of energy conservationWe have implemented and tested the energy buffer described

in section III.C. The variations of the size of the populationsof agents for two crawlers with or without the energy conser-vation buffer are shown in Figure 5.

Fig. 5. Effect of energy conservation on the variations of population in thesystem

To illustrate the effectiveness of the energy buffer thosecrawlers were initialized with a high value for G set to 30,implying brutal variations of the population over long crawlsof 25000 pages. The energy conservation buffer is effectiveand the size of the population is clearly bounded by the valuesgiven by the user. We can see that the size of the populationtends to be equal most of the time to the upper boundary (250here), that is because the precision of the system increasesin time. In the case of longer or more difficult crawlings forwhich the whole accessible topical graph has been visited, theprecision decreases and the size of the population tends to beequal to the lower boundary.

Figure 6 shows the precision for the same crawlers as thosepresented in figure Figure 5. We can see that the precision ofthe crawler with energy conservation is a little lower than itscounterpart. The use of the energy buffer introduces a delayin the scanning of relevant pages: the difference of precisionis progressively reduced to zero.

Fig. 6. Effect of energy conservation on the precision of the system

V. CONCLUSION

We have introduced some improvements to the endogenousfitness model for focused crawling to gain a better control of

the size and activity of the agent-based crawler. Experimentalresults show that we indeed regulate quite well the crawler’spopulation and improve its efficiency thanks to the proposedenergy management strategy. The control parameters we in-troduced are easy to use and their effects are easy to predictThe previous results are encouraging as a preliminary steptoward an efficient and practical multi-agent crawler for Webinformation retrieval.

REFERENCES

[1] F. Menczer and R. K. Belew, “Adaptive retrieval agents: Internalizinglocal context and scaling up to the web,” Machine Learning, vol. 39,no. 2/3, pp. 203–242, 2000.

[2] S. Chakrabarti, M. van den Berg, and B. Dom, “Focused crawling: a newapproach to topic-specific Web resource discovery,” Computer Networks(Amsterdam, Netherlands: 1999), vol. 31, no. 11–16, pp. 1623–1640,1999.

[3] G. Salton, Automatic Text Processing – The Transformation, Analysis,and Retrieval of Information by Computer. Addison–Wesley, 1989.

[4] G. Pant and P. Srinivasan, “Link contexts in classifier-guided topicalcrawlers,” IEEE Trans. on Knowl. and Data Eng., vol. 18, no. 1, pp.107–122, 2006.

[5] G. Pant, P. Srinivasan, and F. Menczer, “Exploration versus exploitationin topic driven crawlers,” 2002.

[6] ——, “Crawling the web,” in Web Dynamics, 2004, pp. 153–178.[7] P. Srinivasan, F. Menczer, and G. Pant, “A general evaluation framework

for topical crawlers,” Inf. Retr., vol. 8, no. 3, pp. 417–447, 2005.[8] F. Menczer, R. K. Belew, and W. Willuhn, “Artificial life applied to

adaptive information agents,” in AAAI Spring Symposium on InformationGathering, 1995.

[9] G. Pant and F. Menczer, “Myspiders: Evolve your own intelligent webcrawlers,” 2002.

[10] A. Revel, “Web-agents inspired by ethology: A population of ”ant”-likeagents to help finding user-oriented information,” in WI ’03: Proceedingsof the 2003 IEEE/WIC International Conference on Web Intelligence.Washington, DC, USA: IEEE Computer Society, 2003, p. 482.

[11] F. Gasparetti and A. Micarelli, “Adaptive web search based on a colonyof cooperative distributed agents,” in Cooperative Information Agents,S. O. A. L. H. Klusch, M.; Ossowski, Ed., vol. 2782. Springer-Verlag,2003, pp. 168–183.

[12] I. Kushchu, “Web-based evolutionary and adaptive information re-trieval,” IEEE Trans. Evolutionary Computation, vol. 9, no. 2, pp. 117–125, 2005.

137