International Journal of Hybrid Computational IntelligenceVolume 4 • Numbers 1-2 • January-December 2011 • pp. 21-26

ViDE: A VISION-BASED APPROACH FOR DEEP WEBDATA EXTRACTION USING VIPSM. Santhosh1

1Department of CSSE, College of Engineering, AndhraUniversity, Visakhapatnam.

Abstract: Deep Web contents are accessed by queries submitted to Web databases and the propose a newevaluation measure revision to capture the amount of human effort needed to returned data records areenwrapped in produce perfect extraction. Our experiments on a structured data from deep Web pages is achallenging problem due to the underlying intricate structures of such pages. Until now, a large number oftechniques have been proposed to address this problem, but all of them have inherent limitations becausethey are Web-page-large set of Web databases show that the proposed vision-based approach is highly effectivefor deep Web data extraction.

Keywords: Web mining, Web data extraction, visual features of programming-language dependent. As thedeep Web pages, wrapper generation popular two-dimensional media, the contents on Web pages are alwaysdisplayed regularly for users to browse. This motivates us to seek a different way for deep Web data extractionto overcome the limitations of previous works by utilizing some interesting common visual features on thedeep Web pages. In this paper, a novel.

1. INTRODUCTIONWe survey the state of art in unsupervised web dataextraction to provide a foundation for developingnext-generation extraction tools. We describe thetoday’s existing approaches in terms of a three-phaseframework, consisting of the identification of thedata-rich area, the identification of the individualvision-based approach that is Web-page records,The alignment of the data in the records programming-language-independent is proposed.

This approach primarily utilizes the visualfeatures on the deep Web pages to implement deepWeb data extraction, including data recordextraction and data item extraction. We also usingthis reference framework, we extract the buildingblocks present in these tools and categorize theminto dealing with the encoded structure, linguisticproperties, the visual structure, and the ontologicalstructure.

The performance of each individual approachcrucially depends on whether its assumptions muchthe result pages to be analyzed and as theseassumptions are so far hard coded into thealgorithms performing the extraction, it is hard toadapt any of these tools for a given site, and toincorporate domain knowledge which goes beyond

ontologies, as used in. The approach we take: Toalleviate this situation, we take a rule-basedapproach to unsupervised data extraction. Since wewant to build on the work done so far in the _eld,we analyzed a spectrum of scholarly described tools,to explicitly state and systematize their underlyingassumptions.

2. RELATED WORKThis chapter first describes necessary componentsthat are not the main contributions of the thesis yetare important components of the proposed methodfor described relationship-based ranking ofdocuments. These components are a populatedontology, semantic annotation of documentcollection to identify the named entities from theontology, indexing and retrieval based on keywordinput from user.

That is, not just the schema of the ontology is ofparticular interest, but also the population(instances, assertions or description base) of theontology. A highly populated ontology (ontologywith instances or assertions) is critical forassessing effectiveness, and scalability ofcore semantic techniques such as semanticdisambiguation.

22 International Journal of Hybrid Computational Intelligence

Large Populated Ontologies reasoning, anddiscovery techniques. Ontology population has beenidentified as a key enabler of The development ofSemantic Web practical semantic applications inindustry; for applications typically involvesprocessing of data example, Semagix reports thatits typical represented using or supported byontologies. An ontology is a specification of aconceptualization. yet the value of ontologies is inthe agreement they are intended to provide(for humans, and/or machines). In the SemanticWeb, ontology can be viewed as a vocabulary usedto describe a world modelcond, commerciallydeveloped ontologies have over one million objects.Another important factor related to the populationof the ontology is that it should be possible to captureinstances that are highly connected (i.e., theknowledge base should be deep with many explicitrelationships among the related previous work isinstances).

A populated ontology is one that containsSWETO Ontology not only theschemaor definitionof the classes/concepts and relationship names butalso a we now review our earlier work for largenumber of entities that constitute the instancebuilding a test-bed ontology, called SWETOpopulation of the ontology.

(Semantic Web Technology EvaluationOntology). SWETO has demonstrated that largepopulated ontologies can be built from dataextracted from a variety of Web sources. We havefound that the richness and diversity of relationshipswithin an ontology is a crucial aspect. SWETOcaptures real world knowledge with over 40 classespopulated with a growing set of relevant facts,currently at about one million instances. The schemawas created in a bottom-up fashion where the data

sources dictate the classes and relationships. Theontology was created using Semagix Freedom, acommercial product which evolved from the LSDISlab’s past research in semantic interoperability andthe SCORE technology. The Freedom toolkit allowsfor the creation of an ontology, in which a user candefine classes and the relationships that it is involvedin using a graphical environment.

We selected as data sources highly reliable Websites that provide instances in a semi-structuredformat, unstructured data with structures easy toparse (e.g., html pages with tables), or dynamic siteswith database back-ends. In addition, the Freedomtoolkit has useful capabilities for focused crawlingby exploiting the structure of Web pages anddirectories. We carefully considered the types andquantity of relationships available in each data sourceby preferring those sources in which instances wereinterconnected. We considered sources whoseintervals. Automatic data extraction and insertion intoinstances would have rich metadata.

Figure 1: Overview of Data Sources for InstancePopulation of SWETO Ontology

All knowledge (or facts that populate theontology) was extracted using Semagix Freedom.Essentially, extractors were created within theFreedom environment, in which regular expressionsare written to extract text from standard html, semi-structured (XML), and database-driven Web pages.Additionally, provenance information, includingsource, time and date of extraction, etc., ismaintained for all extracted data. We later utilizeFreedom’s API for exporting both the ontology andits instances into one of the semantic web

Vide: A Vision-Based Approach for Deep Web Data Extraction Using VIPS 23

representation languages (e.g., RDF). For keepingthe knowledge base up to date, the extractors canbe scheduled to rerun at user specified time and datea knowledge base also raise issues related to thehighly researched area of entity disambiguation. InSWETO, we focused greatly on this aspect ofontology population.

Using Freedom, instances can be disambiguatedusing syntactic matches and similarities (aliases),customizable ranking rules, and relationshipsimilarities among instances. Freedom is thus ableto automatically disambiguate instances as they areextracted Error! Reference source not found.but if itdetects ambiguity among new instances and thosewithin the knowledge base, yet it is unable todisambiguate them within a preset degree ofcertainty, the instances are flagged for manualdisambiguation.

We could not locate the website of a small(http://lsdis.cs.uga.edu/projects/semdis/) builds numberof (arguably local or out of business) publishers. Weassigned them an arbitrary URI using the“example.org” domain name as prefix. A lookupoperation on the respective datasets is in most casesthe key to establish relationships that enrichSwetoDblp. SwetoDblp is publicly available fordownload together with additional datasets thatwere usedforitscreation http://lsdis.cs.uga.edu/projects/semdis/swetodblp/) and other details are also availableError! Reference source not found.

The SemDIS project at the LSDIS Lab upon thevalue of relationships on the Semantic Web. A keynotion to process relationships between entities isthe concept of semantic associations, which are thedifferent sequences of relationships that interconnecttwo entities; semantic associations are based onintuitive notions such as connectivity and semanticsimilarity. Each semantic association can be viewedas a simple path consisting of one or morerelationships, or, pairs of paths in the case ofsemantic similarity. Figure 3 illustrates a small graphof entities and the results of a query for semanticassociations taking two of them as input.

Most useful semantic associations involveDiscovery, Analysis and Ranking of Relationshipssome intermediate entities and associations.

A key element present in Semantic Web is thatof relationships, which a first-class object in RDF is.Relationships provide the context (or meaning) ofentities, depending on how they are interpreted

and/or understood. The value relies on the fact thatthey are named relationships. That is, they refer to a‘type’ defined in ontology. Relationships will playan important role in the continuing evolution of theWeb and it has been argued that people will use websearch not only for documents, but also forinformation about semantic relationships.

Relationships that span several entities may bevery important in domains such as national security,because this may enable analysts to see theconnections between disparate people, places andevents. In fact, applications that utilized the conceptof semantic associations include search of biologicalterms in patent databases, provenance and trust ofdata sources, and national security. The applicabilityof semantic associations in my research comes fromthe need to analyze relationships.

The type of operations needed to discoversemantic associations involve graph-basedtraversals. It has been noted that graph-basedalgorithms help analysts of information tounderstand relationships between the variousentities participating in events, activities, and so on.The underlying technical challenge is also relatedto the common connecting- the-dots applicationsthat are found in a broad variety of fields, includingregulatory compliance, intelligence and nationalsecurity and drug Data records and data items in

Figure 2: Example of Relationships in SwetoDblp Entities

Figure 3: Example Semantic Associations from a Small Graph

24 International Journal of Hybrid Computational Intelligence

them discovery. Additionally, techniques that usesemantic associations have been applied for Peer-to-Peer (P2P) discovery of data and knowledgeaggregation. For example, a P2P approach wasproposed to make the discovery of knowledge moredynamic, flexible, and scalable Since different peersmay have knowledge of related entities andrelationships, they can be interconnected in orderto provide a solution for a scientific problem and/or to discover new knowledge by means ofcomposing knowledge of the otherwise isolatedpeers. ranking” is presented to rank queries returnedwithin semantic Web portals. Their techniquereinterprets query results as “query knowledge-bases”, whose similarity to the original knowledge-base provides the basis for ranking. The actualsimilarity between a query result and the originalknowledge-base is derived from the number ofsimilar super classes of the result and the originalknowledge-base. In our approach, the relevancy ofresults usually depends on a context defined byusers.

Modules:

1. Web crawling and Meta searching

2. Web data record and item Extraction

3. Visual Wrapper generation

4. Precision and recall

1. Web crawling and Meta searching

Ranking of semantic associations has beenaddressed by our colleagues taking the approach ofmachine process able, which is needed in manyapplications such as deep Web crawling and metasearching, the structured data need to be extractedfrom the deep Web pages. Each data record on thedeep Web pages corresponds to an object. Web datarecord and item Extraction.

Data record extraction aims to discover theboundary of data records and extract them from thedeep Web pages. An ideal record extractor shouldachieve the following: 1) all data records in the dataregion are extracted and 2) for each extracted datarecord, no data item is missed and no incorrect dataitem is included.

2. Visual Wrapper generationLetting the user choose among discovery mode andThe complex extraction processes are too slow inconventional mode of discovery/ranking of

supporting real-time applications. Second, therelationships. They considered rare vs. commonextraction processes would fail if there is only oneappearances of relationships in a populatedontology data record on the page. Since all deepWeb pages Research in the area of ranking semanticfrom the same Web database share the same visualrelations also includes, where the notion of“semantic template, once the data records and dataitems on a deep Web page have been extracted, wecan use these they be significantly revised or evenabandoned? Or extracted data records and dataitems to generate the should other approaches beproposed to extraction wrapper for the Webdatabase so that new accommodate the newlanguages? Second, they are deep Web pages fromthe same Web database can be incapable of handlingthe ever-increasing complexity processed using thewrappers quickly without of HTML source code ofWeb pages. Most previous reapplying the entireextraction process. works have not considered thescripts.

3. Precision and recall JavaScript and CSS, in theHTML files. In order to

The basic idea of our vision-based data item makesWeb pages vivid and colorful, Web page wrapperis described as follows: Given a sequence ofattributes Explanation for (f; l; d) fa1; a2; . . . ; angobtained from the sample page and a sequence ofdata items fitem1; item2; . . . ; item m obtained froma new data record, the wrapper processes the dataitems in order to decide which attribute the currentdata item can be matched to. For item i and aj, ifthey are the same on f, l, and d, their match isrecognized. The wrapper then judges whetheritemiþ1 and ajþ1 are matched next, and if not, itjudges item i and ajþ1. Repeat this process until alldata items are matched to their right attributes.

2.1. Existing SystemThe problem of Web data extraction has received alot of attention in recent years and most of theproposed solutions are based on analyzing theHTML source code or the tag trees of the Web pages.These solutions have the following main limitations:First, designers are using more and more complexJavaScript and CSS.. This makes it more difficult forexisting solutions to infer the regularity of thestructure of Web pages by only analyzing the tagstructures.

Vide: A Vision-Based Approach for Deep Web Data Extraction Using VIPS 25

3. PROPOSED SYSTEMIn this paper, we explore the visual regularity ofthe data records and data items on deep Web pagesand propose a novel vision-based approach, Vision-based Data Extractor (ViDE), to extract structuredresults from deep Web pages automatically. ViDEis primarily based on the visual features humanusers can capture on the deep Web pages while alsoUtilizing some simple nonvisual information suchas data types and frequent symbols to make thesolution more robust. ViDE consists of two maincomponents, Vision based Data Record extractor(ViDRE) and Vision-based Data Item extractor(ViDIE). By using visual features for dataextraction, ViDE avoids the they are Web-page-programming-language limitations of thosesolutions that need to analyze dependent, or moreprecisely, HTML-dependent. As most Web pagesare written in HTML, it is not surprising that allprevious solutions are based on analyzing theHTML source code of Web pages. Furthermore,HTML is no longer the exclusive Web pageprogramming language, and other languages havebeen introduced, such as XHTML and XML(combined with XSLT and CSS). The previoussolutions now face the following dilemma: shouldcomplex Web page source files. Our approachemploys a four-step strategy. First, given a sampledeep Web page from a Web database, obtain itsvisual representation and transform it into a VisualBlock tree which will be introduced later; second,extract data records from the Visual Block tree;third, partition extracted data records into dataitems and align the data items of the same semantictogether; and fourth, generate visual wrappers (aset of visual extraction rules) for the Web databasebased on sample deep Web pages for new deepWeb pages that are from the same Web databasecan be carried out more efficiently using the visualwrappers.

4. CONCLUSIONThis work promotes a new architecture for timeseries prediction, tackling recently arising challengesof a generally increasing volume of time series dataexhibiting complex non-linear relationships betweenadopts the proposed architecture. These conclusionswere valid across many different time series, whichgives an inspiration to use the proposed architectureas a guidance to define a general framework forbuilding a combined prediction system.

5. FUTURE WORKAs we know that our website support based ondifferent urls we are getting the same domains,books etc.To reduce the duplicates our presentsystem is very useful. So, mainly based on URL wehave to its multidimensional features and outputs.It reduce the duplicates. This is very useful becausewe combines a multilevel architecture of highlyrobust and diversified individual prediction modelswith operators for fusion and selection that can beapplied at any level of the structure. This article hasbeen accepted for publication in a future issue ofthis journal, but has not been fully edited. Contentmay change prior to final publication. Additionally,the system applies an intelligent smoothingalgorithm as an example of the post-prediction stepthat often leads to significant performance gains,particularly if the predicted time series contains asignificant noise component. The model buildingprocess is supported by the simple, yet effective,greedy feature generation method. The predictedoutput signal is further validated using an originalsmoothing technique to remove excessive noise. Theproposed model has competed in two internationalcompetitions for time series prediction and hasfurthermore been compared with a number ofstandard individual time series forecasting andforecast combination algorithms. It was the winnerof the NISIS Competition 2006 leaving the second-best model with twice as large an error rate, and itwas ranked among the top models in the muchbigger NN3 Forecasting Competition 2006/2007.The results also showed that an ensemble ofindividual models can perform much better if itdon’t gather the same information from differenturls. Actually this is time wasting process. To solvethis problem, we are using this present system.

REFERENCES[1] G.O. Arocena and A.O. Mendelzon, “WebOQL:

Restructuring Documents, Databases, and Webs,”Proc. Int’l Conf. Data Eng. (ICDE), 1998, pp. 24-33.

[2] D. Buttler, L. Liu, and C. Pu, “A Fully AutomatedObject Extraction System for the World Wide Web,”Proc. Int’l Conf. Distributed Computing Systems(ICDCS), 2001, pp. 361-370.

[3] D. CAI, X. He, J.-R. Wen and W.-Y. Ma, “Block-LevelLink Analysis,” Proc. SIGIR, 2004, pp. 440-447.

[4] D. CAI, S. Yu, J. Wen, and W. Ma, “Extracting ContentStructure for Web Pages Based on VisualRepresentation,” Proc. Asia Pacific Web Conf.(APWeb), 2003, pp. 406-417.

26 International Journal of Hybrid Computational Intelligence

[5] C.-H. Chang, M. Kayed, M.R. Girgis, and K.F.Shaalan, “A Survey of Web Information ExtractionSystems,” IEEE Trans. Knowledge and Data Eng., Vol.18, No. 10, 2006, pp. 1411-1428, Oct.

[6] C.-H. Chang, C.-N. Hsu and S.-C. Lui, “AutomaticInformation Extraction from Semi-Structured WebPages by Pattern Discovery”, Decision Support Systems,Vol. 35, No. 1, 2003, pp. 129-147.

[7] V. Crescenzi and G. Mecca, “Grammars HaveExceptions,” Information Systems, Vol. 23, No. 8, 1998,pp. 539-565.

[8] V. Crescenzi, G. Mecca, and P. Merialdo,“Roadrunner: Towards Automatic Data Extractionfrom Large Web Sites,” Proc. Int’l Conf. Very LargeData Bases (VLDB), 2001, pp. 109-118.