ApacheCon 2000 Everything you ever wanted to know about XML Parsing

Everything you always wanted to know about XML parsing

Ted LeungCTO, Enkubator, LLCted@enkubator.com

Thank Youn IBMn ASFn Xerces-Dev

Outlinen Overviewn xml.apache.org projectn Basic XML Conceptsn SAX 1 / 2 n DOM Parsingn Namespacesn XML Scheman Grammar Accessn Round Trippingn Grammar Designn Performance tipsn Xerces Architecture

Overviewn Focus on server side XMLn Not document processingn Benefits to servers / e-business

n Model-View separation for *MLn Ubiquitous format for data exchange

n Hope for network effects from data availability

xml.apache.orgn Xerces XML parser

n Javan C++n Perl

n Xalan XSLT processorn Javan C++ (soon)

n FOPn Cocoon

Xerces-Jn Why Java?

n Unicoden Code motionn Server side cross platform support

n C++ version has lagged Java version

Basic XML Conceptsn Well formednessn Validity

n DTD’sn Schemas

n Entities

Example: RSS<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE rss PUBLIC "-//Netscape Communications//DTD RSS .91//EN""http://my.netscape.com/publish/formats/rss-0.91.dtd">

<channel><title>freshmeat.net</title><link>http://freshmeat.net</link><description>the one-stop-shop for all your Linux software needs</description><language>en</language><rating>(PICS-1.1 "http://www.classify.org/safesurf/" 1 r (SS~~000 1))</rating><copyright>Copyright 1999, Freshmeat.net</copyright><pubDate>Thu, 23 Aug 1999 07:00:00 GMT</pubDate><lastBuildDate>Thu, 23 Aug 1999 16:20:26 GMT</lastBuildDate><docs>http://www.blahblah.org/fm.cdf</docs>

RSS 2<image><title>freshmeat.net</title><url>http://freshmeat.net/images/fm.mini.jpg</url><link>http://freshmeat.net</link><width>88</width><height>31</height><description>This is the Freshmeat image stupid</description></image>

<item><title>kdbg 1.0beta2</title><link>http://www.freshmeat.net/news/1999/08/23/935449823.html</link><description>This is the Freshmeat image stupid</description></item>

<item><title>HTML-Tree 1.7</title><link>http://www.freshmeat.net/news/1999/08/23/935449856.html</link><description>This is the Freshmeat image stupid</description></item>

RSS 3<textinput><title>quick finder</title><description>Use the text input below to search freshmeat</description><name>query</name><link>http://core.freshmeat.net/search.php3</link></textinput>

</channel></rss>

RSS DTD<!ELEMENT rss (channel)><!ATTLIST rss

version CDATA #REQUIRED>

XML Application Tasksn Convert XML into application objectn Convert application object to XMLn Read or write XML into a databasen XSLT conversion to XHTML or HTML

RSS ObjectsLogical View

RSSChannel

+getTitle()+getDescription()+getLink()+getItems()+getImage()+getTextInput()

RSSItem

+getTitle()+getDescription()+getLink()

RSS Image

+getTitle()+getDescription()+getLink()+getURL()+getHeight()+getWidth()

RSSTextInput

+getTitle()+getDescription()+getLInk()+getName()

Parser API’sn What is the job of a parser API?

n Make the structure and contents of a document available

n Report errors

SAX APIn Event callback stylen Representation of elements & atts

n must build own stack

n Development modeln Reasonably openn xml-devn http://www.megginson.com/SAX/SAX2

Handler Strategiesn Single Monolithic

n Requires the handler to know entirely about the grammar

n One per application object and multiplexern Each application object has its own

DocumentHandlern The Multiplexer handler is registered with the

parsern Multiplexer is responsible for swapping SAX

handlers as the context changes.

SAX 1 RSS Handlerpublic void startElement(String name,AttributeList atts) throws SAXException {

currentText = new StringBuffer();textStack.push(currentText);if (name.equals("rss") {

elementStack.push(name);} else if (name.equals("channel")) {

elementStack.push(name);currentChannel = new RSSChannel();currentItems = new Vector();currentChannel.setItems(currentItems);currentImage = new RSSImage();currentChannel.setImage(currentImage);currentTextInput = new RSSTextInput();currentChannel.setTextInput(currentTextInput);currentChannel.setSkipHours(new Vector());currentChannel.setSkipDays(new Vector());

} else if (name.equals("item")) {elementStack.push(name);currentItem = new RSSItem();

} else if (name.equals("image")) {elementStack.push(name);

} else if (name.equals("textinput")) {elementStack.push(name);

} else if (name.equals("skipHours")) {elementStack.push(name);

} else if (name.equals("skipDays")) {elementStack.push(name);

} else {}

SAX 1 RSS Handler 2public void endElement(String name) throws SAXException {

String stackValue = (String) elementStack.peek();String text = ((StringBuffer)textStack.pop()).toString();if (name.equals("rss")) {

elementStack.pop();} else if (name.equals("channel")) {

elementStack.pop();} else if (name.equals("title")) {

if (stackValue.equals("channel")) {currentChannel.setTitle(text);

} else if (stackValue.equals("image")) {currentImage.setTitle(text);

} else if (stackValue.equals("item")) {currentItem.setTitle(text);

} else if (stackValue.equals("textinput")) {currentTextInput.setTitle(text);

} else {}

SAX 1 RSS Handler 3} else if (name.equals("description")) {

if (stackValue.equals("channel")) {currentChannel.setDescription(text);

} else if (stackValue.equals("image")) {currentImage.setDescription(text);

} else if (stackValue.equals("item")) {currentItem.setDescription(text);

} else if (stackValue.equals("textinput")) {currentTextInput.setDescription(text);

} } else if (name.equals("language")) {

currentChannel.setLanguage(text);} else if (name.equals("item")) {

currentItems.add(currentItem);elementStack.pop();

} else if (name.equals("textinput")) {currentChannel.setTextInput(currentTextInput);elementStack.pop();

SAX1 RSS Handler 4public void characters(char[] ch,int start,int length) throws SAXException {

currentText.append(ch, start, length);

SAX 1 RSS Driverpublic class SAX1RSSParse extends Object {

public static void main(String args[]) {Parser p = new SAXParser();DocumentHandler h = new SAX1RSSHandler();p.setDocumentHandler(h);try {

p.parse("file:///Work/ApacheCon/fm0.91_full.rdf");} catch (SAXException se) {

System.out.println("Error during parsing"+se.getMessage());se.printStackTrace();

} catch (IOException e) {System.out.println("I/O Error during parsing+e.getMessage());e.printStackTrace();

}System.out.println(((SAX1RSSHandler)h).getChannel().toString());

SAX 1 Basic Event Handlingn Basic Event Handling

n attribute handling in startElementn characters() for contentn character data handling in endElement

n Multiple characters calls

SAX 1 RSS ErrorHandlerpublic class SAX1RSSErrorHandler implements ErrorHandler {public void warning(SAXParseException ex) throws SAXException {

System.err.println("[Warning] ”+getLocationString(ex)+":”+ex.getMessage());

public void error(SAXParseException ex) throws SAXException {System.err.println("[Error] "+getLocationString(ex)+":”+

ex.getMessage());}

public void fatalError(SAXParseException ex) throwsSAXException {

System.err.println("[Fatal Error]"+getLocationString(ex)+":”+ex.getMessage());

throw ex;}

SAX1 RSS ErrorHandler 2private String getLocationString(SAXParseException ex) {

StringBuffer str = new StringBuffer();

String systemId = ex.getSystemId();

if (systemId != null) {

int index = systemId.lastIndexOf('/');

if (index != -1)

systemId = systemId.substring(index + 1);

str.append(systemId);

str.append(':');

str.append(ex.getLineNumber());

str.append(':');

str.append(ex.getColumnNumber());

return str.toString();

ErrorDriverpublic class SAX1RSSParse extends Object {

public static void main(String args[]) {Parser p = new SAXParser();DocumentHandler h = new SAX1RSSHandler();p.setDocumentHandler(h);ErrorHandler eh = new SAX1RSSErrorHandler();p.setErrorHandler(eh);try {

SAX 1 Error Handlingn Error Handling

n warningn error – parser may recovern fatalError – XML 1.0 spec errors

n Locators

Entity Handlingn <!DOCTYPE rss PUBLIC "-//Netscape Communications//DTD RSS91//EN"

"http://my.netscape.com/publish/formats/rss-0.91.dtd">

n This will do a network fetch!

public class SAX1RSSEntityHandler implements EntityResolver {public SAX1RSSEntityHandler() {}public InputSource resolveEntity(String publicId,String systemId) throws SAXException, IOException {if (publicId.equals("-//Netscape Communications//DTD RSS

91//EN")) {FileReader r = new FileReader("/usr/share/xml/rss91.dtd");return new InputSource(r);

}return null;

EntityDriverpublic class SAX1RSSParse extends Object {

public static void main(String args[]) {Parser p = new SAXParser();DocumentHandler h = new SAX1RSSHandler();p.setDocumentHandler(h);ErrorHandler eh = new SAX1RSSErrorHandler();p.setErrorHandler(eh);EntityResolver r = new SAX1RSSEntityResolver();p.setEntityResolver(r);try {

HandlerBase

n The kitchen sinkn Lets you do DocumentHandler,

ErrorHandler, EntityResolver and DTD Handler

n A good way to go if you are only overriding a small number of methods

SAX 2 Beta

n Configurablen DTDn ContentHandler replaces

DocumentHandler (NS Aware)n Attributes replaces AttributeList (NS

Aware)n XMLReader replaces Parsern XMLFiltern DefaultHandler replaces HandlerBase

SAX 2 Betan Optional LexicalHandler

n Entity boundariesn DTD bounariesn CDATA sectionsn Comments

n Optional DeclHandlern ElementDecls and AttributeDeclsn Internal and External Entity Declsn NO parameter entities

n Compatibility Adapters

SAX 2 Configurablen Uses strings (URI’s) as lookup keysn Features - booleans

n validationn namespaces

n Properties - Objectsn Extensibility by returning an object that

implements additional function

Xerces and Configurablen Standard way to set all parser

configuration settings.n Applies to both SAX and DOM Parsers

Xerces Featuresn Inclusion of general entitiesn Inclusion of parameter entitiesn Dynamic validationn Extra warningsn Allow use of java encoding namesn Lazy evaluating DOMn DOM EntityReference creationn Inclusion of igorable whitespace in DOM

Xerces Propertiesn Name of DOM implementation classn DOM node currently being parsed

DOM APIn Object representation of elements &

attsn Development Model

n W3C Recommendations and process

n http:///www.w3.org/DOM

DOM ArchitectureNode

+getNodeName()+getNodeValue()+getChildNodes()+getFirstChild()+getLastChild()+getNextSibling()+getPreviousSibling()+insertBefore()+appendChild()+replaceChild()+removeChild()

Document

Element

+getAttribute()+setAttribute()+removeAttribute()+getTagName()+getAttributeNode()+setAttributeNode()+removeAttributeNode()+normalize()

Text CDataSection

EntityReference Entity

ProcessingInstruction

Comment

DocumentType

NotationCharacterData

NodeList NamedNodeMap

DOM Instance (XML)

<?xml version="1.0" encoding="US-ASCII"?>

text in c

DOM Tree

Document

Elementa

Elementb

Elementc

Elementd

Text[lf]

Text[lf]text in c[lf]

Text[lf]

Textin b

DOM RSSpublic static void main(String args[]) {

DOMParser p = new DOMParser();

try {p.parse("file:///Work/ApacheCon/fm0.91_full.rdf");System.out.println(dom2Channel(p.getDocument()).toString());

} catch (SAXException se) {System.out.println("Error during parsing "+se.getMessage());se.printStackTrace();

} catch (IOException e) {System.out.println("I/O Error during parsing “+e.getMessage());e.printStackTrace();

DOM2Channelpublic static RSSChannel dom2Channel(Document d) {

RSSChannel c = new RSSChannel();Element channel = null;NodeList nl = d.getElementsByTagName("channel");channel = (Element) nl.item(0);for (Node n = channel.getFirstChild(); n != null;

n = n.getNextSibling()) {if (n.getNodeType() != Node.ELEMENT_NODE)continue;

Element e = (Element) n;e.normalize();String text = e.getFirstChild().getNodeValue();if (e.getTagName().equals("title")) {c.setTitle(text);

} else if (e.getTagName().equals("description")) {c.setDescription(text);

} else if (e.getTagName().equals("link")) {c.setLink(text);

} else if (e.getTagName().equals("language")) {

DOM2Channel 2…

}nl = channel.getElementsByTagName("item");c.setItems(dom2Items(nl));nl = channel.getElementsByTagName("image");c.setImage(dom2Image(nl.item(0)));nl = channel.getElementsByTagName("textinput");c.setTextInput(dom2TextInput(nl.item(0)));nl = channel.getElementsByTagName("skipHours");c.setSkipHours(dom2SkipHours(nl.item(0)));nl = channel.getElementsByTagName("skipDays");c.setSkipDays(dom2SkipDays(nl.item(0)));return c;

DOM Level 1n Attr

n API for result as objectsn API for result as strings

n No DTD supportn Need Import - copying between doms is

DOM Level 2n Core / Namespaces

n Import

n Traversaln Eventsn Rangen Stylesheets

Xerces DOM extensionsn Feature controls insertion of ignorable

whitespacen Feature controls creation of

EntityReference Nodesn User data object provided on

implementation class org.apache.xml.dom.NodeImpl

Xerces Lazy DOMn Delays object creation

n a node is accessed - its object is createdn all siblings are also created.

n Reduces time to complete parsing n Reduces memory usagen Good if you only access part of the

Tricky DOM stuffn item() based access

n use getNextSibling()

n Entity and EntityReference nodes –depends on parser settings

n Ignorable whitespacen Subclassing the DOM

n Use the Document factory class

DOM vs SAXn DOM

n creates a document object treen tree must the be reparsed & converted to

BO’sn Could subclass BO’s from DOM classes

n SAXn event handler based APIn stack managementn no intermediate data structures generated

Validationn When to use validationn non-validating != wf checking

n Not required to read external entities

n validation dial settings

Namespacesn Purpose

n Syntactic discrimination onlyn They don’t point to anythingn They don’t imply schema composition/combination

n Universal Namesn URI + Local Name

{http://www.mydomain.com}Elementn Declarations

n xmlns “attribute”

n Prefixesn Scopingn Validation and DTDs

Namespace Example<xsl:stylesheet version="1.0"

xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

xmlns:html="http://www.w3.org/TR/xhtml1/strict">

<xsl:template match="/">

<html:html>

<html:head>

<html:title>Title</html:title>

</html:head>

<html:body>

<xsl:apply-templates/>

</html:body>

</xsl:template>

</xsl:stylesheet>

Default Namespace Example<xsl:stylesheet version="1.0"

xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

xmlns="http://www.w3.org/TR/xhtml1/strict">

<xsl:template match="/">

<html>

<head>

<title>Title</title>

</head>

<body>

<xsl:apply-templates/>

</body>

</xsl:template>

</xsl:stylesheet>

SAX2 ContentHandlerpublic class NamespaceDriver extends DefaultHandler implements ContentHandler {

public void startPrefixMapping(String prefix,String uri) throws SAXException {

System.out.println("startMapping "+prefix+", uri = "+uri);

public void endPrefixMapping(String prefix) throws SAXException {

System.out.println("endMapping "+prefix);

public void startElement(String namespaceURI,String localName,String

rawName,Attributes atts) throws SAXException {

System.out.println("Start: nsURI="+namespaceURI+" localName= "+localName+"rawName = "+rawName);

printAttributes(atts);

public void endElement(String namespaceURI,String localName,String rawName) throws SAXException {

System.out.println("End: nsURI="+namespaceURI+" localName= "+localName+"rawName = "+rawName);

SAX2 ContentHandler 2public void printAttributes(Attributes atts) {

for (int i = 0; i < atts.getLength(); i++) {

System.out.println("\tAttribute "+i+" URI="+atts.getURI(i)+

" LocalName="+atts.getLocalName(i)+

" Type="+atts.getType(i)+" Value="+atts.getValue(i));

XML Scheman Richer grammar specificationn W3C Working Draftn Structures

n XML Instance Syntaxn Namespacesn Weird content models

n Datatypesn Lots

n Prototype Support in Xerces

RSS Schema<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE schema PUBLIC "-//W3C//DTD XMLSCHEMA 20000225//EN"

"http://www.w3.org/TR/2000/WD-xmlschema-1-20000225/structures.dtd"><xsd:schema

targetNamespace="http://my.netscape.com/publish/formats/rss-0.91"><xsd:element name="rss"><xsd:complexType>

<xsd:element ref="channel"/><xsd:attribute name="version" fixed="0.91" minOccurs="1"/>

</xsd:complexType></xsd:element>

RSS Schema 2<xsd:element name="channel"><xsd:complexType>

<xsd:element ref="title"/><xsd:element ref="description"/><xsd:element ref="link"/><xsd:element ref="language"/><xsd:element ref="item" minOccurs="1" maxOccurs="*"/><xsd:element ref="rating" minOccurs="0"/><xsd:element ref="image" minOccurs="0"/><xsd:element ref="textinput" minOccurs="0"/><xsd:element ref="copyright" minOccurs="0"/><xsd:element ref="pubDate" minOccurs="0"/><xsd:element ref="lastBuildDate" minOccurs="0"/><xsd:element ref="docs" minOccurs="0"/><xsd:element ref="managingEditor" minOccurs="0"/><xsd:element ref="webMaster" minOccurs="0"/><xsd:element ref="skipHours" minOccurs="0"/><xsd:element ref="skipDays" minOccurs="0"/>

</xsd:complexType></xsd:element>

RSS Schema 3...

<xsd:element name="name" type="xsd:string"/><xsd:element name="rating" type="xsd:string"/><xsd:element name="language" type="xsd:string"/><xsd:element name="width" type="xsd:integer"/><xsd:element name="height" type="xsd:integer"/><xsd:element name="copyright" type="xsd:string"/><xsd:element name="pubDate" type="xsd:date"/><xsd:element name="lastBuildDate" type="xsd:date"/><xsd:element name="docs" type="xsd:string"/><xsd:element name="managingEditor" type="xsd:string"/><xsd:element name="webMaster" type="xsd:string"/><xsd:element name="hour" type="xsd:time"/><xsd:element name="day" type="xsd:string"/>

</xsd:schema>

Grammar Accessn No DOM standardn SAX 2n Common Applications

n Editorsn Stub generators

n Also experimental API in Xerces

DTD Access Examplepublic class DTDDumper implements DeclHandler {

public void elementDecl (String name, String model) throws SAXException {

System.out.println("Element "+name+" has this content model: "+model);

public void attributeDecl (String eName, String aName, String type,

String valueDefault, String value) throwsSAXException {

System.out.print("Attribute "+aName+" on Element "+eName+" has type "+type);

if (valueDefault != null)

System.out.print(", it has a default type of "+valueDefault);

if (value != null)

System.out.print(", and a default value of "+value);

System.out.println();

DTD Access Example 2public void internalEntityDecl (String name, String value) throwsSAXException {

String peText = name.startsWith("%") ? "Parameter " : "";

System.out.println("Internal "+peText+"Entity "+name+" has value: "+value);

public void externalEntityDecl (String name, String publicId, StringsystemId) throws SAXException {

String peText = name.startsWith("%") ? "Parameter " : "";

System.out.print("External "+peText+"Entity "+name);

System.out.print("is available at "+systemId);

if (publicId != null)

System.out.print(" which is known as "+publicId);

System.out.println();

DTD Access Driverpublic static void main(String args[]) {

XMLReader r = new SAXParser();

r.setProperty("http://xml.org/sax/properties/declaration-handler",

new DTDDumper());

r.parse(args[0]);

} catch (Exception e) {

e.printStackTrace();

Round Trippingn Use SAX2 - DOM can’t do itn XML Decln Comments (SAX2)n PE Decls

“Serializing”n Several choices

n XMLSerializern TextSerializern HTMLSerializer

n XHTMLSerializer

n Work as either:n SAX DocumentHandlersn DOM serializers

Serializer Example

DOMParser p = new DOMParser();

p.parse("file:///twl/Work/ApacheCon/fm0.91_full.rdf");

Document d = p.getDocument();

// XML

OutputFormat format = new OutputFormat("xml", "UTF-8", true);

XMLSerializer serializer = new XMLSerializer(System.out, format);

serializer.serialize(d);

// XHTML

format = new OutputFormat("xhtml", "UTF-8", true);

serializer = new XHTMLSerializer(System.out, format);

serializer.serialize(d);

Entity Handlingn Entity / Catalog handling

n Catalog handling is mostly for document processing (SGML) applications

n Pluggable entity handling is important for server applications.n Provides access point to DBMS, LDAP, etc.

n predefined character entitiesn amp, lt, gt, apos, quot

Concurrencyn “Thread safety”

n Xerces allows multiple parser instances, one per thread

n Xerces does not allow multiple threads to enter a single parser instance.

n org.apache.xerces.framework.XMLParser#parseSome()

Grammar Designn Attributes vs Elements

n performance tradeoff vs programming tradeoff

n At most 1 attribute, many subelements

n ID & IDREFn NMTOKEN vs CDATAn CDATASECTIONsn Ignorable Whitespace

Grammar Design 2n Everything is Stringsn Data modelling

n Relationships as elementsn modelling inheritance

n Plan to use schema datatypesn avoid weird types

Performance Tipsn skip the DOMn reuse parser instances (code)

n reset() method

n defaulting is slown external anythings are slower, entities

are slowern only validate if you have to, validate

once for all timen use fast encoding (US-ASCII)

Xerces Architecturen Use SAX based infrastructure where

possible to avoid reinventing the wheeln Modular / Framework designn Prefer composition for system

components

Xerces Architecture Diagram

SAXParser DOMParser DOM Implementations

Error Handling XMLParser XMLDTDScanner DTDValidator

XMLDocumentScanner XSchemaValidator

Entity Handling

Readers

Things we don’t don HTML parsing (yet)

n java Tidy http://www3.sympatico.ca/ac.quick/jtidy.html

n Grammar caching (yet)n Parse several documents in a stream

ApacheCon 2000 Everything you ever wanted to know about XML Parsing

Technology

Shindig Apachecon Asia 09

Rave in Context @ ApacheCon 2011

Geode Meetup Apachecon

Kitty Apachecon 2010

Apache Olingo - ApacheCon Denver 2014

Query Parsing Tips & Tricks - ApacheCon

ApacheCon 2005

ApacheCon NA 2011 report

ApacheCon - Mentoring

Weighted Parsing, Probabilistic Parsing

FlexJS ApacheCon 2015

Contextual Dependency Injection for Apachecon 2010

Apachecon 2002 Struts

7 Deadly Hadoop Misconfigurations - ApacheCon

ApacheCon NA 2013 VFASTR

Apachecon cassandra transport

ApacheCon North Amaerica 2013 Presentation Templatearchive.apachecon.com/na2013/presentations/27... · Solr has nested query parsing capability, allowing for multiple query parsers

Apachecon Rails

Everything You Always Wanted to Know About Parsing Part V ...satta/tutorial/esslli13/part_5.pdfEverything You Always Wanted to Know About Parsing Part V : LR Parsing Giorgio Satta

Karaf ee-apachecon eu-2012